Web Services And Data Exchange

I may not write web code for a living (not much backend certainly and definitely no front-end stuff, as you can see around here), but interacting with webservices to the point of sometimes having to “fix” or “enhance” them? Often enough to have an Opinion.

There is a very strong divide between web development and more “traditional” heavy client/app development : most of the time, I tell people I write code for that these are two very distinct ways of looking at code in general, and user interaction in particular. I have strong reservations about the current way webapps are rendered and interacted with on my screen, but I cannot deny the visual and overall usage quality of some of them. When I look at what is involved in displaying that blog in my browser window, from the server resources that it takes to the hundreds of megabytes of RAM to run a couple paltry JS scripts in the window, the dinosaur that I am reels back in disgust, I won’t deny it.

But I also tell my students that you use the tool that’s right for the job, and I am not blind : it works, and it works well enough for the majority of the people out there.

I just happen to be performance-minded and nothing about the standard mysql-php-http-html-css-javascript standard pipeline of delivering stuff to my eyeballs is exactly doing that. Sure, individually, these nodes have come a long long way, but as soon as you start passing data along the chain, you stack up transformation and iteration penalties very quickly.

The point

It so happens that I wanted to do a prototype involving displaying isothermic-like areas on a map, completely dynamic, and based on roughly 10k points whenever you move the camera a bit in regards to the map you’re looking at.

Basically, 10 000 x 3 numbers (latitude, longitude, and temperature) would transit from a DB to a map on a cell phone every time you moved the map by a significant factor. The web rendering on the phone was quickly abandoned, as you can imagine. So web service it is.

Because I’m not a web developer, and fairly lazy to boot, I went with something that even I could manage writing in : Silex (I was briefly tempted by Kitura but it’s not yet ready for production when involved with huge databases).

Everyone told me since forever that SOAP (and XML) was too verbose and resource intensive to use. It’s true. I kinda like the built-in capability for data verification though. But never you mind, I went with JSON like everyone else.

JSON is kind of anathema to me. It represents everything someone who’s not a developer thinks data should look like :

  • there are 4 types that cover everything (dictionary, array, float, and string)
  • it’s human readable
  • it’s compact enough
  • it’s text

The 4 types thing, combined with the lack of metadata means that it’s not a big deal to any of the pieces in the chain to swap between 1, 1.000, “1”, and “1.000”, which, to a computer, is 3 very distinct types with hugely different behaviors.

But in practice, for my needs, it meant that my decimal numbers, like, say, a latitude of 48.8616138, gobbles up a magnificent 10 bytes of data, instead of 4 (8 if you’re using doubles). That’s only the start. Because of the structure of the JSON, you must have colons and commas and quotes and keys. So for 3 floats (12 bytes – 24 bytes for doubles), I must use :

{lat:48.8616138,lng:2.4442788,w:0.7653901}

That’s the shortest possible form – and not really human readable anymore when you have 10k of those -, and it takes 41 bytes. That’s almost four times as much.

Furthermore

Well, for starters, the (mostly correct) assumption that if you have a web browser currently capable of loading a URL, you probably have the necessary bandwidth to load the thing – or at least that the user will understand page load times – fails miserably on a mobile app, where you have battery and cell coverage issues to deal with.

But even putting that aside, the JSON decoding of such a large datasets was using 35% of my cpu cycles. Four times the size, plus a 35% performance penalty?

Most people who write webservices don’t have datasets large enough to really understand the cost of transcoding data. The server has a 4×2.8Ghz CPU with gazillions of bytes in RAM, and it doesn’t really impact them, unless they do specific tests.

At this point, I was longingly looking at my old way of running CGI stuff in C when I discovered the pack() function in PHP. Yep. Binary packing. “Normal” sized data.

Conclusion

Because of the large datasets and somewhat complex queries I run, I use PostgreSQL rather than MySQL or its infinite variants. It’s not as hip, but it’s rock-solid and predictable. I get my data in binary form. And my app now runs at twice the speed, without any optimization on the mobile app side (yet).

It’s not that JSON and other text formats are bad in themselves, just that they use a ton more resources than I’m ready to spend on “just” getting 30k numbers. For the other endpoints (like authentication and submitting new data), where performance isn’t really critical, they make sense. As much sense as possible given their natural handicap anyways.

But using the right tool for the right job means it goes both ways. I am totally willing to simplify backend-development and make it more easily maintainable. But computers work the same way they have always done. Having 8 layers of interpretation between your code and the CPU may be acceptable sometimes but remember that the olden ways of doing computer stuff, in binary, hex, etc, also provide a way to fairly easily improve performance : less layers, less transcoding, more cpu cycles for things that actually matter to your users.

  

Bluetooth, or the new old technology

Connected devices are the new thing. Thanks to the growing support of BTLE, we now have thingies that can perform a variety of tasks, and take commands or give out information to a smartphone.

When I started a project that relies on BTLE, my first instinct was to take a look at the documentation for the relevant frameworks and libraries, both on iOS and Android. According to these documentation files, everything is dandy… There’s a couple functions to handle scanning, a few functions/methods/callbacks that manage the (dis)connections, and some read/write functions. Simple and peachy!

Well… Yes and no. Because of its history, BTLE has a few structural quirks, and because of its radio components, a few pitfalls.

The very first thing you have to understand when doing hardware realted projects is that there a huge number of new (compared to pure software development) things that can go wrong and have to be taken into account: does it matter if the device disconnected in the middle of an operation? If so, what kind of power management strategy should you employ? The failsafe mechanisms in general become highly relevant, since we’re not in a tidy sandbox anymore. Also, hardware developers have a different culture and a different set of priorities, so an adjustment might be necessary.

Bluetooth is an old serial-over-radio thing

Yes. Not only is it radio, with all its transmission potential hiccups, it also is a 20 years old technology, made to have a wireless RS-232 like protocol. Its underpinnings are therefore way less network-ish than most remote-toting APIs. There’s very little in the way of control mechanisms, and no guarantee the packets will arrive in a complete and orderly fashion.

As far as I can understand it, it’s kind of like AT commands on a modem, and therefore prone to errors.

On top of it, BTLE adds a “smart” layer (smart being the name, not a personal opinion), which as a very singular purpose: syncing states.

Again, I see it from the vantage point of having a few successful RC projects under my belt, not an expert in the BTLE stack of the OS or the device. But as far as I can see, BTLE is a server over serial connection that exposes a hierarchy of storage units. These storage units have contents, and the structure of the storage, as well as the bytes it contains are periodically synced with the ghost copy in the other end of the link.

So, a device handles its Bluetooth connection as usual (pairing, bonding, etc), then exposes a list of services (top level storage containers of the tree, akin to groups), each of which containing characteristics (a storage space with a few properties, a size, and a content).

In theory, once you’re connected to the device, all you have to do is handle the connection, and read/write bytes (yep, bytes… No high level thing here) in characteristics, for Things to happen.

As you can see, having a good line of communication with the hardware developers is a necessity: everything being bytes means being sure about the format of the data, and knowing which characteristics require a protocol of their own, as well as which writes will trigger a reboot.

All in all, provided you are methodical and open minded enough, it can be fun to figure out a way to copy megabytes of data using two characteristics that can handle 16 bytes at the most. Welcome back to the dawn of serial protocols!

BTLE is all about data sync

Since the device is likely to be a state machine, most of the APIs mirror that: the overall connection has disconnected, connecting, connected, and disconnecting states, and synchronizing the copy in-system of the data with the copy in-device is highly unsynchronous. Not only that, but there’s no guarantee as to order in which they are transmitted, received, or timing thereof. You are a blind man playing with the echo, here.

If you want to transmit long structured data, you have to have a protocol of your own on top of BTLE. This includes, but is not restricted to, in-device offset management, self correcting algorithms, replayable modularity, etc etc.

Not to mention that more often than not the OS tells you you are disconnected after an inordinate amount of time, sometimes never. So the constant monitoring of the connection state is also paramount.

Last but not least, background execution of said communication is patchy at best. Don’t go into that kind of development expecting an easy slamdunk, because that is the very best way to set some customer’s house on fire, as the thingie you are managing was stuck on “very hot” in the middle of a string of commands and you never realized this was the case.

The software side, where it’s supposed to come together

Let’s imagine you mastered the communication completely, and know for sure the state and values of the device at any given time. First, let me congatulate you. Applaud non ironically, even.

Conveying the relative slowness of serial communications in the age of high speed communication in a manner the user will a/ accept and b/ understand is not a small feat whatsoever. It takes up to 4 minutes for us to push 300k of data in mega-slow-but-safe mode. Reading is usually 10 times faster, but we are still talking orders of magnitudes above what’s considered normal on these devices for these sizes.

One trick is to make the data look big: plenty of different ways to visualize the same 12 entries, another is to make it look like the app is waiting for a lot of small data points to compute a complex looking graph. All in all, it’s about making the user feel at home with the comparatively large “doing nothing visible” moments.

What I am hoping for the (very near) future

Bluetooth has always felt kind of wonky and temperamental. It actually has a very stable radio side, the problem largely lays in the total absence of the kind of control structures protocols like TCP uses to recover from errors, or slow and changing connections. At its core, the whole system seems to be built on the assumption that “once the connection is established, everything will be alright”. A lot of effort has therefore been put on discovery and power management issues, rather than any kind of self-correcting way to talk. It is a very complex system intended to establish a serial connection, then a level of abstraction on top of it in the form of a “server” that organizes the data in a somewhat discreete fashion. And that’s it.

If changing the protocol is too hard, and I’m totally ready to assume that it is, then the API providers need to figure out a way to manage the various states in a way that represents way more the reality of the transmission. Otherwise it’s a permanent struggle to circumvent the system or to coax it in doing what you know is what should happen.

  

The Bane Of Reality

Fiction is not enough. Apparently the masses want reality. The superheroes and master spies have to be explained and “fit in” the real world (by the way, thanks to the people who did Iron Sky, it was a breather of absurdity and laughter).

In software terms, it gave us (rolling drums) skeuomorphism, the art to mimick real objects to help us poor humans deal with apparently complex functions.

Last one to date, the podcast application from Apple, and it looks like a tape deck. Seriously. Man, I mastered the art of obscure VCR controls a long time ago… And now you want to simplify my life by analoguing a defunct technology?

Don’t get me wrong, I really think interfaces should be thought about and self-explanatory, but really? Who uses a binder these days? So what’s the point of the spirals on the left of your writing interface? I’ve actually never used an agenda that I recall, so why give me that faux-leather look?

Some ideas are not based in the real world, but they quickly become THE way to do it, like pull-for-refresh, for instance, or pinch to zoom in and out. What’s the real world equivalent of those? Do we need any equivalent?

I guess I’m not a natural target for software anyway: when I take a look at a program, I want to know what it does for me. Let’s say I want an app that gives me remote control of my coffee maker. That way, I’m heading back home after a tiring day, and I want a coffee that’s strong (more coffee in it) and has been finished 5 minutes before I get home (because coffee has to cool down a little bit). Do I want to drag and drop the number of spoons from one half of the screen to the next to simulate the amount to pour in? Do I want the same kind of clumsy timers-with-arrows that exists already on these machines? Nope.

But I want to know if the coffee maker can make me coffee (because it’s all washed up and ready to go), the amount of coffee left in the reservoir, as well as the water level, I want to set up the amount in as little movements as possible while being totally reliable and I want to be able to just say “ready 5 minutes before I’m in” and let the location manager deal with it (one man can dream, right?)

There is a history behind physical controls. Some designers, ergonomists, and engineers took the time to fine tune them for daily use (with mixed results), and the ones that stayed with us for 20 years or more stayed because people “got them”, not because they liked them or thought it was a good analogy to whatever they were using before. Thank goodness, we’re not driving cars with reins-analogues, or bicycle-horns-analogues.

It’s time to do the same with software. Until we have 3d-manipulation interfaces, we’re stuck in Flatland. And that means that any control that was built for grabbing with multiple fingers at several depths, is out (you hear me rotating dial analogue?).

If you want your users to feel comfortable with your software, make sure the function of it is clear to the intended audience. Then prettify it with the help of a computer designer. Different world, different job.

  

More Isn’t Better

The high tech world we live in generates some really high tech expectations.

Whether it’s needed or not, we see countless “features” and “upgrades” being thrown at us, causing for the most part more confusion than anything else. If there is a lesson the humongous sales of iPads hasn’t taught our beloved decision makers, it’s that sometimes simpler is better than more.

Most of the time, a subtle nudge works better than a huge wink, and thankfully, some designers took the hint.

But it seems that flashy now means modern in some people’s minds. Case in point: a very good friend of mine was put in charge of creating from scratch a website for his company. The company deals in service for professionals (read “something most of you, and myself, will never need”). Their business is all mouth-to-ear anyway, so the website isn’t really needed, it’s mostly so that people could look them up if needed.

He worked with some friends of his, very good at designing websites in their own right, and came up with something Apple-y. Clear and concise, no-nonsense, but clearly not really funny either.

This idea got rejected immediately. “Where are all the animations?” and “can we add a little more bang to it, to show that, you know, we’re modern and all?” seemed to be the major reason for rejection.

Now, picture this: you are tasked by your company to find a suitable service provider. You ask a few colleagues and/or friends from the business, you look companies up, and you come up with two possible candidates.

You go on one’s website, it’s clean but contains little except for a list of current customers, and contact information, maybe with a little side of demo/pr.

On the other’s website, you see animations everywhere, it takes a good couple of minutes for everything to settle down, and maybe take you to the place you were looking for: contact and price information.

In all honestly, which is most likely to annoy?

This is something that, as a developer who knows I suck at design, I have to face on a regular basis. For a project, I would get only screens, and not a word about navigation. A beta I would offer would get criticized at length because “the cool flipover double axel animation thingie is not in, yet”. I would have detailed sketches as to wooshing sound effects and glow-in-the-dark animations, but when I ask “ok, but once you are on that screen, you’re stuck, and have no way to go back, right?”, I would get looked at as if I had rabies.

Every once in a while, I have the chance of working with designers who actually think all of this through carefully. And man, does it feel great to have someone who can sometimes say “you know what, I honestly didn’t think that case would present itself, but now that I see it, I’ll think about how to deal with it”, and do so. And as a user, I even agree with the final decision. Talk about sweet. That was the case on that huge Java project I was working on, and given the scope of the project (think small OS), it was a very welcome change in the type of people I sometimes have to deal with.

In my mind, usability should come first, graphics second. This is why for a long time, Linux, while vastly superior in many ways on the technical level to its competitors, could not gain a foothold in the desktop business: unusable by my granny. That’s why some really really cool projects (from a geek perspective) such as automated households don’t really appeal to most people: they know how to use a dial and a button, and fidgeting with an LCD display and a keyboard seems over-complex. Even if in the end, they won’t have to touch the thing ever again.

If you are thinking of a new and wonderful project, think about 3 major factors before handing the making to somebody:
– the user needs to find what he/she is looking for in less than 20s, or at least understand how to get there in that time frame. (depth)
– do at least a rough storyboard of the navigation. Where do you start? Where can you go from there? Should you be able to go back, or forward only? Repeat. (width)
– “animations” is cool. But only if it highlights a feature you want to bring forward, draws the attention towards it, never away from it.

Now, I’m only a developer. I have no track record in design or graphics. But after a decade of writing code, I start to get a sense of what the user wants. And if a developer can, you can too.

  

Sectar Wars II : Revenge Of The Lingo

Every so often, you get a tremor of a starting troll whenever you express either congratulations or displeasure at a specific SDK, language, or platform.

Back in the days where people developing for Apple’s platforms were very very few (yea, I know, it was all a misunderstanding), I would get scorned at for not having the wondrous MFC classes and Visual Basic and the other “better” and “easier” ways of having an application made.  You simply couldn’t do anything remotely as good as the Windows equivalent, because, face it, Mac OS was a “closed system”, with a very poor toolbox, and so few potential users. But hey, I was working in print and video, and MacOS had the best users in both fields at the time. And the wonders of QuickTime… sigh

Then it would be a ProjectBuilder versus Codewarrior (I still miss that IDE every now and then…). Choosing the latter was stupid: it was expensive, with minimal support for NIBs, was sooooooo Carbon,… But it also had a great debugger, a vastly superior compiler, and could deal with humongous files just fine on my puny iBook clamshell…

Once everyone started jumping on the iOS bandwagon, it was stupid to continue developing for Mac.

Every few months, it’s ridiculous to develop in Java.

There seems to be something missing for the arguments of every single one of these trolls: experience.

Choosing a set of tools for a task is a delicate thing. Get the wrong language, IDE, library, for a project and you will end up working 20 times as more for the same result. Granted, you can always find a ton of good examples why this particular choice of yours at that moment in time is not ideal. But that doesn’t mean it’s not good in general.

“QuickTime is dead”, but it’s still everywhere in the Mac OS. “Java is slow” is the most recurrent one. Well for my last project I reimplemented the “Spaces” feature in Java. Completely. Cross platformly. And at a decent speed. I’d say that’s proof enough that, when someone puts some care and craft in his/her work, any tool will do.

It all boils down to experience: with your skillset, can you make something good with the tools at your disposal? If the answer is yes, does it matter which tools you use? Let the trolls rant. The fact that they can’t do something good with this or that platform/tool doesn’t mean no one can.

  

Time? Time? Who’s Got The Time?

As I sat down at my desk, with the clear objective to update WordPress here, I had a few revelations:

1/ I can’t update WordPress. The  PHP version on that server is too old. Half an hour wasted. And each time I try doing something with this corner of the World Wide Web, I realize it’s been almost ten years I have decided to do something worthwhile with it, and never quite found time enough to do so.

Of course, I want to project a decent image as a technophile (which I am), a cool guy (which I may be), and a somewhat successful business guy (results may vary). I also obviously want to be able to disclose information, in long form (as I’ve done here for a while, with sporadic energy), or in short form (mostly via Twitter). Combining that with information about what I could do for potential customers, what I have done for low-and-high-profile customers is something you have to do in the current Web2.0 bonanza, and something most companies, and even indies, have no trouble doing.

But I find myself facing a conundrum: I want complete control (so I need to be able to edit the pages, the CRM, and whatever else might be on the site), and I am utterly incompetent at deciding what’s best.

This is why, every time I decide to spend half a day looking around for technical solutions to my “needs”, I end up lost in unfamiliar territory. There are 3 dozens CRM systems, all requiring a boatload of dependancies. There is a plethora of CSS and whatnot design styles, none of which I could fine-tune if my life depended on it (and it doesn’t).

Sure, there are services that would do it all for me, but it would be worse, to my mind: I wouldn’t understand how it does what it does, which to any self-styled developer feels extremely weird.

At least I understand the underpinnings of wordpress pretty well. I can (and have) modified it to suit my needs. But now that this PHP requirement prevents me from upgrading, I will have to do another weird “let me play!” session. And the end result will probably be the same in the end, which is a little depressing.

Over the years, a bunch of people asked me why I had such a reluctance with web interface, and truth be told, I don’t know for sure. That might be the topic of another post sometime in the future.

Which brings us to…

2/ I have 5 blog drafts I have started and never finished. That’s right. 5 pieces of 500-1000 words I have written but was never satisfied of enough to hit the publish button.

Some geeky analysis of a book, a piece of reusable code I’m proud of, and a somewhat heartfelt eulogy that have been sitting in my (virtual) out-tray for 6 months.

I guess that speaks volumes about a definite flaw: When it concerns only me, I am not a closing kind of guy. Making prototypes and brainstorming commando-style? pschhhhhh easy. Spending 4h navigating ARM-assembly to find a bug that prevents some obscure piece of software from working? stroll in the park. Spending the necessary 4h for a personal project of mine to be finished and released? Apparently can’t be done… If you guys have any motivational tips and tricks for that, I’m all ears.

3/ However…

I am currently working on 2 projects that I really really want to see through. Unfortunately, they are time consuming, at roughly the same level as my paid gigs. That means every hour I’m working on these pet projects, I’m stealing from a customer. And given that I’m at ~120% capacity due mostly to “not enough pressuring the customers to give me what I need to move forward”, these stolen hours are just really hard to pull off.

But I’m still announcing that these two projects (an iOS application that interfaces with RT for bug/incident tracking, and a podcast with Dam and Marc) will see the light of day before the end of Q1-2012. Because we all know the end of the world at the end of aliens who think we are pathetic comes around the 2012 equinox or somesuch.

Happy holidays to everyone!