I Know People Call Me Crazy, But…

Some days in the life of a developer are a rabbit hole, especially rainy saturdays, for some reason. This following story will be utter gibberish to non developers or people who haven’t followed the news about Apple’s new language: Swift.

You have been warned and are still reading? Gosh, ok, I guess I have to deliver now. No story would be complete with a bit of background about the main protagonist: yours truly.

The story, till that fateful morning.

Despite being a mostly Apple oriented developer (it’s not exactly easy to change more than 15 years of habits), I freely admit I can be skeptical about some seemingly random changes the company makes. Swift can have lofty goals like being able to work for any task a programmer is given, or being learned super easily, it still has to deal with the grim reality that our development work is: understanding what an apparent human being has in its head, and translate it in a way a computer, which is not just apparently stupid, can understand and act as if it understood the original idea/concept/design, whatever youngsters call it these days.

It doesn’t help that Swift isn’t source compatible: any program written up to a month ago simply does not compile anymore. Yea. If there’s anything a computer is good at and humans suck at, it’s redoing the same task over and over again. I know I feel resentful when that happens.

For these reasons and more, I’ve held up switching to Swift as a primary language for my Apple related work. It doesn’t mean I don’t like it or can’t do it, just that I earn a living by writing code and making sure my customers don’t come back a month later with a little request that means I have to rewrite a bunch of code, because the language changed in between then and now.

The premise

I was excited about some things shown during WWDC2016, including the fact I may not have to learn javascript or upgrade my old and shabby knowledge of php to write decent backends for some projects I tend to refuse. IBM is hard at work on an open source and actually pretty decent Swift “Web Framework” (i.e. a server written in and extensible in Swift) named Kitura. I know there are alternatives, but Apple endorsed this one, and judging by the github activity, I have some faith in the longevity of the project.

Armed with my Contact List of Dinosaurus (+3 resist to Baffling), my Chair of Comfortness (-1 Agility, +3 Stamina), and my Giant Pot of Coffee (grants 3 Random Insights per day), I embarked on a quest that was probably meant for Lvl 9 Swift Coders rather that Lvl 10 Generalists, but it’s an adventure, right?

The beginning of the adventure, or the “But”

The aforementioned source instability means that the brave folks at IBM and their external contributors have to deal with rewrites of the API roughly every other week. Sometimes they are big, sometimes small, but it’s non trivial. At the time your truly embarks on the journey, Kitura only works on the June 6th snapshot of Swift. One notable thing about this is that it’s actually different than the one that ships with the betas Apple provided to developers for the WWDC. It requires installing a separate toolchain that, while it can work with Xcode is, to put it bluntly, a pain to work with. Special shell variables, switching Xcode to the new toolchain (which, incidentally, isn’t a project setting, but will apply to everything. You have to switch and restart the IDE every time).

After a solid hour of grinding through the process, the very simple Hello World sample finally loads in Safari, and our hero grins when the url http://localhost:8090/hello/Zino spit out

Hello, Zino!

Incidentally, it can also do the same thing using http://localhost:8090/hello/?name=Zino and its POST variant, because why the hell not, while I’m at it?

Whereupon, invigorated by the victory over the Guardian of the Door, we enter the rabbit hole proper and start making much longer headers

Once that particular beast is slain, I decide that experimenting with passing arguments through every possible means at my disposal is childish, and set my sights on middleware, and more specifically authentication. For those of you unfamiliar with the topic, good for you. It’s a mess that no amount of reading will make clearer. Encryption, storage, and tokens feature prominently, and the more you read about it the more you go ‘huh?’. Best practices are as varied as they are counter intuitive, and quite frankly, the reason why most websites out there either fail spectacularly at it or resign themselves to trusting third parties like facebook, goolge, or twitter, is because any way you look at it, it’s a tradeoff between security, sanity, and ease of use. And you can’t satisfy all three without going insane. It basically requires a complete rewrite about either how the web or humans work. And we know both are really hard to do.

Since the frameworks are in the middle of a transition to something that is already obsolete anyways, you can tell that some dependencies aren’t as much used as others. But let’s be clear on one thing: I do not blame anyone. It’s a stern chase, and a classic catch 22: why pour time and effort beyond a certain point since Swift will break everything again soon? Hopefully, now that source compatibility is on the agenda, we’ll be able to catch up.

But that’s not the point of the adventure, says the now tired Zino, it’s to gauge both my capacity to learn new tricks and my adaptability, so let’s pretend it actually serves a purpose! HTTP Basic and Digest are quickly worked out and I turn my sights to making my sample code actually do something realistic, like talking to a database. That’s what backends do, you know? They talk to stuff like PostgreSQL instances and construct responses to queries, isolating the base itself from prying eyes.

Medieval or primeval, call it what you wish

A not so quick trawling through the interwebs reveals almost nothing that deals with direct database access. There is simply no demand yet for it. Swift programs are currently user facing apps on iOS, and therefore do not need to interact with servers beyond happily chatting away with the kind of API I am currently trying to build. So, it starts looking like I will have to actually create a database driver to test my abilities. Quite honestly, I have been at it for a long time, if you look at the result. So writing a PDO-like framewok will have to wait. The only alternative that seems reasonably maintained works with CouchDB, and there is absolutely no frigging way I am going to “embrace the web” by ditching a performance-oriented piece of software for a javascript derivated “storage format”. Call me a snob, but no thank you.

Then I remember I have used successfully a SQLite wrapper in a project using Swift. SQLite may not be PostgreSQL, but at least it tries. The library in question is SwiftyDB, and I highly recommend you take a peek at it if you want to use a Swift class to SQLite (and back) that works as advertised. It doesn’t handle complex relationships yet, but for those who, like me, don’t want to have to deal with the idiosyncrasies of the SQL language and the various incompatibilities it entails, it’s quite a nifty piece of software.

So, thinks little me, all there is to do, is to port SwiftyDB to Swift 3 (moar XP!). Too easy, why not port it to the package manager Kitura and Apple use instead of bloody Cocoapods (don’t get me started on pods. It’s a good idea, a necessity, even, but implemented in a completely baffling and fragile way). Port it to the Future, kind of thing. Yea, it’s only 18:00, and it will be a good source of XP as well. From medieval Swift to the Modern Era!

Modern, my %€!&@#$~!

The Swift Package Manager is the Nth re-invention of something we’ve been doing since forever in every language, and has its own particular twists, because it’s new, and why not do things in a new way?

It relies on git. Heavily. The version of the library you’re using as a dependency is predicated on the git tags. It clones whatever URL you said the manager would find it at, looks for the version you want to use, based on the ubiquitous major, minor and build, numbers. If you want to have letters in there or a versionning system that uses some other scheme, you’re screwed. Yup. Well, that’s not a concession that would warrant much outrage, even if more than half of my versioning naming schemes don’t work like that. Never mind says I, I’ll bottle the cry for freedom, and get to it.

Did I mention that I had git issues earlier in the process? Apparently, git 2.9.0 has a bug that prevents the package manager from working correctly, and you can’t use dependencies that aren’t git based. That took a while to figure out. But figure it out I did, and everything’s dandy again.

So, let’s take a look at SwiftyDB. It depends on TinySQL (by the same guy), which depends on sqlite3. A handful of swift files in each of the two frameworks, and a lib that is installed on every mac out there. What could go wrong?

The sad story of the poor toolchain that was shoved aside

In order to use C functions (which sqlite is made of), you need to trick Swift in importing them. This is done through the use of a modulemap, which is basically a file that says “if the developer says they want to use this package, what they really mean is that you should include this C header, and link toward that library”. The file sqlite.h lives in /usr/include, so I start with that. Except it is incompatible with the Swift development snapshot for Reasons. The package build system requiring git tags and stuff for the dependencies, it’s kind of a long process to make a small modification and recompile, but after half an hour I manage to find the right combination of commands to have a working module map. Short story is: you need to use the .sdk header, not the system one. I went through gallons of coffee and potential carpal tunnel syndrome so that other adventurers don’t have to. You’re welcome.

Once that little (and awesomely frustrating) nugget has been dug up, all that’s left to do is port TinySQL, then SwiftyDB.

The package system is quirky. It requires a very specific structure and git shenanigans, but once you’ve got where everything should go, it’s just a matter of putting all the swift files in Sources and editing your Package.swift file carefully. It’s a shame that file isn’t more flexible (or expressive) but what are you going to do?

Now the Swift 2 to Swift 3 migration isn’t as simple and self explanatory as Chris lets on, let me tell you. Sure, the mandatory labels where they were optional before aren’t a big deal, but it throws the type inference system and the error checker into a fugue state that spouts gibberish such as “can’t find overloaded functor” of something or other, when I, who haven’t spent that much time writing the SwiftyDB code, can see there’s only one possible match.

Anyways, after much massaging and coaxing, and even cajoling, my SwiftyDB fork is finally swift-package-manager-compatible. With glazed eyes, I look up at the clock to see it’s now 21:00 and on a friggin saturday, too. Very few lines of actual code were involved. And that still was way longer than expected. At this point, the hero of the story wonders if he should go on with his life, or if he should at least try to make everything explode by incorporating SwiftyDB in the sample project that sits there, stupidly saying hello to everyone who type in the right URL…

Aren’t you done yet? I’m hungry!

Sorry sorry. The end was totally anti-climactic. The package integration worked first go. And SwiftyDB delivered without a hitch, I now have a database that holds stuff and a simple web-service that supports authentication and lets you store “notes” (aka blurbs of whatever text you want to throw at it) without so much as a hiccup.

But, as I’m sitting here, on the body of that slain foe, recounting my adventure for the folks who have no idea what the process of such an endeavor is, and intermittently looking at a boiling pot containing my well deserved dinner, I wonder if anyone will see what makes this an adventure: the doubts, the obstacles, the detours, and finally, hopefully, the victory. All that for something that isn’t even needed, or for all I know wanted, by anyone but a lone developer looking for an excuse to take on a challenge.

If, for reasons of your own, you want to use my work in your own package manager experiment, be it with or without Kitura, all you have to do is include this as a dependency:

.Package(url: "https://github.com/krugazor/swiftydb", majorVersion: 1, minor: 2)

Currently, it works with the same dependencies as Kitura (snapshot 06 06), and I may even update it along with it.

  

[Rant] Collaboration on Code

It so happens that we just found a bug that plagued us for 6 months, in the form of a commit that ignored completely the existing code in favor of a copy/paste from an internet source (I assume).

Now, let’s get one thing out first. I really really don’t mind having stack overflow code in my source. It has, over the years, established itself as a reliable way to fix thorny issues, or at least find pointers, and is probably todays numero uno origin of most code written by new programmers. That’s fine. Better than fine, it’s totally rad to have a community help students and people exploring new fields of our wonderful pitfall-laden world.

The second thing I want to point out is that I have worked freelance for 16 years. That should tell you my teamwork leaves a lot to be desired. I know that.

Now, every now and again, I still have to work with existing code, being updated while I work on the project. That’s fine too. However, successful collaboration, especially on code, relies heavily on communication, and prepwork.

So, without further ado, the Cardinal Rules of Team Code Editing (at least when I’m involved):

  • Thou shalt have a clearly defined “ownership” system. What parts of the code are you responsible for? What parts has someone else in charge?
  • Thou shalt have a clear way to communicate the changes thou art making. No, commit logs aren’t enough. At least not on their own.
  • Thou shalt always check thrice as much when changing code thou doth not have ownership thereof. If you “fix” something in someone else’s code, you’d better be pretty bloody sure you understood how the owner made that piece work.
  • Thou shalt notify the owner of the code of thy changes. Email, Slack, IM, Skype, I don’t care how.
  • Thou shalt make sure your changes art reversible. Possibly by having separate commits for each piece of the project that is owned by a different author.
  • Thou shalt not update critical pieces of thy project that are tied to a specific setup. Kind of an off the cuff unrelated item, but next time I have someone update the friggin project file with their own heavily personal paths, I’ll ‘git rm -rf’ everything. Be warned.

I know it sounds like a rant from a totally pissed lunatic, but beyond the anger, I actually believe these are sensible (and scalable, even if you have different layers of ownership) rules that would make everybody’s life better if thoroughly applied.

Feel free to add your own in the comments.

  

CoreData, iCloud, And “Failure”

CoreData is a very sensitive topic. Here and elsewhere, it’s a recurrent theme. Just last week I had a hair-pulling problem with it that was solved in a ridiculous manner. I’ll document it later for future reference.

This week, triggered by an article on the verge, the spotlight came once again on the difficulties of that technology, namely that it just doesn’t work with iCloud, which by all other accounts works just fine.

It is kind of frustrating (yet completely accurate) to hear from pundits and users that iCloud just works for them for most things, especially Apple’s own products, and that CoreData-based apps work unreliably, if at all. The perception of people not actually trying to make it work is that it’s somehow the developer’s fault for not supporting it. Hence this article on the verge, which highlights the fact that it’s not the developer’s fault. This is a good intent, but unfortunately doesn’t solve anything, since it kind of waggles the finger at Apple and doesn’t explain anything.

But what is the actual problem?

CoreData is a framework for storing an application’s data in an efficient (hopefully) and compact way. It was introduced in 2005 for a very simple purpose: stopping the developers from storing stuff on the user’s disk in “messy” ways. By giving access to a framework that would help keeping everything tidied up in a single (for the “messy” part) database (for the “efficient” part), Apple essentially said that CoreData was a solution to pretty much every storage ailment that plagued the applications: custom file formats that could be ugly and slow, the headache of having “relationships” between parts of documents that would end up mangled or inefficient, etc.

CoreData is a simplification of storage techniques maintained by Apple and therefore reliable, is the underlying tenet. And for the most part, it is reliable and efficient.

iCloud, on the other hand, is addressing another part of the storage problem : syncing. It is a service/framework meant to make the storage on every device a user owns kind of the same storage space. Meaning, if I create a file on device A, it is created on B and C as well. If I modify it on C, the modification is echoed on A and B without any user interaction. Behind the scene, the service keeps track of the modifications in the storage it’s responsible for, pushes them through the network, and based on the last modification date and some other factors, every device decides which files on disk to replace with the one “in the cloud”. The syncing problem is a hard one, because of all the fringe cases (what if I modified a file on my laptop, then closed it before it sent something, then made another modification on my iPad? Which version is the right one? can we mix them safely?), but for small and “atomic” files, it works well enough.

iCloud is a simplification of syncing techniques maintained by Apple, and therefore reliable, to keep the tune playing. And for the most part, it does work as advertised.

But when you mix the two, it doesn’t work.

When you take a look at the goals of the two technologies, you can see why it’s a hard problem to solve: CoreData aims at making a monolithic “store-it-all” file for coherence and efficiency purposes, while iCloud aims at keeping a bunch of files synchronized across multiple disks, merging them if necessary. These two goals, while not completely opposed, are at odds: ideally, iCloud should sync the difference between two files.

But with a database file, it’s hard. It’s never a couple of bytes that are modified, it’s the whole coherence tracking metadata, plus all the objects referenced by the actual modification. Basically, if you want to be sure, you’d have to upload and replace the whole database. Because, once again, the goal of CoreData is to be monolithic and self-contained.

The iCloud philosophy would call for incremental changes tracking to be efficient: the original database, then the modification sets, ideally in separate files. The system would then be able to sync “upwards” from any given state to the current one, by playing the sets one by one until it reaches the latest version.

As you can see, a compromise cannot be reached easily. A lot of expert developers I highly respect have imagined a number of ways to make CoreData+iCloud work. Most of them are good ideas. But are they compatible with Apple’s vision of what the user experience should be? Syncing huge files that have been partially modified isn’t a new problem. And it’s one none of my various version control systems have satisfactorily addressed. Most of them just upload the whole thing.

Just my $.02.

  

The Joy Of Dependencies

Whether on our favorite UNIX flavor, or on our projects, we are pretty much all dreading the time when we have to update the dependencies. What will be broken? How long will it take? How many platforms/users will we have to leave hanging?

We obviously depend on a number of libraries and pieces of code written in another time, for another project, and/or by somebody else. Whether it’s something big (like the OS itself — just read the forums for cries of despair), or small (OK, so, apparently, that lib takes UTF-8 strings instead of UTF-16 ones, now…), no dependency update is hassle-free and painless. None. Never.

Once upon a time, a long long time ago, in a city not that far away, I had to write tools for a small printing business. My major dependencies were the print driver (which drove a huge triple-rotor, 12 ink barrels, monstrosity), Quark XPress, QuickTime and MacOS (Classic, but I can’t recall the exact version number).

Once it was up and running, after a lot of sweating and swearing, they didn’t dare upgrade anything. If a customer came with a newer version of their XPress document, they just told them to export it again in a more compatible format. And that was it.

Nowadays, with users being informed constantly about updates (and not trained for it), it’s up to us developers to make sure we have the backward compatibility (what if a user upgrades our app, but not the system?), and the forward one (try to keep up with the general thrust of evolution and prepare the code for “easy” updates), and of course fixing the existing bugs.

This obviously adds a lot of overhead to our work, which is very hard to convey. “It’s the fault of that OS maker”, or “yeah but they broke it in that version of the lib”, is something a fellow developer might accept as an excuse, but a paying customer?

And we can’t blame them! How about if someone told you your car could only go on this freeway or in this town, but nowhere else, until they have upgraded the road? Especially if you don’t actually see the difference…

So we’re stuck between dependencies which evolve according to their own constraints and objectives, and users who legitimately want the thing they paid for to work. Not mentioning the apparently dreadful one-star-review (for me, it’s mostly people who don’t read the instructions, then complain about things that are stated somewhere, but hey…), it’s a reputation do-or-die. And our code is literally filling up with workarounds for this or that version of the OS, the libs we depend on.

Why do I go on about this anyway? Well three things: Yoann (@ygini on twitter) and I both had to upgrade our ports on a BSD, which is long and/or painful, because each lib depends on at least 3 others, and I tried to fix a few bugs in an old project of mine I will definitely release someday. Both of them took wayyyyyyyyyyy longer than any “regular user” would accept. We are talking weeks or days of down or crippled-service time here.

I am still looking for a way to make sure people depending on me make the difference between a lame excuse (“they changed something so it doesn’t work anymore”) and a good excuse (“they changed something so it doesn’t work anymore”), and will take any good advice on the topic.

  

Wall? What Wall?

The excellent Mike Lee (@bmf on twitter) has a hilarious way of handling theoretical problems: he ignores them to solve them.

In a case of life imitating art imitating life, programmer Mike Lee explained his writing a solution to the halting problem, with the simple explanation that, lacking a formal education in computer science, he didn’t realize it was considered unsolvable.

To solve the halting problem is to write a function that takes any function—including itself—to determine whether it will run forever, or eventually stop.

The classical approach is to wait and see if the function halts, but the necessity to accept itself as input means it will end up waiting for itself.

This paradox is what makes the problem unsolvable, but Lee’s function avoids the paradox by using a different approach entirely.

“It simply returns true,” Lee explained. “Of course it halts. Everything halts. No computer is a perpetual motion machine.”

That being said, the scientists vs engineers problem is an old one. Computer science started out as a branch of mathematics, and was treated as such for the longest time. When I was in college, we didn’t have any exam on an actual machine. It was all pen and paper!

Part of the problem of any major “it can’t be done” block on the road is the sacrosanct “it’s never been done before” or “such and such guys have said it can’t be done”. The truth, though, is that technology and knowledge make giant leaps forward these days, mostly because of people like Mike who just want to get things done.

Just remember that a few decades ago, multi-threading was science fiction. Nowadays, any programmer worth their salt can have a builtin “hang detector” to monitor if a part of their program is stuck in an infinite loop, or has exited abnormally. Hell, it’s hard to even buy a single-core machine!

I distinctly remember sitting in a theoretical computer science class, listening to a lesson on Gödel’s numbers. To oversimplify what I was hearing, the theorem was about how any program could be represented by a single number, however long. And about 5 minutes in, I was saying in my head “duh, it’s called compiling the program”. Had I said that out loud though, I’d probably have gotten in a lot of trouble.

Don’t get me wrong though, I think that mathematical analysis of computer programs is important and worthwhile. I’d like to be able to talk about optimization to a whole lot more people (how you just don’t use an O(n³) sorting algorithm, please…). But whereas I see it as a valuable tool to prove something positive, I stop listening whenever something is deemed impossible.

Trust Mike (and to a lesser extent me) on this: if something is impossible, it’s probably because the right tools haven’t been used yet. Maybe they don’t exist. And I’m ready to acknowledge that there is a probability they won’t exist any time soon. But “never” is a long time for anything to (not) happen.

UPDATE: it seems that people link it with the skeuomorphism ranting from before. True, it does ring familiar: we do things like we’ve always done, because we can’t do otherwise. Right?

  

[CoreData] Duplicating an object

As any of you knows, duplicating an object in coredata is just a nightmare : you basically have to start afresh each and every single time for each object, then iterate over attributes and relationships.

It so happens I have to do that often in one of my projects. I have to duplicate them except for a couple of attributes and relationships, and there are 20 of each on average (I didn’t come up with the model, OK?).

So, I came up with this code. Feel free to use it, just say hi in the comments, via mail, or any other way if you do!

@implementation NSManagedObject (Duplication)
+ (BOOL) duplicateAttributeValuesFrom:(NSManagedObject*)source To:(NSManagedObject*)dest ignoringKeys:(NSArray*)ignore {
    if(source == nil || dest == nil) return NO;
    if(![[source entity] isEqual:[dest entity]]) return NO;
 
    for(NSString *attribKey in [[[source entity] attributesByName] allKeys]) {
        if([ignore containsObject:attribKey]) continue;
 
        [dest setValue:[source valueForKey:attribKey] forKey:attribKey];
    }
 
    return YES;
}
 
+ (BOOL) duplicateRelationshipsFrom:(NSManagedObject*)source To:(NSManagedObject*)dest ignoringKeys:(NSArray*)ignore {
    if(source == nil || dest == nil) return NO;
    if(![[source entity] isEqual:[dest entity]]) return NO;
 
    NSDictionary *relationships = [[source entity] relationshipsByName];
    for(NSString *attribKey in [relationships allKeys]) {
        if([ignore containsObject:attribKey]) continue;
 
        if([((NSRelationshipDescription*)[relationships objectForKey:attribKey]) isToMany]) {
            [dest setValue:[NSSet setWithSet:[source valueForKey:attribKey]] forKey:attribKey];
 
        } else {
            [dest setValue:[source valueForKey:attribKey] forKey:attribKey];
        }
 
    }
 
    return YES;
}
 
@end