Some More Adventures in Swiftland

Last summer, I tried to learn new tricks and got some small success. Because the pace at which Swift 3 is changing finally settled a bit, I decided to finish what I set out to do and make SwiftyDB both up to date with the latest version and working on Linux.

The beginning

In August, SwiftyDB was working fine on MacOS but, while it compiled fine, it didn’t on Linux for a variety of reasons.

Swift was flimsy on that platform. The thing “worked”, but caused weird errors, had strange dependancies and was severely lacking stuff in the Foundation library. The version I had then crashed all the time, but for random and different reasons, so I decided to wait till it stabilized.

With Swift 3.0.2 and the announcement that Kitura was to become one of the pillars of the official built-in server apis (called it, by the way), I figured it was time to end the migration.

The problem

The main issue is that swift for Linux lacks basic things Foundation on the Mac has. I mean, it doesn’t even have NSDate’s timeIntervalSinceReferenceDate… But beyond that, the port lacks something that is truly important for the kind of framework that SwiftyDB is : introspection.

The typechecker is the absolute core of Swift. Don’t get me wrong, it’s great. It forces people to mind the type of the data they are manipulating, and throws errors early rather than late. But it comes at a cost : the compiler does all kinds of crazy operations to try to guess the type and too often for my tastes, fails miserably. If you’ve ever seen “IDE internal error” dialogs in XCode, that’s probably the reason.

But even if it worked perfectly, data manipulation that’s needed to get rows in and out of the db requires either to work with formless data (JSON) or to have a way to coerce and map types at run time. And boy, swift doesn’t like that at all.

So, SwiftyDB handles it in a hybrid way, passing along dictionaries of type [String : Any] and suchlikes. It’s kind of gross, to be honest, but that’s the only way this is going to work.

But that was kind of solved in August, albeit in a crash-prone way. What was the issue this time?

The swift team made huge strides to unify MacOS and Linux from an API point of view. If you read the doc, it “just works”, more or less. And that’s true, except for one tiny little thing : toll-free bridging.

Data type conversion

Swift, as ObjectiveC before it, deals with legacy stuff by having a toll-free bridging mechanism. Basically, to the compiler, NSString and String are interchangeable, and will use the definition (and the methods) it needs based on the line it’s at, rather than as a static type.

Something you surely know if you’ve done any kind of object oriented programming, typecasting is hard. If String inherits from NSString, I can use an object of the former type in any place I would have to use the latter. Think of the relationship between a square and a rectangle. The square is a rectangle, but the rectangle isn’t necessarily a square. It’s an asymmetrical relationship. And you can’t make it work by also having NSString inheriting from String, because that’s not allowed for a lot of complicated reasons, but with effects you can probably guess.

So, how does this toll-free bridging work? By cheating. But that’s neither here nor there. The fact is that it works just fine on MacOS, and not on Linux.

A solution

The easiest way to solve that is to have constructors in both classes that take the other as a parameter. And that’s the way it’s solved on Linux. True it’s a bit inelegant, and negates most of the “pure sexiness” that swift is supposed to have, but what are you gonna do? This, after all, is still a science.

Once those extensions are in place, as well as a few replacement additions to the stock Foundation (such as the infamous timeIntervalSinceReferenceDate), and a few tweaks to the way the system finds the SQLite headers, everything finally works.

Use it

As I said before it’s mostly an intellectual exercise, and a way to see if I could some day write some serverside stuff, but in the meantime it works just fine and you can find it here. Feel free to submit pull requests and stuff, but as it is, it works as intended : swift objects to sqlite storage and vice versa.

As usual, if you want to use it as a swift package, just use:

.Package(url: "", majorVersion: 1, minor: 2)


I Know People Call Me Crazy, But…

Some days in the life of a developer are a rabbit hole, especially rainy saturdays, for some reason. This following story will be utter gibberish to non developers or people who haven’t followed the news about Apple’s new language: Swift.

You have been warned and are still reading? Gosh, ok, I guess I have to deliver now. No story would be complete with a bit of background about the main protagonist: yours truly.

The story, till that fateful morning.

Despite being a mostly Apple oriented developer (it’s not exactly easy to change more than 15 years of habits), I freely admit I can be skeptical about some seemingly random changes the company makes. Swift can have lofty goals like being able to work for any task a programmer is given, or being learned super easily, it still has to deal with the grim reality that our development work is: understanding what an apparent human being has in its head, and translate it in a way a computer, which is not just apparently stupid, can understand and act as if it understood the original idea/concept/design, whatever youngsters call it these days.

It doesn’t help that Swift isn’t source compatible: any program written up to a month ago simply does not compile anymore. Yea. If there’s anything a computer is good at and humans suck at, it’s redoing the same task over and over again. I know I feel resentful when that happens.

For these reasons and more, I’ve held up switching to Swift as a primary language for my Apple related work. It doesn’t mean I don’t like it or can’t do it, just that I earn a living by writing code and making sure my customers don’t come back a month later with a little request that means I have to rewrite a bunch of code, because the language changed in between then and now.

The premise

I was excited about some things shown during WWDC2016, including the fact I may not have to learn javascript or upgrade my old and shabby knowledge of php to write decent backends for some projects I tend to refuse. IBM is hard at work on an open source and actually pretty decent Swift “Web Framework” (i.e. a server written in and extensible in Swift) named Kitura. I know there are alternatives, but Apple endorsed this one, and judging by the github activity, I have some faith in the longevity of the project.

Armed with my Contact List of Dinosaurus (+3 resist to Baffling), my Chair of Comfortness (-1 Agility, +3 Stamina), and my Giant Pot of Coffee (grants 3 Random Insights per day), I embarked on a quest that was probably meant for Lvl 9 Swift Coders rather that Lvl 10 Generalists, but it’s an adventure, right?

The beginning of the adventure, or the “But”

The aforementioned source instability means that the brave folks at IBM and their external contributors have to deal with rewrites of the API roughly every other week. Sometimes they are big, sometimes small, but it’s non trivial. At the time your truly embarks on the journey, Kitura only works on the June 6th snapshot of Swift. One notable thing about this is that it’s actually different than the one that ships with the betas Apple provided to developers for the WWDC. It requires installing a separate toolchain that, while it can work with Xcode is, to put it bluntly, a pain to work with. Special shell variables, switching Xcode to the new toolchain (which, incidentally, isn’t a project setting, but will apply to everything. You have to switch and restart the IDE every time).

After a solid hour of grinding through the process, the very simple Hello World sample finally loads in Safari, and our hero grins when the url http://localhost:8090/hello/Zino spit out

Hello, Zino!

Incidentally, it can also do the same thing using http://localhost:8090/hello/?name=Zino and its POST variant, because why the hell not, while I’m at it?

Whereupon, invigorated by the victory over the Guardian of the Door, we enter the rabbit hole proper and start making much longer headers

Once that particular beast is slain, I decide that experimenting with passing arguments through every possible means at my disposal is childish, and set my sights on middleware, and more specifically authentication. For those of you unfamiliar with the topic, good for you. It’s a mess that no amount of reading will make clearer. Encryption, storage, and tokens feature prominently, and the more you read about it the more you go ‘huh?’. Best practices are as varied as they are counter intuitive, and quite frankly, the reason why most websites out there either fail spectacularly at it or resign themselves to trusting third parties like facebook, goolge, or twitter, is because any way you look at it, it’s a tradeoff between security, sanity, and ease of use. And you can’t satisfy all three without going insane. It basically requires a complete rewrite about either how the web or humans work. And we know both are really hard to do.

Since the frameworks are in the middle of a transition to something that is already obsolete anyways, you can tell that some dependencies aren’t as much used as others. But let’s be clear on one thing: I do not blame anyone. It’s a stern chase, and a classic catch 22: why pour time and effort beyond a certain point since Swift will break everything again soon? Hopefully, now that source compatibility is on the agenda, we’ll be able to catch up.

But that’s not the point of the adventure, says the now tired Zino, it’s to gauge both my capacity to learn new tricks and my adaptability, so let’s pretend it actually serves a purpose! HTTP Basic and Digest are quickly worked out and I turn my sights to making my sample code actually do something realistic, like talking to a database. That’s what backends do, you know? They talk to stuff like PostgreSQL instances and construct responses to queries, isolating the base itself from prying eyes.

Medieval or primeval, call it what you wish

A not so quick trawling through the interwebs reveals almost nothing that deals with direct database access. There is simply no demand yet for it. Swift programs are currently user facing apps on iOS, and therefore do not need to interact with servers beyond happily chatting away with the kind of API I am currently trying to build. So, it starts looking like I will have to actually create a database driver to test my abilities. Quite honestly, I have been at it for a long time, if you look at the result. So writing a PDO-like framewok will have to wait. The only alternative that seems reasonably maintained works with CouchDB, and there is absolutely no frigging way I am going to “embrace the web” by ditching a performance-oriented piece of software for a javascript derivated “storage format”. Call me a snob, but no thank you.

Then I remember I have used successfully a SQLite wrapper in a project using Swift. SQLite may not be PostgreSQL, but at least it tries. The library in question is SwiftyDB, and I highly recommend you take a peek at it if you want to use a Swift class to SQLite (and back) that works as advertised. It doesn’t handle complex relationships yet, but for those who, like me, don’t want to have to deal with the idiosyncrasies of the SQL language and the various incompatibilities it entails, it’s quite a nifty piece of software.

So, thinks little me, all there is to do, is to port SwiftyDB to Swift 3 (moar XP!). Too easy, why not port it to the package manager Kitura and Apple use instead of bloody Cocoapods (don’t get me started on pods. It’s a good idea, a necessity, even, but implemented in a completely baffling and fragile way). Port it to the Future, kind of thing. Yea, it’s only 18:00, and it will be a good source of XP as well. From medieval Swift to the Modern Era!

Modern, my %€!&@#$~!

The Swift Package Manager is the Nth re-invention of something we’ve been doing since forever in every language, and has its own particular twists, because it’s new, and why not do things in a new way?

It relies on git. Heavily. The version of the library you’re using as a dependency is predicated on the git tags. It clones whatever URL you said the manager would find it at, looks for the version you want to use, based on the ubiquitous major, minor and build, numbers. If you want to have letters in there or a versionning system that uses some other scheme, you’re screwed. Yup. Well, that’s not a concession that would warrant much outrage, even if more than half of my versioning naming schemes don’t work like that. Never mind says I, I’ll bottle the cry for freedom, and get to it.

Did I mention that I had git issues earlier in the process? Apparently, git 2.9.0 has a bug that prevents the package manager from working correctly, and you can’t use dependencies that aren’t git based. That took a while to figure out. But figure it out I did, and everything’s dandy again.

So, let’s take a look at SwiftyDB. It depends on TinySQL (by the same guy), which depends on sqlite3. A handful of swift files in each of the two frameworks, and a lib that is installed on every mac out there. What could go wrong?

The sad story of the poor toolchain that was shoved aside

In order to use C functions (which sqlite is made of), you need to trick Swift in importing them. This is done through the use of a modulemap, which is basically a file that says “if the developer says they want to use this package, what they really mean is that you should include this C header, and link toward that library”. The file sqlite.h lives in /usr/include, so I start with that. Except it is incompatible with the Swift development snapshot for Reasons. The package build system requiring git tags and stuff for the dependencies, it’s kind of a long process to make a small modification and recompile, but after half an hour I manage to find the right combination of commands to have a working module map. Short story is: you need to use the .sdk header, not the system one. I went through gallons of coffee and potential carpal tunnel syndrome so that other adventurers don’t have to. You’re welcome.

Once that little (and awesomely frustrating) nugget has been dug up, all that’s left to do is port TinySQL, then SwiftyDB.

The package system is quirky. It requires a very specific structure and git shenanigans, but once you’ve got where everything should go, it’s just a matter of putting all the swift files in Sources and editing your Package.swift file carefully. It’s a shame that file isn’t more flexible (or expressive) but what are you going to do?

Now the Swift 2 to Swift 3 migration isn’t as simple and self explanatory as Chris lets on, let me tell you. Sure, the mandatory labels where they were optional before aren’t a big deal, but it throws the type inference system and the error checker into a fugue state that spouts gibberish such as “can’t find overloaded functor” of something or other, when I, who haven’t spent that much time writing the SwiftyDB code, can see there’s only one possible match.

Anyways, after much massaging and coaxing, and even cajoling, my SwiftyDB fork is finally swift-package-manager-compatible. With glazed eyes, I look up at the clock to see it’s now 21:00 and on a friggin saturday, too. Very few lines of actual code were involved. And that still was way longer than expected. At this point, the hero of the story wonders if he should go on with his life, or if he should at least try to make everything explode by incorporating SwiftyDB in the sample project that sits there, stupidly saying hello to everyone who type in the right URL…

Aren’t you done yet? I’m hungry!

Sorry sorry. The end was totally anti-climactic. The package integration worked first go. And SwiftyDB delivered without a hitch, I now have a database that holds stuff and a simple web-service that supports authentication and lets you store “notes” (aka blurbs of whatever text you want to throw at it) without so much as a hiccup.

But, as I’m sitting here, on the body of that slain foe, recounting my adventure for the folks who have no idea what the process of such an endeavor is, and intermittently looking at a boiling pot containing my well deserved dinner, I wonder if anyone will see what makes this an adventure: the doubts, the obstacles, the detours, and finally, hopefully, the victory. All that for something that isn’t even needed, or for all I know wanted, by anyone but a lone developer looking for an excuse to take on a challenge.

If, for reasons of your own, you want to use my work in your own package manager experiment, be it with or without Kitura, all you have to do is include this as a dependency:

.Package(url: "", majorVersion: 1, minor: 2)

Currently, it works with the same dependencies as Kitura (snapshot 06 06), and I may even update it along with it.