First part of doing RNN text prediction with TensorfFlow, in Swift

For all intents and purposes, it's about statistics. The question we are trying to solve is either something along the lines of "given an input X, what is the most probable Y?", or along the lines of "given an input X, what is the probability of having Y?"

Of course, simple probability problems have somewhat simple solutions: if you take a game of chess and ask for a next move based on the current board, you can do all the possible moves and sort them based on the probability of having a piece taken off the board, for instance. If you are designing an autopilot of some kind, you have an "ideal" attitude (collection of yaw, pitch and roll angles), and you calculate the movements of the stick and pedals that will most likely get you closer to that objective. If your last shot went left of the target, chances are, you should go right. Etc etc etc.

But the most interesting problems don't have obvious causality. If you have pasta, tomatoes and ground meat in your shopping basket, maybe your next item will be onions, because you're making some kind of bolognese, maybe it will be soap, because that's what you need, maybe it will be milk, because that's the order of the shelves you are passing by.

Machine learning is about taking a whole bunch of hopefully consistent data (even if you don't know for sure that it's consistent), and use it to say "based on this past data, the probabilities for onions, soap and milk are X, Y, and Z, and therefore the most probable is onions.

The data your are basing your predictive model on is really really important. Maybe your next item is consistent with the layout of the shop. Maybe it is consistent with what other customers got. Maybe it's consistent to your particular habits. Maybe it's consistent with people who are in the same "category" as you (your friends, your socio-economic peers, your cultural peers, ... pick one or more).

So you want a lot of data to have a good prediction, but not all the data, because noise (random data) does not reveal a bias (people in the shop tend to visit the shelves in that order) or doesn't exploit a bias (people following receipes want those ingredients together).

Oh yes, there are biases. Lots of them. Because ML uses past data to predict the future, if the data we use was based on bad practices, recommendations won't be a lot better.

There is a branch of machine learning that starts ex nihilo but it is beyond the scope of this introduction, and generates data based on a tournament rather than on actual facts. Its principle is roughly the same, though.

So, to recap:

• We start with a model with random probabilities, and a series of "truths" ( X leads to Y )
• We try with a Z, see what the model predicts
• We compare it to a truth, and fix the probabilities a little bit so that it matches
• Repeat with as many Zs as possible to be fairly confident the prediction isn't too far off

If you didn't before, now you know why ML takes a lot of resources. Depending on the number of possible Xs, Ys and the number of truths, the probability matrix is potentially humongous. Operations (like fixing the probabilities to match reality) on such enormous structures aren't cheap.

If you want a more detailed oriented explanation, with maths and diagrams, you can read my other attempt at explaining how it works.

#### Swift TensorFlow

There are a few contenders in the field of "ML SDK", but one of the most known is TensorFlow, backed by Google. It also happens to have a Swift variant (almost every other ML environment out there is either Python or R).

And of course, the documentation is really... lacking, making this article more useful than average along the way.

In their "Why Swift?" motivation piece, the architects make a good case, if a little bit technical, as to why swift makes a good candidate for ML.

The two major takeaways you have to know going in are:

• It's a different build of Swift. You cannot use the one that shipped with Xcode (yet)
• It uses a lot of Python interoperability to work, so some ways of doing things will be a bit alien

The performance is rather good, comparable or better than the regular Python TensorFlow for the tasks I threw at it, so there's that.

But the documentation... My oh my.

Let's take an example: Tensor is, as the name of the framework implies, the central feature of the system. Its documentation is here: https://www.tensorflow.org/swift/api_docs/Structs/Tensor

Sometimes, that page greets me in Greek... But hey, why not. There is little to no way to navigate the hierarchy, other than going on the left side, opening the section (good luck if you don't already know if it's a class, a protocol or a struct you're looking for), and if you use the search field, it will return pages about... the Python equivalents of the functions you're looking for.

Clearly, this is early in the game, and you are assumed to know how regular TensorFlow works before attempting to do anything with STF.

But fret not! I will hold your hand so that you don't need to look at the doc too much.

The tutorials are well written, but don't go very far at all. Oh and if you use that triple Dense layers on more than a toy problem (flower classification that is based on numbers), your RAM will fill so fast that your computer will have to be rebooted. More on that later.

And, because the center of ML is that "nudge" towards a better probability matrix (also called a Tensor), there is the whole @differentiable thing. We will talk about it later.

A good thing is that Python examples (there are thousands of ML tutorials in Python) work almost out of the box, thanks to the interop.

#### Data Preparation

Which Learning will my Machine do?

I have always thought that text generation was such a funny toy example (if a little bit scary when you think about some of the applications): teach the machine to speak like Shakespeare, and watch it spit some play at you. It's also easy for us to evaluate in terms of what it does and how successful it is. And the model makes sense, which helps when writing a piece on how ML works.

A usual way of doing that is using trigrams. We all know than predicting the next word after a single word is super hard. And our brains tend to be able to predict the last word of a sentence with ease. So, a common way of teaching the machine is to have it look at 3 words to predict a 4th.

I am hungry -> because, the birds flew -> away, etc

Of course, for more accurate results, you can extend the number of words in the input, but it means you must have a lot more varied sentence examples.

What we need to do here is assign numbers to these words (because everything is numbers in computers) so that we have a problem like "guess the function f if f(231,444,12)->123, f(111,2,671)->222", which neural networks are pretty good at.

So we need data (a corpus), and we need to split it into (trigram)->result

Now, because we are ultimately dealing with probabilities and rounding, we need the input to be in Float, so that the operations can wriggle the matrices by fractions, and we need the result to be an Int, because we don't want something like "the result is the word between 'hungry' and 'dumb'".

The features (also called input) and the labels (also called outputs) have to be stored in two tensors (also called matrices), matching the data we want to train our model on.

That's where RAM and processing time enter the arena: the size of the matrix is going to be huge:

• Let's say the book I chose to teach it English has 11148 words in it (it's Tacitus' Germany), that's 11148*3-2 trigrams (33442 lines in my matrices, 4 columns total)
• The way neural networks function, you basically have a function parameter per neuron that gets nudged at each iteration. In this example, I use two 512 parameters for somewhat decent results. That means 2 additional matrices of size 33442*512.
• And operations regularly duplicate these matrices, if only for a short period of time, so yea, that's a lot of RAM and processing power.

Here is the function that downloads a piece of text, and separates it into words:

func loadData(_ url: URL) -> [String] {
let sem = DispatchSemaphore(value: 0)
var result = [String]()

let session = URLSession(configuration: URLSessionConfiguration.default)
//     let set = CharacterSet.punctuationCharacters.union(CharacterSet.whitespacesAndNewlines)
let set = CharacterSet.whitespacesAndNewlines
session.dataTask(with: url, completionHandler: { data, response, error in
if let data = data, let text = String(data: data, encoding: .utf8) {
let comps = text.components(separatedBy: set).compactMap { (w) -> String? in
// separate punctuation from the rest
if w.count == 0 { return nil }
else { return w }
}
result += comps
}

sem.signal()
}).resume()

sem.wait()
return result
}

Please note two things: I make it synchronous (I want to wait for the result), and I chose to include word and word, separately. You can keep only the words by switching the commented lines, but I find that the output is more interesting with punctuation than without.

Now, we need to setup the word->int and int->word transformations. Because we don't want to look at all the array of words every time we want to search for one, there is a dictionary based on the hashing of the words that will deal with the first, and because the most common words have better chances to pop up, the array for the vocabulary is sorted. It's not optimal, probably, but it helps makes things clear, and is fast enough.

func loadVocabulary(_ text: [String]) -> [String] {
var counts = [String:Int]()

for w in text {
let c = counts[w] ?? 0
counts[w] = c + 1
}

let count = counts.sorted(by: { (arg1, arg2) -> Bool in
let (_, value1) = arg1
let (_, value2) = arg2
return value1 > value2
})

return count.map { (arg0) -> String in
let (key, _) = arg0
return key
}
}

func makeHelper(_ vocabulary: [String]) -> [Int:Int] {
var result : [Int:Int] = [:]

vocabulary.enumerated().forEach { (arg0) in
let (offset, element) = arg0
result[element.hash] = offset
}

return result
}

Why not hashValue instead of hash? turns out, on Linux, which this baby is going to run on, the values are more stable with the latter rather than the former, according to my tests.

The data we will work on therefore is:

struct TextBatch {
let original: [String]
let vocabulary: [String]
let indexHelper: [Int:Int]
let features : Tensor<Float> // 3 words
let labels : Tensor<Int32> // followed by 1 word
}

We need a way to initialize that struct, and a couple of helper functions to extract some random samples to train our model on, and we're good to go:

extension TextBatch {
public init(from: [String]) {
let h = makeHelper(v)
var f : [[Float]] = []
var l : [Int32] = []
for i in 0..<(from.count-3) {
if let w1 = h[from[i].hash],
let w2 = h[from[i+1].hash],
let w3 = h[from[i+2].hash],
let w4 = h[from[i+3].hash] {
f.append([Float(w1), Float(w2), Float(w3)])
l.append(Int32(w4))
}
}

let featuresT = Tensor<Float>(shape: [f.count, 3], scalars: f.flatMap { $0 }) let labelsT = Tensor<Int32>(l) self.init( original: from, vocabulary: v, indexHelper: h, features: featuresT, labels: labelsT ) } func randomSample(of size: Int) -> (features: Tensor<Float>, labels: Tensor<Int32>) { var f : [[Float]] = [] var l : [Int32] = [] for i in 0..<(original.count-3) { if let w1 = indexHelper[original[i].hash], let w2 = indexHelper[original[i+1].hash], let w3 = indexHelper[original[i+2].hash], let w4 = indexHelper[original[i+3].hash] { f.append([Float(w1), Float(w2), Float(w3)]) l.append(Int32(w4)) } } var rf : [[Float]] = [] var rl : [Int32] = [] if size >= l.count || size <= 0 { let featuresT = Tensor<Float>(shape: [f.count, 3], scalars: f.flatMap {$0 })
let labelsT = Tensor<Int32>(l)
return (featuresT, labelsT)
}
let idx = Int.random(in: 0..<l.count)
rf.append(f[idx])
rl.append(l[idx])
}
}

let featuresT = Tensor<Float>(shape: [f.count, 3], scalars: f.flatMap { $0 }) let labelsT = Tensor<Int32>(l) return (featuresT, labelsT) } func randomSample(splits: Int) -> [(features: Tensor<Float>, labels: Tensor<Int32>)] { var res = [(features: Tensor<Float>, labels: Tensor<Int32>)]() var alreadyPicked = Set<Int>() let size = Int(floor(Double(original.count)/Double(splits))) var f : [[Float]] = [] var l : [Int32] = [] for i in 0..<(original.count-3) { if let w1 = indexHelper[original[i].hash], let w2 = indexHelper[original[i+1].hash], let w3 = indexHelper[original[i+2].hash], let w4 = indexHelper[original[i+3].hash] { f.append([Float(w1), Float(w2), Float(w3)]) l.append(Int32(w4)) } } for part in 1...splits { var rf : [[Float]] = [] var rl : [Int32] = [] if size >= l.count || size <= 0 { let featuresT = Tensor<Float>(shape: [f.count, 3], scalars: f.flatMap {$0 })
let labelsT = Tensor<Int32>(l)
return [(featuresT, labelsT)]
}
let idx = Int.random(in: 0..<l.count)
rf.append(f[idx])
rl.append(l[idx])
}
}

let featuresT = Tensor<Float>(shape: [f.count, 3], scalars: f.flatMap { $0 }) let labelsT = Tensor<Int32>(l) res.append((featuresT,labelsT)) } return res } } In the next part, we will see how to set the model up, and train it. From this list, the gist is that most languages can't process 9999999999999999.0 - 9999999999999998.0 Why do they output 2 when it should be 1? I bet most people who've never done any formal CS (a.k.a maths and information theory) are super surprised. Before you read the rest, ask yourself this: if all you have are zeroes and ones, how do you handle infinity? If we fire up an interpreter that outputs the value when it's typed (like the Swift REPL), we have the beginning of an explanation: Welcome to Apple Swift version 4.2.1 (swiftlang-1000.11.42 clang-1000.11.45.1). Type :help for assistance. 1> 9999999999999999.0 - 9999999999999998.0$R0: Double = 2
2> let a = 9999999999999999.0
a: Double = 10000000000000000
3> let b = 9999999999999998.0
b: Double = 9999999999999998
4> a-b
$R1: Double = 2 Whew, it's not that the languages can't handle a simple substraction, it's just that a is typed as 9999999999999999 but stored as 10000000000000000. If we used integers, we'd have:  5> 9999999999999999 - 9999999999999998$R2: Int = 1

Are the decimal numbers broken? 😱

##### A detour through number representations

Let's  look at a byte. This is the fundamental unit of data in a computer and  is made of 8 bits, all of which can be 0 or 1. It ranges from 00000000 to 11111111 ( 0x00 to 0xff in hexadecimal, 0 to 255 in decimal, homework as to why and how it works like that due by monday).

Put like that, I hope it's obvious that the question "yes, but how do I represent the integer 999 on a byte?" is meaningless. You can decide that 00000000 means 990 and count up from there, or you can associate arbitrary values to the 256 possible combinations and make 999 be one of them, but you can't have both the 0 - 255 range and 999. You have a finite number of possible values and that's it.

Of  course, that's on 8 bits (hence the 256 color palette on old games). On  16, 32, 64 or bigger width memory blocks, you can store up to 2ⁿ different values, and that's it.

##### The problem with decimals

While  it's relatively easy to grasp the concept of infinity by looking at  "how high can I count?", it's less intuitive to notice that there is the same amount of numbers between 0 and 1 as there are integers.

So,  if we have a finite number of possible values, how do we decide which  ones make the cut when talking decimal parts? The smallest? The most  common? Again, as a stupid example, on 8 bits:

• maybe we need 0.01 ... 0.99 because we're doing accounting stuff
• maybe we need 0.015, 0.025,..., 0.995 for rounding reasons
• We'll just encode the numeric part on 8 bits ( 0 - 255 ), and the decimal part as above

But that's already  99+99 values taken up. That leaves us 57 possible values for the rest of infinity. And that's not even mentionning the totally arbitrary nature of the  selection. This way of representing numbers is historically the first  one and is called "fixed" representation. There are many ways of  choosing how the decimal part behaves and a lot of headache when coding  how the simple operations work, not to mention the complex ones like  square roots and powers and logs.

##### Floats (IEEE 754)

To  make it simple for chips that perform the actual calculations, floating  point numbers (that's their name) have been defined using two  parameters:

• an integer n
• a power (of base b) p

Such that we can have n x bᵖ, for instance 15.3865 is 153863 x 10^(-4). The question is, how many bits can we use for the n and how many for the p.

The standard is to use 1 bit for the sign (+ or -), 23 bits for n, 8 for p, which use 32 bits total (we like powers of two), and using base 2, and n is actually 1.n.  That gives us a range of ~8 million values, and powers of 2 from -126  to +127 due to some special cases like infinity and NotANumber (NaN).

$$(-1~or~1)(2^{[-126...127]})(1.[one~of~the~8~million~values])$$

In theory, we have numbers from -10⁴⁵ to 1038 roughly, but some numbers can't be represented in that form. For  instance, if we look at the largest number smaller than 1, it's 0.9999999404. Anything between that and 1 has to be rounded. Again, infinity can't be represented by a finite number of bits.

##### Doubles

The  floats allow for "easy" calculus (by the computer at least) and are  "good enough" with a precision of 7.2 decimal places on average. So when  we needed more precision, someone said "hey, let's use 64 bits instead  of 32!". The only thing that changes is that n now uses 52 bits and p 11 bits.

Coincidentally, double has more a meaning of double size than double precision, even though the number of decimal places does jump to 15.9 on average.

We  still have 2³² more values to play with, and that does fill some  annoying gaps in the infinity, but not all. Famously (and annoyingly),  0.1 doesn't work in any precision size because of the base 2. In 32 bits  float, it's stored as 0.100000001490116119384765625, like this:

(1)(2⁻⁴)(1.600000023841858)

Conversely, after double size (aka doubles), we have quadruple size (aka quads), with 15 and 112 bits, for a total of 128 bits.

##### Back to our problem

Our value is 9999999999999999.0. The closest possible value encodable in double size floating point is actually 10000000000000000, which should now make some kind of sense. It is confirmed by Swift when separating the two sides of the calculus, too:

2> let a = 9999999999999999.0
a: Double = 10000000000000000

Our  big brain so good at maths knows that there is a difference between  these two values, and so does the computer. It's just that using  doubles, it can't store it. Using floats, a will be rounded to 10000000272564224 which isn't exactly better. Quads aren't used regularly yet, so no luck there.

It's  funny because this is an operation that we puny humans can do very  easily, even those humans who say they suck at maths, and yet those  touted computers with their billions of math operations per second can't  work it out. Fair enough.

The kicker is, there is a litteral infinity of examples such as this one, because trying to represent infinity in a finite number of digits is impossible.