As promised in Part 1, today, we’re looking at making OpenWhisk work on CentOS 7. The process actually helps making it work for almost every Linux distribution, but I wanted something with a different package management system, and a different way to handle security. Adapt and hack, that’s the OSS way.
This is the process as has been tested and validated on a fresh minimal in a a Virtual Box VM, but, as usual, your mileage may vary. Beyond the comfort of staying in my own environment for note-taking and web browsing, this allows to perfectly control the context. It may, however, introduce some weirdness due to virtualization inside a virtualized box. You have been warned.
Here’s the exhaustive list I had to install to get anywhere:
- unzip (yes)
Ansible is going to play a huge part later, but various repos have different versions of it and they may or may not support docker images. I installed it using pip, as well as docker-py and jinja2
pip install ansible pip install docker-py pip install jinja2==2.9.6
You will also need scala. I used version 2.12.4
wget https://downloads.lightbend.com/scala/2.12.4/scala-2.12.4.rpm rpm -ihv scala-2.12.4.rpm
An extremely important piece of the puzzle is of course Docker, our container system. You probably have to install docker 1.12, as it is arguably the most stable and performance oriented version. OpenWhisk developers as well as others told me repeatedly I should use this version and none other. So I did.
I just used this guide as a base and installed the most recent 1.12.x release.
A couple of notes regarding CouchDB
At the heart of the orchestrator in OpenWhisk, there is a CouchDB database. The doc says you can use a transient one that will get reset every time the docker image is restarted, or a more permanent one. I chose the latter, having an instance running locally in the VM.
Of course, in this case you need to install it and configure it. Two things you have to make sure of:
- step 1 is always securing it by enabling the admin account
- given that your couchdb instance is probably only going to be used by your openwhisk installation, you should probably have it listen only on the docker0 interface. Mine is bound to
The easy part : Gradle
OpenWhisk’s first step is to build all the ancillary images and scaffolding needed for the ansible scenarios. It takes a long time, because it pulls every image it needs and builds stuff. On a HDD machine with 2 processors I tested this on, it took well over 30 minutes. Of course, on a fiber connection and SSD machine, it was significantly faster. Don’t panic, it’s long, regardless.
You’ll obviously need to clone the git project to get started:
git clone https://github.com/apache/incubator-openwhisk openwhisk cd openwhisk
Then you just let gradle work. For those of you doing Android development, it’s not really weird, but a lot of people have no idea what gradle is. Think of it like a
make, written in Java, and with extra features.
“it fails, with random errors!”
Because of all the docker things it has to do, as well as some side effects of the gradle build itself, I always run these commands as a superuser. Try it to see if it solves your problem. It also may help to have your current user allowed to horse around with the docker daemon, if only to debug issues. See this askubuntu thread for both a solution, and very important warnings.
After a (long) time, it should say that 40 or so tasks have been completed successfully. 🧞♂️ magic at work.
By default, CentOS is strict in regards to network segregation, and security in general. You will need to enable inter-docker networking manually:
iptables -I INPUT 1 -i docker 0 -j ACCEPT
If you want your networking to survive reboots, it’s quite a lot more involved:
firewalld have to be brought to heel. The issue (and the solution) has been best summarized in this issue as far as I can tell.
The TL;DR version is:
nmcli connection modify docker0 connection.zone trusted systemctl stop NetworkManager.service firewall-cmd --permanent --zone=trusted --change-interface=docker0 systemctl start NetworkManager.service # (optional, to make absolutely sure) # nmcli connection modify docker0 connection.zone trusted
Now, if you type
firewall-cmd --get-zone-of-interface=docker0, it should say
Don’t forget to
systemctl restart docker.service
So now comes the part that has most developers who aren’t used to server maintenance really baffled: ansible playbooks. The short version of it is, it’s another build system, but you can run it simultaneously on multiple machines in parallel. Don’t worry about it for now if you’re not embarking on deploying a cluster of computers.
The important part of it for the purpose of this writeup is that it’s a series of commands that configure and launch your services, and generates the configuration files.
You will need to be in the ansible directory, so it’s time to
The first step is to generate the various and very important config files the process will use:
The very important file that comes out of this is
db_local.ini. Edit it with your CouchDB credentials and IP address. If they aren’t correct, or you haven’t setup the network correctly (see above), pretty much everything else will fail.
The rest of the playbooks should be a cakewalk:
ansible-playbook couchdb.yml --tags ini ansible-playbook initdb.yml ansible-playbook wipe.yml ansible-playbook apigateway.yml ansible-playbook openwhisk.yml ansible-playbook postdeploy.yml
In the next installment, I’ll showcase a tricky swift 4.0 series of actions that need a non-standard whisk image to run (only swift 3.1.1 is supported as of this post, and you can’t really have code that relies on system libs like libxml2 or postgresql because they aren’t included in the base image.
Bonus round nb 1
On some systems (I’m looking at you ubuntu) and for some versions of openwhisk, the nginx reverse proxy fails to reload upon reboot. You will need to save and restore its configuration files, or set it up differently in the ansible deployment procedure. I chose the former because I’m just testing for now: after openwhisk has been successfully deployed, I copied the
/tmp/wskconf folder somewhere “safe”, and in my rc.local, I restore it, as well as the log folders:
mkdir -p /tmp/wsklogs/controller0 mkdir -p /tmp/wsklogs/invoker0 mkdir -p /tmp/wsklogs/nginx cp -rf /usr/share/openwhisk/wskconf /tmp/
This procedure is almost certainly going to change as the community evolves the deployment process, but you should be aware of it.
Bonus round nb 2
OK, I need to update OpenWhisk because it now supports feature XXX. Do I need to wipe everything and start anew?
Well, you certainly can. Personally, I just stop and remove the OW dockers and replay part of the installation:
docker stop nginx invoker1 invoker0 controller1 controller0 zookeeper kafka controller zookeeper0 kafka0 controller0 zookeeper1 kafka1 controller1 apigateway redis docker rm nginx invoker1 invoker0 controller1 controller0 zookeeper kafka controller zookeeper0 kafka0 controller0 zookeeper1 kafka1 controller1 apigateway redis # SUPER IMPORTANT!!!! ./gradlew distDocker cd ansible rm -rf /tmp/wskconf/nginx # don't forget to look at bonus round 1 if you want to survive a reboot ansible-playbook setup.yml ansible-playbook couchdb.yml ansible-playbook initdb.yml ansible-playbook apigateway.yml ansible-playbook openwhisk.yml