[OpenWhisk] CentOS installation (Part 2)

As promised in Part 1, today, we’re looking at making OpenWhisk work on CentOS 7. The process actually helps making it work for almost every Linux distribution, but I wanted something with a different package management system, and a different way to handle security. Adapt and hack, that’s the OSS way.

The Setup

This is the process as has been tested and validated on a fresh minimal in a a Virtual Box VM, but, as usual, your mileage may vary. Beyond the comfort of staying in my own environment for note-taking and web browsing, this allows to perfectly control the context. It may, however, introduce some weirdness due to virtualization inside a virtualized box. You have been warned.

Prerequisites

Here’s the exhaustive list I had to install to get anywhere:

  • git
  • ntp
  • zip
  • unzip (yes)
  • python2-pip
  • python-argcomplete
  • java-1.8.0-openjdk-headless
  • nc

Ansible is going to play a huge part later, but various repos have different versions of it and they may or may not support docker images. I installed it using pip, as well as docker-py and jinja2

pip install ansible
pip install docker-py
pip install jinja2==2.9.6

You will also need scala. I used version 2.12.4

wget https://downloads.lightbend.com/scala/2.12.4/scala-2.12.4.rpm
rpm -ihv scala-2.12.4.rpm

Docker

An extremely important piece of the puzzle is of course Docker, our container system. You probably have to install docker 1.12, as it is arguably the most stable and performance oriented version. OpenWhisk developers as well as others told me repeatedly I should use this version and none other. So I did.

I just used this guide as a base and installed the most recent 1.12.x release.

A couple of notes regarding CouchDB

At the heart of the orchestrator in OpenWhisk, there is a CouchDB database. The doc says you can use a transient one that will get reset every time the docker image is restarted, or a more permanent one. I chose the latter, having an instance running locally in the VM.

Of course, in this case you need to install it and configure it. Two things you have to make sure of:

The easy part : Gradle

OpenWhisk’s first step is to build all the ancillary images and scaffolding needed for the ansible scenarios. It takes a long time, because it pulls every image it needs and builds stuff. On a HDD machine with 2 processors I tested this on, it took well over 30 minutes. Of course, on a fiber connection and SSD machine, it was significantly faster. Don’t panic, it’s long, regardless.

You’ll obviously need to clone the git project to get started:

git clone https://github.com/apache/incubator-openwhisk openwhisk 
cd openwhisk

Then you just let gradle work. For those of you doing Android development, it’s not really weird, but a lot of people have no idea what gradle is. Think of it like a make, written in Java, and with extra features.

./gradlew distDocker

Preempting:

“it fails, with random errors!”

Because of all the docker things it has to do, as well as some side effects of the gradle build itself, I always run these commands as a superuser. Try it to see if it solves your problem. It also may help to have your current user allowed to horse around with the docker daemon, if only to debug issues. See this askubuntu thread for both a solution, and very important warnings.

After a (long) time, it should say that 40 or so tasks have been completed successfully. 🧞‍♂️ magic at work.

Networking

By default, CentOS is strict in regards to network segregation, and security in general. You will need to enable inter-docker networking manually:

iptables -I INPUT 1 -i docker 0 -j ACCEPT

If you want your networking to survive reboots, it’s quite a lot more involved: NetworkManager and firewalld have to be brought to heel. The issue (and the solution) has been best summarized in this issue as far as I can tell.

The TL;DR version is:

nmcli connection modify docker0 connection.zone trusted
systemctl stop NetworkManager.service
firewall-cmd --permanent --zone=trusted --change-interface=docker0
systemctl start NetworkManager.service
 
# (optional, to make absolutely sure)
# nmcli connection modify docker0 connection.zone trusted

Now, if you type firewall-cmd --get-zone-of-interface=docker0, it should say trusted.

Don’t forget to

systemctl restart docker.service

Ansible

So now comes the part that has most developers who aren’t used to server maintenance really baffled: ansible playbooks. The short version of it is, it’s another build system, but you can run it simultaneously on multiple machines in parallel. Don’t worry about it for now if you’re not embarking on deploying a cluster of computers.

The important part of it for the purpose of this writeup is that it’s a series of commands that configure and launch your services, and generates the configuration files.

You will need to be in the ansible directory, so it’s time to cd ansible

The first step is to generate the various and very important config files the process will use: ansible-playbook setup.yml

The very important file that comes out of this is db_local.ini. Edit it with your CouchDB credentials and IP address. If they aren’t correct, or you haven’t setup the network correctly (see above), pretty much everything else will fail.

The rest of the playbooks should be a cakewalk:

ansible-playbook couchdb.yml --tags ini
ansible-playbook initdb.yml
ansible-playbook wipe.yml
ansible-playbook apigateway.yml
ansible-playbook openwhisk.yml
ansible-playbook postdeploy.yml

In the end, you should have a working openwhisk installation. If you wonder what you can use it for, see part 1, but your first step should probably be setting up the cli.

In the next installment, I’ll showcase a tricky swift 4.0 series of actions that need a non-standard whisk image to run (only swift 3.1.1 is supported as of this post, and you can’t really have code that relies on system libs like libxml2 or postgresql because they aren’t included in the base image.

Bonus round nb 1

On some systems (I’m looking at you ubuntu) and for some versions of openwhisk, the nginx reverse proxy fails to reload upon reboot. You will need to save and restore its configuration files, or set it up differently in the ansible deployment procedure. I chose the former because I’m just testing for now: after openwhisk has been successfully deployed, I copied the /tmp/wskconf folder somewhere “safe”, and in my rc.local, I restore it, as well as the log folders:

mkdir -p /tmp/wsklogs/controller0
mkdir -p /tmp/wsklogs/invoker0
mkdir -p /tmp/wsklogs/nginx
 
cp -rf /usr/share/openwhisk/wskconf /tmp/

This procedure is almost certainly going to change as the community evolves the deployment process, but you should be aware of it.

Bonus round nb 2

OK, I need to update OpenWhisk because it now supports feature XXX. Do I need to wipe everything and start anew?

Well, you certainly can. Personally, I just stop and remove the OW dockers and replay part of the installation:

docker stop nginx invoker1 invoker0 controller1 controller0 zookeeper kafka controller zookeeper0 kafka0 controller0 zookeeper1 kafka1 controller1 apigateway redis
docker rm nginx invoker1 invoker0 controller1 controller0 zookeeper kafka controller zookeeper0 kafka0 controller0 zookeeper1 kafka1 controller1 apigateway redis
 
# SUPER IMPORTANT!!!!
./gradlew distDocker
 
cd ansible
rm -rf /tmp/wskconf/nginx
# don't forget to look at bonus round 1 if you want to survive a reboot
 
ansible-playbook setup.yml
ansible-playbook couchdb.yml
ansible-playbook initdb.yml
ansible-playbook apigateway.yml
ansible-playbook openwhisk.yml