[Dev Diaries] Code Coverage?

As you probably know by now, I'm fairly obsessed with tools that give me metrics about quality: linting, docs, tests...

Unfortunately, code coverage is fairly hard to get with SPM in a way that is usable.

What?

Code coverage is the amount of code in the package that is covered with your tests. If you run your tests, are these lines run? Are those? Your tests pass, and that's fine, but have you forgotten to test anything?

You can enable it in Xcode using the option "gather code coverage" in the scheme you are running tests for, which allows you to find a decent visualization if you know where to look for it (a new gutter appears in your code editors).

In Swift Package Manager though, it's fairly obscure:

• first you have to use the --enable-code-coverage option of the testing phase
• then you have to grab the output json path by using --show-codecov-path
• then you get a collection of things that is unique to SPM, and therefore unusable elsewhere

Now, if you look closely at the output, you can see it's fairly close to the lcov format, which is more or less a standard.

Let's make a script!

Because I'm very attached to my packages running on both Linux and MacOS, I need to grab the correct values from the environment (makes them dockerizable too).

I need:

• the output of the swift test phase
• llvm-cov which is used by the swift toolchain and can extract usable information
• A few frills here and there

Looking here and there if anything existed already, I stumbled upon a good writeup setting some of the bricks up. I would suggest reading this first if you want to get the nitty gritty details.

Reusing parts of this and making my own script that can spit either human readable or lcov-compatible, and can work either on Linux or MacOS, and dockerizable, here's what I end up with:

#!/bin/bash

swift test --enable-code-coverage > /dev/null 2>&1

if [ $? -ne 0 ]; then
echo "tests unsuccessful"
exit 1
fi

BIN_PATH="$(swift build --show-bin-path)"
XCTEST_PATH="$(find ${BIN_PATH} -name '*.xctest')"

if [[ "$OSTYPE" == "darwin"* ]]; then
	COV_BIN="/usr/bin/xcrun llvm-cov"
	MODULE="$(basename $XCTEST_PATH .xctest)"
	XCTEST_PATH="$XCTEST_PATH/Contents/MacOS/$MODULE"
else
	COV_BIN=`which llvm-cov || echo "false"`
fi

if [ $# -eq 0 ]; then
	$COV_BIN report -ignore-filename-regex=".build|Tests" \
	-instr-profile=.build/debug/codecov/default.profdata -use-color \
	"$XCTEST_PATH" 
elif [ $# -eq 1 ] && [ $1 = "lcov" ]; then
	$COV_BIN export -ignore-filename-regex=".build|Tests" \
	-instr-profile=.build/debug/codecov/default.profdata \
	--format=lcov "$XCTEST_PATH" 
else
	echo "Usage:"
	echo "codecov.sh [lcov]"
	echo "use without argument for human readable output"
	echo "use with lcov as argument for lcov format output"
	exit 1
fi

codecov.sh spits something like:

Filename                      Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SEKRET.swift                       67                20    70.15%          23                 4    82.61%         157                27    82.80%
Extensions.swift                   15                 3    80.00%           2                 0   100.00%          20                 0   100.00%
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                              82                23    71.95%          25                 4    84.00%         177                27    84.75%

codecov.sh lcov spits the corresponding lcov output.

Hurray for automation!