Rust Code Coverage Guide: kcov + Travis CI + Codecov / Coveralls25 Jul 2016
Last Updated: October 31, 2016
For questions about or problems with Rust code coverage, use the Rust IRC channels, StackOverflow or contact me. This website is open-source so if you find a problem with this guide, please open a new issue.
The de facto code coverage tutorial for Rust was published on March 15, 2015. If you Google “rust code coverage” you will largely find links to that tutorial. It is a great guide, many things have changed in the year since it was published.
This guide covers everything you need to know to get code coverage working for your Rust project. It goes over some common problems you may run into as well as their solutions. Travis CI integration and publishing your coverage results to either Codecov or Coveralls will be discussed near the end.
I recently dove quite deeply into the current state of Rust code coverage when I set it up for my own project and also helped the Codecov team fully integrate Rust into their platform. I wrote the Codecov Rust example which contains detailed instructions for sending your coverage data to Codecov. This guide will go into even more detail about Rust code coverage.
You should understand how testing works in Rust and how to run your tests with cargo. The Rust book has a great section on this.
You should also know the basics of how to use Travis CI. The Travis CI website is an excellent resource for getting setup.
The tool being used, kcov, is not Rust-specific. It uses DWARF debugging information in your generated executable to determine which lines have been covered. kcov does not always generate the most accurate coverage information.
kcov will likely only work for x86 and x86_64 Linux. This is usually enough for Travis CI integration but you may need some extra steps to get it working locally if you are not on that operating system.
As of writing this guide, there is no way to generate code coverage data for Rust doctests. This is because the doctest executable is only generated temporarily while it is being run. If you find a way to collect coverage for doctests, please open a new issue or contact me to let me know.
You need to use a very recent version of kcov in order to get it to work with Rust executables.
Before you begin, see if your package manager has a kcov version greater than or equal to kcov 31. If it does, you will likely be able to install kcov using your package manager and skip the Manually Compiling kcov section entirely. Earlier versions of kcov may also work, but I have only verified kcov 31.
Update (October 2, 2016): Thanks to a contribution by Ragnaroek there is now a third way to get kcov in addition to using apt or manually compiling. If you are using Docker, he has provided a section below which describes exactly how to use kcov from a public Docker image.
To install using apt:
sudo apt-get install kcov
Once installed, check your kcov version use the
Very old versions do not support the
--version argument so if you get
an error message instead of a version, your kcov version is too old.
Manually Compiling kcov
Install the dependencies for kcov:
sudo apt-get install libcurl4-openssl-dev libelf-dev libdw-dev cmake gcc binutils-dev libiberty-dev zlib1g-dev
In the original Rust code coverage tutorial,
gcc were required. However,
shortly after that was published,
libiberty as dependencies as well.
The new dependencies allow you to run kcov with the
option as shown below.
To compile the kcov source code, run the following commands:
wget https://github.com/SimonKagstrom/kcov/archive/master.tar.gz tar xzf master.tar.gz cd kcov-master mkdir build cd build cmake .. make # puts kcov in the default location usually /usr/local/bin/kcov sudo make install
To make kcov install in a different location, run
cmake like this:
cmake -DCMAKE_INSTALL_PREFIX:PATH=/usr ..
To avoid running
make install, copy both kcov and the generated
files from the
build/src directory. The shared objects are necessary for
working with Rust executables.
Using kcov From Docker
Update (October 2, 2016): This additional section was contributed by Ragnaroek. Thanks to him, there is now an easy way for Docker users to setup kcov without going through all of the steps above.
To use kcov from Docker, pull the kcov image from DockerHub:
docker pull ragnaroek/kcov:<version>
<version> is the kcov version tag (e.g.
v31) as published in the
Ragnaroek/kcov_docker GitHub repository.
The docker images are provided starting from version v31.
If you pulled the image you can run kcov from the image with:
docker run --security-opt seccomp=unconfined -v $(pwd):/source kcov:v31 <kcov-commands>
<kcov-commands> to provide arguments to the kcov binary. kcov is
automatically executed in the image.
--security-opt seccomp=unconfined is important. Without it
docker will not allow the
personality call that kcov executes.
You can also build your own custom kcov image, maybe with a non-released beta version or with your own dependencies installed. Detailed instructions are provided in the Ragnaroek/kcov_docker GitHub repository.
Collecting Coverage Data
kcov runs your test executables and then outputs a report showing which lines were covered and which lines were not.
To collect coverage data, first generate your test executable without running it:
cargo test --no-run
The compiled binaries are placed in
target/debug. Cargo may create multiple
test executables if you have multiple binaries. Note that Cargo will postfix the test binary names with a hash, e.g.
<executable name>-012954d6a8535cff, so make sure to pick the latest of these binaries for the next step.
To run your tests and collect coverage, run kcov with the following command:
kcov --exclude-pattern=/.cargo,/usr/lib --verify target/cov target/debug/<executable name>
You know it worked if you get the same output you see when you run
your tests with
cargo test. You can see your coverage data in
target/cov. Change that path in the command above to output
coverage somewhere else.
Be careful here because kcov will only run a single executable. If you
<executable name> with a pattern that matches multiple executables,
only one of them will run.
Note: If collecting coverage for multiple test executables, make sure you are not inadvertently overwriting the coverage data of one of your other executables. When you run kcov, it will automatically store coverage in a directory named after the full name of the executable you pass in. It will also merge any coverage it finds each time you run it for a different executable. This may cause some lines to appear uncovered even though they are covered in another test executable. Whether or not things are merged should not be an issue for Codecov because it will automatically search for and merge all of your coverage data automatically. Coveralls should be able to do this as well.
/usr/lib is included in
--exclude-pattern so that shared
libraries you may be using are not included. You may not always need this.
You can exclude any path you don’t want showing up in your coverage data
by adding to the comma separated list in the
--verify option is used by kcov to “verify breakpoint setup (to catch compiler bugs)” – see
Without it you will get an error that looks something like this:
kcov: Process exited with signal 4 (SIGILL) at 0x7ffff73c0301 kcov: Illegal instructions are sometimes caused by some GCC versions kcov: miscompiling C++ headers. If the problem is persistent, try running kcov: with --verify. For more information, see kcov: http://github.com/SimonKagstrom/kcov/issues/18
If you get this error (or something like it):
kcov: error while loading shared libraries: libopcodes-2.24-system.so: cannot open shared object file: No such file or directory
It means the libraries used to compile kcov are out of date. Recompiling kcov using the instructions above or reinstalling it using your package manager should solve the problem.
Example coverage report generated by kcov for lion:
The live Codecov coverage report generated using the kcov coverage data is available on Codecov.
Each time you run kcov using the instructions above, it will
generate a coverage report for your executable and place it in
target/cov/<executable name>/ directory. If you run kcov
multiple times, it will merge the coverage data into a single report and
place that in a special
find . -name "cobertura.xml" ./target/cov/<test-executable-1>/cobertura.xml ./target/cov/<test-executable-2>/cobertura.xml ./target/cov/kcov-merged/cobertura.xml
Full instructions for how to upload this data to Codecov or Coveralls is provided below. For any other coverage provider, these are likely the files you will need to upload.
Travis CI Integration
Update (July 29, 2016): As pointed out in #1, there is a script called travis-cargo that handles Travis and Coveralls integration for you. The Travis configuration you would use to integrate that is similar to the one below except that it has the kcov installation details hidden behind the script.
These instructions are specific to Travis CI, but should be ubiquitous enough to translate to any CI provider. We essentially need to take the above instructions and translate them into a Travis CI configuration.
As of July 25, 2016, the version of kcov that can be downloaded with
is not recent enough to run Rust executables. That means that you need to
compile kcov manually in your Travis CI configuration for this to work.
Here is the complete Travis CI configuration:
language: rust rust: - 1.9.0 addons: apt: packages: - libcurl4-openssl-dev - libelf-dev - libdw-dev - cmake - gcc - binutils-dev after_success: | wget https://github.com/SimonKagstrom/kcov/archive/master.tar.gz && tar xzf master.tar.gz && cd kcov-master && mkdir build && cd build && cmake .. && make && sudo make install && cd ../.. && rm -rf kcov-master && kcov --exclude-pattern=/.cargo,/usr/lib --verify target/cov target/debug/<executable name>-* && bash <(curl -s https://codecov.io/bash) && echo "Uploaded code coverage"
Make sure you replace
<executable name> with the name of your test
-* afterwards is a pattern that will automatically match
the hash generated by cargo after your executable name.
Once the version of kcov available with
apt is updated, this configuration will become:
language: rust rust: - 1.9.0 addons: apt: packages: - kcov after_success: | kcov --exclude-pattern=/.cargo,/usr/lib --verify target/cov target/debug/<executable name>-* && bash <(curl -s https://codecov.io/bash) && echo "Uploaded code coverage"
Let’s go through each section of the configuration.
Tells Travis CI that this is a Rust project. Travis CI has a default rust-specific configuration that gets loaded when you use this option.
rust: - 1.9.0
Update this with the versions of Rust you plan to support.
addons: apt: packages: - libcurl4-openssl-dev - libelf-dev - libdw-dev - cmake - gcc - binutils-dev
Installs the necessary dependencies for compiling kcov.
after_success: | wget https://github.com/SimonKagstrom/kcov/archive/master.tar.gz && tar xzf master.tar.gz && cd kcov-master && mkdir build && cd build && cmake .. && make && sudo make install && cd ../.. && rm -rf kcov-master && kcov --exclude-pattern=/.cargo,/usr/lib --verify target/cov target/debug/<executable name>-* && bash <(curl -s https://codecov.io/bash) && echo "Uploaded code coverage"
This is the largest step in the Travis configuration. It essentially just follows the exact steps outlined above for manually building kcov.
Let’s break this down even further:
wget https://github.com/SimonKagstrom/kcov/archive/master.tar.gz && tar xzf master.tar.gz && cd kcov-master && mkdir build && cd build && cmake .. && make && sudo make install &&
These are all the manual compilation steps.
cd ../.. && rm -rf kcov-master &&
This removes the kcov-master folder so that it is not accidentally picked up when coverage is uploaded to Codecov.
kcov --exclude-pattern=/.cargo,/usr/lib --verify target/cov target/debug/<executable name>-* &&
This runs kcov with the first executable found with the
<executable name>-* pattern. The
-* will match the hash that cargo
automatically appends to executable names.
If you have multiple test executables with the same name but a different hash, you can replace the kcov command with this bash for loop to avoid any conflicts:
for file in target/debug/<executable name>-*; do mkdir -p "target/cov/$(basename $file)"; kcov --exclude-pattern=/.cargo,/usr/lib --verify "target/cov/$(basename $file)" "$file"; done &&
This will make a folder for each set of results before running the tests. Using this will not impact Codecov because it will automatically find all coverage results regardless of the folder structure.
You only need to use this loop if your
<executable name>-* pattern matches
more than one executable. Without it, only some of your test results will be
reflected in the kcov report.
bash <(curl -s https://codecov.io/bash) &&
This line uploads the coverage to Codecov and is explained in more detail below. Make sure you remove this line if you do not plan to use Codecov.
Caveats of this setup
Tests run twice, once during the build and once after success to collect coverage. You can customize this behaviour by refining the Travis CI configuration
kcov is recompiled during every build. This adds some time to every build. This can potentially be improved using Travis CI’s caching feature. Cache the kcov executable to avoid repeating this step in every build.
A future version of this guide may resolve these issues.
Integrating Codecov with Travis CI requires that you add a single bash
command to the
after_success: section of your configuration.
bash <(curl -s https://codecov.io/bash)
In the configuration above, this is included after building and running kcov. You do not need to add this line again if you are using the Travis CI configuration above.
Update (July 29, 2016): As pointed out in #1, there is a
script called travis-cargo that handles Travis and
Coveralls integration for you. Using Codecov with the configuration provided
by the travis-cargo documentation should be exactly the same. Just add
the line above to the
after_success: section provided in their example
This loads and runs a script that automatically finds and uploads coverage
data to Codecov from a Travis CI machine. Only the
that kcov generates should be uploaded.
The source code of the script is available on GitHub and you can see which files it is uploading in the build output.
Example Travis CI output, for the lion project:
==> Travis CI detected. project root: . ==> Running gcov disable via -X gcov -> Running gcov in . ==> Searching for coverage reports under: + . -> Found 2 reports ==> Python coveragepy not found ==> Detecting git/mercurial file structure ==> Reading reports + ./target/cov/lion-d992d5a006a939c3/lion-d992d5a006a939c3/cobertura.xml bytes=63276 + ./target/cov/lion-ffbc62e55890c961/lion-ffbc62e55890c961/cobertura.xml bytes=608 ==> Uploading reports url: https://codecov.io query: branch=master&commit=b6e5d84873cc5702080470f5a2546edd4da247ea&build=41.1&build_url=&tag=&slug=sunjay/lion&yaml=&service=travis&flags=&pr=false&job=143150794 -> Pinging Codecov -> to S3 https://codecov.s3.amazonaws.com -> View reports at http://codecov.io/github/sunjay/lion/commit/b6e5d84873cc5702080470f5a2546edd4da247ea Uploaded code coverage
Coveralls integration is built in to kcov. To upload your coverage to
Coveralls, add the
--coveralls-id to your kcov command.
For Travis CI, you can update the kcov command to be:
kcov --coveralls-id=$TRAVIS_JOB_ID --exclude-pattern=/.cargo,/usr/lib --verify target/cov target/debug/<executable name>
Make sure you delete the Codecov specific line if you are using the Travis CI configuration above with Coveralls. See the Codecov Integration section for more details about which part is Codecov specific.
Docker Security Settings
Update (September 25, 2016): This additional section was added thanks to a contribution by Ragnaroek. He emailed me when he ran into this issue on Docker. I added this section since it may be a useful tip for anyone who sets up kcov on Docker. See his GitHub issue for the original problem description and discussion.
Docker users may run into the following error message when executing kcov on a Rust executable on a Docker image.
kcov --exclude-pattern=/.cargo,/usr/lib --verify target/cov target/debug/<executable name> Can't set personality: Operation not permitted kcov: error: Can't start/attach to target/debug/<executable name> Child hasn't stopped: ff00 kcov: error: Can't start/attach to target/debug/<executable name>
This happens because kcov uses the personality syscall to be able to run
PIE executables without address space randomization (source).
The fix is to update the docker security settings using this command:
docker run -it --rm --security-opt seccomp=unconfined -v $(pwd):/source kcov
You should now have the necessary tools and knowledge to:
- Generate code coverage for your Rust project locally
- Integrate code coverage into your Travis configuration
- Upload code coverage data to either Codecov or Coveralls