Rust describes itself as:

a systems programming language that runs blazingly fast, prevents segfaults, and guarantees thread safety. ### Featuring * zero-cost abstractions * minimal runtime *efficient C bindings

So, it’s likely that developers who choose to program in Rust are focused on performance. You can make sure your code is efficient by writing benchmarks, but in order to prevent performance regressions, you’ll need to run benchmarks on your Pull Requests or patches and somehow compare before and after. Doing this can be tedious, especially as the changeset evolves over the course of code review or miscellaneous refactoring.

Let’s see how we can get automated benchmark comparisons across commits on Travis CI.

Putting benchmarks in your project

First off, you’ll need to have benchmarks in your codebase. There are a few ways to do this:

• The standard way documented in the Rust Book
• Making a benches directory in your project root, putting your benchmarks there, and running cargo bench (this is how I’ve done it in Frunk)

Running benchmarks on Travis

Next, in order to run benchmarks on Travis, we’ll need to make sure that your .travis.yml file has nightly listed as one of the Rust versions that your project is built with:

1
2
3

rust:
  - stable
  - nightly # so we can run benchmarks (required as of writing)

Then, in after_success, we’ll want the following in order to have benchmarks run when we are on a build that uses Rust nightly:

1
2
3
4

after_success:
  - if [ "$TRAVIS_RUST_VERSION" == "nightly" ]; then
        cargo bench;
    fi

Some readers might be wondering why I’m not using travis-cargo here. The reason is because travis-cargo doesn’t support arbitrary cargo libraries/commands, which is needed in the next section ;)

Getting benchmark comparisons in Pull Requests

So we have benchmarks running automatically on Travis, but what about the before-after comparisons that we talked about earlier? This is where the cargo-benchcmp library comes into play. benchcmp is:

A small utility for comparing micro-benchmarks produced by cargo bench. The utility takes as input two sets of micro-benchmarks (one “old” and the other “new”) and shows as output a comparison between each benchmark.

What we’ll want to do next is add a condition to only run these benchmarks when we’re building a Pull Request (henceforth PR), install the benchcmp tool, and use it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

#!/usr/bin/env bash

if [ "${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}" != "master" ] && [ "$TRAVIS_RUST_VERSION" == "nightly" ]; then
    REMOTE_URL="$(git config --get remote.origin.url)";
    # Clone the repository fresh..for some reason checking out master fails
    # from a normal PR build's provided directory
    cd ${TRAVIS_BUILD_DIR}/.. && \
    git clone ${REMOTE_URL} "${TRAVIS_REPO_SLUG}-bench" && \
    cd  "${TRAVIS_REPO_SLUG}-bench" && \
    # Bench master
    git checkout master && \
    cargo bench > benches-control && \
    # Bench variable
    git checkout ${TRAVIS_COMMIT} && \
    cargo bench > benches-variable && \
    cargo install cargo-benchcmp --force && \
    cargo benchcmp benches-control benches-variable;
fi

The first conditional is simply to check that the current branch being built is not master. It’s a bit verbose because $TRAVIS_BRANCH does not always provide the current branch name. So instead, we use ${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}, which consists of $TRAVIS_PULL_REQUEST_BRANCH because it gives us the current branch if the build was triggered by a PR, and a default of $TRAVIS_BRANCH, which gives us the branch name of non-PR builds.

The second conditional checks that the current Travis build is using nightly, which is a requirement for running benchmarks (as of writing).

Inside the if statements body, we first cd out of our provided directory and clone our project anew. I’m not entirely sure why, but in my testing, I was unable to checkout another branch (e.g. master) otherwise. Next, we run cargo bench on the master branch, sending the output to benches-control. Afterwards, we checkout the commit for the current build by using TRAVIS_COMMIT, and run cargo bench again, sending the output to benches-variable.

Lastly, we install and run cargo benchcmp, passing the path of the control and variable benchmark result files as arguments, letting cargo-benchcmp do its job.

Oh, we shouldn’t forget to add our script to the after_success block in our Travis file.

1
2

after_success:
  - ./travis-after-success.sh

Here is some sample output from my Rust functional programming library, Frunk.

The benchmark comparisons show up in the build log.

Conclusion

That’s it. Now, you can go to the Travis build log of your PRs and see how performance has been affected. Please give it a try, and send any questions or feedback. Oh, if you’re interested in a library that does this for you or if you want to turn this into some kind of a service, do let me know ;-)