TravisCI with multiple concurrent Docker images

By Chaim Sanders

The Problem

At the OWASP Core Rule Set project (CRS) , we recently wanted to expand our testing to run against multiple version of ModSecurity, and even some non-ModSecurity WAFs, like WAFLZ. CRS can be a tricky project because we have to run tests against not only different WAFs but different WAFs on different OSs (For instance, the NGINX, Apache, and IIS version of ModSecurity all behave differently in various edge cases).

Out of necessity over the past few years CRS has been developing a suite of tests designed to work with the Framework for Testing WAFs (FTW). These tests are meant to ensure that the CRS and by extension the WAF, works as it is expected to. We historically had been using one Docker container, running ModSecurity 2.9.x on Apache to test against. While this is the most common deployment, it left some room for improved methodology.

To increase the complexity of our testing effort, we wanted to respect the fact that most WAFs, including ModSecurity, are designed to be ruleset neutral. While OWASP CRS is, by far, the most commonly used ruleset, we want to maintain the neutrality of the underlying WAFs. As a result, we end up maintaining two separate Docker images (Modsecurity, OWASP CRS) , and Docker Hub Repositories (Modsecurity, OWASP CRS). This allowed us to support the community by supplying Modsecurity images while simultaneously allowing us to test across multiple platforms.

Ultimately however, we needed Travis CI to support running multiple concurrent Docker images and test suites. Fortunately, this is possible! While we first found a hint about this on the following Stackoverflow question, it doesn’t really provide a lot of context. This is made worse by observing that the Travis CI documentation on matrices doesn’t describe the use case nor does it really demonstrate simple example cases that aren’t using multiple languages.

Using Travis CI Matrices

When you create a matrix of environment variables, each of these will create a new environment that will run independently. Take as an example the following excerpt from a .travis.yml file (The very top to be specific)

env:
  matrix:
    - DOCKER_TAG=modsecurity-crs-2.9-apache
      DOCKERFILE=Dockerfile-2.9-apache
    - DOCKER_TAG=modsecurity-crs-3.0-apache
      DOCKERFILE=Dockerfile-3.0-apache
    - DOCKER_TAG=modsecurity-crs-3.0-nginx
      DOCKERFILE=Dockerfile-3.0-nginx

This will run a TravisCI job with three separate build jobs. It will appear as follows:

Now, of course knowing that it will run three build jobs is not the whole story. You want to know how you control what is run. Essentially, the only way to do this is to use the enviorvment variables you are specifying in the Matrix. Each build can have multiple parameters as you see in the above example (DOCKER_TAG and DOCKERFILE). You will use these in the later stages to control how each job will differ.

For example, here is our before_install block that spins up a different Docker container for each of the three build jobs using the variables from above.

before_install:
  - |
    if [[ "$TRAVIS_PULL_REQUEST" != "false" ]]; then
      docker build --build-arg REPO=$TRAVIS_PULL_REQUEST_SLUG --build-arg COMMIT=$TRAVIS_PULL_REQUEST_SHA -f ./util/docker/$DOCKERFILE -t $DOCKER_TAG ./util/docker/
    else
      docker build -f ./util/docker/$DOCKERFILE -t $DOCKER_TAG ./util/docker/
    fi
  
  - docker run -ti -e PARANOIA=5 -d -p 80:80 -v /var/log/apache2:/var/log/apache2/ --name "$TRAVIS_BUILD_ID" $DOCKER_TAG

It’s important to note that all programatic phases (i.e before_install, install, and script) will run for each matrix build job. Of course, each build job is isolated from the other so feel free to reuse ports, as we do in our example.

Caveats

We have demonstrated the ‘matrix expansion’ method in the above example (https://docs.travis-ci.com/user/build-matrix/). It is important to note that there is also the ‘matrix.include’ method. The ‘matrix.include‘ method has an advantage of being slightly more readable (IMHO) and being able to use the ‘name‘ keyword, which is fairly undocumented (https://github.com/travis-ci/travis-ci/issues/5898). It does however have a disadvantage. As the documentation notes, “all the keys in allow_failures element must exist in the top level of the build matrix (i.e., not in matrix.include)”. Therefore, I’d caution against the use of ‘matrix.include’ from a future proofing perspective.

When using allow_failures, there is one other major caveat! I was unable, after many attempts, to make allow_failures work when using multi line environment variables. While these work fine in the general use case, allow_failures is space sensitive, as such you’re likely to run into issues on this front. For us, this resulted in a slightly less readable travis.yml file.

Our Final Product

We have demonstrated a somewhat simplified setup using TravisCI Matrices for running tests against multiple Docker instances. Our whole .travis.yaml file ultimately looked as follows:

language:         python
python:
  - 2.7
env:
  matrix:
    - DOCKER_TAG=modsecurity-crs-2.9-apache DOCKERFILE=Dockerfile-2.9-apache LOG_LOCATION=/var/log/apache2/ LOG_FILE=error.log FTW_DATE_REGEX="\\[([A-Z][a-z]{2} [A-z][a-z]{2} \\d{1,2} \\d{1,2}\\:\\d{1,2}\\:\\d{1,2}\\.\\d+? \\d{4})\\]" FTW_DATE_FORMAT="%a %b %d %H:%M:%S.%f %Y"
    - DOCKER_TAG=modsecurity-crs-3.0-nginx DOCKERFILE=Dockerfile-3.0-nginx LOG_LOCATION=/var/log/nginx/ LOG_FILE=error.log FTW_DATE_REGEX="(\\d{4}\\/\\d{2}\\/\\d{2} \\d{2}:\\d{2}:\\d{2})" FTW_DATE_FORMAT="%Y\/%m\/%d %H:%M:%S.%f"
    - DOCKER_TAG=modsecurity-crs-3.0-apache DOCKERFILE=Dockerfile-3.0-apache LOG_LOCATION=/var/log/apache2/ LOG_FILE=error.log FTW_DATE_REGEX="\\[([A-Z][a-z]{2} [A-z][a-z]{2} \\d{1,2} \\d{1,2}\\:\\d{1,2}\\:\\d{1,2}\\.\\d+? \\d{4})\\]" FTW_DATE_FORMAT="%a %b %d %H:%M:%S.%f %Y"

matrix:
  fast_finish:    true
  allow_failures:
  - python:       2.7
    env:          DOCKER_TAG=modsecurity-crs-3.0-apache DOCKERFILE=Dockerfile-3.0-apache LOG_LOCATION=/var/log/apache2/ LOG_FILE=error.log FTW_DATE_REGEX="\\[([A-Z][a-z]{2} [A-z][a-z]{2} \\d{1,2} \\d{1,2}\\:\\d{1,2}\\:\\d{1,2}\\.\\d+? \\d{4})\\]" FTW_DATE_FORMAT="%a %b %d %H:%M:%S.%f %Y"

sudo:             required

services:
  - docker

before_install:
  - |
    if [[ "$TRAVIS_PULL_REQUEST" != "false" ]]; then
      docker build --build-arg REPO=$TRAVIS_PULL_REQUEST_SLUG --build-arg BRANCH=$TRAVIS_PULL_REQUEST_BRANCH --build-arg COMMIT=$TRAVIS_PULL_REQUEST_SHA -f ./util/docker/$DOCKERFILE -t $DOCKER_TAG ./util/docker/
    else
      docker build -f ./util/docker/$DOCKERFILE -t $DOCKER_TAG ./util/docker/
    fi
  - docker run -ti -e PARANOIA=5 -d -p 80:80 -v $LOG_LOCATION:$LOG_LOCATION --name "$TRAVIS_BUILD_ID" $DOCKER_TAG
install:
  - pip install -r ./util/integration/requirements.txt
  - pip install -r ./util/regression-tests/requirements.txt
script:
  - |
    docker ps | grep -q modsecurity-crs
    if [[ $? -ne 0 ]]; then
      docker logs "$TRAVIS_BUILD_ID"
      docker rm -f "$TRAVIS_BUILD_ID"
      exit 1
    fi
  - git clone https://github.com/CRS-support/secrules_parsing
  - pip install -r secrules_parsing/requirements.txt
  - python secrules_parsing/secrules_parser.py -c -f rules/*.conf
  - py.test -vs ./util/integration/format_tests.py
  # FTW custon settings
  - echo "log_location_linux = '$LOG_LOCATION$LOG_FILE'" > util/regression-tests/config.py
  - echo "log_date_regex = '$FTW_DATE_REGEX'" >> util/regression-tests/config.py
  - echo "log_date_format = '$FTW_DATE_FORMAT'" >> util/regression-tests/config.py
  - py.test -vs util/regression-tests/CRS_Tests.py --rule=util/regression-tests/tests/test.yaml
  - py.test -vs util/regression-tests/CRS_Tests.py --ruledir_recurse=util/regression-tests/tests/
  - docker rm -f "$TRAVIS_BUILD_ID"
# safelist
branches:
  only:
  - v3.0/dev
  - v3.0/master
  - v3.1/dev
  - v3.2/dev
notifications:
  irc:            "chat.freenode.net#modsecurity"

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s