Fast and Resilient Integration Testing

Continuous Lifecycle 2015

  Dr. Thomas Schank

  Max F. Albrecht

http://drtom.ch/talks/2015/CL

Version 1.1.0

This work is licensed under a Creative Commons Attribution-NoDerivatives 4.0 International License.

Madek Team & Us

Zurich University of the Arts

  • Thomas: Software-Architect, Developer, CI-Infrastructure
  • Max: Frontend Software-Engineer, Meta-Data Concepts

Madek - Medienarchiv der Künste

Architecture & Technologies

  • Ruby on Rails, Clojure
  • React with progressive enhancement
  • 3-tier web-application
  • towards micro-services
  • deployment via Ansible to private cloud

Madek Testing

"specification by example"

→ integration testing

→ components interaction

1. The Problem

Madek Project 2012

many new features, many new tests

  • testing time 1 1/2 - 2 hours, increasing
  • more and more failing tests: false negatives
  • 1/8 builds pass

Try to Improve Tests

  • very time and resources consuming
  • improvement for some time
  • new features and new tests made efforts futile

Manual Retrying

automated tests, local retries

automatic → semi automatic testing

2. Comprehension

Probability of a False Negative for a whole Test-Suite

  Expression Example
probability false negative single test 3%
probability "success" 0.97
number of tests 100
probability "success" whole suite

→ only one out of 20 will pass as it should

"succes" = true positive

Why retrying works so well

let number of independent retries per test

Expected successful outcome for and

k
1 5%
2 91%
3 99.7%

2. Comprehension - Conclusion

  • more tests → exponential increase of likeliness for false negatives

  • compensate by retrying single tests just a few times

→ retrying is not an anti-pattern

→ it can be a necessity

3. Implementation

Jenkins

  • fall 2012
  • build creates other builds via the Jenkins API
  • last build aggregates
  • solved false negative problem (partly)
  • testing time: 15 - 25 minutes

→ it worked

  • frequent code pushes interfere

  • "REST-like style API" → not much like REST

  • considerable effort and maintenance

→ Jenkins and "CI-X" just aren't made for this

Cider-CI

homegrown solution, started in spring 2013

  • inherent support for retries and parallelization
  • test reproducibility
  • tight integration with source code
  • manage services while testing
  • support everything from testing to deployment

ready to use in fall 2013, never looked back

4. Concepts in Context

Jobs

Examples

  • run test-suite
  • perform static code checks
  • build
  • deploy

jobs can be triggered and can depend on each other

Project Configuration

cider-ci.yml file in the project

jobs:
  deploy_test:
    name: Deploy to test

    depends-on:
    - type: job
      job: integration-tests
      states: [passed]

    run-on:
    - type: branch
      include-match: ^master$

    # specify tasks etc

The source is the truth.

configuration: reproducible, reviews, audits ???

Cider-CI and the Source Code

Cider-CI "knows" about commits, branches, submodules,…

tree-id: fingerprint of your source code

  • reproducibility
  • jobs can be run at any time (later)
  • binary search for "bad" commits
  • commit amends, squashing: existing job remains valid

Jobs & Tasks

job: container and state aggregate for tasks

→ parallelization

Tasks & Trials

  • blueprint
  • container and state aggregate for trials

→ resilience

Trial & Scripts

  • actual unit of execution
  • executed in the same context
  • depend on each other

Script Dependencies

  • traditional CI: one "build" ⇔ one script
  • more modern: one main script + before and after "hooks"
  • Cider-CI: scripts with dependencies

4. Demos

5. Addendum

Managing false positives

Retrying randomized tests can hide problems!

"Generative Testing" e.g.

Solution:

  • reproducibility by initializing the pseudo random generator (we use the tree_id e.g.)
  • statistics

Git SCM and Git only

  • don't compromise
  • can't support everything with reasonable effort

Security & Trust

  • Cider-CI server itself never runs any code from projects
  • "blessed" executors only accept trials for a particular project (repository)

Matching Trials to Executors

  • task specifies required traits, e.g: [bash, ruby-2.2 ]

  • executors advertise available traites, e.g. [bash, maven, postgresql, ruby-2.1, ruby-2.2, …]

Cider-CI will determine a suitable executor.

Deployment

  • Ansible
  • Cider-CI deploy project, SCM managed, reproducible

Cider-CI is an Expert System

it is about making the hard possible, and not not about making the simple easy*

  • for professionals
  • no compromises
  • steep learning curve
  • high rewards

→ swiss army knife for devops

*see "Simple Made Easy" by Rich Hickey

Conclusion

  • A false negative outcome becomes likely with an increasing number of tests.
  • The problem must be solved by retrying single tests.
  • Consider to build your own pipeline.
  • Try Cider-CI, open source, installs with two commands:

http://docs.cider-ci.info/introduction/quick-start/
Thank You!