Subsetting your test runs

Overview
Soon after you've started sending data, you can start using Launchable to subset your test runs and save time.
The high level flow for subsetting is:
1.Get the full list of tests (or test files, targets, etc.) and pass that to launchable subset along with:
1.an optimization target for the subset
2.a build name, so Launchable can choose the best tests for the changes in the build being tested
2.launchable subset will get a subset from the Launchable platform and output that list to a text file
3.Pass the text file into your test runner to run only those tests
The diagram below illustrates the interactions between your tools, the Launchable CLI, and the Launchable platform:
The diagram above uses the generic term test files, but the real object type may be different depending on your stack (e.g. test classes, test targets, etc.).
Preparing your pipeline for subsetting
Depending on your goal, you might need to make a few changes to your pipeline to adopt subsetting.
Goal: Run a subset of tests at the same stage of your software delivery lifecycle
After subsetting your tests, you should make sure to run the full suite of tests at some point later in your pipeline.
For example, once you start running a subset of an integration test suite that runs on pull requests, you should make sure to run the full integration test suite after a PR is merged (and record the outcome of those runs with launchable record tests).
Run the full suite after merging
Goal: Run a subset of tests earlier in your software delivery lifecycle ("shift left")
If your goal is to run a short subset of a long test suite earlier in the development process, then you may need to set up a new pipeline to run tests in that development phase. For example, if you currently run a long nightly test suite, and you want to run a subset of that suite every hour, you may need to create a pipeline to build, deploy, and run the subset if one doesn't exist already.
You'll also want to continue running the full test suite every night (and recording the outcome of those runs with launchable record tests).
Shift nightly tests left
Choosing an optimization target
The optimization target you choose determines how Launchable populates a subset with tests. You can use the Confidence and Comprehensiveness curves shown in the Launchable dashboard to choose an optimization target.
Confidence and comprehensiveness curves
Confidence target (--confidence)
The confidence target is designed for use with test suites where the total duration of each run used to train the model is relatively stable. If your runs have highly variable duration, the percentage time target may be more useful.
Confidence is shown on the y-axis of a confidence curve. When you request a subset using --confidence 90%, Launchable will populate the subset with relevant tests up to the corresponding expected duration value on the x-axis. For example, if the corresponding duration value for 90% confidence is 3 minutes, Launchable will populate the subset with up to 3 minutes of the most relevant tests for the changes in that build. This is useful to start with because the duration should decrease over time as Launchable learns more about your changes and tests.
Fixed time target (--time)
The fixed time target is designed for use with test suites where the total duration of each run used to train the model is relatively stable. If your runs have highly variable duration, the percentage time target may be more useful.
Time is shown on the x-axis of a confidence curve. When you request a subset using --time 10m, Launchable will populate the subset with up to 10 minutes of the most relevant tests for the changes in that build. This is useful if you have a maximum test runtime in mind.
Percentage time target (--target)
Percentage time is not yet shown in any charts at app.launchableinc.com. When you request a subset using --target 20%, Launchable will populate the subset with 20% of the expected duration of the most relevant tests. For example, if the expected duration of the full list of tests passed to launchable subset is 100 minutes, Launchable will return up to 20 minutes of the most relevant tests for the changes in that build. This is useful if your test runs vary in duration.
Requesting and running a subset
To retrieve a subset of tests, first list all the tests you would normally run and pass that to launchable subset. Here's an example using Ruby/minitest and --confidence:
1
launchable subset \
2
  --build <BUILD NAME> \
3
  --confidence 90% \
4
  minitest test/**/*.rb > launchable-subset.txt
Copied!
The --build option should use the same <BUILD NAME> value that you used in launchable record build.
This creates a file called launchable-subset.txt that you can pass into your command to run tests:
1
bundle exec rails test $(cat launchable-subset.txt)
Copied!
Subsetting instructions depend on the test runner or build tool you use to run tests. Click the appropriate link below to get started:
​Android Debug Bridge (adb)​
​Ant​
​Bazel​
​Behave​
​CTest​
​Cypress​
​GoogleTest​
​Go Test​
​Gradle​
​Jest​
​Maven​
​minitest​
​nose​
​pytest​
​Robot​
​RSpec​
If you're not using any of these, use the generic 'file-based' runner integration, the raw profile for custom test runners, or request a plugin.
Inspecting subset details
You can use launchable inspect subset to inspect the details of a specific subset, including rank and expected duration. This is useful for verifying that you passed the correct tests or test directory path(s) into launchable subset.
The output from launchable subset includes a tip to run launchable inspect subset:
1
$ launchable subset --build 123 --confidence 90% minitest test/*.rb > subset.txt
2
​
3
< summary table >
4
​
5
Run `launchable inspect subset --subset-id 26876` to view full subset details
Copied!
Running that command will output a table containing a row for each test including:
Rank/order
Test identifier
Whether the test was included in the subset
Launchable's estimated duration for the test
Tests with a duration of .001 seconds were not recognized by Launchable
Note that the hierarchy level of the items in the list depends on the test runner in use.
For example, since Maven can accept a list of test classes as input, launchable inspect subset will output a prioritized list of test classes. Similarly, since Cypress can accept a list of test files as input, launchable inspect subset will output a list of prioritized test files. (And so on.)
Other tips
"Training wheels" mode with the --rest option
You can start subsetting by just splitting your existing suite into an intelligent subset and then the rest of the tests. After you've dialed in the right subset target, you can then remove the remainder and run the full suite less frequently. See the diagram below for a visual explanation.
The middle row of the diagram shows how you can start by splitting your existing test run into two parts:
1.A subset of dynamically selected tests, and
2.The rest of the tests
The example below shows how you can generate a subset (launchable-subset.txt) and the remainder (launchable-remainder.txt) using the --rest option. Here we're using Ruby and minitest:
1
launchable subset \
2
  --build <BUILD NAME> \
3
  --confidence 90% \
4
  --rest launchable-remainder.txt \
5
  minitest test/**/*.rb > launchable-subset.txt
Copied!
This creates two files called launchable-subset.txt and launchable-remainder.txt that you can pass into your command to run tests in two stages. Again, using Ruby as an example:
1
bundle exec rails test $(cat launchable-subset.txt)
2
​
3
bundle exec rails test $(cat launchable-remainder.txt)
Copied!
You can remove the second part after you're happy with the subset's performance. Once you do this, make sure to continue running the full test suite at some stage as described in Preparing your pipeline.
Replacing static parallel suites with a dynamic parallel subset
Some teams manually split their test suites into several "bins" to run them in parallel. This presents a challenge adopting Launchable, because you don't want to lose the benefit of parallelization.
Luckily, with split subsets you can replace your manually selected bins with automatically populated bins from a Launchable subset.
For example, let's say you currently run ~80 minutes of tests split coarsely into four bins and run in parallel across four workers:
Worker 1: ~20 minutes of tests
Worker 2: ~15 minutes of tests
Worker 3: ~20 minutes of tests
Worker 4: ~25 minutes of tests
With a split subset, you can generate a subset of the full 80 minutes of tests and then call Launchable once in each worker to get the bin of tests for that runner.
The high level flow is:
1.Request a subset of tests to run from Launchable by running launchable subset with the --split option. Instead of outputting a list of tests, the command will output a subset ID that you should save and pass into each runner.
2.Start up your parallel test worker, e.g. four runners from the example above
3.In each worker, request the bin of tests that worker should run. To do this, run launchable split-subset with:
1.the --subset-id option set to the ID you saved earlier, and
2.the --bin value set to bin-number/bin-count.
4.Run the tests in each worker.
5.After each run finishes in each worker, record test results using launchable record tests with the --subset-id option set to the ID you saved earlier.
In pseudocode:
1
# main
2
$ launchable record build --name $BUILD_ID --source src=.
3
$ launchable subset --split --confidence 90% --build $BUILD_ID bazel .
4
subset/12345
5
​
6
...
7
​
8
# worker 1
9
$ launchable split-subset --subset-id subset/12345 --bin 1/3 --rest rest.txt bazel > subset.txt
10
$ bazel test $(cat subset.txt)
11
$ launchable record tests --subset-id subset/12345 bazel .
12
​
13
​
14
# worker 2
15
$ launchable split-subset --subset-id subset/12345 --bin 2/3 --rest rest.txt bazel > subset.txt
16
$ bazel test $(cat subset.txt)
17
$ launchable record tests --subset-id subset/12345 bazel .
18
​
19
# worker 3
20
$ launchable split-subset --subset-id subset/12345 --bin 3/3 --rest rest.txt bazel > subset.txt
21
$ bazel test $(cat subset.txt)
22
$ launchable record tests --subset-id subset/12345 bazel .
Copied!