Using the generic file-based runner integration

Search…
This is a reference page. See Getting started, Sending data to Launchable, and Subsetting your test runs for more comprehensive usage guidelines.
About
The "file based" test runner integration is primarily designed to work with test runners that are not explicitly supported, such as custom test runners built in-house.
In order to work with Launchable through this integration mechanism, your test runner has to satisfy the following conditions:
File based: your test runner accepts file names as an input of a test execution, to execute just those specified set of tests.
JUnit XML reports include file names/paths: your test runner has to produce results of tests in a JUnit compatible format, with additional attributes that capture the file names/paths of the tests that run. If not, see converting test reports to JUnit.
For example, Mocha is a test runner that meets those criteria. You write tests in JavaScript files:
1
$ cat foo.js
2
var assert = require('assert');
3
describe('Array', function() {
4
  describe('#indexOf()', function() {
5
    it('should return -1 when the value is not present', function() {
6
      assert.equal([1, 2, 3].indexOf(4), -1);
7
    });
8
  });
9
});
Copied!
The Mocha test runner takes those files as arguments:
1
$ mocha --reporter mocha-junit-reporter foo.js
Copied!
...and produces JUnit report files, where the name of the test file is captured, in this case, in the file attribute:
1
$ cat test-results.xml
2
<?xml version="1.0" encoding="UTF-8"?>
3
<testsuites name="Mocha Tests" time="0.0000" tests="1" failures="0">
4
  <testsuite name="#indexOf()" file="/home/kohsuke/ws/foo.js" ...>
5
    <testcase  ... />
6
...
Copied!
The rest of this document uses Mocha as an example.
Recording test results
After running tests, point the CLI to your test report files to collect test results and train the model:
1
launchable record tests --build <BUILD NAME> file ./reports/*.xml
Copied!
You might need to take extra steps to make sure that launchable record tests always runs even if the build fails. See Always record tests.
Subsetting your test runs
The high level flow for subsetting is:
1.
Get the full list of test files and pass that to launchable subset with an optimization target for the subset
2.
launchable subset will get a subset from the Launchable platform and output that list to a text file
3.
Pass the text file into your test runner to run only those tests
To retrieve a subset of tests, first pass the full list of test candidates to launchable subset. For example:
1
find ./test -name '*.js' | 
2
launchable subset \
3
    --build <BUILD NAME> \
4
    --target 10% \
5
    --rest launchable-remainder.txt \
6
    file > subset.txt
Copied!
The --build should use the same <BUILD NAME> value that you used before in launchable record build.
The --confidence option should be a percentage; we suggest 90% to start. You can also use --time or --target; see Subsetting your test runs for more info.
This creates a file called launchable-subset.txt that you can pass into your command to run tests:
1
mocha $(< launchable-subset.txt)
Copied!
Use 'flavors' to run the best tests for an environment
Next - 💡Insights
Flaky tests
Last modified 1mo ago
Copy link
Contents
About
Recording test results
Subsetting your test runs