public
Resources
nose
This page outlines how the Launchable CLI interfaces with nose.
This is a reference page. See Getting started, Sending data to Launchable, and Subsetting your test runs for more comprehensive usage guidelines.
Recording test results
The nose plugin automatically sends test results to Launchable when you run nose with
--launchable-subset enabled.However, if you want to submit test reports from a full test run to help train the model, run your tests with the
--launchable-record-only flag:1
nosetests --launchable-build-number <BUILD NAME> \
2
--launchable-record-only
Copied!
Subsetting your test runs
First, install the Launchable plugin for nose using PIP:
1
$ pip install nose-launchable
Copied!
To run a meaningful subset of tests, invoke
nosetests with three extra flags:1
# subset tests with Launchable
2
nosetests --launchable-build-number <BUILD NAME> \
3
--launchable-subset \
4
--launchable-subset-options <LAUNCHABLE CLI SUBSET OPTIONS>
Copied!
The
--launchable-build-number flag tells Launchable which build is being tested. This should be the same value you used for --name in launchable record build before. The --launchable-subset flag enables subsetting.The
--launchable-subset-options flag tells the plugin which Launchable CLI subset options to use. For example, to use the Launchable CLI's --target option, the flag should look like --launchable-subset-options '--target 10%'. This runs a subset of the most useful test targets that will run in 10% of the full execution time. You can also use --launchable-subset-options '--time 100' to run a subset of the most useful test targets that will run in 100 seconds. See the CLI reference for more options.The
--launchable-subset-target PERCENTAGE option is still available; it functions the same as --launchable-subset-options '--target PERCENTAGE'. To accommodate future CLI options, however, we recommend using --launchable-subset-options where convenient.Running a subset across parallel nose invocations
Some teams manually split their test suites into several "bins" to run them in parallel. This presents a challenge adopting Launchable, because you don't want to lose the benefit of parallelization. Luckily, with split subsets you can replace your manually selected bins with automatically populated bins from a Launchable subset.
For example, let's say you currently run ~35 minutes of tests split coarsely into four bins and run in parallel across four workers:
- Worker 1: ~20 minutes of tests
- Worker 2: ~15 minutes of tests
With a split subset, you can call Launchable once in each worker to get the bin of tests from a subset for that runner.
The high level flow is:
- 1.
- 2.Manually record a test session against that build and store the value returned from Launchable for use later
- 3.In each nosetests invocation, request the bin of tests that worker should run. To do this, run
nosetestswith:- 1.the
--launchable-test-sessionoption set to the session ID value you saved earlier, and - 2.the
--binvalue in--launchable-subset-optionsset tobin-number/bin-count.
In pseudocode:
1
# main
2
$ launchable record build --name 12345 --source src=.
3
$ launchable record session --build 12345
4
builds/12345/test_sessions/678910
5
​
6
# machine 1
7
$ nosetests --launchable-subset --launchable-test-session builds/12345/test_sessions/678910 --launchable-subset-options "--target 40% --bin 1/2"
8
​
9
# machine 2
10
$ nosetests --launchable-subset --launchable-test-session builds/12345/test_sessions/678910 --launchable-subset-options "--target 40% --bin 2/2"
Copied!
Troubleshooting
If you encounter issues running nose with the Launchable plugin, you can set the
LAUNCHABLE_DEBUG environment variable to 1 before running tests to print debug logs.1
# enable debug logs
2
export LAUNCHABLE_DEBUG=1
3
​
4
# run tests with Launchable
5
nosetests --launchable-subset ...
Copied!
Development
The Launchable nose plugin is open source and available on GitHub. Pull requests are always appreciated!
Last modified 13d ago