Managing complex test session layouts

Background

The test session is one of Launchable's core concepts. When you request a subset of tests from Launchable, the subset is linked to a test session. Similarly, you record test results, those results are recorded to the same test session. This concept is useful because tests might run several times against the same build; it helps disambiguate those runs.
By default, the Launchable CLI handles creating, saving, and retrieving a session in the background. When you run launchable subset or launchable record tests, the CLI checks for an existing session file in ~/.config/launchable. If it finds one, it uses it. If not, the CLI gets a new session ID from Launchable and saves it to a file in ~/.config/launchable.
Recording tests:
Subsetting and recording tests:
This ensures that launchable subset and launchable record tests commands are 'linked', which is important for proper instrumentation.
However, there are a few cases where you might need to override this implicit behavior and create a session explicitly:

Test report collection happens on a different machine

Sometimes, the build, test, and reporting steps are split between different machines. This can cause two issues:

Extra sessions

This can cause an extra session to be created, because launchable record tests doesn't find the session created by launchable subset:

Reused sessions

Or, if the reporting machine is reused and persistent, the saved session from launchable record testsmay be accidentally reused, combining tests from different sessions against the same build:

Solution

The best way to resolve these issues is to use launchable record session to explicitly create and store a session before you run your tests. You'll use that value with launchable subset and then pass this it through to the reporting phase.
You'll use --session instead of --build in launchable subset and launchable record tests. Here's some pseudocode to illustrate:
1
# machine 1
2
3
# build step
4
launchable record build --name <BUILD NAME> [OPTIONS]
5
6
# build software
7
bundle install
8
9
# machine 2
10
11
# before running tests, save a session token
12
# you'll use this token again later
13
launchable record session --build <BUILD NAME> > launchable-session.txt
14
15
# get a subset, if applicable
16
launchable subset --session $(cat launchable-session.txt) [OPTIONS]
17
18
# run tests
19
bundle exec rails test [OPTIONS]
20
21
# machine 3
22
23
# record test results
24
launchable record tests --session $(cat launchable-session.txt) [OPTIONS]
Copied!

Combining test reports from multiple runs

Some pipelines execute multiple test runs against a build in a single test session, outputting distinct test report(s) across several machines.
For example, one test session might include a Cypress run, a GoogleTest run, and a Bazel run, all executed separately, generating their own test report XML file(s) on separate machines.
These are all ostensibly part of a single test session, so it's desirable to treat them as such.
This may also be the case if you execute tests of a single type across several parallel runs, but usually the test runner can combine reports from parallel runs for consumption from a single place.
If all the test reports for a session can be collected from a single machine, you don't need to use this method.
So, if you need to capture test reports from several machines, you can tell Launchable that they all relate to the same test session using the launchable record session command and the corresponding --session parameter (note: pseudocode):
1
## build step
2
3
# before building software, send commit and build info
4
# to Launchable
5
launchable record build --build <BUILD NAME> [OPTIONS]
6
7
# build software the way you normally do, for example
8
bundle install
9
10
## test step
11
12
# before running tests, save a session token
13
# you'll use this token to group the test reports together later
14
launchable record session --build <BUILD NAME> > launchable-session.txt
15
16
# start multiple test runs
17
18
# machine 1
19
20
# run tests
21
bundle exec rails test
22
23
# send test results to Launchable from machine 1
24
# Note: You need to configure the line to always run whether test run succeeds/fails.
25
# See each integration page.
26
launchable record tests --session $(cat launchable-session.txt) [OPTIONS]
27
28
# machine 2
29
30
# run tests
31
bundle exec rails test
32
33
# send test results to Launchable from machine 2
34
# Note: You need to configure the line to always run whether test run succeeds/fails.
35
# See each integration page.
36
launchable record tests --session $(cat launchable-session.txt) [OPTIONS]
37
38
## repeat as needed...
39
40
## finish multiple test runs
Copied!
You can read more about launchable record session in the CLI reference.
Last modified 1mo ago