Control LaunchDarkly From Cypress Tests

How to set the feature flag values using targeting to test each variation.

Let's say you are using LaunchDarkly to develop and test new web application features behind a flag. You have end-to-end tests too. How do you test the features behind the flag? In this blog post, I will show how to target features using individual user IDs. We will use the plugin cypress-ld-control to set the user ID as an explicit target for the experiment and then confirm the web application behaves correctly using Cypress test.

LaunchDarkly project

I have created a new LaunchDarkly project with project "Demo Project" and two environments. We will concentrate on the "Test" environment.

LaunchDarkly demo project

In the project, I have created a new String feature flag testing-launch-darkly-control-from-cypress with three variations.

Test feature flag variations

Because we want to turn different flag variations for specific users, we will turn on the "Targeting" option. Currently there are no targets yet.

Turn the feature flag targeting on

Demo React application

To demonstrate controlling the feature flags from Cypress tests, I grabbed a copy of the LD's React application. I got a copy using the degit command.

1
2
3
$ npx degit launchdarkly/react-client-sdk/examples/hoc ld-example
> cloned launchdarkly/react-client-sdk#HEAD to ld-example
$ cd ld-example

๐ŸŽ You can find my version of the application used in this blog post in the repo bahmutov/cypress-ld-control-example.

I have changed the code to use my project's Client SDK ID and show the current greeting using the feature flag value. For the demo, I have passed a made-up user ID (in the real application, the user ID would be set after authentication step)

universal/app.js
1
2
3
4
5
6
7
8
9
10
11
import { withLDProvider } from 'launchdarkly-react-client-sdk';
const App = () => (
...
);
// Set clientSideID to your own Client-side ID. You can find this in
// your LaunchDarkly portal under Account settings / Projects
// https://docs.launchdarkly.com/sdk/client-side/javascript#initializing-the-client
const user = {
key: 'USER_1234'
};
export default withLDProvider({ clientSideID: 'YOUR_CLIENT_SIDE_ID', user: 'USER_1234' })(App);

My Home page uses the flag's value to show the greeting.

universal/home.js
1
2
3
4
5
6
7
8
9
10
11
import { withLDConsumer } from 'launchdarkly-react-client-sdk';
const Home = ({ flags }) => (
<Root>
<Heading>{flags.testingLaunchDarklyControlFromCypress}, World</Heading>
<div>
This is a LaunchDarkly React example project. The message above changes the greeting,
based on the current feature flag variation.
</div>
</Root>
);
export default withLDConsumer()(Home);

When I start the application, it shows the default causal greeting

The application uses the casual greeting variation of the flag

Great. Let's target the user USER_1234 with a more formal greeting. At LaunchDarkly app, I will create a new targeting list. Don't forget to save the changes for them to be applied!

Target the user by ID

LaunchDarkly SDK includes real-time updates using server-side events, thus the Home page immediately changes to the formal greeting.

The current user receives its own feature flag value

Nice, let's do the same from a Cypress test.

Cypress setup

Let's install Cypress test runner

1
2
3
4
$ yarn add -D cypress
success Saved 1 new dependency.
info Direct dependencies
โ””โ”€ [email protected]

We will need to control LaunchDarkly flags via HTTP calls. While you can make HTTP calls from Node and from Cypress easily, there is higher-level logic LaunchDarkly uses that makes implementing feature flag changes a chore. I have abstracted everything necessary to add individual user targets into a plugin cypress-ld-control that Cypress tests can use to avoid the complexity. Let's install this plugin and start using it.

1
2
3
$ yarn add -D cypress-ld-control
info Direct dependencies
โ””โ”€ [email protected]

To change the flag values and add individual user targets, the plugin needs to access the LaunchDarkly REST API. We need to make an access token.

LaunchDarkly token

Let's load the plugin and create tasks for the Cypress tests to call using the cy.task - after all, cy.task is very powerful command.

cypress/plugins/index.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
const { initLaunchDarklyApiTasks } = require('cypress-ld-control')
module.exports = (on, config) => {
const tasks = {
// add your other Cypress tasks if any
}

// https://github.com/bahmutov/cypress-ld-control
if (
process.env.LAUNCH_DARKLY_PROJECT_KEY &&
process.env.LAUNCH_DARKLY_AUTH_TOKEN
) {
const ldApiTasks = initLaunchDarklyApiTasks({
projectKey: process.env.LAUNCH_DARKLY_PROJECT_KEY,
authToken: process.env.LAUNCH_DARKLY_AUTH_TOKEN,
environment: 'test', // the key of the environment to use
})
// copy all LaunchDarkly methods as individual tasks
Object.assign(tasks, ldApiTasks)
} else {
console.log('Skipping cypress-ld-control plugin')
}

// register all tasks with Cypress
on('task', tasks)

// IMPORTANT: return the updated config object
return config
}

Whenever we open Cypress locally or run on a continuous integration system, we need to provide two environment variables LAUNCH_DARKLY_PROJECT_KEY and LAUNCH_DARKLY_AUTH_TOKEN. The token is your private secret key we have just created. The project key is the unique string identifying each project shown in the URL and on the https://app.launchdarkly.com/settings/projects page.

LaunchDarkly project key

Sensitive variables

You can store sensitive values and inject them as needed using my as-a CLI tool. In my case, I have .as-a.ini file that is never checked into source control

1
2
3
4
; https://github.com/bahmutov/as-a
[cypress-ld-control-demo]
LAUNCH_DARKLY_PROJECT_KEY=...
LAUNCH_DARKLY_AUTH_TOKEN=...

I open Cypress using the command as-a cypress-ld-control-demo yarn cypress open.

Starting the app and the tests

I typically use start-server-and-test to start the application and open / run Cypress tests. In my package.json file I have set up the command dev:

package.json
1
2
3
4
5
6
{
"scripts": {
"start": "node src/server/index.js",
"dev": "start-test 3000 'cypress open'"
}
}

I run the application and inject the sensitive variables into Cypress tests using the terminal command

1
$ as-a cypress-ld-control-demo yarn run dev

For more, see video Start server and test.

Make LaunchDarkly optional

It is up to you to require these environment variables or gracefully handle it and only require them in some tests. For example, you could set an environment variable if the LaunchDarkly has been initialized:

cypress/plugins/index.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
const { initLaunchDarklyApiTasks } = require('cypress-ld-control')
module.exports = (on, config) => {
// https://github.com/bahmutov/cypress-ld-control
if (
process.env.LAUNCH_DARKLY_PROJECT_KEY &&
process.env.LAUNCH_DARKLY_AUTH_TOKEN
) {
const ldApiTasks = initLaunchDarklyApiTasks({
projectKey: process.env.LAUNCH_DARKLY_PROJECT_KEY,
authToken: process.env.LAUNCH_DARKLY_AUTH_TOKEN,
environment: 'test', // the key of the environment to use
})
// copy all LaunchDarkly methods as individual tasks
Object.assign(tasks, ldApiTasks)
// set an environment variable for specs to use
// to check if the LaunchDarkly can be controlled
config.env.launchDarklyApiAvailable = true
} else {
console.log('Skipping cypress-ld-control plugin')
}

// register all tasks with Cypress
on('task', tasks)

// IMPORTANT: return the updated config object
return config
}

In the test files that really need to call LaunchDarkly API we can check the variable once:

cypress/integration/spec.js
1
2
3
before(() => {
expect(Cypress.env('launchDarklyApiAvailable'), 'LaunchDarkly').to.be.true
})

The plugin API

The "cypress-ld-control" plugin can be used by itself without Cypress to target users. When you initialize the ldApi object, it has the following methods: getFeatureFlag, setFeatureFlagForUser, and others, see README#API section. When using the plugin from Cypress specs, you need to call these methods via the cy.task command. To avoid clashing with other tasks, and conform to the cy.task semantics, the plugin follows the following rules:

  • every task it returns is prefixed with cypress-ld-control: string. Thus you to get the feature flag you would call cy.task('cypress-ld-control:getFeatureFlag') command.
  • every command takes zero or a single options object as an argument, for example: cy.task('cypress-ld-control:setFeatureFlagForUser', ({featureFlagKey, userId, variationIndex}))
  • every command returns either an object or a null, never undefined

The tests

For each experiment variation, I wrote a test placeholder to verify the application's behavior. In every test we can set the feature flag target that specific user, load the application, and check its behavior.

cypress/integration/spec.js
1
2
3
4
5
6
7
8
9
before(() => {
expect(Cypress.env('launchDarklyApiAvailable'), 'LaunchDarkly').to.be.true
})

it('shows the casual greeting')

it('shows formal greeting')

it('shows vacation greeting')

Let's start with the first test. Set the flag, load the app, confirm the user sees a casual greeting heading.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
before(() => {
expect(Cypress.env('launchDarklyApiAvailable'), 'LaunchDarkly').to.be.true
})

const featureFlagKey = 'testing-launch-darkly-control-from-cypress'
const userId = 'USER_1234'

it('shows the casual greeting', () => {
// target the given user to receive the first variation of the feature flag
cy.task('cypress-ld-control:setFeatureFlagForUser', {
featureFlagKey,
userId,
variationIndex: 0,
})
cy.visit('/')
cy.contains('h1', 'Hello, World').should('be.visible')
})

The test sets the feature flag and confirm the user sees the first variation

Similarly, the second test can target the user and confirm the second variant (index 1) is working as expected.

1
2
3
4
5
6
7
8
9
it('shows formal greeting', () => {
cy.task('cypress-ld-control:setFeatureFlagForUser', {
featureFlagKey,
userId,
variationIndex: 1,
})
cy.visit('/')
cy.contains('h1', 'How do you do, World').should('be.visible')
})

Testing the second variation that shows the formal greeting

Finally, when on vacation, the greeting is relaxed

Testing the Aloha greeting

Note: you can see a flash of empty content while the application is fetching the features from LaunchDarkly. See LaunchDarkly docs on how to avoid it. For my simple application it was fine to have the flash.

Cleaning up

When the tests are finished, the last target for the user remains. We should clean up these targets to avoid adding more and more individual test users to LaunchDarkly. I am not sure, but it probably makes it slower to fetch the status for a particular user, and makes the web UI noisier. Let's remove any targeting after all tests are done.

The user target remains after the last test

1
2
3
after(() => {
cy.task('cypress-ld-control:removeUserTarget', { featureFlagKey, userId })
})

Nice, the user target is automatically removed.

The test removed the individual user target after finishing

Note: the after hook runs even if any of the tests fail. The only reason it can be completely skipped is if the test runner crashes.

Inspecting a feature flag

If you are just interested in the feature flag and its variations, you can fetch the flag' state using the "cypress-ld-control:getFeatureFlag" task

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
it('shows vacation greeting', () => {
cy.task('cypress-ld-control:setFeatureFlagForUser', {
featureFlagKey,
userId,
variationIndex: 2,
})
cy.visit('/')
cy.contains('h1', 'Aloha, World').should('be.visible')

// print the current state of the feature flag and its variations
cy.task('cypress-ld-control:getFeatureFlag', featureFlagKey)
.then(console.log)
// let's print the variations to the Command Log side panel
.its('variations')
.then((variations) => {
variations.forEach((v, k) => {
cy.log(`${k}: ${v.name} is ${v.value}`)
})
})
})

The entire feature flag object is quite large, as we see in the DevTools console. The highlighted variations are visible in the Command Log.

The feature flag object

Running tests on CI

Let's use GitHub Actions to run the same tests on CI. I will use cypress-io/github-action to install the dependencies, cache Cypress, start the application, and run the tests. I will need to set the same environment secrets in the repo.

Setting secrets on GitHub

The CI workflow ci.yml uses the GH action and injects the secrets as environment variables.

.github/workflows/ci.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
name: ci
on: push
jobs:
test:
runs-on: ubuntu-20.04
steps:
- name: Checkout ๐Ÿ›Ž
uses: actions/[email protected]

- name: Run tests ๐Ÿงช
# https://github.com/cypress-io/github-action
uses: cypress-io/[email protected]
with:
start: 'yarn start'
env:
LAUNCH_DARKLY_PROJECT_KEY: ${{ secrets.LAUNCH_DARKLY_PROJECT_KEY }}
LAUNCH_DARKLY_AUTH_TOKEN: ${{ secrets.LAUNCH_DARKLY_AUTH_TOKEN }}

The terminal output on CI shows the messages from cypress-ld-control plugin as it calls LaunchDarkly API

The plugin logs its LaunchDarkly operations

Single test

Because LaunchDarkly client-side SDK includes real-time updates, we can write a single test that goes through every variation of the flag without visiting the page again or even reloading it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
it.only('shows all greetings', () => {
cy.visit('/')
cy.task('cypress-ld-control:setFeatureFlagForUser', {
featureFlagKey,
userId,
variationIndex: 0,
})
cy.contains('h1', 'Hello, World')
.should('be.visible')
// I have added one second waits for clarity
.wait(1000)

cy.task('cypress-ld-control:setFeatureFlagForUser', {
featureFlagKey,
userId,
variationIndex: 1,
})
cy.contains('h1', 'How do you do, World').should('be.visible').wait(1000)

cy.task('cypress-ld-control:setFeatureFlagForUser', {
featureFlagKey,
userId,
variationIndex: 2,
})
cy.contains('h1', 'Aloha, World').should('be.visible')
})

A single test will all feature flag states

Note: the plugin "cypress-ld-control" handles LaunchDarkly rate-limiting, retrying API calls if the test runner receives 429 HTTP response code.

Note 2: in the test above you see a flash of "Hello, World" default flag state between the formal and the vacation greetings. The plugin automatically removes the current user target from a variation before adding it to another one (the same user cannot be target of two variations at the same time). Thus you see that brief moment between the remove and add commands.

See also