products

Control LaunchDarkly From Cypress Tests

How to set the feature flag values using targeting to test each variation.

Let's say you are using LaunchDarkly to develop and test new web application features behind a flag. You have end-to-end tests too. How do you test the features behind the flag? In this blog post, I will show how to target features using individual user IDs. We will use the plugin cypress-ld-control to set the user ID as an explicit target for the experiment and then confirm the web application behaves correctly using Cypress test.

LaunchDarkly project

I have created a new LaunchDarkly project with project "Demo Project" and two environments. We will concentrate on the "Test" environment.

LaunchDarkly demo project

In the project, I have created a new String feature flag testing-launch-darkly-control-from-cypress with three variations.

Test feature flag variations

Because we want to turn different flag variations for specific users, we will turn on the "Targeting" option. Currently there are no targets yet.

Turn the feature flag targeting on

Demo React application

To demonstrate controlling the feature flags from Cypress tests, I grabbed a copy of the LD's React application. I got a copy using the degit command.

1
2
3
$ npx degit launchdarkly/react-client-sdk/examples/hoc ld-example
> cloned launchdarkly/react-client-sdk#HEAD to ld-example
$ cd ld-example

🎁 You can find my version of the application used in this blog post in the repo bahmutov/cypress-ld-control-example.

I have changed the code to use my project's Client SDK ID and show the current greeting using the feature flag value. For the demo, I have passed a made-up user ID (in the real application, the user ID would be set after authentication step)

universal/app.js
1
2
3
4
5
6
7
8
9
10
11
import { withLDProvider } from 'launchdarkly-react-client-sdk';
const App = () => (
  ...
);
// Set clientSideID to your own Client-side ID. You can find this in
// your LaunchDarkly portal under Account settings / Projects
// https://docs.launchdarkly.com/sdk/client-side/javascript#initializing-the-client
const user = {
  key: 'USER_1234'
};
export default withLDProvider({ clientSideID: 'YOUR_CLIENT_SIDE_ID', user: 'USER_1234' })(App);

My Home page uses the flag's value to show the greeting.

universal/home.js
1
2
3
4
5
6
7
8
9
10
11
import { withLDConsumer } from 'launchdarkly-react-client-sdk';
const Home = ({ flags }) => (
  <Root>
    <Heading>{flags.testingLaunchDarklyControlFromCypress}, World</Heading>
    <div>
      This is a LaunchDarkly React example project. The message above changes the greeting,
      based on the current feature flag variation.
    </div>
  </Root>
);
export default withLDConsumer()(Home);

When I start the application, it shows the default causal greeting

The application uses the casual greeting variation of the flag

Great. Let's target the user USER_1234 with a more formal greeting. At LaunchDarkly app, I will create a new targeting list. Don't forget to save the changes for them to be applied!

Target the user by ID

LaunchDarkly SDK includes real-time updates using server-side events, thus the Home page immediately changes to the formal greeting.

The current user receives its own feature flag value

Nice, let's do the same from a Cypress test.

Cypress setup

Let's install Cypress test runner

1
2
3
4
$ yarn add -D cypress
success Saved 1 new dependency.
info Direct dependencies
└─ [email protected]

We will need to control LaunchDarkly flags via HTTP calls. While you can make HTTP calls from Node and from Cypress easily, there is higher-level logic LaunchDarkly uses that makes implementing feature flag changes a chore. I have abstracted everything necessary to add individual user targets into a plugin cypress-ld-control that Cypress tests can use to avoid the complexity. Let's install this plugin and start using it.

1
2
3
$ yarn add -D cypress-ld-control
info Direct dependencies
└─ [email protected]

To change the flag values and add individual user targets, the plugin needs to access the LaunchDarkly REST API. We need to make an access token.

LaunchDarkly token

Let's load the plugin and create tasks for the Cypress tests to call using the cy.task - after all, cy.task is very powerful command.

cypress/plugins/index.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
const { initLaunchDarklyApiTasks } = require('cypress-ld-control')
module.exports = (on, config) => {
  const tasks = {
    // add your other Cypress tasks if any
  }

  // https://github.com/bahmutov/cypress-ld-control
  if (
    process.env.LAUNCH_DARKLY_PROJECT_KEY &&
    process.env.LAUNCH_DARKLY_AUTH_TOKEN
  ) {
    const ldApiTasks = initLaunchDarklyApiTasks({
      projectKey: process.env.LAUNCH_DARKLY_PROJECT_KEY,
      authToken: process.env.LAUNCH_DARKLY_AUTH_TOKEN,
      environment: 'test', // the key of the environment to use
    })
    // copy all LaunchDarkly methods as individual tasks
    Object.assign(tasks, ldApiTasks)
  } else {
    console.log('Skipping cypress-ld-control plugin')
  }

  // register all tasks with Cypress
  on('task', tasks)

  // IMPORTANT: return the updated config object
  return config
}

Whenever we open Cypress locally or run on a continuous integration system, we need to provide two environment variables LAUNCH_DARKLY_PROJECT_KEY and LAUNCH_DARKLY_AUTH_TOKEN. The token is your private secret key we have just created. The project key is the unique string identifying each project shown in the URL and on the https://app.launchdarkly.com/settings/projects page.

LaunchDarkly project key

Sensitive variables

You can store sensitive values and inject them as needed using my as-a CLI tool. In my case, I have .as-a.ini file that is never checked into source control

1
2
3
4
; https://github.com/bahmutov/as-a
[cypress-ld-control-demo]
LAUNCH_DARKLY_PROJECT_KEY=...
LAUNCH_DARKLY_AUTH_TOKEN=...

I open Cypress using the command as-a cypress-ld-control-demo yarn cypress open.

Starting the app and the tests

I typically use start-server-and-test to start the application and open / run Cypress tests. In my package.json file I have set up the command dev:

package.json
1
2
3
4
5
6
{
  "scripts": {
    "start": "node src/server/index.js",
    "dev": "start-test 3000 'cypress open'"
  }
}

I run the application and inject the sensitive variables into Cypress tests using the terminal command

1
$ as-a cypress-ld-control-demo yarn run dev

For more, see video Start server and test.

Make LaunchDarkly optional

It is up to you to require these environment variables or gracefully handle it and only require them in some tests. For example, you could set an environment variable if the LaunchDarkly has been initialized:

cypress/plugins/index.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
const { initLaunchDarklyApiTasks } = require('cypress-ld-control')
module.exports = (on, config) => {
  // https://github.com/bahmutov/cypress-ld-control
  if (
    process.env.LAUNCH_DARKLY_PROJECT_KEY &&
    process.env.LAUNCH_DARKLY_AUTH_TOKEN
  ) {
    const ldApiTasks = initLaunchDarklyApiTasks({
      projectKey: process.env.LAUNCH_DARKLY_PROJECT_KEY,
      authToken: process.env.LAUNCH_DARKLY_AUTH_TOKEN,
      environment: 'test', // the key of the environment to use
    })
    // copy all LaunchDarkly methods as individual tasks
    Object.assign(tasks, ldApiTasks)
    // set an environment variable for specs to use
    // to check if the LaunchDarkly can be controlled
    config.env.launchDarklyApiAvailable = true
  } else {
    console.log('Skipping cypress-ld-control plugin')
  }

  // register all tasks with Cypress
  on('task', tasks)

  // IMPORTANT: return the updated config object
  return config
}

In the test files that really need to call LaunchDarkly API we can check the variable once:

cypress/integration/spec.js
1
2
3
before(() => {
  expect(Cypress.env('launchDarklyApiAvailable'), 'LaunchDarkly').to.be.true
})

The plugin API

The "cypress-ld-control" plugin can be used by itself without Cypress to target users. When you initialize the ldApi object, it has the following methods: getFeatureFlag, setFeatureFlagForUser, and others, see README#API section. When using the plugin from Cypress specs, you need to call these methods via the cy.task command. To avoid clashing with other tasks, and conform to the cy.task semantics, the plugin follows the following rules:

  • every task it returns is prefixed with cypress-ld-control: string. Thus you to get the feature flag you would call cy.task('cypress-ld-control:getFeatureFlag') command.
  • every command takes zero or a single options object as an argument, for example: cy.task('cypress-ld-control:setFeatureFlagForUser', ({featureFlagKey, userId, variationIndex}))
  • every command returns either an object or a null, never undefined

The tests

For each experiment variation, I wrote a test placeholder to verify the application's behavior. In every test we can set the feature flag target that specific user, load the application, and check its behavior.

cypress/integration/spec.js
1
2
3
4
5
6
7
8
9
before(() => {
  expect(Cypress.env('launchDarklyApiAvailable'), 'LaunchDarkly').to.be.true
})

it('shows the casual greeting')

it('shows formal greeting')

it('shows vacation greeting')

Let's start with the first test. Set the flag, load the app, confirm the user sees a casual greeting heading.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
before(() => {
  expect(Cypress.env('launchDarklyApiAvailable'), 'LaunchDarkly').to.be.true
})

const featureFlagKey = 'testing-launch-darkly-control-from-cypress'
const userId = 'USER_1234'

it('shows the casual greeting', () => {
  // target the given user to receive the first variation of the feature flag
  cy.task('cypress-ld-control:setFeatureFlagForUser', {
    featureFlagKey,
    userId,
    variationIndex: 0,
  })
  cy.visit('/')
  cy.contains('h1', 'Hello, World').should('be.visible')
})

The test sets the feature flag and confirm the user sees the first variation

Similarly, the second test can target the user and confirm the second variant (index 1) is working as expected.

1
2
3
4
5
6
7
8
9
it('shows formal greeting', () => {
  cy.task('cypress-ld-control:setFeatureFlagForUser', {
    featureFlagKey,
    userId,
    variationIndex: 1,
  })
  cy.visit('/')
  cy.contains('h1', 'How do you do, World').should('be.visible')
})

Testing the second variation that shows the formal greeting

Finally, when on vacation, the greeting is relaxed

Testing the Aloha greeting

Note: you can see a flash of empty content while the application is fetching the features from LaunchDarkly. See LaunchDarkly docs on how to avoid it. For my simple application it was fine to have the flash.

Cleaning up

When the tests are finished, the last target for the user remains. We should clean up these targets to avoid adding more and more individual test users to LaunchDarkly. I am not sure, but it probably makes it slower to fetch the status for a particular user, and makes the web UI noisier. Let's remove any targeting after all tests are done.

The user target remains after the last test

1
2
3
after(() => {
  cy.task('cypress-ld-control:removeUserTarget', { featureFlagKey, userId })
})

Nice, the user target is automatically removed.

The test removed the individual user target after finishing

Note: the after hook runs even if any of the tests fail. The only reason it can be completely skipped is if the test runner crashes.

Inspecting a feature flag

If you are just interested in the feature flag and its variations, you can fetch the flag' state using the "cypress-ld-control:getFeatureFlag" task

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
it('shows vacation greeting', () => {
  cy.task('cypress-ld-control:setFeatureFlagForUser', {
    featureFlagKey,
    userId,
    variationIndex: 2,
  })
  cy.visit('/')
  cy.contains('h1', 'Aloha, World').should('be.visible')

  // print the current state of the feature flag and its variations
  cy.task('cypress-ld-control:getFeatureFlag', featureFlagKey)
    .then(console.log)
    // let's print the variations to the Command Log side panel
    .its('variations')
    .then((variations) => {
      variations.forEach((v, k) => {
        cy.log(`${k}: ${v.name} is ${v.value}`)
      })
    })
})

The entire feature flag object is quite large, as we see in the DevTools console. The highlighted variations are visible in the Command Log.

The feature flag object

Running tests on CI

Let's use GitHub Actions to run the same tests on CI. I will use cypress-io/github-action to install the dependencies, cache Cypress, start the application, and run the tests. I will need to set the same environment secrets in the repo.

Setting secrets on GitHub

The CI workflow ci.yml uses the GH action and injects the secrets as environment variables.

.github/workflows/ci.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
name: ci
on: push
jobs:
  test:
    runs-on: ubuntu-20.04
    steps:
      - name: Checkout 🛎
        uses: actions/checkout@v2

      - name: Run tests 🧪
        # https://github.com/cypress-io/github-action
        uses: cypress-io/github-action@v3
        with:
          start: 'yarn start'
        env:
          LAUNCH_DARKLY_PROJECT_KEY: ${{ secrets.LAUNCH_DARKLY_PROJECT_KEY }}
          LAUNCH_DARKLY_AUTH_TOKEN: ${{ secrets.LAUNCH_DARKLY_AUTH_TOKEN }}

The terminal output on CI shows the messages from cypress-ld-control plugin as it calls LaunchDarkly API

The plugin logs its LaunchDarkly operations

Single test

Because LaunchDarkly client-side SDK includes real-time updates, we can write a single test that goes through every variation of the flag without visiting the page again or even reloading it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
it.only('shows all greetings', () => {
  cy.visit('/')
  cy.task('cypress-ld-control:setFeatureFlagForUser', {
    featureFlagKey,
    userId,
    variationIndex: 0,
  })
  cy.contains('h1', 'Hello, World')
    .should('be.visible')
    // I have added one second waits for clarity
    .wait(1000)

  cy.task('cypress-ld-control:setFeatureFlagForUser', {
    featureFlagKey,
    userId,
    variationIndex: 1,
  })
  cy.contains('h1', 'How do you do, World').should('be.visible').wait(1000)

  cy.task('cypress-ld-control:setFeatureFlagForUser', {
    featureFlagKey,
    userId,
    variationIndex: 2,
  })
  cy.contains('h1', 'Aloha, World').should('be.visible')
})

A single test will all feature flag states

Note: the plugin "cypress-ld-control" handles LaunchDarkly rate-limiting, retrying API calls if the test runner receives 429 HTTP response code.

Note 2: in the test above you see a flash of "Hello, World" default flag state between the formal and the vacation greetings. The plugin automatically removes the current user target from a variation before adding it to another one (the same user cannot be target of two variations at the same time). Thus you see that brief moment between the remove and add commands.

See also

Bonus 1: control the feature lifetime

When someone introduces a new experiment behind the feature flag, one has to be careful not to break all existing tests. Here is what I think the feature lifetime should be:

  1. an experiment A new feature flag is added to enable the new behavior. At first, the developer is experimenting with the behavior, thus the feature is strictly opt-in. All existing users are seeing the existing default behavior. No updates to the tests are necessary.
  2. a prototype The new feature seems to be a success and has been given a go. Now the team is implementing it and is planning to release it. A few end-to-end tests are added that use the LD plugin to turn the feature on and test the new feature flow. All existing tests are still seeing the old behavior because the old behavior is the default one.
  3. an alternative In this stage, more and more users are seeing the new feature, and the old behavior will be removed in the future. We need to start thinking about the test changes. We now switch the existing tests to explicitly opt-in to turn the old behavior. So some tests opt-in and test the new feature, and other tests opt-in and test the new feature.
  4. the switch The feature is being turned on by default for most or all users. The tests written while developing the feature are now working without opt-in. The old tests are still running with opt-in to the old behavior.
  5. the removal The old behavior is removed and all old tests are disabled. The feature flag is now always pointing to the new behavior.

Bonus 2: custom commands

To simplify using the plugin, I have added Cypress custom commands to get, set, and remove feature flags.

1
2
3
4
5
6
if (Cypress.isLaunchDarklyControlInitialized()) {
  // we can control the LaunchDarkly flags
}
cy.getFeatureFlag(featureFlagKey).then(flag => ...)
cy.setFeatureFlagForUser(featureFlagKey, userId, variationIndex)
cy.removeUserTarget(featureFlagKey, userId)

See the README for the latest command syntax.

The plugin can also control multiple LaunchDarkly projects at once. When initializing the plugin from the plugins file or from the cypress.config.js file, list the projects you want to control (and their environments):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
// list all the LD projects you want to use
const projects = [
  {
    name: 'web-flags',
    projectKey: 'default',
    environment: 'test',
  },
  {
    projectKey: 'api-project',
    environment: 'test',
  },
]

initCypressMultipleProjects(projects, on, config)

From the spec file, use the name or the key of the project you want to use when controlling a feature flag.

1
2
cy.getFeatureFlag('my-flag', 'default')
cy.getFeatureFlag('my-flag', 'web-flags')