Diff Feature Flags Before Running Tests

As I explained in my previous blog post on feature flags and testing, you need the flags to be the same to ensure consistent application behavior during end-to-end tests. If you want to test the specific feature, control the flag from the test. But sometimes the tests unexpectedly fail when one of the developers or QA engineers is testing something and modifies the flag accidentally (and forgets to turn it back off to the default).

Here is how you can quickly understand why some tests failed. Imagine we have a LaunchDarkly project with a flag "testing-launch-darkly-control-from-cypress"

The flag has several variations

Currently, our application is receiving the "Formal" greeting, and the tests confirm it

Store all current feature flags as JSON

In my case, I am using LaunchDarkly and I control it via my cypress-ld-control plugin. This plugin has a CLI utility for grabbing one or more project flags and printing the JSON to STDIO.

$ LAUNCH_DARKLY_AUTH_TOKEN=... npx list-ld-flags --project demo-project --environment test

You can save the produced output into a JSON file

$ npx list-ld-flags --project demo-project --environment test > ld-flags.json

ld-flags.json

{
  "demo-project//test": [
    {
      "key": "testing-launch-darkly-control-from-cypress",
      "name": "Testing LaunchDarkly Control From Cypress",
      "description": "Demo feature flag",
      "kind": "multivariate",
      "deprecated": false,
      "archived": false,
      "temporary": false,
      "defaults": {
        "offVariation": 1,
        "onVariation": 0
      },
      "variations": [
        {
          "_id": "7d910381-5bf1-48c3-9ead-2d99799714ff",
          "description": "Casual greeting",
          "name": "Casual",
          "value": "Hello"
        },
        {
          "_id": "762b2ef1-6f38-48c2-92c9-1d73dc39d472",
          "description": "Formal greeting",
          "name": "Formal",
          "value": "How do you do"
        },
        {
          "_id": "bd529c6d-4287-4128-aaeb-d4dddc273b6b",
          "description": "Vacation greeting",
          "name": "Vacation",
          "value": "Aloha"
        }
      ],
      "environment": {
        "prerequisites": 0,
        "variations": {
          "0": {
            "contextTargets": 0,
            "isFallthrough": true,
            "nullRules": 0,
            "rules": 0,
            "targets": 0
          },
          "1": {
            "contextTargets": 0,
            "isOff": true,
            "nullRules": 0,
            "rules": 0,
            "targets": 0
          }
        }
      }
    }
  ]
}

Tip: you can dump feature flags from multiple projects / environments

$ npx list-ld-flags --project project1,project2,project3 --environment e1,e2,e3
# grabs project1 environment e1
# + project2 environment e2
# + project3 environment e3

Commit the JSON file and store in your source control, as it should not change too often.

Diff flags before running tests

Before launching end-to-end tests, output a diff of the current feature flags and the saved flags JSON file. For example, using GitHub Actions:

- name: LD flag differences
  run: node bin/list-ld-flags.js --environment test --diff ld-flags.json
  env:
    # our CI has the project key as an environment variable
    LAUNCH_DARKLY_PROJECT_KEY: ${{ secrets.LAUNCH_DARKLY_PROJECT_KEY }}
    LAUNCH_DARKLY_AUTH_TOKEN: ${{ secrets.LAUNCH_DARKLY_AUTH_TOKEN }}

Let's say something changes, maybe someone is testing the new greeting and flipped the LaunchDarkly switch. The CI output shows the change:

Nice, if any E2E tests fail we will quickly know the suspected reason. The tool also outputs nice GitHub Actions summary with each flag object's diff:

Easy to spot any potential feature flags that might affect the tests.

Update the flags JSON when needed

If the feature flag defaults change, you might need to update the JSON file. I have a little workflow that the user can run:

.github/workflows/ld-flags.yml

name: ld-flags
on:
  workflow_dispatch:
    inputs:
      update-file:
        description: 'Update the ld-flags.json file with the latest flags from LaunchDarkly'
        required: false
        default: false
        type: boolean

jobs:
  diff-ld-flags:
    runs-on: ubuntu-latest
    steps:
      // check out code and run the NPM install

      - name: Diff LD flags
        run: npx list-ld-flags --environment test --diff ld-flags.json
        env:
          # our CI has the project key as an environment variable
          LAUNCH_DARKLY_PROJECT_KEY: ${{ secrets.LAUNCH_DARKLY_PROJECT_KEY }}
          LAUNCH_DARKLY_AUTH_TOKEN: ${{ secrets.LAUNCH_DARKLY_AUTH_TOKEN }}

      - name: Update ld-flags.json
        if: ${{ github.event.inputs.update-file == 'true' }}
        run: npx list-ld-flags --environment test > ld-flags.json
        env:
          LAUNCH_DARKLY_PROJECT_KEY: ${{ secrets.LAUNCH_DARKLY_PROJECT_KEY }}
          LAUNCH_DARKLY_AUTH_TOKEN: ${{ secrets.LAUNCH_DARKLY_AUTH_TOKEN }}

      - name: Commit updated ld-flags.json
        if: ${{ github.event.inputs.update-file == 'true' }}
        # https://github.com/planetscale/ghcommit-action
        uses: planetscale/[email protected]
        with:
          commit_message: Updated ld-flags.json file
          file_pattern: ld-flags.json
        env:
          GITHUB_TOKEN: ${{ secrets.token }}

Any developer can launch the workflow and check the box to commit the changes

Easy.