products

Keep passwords secret in E2E tests

How to pass passwords and tokens during Cypress tests to avoid accidentally revealing them in screenshots, videos and logs

Note: source code for this blog post is the repository bahmutov/keep-password-secret.

Visible password

Imagine we have a password-protected web application. For example I have cloned passport/express-4.x-local-example - you need to enter login and password to see your personal profile. We can easily write an end-to-end test for this application using Cypress test runner.

cypress/integration/login-spec.js
1
2
3
4
5
6
7
8
9
10
it('logs in', () => {
  cy.visit('/login')
  cy.get('[name=username]').type('jack')
  cy.get('[name=password]').type('secret')
  cy.get('[type=Submit]').click()

  cy.contains('a', 'profile').should('be.visible').click()
  cy.url().should('match', /profile$/)
  cy.contains('Email: [email protected]')
})

This test passes, as you can see from the test run video. It also shows the danger: the password is clearly shown in the video during "TYPE" command!

Login spec showing the password during cy.type

In this blog post I will show how to ways to mitigate the security risk of showing secret information like the password text during end-to-end tests. In this case, the password is local - and it is probably fine to show it in plain text. But in other situations, when testing staging or production environment, the password to the test account should NOT be visible.

Do not hardcode passwords

The first thing I would do is to remove the hard-coded password value from the test files. Right now we have the password in each spec file source code as a string cy.type('secret'), and instead I will pass it as an environment variable.

cypress/integration/no-password-spec.js
1
2
3
4
5
6
7
8
9
10
11
12
13
it('logs in using env variables', () => {
  const username = Cypress.env('username')
  const password = Cypress.env('password')

  cy.visit('/login')
  cy.get('[name=username]').type(username)
  cy.get('[name=password]').type(password)
  cy.get('[type=Submit]').click()

  cy.contains('a', 'profile').should('be.visible').click()
  cy.url().should('match', /profile$/)
  cy.contains('Email: [email protected]')
})

I believe in most cases, the username can be public, while password should be secret. Thus it is ok to store the username in the env object of cypress.json file. And for completeness and clarity, I store an empty value for the password key - this tells anyone reading the tests to expect to pass the password to make the tests work.

cypress.json
1
2
3
4
5
6
7
{
  "baseUrl": "http://localhost:3000",
  "env": {
    "username": "jack",
    "password": ""
  }
}

Great - if we run Cypress now, the test will fail, because cy.type refuses to type an empty string.

cy.type will not type an empty password string

Cypress allows several ways to pass the environment variables, in this case, the secure way is to use an environment variable CYPRESS_password=... when running Cypress. Cypress will stick all unknown environment variables that start with prefix CYPRESS_ into the env object automatically.

1
$ CYPRESS_password=secret npx cypress open|run

When running tests locally I strongly recommend using my as-a utility to load groups of environment variables conveniently and cross-platform. In file ~/.as-a/.as-a.ini I will add one more INI section and will place the secret password variable there.

~/.as-a/.as-a.ini
1
2
3
[kps]
; group of environment variables for keep-password-secret app
CYPRESS_password=secret

When running Cypress from my terminal I use

1
as-a kps npx cypress open

The password variable (and any other values in the block kps) will be injected just for the duration of the above command.

On Continuous Integration server, just set a secure environment variable CYPRESS_password to value secret. Most CIs mask such values automatically in the logs.

Validate password

If the user forgets to open Cypress with CYPRESS_password=..., or if the variable is not set on CI we get a cryptic error "TYPE cannot type an empty string". It would be much nicer to give a meaningful error if the password string is empty, right. Here is an updated test that first verifies the username and password values before using them.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
it('logs in using env variables', () => {
  const username = Cypress.env('username')
  const password = Cypress.env('password')

  expect(username, 'username was set').to.be.a('string').and.not.be.empty
  expect(password, 'password was set').to.be.a('string').and.not.be.empty

  cy.visit('/login')
  cy.get('[name=username]').type(username)
  cy.get('[name=password]').type(password)
  cy.get('[type=Submit]').click()

  cy.contains('a', 'profile').should('be.visible').click()
  cy.url().should('match', /profile$/)
  cy.contains('Email: [email protected]')
})

It is working - if the developer is running Cypress test without CYPRESS_password=... they will get a nice error.

Explicit empty password error

But this is dangerous too - notice that the username assertion expect(username, 'username was set').to.be.a('string').and.not.be.empty prints the value in the Command Log. If the password is set it will be visible in the video and any screenshot.

Actual password is shown when assertion passes

To avoid assertion values reflected in the Command Log, we must throw an error ourselves like this.

1
2
3
4
5
6
7
8
9
const username = Cypress.env('username')
const password = Cypress.env('password')

// it is ok for the username to be visible in the Command Log
expect(username, 'username was set').to.be.a('string').and.not.be.empty
// but the password value should not be shown
if (typeof password !== 'string' || !password) {
  throw new Error('Missing password value, set using CYPRESS_password=...')
}

Finally, we can avoid cy.type(password) showing the value it is typing by using cy.type(password, {log: false}) option.

1
cy.get('[name=password]').type(password, {log: false})

Now the Command Log will not reveal the secret password value in the video or screenshot

Password is no longer shown

Finally, sometimes I see assertions chained at the end of cy.type to confirm right away the value of the input element like this:

1
2
3
4
5
6
cy.get('[name=username]')
  .type(username)
  .should('have.value', username)
cy.get('[name=password]')
  .type(password, {log: false})
  .should('have.value', password)

The assertion .should('have.value', ...) also prints the value in the Command Log as shown below, revealing the password, even if just part of it in this case.

Implicit assertion revealing a part of the asserted value

To avoid this, again we need to throw our own error by using should(callback) assertion form.

1
2
3
4
5
6
7
cy.get('[name=password]')
  .type(password, { log: false })
  .should(el$ => {
    if (el$.val() !== password) {
      throw new Error('Different value of typed password')
    }
  })

The custom errors are not shown in the Command Log.

Should with callback function with custom error does not reveal value

Avoid UI login

It is fine to test the Login user interface - but every test after that should NOT use UI to log in. It is slow and unnecessary. Instead, check how the web application performs the login by looking at the DevTools during UI login. In our case, the application is performing a POST request to /login with the username and password form.

Inspecting login network call to find POST form submission

Find the recipe that matches this method among Cypress Logging in recipes. In this case it is HTML Web Form recipe. Copying the recipe to get the following test that uses cy.request to make the form submission post and automatically sets any returned cookies.

cypress/integration/api-login-spec.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
it('logs in using cy.request', () => {
  const username = Cypress.env('username')
  const password = Cypress.env('password')

  // it is ok for the username to be visible in the Command Log
  expect(username, 'username was set').to.be.a('string').and.not.be.empty
  // but the password value should not be shown
  if (typeof password !== 'string' || !password) {
    throw new Error('Missing password value, set using CYPRESS_password=...')
  }

  cy.request({
    method: 'POST',
    url: '/login',
    form: true,
    body: {
      username,
      password
    }
  })
  cy.getCookie('connect.sid').should('exist')

  // now visit the profile page
  cy.visit('/profile').contains('Email: [email protected]')
})

Test that logs in using cy.request API call

Beautiful, it is working, and we can make the login reusable by making it into a Cypress custom command or by creating a reusable function (which is simpler in my opinion). I will create and export login function from cypress/support/index.js file to be used in any test that wants to log in via API. Only tests that explicitly test the login page should not use this function - the rest can quickly login without going through the UI.

cypress/support/index.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
/**
 * Logs the user by making API call to POST /login.
 * Make sure "cypress.json" + CYPRESS_ environment variables
 * have username and password values set.
 */
export const login = () => {
  const username = Cypress.env('username')
  const password = Cypress.env('password')

  // it is ok for the username to be visible in the Command Log
  expect(username, 'username was set').to.be.a('string').and.not.be.empty
  // but the password value should not be shown
  if (typeof password !== 'string' || !password) {
    throw new Error('Missing password value, set using CYPRESS_password=...')
  }

  cy.request({
    method: 'POST',
    url: '/login',
    form: true,
    body: {
      username,
      password
    }
  })
  cy.getCookie('connect.sid').should('exist')
}

I really like small reusable function, partly because I can document them using JSDoc comments, as above. This gives me intelligent code completion in any place I import and use login function, for example from the spec file.

Hovering over `login` shows JSDoc comments

Continuous integration

To run Cypress test on CI I will use CircleCI - it is simple to set up, especially by using Cypress CircleCI Orb.

circle.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
version: 2.1
orbs:
  cypress: cypress-io/cypress@1
workflows:
  build:
    jobs:
      - cypress/run:
          start: 'npm start'
          wait-on: 'http://localhost:3000'
          # save test run videos and screenshots on CircleCI
          # and they are publicly viewable, so make sure there
          # are no passwords in clear text!
          store_artifacts: true

You can find the project at https://circleci.com/gh/bahmutov/keep-password-secret. The first run fails - because I have not set CYPRESS_password environment variable yet.

Run fails because we have not set CYPRESS_password yet

We can set the required environment variable on CircleCI, but I really like the new CircleCI security contexts because they:

  • allow explicitly listing the context that a job expects in the circle.yml file
  • inject or stop the job depending on the security permission of the context

So I will create a new security context keep-password-secret and will add the variable password there

Created security context with the password environment variable

Tip: longer passwords are better, because UI masking still reveals a half of the word!

Change the cypress/run job by requiring the new context we have created

circle.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
version: 2.1
orbs:
  cypress: cypress-io/cypress@1
workflows:
  build:
    jobs:
      - cypress/run:
          context: keep-password-secret
          start: 'npm start'
          wait-on: 'http://localhost:3000'
          # save test run videos and screenshots on CircleCI
          # and they are publicly viewable, so make sure there
          # are no passwords in clear text!
          store_artifacts: true

And the tests pass! Check out the test artifacts - you can see password in the video login-spec.js.mp4 that shows the original, insecure spec file.

Password is visible in the public video stored on the CI

But the password should not be visible in any other video - because we have modified the spec code to be less "chatty".

If you are using GitHub Actions, you can set the secrets using GitHub UI, and then pass explicitly each secret to an environment variable:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
name: Cypress tests
on: push
jobs:
  cypress-run:
    name: Cypress run
    runs-on: ubuntu-22.04
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Cypress run
        # https://github.com/cypress-io/github-action
        uses: cypress-io/github-action@v6
        with:
          record: true
        env:
          # pass the Cypress Cloud record key as an environment variable
          CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
          # password on CI to be set as Cypress.env('password')
          CYPRESS_password: ${{ secrets.USER_PASSWORD }}

Tip: the more about Cypress and GitHub Actions, take my course Testing The Swag Store.

Plugins

Watch out for any plugins you use with Cypress Test Runner - they might be printing secrets too. For example, I will add cypress-failed-log plugin that prints commands to the terminal.

1
$ npm i -D cypress-failed-log

The add to the support file the following

cypress/support/index.js
1
require('cypress-failed-log')

and to the plugins file

cypport/plugins/index.js
1
2
3
4
5
6
7
module.exports = (on, config) => {
  // `on` is used to hook into various events Cypress emits
  // `config` is the resolved Cypress config
  on('task', {
    failed: require('cypress-failed-log/src/failed')()
  })
}

Let's try this module. Let's run it locally WITH CYPRESS_password=secret set. I will be using npm test script

package.json
1
2
3
4
5
6
7
{
  "scripts": {
    "start": "node ./server",
    "dev": "start-test 3000 'cypress open'",
    "test": "start-test 3000 'cypress run'"
  }
}

When running as-a kps npm test everything runs ok - nothing sensitive is printed to the terminal. But make a test fail - for example like this

cypress/integration/no-password-spec.js
1
2
3
4
5
6
7
8
it('logs in using env variables', () => {
  const username = Cypress.env('username')
  const password = Cypress.env('password')

  ... the rest of the test
  // and now an incorrect assertion
  cy.contains('Email: [email protected]')
})

Let's test this - to make sure no sensitive information is printed on failure.

1
2
$ as-a kps npx start-test 3000 \
  'cypress run --spec cypress/integration/no-password-spec.js'

The spec runs, fails - and the cypress-failed-log plugin prints the each command - but does not reveal anything sensitive.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
  1) logs in using env variables
assert username was set: expected jack to be a string
assert username was set: expected jack not to be empty
visit /login
get [name=username]
type jack
assert expected <input> to have value jack
get [name=password]
get [type=Submit]
click
form sub --submitting form--
page load --waiting for new page to load--
new url http://localhost:3000/
contains a, profile
assert expected <a> to be visible
click
page load --waiting for new page to load--
new url http://localhost:3000/profile
url
assert expected http://localhost:3000/profile to match /profile$/
contains Email: [email protected]

  0 passing (8s)
  1 failing

  1)  logs in using env variables:
     CypressError: Timed out retrying: Expected to find content: 'Email: [email protected]' but never did.

Perfect. This plugin also saves a JSON file with the error - and the file does not reveal the password either.

cypress/logs/failed-logs-in-using-env-variables.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "specName": "no-password-spec.js",
  "title": "logs in using env variables",
  "suiteName": "",
  "testName": " logs in using env variables",
  "testError": "Timed out retrying: Expected to find content: 'Email: [email protected]' but never did.",
  "testCommands": [
    "assert username was set: expected **jack** to be a string",
    "assert username was set: expected **jack** not to be empty",
    "visit /login",
    "get [name=username]",
    "type jack",
    "assert expected **<input>** to have value **jack**",
    "get [name=password]",
    "get [type=Submit]",
    "click ",
    "form sub --submitting form--",
    "page load --waiting for new page to load--",
    "new url http://localhost:3000/",
    "contains a, profile",
    "assert expected **<a>** to be **visible**",
    "click ",
    "page load --waiting for new page to load--",
    "new url http://localhost:3000/profile",
    "url ",
    "assert expected **http://localhost:3000/profile** to match /profile$/",
    "contains Email: [email protected]"
  ],
  "screenshot": "logs-in-using-env-variables-failed.png"
}

Great - this plugin seems safe to use.

Conclusions

Keeping sensitive information out of public logs, screenshots and videos is a very important and ongoing concern. Make sure that every commit even if it only changes the tests goes through code review. If a password has been accidentally revealed, even during the pull request, revoke it and use a new one. For example, here is a quick way to get a random password using Node from the terminal

1
2
$ node -p "crypto.randomBytes(16).toString('hex')"
d01ce7056fb2eab425161dcfa5bdb502

Read more about Cypress and ways to use it in the linked resourced

Bonus 1: hide the email

If you want to hide the text entered in other input elements, like the user email, you can change the element's type attribute before typing.

1
2
3
4
5
6
7
cy.get('form').within(() => {
  cy.get('[name=email]')
    .invoke('attr', 'type', 'password')
    .type('[email protected]', { log: false })
  cy.get('[name=password]').type('Secret!', { log: false })
  cy.get('button[type=submit]').click()
})

Watch the video Hide The Entered Email In The Form for details.