products

Repeat Test to Fight Flake

Repeat the same test and the entire project to find flakey tests

Recently I have looked at the reports of cy.route command failing to intercept a network request made by the web application. The following code failed sometimes according to the user:

1
2
3
4
5
6
7
8
9
10
11
12
13
describe("cypress bug", () => {
  it("should fail sometimes, rerun if not ;)", () => {
    cy.server();
    cy.route("get", "/api/posts").as("getPosts");

    cy.visit("http://localhost:3000");

    cy.wait("@getPosts");
    cy.wait("@getPosts");
    cy.wait("@getPosts");
    cy.wait("@getPosts");
  });
});

Note: you can find my fork of the above project at bahmutov/cypress-bug.

When trying the test locally, I do not see a problem. In the video below I am repeatedly re-running the test again and again - and it always passes.

Re-running the test never fails

Hmm, seems the project works in cypress open mode. Maybe the test fails or is unreliably during cypress run mode? To run the same spec again and again during cypress run we can use cypress-repeat utility, one of my Cypress wrappers. Let's install it

1
2
3
$ yarn add -D cypress-repeat
info Direct dependencies
└─ [email protected]

In package.json instead of cypress run we can use cypress-repeat run. Let's run the spec 10 times

1
2
3
4
5
6
{
  "scripts": {
-    "test": "cypress run"
+    "test": "cypress-repeat run -n 10"
  }
}

Run the NPM test script from the terminal to see if cy.route fails.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
$ npm t

> [email protected] test /Users/gleb/git/cypress-bug
> cypress-repeat run -n 10

cypress-repeat: will repeat Cypress run 10 time(s)
***** cypress-repeat: 1 of 10 *****

...
***** cypress-repeat: 2 of 10 *****

...
***** cypress-repeat: 10 of 10 *****

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:    6.0.0                                                                              │
  │ Browser:    Electron 87 (headless)                                                             │
  │ Specs:      1 found (index.js)                                                                 │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


────────────────────────────────────────────────────────────────────────────────────────────────────

  Running:  index.js                                                                        (1 of 1)


  cypress bug
    ✓ should fail sometimes, rerun if not ;) (248ms)


  1 passing (1s)


  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        1                                                                                │
  │ Passing:      1                                                                                │
  │ Failing:      0                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      0                                                                                │
  │ Screenshots:  0                                                                                │
  │ Video:        true                                                                             │
  │ Duration:     1 second                                                                         │
  │ Spec Ran:     index.js                                                                         │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


  (Video)

  -  Started processing:  Compressing to 32 CRF
  -  Finished processing: /Users/gleb/git/cypress-bug/cypress/videos/index.js.mp4        (0 seconds)


====================================================================================================

  (Run Finished)


       Spec                                              Tests  Passing  Failing  Pending  Skipped
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  index.js                                 00:01        1        1        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                        00:01        1        1        -        -        -

***** finished 10 run(s) successfully *****

Ok, the project does seem to pass 10 times in the row in cypress run mode. Maybe the flake is rarer than that.

Instead of repeating a project with a single test, let's repeat the same test 100 times. We can use the bundled Lodash to create test functions on the fly.

1
2
3
4
5
6
7
8
9
10
11
12
13
Cypress._.times(100, (k) => {
  it(`test ${k} of 100`, () => {
    cy.server();
    cy.route("get", "/api/posts").as("getPosts");

    cy.visit("http://localhost:3000");

    cy.wait("@getPosts");
    cy.wait("@getPosts");
    cy.wait("@getPosts");
    cy.wait("@getPosts");
  });
})

Running same test 100 times

Bingo! One of the tests failed. So there is some flake. Let's see if we can solve the flake problem by switching from the deprecated cy.route to much more powerful cy.intercept command. We no longer need cy.server() call.

1
2
3
4
5
6
7
8
9
10
it(`test ${k} of 100`, () => {
  cy.intercept("GET", "/api/posts").as("getPosts");

  cy.visit("http://localhost:3000");

  cy.wait("@getPosts");
  cy.wait("@getPosts");
  cy.wait("@getPosts");
  cy.wait("@getPosts");
});

The new command runs reliably 100 times.

Running 100 times a test with cy.intercept

Finally, let's run the same 100 tests 10 times in a row using cypress-repeat command from the terminal. We can use npm test or call cypress-repeat straight using NPX/Yarn

1
2
3
4
5
6
7
8
$ yarn cypress-repeat run -n 10
yarn run v1.22.5
$ /Users/gleb/git/cypress-bug/node_modules/.bin/cypress-repeat run -n 10
cypress-repeat: will repeat Cypress run 10 time(s)
***** cypress-repeat: 1 of 10 *****
...
***** finished 10 run(s) successfully *****
✨  Done in 271.29s.

The run completes 10 * 100 = 1000 tests in a row. Seems the test is now flake-free.

Tip: if this test shows very low frequency of flake, for example < 1%, I would simply add the test retries to repeat it automatically.

1
2
3
4
// repeat the test if it fails, up to 3 additional attempts
it(`test ${k} of 100`, {retries: 3}, () => {
  ...
})

See also