products

Cypress Cloud Test Triage Using AI

Use Cypress CLI or Cloud skill to do a quick test run triage.

Let's take a sample Cypress project bahmutov/cypress-cloud-skill-example with recorded Cypress test runs on CI. For simplicity there is just one spec cypress/e2e/spec.cy.js with 3 passing tests

cypress/e2e/spec.cy.js
1
2
3
4
5
describe('Suite A', () => {
  it('works 1', () => {})
  it('works 2', () => {})
  it('works 3', () => {})
})

We run the tests using GitHub Actions

.github/workflows/ci.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
name: ci
on: push
jobs:
  cypress-run:
    runs-on: ubuntu-24.04
    steps:
      - name: Checkout
        uses: actions/checkout@v7
      # Install dependencies with caching
      # and run all Cypress tests
      - name: Cypress run
        # https://github.com/cypress-io/github-action
        uses: cypress-io/github-action@v7
        with:
          record: true
        env:
          CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}

Tests are green

All 3 tests are passing

Break one test

Great. Now let's break one test on purpose

cypress/e2e/spec.cy.js
1
2
3
4
5
6
7
8
describe('Suite A', () => {
  it('works 1', () => {})
  it('works 2', () => {
    // make this test fail
    cy.wrap(42).should('equal', 43)
  })
  it('works 3', () => {})
})

The 2nd test is now broken

The Cypress Cloud shows the error message and the stack pretty well. We could also look at the test replay to see each command, but our tests are simple enough to get the gist from the text of the message.

Break one more test

Imagine someone breaks the 3rd test

cypress/e2e/spec.cy.js
1
2
3
4
5
6
7
8
9
10
11
describe('Suite A', () => {
  it('works 1', () => {})
  it('works 2', () => {
    // make this test fail
    cy.wrap(42).should('equal', 43)
  })
  it('works 3', () => {
    // make this test fail too
    cy.wrap('hello').should('equal', 'bye')
  })
})

Two broken tests out of 3

So now we have 2 broken tests. One was broken for two runs, another test just switched from passing to failing (read about Cypress test statuses in the blog post Cypress Test Statuses). Let's look at the test run; of course we can just use the browser, but wouldn't it be nicer to access the test results from our terminal, or use them in our pipelines and agent chats?

Cloud CLI

My first tool for accessing the recorded test results is the Cloud CLI

1
2
3
4
5
6
# install the tool once
$ npm install --global @cypress/cloud
# log in
$ cy-cloud login
# any time you need to remember CLI commands
$ cy-cloud --help

Quick note: Cloud CLI allows an interactive login using a browser and OAuth, or you can use a personal token for CI environments (read-only).

Let's see the latest run results

1
2
$ cy-cloud run list
Optional argument '--projectId' is required

Oops, we need to tell CLI what project we are working on. Our project ID is public, written in the cypress.config.js file

cypress.config.js
1
2
3
4
5
6
7
const { defineConfig } = require('cypress')

module.exports = defineConfig({
  // Cypress cloud project id
  projectId: 'pcgkb1',
  ...
})

So we really need to copy/paste the project ID in each CLI command

1
$ cy-cloud run list --projectId pcgkb1
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
{
  "runs": [
    {
      "runNumber": 8,
      "branch": "main",
      "createdAt": "2026-09-22T15:41:34.716Z",
      "duration": 11.296,
      "completedAt": "2026-09-22T15:41:46.111Z",
      "status": "failed"
    },
    {
      "runNumber": 7,
      "branch": "main",
      "createdAt": "2026-09-22T15:34:08.544Z",
      "duration": 7.021,
      "completedAt": "2026-09-22T15:34:15.965Z",
      "status": "failed"
    },
    {
      ...
    }
  ]
}

Cloud CLI outputs JSON, making it extremely convenient for connecting with other tools. I personally prefer focusing on the last N runs, lets see the last three runs.

1
$ cy-cloud run list --projectId pcgkb1 --limit 3
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
{
  "runs": [
    {
      "runNumber": 8,
      "branch": "main",
      "createdAt": "2026-09-22T15:41:34.716Z",
      "duration": 11.296,
      "completedAt": "2026-09-22T15:41:46.111Z",
      "status": "failed"
    },
    {
      "runNumber": 7,
      "branch": "main",
      "createdAt": "2026-09-22T15:34:08.544Z",
      "duration": 7.021,
      "completedAt": "2026-09-22T15:34:15.965Z",
      "status": "failed"
    },
    {
      "runNumber": 6,
      "branch": "main",
      "createdAt": "2026-09-22T15:28:36.669Z",
      "duration": 3.285,
      "completedAt": "2026-09-22T15:28:40.090Z",
      "status": "passed"
    }
  ],
  "pagination": {
    "page": 1,
    "perPage": 3,
    "total": 8
  }
}

Even better, let's see the last 2 failed test runs:

1
$ cy-cloud run list --projectId pcgkb1 --limit 2 --status failed

Tip: you can find the last run number

1
$ cy-cloud project get --projectId pcgkb1
1
2
3
4
5
6
{
  "name": "cypress-cloud-skill-example",
  "latestRunNumber": 8,
  "latestCompletedRunNumber": null,
  "projectId": "pcgkb1"
}

Get the failed tests using CLI

We know that the test runs 8 and 7 failed. Let's find which tests failed in the run 8

1
$ cy-cloud test list --projectId pcgkb1 --runNumber 8 --status failed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
{
  "tests": [
    {
      "projectId": "pcgkb1",
      "runNumber": 8,
      "specFilepath": "cypress/e2e/spec.cy.js",
      "specName": "spec.cy.js",
      "testName": [
        "Suite A",
        "works 2"
      ],
      "testId": "73640d79-f57f-4f06-93e3-cd303c1b31e8",
      "testReplayUrl": "https://cloud.cypress.io/projects/pcgkb1/runs/8/overview/73640d79-f57f-4f06-93e3-cd303c1b31e8/replay",
      "browserName": "Electron",
      "status": "failed",
      "attempts": [
        {
          "attemptNumber": 1,
          "errorName": "AssertionError",
          "errorMessage": "Timed out retrying after 4000ms: expected 42 to equal 43",
          "stackTrace": "    at Context.eval (webpack://cypress-cloud-skill-example/./cypress/e2e/spec.cy.js:7:16)"
        }
      ],
      "flaky": false
    },
    {
      "projectId": "pcgkb1",
      "runNumber": 8,
      "specFilepath": "cypress/e2e/spec.cy.js",
      "specName": "spec.cy.js",
      "testName": [
        "Suite A",
        "works 3"
      ],
      "testId": "3590c6c2-6bca-4a28-9beb-4fd9f8592258",
      "testReplayUrl": "https://cloud.cypress.io/projects/pcgkb1/runs/8/overview/3590c6c2-6bca-4a28-9beb-4fd9f8592258/replay",
      "browserName": "Electron",
      "status": "failed",
      "attempts": [
        {
          "attemptNumber": 1,
          "errorName": "AssertionError",
          "errorMessage": "Timed out retrying after 4000ms: expected 'hello' to equal 'bye'",
          "stackTrace": "    at Context.eval (webpack://cypress-cloud-skill-example/./cypress/e2e/spec.cy.js:11:21)"
        }
      ],
      "flaky": false
    }
  ],
  "pagination": {
    "page": 1,
    "perPage": 30,
    "total": 2
  }
}

We see the 2 failed tests with the error messages. Now I might want to investigate further. Which test has failed previously? Because that is what I want to prioritize: investigating those tests that used to pass and now are reliably failing. In Cypress Dashboard, the "Previous runs" panel gives me this information visually

Individual test status across multiple runs

We can check the test status from the CLI by using its testId property and fetching the status for each run. We know that the test with id 73640d79-f57f-4f06-93e3-cd303c1b31e8 has failed in the latest run 8. Let's find out its status in the run 7. Because the test ids change between the runs, we need to fetch the failed tests in the run 8

1
$ cy-cloud test list --projectId pcgkb1 --runNumber 7 --status failed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
{
  "tests": [
    {
      "projectId": "pcgkb1",
      "runNumber": 7,
      "specFilepath": "cypress/e2e/spec.cy.js",
      "specName": "spec.cy.js",
      "testName": [
        "Suite A",
        "works 2"
      ],
      "testId": "1c40a53b-98f5-4834-9b10-2967d91fdc54",
      "testReplayUrl": "https://cloud.cypress.io/projects/pcgkb1/runs/7/overview/1c40a53b-98f5-4834-9b10-2967d91fdc54/replay",
      "browserName": "Electron",
      "status": "failed",
      "attempts": [
        {
          "attemptNumber": 1,
          "errorName": "AssertionError",
          "errorMessage": "Timed out retrying after 4000ms: expected 42 to equal 43",
          "stackTrace": "    at Context.eval (webpack://cypress-cloud-skill-example/./cypress/e2e/spec.cy.js:7:16)"
        }
      ],
      "flaky": false
    }
  ],
  "pagination": {
    "page": 1,
    "perPage": 30,
    "total": 1
  }
}

We can match the test titles between the test run using the "testName" array which has strings for the suite names plus the test title itself. To simplify it all, we can write a script

Triage script

Let's find all tests that failed in the latest test run AND failed previously. Here is my version in triage.js.

triage.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
#!/usr/bin/env node

console.log('Triage the last two Cypress runs')

// call the "cy-cloud project get --projectId pcgkb1"
// and parse the JSON output to get the property "latestRunNumber"

const { execSync } = require('child_process')

const projectId = 'pcgkb1'
const output = execSync(
  `cy-cloud project get --projectId ${projectId}`,
  { encoding: 'utf-8' },
)
const latestRunNumber = JSON.parse(output).latestRunNumber
console.log('Latest run number:', latestRunNumber)

// parse the JSON output of the "cy-cloud test list --projectId <projectId> --runNumber <latestRunNumber> --status failed"
const failedTestsOutput = execSync(
  `cy-cloud test list --projectId ${projectId} --runNumber ${latestRunNumber} --status failed`,
  { encoding: 'utf-8' },
)
const failedTests = JSON.parse(failedTestsOutput).tests
// console.log('Failed tests:', failedTests)

if (failedTests.length === 0) {
  console.log('No failed tests')
  process.exit(0)
}

const testTitles = failedTests.map((test) =>
  test.testName.join(' > '),
)
console.log('Found %d failed tests', testTitles.length)
testTitles.forEach((title) => console.log(' -', title))

// fetch the failed tests for the previous run
const previousRunNumber = latestRunNumber - 1
const failedTestsPreviousOutput = execSync(
  `cy-cloud test list --projectId ${projectId} --runNumber ${previousRunNumber} --status failed`,
  { encoding: 'utf-8' },
)
const failedTestsPrevious = JSON.parse(
  failedTestsPreviousOutput,
).tests
const testTitlesPrevious = failedTestsPrevious.map((test) =>
  test.testName.join(' > '),
)
console.log(
  'Found %d failed tests in the previous run',
  testTitlesPrevious.length,
)
testTitlesPrevious.forEach((title) => console.log(' -', title))

// filter the latest failed tests by title matching previously failed test
const oldFailedTests = testTitles.filter((title) =>
  testTitlesPrevious.includes(title),
)
console.log(
  'Found %d failed tests in the latest run that failed before',
  oldFailedTests.length,
)
oldFailedTests.forEach((title) => console.log(' -', title))

Let's run it

1
2
3
4
5
6
7
8
9
10
$ ./triage.js
Triage the last two Cypress runs
Latest run number: 8
Found 2 failed tests
 - Suite A > works 2
 - Suite A > works 3
Found 1 failed tests in the previous run
 - Suite A > works 2
Found 1 failed tests in the latest run that failed before
 - Suite A > works 2

Of course, you can flip the logic and report newly failed tests.

Triage using AI

Writing a script just to triage the tests might be tedious. Often we want to "explore" the test results, without hardcoding a strict algorithm. We also don't want to remember Cloud CLI arguments and juggle multiple command line arguments while thinking about test results. We just want to fetch some information:

  • find the latest Cypress CI run number
  • find all failed tests
  • fetch results for the previous CI run
  • report the tests that failed in both

Calling Cloud CLI that knows how to call Cypress results API is just details. Thus we can let an AI skill released by Cypress Company manage these details.

1
$ npx skills add https://github.com/cypress-io/ai-toolkit --skill cypress-cloud-cli

I am using GitHub Copilot, which is included in the default agents for skills tool. The installation creates several files

  • skills-lock.json file keeps track of the locally installed AI skills in my project
  • .agents/skills/cypress-cloud-cli/SKILL.md is produced by Cypress and describes to AI agent all the cy-cloud project ... commands

Let's use Copilot to triage the recorded Cypress test results. First, the latest project run number.

Copilot finds the latest run number

Expand the command summary to see the agent using cypress-cloud-cli skill to call cy-cloud commands

Expand the chat command block to see the executed Cloud CLI commands

What are the failed tests in the latest run 8? Ask AI agent, it should be able to fetch the individual tests

The list of failed test in the latest run

Note: since we are running the prompts in the same chat, the agent "knows" that the latest run number was already discovered "8". We can now go to the previous run

The list of failed test in the previous run 7

The agent should be smart enough to understand that the "previous run" can be computed from number "8". Now let's make the agent use the chat context and match the two list of failed tests

AI match lists the only test that failed in the two runs

So there you have it:

  • you can access Cypress Cloud test results via its CLI tool
  • you can write small scripts to call multiple CLI commands fetching the information needed
  • or you can make your AI agent use the Cypress Cloud CLI skill to call the correct Cloud CLI commands and then report the results

The AI approach is flexible: you can explore the test runs, analyze the test errors, group errors in "buckets", etc. The scripting approach is fast and does not cost you token money; if you are always accessing the same test run information, a script might be the best approach. Finally, you can call Cloud CLI yourself when needed.