products

Cypress Vs SafeTest

Comparing component testing using Cypress vs SafeTest.

Recently Netflix has released SafeTest - a component testing framework built on top of Playwright. In this blog post I want to compare it to my Cypress component testing feature. To compare these test runners using concrete examples I grabbed an example from the SafeTest's own repo and placed with additional Cypress tests into the repo bahmutov/safetest-vs-cypress.

The bahmutov/safetest-vs-cypress repo

Let's see if can learn what SafeTest can and cannot do.

Is it dev or prod dependency?

The first thing I have noticed was that SafeTest seems to be a production dependency. Yes, the docs say to install SafeTest using npm install --save-dev safetest, but then you should include it in your src/index.tsx file:

1
2
3
4
import ReactDOM from "react-dom";
import { bootstrap } from 'safetest/react';
import App from "./App";
...

The example folders in kolodny/safetest all list safetest as a prod dependency:

kolodny/safetest/blob/main/examples/cra/package.json
1
2
3
4
5
{
  "dependencies": {
    "safetest": "file:../.."
  }
}

Call me old-fashioned, but I feel a testing library should be a dev dependency, just like Cypress.

Feature SafeTest Cypress Component Testing (CT)
Dependency prod 👎 dev 👍

Installation

With SafeTest you create a config file setup-safetest.tsx, add package.json scripts, and modify your src/index.tsx file.

Installing SafeTest

You also need to be running the application before you can start testing, just as if this was an end-to-end test against the local environment.

With Cypress, you just open it and click on the "Component Testing".

Pick Component Testing

Cypress finds from the source code the framework your are using and creates appropriate files automatically.

React with Vite

Scaffolded files

The modifications to the cypress.config.ts file:

cypress.config.ts
1
2
3
4
5
6
7
8
9
10
import { defineConfig } from "cypress";

export default defineConfig({
  component: {
    devServer: {
      framework: "react",
      bundler: "vite",
    },
  },
});

That it is. You can start writing component tests.

Feature SafeTest Cypress Component Testing (CT)
Dependency prod 👎 dev 👍
Installation manual 👎 auto 👍
Start the app needed 👎 not needed 👍

Hello World test

Let's take a look at the example test shown in the SafeTest folder.

safetest-example/src/misc.safetest.tsx
1
2
3
4
5
6
7
8
9
import { describe, it, expect } from 'safetest/vitest';
import { render } from 'safetest/react';

describe('simple', () => {
  it('s', async () => {
    const { page } = await render(<div>Test1</div>);
    await expect(page.locator('text=Test1')).toBeVisible();
  });
});

We are rendering a simple React component with the text "Test1" and confirm it is visible. SafeTest is built on top of Playwright, thus the test runs in a real browser. The test confirms that the component is visible in the real browser. Nice. The same test can be written using Cypress:

cypress-example/src/misc.cy.tsx
1
2
3
4
5
6
describe('simple', () => {
  it('s', () => {
    cy.mount(<div>Test1</div>)
    cy.contains('Test1').should('be.visible')
  })
})

Let's interact with a component. Here is the example test that clicks on the component 500 times.

safetest-example/src/App.safetest.tsx
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
it('can do many interactions fast', async () => {
  const Counter = () => {
    const [count, setCount] = React.useState(0);
    return (
      <div>
        <button onClick={() => setCount(count + 1)}>Count is {count}</button>
      </div>
    );
  };
  const { page } = await render(<Counter />);
  await expect(page.locator('text=Count is 0')).toBeVisible();
  for (let i = 1; i <= 500; i++) {
    await page.locator('button:not(a)').click();
    await expect(page.locator(`text=Count is ${i}`)).toBeVisible();
  }
});

And an equivalent Cypress component test

cypress-example/src/App.cy.tsx
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
it('can do many interactions fast', () => {
  const Counter = () => {
    const [count, setCount] = React.useState(0)
    return (
      <div>
        <button onClick={() => setCount(count + 1)}>Count is {count}</button>
      </div>
    )
  }
  cy.mount(<Counter />)
  cy.contains('Count is 0').should('be.visible')
  for (let i = 1; i <= 500; i++) {
    cy.get('button:not(a)').click()
    cy.contains(`Count is ${i}`).should('be.visible')
  }
})

Ok. How about using shortcut to change the state of the component to perform an "app action"? SafeTest can give you a "bridge" function, whatever it is:

safetest-example/src/App.safetest.tsx
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
it('can use the bridge function', async () => {
  let count = 0;
  let forceNumber: (num: number) => void = () => {};
  const Counter = () => {
    const forceRender = React.useReducer(() => count, 0)[1];
    forceNumber = (n) => {
      count = n;
      forceRender();
    };
    return (
      <div>
        <button
          onClick={() => {
            count++;
            forceRender();
          }}
        >
          Count is {count}
        </button>
      </div>
    );
  };

  const { page, bridge } = await render(<Counter />);
  await expect(page.locator('text=Count is 0')).toBeVisible();
  await page.click('button');
  await expect(page.locator('text=Count is 1')).toBeVisible();
  await bridge(() => forceNumber(50));
  await expect(page.locator('text=Count is 50')).toBeVisible();
  await page.click('button');
  await expect(page.locator('text=Count is 51')).toBeVisible();
});

Seems we need the bridge to call the component code running in the browser from the Playwright test running in Node. Cypress can simply interact with the component, since it is the same browser code, so nothing special is needed.

cypress-example/src/App.cy.tsx
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
it('can use the bridge function', () => {
  let count = 0
  let forceNumber: (num: number) => void = () => {}
  const Counter = () => {
    const forceRender = React.useReducer(() => count, 0)[1]
    forceNumber = (n) => {
      count = n
      forceRender()
    }
    return (
      <div>
        <button
          onClick={() => {
            count++
            forceRender()
          }}
        >
          Count is {count}
        </button>
      </div>
    )
  }

  cy.mount(<Counter />)
  cy.contains('Count is 0')
  cy.get('button').click()
  cy.contains('Count is 1').then(() => {
    forceNumber(50)
  })
  cy.contains('Count is 50')
  cy.get('button').click()
  cy.contains('Count is 51')
})

People new to Cypress tip over its "schedule all commands, then run them with retries" execution model. This is why we use the following syntax to call the forceNumber after we confirm the page confirms the component has text "Count is 1"

1
2
3
4
cy.contains('Count is 1').then(() => {
  forceNumber(50)
})
cy.contains('Count is 50')

I always though the cy.then command should have been called cy.later, as I wrote in the blog post Replace The cy.then Command.

So let's add a few rows to our comparison:

Feature SafeTest Cypress Component Testing (CT)
Dependency prod 👎 dev 👍
Installation manual 👎 auto 👍
Start the app needed 👎 not needed 👍
Test syntax easy 👍 easy 👍
Execution environment mix of Node and browser 👎 browser 👍

Test speed

Let's run the same test "can use the bridge function" by itself to see how long it takes in SafeTest vs Cypress

We start the SafeTest app with npm run dev and run the single spec. The test is isolated to run by itself using it.only

SafeTest runs the test in 1200ms

Let's run the same test using Cypress component testing

Cypress runs the test in 300ms

Cypress component testing is very fast because it bundles only the component under the test plus the test itself. SafeTest loads the full E2E bundle and extracts the component to be tested. This is why you need to include it in the bundling entry and execute the application while running the component tests. In the CI mode, Cypress disables time-traveling debugger, thus alleviating the overhead. Thus Cypress CT can be faster than the SafeTest. Still, it does not matter. The component tests are fast enough in both cases.

Feature SafeTest Cypress Component Testing (CT)
Dependency prod 👎 dev 👍
Installation manual 👎 auto 👍
Start the app needed 👎 not needed 👍
Test syntax easy 👍 easy 👍
Execution environment mix of Node and browser 👎 browser 👍
Speed fast 👍 fast 👍

📝 SafeTest is extracting components / functions for testing from the production bundle. It is is a pretty cool idea. I have described doing it for Angular.JS (!) in the blog post Unit testing Angular from Node like a boss in 2015. Dmitriy Tishin described how to grab React components from Storybook bundles to use my Cypress component testing library cypress-react-unit-test in 2020.

Mocks and Spies

SafeTest provides Jest mocks and spies

1
2
3
4
5
6
7
8
9
10
11
12
13
14
import { describe, it, expect, browserMock } from 'safetest/jest';
import { render } from 'safetest/react';
import { Header } from './Header';

describe('Header', () => {
  /* ... */

  it('calls the passed logout handler when clicked', async () => {
    const spy = browserMock.fn();
    const { page } = await render(<Header handleLogout={spy} />);
    await page.locator('text=Logout').click();
    expect(await spy).toHaveBeenCalled();
  });
});

Cypress includes Sinon.js mocks and spies.

1
2
3
4
5
it('calls the passed logout handler when clicked', () => {
  cy.mount(<Header handleLogout={cy.stub().as('logout')} />);
  cy.contains('Logout').click();
  cy.get('@logout').should('have.been.called')
});
Feature SafeTest Cypress Component Testing (CT)
Dependency prod 👎 dev 👍
Installation manual 👎 auto 👍
Start the app needed 👎 not needed 👍
Test syntax easy 👍 easy 👍
Execution environment mix of Node and browser 👎 browser 👍
Speed fast 👍 fast 👍
Spies and stubs present 👍 present 👍

Overrides, providers, etc

SafeTest exposes createOverride that let's you change the behavior of the component; one more proof that SafeTest is a production dependency. You can call use the override from the test

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
 // Records.tsx
const UseGetRecordsQuery = createOverride(useGetRecordsQuery)

export const Records = () => {
  const useGetRecordQuery = UseGetRecordsQuery.useValue()
  const { records, loading, error } = useGetRecordsQuery();
  if (loading) return <Loader />;
  if (error) return <Error error={error} />;
  return <RecordList records={records} />;
};
// from the safetest
it('Has an error state', async () => {
  const { page } = render(
    <UseGetRecordQuery.Override
      with={(old) => ({ ...old(), error: new Error('Test Error') })}
    >
      <Records />
    </UseGetRecordQuery.Override>
  );
  await expect(await page.locator('text=Test Error')).toBeVisible();
});

In Cypress, you could pass the overrides via the shared window object to the component to implement the same substitution. I strongly discourage using the implementation-specific overrides. Test the interface of the component. You want to see how the component reacts to the loader error? How the loader shows? Stub the network call and confirm.

src/components/Overlay.cy.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
import React from 'react'
import { Overlay } from './Overlay'
import '../App.css'

it('is visible and clickable', () => {
  cy.mount(
    <Overlay
      overlay={true}
      onClickOverlay={cy.stub().as('click')}
    />,
  )
  cy.get('.overlay').should('be.visible').click()
  cy.get('@click').should('have.been.called')
})

it('shows the loading element', () => {
  cy.intercept('GET', '/times/90', {
    delay: 1000,
    statusCode: 404,
    body: [],
  }).as('times')
  cy.mount(<Overlay overlay={true} time={90} />)
  cy.contains('.overlay__loading', 'Loading').should('be.visible')
  cy.wait('@times')
  // the loader goes away
  cy.get('.overlay__loading').should('not.exist')
})

it('shows the top times', () => {
  cy.intercept('GET', '/times/90', {
    fixture: 'times.json',
  }).as('scores')
  cy.mount(<Overlay overlay={true} time={90} />)
  cy.wait('@scores')
  cy.get('.overlay__times li').should('have.length', 4)
  cy.contains('.overlay__times li', '01:30').should(
    'have.class',
    'overlay__current',
  )
})

💻 The above example comes from my presentation Learn Cypress React component testing by playing Sudoku. The source code with all component tests is in the repo bahmutov/the-fuzzy-line.

Feature SafeTest Cypress Component Testing (CT)
Dependency prod 👎 dev 👍
Installation manual 👎 auto 👍
Start the app needed 👎 not needed 👍
Test syntax easy 👍 easy 👍
Execution environment mix of Node and browser 👎 browser 👍
Speed fast 👍 fast 👍
Spies and stubs present 👍 present 👍
Overrides present ⚠️ possible ⚠️

Dev experience

With Cypress you get an excellent interactive mode that shows the component and let's you write full tests quickly. It is no wonder that the SafeTest repo itself comes with lots of React components and none of them are tested.

The reporter components

But it is easy to write Cypress tests, here are a couple.

cypress-example/src/report/label.cy.tsx
1
2
3
4
5
6
import { Label } from './label.tsx'

it('labels', () => {
  cy.mount(<Label>hello</Label>)
  cy.contains('div', 'hello').should('be.visible')
})
cypress-example/src/report/expandable.cy.tsx
1
2
3
4
5
6
import { Expandable } from './expandable'

it('expands', () => {
  cy.mount(<Expandable expanded={true}>Expandable content</Expandable>)
  cy.contains('Expandable content').should('be.visible')
})

The reporter components with Cypress CT specs

A good test example is a sanity test for the Accordion component

cypress-example/src/report/accordion.cy.tsx
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
import { Accordion } from './accordion.tsx'
import { ComponentsContext } from './report'
import { Expandable } from './expandable.tsx'

it('works', () => {
  cy.mount(
    <ComponentsContext.Provider
      value={{
        Expandable,
      }}
    >
      <Accordion summary="All is good" defaultOpen>
        Accordion content is here
      </Accordion>
    </ComponentsContext.Provider>,
  )
  cy.contains('Accordion content is here').should('be.visible')
  cy.contains('[data-testid=toggle]', 'All is good').click()
  cy.contains('Accordion content is here').should('not.be.visible')
})
Feature SafeTest Cypress Component Testing (CT)
Dependency prod 👎 dev 👍
Installation manual 👎 auto 👍
Start the app needed 👎 not needed 👍
Test syntax easy 👍 easy 👍
Execution environment mix of Node and browser 👎 browser 👍
Speed fast 👍 fast 👍
Spies and stubs present 👍 present 👍
Overrides present ⚠️ possible ⚠️
Want to write tests maybe 🤷‍♂️ yes 👍

Dev support

SafeTest was announced by Netflix. If they seriously use it to standardize on Playwright for the E2E tests and SafeTest for the component tests, it is going to be well-maintaned and supported. Cypress is backing its component testing. Judging from previous OSS library experience, I would say

Feature SafeTest Cypress Component Testing (CT)
Dependency prod 👎 dev 👍
Installation manual 👎 auto 👍
Start the app needed 👎 not needed 👍
Test syntax easy 👍 easy 👍
Execution environment mix of Node and browser 👎 browser 👍
Speed fast 👍 fast 👍
Spies and stubs present 👍 present 👍
Overrides present ⚠️ possible ⚠️
Want to write tests maybe 🤷‍♂️ yes 👍
Backed by Netflix 👍 Cypress.io 👎

The final tally

SafeTest has other features that I like, for example the built-in visual comparisons

1
2
3
const { page } = await render(<Header />);
await expect(page.locator('text=Logout')).toBeVisible();
expect(await page.screenshot()).toMatchImageSnapshot();

I do not count this feature towards SafeTest benefits, since it is built into Playwright.

Feature SafeTest Cypress Component Testing (CT)
Dependency prod 👎 dev 👍
Installation manual 👎 auto 👍
Start the app needed 👎 not needed 👍
Test syntax easy 👍 easy 👍
Execution environment mix of Node and browser 👎 browser 👍
Speed fast 👍 fast 👍
Spies and stubs present 👍 present 👍
Overrides present ⚠️ possible ⚠️
Want to write tests maybe 🤷‍♂️ yes 👍
Backed by Netflix 👍 Cypress.io 👎
Total 4 👍 3 👎 8 👍 1 👎

Do you have a good SafeTest spec example and want to see how it looks in Cypress? Make the code public and send it my way.

Update 1: SafeTest vs Cypress vs WebDriver

Christian Bromann has published a nice comparison following the same metrics COMPONENT TESTING WITH SAFETEST VS. CYPRESS VS. WEBDRIVERIO.