How We Run The Mobile Web Browser Tests at Mercari US

At Mercari US users can buy and sell things. While most of the application traffic comes from our mobile application (which we are rewriting as a ReactNative App right now), a big chunk of users use Mercari via web browsers, both desktop and mobile. The mobile web application is not simply a responsive version of the app, but it renders a lot of web components differently, and processes different user events, like "touch" instead of "click.

In the right image above, I simulate a mobile web browser using Chrome DevTools mobile browser emulation.

The mobile web browser emulation works by attaching a custom "user-agent" header to every outgoing request. The Mercari web server looks at that request to determine what page to serve (including different CSS and JS bundles). Thus the page looks and behaves differently from the normal desktop page. How do we test the mobile version of the web page using Cypress Test Runner?

Mobile tests

We must request pages using the same emulation "user-agent" header. Cypress includes a configuration option that let's you set an arbitrary userAgent value to sent with each request. The value does not even have to be very complicated. For example, in our NPM scripts, we have a command to run the "normal" desktop Cypress tests, and another command to open Cypress with a mobile user agent and set the app viewport to 400 by 600 pixels:

package.json

{
  "scripts": {
    "cy:open": "cypress open",
    "cy:open:mobile": "cypress open --config viewportWidth=400,viewportHeight=600,
      userAgent=\"Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_1 like Mac OS X) Mobile/14E304\""
  }
}

When we execute npm run cy:open:mobile or yarn cy:open:mobile the configuration options --config ... are applied and set the viewport of the browser to 400x600 pixels, and set the "user-agent" header to a simplified string the backend server still considers a mobile browser client. Here is how a typical mobile test looks when running:

Not every test needs to be executed in the mobile web browser emulation mode. While we have 500+ full end-to-end tests, only about 75 tests are meant to exercise mobile-specific web flows. We tag tests to mark the tests that have mobile-specific features. The test above is defined as both a regression and a mobile test.

buy-and-change-payment-method.js

it('Can change payment method on checkout',
  { tags: ['@regression', '@mobile'] },
  () => {
    cy.signup(seller)
    cy.createListing({ name, price })
    ...
  }
)

💡 For tagging individual tests and suites of tests, we use my cypress-grep plugin. For finding all tests tagged "@mobile" we use my find-cypress-specs utility.

Fast feedback

It is extremely important to get early results when running lots of end-to-end tests. Whenever someone opens a pull request, we first run a CI job that only runs any new and modified spec files. Read my blog posts Get Faster Feedback From Your Cypress Tests Running On GitHub Actions and Get Faster Feedback From Your Cypress Tests Running On CircleCI how we do this. For mobile spec specifically, we have a separate CI step that runs any specs with the found @mobile tag inside using cypress run --config viewport,userAgent=mobile.. command. The relevant CircleCI command is below

# find all changed or new specs with the test tag "@mobile" inside
specs=$(npx find-cypress-specs --branch main --parent --tagged @mobile)
n=$(npx find-cypress-specs --branch main --parent --tagged @mobile --count)
echo ""
echo "Changed and added ${n} Cypress mobile specs"
echo ${specs}
echo ""

if [ ${n} -lt 1 ]; then
  echo "No Cypress mobile specs changed, exiting..."
  exit 0
fi

# we have to form the Cypress run command ourselves
# to only execute the changed specs using mobile user agent
npx cypress run --record --parallel \
  --group "0. Changed mobile specs" \
  --browser chrome \
  --config viewportWidth=400,viewportHeight=600, \
    userAgent="Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_1 like Mac OS X) Mobile/14E304" \
  --spec ${specs}

Any changed "normal" specs execute in a different CircleCI job. Only if changed (and added) specs pass, we run a sanity set of tests for each pull request.

Because the changed specs passed in mobile and desktop modes, we can proceed with running all sanity tests in the "Other specs" group.

Tip: the pull request text allows our engineers to run tests by topic or run a larger regression suite of tests, or even run all specs just by checking the PR checkboxes, see the blog post Pick Tests To Run Using The Pull Request Text.

Mobile-specific steps

Some tests should run in desktop and in mobile modes. Thus we need to have the conditional Cypress steps that look at the current user agent configuration setting.

utils.js

export const isMobile = () => {
  const userAgent = Cypress.config('userAgent')
  return userAgent && userAgent.includes('Mobile')
}

spec.js

import {isMobile} from './utils.js'
const isMobilePage = isMobile()

it('runs this test', () => {
  ...
  if (isMobilePage) {
    // mobile page does not have user icon, only nav
    cy.byTestId('InquiriesButtonNav').should('be.visible')
  } else {
    cy.get('nav img[alt=avatar]').should('be.visible')
  }
})

Because the user agent cannot change while the tests are running, we can check the Cypress.config('userAgent') once and then use the value through the spec file.

The next steps

If you follow my Cypress Network Testing Exercises course, you might have seen "Bonus 20" lesson that simulates the mobile mode using the cy.intercept command. I am exploring ways to NOT use the userAgent configuration option to run mobile tests, and instead attaching the request header to every outgoing request. Stay tuned.