Crawl Local Pages Using Cypress

Sometimes you want to simply visit every local page on your site to make sure the links are correct and every page loads. Cypress is not a crawler, but it can definitely handle the crawl for smaller sites. In the videos below I show how to collect every anchor link, filter external links, and visit every collected URL once.

🎁 You can find the full source code in my repository bahmutov/cypress-crawl-example.

Collect the URLs

The best way to write a crawler is to think about the actions on every page. The crawler needs to:

1. grab the first URL to visit from a queue

• if there are no URLs to visit, we are done

2. call cy.visit(url)
3. collect all anchor elements

• filter external links
• filter links we have already visited
• filter links we have already queued up to visit
• add the filtered URLs to the queue

4. go to step 1

You can see my implementation of the above steps in the test file spec.js and watch the implementation in the video below:

Perfect, at the end of the test each URL has been visited, but some pages were visited twice - because the crawler does not know that links to /tos.html and /tos lead to the same page.

Resolving URLs

To prevent visiting the same page via different links, we need to check if a given URL leads to a page we have visited already. We can do this by using the cy.request command and inspecting the redirects array.

// check URLs by requesting them and getting through the redirects
const redirected = []
localUrls.forEach((url) => {
  cy.request({ url, log: false })
    .its('redirects', { log: false })
    // resources without redirects will
    // not have the property "redirects"
    // so prevent Cypres from throwing an error
    .should(Cypress._.noop)
    .then((redirects) => {
      if (Array.isArray(redirects) && redirects.length > 0) {
        // each redirect record is like "301: URL"
        // so grab the last redirect and parse it
        // that will be the final address
        const redirectedUrl =
          redirects[redirects.length - 1].split(' ')[1]
        // keep just the local part of the full URL
        const parsed = new URL(redirectedUrl)
        redirected.push(parsed.pathname)
      } else {
        redirected.push(url)
      }
    })
})

You can find the full source code in the file spec2.js and the explanation in the video below.

Bonus: check the 404 resource

The crawl example has one additional test file 404-spec.js that shows how to verify the error page the site serves when you try to visit a non-existent URL. Again, we can use a combination of cy.request and cy.visit commands to verify the status code and the error page served. We do need to let the commands work on the status code 4xx by using failOnStatusCode: false option

it('shows 404 error', () => {
  const url = '/does-not-exist'
  cy.request({ url, failOnStatusCode: false })
    .its('status', { timeout: 0 })
    .should('eq', 404)
  cy.visit(url, { failOnStatusCode: false })
  cy.contains('span', '404').should('be.visible')
})

You can find the explanation in the video below

Happy Crawling 🕷