process

Minimize Mailosaur Use

How to avoid the high Mailosaur costs when running Cypress email tests.

Recently I have described hot to testing email flows using Mailosaur. Unfortunately, I must report that Mailosaur uses dark design UI pattern to trick you to sign up to a monthly plan that has 1/20 of the usage limits you expect! The trial period has usage limit of 2000 emails per day, which is so generous, you never hit any limits while trying it out.

Mailosaur usage during the trial period

The misleading Mailosaur pricing table

When you go to the pricing page to sign up for real, you see the following:

Mailosaur pricing table

Looks reasonable, right? A simple monthly plan with 1 developer account and 1000 emails per day ... is only $9. Right? WRONG. Small gray font gives the weasel words: "Up to 1000 emails per day" and default being ... only 50 per day.

Single monthly plan

Ok, how do you increase the daily email limit? You change it in that input field from 50 to 1000. Ooops, you monthly price is now $9 + $25.

Real single monthly plan

Hmm. That is completely different ballpark, the price to get what you think we were getting (up to 1000 emails per day) just went up 4x. Ok, no biggie, let's use the "Most popular Business" monthly plan. This plan says "From 5000 emails per day" which is very generous. You must be thinking how is it so much cheaper than the "Starter" plan for much higher limits?

Advertised Business plan.

It is not. But the table does not even show the real price. The catch is in the words "Starts at 5 seats". Once you pick the Business plan and go to the checkout page you get hit with $80 per month... because $16 * 5 users = $80 total monthly cost.

The real Business plan monthly cost

Even that is not all. The dark UI Mailosaur patterns continue. When picking the plan to sign up... you don't see the total price. Yup, you pick the new usage limits and do not see the price until you fill everything and proceed to the Payment page.

Mailosaur does not show the total monthly price

Ohh, you want to increase your daily email limit above 50? Ok, how about ... you can only increase it to 500 or 750 or 1000?!!! What kind of the step up is this?

Mailosaur plan limits are ... weird

Am I crazy, or is this a really shitty way of tricking customers? I do think developers and companies providing services must be paid, but this is just tricking you to pay for a lot less service than you expect.

Do not hit usage limits

Ok, let's see if we can get away with the lowest usage limit of 50 emails per day. One thing that is very annoying: you run your CI tests several times a day, and suddenly hit the email usage limits and all your Mailosaur cy commands fail with the "Usage limit" errors. Here is how I protected my builds and simply skip the email tests if we are close to the daily usage limit. Let's say at most my Cypress tests generate 10 emails per run. Thus I disable them if we are close to "daily limit - 10" number.

🎁 You can find the full source code shown in this blog post in the repo bahmutov/cypress-sendgrid-mailosaur-example.

First, to get the Mailosaur usage limit, you can hit the API endpoint using the same API key used during email testing.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
const axios = require('axios')

/**
 * Pings Mailosaur to fetch the current usage
 * @see https://mailosaur.com/docs/api/usage/
 */
async function getEmailUsage() {
  try {
    const usage = await axios.get('https://mailosaur.com/api/usage/limits', {
      auth: {
        username: 'api',
        password: process.env.CYPRESS_MAILOSAUR_API_KEY,
      },
    })
    return usage.data.email
  } catch (e) {
    console.error('problem fetching Mailosaur usage')
    console.error(e.message)
  }
}

The returned object data.email has only two properties limit and current. In our case the limit: 50. Let's disable email testing by checking the usage limit before Cypress tests start. From the project config file, grab the usage limits and set the env.skipEmailTests = true value if we are over the limit.

cypress.config.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
const { defineConfig } = require('cypress')
const axios = require('axios')

async function getEmailUsage() { ... }

/**
 * Returns true if we are inside "cypress open" command
 */
const isInteractive = (config) => {
  return !config.isTextTerminal
}

/**
 * Checks if we can still perform email testing, depending on the 3rd party usage.
 * If we are over the limit - hard no.
 * If we are close to the limit:
 *  If we are in the interactive mode "cypress open" allow testing
 *  Else do not allow email testing
 * This gives preference to human email test writing
 * Note: modifies the config.env object to disable email testing.
 */
async function checkIfEmailTestingIsPossible(config) {
  const emailUsage = await getEmailUsage()
  console.log('email usage limits', emailUsage)
  // hard limit
  if (emailUsage.current >= emailUsage.limit) {
    console.log(
      '📧 current email usage %d is close to the limit %d',
      emailUsage.current,
      emailUsage.limit,
    )
    console.log('📧🛑 will skip email tests')
    config.env.skipEmailTests = true
  } else if (emailUsage.current + 10 >= emailUsage.limit) {
    // soft limit - only allow human interactive testing
    // during "cypress open"
    if (isInteractive(config)) {
      console.log(
        '📧 current email usage %d is close to the soft limit %d',
        emailUsage.current,
        emailUsage.limit,
      )
      console.log(
        '📧 but we are running in the interactive mode, allow email testing',
      )
    } else {
      console.log(
        '📧 current email usage %d is close to the soft limit %d',
        emailUsage.current,
        emailUsage.limit,
      )
      console.log('📧🛑 will skip email tests')
      config.env.skipEmailTests = true
    }
  }
}

module.exports = defineConfig({
  e2e: {
    async setupNodeEvents(on, config) {
      await checkIfEmailTestingIsPossible(config)

      // IMPORTANT: return the config object
      return config
    },
  },
})
  • If we are above the daily usage limit, disable all email testing by setting config.env.skipEmailTests = true
  • If we are approaching the daily usage limit:
    • still allow testing if we are using cypress open command. This way a developer can still work on email specs

In each Cypress spec file, we skip the email test if the skipEmailTests value is set to true.

1
2
3
4
5
6
7
8
9
10
11
12
13
describe('An email', () => {
  if (Cypress.env('skipEmailTests')) {
    it('shows the code by itself')
    it('has the confirmation code link')
    it('has the working code')
    return
  }

  // real email tests
  it('shows the code by itself', () => ...)
  it('has the confirmation code link', () => ...)
  it('has the working code', () => ...)
})

The value skipEmailTests set in the cypress.config.js is static and is available in every spec right away and allows us to determine which tests to define. If we are at the limit, the tests are showing as pending.

Email tests are skipped

If we are below the usage limit, the tests execute.

Reuse the email

A single test flow can generate multiple emails. For example, during Mercari US testing, we might create a buyer and a seller users. That is two confirmation emails. If the buyer buys an item, both users get an email. The buyer receives an email "You made a sale..." and the seller receives an email "You made a purchase..."

Seller and buyer emails

You can imagine how we might write multiple tests based on those 2 emails. Can the seller confirm the shipment by clicking on the item? What if the seller is not logged in into the browser, will the email lead to the "Sign in" screen and then to the right item's page? What about the user: can they go to the order page? Can the buyer that is not logged in click on the email, sign in, and then see their purchase item? Instead of writing separate email flow tests, each generating 4 emails, let's grab the emails just once and reuse them.

In my example, I will use the following confirmation email from bahmutov/cypress-sendgrid-mailosaur-example repo.

The confirmation email to test

We get this email by filling out the form and retrieving the sent email, and writing the HTML into the Document

store-email.cy.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
// https://github.com/mailosaur/cypress-mailosaur
// register email commands
import 'cypress-mailosaur'

describe('An email', () => {
  if (Cypress.env('skipEmailTests')) {
    it('shows the code by itself')
    it('has the confirmation code link')
    it('has the working code')
    return
  }

  beforeEach(() => {
    const userName = 'Joe Bravo'
    const serverId = Cypress.env('MAILOSAUR_SERVER_ID')
    const randomId = Cypress._.random(1e6)
    const userEmail = `user-${randomId}@${serverId}.mailosaur.net`
    cy.log(`📧 **${userEmail}**`)

    cy.visit('/')
    cy.get('#name').type(userName)
    cy.get('#company_size').select('3')
    cy.get('#email').type(userEmail)
    cy.get('button[type=submit]').click()

    cy.log('**shows message to check emails**')
    cy.get('[data-cy=sent-email-to]')
      .should('be.visible')
      .and('have.text', userEmail)

    cy.mailosaurGetMessage(serverId, {
      sentTo: userEmail,
    })
      .then(console.log)
      .its('html.body')
      // store the HTML under an alias
      .as('email)
  })

  beforeEach(function () {
    cy.document({ log: false }).invoke({ log: false }, 'write', this.email)
  })

  // real email tests
  it('shows the code by itself', () => ...)
  it('has the confirmation code link', () => ...)
  it('has the working code', () => ...)
})

There are multiple tests we could write to validate different features of this email. There are codes to confirm, buttons to click, etc. If we let the tests run right now, each test signs up and receives its own email. But we can quickly transform this spec into one email by using cypress-data-session plugin. Just wrap te code in the first beforeEach hook with cy.dataSession "setup" method.

store-email.cy.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
// BEFORE
beforeEach(() => {
  const userName = 'Joe Bravo'
  const serverId = Cypress.env('MAILOSAUR_SERVER_ID')
  const randomId = Cypress._.random(1e6)
  const userEmail = `user-${randomId}@${serverId}.mailosaur.net`
  cy.log(`📧 **${userEmail}**`)
  // fill the form
  cy.mailosaurGetMessage(serverId, {
    sentTo: userEmail,
  })
    .then(console.log)
    .its('html.body')
    // store the HTML under an alias
    .as('email)
})

beforeEach(function () {
  cy.document({ log: false }).invoke({ log: false }, 'write', this.email)
})
store-email.cy.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
// AFTER
// https://github.com/bahmutov/cypress-data-session
import 'cypress-data-session'

beforeEach(() => {
  cy.dataSession({
    name: 'email',
    setup() {
      const userName = 'Joe Bravo'
      const serverId = Cypress.env('MAILOSAUR_SERVER_ID')
      const randomId = Cypress._.random(1e6)
      const userEmail = `user-${randomId}@${serverId}.mailosaur.net`
      cy.log(`📧 **${userEmail}**`)
      // fill the form
      cy.mailosaurGetMessage(serverId, {
        sentTo: userEmail,
      })
        .then(console.log)
        .its('html.body')
    },
    shareAcrossSpecs: true,
  })
})

beforeEach(function () {
  cy.document({ log: false }).invoke({ log: false }, 'write', this.email)
})

Each data session creates a Cypress alias automatically, thus we don't need .its('html.body').as('email') command. Note the shareAcrossSpecs: true parameter. It saves the email in the Cypress config process, thus the same email stays cached as long as we need it. Only the very first run of the spec signs up the user and retries the email - all tests afterwards, and even browser spec reloads just fetch the already cached copy. One Mailosaur email = infinite number of tests we can run.

The first time the beforeEach hook runs it signs up and retries the HTML email

All test runs afterwards simply use the cached HTML email from that data session.

All tests afterwards use the cached HTML email

Reusing the cached email is fast. When the first test signs up the user to retrieve the email, it takes about 8 seconds. By reusing the email, the other two tests save 16 seconds. If we reload the specs, all tests use the cached email, and the total spec time drops from 12 to 4 seconds.

Reusing the same email in all tests

Reusing the email is very convenient when working on the tests themselves, as it gives you instant feedback while you are editing the spec source code. Once you are done, you might want to clear the data session cached value to force the full test flow to be tested on each spec run. You can do it by adding the command to the after hook.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
// https://github.com/bahmutov/cypress-data-session
import 'cypress-data-session'

beforeEach(() => {
  cy.dataSession({
    name: 'email',
    setup() {
      ...
    }
  })
})

beforeEach(function () {
  cy.document({ log: false }).invoke({ log: false }, 'write', this.email)
})

// during interactive work, comment this "after" hook
// to keep reusing the same email for speed
after(() => {
  Cypress.clearDataSession('email')
})

Note: I wish Cypress team realized that their plugins ecosystem and the ability to show UI controls for plugins is a super power that the test runners have. If I am working with cypress-data-session plugin, I have to use the browser command log to clear data session by manually calling.

Have to call the plugin command via DevTools console

Why can't Cypress allow the plugins to create buttons and labels through an API so that the users could interact with them from the Test Runner? What a missed opportunity.