Schema-shot - snapshot testing for dynamic data

I have recently started using snapshot testing using my own framework-agnostic snap-shot library and I love it. But there is one limitation - it only works if the data compared with snapshot is static. For example fetching top item from an API endpoint cannot be tested this way.

// spec.js
const snapShot = require('snap-shot')
it('returns most popular item', () => {
  return snapShot(api.getMostPopularItem())
  // api.getMostPopularItem() returns different item every day
  // day 1 {id: 101, name: 'foo', ...}
  // day 2 {id: 241, name: 'bar', ...}
})

What can we do to overcome this? Well, when does our endpoint work correctly? Do we care about specific 'id' in the returned object or the fact that the object has one and it is a positive integer?

I suggested that one should deal with dynamic data by deriving invariant object from the result. For example, we could check the above case like this

it('returns most popular item', () => {
  return snapShot(
    api.getMostPopularItem()
      .then(o => ({
        id: typeof o.id === 'number' && o.id > 0,
        name: typeof o.name === 'string' && o.name
      }))
  )
})

The above snapshot test works - it saves the properties of the object we care about in data-independent way. The snapshot is simply {"id": true, "name": true} which works for any well-formed result. If the endpoint really goes down, or returns an item with {uuid: 101, ...} instead of {id: 101, ...} our test will catch it.

JSON Schema

What should the transformation be? The above transform is manual and verbose. Luckily, there is a JSON schema convention (it is not a standard) that can be used in this case. For example, {id: 101} would be simply

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "properties": {
    "id": {
      "type": "number"
    }
  }
}

An object could be verified against an existing schema, there are lots of validators in every language. Some of them even allow specifying the "required" property flag, and disallowing additional properties. For example is-my-json-valid can understand the following

const schema = {
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "properties": {
    "id": {
      "type": "number",
      required: true
    }
  },
  additionalProperties: false
}
const isMyJsonValid = require('is-my-json-valid')
const validate = isMyJsonValid(schema)
const result = validate({uuid: 101})
console.log(result) // false
console.log(validate.errors)
/*
[{
  field: 'data.id', message: 'is required'
}, {
  field: 'data', message: 'has additional properties'
}]
*/

Schema shot

We need to get the schema somehow. Relying on the API generation tools, like Swagger is too much effort and limits us to APIs. I want to use schema to check the shape of any object!

Luckily, given an object we can "derive" or "describe" its JSON schema automatically. Good projects are generate-schema and json-schema-trainer. We start with an object and get a schema. Using a schema any other object can be validated. I put these two features together in validate-by-example

const {train, validate} = require('validate-by-example')
const schema = train({id: 101, name: 'foo'})
const result = validate(schema, {uuid: 101})
if (!result.valid) {
  // see result.errors list
}

If we derive the schema from the first seen object, we can store it and then load any time we need to verify a new object. This is the definition of snapshot testing! I factored out the snapshot save / load logic into snap-shot-core and made schema-shot. Schema-shot saves the trained schema the first time it is called for a given location (derived automatically from the call stack). Every time after that, the schema is used to check the object's compliance.

const schemaShot = require('schema-shot')
it('returns most popular item', () => {
  return schemaShot(api.getMostPopularItem())
})

The snap shot file will have something like this

exports['returns most popular item 1'] = {
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "properties": {
    "id": {
      "type": "number",
      "required": true
    }
  },
  "additionalProperties": false
}

Perfecto!