Solve The User Problem

The problem

Recently a user has opened an issue #14532 in the Cypress repo. The subject of the user issue:

Module API: cypress.run is unable to pick a specific test from within the integration folder

The details were very well described, the user has noted that:

Once the user has explained how they search for specific test in the Cypress Desktop GUI, they have noted a discrepancy: using the same filename does not work when calling either cypress run --spec <filename> or require('cypress').run({ spec: <filename> }) commands. Hmm, so how does one pass the spec filename when trying to run a single spec in non-interactive mode? And why does it work differently in the interactive cypress open mode? The user then notes the confusing part: the same filename that works in the Desktop Gui does not work from the command line or via Cypress NPM module call.

The background information

In my opinion, the problem is caused by the difference in what is a spec <filename> means when the project is already loaded vs when the user tries to pick a spec to run. When Cypress Desktop GUI opens we already know the integration folder with all the specs, and thus show just the spec filenames relative to the integration folder. Thus if you have for example the Cypress specs inside cypress/integration/ui folder, we show just the files in that folder.

On the other hand, if you are trying to start Cypress and pass the test file to run, you are likely to use the filename on the disk - the full filename from the current working directory. When entering the type, it is so convenient to type it relative to the current folder, watch as I type the relative name using the shell's tab completion:

Similarly, when using Cypress NPM module, the relative spec filename is convenient. My VSCode editor even does file completion, just like the shell.

So the user's issue has to do with the folder: in Cypress Desktop GUI the specs are relative to the integration folder, while the command line and the module API evaluate the spec parameter from the current working folder.

Decision time

What do we do about the issue #14532? Do we change the Cypress test runner? Do we modify the way --spec works? Do we search the integration folder first, then fall back to the current working folder?

We can solve the user's problem in multiple ways. Every solution has its trade offs such as:

• difficulty
• speed

For example, if we modify the test runner's code, and search the integration folder search, then fall back to the current working directory:

• the solution would be more difficult to implement, since there are edge cases to consider, like: what happens if there are some files found in both places? Do we merge them? Do we just take the first found file? It would require modifying the existing tests and adding few new ones. It requires a pull request and a code review since it changes the existing logic.
• the solution would be released with the next Cypress version, which usually happens every two weeks. Even then, the user has to upgrade to the new Cypress version to resolve the problem.

The above solution might appeal to someone who likes working on the code itself. But I think we can do better. Put yourself in the user's shoes. They have a problem. They pinged the issue two weeks later asking for an update:

🤷‍♂️ If you are reading this, please, do not post comments like this UNLESS there was a previous comment from a Cypress team member saying "I am about to do this! Here we go! Get ready to get your mind blown 🤯". We always update the issues with new information as we get to them, there are GitHub automations that update the issue labels whenever there is work being done, etc. If there are no updates on an issue, that means we did not have time to work on it.

Fine. What can we do now to solve the user's issue? Here are our options:

1. Close the issue itself explaining the problem and the workaround:
   • Difficulty: simple
   • Speed: now
   • Resolution: comment on the issue
2. Update the documentation for --spec CLI parameter and cypress.run({ spec: ... }) property explaining the relative path
   • Difficulty: simple
   • Speed: one day
   • Resolution: documentation PR #3525
3. Modify the Cypress error message to avoid the confusion
   • Difficulty: medium
   • Speed: between one day and two weeks
   • Resolution: Cypress PR #14810

I perform the listed actions in order. The user can understand the problem and use the correct spec path right away after #1 is complete. We will merge the documentation PR clarifying the spec path, thus the users will find the answer (hopefully). The Test Runner PR is the most complex: it adds the folder to the error message, making it clear how the spec files are searched. Here are the error messages before and after when using --spec or cypress.run and the specs were not found:

Before:

Can't run because no spec files were found.

We searched for any files matching this glob pattern:

my-spec.js

After:

Can't run because no spec files were found.

We searched for any files matching this glob pattern:

my-spec.js

With respect to the project root folder:

/projects/my-folder

Nice! A few tests had to be modified, and a few snapshots had to be updated, but once this PR merges, any user that hits this problem will (hopefully) just correct the spec parameter and move on.

Finally, I will NOT make the Cypress Test Runner more complex by implementing the folder search fallback logic; it will make the code more complex. I might revisit this question again, if I see that the above 3 solutions are not enough.

Conclusion

It is simple to see every issue as an opportunity to do what you like doing the most: if you like coding, you might code to "fix" every GitHub issue, and if you like writing, you might answer every user question . But let us keep the focus on the current users. They hit a snag, and they are struggling. Maybe their boss is breathing down their neck. They need a workaround now. While we are providing an explanation answer, we think about the other users who might hit the same problem tomorrow. We can update the documentation for them, and finally, we can change the product to prevent the future problems. Every decision is a trade-off between difficulty and speed, so our answers should be staggered.

See more

• Rocks and Stones
• Multiple Benefits