Well, I am happy to report Cypress-the-company is all in on test auto-healing in its new cy.prompt command (which I recreated locally as cypress-think plugin). Let's look at the screenshots Cypress put in the blog post:
The test shows Cypress auto-healing in practice. The test had a prompt like "Turn on dark mode", which used to resolve to some button (unclear). Now the button is gone, but the test has been "auto-healed" to find another button. We can see the details of the healing by clicking on the "Turn on dark mode" step in the Command Log:
Please do not use any auto-healing in your tests.
It is a huge red flag that your development and testing process is not working as intended. Let's look at the situation above: the original test found a "button.group" element. Which element was it? Did anyone confirm or review this button to ensure the user would understand its purpose? Did the button have a "title" attribute? An "on/off" state to ensure the user understands its role? Did anyone review the initial implementation code and its independent testing code?
I guarantee that NO ONE will be looking at the AI-generated or AI-healed tests to ensure the actual executed test commands are correct and the test is exercising the app correctly. NO ONE. EVER. EVER (repeat like the Outkast song) How often do you look at successful CI runs? Unless you are debugging a sudden failure, you are NOT looking at the CI logs. Same with tests: if the test is green, it is a black box to be ignored in the deepest corner of the basement.
Adding auto-healing is like adding a giant wildcard character to every test command, element selector, assertion, etc.
- Not updating the expectations when updating the front-end code is a red flag.
- Not reviewing the test changes is a red flag.
- Not running end-to-end tests locally when updating the code is a red flag.
- Not having explicit tests is a red flag.
Imagine a login form:
1 | <input type="email" placeholder="Enter your email" /> |
An AI prompt "Enter the user email" can correctly pick this input field and use cy.get('input[type="email"]').type(...) command. Then something changes and the placeholder is showing a pretty interesting attitude:
1 | <input type="text" placeholder="Dummy email" /> |
Well, the auto-healing can certainly "guess" that this is the same email input field, even if its type has changed from "email" to "text". But the changed placeholder is probably NOT something that we should show on the login page! Yet, no one will review the actual test commands until an angry user complains on a Reddit form "Acme.co called me a dummy!".
Auto-healing tests is like a medical doctor telling its patients to use ChatGPT to self-medicate. I'm sure it will be fine /s