[qase-playwright-reporter@v2.1.7] Tests marked as Invalid when multiple assertion errors
T
Taber Hust
We have several tests that are failing with multiple assertion errors, but in Qase they are marked as Invalid. We use
expect.soft()
to report every assertion error when evaluation the response body of our API tests using playwright.But for some reason, these tests are not being properly marked as Failed, but instead show as Invalid - which confuses the team and makes them think the test errored out instead of just failing due to bugs.
This began happening when upgrading to 2.1.7 of the playwright-qase-reporter. Can someone please update the status evaluation logic to ensure it works for multiple failed expect.soft() playwright assertions?
Update: this may not be limited to array of errors, but instead may affect GenericAssertions (https://playwright.dev/docs/api/class-genericassertions) that don't result in the expected error message.
Attached is an example, where we have two expect.soft assertions asserting the value of some json data (results of an accessibility scan) match some criteria. The error uses a custom message in the
expect()
call rather than the standard messagingMay Oktan
In addition to the multiple expect.soft() assertion case, we’re also seeing tests that fail in beforeEach / beforeAll hooks being marked as Invalid instead of Failed.
From a user perspective, these tests have still failed, regardless of whether the failure occurred in the test body or in a setup hook. Marking them as Invalid is confusing for the team and interferes with reporting and analysis (e.g. flaky test dashboards), since Invalid tests are not counted in failure metrics.
At minimum, it would be very helpful to:
• Treat hook failures as Failed rather than Invalid, or
• Provide a configuration option to disable the Invalid status or automatically map Invalid → Failed.
This would make test results more accurate and actionable when analyzing stability and flakiness.