Fail the test-case if one of the related automated tests is failed
A
Arthur Harutyunyan
The situation is the following:
1) I have test case ABC-123 in qase.io.
2) I have integrated Cypress with Qase
3) I have a test spec file with 3 tests in Cypress
4) All those 3 tests are related to ABC-123 test-case
The thing is, that when running this single spec file with qase-reporter, and I make one test (one "it" block ) to fail, I see that the test-run is passed in Qase.io with that single ABC-123.
Feature Request: I think, it is more than logical to fail the test-run/case in qase.io, if one of the related automated tests is failed. Also provide an informative UI to easily understand which one is failed and passed.
Manjunatha Reddy
Hi Arthur Harutyunyan,
We've added parameterization support for
qase-cypress
in the latest version, and here's an example test, for syntax reference -https://github.com/qase-tms/qase-javascript/blob/main/qase-cypress/docs/usage.md#adding-parameters-to-a-test
Could you please give this a try, and let us know if this addresses the problem you've been facing?
A
Arthur Harutyunyan
Hi Manjunatha Reddy,
Thanks for the update!!
Could you please share more details on how to configure qase to cover my needs?
Manjunatha Reddy
Hi Arthur Harutyunyan!
We'll be glad to assist, may I suggest continuing on our support chat? Please, send us a message and we'll be with you right away!
Manjunatha Reddy
Hi everyone!
Apologies for the delay in providing an update on this issue.
I wanted to follow up now that I have some news. The issue has been escalated to our engineering team, and we believe the solution lies in parameterising the results.
The team is currently working on implementing this in the Robot Framework reporter, so the results will be displayed separately rather than as retries.
For the Cypress reporter, we’ll soon be releasing a stable version that includes support for parameters. We’ll keep you posted here once it’s live.
Vladyslav Mykytenko, I apologize for the delay in getting back to you. The Playwright reporter already supports parameterization. Could you let us know if this option solves your issue, or is there something specific you’re looking for? Feel free to share more details here or reach out to us via live chat.
We’ll continue to keep you updated on any further developments.
Vladyslav Mykytenko
Manjunatha Reddy Hi! What kind of parametrization are you talking about? The problem described in the ticket is not a reporter problem. We described all details to ivolkov@qase.io. Pls, contact him for more details if needed.
Manjunatha Reddy
Hi Vladyslav Mykytenko ,
I checked with Ilya, about your requirement and I understand that parameterisation would not completely address it.
I understand that you want all retries of the same test to be considered as one result. If any retry fails, it should be counted as a failure of the test overall, rather than only reflecting the result of the last attempt as we currently do.
I’ll keep this in mind and let you know if there’s any development.
Vladyslav Mykytenko
Manjunatha Reddy any updates?
Vladyslav Mykytenko
Have the same problem but in JS Playwright
Manjunatha Reddy
Merged in a post:
[qase-robotframework] Improve Reporting for Data-Driven Tests (DDT) to Accurately Reflect Failures
F
FP
I currently have no way to see in the Qase UI if runs from my DDT fail. The status goes to PASS although some of the runs are set to FAIL
For example, 34 runs are made with a test case. If out of the 34, one fails, Qase shows the run status PASS and +34. I would expect the status to be Fail and +33 PASS -1FAIL
[qase-support] Adding additional information
In Data-Driven Tests (DDT), the current method of representing multiple results for the same test case leads to inaccurate final statuses and results.
For example, if you have a dataset of 20 test cases and each test case posts 20 results to the test run, even if just 1 out of the 20 tests fails, the overall status for that test case should be marked as ‘Failed’. However, the status is incorrectly shown as ‘Passed’ due to the way results are aggregated and displayed.
Currently, the system shows a ‘Passed’ status even when there are failed results because the final status is based on the most recent results for all test cases in the run. As a result, failed results are not effectively highlighted.
Example -
Scenario:
Dataset size: 20 test cases
Each test case has 20 results
Results Posting:
If 1 of 20 tests fails for a particular test case, there are still 19 passing results.
Expected status: ‘Failed’ (because there’s at least one failure).
Current Behavior:
Test case status: ‘Passed’ (since the most recent results for that test case may show as ‘Passed’ even if earlier results were ‘Failed’).
Issue:
The final status of a test case is misleading because failed results are not properly reflected in the overall status due to the aggregation method. (only the latest result status of each test case in the run is used for calculation of final run status)