Reporting questions come up in every SDET interview. They are testing two things: do you know the tools, and do you think about communication with your team. Here are the most common questions with strong answers.
Q: What reporting tools do you use in your framework?
A: We use Allure as our primary reporting tool integrated with Jenkins CI. Every test class has @Epic and @Feature annotations for business-readable organization, and all page object methods have @Step annotations for step-by-step tracing. On failure, a TestNG listener captures screenshots, page URLs, and browser logs. The report is automatically generated and hosted on Jenkins after every build. We also have TestNG default reports as a fallback for quick local checks. I have used ExtentReports in previous projects as well.
Q: How do you capture screenshots on failure?
A: I use a custom TestNG ITestListener. The onTestFailure() method gets the WebDriver instance from the test class using reflection, captures a screenshot using TakesScreenshot.getScreenshotAs(OutputType.BYTES), and attaches it to the Allure report using Allure.addAttachment(). I also attach the page URL and page source for additional context. The listener is registered in testng.xml, so it works automatically for all tests without modifying test classes.
Q: What Allure annotations do you use?
A: @Epic and @Feature at the class level for business grouping. @Story, @Description, @Severity at the method level for test details. @Step on page object methods for action tracing. @Issue to link to JIRA tickets and @TmsLink to link to test management system. @Owner to assign test ownership. @Link for external documentation references. These annotations organize the Behaviors tab into a three-level hierarchy that stakeholders can navigate.
Q: How do you generate reports in CI/CD?
A: In Jenkins, I use the Allure Jenkins Plugin. After the test stage, the post-always block calls allure([results: [[path: "target/allure-results"]]]) which generates the HTML report and hosts it on the build page. Historical trends populate automatically. For GitHub Actions, I use the allure-report-action to generate the report and publish it to GitHub Pages. In both cases, the report is accessible via a shared link — no one needs to ask for it.
Q: How do you handle flaky tests in reports?
A: Allure has a built-in Retries tab that shows tests which failed and then passed on retry. We use TestNG IRetryAnalyzer to retry failed tests once. If a test passes on retry, it appears in the Retries section as "flaky". We review this section weekly in our QA standup. Most flaky tests are caused by timing issues (fixed with better waits), test data dependencies (fixed with data isolation), or race conditions in parallel execution (identified using the Timeline view). The trend charts help us track if flakiness is increasing or decreasing over builds.
Q: How do you share test results with non-technical stakeholders?
A: The Allure report is hosted on Jenkins, so anyone with access can view it. For standups, I share the Overview dashboard link showing the pass/fail donut chart and trend. For specific failures, I share the direct link to the failed test showing steps, screenshot, and error. For management reviews, the Behaviors tab gives a business-readable view organized by Epic > Feature > Story. For compliance audits, we archive reports as build artifacts. We also send automated email notifications with the report link when BLOCKER or CRITICAL tests fail.
Q: What is the difference between allure-results and allure-report?
A: allure-results is the raw data directory — it contains JSON files generated during test execution. Each test writes one JSON file with its steps, status, and metadata. allure-report is the generated HTML report directory — it contains the interactive dashboard created by the Allure CLI. The flow is: tests generate allure-results, then the CLI reads allure-results and creates allure-report. You can regenerate allure-report at any time from allure-results. In CI, we archive both — allure-results for regeneration if needed, and allure-report for immediate viewing.
Q: Can you customize the failure categories in Allure?
A: Yes, using a categories.json file placed in the allure-results directory. You define category names, descriptions, matched statuses (failed/broken), and regex patterns that match error messages. For example, I categorize NoSuchElementException as "Element Not Found", TimeoutException as "Timeout Failures", and AssertionError as "Assertion Failures". This helps during triage — the team can quickly see how many failures are real bugs versus test infrastructure issues versus timing problems.
Q: Why do you put @Step on page objects instead of test methods?
A: Because page objects represent user actions — "Enter username", "Click login", "Select account". These are the meaningful steps I want in the report. If I put @Step on test methods, the report would just show "testLogin" — not useful. With @Step on page objects, when a test fails, the report shows the exact user action that failed. And because methods like loginAs() call enterUsername(), enterPassword(), and clickLogin() internally, Allure creates nested steps automatically, giving a detailed action trace. This makes debugging significantly faster.
When answering reporting questions in interviews, always connect it back to the team. Do not just say "I use Allure." Say "I use Allure because it helps the team — developers see the screenshot, managers see the trend, QA leads see which features are unstable." That shows you think beyond just writing code.
Key Point: Reporting interview questions test your communication skills as much as your technical knowledge. Always connect your answer back to how it helps the team.
Answer all 5 questions, then submit to see your score.
1. What does the AspectJ weaver do in an Allure setup?
2. Where should you put @Step annotations for the best Allure report?
3. What is the difference between "allure serve" and "allure generate"?
4. Which Allure report section organizes tests by @Epic > @Feature > @Story?
5. What file defines custom failure categories in an Allure report?