Home / Framework / Milestone 8

Milestone 8 of 9

Reliability, reports and CI/CD

The suite runs in parallel, every failure explains itself with a report and a screenshot, and GitHub Actions runs it on every push, every pull request and on a schedule.

Why it matters

A suite that only runs on your laptop doesn't protect anyone. CI is where tests earn their keep, and a readable report is what lets a teammate fix a failure without you.

What to build

  1. Parallel runs. Run tests in parallel safely: each worker has its own browser, its own session and its own data, so nothing is shared.
  2. Reports. An HTML report (Allure, Extent, pytest-html or Playwright's own) with each test's steps, plus a screenshot, the page URL and the browser log on failure. Playwright's trace viewer counts.
  3. A GitHub Actions workflow. A file in .github/workflows/ that installs your tools, runs the suite headless and uploads the report as an artifact, on push and pull_request.
  4. Secrets. Store your DojoCover user ID and password as repository secrets and pass them to the job as environment variables. They must never appear in the workflow file or the logs.
  5. A schedule. Add a schedule trigger, nightly or weekly, at an odd minute such as 37 2 * * * rather than on the hour, and a workflow_dispatch trigger to start it by hand.
  6. A status badge. Show the workflow's badge at the top of your README.

Things to know about GitHub's schedules:

  • Scheduled runs can start late when GitHub is busy, especially at the top of the hour.
  • On a public repository, GitHub turns schedules off after 60 days without activity in the repo. Any commit turns them back on.
  • Schedules run on the default branch only.

Other CI tools such as GitLab CI, Jenkins and Azure Pipelines use the same ideas: a trigger, a clean machine, secrets, the same command, and a kept report.

Rules

  • Retries are allowed only if the report still shows which tests needed them; a retry is not a fix.
  • The CI run uses the same command you run locally.
  • Run the schedule nightly or weekly at most: DojoCover is a shared practice server.

Hints

Selenium

TestNG parallel="methods", JUnit's parallel execution or pytest-xdist need one driver per thread: keep the driver in a ThreadLocal (or a fixture) rather than a static field. In CI, run Chrome with --headless=new.

Playwright

Playwright Test runs files in parallel by default; set workers and retries for CI in playwright.config, keep trace: 'on-first-retry', and upload playwright-report with actions/upload-artifact.

These are pointers, not solutions. TestDojo doesn't publish a finished framework: the point is that you build and understand your own.

Done when

Ticks are saved in this browser only, for your own tracking.

In an interview

Questions this milestone prepares you for, and what a good answer covers. Answer them in your own words, about your own framework.

How do you run tests in CI?
Triggers, a clean environment, secrets, headless browsers, the same command as locally, and reports kept as artifacts.
How do you make parallel runs safe?
Nothing shared: a driver per thread or worker, separate sessions and data.
Do you use retries?
Only with visibility: the report must show which tests were retried, and flaky tests are fixed, not hidden.
What happens when the nightly run fails?
Who is told, how the report shows the cause, and how a real bug is told apart from a flaky test.

Note the decision you made in this milestone and why. Those notes become your README in milestone 9.