Skip to main content
A check that passes on a retry is flaky. Mobster marks it, keeps its history per device, and suggests quarantine. It never quarantines a check on its own.

Retry a failed check

From your project folder, retry each failed check once and fail the suite if a retry passes:
Mobster tries a failed run again once (--retries N, from 0 to 5) with the check’s own reset, which reinstalls the app or clears its data unless the check says otherwise. A check that couldn’t run because of the machine (WebDriverAgent, the simulator, the launch, a busy device) gets another try too. A usage error, a missing key, the spend cap, a refused app or a phone that stopped answering mid-run (device_lost) doesn’t. A check that passes on a retry passed, and is flaky: the summary marks it ~ flaky, the report lists it in its Flaky table, and JUnit gives it the property flaky=true.

Measure a flaky check

With --repeat K, each check runs K times, retries are off unless you pass --retries, and each result has a pass rate. A check passes only when all K runs passed. For a check with steps, first record a passing path with mobster test --fresh CHECK, then run mobster test --repeat K CHECK. Repetition needs that complete approved path with no model, and a missing path refuses before app reset, install or launch. --fresh --repeat and --spread --repeat are refused. Launch-only checks can repeat without a model. The repeated run pins the check and its expectations, its fixture and condition files, the selected model, the full iOS runtime, and the approved replay path for each device. If one changes before a later attempt, Mobster refuses that attempt before app reset, install or launch.

Know what a suite pins

Every suite pins each check’s file, effective steps and assertion files, and the bytes at app.path, when it starts. If a pinned input changes before a later attempt, or a fixture, linked step or accessibility baseline changed between planning and the start, Mobster refuses that attempt before app reset, install or launch, and the suite stops with exit 3. Mobster can’t pin a missing file, a symlink, a seed folder holding a symlink, a seed or media file over its limit, an app holding a symlink, or an app over 2 GB or 50,000 files. That input stays unpinned, the run prints a note naming it, and its digest is null in results.json. The check still runs, and the other checks carry on. A missing app fails only its own check, and so does a symlink in a seed folder. --repeat refuses an app it can’t pin, for that check only, because its runs could otherwise span two builds of one version.

Quarantine a flaky check

A check you know is flaky can keep running without failing the run. List it in .mobster/quarantine.yaml:
A quarantined check runs, and its ordinary failures appear as JUnit <skipped> without changing the exit code. A safety stop still ends the suite and appears as a JUnit error. --quarantine FILE reads another file, and stderr says when a listed name matches no check.

Find your flaky checks

Every mobster test run adds each check’s result, per device, to .mobster/test-results/history.json, and keeps the last 20. The folder is in .mobster/.gitignore, so the history stays on your Mac.
--history runs nothing and takes a project path. --json prints the same rows as JSON. Each row has the pass rate, the duration of every run, the duration trend (slower, faster or steady, newer half of the runs against the older half) and a flake score from 0 to 1. Under each row, Last run: gives the launch time, memory and hangs of the check’s last run (Performance). The flake score is the larger of two shares: the runs that passed only after a retry or a repeat that failed, and the neighboring runs where one passed and the other didn’t. It counts only verdicts of the same check text on the same app build, so a change after you edit the check or rebuild the app isn’t a flake. A run that ended budget_unmeasured counts in neither share. With 5 or more runs a check is: A run that couldn’t run (a lost device, a busy phone, a stopped suite) says nothing about the check, and Mobster doesn’t keep it. Mobster suggests quarantining a flaky check in the summary, in results.json (history.suggestQuarantine, and history on each result) and in the HTML report’s “Flaky over time” table. With --auto-quarantine, Mobster quarantines the checks the history calls flaky, for that run only: they run and are reported, and they don’t change the exit code. It never does this for a check whose latest result failed on any device, and .mobster/quarantine.yaml stays as you wrote it. A history file Mobster can’t read is set aside as history.json.corrupt, and the history starts again. Mobster for Mac reads the same numbers from GET /api/tests/history, with ?project=PATH for one project.

Split a suite with tags and shards

Give a check tags (tags: [smoke, checkout]) and pick by them: mobster test --tag smoke. A tag no check has is a usage error, so a typo can’t pass CI by running nothing. --shard 2/4 runs the second of four even shards, every fourth check in file order, so four CI machines split one suite.

Rerun the checks that failed

You get this in the next release, not in Mobster for Mac 0.4.0 or Mobster CLI 0.4.3.
mobster test --failed runs only the checks that didn’t pass in your project’s newest finished run: failed, couldn’t run or needs review on any device. A check that passed on a retry counts as passed, unless that run used --strict. A quarantined check that failed still runs. The last run is the newest suite in .mobster/test-results. If it ran only some of the checks, --failed picks from those. The rerun uses the checks as your project is now, on the devices you select for this run: the simulators in .mobster/matrix.yaml, or each check’s own simulator. Its first line names the suite it picks from, and the device change when the devices differ. --failed runs nothing and exits 3 when:
  • the newest suite has no results yet, because it is still running or stopped first
  • a check that didn’t pass has no file in your project now, unless it is quarantined
  • an XCTest result didn’t pass and you didn’t add --xctest, or you limit the tests with --only-testing, or --scheme or --test-plan names something the last run didn’t use
  • your project has no finished run
When no check needs a rerun, --failed exits 0 if the last finished run passed, else 3:
You can’t combine --failed with check paths, --check, --tag or --shard. Each machine’s last run is already its shard, so mobster test --shard 2/3 || mobster test --failed reruns the checks that didn’t pass in that shard. failed: true on run_tests selects the same checks over MCP, and Run failed checks does it on your project’s page (Projects).