Benchmarking
jacket bench builds the @bench procedures optimized and measures them. jacket bench covers baselines and comparisons.
Build and measure the @bench procedures (optimized by default; the options of test select and build them)
jacket bench [OPTIONS] [FILTER]... [-- <FLAGS>...]
Arguments
| Argument | Meaning |
|---|---|
[FILTER]... | Run the tests whose name contains one of these; test/subtest runs one subtest |
[FLAGS]... | Flags for jaic, after -- |
Options
| Option | Meaning |
|---|---|
--exact | Filters are whole names, not parts of names |
-E, --filter-expr <EXPR> | Run the tests a filter expression selects, as module(Sort) and not kind(property); with FILTERs, the tests both select (predicates: test, module, file, kind, all, none) |
--list | Print the tests (and benchmarks) instead of running them |
--list-target | With --list: discover the selected target's compile-time cases instead of the host's |
--roots | Print the test roots (the folders [test] paths expands to, narrowed by the current directory), one per line, and nothing else |
--deps | With --list: print the source files the test programs read (what jaic -deps prints for each), instead of the tests |
--changed <FILE> | Run only the tests a change to FILE can affect: those whose programs read it (as --list --deps shows), the tests of a root it adds an unlisted file to, and tests whose program does not compile. Repeat it for each changed file; it applies before --partition |
--coverage | Report which lines of the code the tests ran (unit tests, golden programs and doc examples), and write them as lcov |
--coverage-out <FILE> | Where --coverage writes its report [default: target/lcov.info, or target/coverage.html] |
--coverage-format <COVERAGE_FORMAT> | The report's format Possible values: - lcov: lcov's text format, with line counts and procedures - html: One self-contained page: modules, files and the source with the count of each line [default: lcov] |
--coverage-by <COVERAGE_BY> | What a row of the printed table is Possible values: - file - module: The nearest folder with a module.jai [default: file] |
--coverage-baseline <FILE> | Fail when a module (or the total) covers fewer lines than this recorded baseline by more than --coverage-margin |
--coverage-margin <POINTS> | Percentage points a module may lose against --coverage-baseline [default: 0.5] |
--coverage-record <FILE> | Record this run's lines per module as a baseline |
--coverage-summary <FILE> | Add a markdown table of the modules to this file ($GITHUB_STEP_SUMMARY) |
--coverage-include <DIR> | Also report the code below this folder (repeatable); by default the test roots and the folders of -import_dir |
--watch | Run again whenever a .jai file of the tests or the code they import changes |
-j, --jobs <N> | Tests run at once, as jaic's -j [default: the machine's threads] |
--timeout <SECONDS> | Seconds before a test without @timeout is killed; 0 for no limit [default: 120] |
--retries <N> | Run a failed test again up to N times; one that then passes is FLAKY |
--fail-fast | Start no more tests after the first failure |
--no-fail-fast | Run every test even when a profile sets fail_fast |
--no-capture | Show the tests' output as it is written (runs one test at a time) |
--no-isolation | Run a root's tests in one process instead of one process each |
-q, --quiet | Print only the tests that fail, time out or are flaky, then the summary |
--format <FORMAT> | human text, or json: one event per line on standard output Possible values: - human - json: One JSON object per event on standard output [default: human] |
--junit <FILE> | Also write the results as JUnit XML |
--partition <SPEC> | Run one part of the tests: hash:K/N (by name) or count:K/N (by position) |
--interp | Run the tests in jaic's interpreter |
--os <OS> | Target OS, as jaic's -os (also spelt -os) [possible values: linux, windows, macos, wasm] |
--cpu <CPU> | Target CPU, as jaic's -cpu (also spelt -cpu) |
--runner <COMMAND> | Command that runs test programs built for another target (node tools/wasi_run.ts) |
--needs <LIST> | Capabilities this machine has, for tests with @needs (network,gpu) |
--profile <NAME> | Use [test.profile.NAME] from jacket.toml |
--path <DIR> | Test the tests/ of these folders instead of the project's |
--doc | Run only the doc tests: the jai code blocks in a library's doc comments |
--check-only | Only type-check the doc examples, without building or running them: with --os, the examples for that target from any machine |
--no-doc | Leave the doc tests out |
--keep-temp | Keep every test's temporary folder (failed tests keep theirs anyway) |
--accept | Write what golden and compile-fail tests printed into their expected files, and keep the snapshots that differ or are new |
--review | Go through the pending .snap.new snapshots one by one (only lists them without a terminal) |
--seed <N> | The seed property tests start from [default: random for each run, shown with a failure] |
--samples <N> | Samples per benchmark, over @samples [default: 50] |
--warmup <MS> | Warm-up per benchmark in milliseconds, at least; it goes on until the time per iteration is steady [default: 100] |
--measure-ms <MS> | Time to spend sampling each benchmark in milliseconds, over @min_time [default: 1000] |
--resamples <N> | Bootstrap resamples for the confidence intervals and the comparison [default: 100000] |
--noise-threshold <PCT> | A change smaller than this many percent is noise [default: 2] |
--significance <P> | A change counts when its p-value is below this [default: 0.05] |
--save-baseline <NAME> | Write the results to target/bench-baselines/NAME.json |
--baseline <NAME> | Compare with the results saved as NAME and show the change per benchmark |
--max-regression <PCT> | Exit 1 when a benchmark regressed (the change is significant and beyond the noise threshold) by more than PCT percent |
-h, --help | Print help (see a summary with '-h') |