Ashes Testing

This document defines the supported surface of ashes test and the expected behavior of Ashes tests. It serves as a reference for test authors and a specification for test runner implementations.

Ashes tests are ordinary .ash programs annotated with leading // comment directives. The test runner compiles each test to a native executable, runs it, and compares the observed result against the declared expectations.

Discovery

Default discovery:

• ashes test looks for tests under tests/.
• Discovery is recursive.
• Only .ash files are included.
• Hidden directories such as .git and .vscode are ignored.
• Files are executed in lexicographic path order.

Project-aware behavior:

• If ashes.json is discovered by searching upward from the current directory, ashes test runs in project mode.
• ashes test --project path/to/ashes.json forces project mode for that project.
• In project mode, each discovered test is compiled as the entry module while reusing the project sourceRoots, include, and shipped standard library resolution.

Explicit paths:

• ashes test some/path runs tests only from the provided file or directory.
• If a project is active and a relative path does not exist from the current directory, the runner also tries resolving it relative to the project root.

Execution Model

For each discovered test, the runner:

1. Reads the leading directive block.
2. Materializes any file fixtures into a temporary working directory.
3. Compiles the test to a native executable, optionally beneath a declared fixture-relative executable directory.
4. Runs the executable from the declared fixture-relative working directory, or the fixture root by default.
5. Compares the observed exit code and stdout with the directive expectations.
6. Continues to the next test even if the current test fails.

Tests run sequentially.

// executable-directory: and // working-directory: let host-tool tests model an installed binary launched from an unrelated project directory. Both values are relative to the isolated fixture root; absolute paths and paths that escape it are rejected. The runner creates the declared directories.

Matching Rules

Stdout matching:

• The runner compares stdout after trimming trailing newlines and trailing whitespace on the captured output.
• The expected text from // expect: is also trimmed at the end.
• Matching is otherwise exact.

Examples:

• // expect: 42 matches program output 42\n.
• // expect: hello world matches only hello world, not hello world.

Stderr behavior:

• Stderr does not participate in pass/fail matching.
• If a test fails and stderr is non-empty, stderr is appended to the rendered failure output for diagnostics.

Timeouts:

• The current runner has no built-in timeout.

Compile-error tests:

• // expect-compile-error: matches by substring containment within the rendered compiler diagnostic output.

Supported Directives

Directives must appear in the leading comment block before the first non-comment, non-empty source line.

// expect: ...

Syntax:

// expect: 42

Meaning:

• Declares the exact expected stdout for a successful test.
• Implies default expected exit code 0 unless overridden by // exit:.

Examples:

// expect: ok
Ashes.IO.print("ok")

// expect: empty
Ashes.IO.print("empty")

Common mistakes:

• // expect: empty does not mean empty stdout. It literally expects the text empty.
• Multi-line expected output is not supported as a single directive; only the remainder of the directive line is captured.

// expect-stderr: ...

Syntax:

// expect-stderr: compilation failed

Meaning:

• Declares the exact expected stderr, independently of // expect: stdout.
• Trailing whitespace is trimmed in the same way as stdout.
• May be combined with // exit: to test controlled failure reporting.

// expect-stderr-contains: ...

Syntax:

// expect-stderr-contains: compilation failed

Meaning:

• Requires stderr to contain the declared text.
• Use this for cross-target tests launched through a host adapter such as Wine, which may write its own diagnostics to the inherited stderr stream.
• May be combined with // expect: and // exit:.

// expect-compile-error: ...

Syntax:

// expect-compile-error: Could not resolve module 'Missing'

Meaning:

• The test is expected to fail during compilation.
• The runner expects exit code 1.
• The provided text must appear in the rendered compiler diagnostics.

Example:

// expect-compile-error: Undefined variable
Ashes.IO.print(missing)

Common mistakes:

• This is for compile-time failures only, not runtime panics.

// exit: N

Syntax:

// exit: 1

Meaning:

• Overrides the expected process exit code.
• If omitted, the default is 0.

Example:

// exit: 1
// expect: boom
Ashes.IO.panic("boom")

Common mistakes:

• If you expect a runtime failure, set both // exit: and // expect:.

// executable-directory: path

Places the generated test executable beneath path in the isolated fixture. For example, // executable-directory: install/bin makes Ashes.IO.Environment.executableDirectory(Unit) report the fixture's install/bin directory. The default is the fixture root.

// working-directory: path

Starts the generated executable from path in the isolated fixture. For example, // working-directory: workspace/project/src lets a test exercise upward project discovery while the executable remains in an installed layout. The default is the fixture root.

// stdin: ...

Syntax:

// stdin: hello\nworld\n

Meaning:

• Sends the decoded string to the test process standard input.
• Supported escapes: \n, \r, \t, \\.

Example:

// stdin: hello\n
// expect: hello
match Ashes.IO.readLine() with
    | Some(text) -> Ashes.IO.print(text)
    | None -> Ashes.IO.print("none")

// file: path = content

Syntax:

// file: input.txt = hello

Meaning:

• Creates a UTF-8 text fixture file beneath the isolated fixture root before execution.
• Paths must be relative and cannot escape the temporary test directory.

Example:

// file: data/input.txt = hello
// expect: hello
match Ashes.IO.File.readText("data/input.txt") with
    | Ok(text) -> Ashes.IO.print(text)
    | Error(msg) -> Ashes.IO.print(msg)

// file-bytes: path = HEX HEX ...

Syntax:

// file-bytes: bad.bin = FF FE FD

Meaning:

• Creates a binary fixture file using hexadecimal byte values.

Example:

// file-bytes: bad.bin = FF FE FD
// expect: Ashes.IO.File.readText() encountered invalid UTF-8
match Ashes.IO.File.readText("bad.bin") with
    | Ok(text) -> Ashes.IO.print(text)
    | Error(msg) -> Ashes.IO.print(msg)

// tcp-server: accept

Syntax:

// tcp-server: accept

Meaning:

• Starts a loopback TCP server fixture on an ephemeral port.
• The test source may reference the placeholder __TCP_PORT__, which is substituted before compilation.

// tcp-expect: ...

Syntax:

// tcp-expect: hello

Meaning:

• The loopback TCP fixture expects the client to send exactly the provided UTF-8 text.

// tcp-send: ...

Syntax:

// tcp-send: hello

Meaning:

• The loopback TCP fixture sends the provided UTF-8 text to the client after accept.

Unknown Directives

Unknown directives are currently ignored by the TestRunner.

That means:

• they do not fail the test by themselves
• they are not interpreted as supported behavior
• they should not be relied on as part of the stable tooling surface

Formatting Interaction

• Tests may include leading // directives; examples should not use test directives.
• ashes fmt formats .ash source but preserves the leading comment block.
• // fmt-skip: ... is not interpreted by the TestRunner. It is used by CI and formatting verification scripts to exempt intentionally malformed fixtures from formatting checks.

Compilation Pipelines

ashes test uses the normal optimized compiler path by default. The --pipeline option selects the Ashes semantic pipeline independently of LLVM's -O0 through -O3 setting:

• optimized lowers, runs IrOptimizer, and then invokes LLVM. This matches ashes compile and ashes run and is the default.
• lowered bypasses IrOptimizer and sends raw lowered IR to LLVM. LLVM still uses the selected -O level.
• both compiles and runs every runnable test through both paths, and the test passes only when both executions satisfy the same directives. Compile-error tests compile once because they stop before the semantic optimizer boundary.

Full CI uses --pipeline both so semantic optimization cannot hide a lowering/runtime regression and the optimizer cannot introduce a regression that the raw-lowered path misses. In this mode, native execution timings are labeled opt=<time> and lowered=<time>; compilation remains excluded.

Failure Reporting

When a test fails, the runner reports:

• the file name
• expected exit code
• actual exit code
• expected output
• actual output
• stderr, when present and relevant to the failure

This keeps failures deterministic and suitable for CI output.

Compiler Memory Regressions

The .ash directive runner checks output and exit status; it does not measure resident memory. Compiler/runtime memory regressions live in src/Ashes.Tests/LinuxBackendCoverageTests.cs, where the test compiles a native program and measures child ru_maxrss through Python's resource.getrusage.

Memory-management changes must test growth, not only one peak:

1. Run the same workload at three increasing scales (normally 2,000, 10,000, and 50,000 iterations).
2. Verify program output at every scale so bounded memory cannot hide corruption or premature release.
3. Bound both total growth (first to last) and late growth (middle to last). A fixed allocator high-water mark may raise the first sample; proportional late growth indicates retained objects.
4. Cover unique and shared ownership where relevant. For graph changes also exercise transfer, duplicate, recursive drop, null reuse fallback, and unreturned branch/loop owners.

The permanent matrix includes lists/ADTs/records/tuples, Strings, Bytes, BigInts, closures, TCO accumulators, task/capability regions, HTTP keep-alive, parallel workers, persistent Map/HashMap updates, and the shipped 1BRC program. An executable arena or RC leak is a release blocker even when a different allocation path passes.

Use TUnit filters while iterating:

dotnet run --project src/Ashes.Tests -- --no-progress \
  --treenode-filter "/*/*/LinuxBackendCoverageTests/<test-name>"

Run the full compiler suite at phase boundaries. RSS tests are Linux-native; cross-target correctness still runs through qemu/Wine or structural target checks as described in the development guide. A win-x64 test run on Linux starts one Wine server with a 15-second idle timeout before the first program (wineserver -p15), so the programs share it instead of each booting and tearing down a server of their own; the server exits by itself after the run.