Testing tips ============ ``tractor``'s test suite is a different kind of beast than your average single-proc pytest run: nearly every test spawns a real **process tree**, hammers on cancellation under structured concurrency (SC), and tears the whole thing down again — hundreds of times per session. This page collects the tips, knobs and one-liners that make hacking on (and with) the suite pleasant. Running the suite ----------------- This is a uv_-managed project, so after cloning it's just:: uv sync --dev uv run pytest tests/ Expect a *lot* of process churn; the suite is effectively a rolling chaos exercise for the runtime. The classic fix-iterate loop when something breaks:: # stop at the first failure uv run pytest tests/ -x # then iterate on just the failures til green uv run pytest --lf -x ``--lf`` (last-failed) re-runs only what failed previously, so combined with ``-x`` you get a tight one-test-at-a-time repair loop. Suite-specific flags ******************** The repo auto-loads the bundled ``tractor._testing.pytest`` plugin (via ``addopts`` in ``pyproject.toml``) which adds a few extra flags: - ``--spawn-backend ``: pick the process spawn backend for the session (default ``'trio'``); same keys as the ``start_method`` runtime argument, - ``--tpt-proto [...]``: which IPC transport(s) opting-in suites should run against, eg. ``--tpt-proto uds``, - ``--tpdb`` / ``--debug-mode``: flip on the ``debug_mode`` fixture so debugger-aware tests boot their trees with the crash-REPL enabled, - ``--enable-stackscope``: install the ``SIGUSR1`` task-tree dump handler in pytest *and* every spawned subactor — much lighter than a full debug-mode run when you only need stack visibility during a hang hunt, - ``--ll `` / ``--tl ``: console loglevels; ``--tl`` targets the ``tractor``-as-runtime logger and accepts a per-subsystem spec like ``'devx:runtime,trionics:cancel'``. Watch the tree grow ------------------- The single most useful trick while the suite (or any ``tractor`` app) runs: keep a live ``pstree`` view going in a side terminal:: watch -n 0.1 "pstree -a $(pgrep -f pytest)" You'll see actor processes pop in and out of existence as each test builds and reaps its tree. Launch it *after* pytest is up (the pid is substituted once, at ``watch`` startup). Every subactor also sets its OS process title (via ``setproctitle``) to ``_subactor[@]`` so the tree view shows *which actor is which* at a glance — and targeted greps stay easy:: pgrep -af '_subactor\[' For a single example script, the repo's signature incantation spawns the watcher alongside your program and cleans it up after:: $TERM -e watch -n 0.1 "pstree -a $$" \ & python examples/parallelism/single_func.py \ && kill $! Env-var knobs ------------- Two env-vars override their corresponding runtime arguments *globally* — no application (or test) code changes required: ``TRACTOR_SPAWN_METHOD`` Wins over any caller-passed ``start_method`` so you can drive the whole suite (or any app) under a different spawn backend:: TRACTOR_SPAWN_METHOD=mp_spawn uv run pytest tests/ -x ``TRACTOR_LOGLEVEL`` Wins over any caller-passed ``loglevel``; crank (or silence) runtime console verbosity wholesale:: TRACTOR_LOGLEVEL=cancel uv run pytest tests/ -x -s ``TRACTOR_ENABLE_STACKSCOPE`` Force-install the ``SIGUSR1`` task-tree dump handler in every actor, debug-mode or not; then ``pkill --signal SIGUSR1 -f `` dumps every actor's live ``trio`` task tree. Debug mode vs. pytest capture ----------------------------- The tree-wide crash-to-REPL experience (``debug_mode=True`` plus ``await tractor.pause()``) requires a **real tty**, and pytest's default output capturing swallows exactly that. When you want to interact with the REPL from inside a test run, disable capture:: uv run pytest tests/test_foo.py -x -s (``-s`` is shorthand for ``--capture=no``.) Tests should request the ``debug_mode`` fixture (driven by the ``--tpdb`` flag) rather than hard-coding it, so that normal CI runs stay non-interactive. For *automated* REPL interaction — asserting on prompt output, sending debugger commands — you can't just turn capture off; instead do what ``tests/devx/`` does: drive a child Python program through pexpect_ on a real pseudo-tty and pattern-match the ``(Pdb+)`` prompts. See ``tests/devx/test_debugger.py`` for many worked patterns. Examples *are* tests -------------------- Every script under ``examples/`` is run as a subprocess by ``tests/test_docs_examples.py``; since these docs ``literalinclude`` those same scripts, the code you read here is CI-verified on every push and can never silently rot B) Conventions when adding a new example: - make it a standalone runnable script with the usual guard:: if __name__ == '__main__': trio.run(main) - it must exit cleanly (returncode ``0``) within the per-example timeout (~16s locally, with headroom auto-added in CI and under cpu-freq scaling) — keep sleeps short, - any stderr line containing ``Error`` fails the test, so silence or assert-around expected error output, - don't crank ``tractor`` logging inside an example: subprocess pipe **backpressure can deadlock** the run (ask us how we know..), - filenames starting with ``_`` are skipped (the WIP convention), as are the special subdirs (``debugging/``, ``integration/``, ``advanced_faults/``, ``trio/``) which are driven by their own dedicated suites instead. Drop your script in, run the example suite, profit:: uv run pytest tests/test_docs_examples.py -x Zombie cleanup -------------- First, the contract: ``tractor`` **always** reaps its children — if you can create a zombie process (without resorting to untrappable signals) it **is a bug**, please report it! That said, while hacking on the *runtime itself* you can definitely wedge things — a ``SIGKILL``-ed pytest, a half-broken spawn backend — and strand subactor procs plus their shm segments and UDS socket files. The repo ships a dedicated cleanup tool:: uv run scripts/tractor-reap --shm --uds It's SC-polite even as a reaper: matched processes get ``SIGINT`` first with a bounded grace window — so actor runtimes can run their ``trio`` teardown paths — escalating to ``SIGKILL`` only as a last resort. The ``--shm`` sweep unlinks ``/dev/shm/`` segments that no live process has open (it leans on psutil_, already in your dev venv, to check live mappings and fds) and ``--uds`` clears socket files whose binder pid is dead. Testing your own ``tractor`` app -------------------------------- The same plugin the suite uses ships in the package, so your project can load it too:: [tool.pytest.ini_options] addopts = ['-p tractor._testing.pytest'] That buys you the CLI flags above plus a set of fixtures — ``loglevel``, ``debug_mode``, ``reg_addr`` (a session-unique registrar address so concurrent runs and other live ``tractor`` apps on the host can't cross-talk) — and the ``@tractor_test`` decorator: .. code:: python import tractor from tractor._testing import tractor_test @tractor_test async def test_my_service( reg_addr: tuple, loglevel: str, ): # already inside a root actor's trio task! async with tractor.open_nursery() as an: ... The decorator boots a root actor around your (async) test fn, wires any of the special fixtures you declare (``reg_addr``, ``loglevel``, ``start_method``, ``debug_mode``) into ``open_root_actor()``, and runs the body as the root-most task under a wall-clock ``trio.fail_after()`` guard. General advice that has served this suite well: - bound waits with ``trio.fail_after()`` *inside* tests; global pytest timeout plugins interact badly with multi-process ``trio`` teardown, - use the ``reg_addr`` fixture (or otherwise randomize your registry addrs) so leftover registrars from prior runs can't contaminate lookups, - assert on **structured outcomes** — eg. ``RemoteActorError.boxed_type`` or ``ContextCancelled.canceller`` — not on log text. .. note:: ``tractor._testing`` is still an underscore-internal namespace: shipped and handy, but its API may shift between alpha releases. (This page exists thanks to the ask in `#126`_.) .. seealso:: - :doc:`/guide/discovery` — how registrar wiring (the thing ``reg_addr`` randomizes) works in the runtime proper. - :doc:`/project/dev-tips` — contributor-oriented extras: releases, log-system tracing, tree-monitoring recipes. .. _uv: https://docs.astral.sh/uv/ .. _pexpect: https://pexpect.readthedocs.io/en/stable/ .. _psutil: https://psutil.readthedocs.io/en/latest/ .. _#126: https://github.com/goodboy/tractor/issues/126