tractor/docs/guide/testing.rst

Testing tips
============

``tractor``'s test suite is a different kind of beast than your
average single-proc pytest run: nearly every test spawns a real
**process tree**, hammers on cancellation under structured
concurrency (SC), and tears the whole thing down again — hundreds
of times per session. This page collects the tips, knobs and
one-liners that make hacking on (and with) the suite pleasant.

Running the suite
-----------------

This is a uv_-managed project, so after cloning it's just::

    uv sync --dev
    uv run pytest tests/

Expect a *lot* of process churn; the suite is effectively a
rolling chaos exercise for the runtime.

The classic fix-iterate loop when something breaks::

    # stop at the first failure
    uv run pytest tests/ -x

    # then iterate on just the failures til green
    uv run pytest --lf -x

``--lf`` (last-failed) re-runs only what failed previously, so
combined with ``-x`` you get a tight one-test-at-a-time repair
loop.

Suite-specific flags
********************

The repo auto-loads the bundled ``tractor._testing.pytest`` plugin
(via ``addopts`` in ``pyproject.toml``) which adds a few extra
flags:

- ``--spawn-backend <key>``: pick the process spawn backend for
  the session (default ``'trio'``); same keys as the
  ``start_method`` runtime argument,

- ``--tpt-proto <key> [...]``: which IPC transport(s) opting-in
  suites should run against, eg. ``--tpt-proto uds``,

- ``--tpdb`` / ``--debug-mode``: flip on the ``debug_mode``
  fixture so debugger-aware tests boot their trees with the
  crash-REPL enabled,

- ``--enable-stackscope``: install the ``SIGUSR1`` task-tree dump
  handler in pytest *and* every spawned subactor — much lighter
  than a full debug-mode run when you only need stack visibility
  during a hang hunt,

- ``--ll <level>`` / ``--tl <level-or-spec>``: console loglevels;
  ``--tl`` targets the ``tractor``-as-runtime logger and accepts
  a per-subsystem spec like ``'devx:runtime,trionics:cancel'``.

Watch the tree grow
-------------------

The single most useful trick while the suite (or any ``tractor``
app) runs: keep a live ``pstree`` view going in a side terminal::

    watch -n 0.1 "pstree -a $(pgrep -f pytest)"

You'll see actor processes pop in and out of existence as each
test builds and reaps its tree. Launch it *after* pytest is up
(the pid is substituted once, at ``watch`` startup).

Every subactor also sets its OS process title (via
``setproctitle``) to ``_subactor[<name>@<uuid-prefix>]`` so the
tree view shows *which actor is which* at a glance — and targeted
greps stay easy::

    pgrep -af '_subactor\['

For a single example script, the repo's signature incantation
spawns the watcher alongside your program and cleans it up after::

    $TERM -e watch -n 0.1 "pstree -a $$" \
        & python examples/parallelism/single_func.py \
        && kill $!

Env-var knobs
-------------

Two env-vars override their corresponding runtime arguments
*globally* — no application (or test) code changes required:

``TRACTOR_SPAWN_METHOD``
    Wins over any caller-passed ``start_method`` so you can drive
    the whole suite (or any app) under a different spawn backend::

        TRACTOR_SPAWN_METHOD=mp_spawn uv run pytest tests/ -x

``TRACTOR_LOGLEVEL``
    Wins over any caller-passed ``loglevel``; crank (or silence)
    runtime console verbosity wholesale::

        TRACTOR_LOGLEVEL=cancel uv run pytest tests/ -x -s

``TRACTOR_ENABLE_STACKSCOPE``
    Force-install the ``SIGUSR1`` task-tree dump handler in every
    actor, debug-mode or not; then
    ``pkill --signal SIGUSR1 -f <part-of-cmd>`` dumps every
    actor's live ``trio`` task tree.

Debug mode vs. pytest capture
-----------------------------

The tree-wide crash-to-REPL experience (``debug_mode=True`` plus
``await tractor.pause()``) requires a **real tty**, and pytest's
default output capturing swallows exactly that. When you want to
interact with the REPL from inside a test run, disable capture::

    uv run pytest tests/test_foo.py -x -s

(``-s`` is shorthand for ``--capture=no``.)

Tests should request the ``debug_mode`` fixture (driven by the
``--tpdb`` flag) rather than hard-coding it, so that normal CI
runs stay non-interactive.

For *automated* REPL interaction — asserting on prompt output,
sending debugger commands — you can't just turn capture off;
instead do what ``tests/devx/`` does: drive a child Python program
through pexpect_ on a real pseudo-tty and pattern-match the
``(Pdb+)`` prompts. See ``tests/devx/test_debugger.py`` for many
worked patterns.

Examples *are* tests
--------------------

Every script under ``examples/`` is run as a subprocess by
``tests/test_docs_examples.py``; since these docs
``literalinclude`` those same scripts, the code you read here is
CI-verified on every push and can never silently rot B)

Conventions when adding a new example:

- make it a standalone runnable script with the usual guard::

      if __name__ == '__main__':
          trio.run(main)

- it must exit cleanly (returncode ``0``) within the per-example
  timeout (~16s locally, with headroom auto-added in CI and under
  cpu-freq scaling) — keep sleeps short,

- any stderr line containing ``Error`` fails the test, so silence
  or assert-around expected error output,

- don't crank ``tractor`` logging inside an example: subprocess
  pipe **backpressure can deadlock** the run (ask us how we
  know..),

- filenames starting with ``_`` are skipped (the WIP convention),
  as are the special subdirs (``debugging/``, ``integration/``,
  ``advanced_faults/``, ``trio/``) which are driven by their own
  dedicated suites instead.

Drop your script in, run the example suite, profit::

    uv run pytest tests/test_docs_examples.py -x

Zombie cleanup
--------------

First, the contract: ``tractor`` **always** reaps its children —
if you can create a zombie process (without resorting to
untrappable signals) it **is a bug**, please report it!

That said, while hacking on the *runtime itself* you can
definitely wedge things — a ``SIGKILL``-ed pytest, a half-broken
spawn backend — and strand subactor procs plus their shm segments
and UDS socket files. The repo ships a dedicated cleanup tool::

    uv run scripts/tractor-reap --shm --uds

It's SC-polite even as a reaper: matched processes get ``SIGINT``
first with a bounded grace window — so actor runtimes can run
their ``trio`` teardown paths — escalating to ``SIGKILL`` only as
a last resort. The ``--shm`` sweep unlinks ``/dev/shm/`` segments
that no live process has open (it leans on psutil_, already in
your dev venv, to check live mappings and fds) and ``--uds``
clears socket files whose binder pid is dead.

Testing your own ``tractor`` app
--------------------------------

The same plugin the suite uses ships in the package, so your
project can load it too::

    [tool.pytest.ini_options]
    addopts = ['-p tractor._testing.pytest']

That buys you the CLI flags above plus a set of fixtures —
``loglevel``, ``debug_mode``, ``reg_addr`` (a session-unique
registrar address so concurrent runs and other live ``tractor``
apps on the host can't cross-talk) — and the ``@tractor_test``
decorator:

.. code:: python

    import tractor
    from tractor._testing import tractor_test

    @tractor_test
    async def test_my_service(
        reg_addr: tuple,
        loglevel: str,
    ):
        # already inside a root actor's trio task!
        async with tractor.open_nursery() as an:
            ...

The decorator boots a root actor around your (async) test fn,
wires any of the special fixtures you declare (``reg_addr``,
``loglevel``, ``start_method``, ``debug_mode``) into
``open_root_actor()``, and runs the body as the root-most task
under a wall-clock ``trio.fail_after()`` guard.

General advice that has served this suite well:

- bound waits with ``trio.fail_after()`` *inside* tests; global
  pytest timeout plugins interact badly with multi-process
  ``trio`` teardown,

- use the ``reg_addr`` fixture (or otherwise randomize your
  registry addrs) so leftover registrars from prior runs can't
  contaminate lookups,

- assert on **structured outcomes** — eg.
  ``RemoteActorError.boxed_type`` or
  ``ContextCancelled.canceller`` — not on log text.

.. note::
   ``tractor._testing`` is still an underscore-internal namespace:
   shipped and handy, but its API may shift between alpha
   releases.

(This page exists thanks to the ask in `#126`_.)

.. seealso::
   - :doc:`/guide/discovery` — how registrar wiring (the thing
     ``reg_addr`` randomizes) works in the runtime proper.
   - :doc:`/project/dev-tips` — contributor-oriented extras:
     releases, log-system tracing, tree-monitoring recipes.

.. _uv: https://docs.astral.sh/uv/
.. _pexpect: https://pexpect.readthedocs.io/en/stable/
.. _psutil: https://psutil.readthedocs.io/en/latest/
.. _#126: https://github.com/goodboy/tractor/issues/126