Compare commits

...

511 Commits

Author SHA1 Message Date
Gud Boi 90a7a04b72 Add snapshot evidence to cancel-cascade MTF issue doc
Append "Snapshot evidence (2026-05-13)" section to
`cancel_cascade_too_slow_under_main_thread_forkserver_issue.md`
documenting `fail_after_w_trace` diag capture results for
`test_nested_multierrors` under the MTF backend — reproduction cmd,
ptree analysis, observed hang signature, and updated triage plan.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 5372fd178a)
2026-06-29 19:20:21 -04:00
Gud Boi 48c4714025 Add `main_thread_forkserver` CI matrix rows
Add `capture` dimension to CI matrix so fork-based
backends run `--capture=sys` (fork-child × `--capture=fd`
is a known deadlock). Non-fork backends keep `fd`.

Deats,
- two `include:` rows for `main_thread_forkserver` on
  linux py3.13: tcp + uds, both `capture: 'sys'`
- job name updated to show `capture=` mode
- timeout bumped 16 -> 20 min to accommodate the
  additional matrix cells
- `--capture=${{ matrix.capture }}` replaces hardcoded
  `--capture=fd`

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit a24600f1a7)
2026-06-29 19:20:21 -04:00
Gud Boi b17ff23a53 Adjust `subint_forkserver` docs to match stub impl
Comment/docstring updates: `subint_forkserver` is a clean
`NotImplementedError` stub — not an alias to variant-1
(`main_thread_forkserver`). Key reserved in-place (not aliased) so
the subint-hosted-child impl can flip without API churn once
jcrist/msgspec#1026 unblocks PEP 684 subints.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 7d1e4462d4)
2026-06-29 19:20:21 -04:00
Gud Boi 7256570954 Add `wait_for_peer_or_proc_death()` to `_spawn`
Race `IPCServer.wait_for_peer(uid)` against the sub-proc's
`.wait()` inside a `trio` nursery; whichever completes first
cancels the other.

Prevents the spawning task from parking forever on an unsignalled
`_peer_connected[uid]` event when a sub-actor dies during boot
(e.g. crashed on import before reaching `_actor_child_main`).
Instead of hanging, raises `ActorFailure` w/ the proc's exit code
for clean supervisor error reporting.

Also,
- use the new racer in `main_thread_forkserver_proc()` spawn path.
- keep `proc_wait` generic so each backend passes its own callable
  (`trio.Process.wait`, `_ForkedProc.wait`, etc.).

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 3b0724eba8)
2026-06-29 19:20:21 -04:00
Gud Boi c47ff10047 Add `terminate()` to `_ForkedProc`
Sends `SIGTERM` (graceful shutdown) instead of the existing `kill()`
which sends `SIGKILL`. Mirrors the `trio.Process.terminate()`
/ `multiprocessing.Process.terminate()` interface.

Used by `ActorNursery.cancel()`'s per-child escalation when
`Portal.cancel_actor()` raises `ActorTooSlowError`, and by the legacy
`hard_kill=True` branch. Swallows `ProcessLookupError` (child already
dead) same as `kill()`.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 4c00913b3b)
2026-06-29 19:20:21 -04:00
Gud Boi d28acfee25 Add cancel-cascade `TooSlowError` flake analysis
Document the ~0.3% rotating `trio.TooSlowError`
flake under `--spawn-backend=main_thread_forkserver`
full-suite runs. Root cause: `hard_kill`'s per-sub
1.6s graceful timeout compounding across N subactors
in a cancel cascade, plus cumulative autouse-reaper
teardown overhead.

Covers symptom, observed flaking tests, root-cause
family, ranked mitigations (cap bump -> CPU-count-
aware cap -> `pytest-rerunfailures` -> `hard_kill`
tuning -> targeted profiling), and a verification
protocol.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 60ce713016)
2026-06-29 19:20:21 -04:00
Gud Boi bd25544a6b Drop subint-family gate from `main_thread_forkserver`
`main_thread_forkserver` doesn't actually need py3.14
`concurrent.interpreters` (PEP 734) — it forks from a
non-trio worker thread and runs `_trio_main` in the child,
same shape as `trio_proc`. The previous `_has_subints`
gate + subint-family `case` arm were a copy-paste error.

In `tractor.spawn._main_thread_forkserver`,
- drop the `_has_subints` import + the `RuntimeError`
  raise in `main_thread_forkserver_proc()`.
- drop the now-unused `import sys` (only used by the
  prior error msg).

In `tractor.spawn._spawn.try_set_start_method()`,
- pull `'main_thread_forkserver'` out of the subint-
  family arm (which still gates on `_has_subints`).
- merge it into the `'trio'` arm — both set `_ctx = None`
  bc neither needs an `mp.context`.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit fc5e80fea5)
2026-06-29 19:20:21 -04:00
Gud Boi 76f6440f91 Refine fork-survival docs + `EBADF` handling
Two cleanup tweaks in `_main_thread_forkserver`:

Doc, "what survives the fork?" section — expand the
"non-calling threads are gone in the child" claim with
the precise execution-vs-memory split that reconciles
this module's prior framing with trio's (canonical
[python-trio/trio#1614][trio-1614]) "leaked stacks"
framing:

- execution-side: only the calling thread runs
  post-fork; all others never execute another
  instruction.
- memory-side: those non-running threads' stacks +
  per-thread heap structures are still COW-inherited
  as orphaned bytes — what trio means by "leaked".

Same POSIX reality, opposite sides; the table is
extended to a 4-col `parent | child (executing) |
child (memory)` layout to make both views explicit.
Also blank-line-padded the bulleted hazard classes
for cleaner markdown rendering.

[trio-1614]: https://github.com/python-trio/trio/issues/1614

Code, `_close_inherited_fds()` log noise — split the
catch-all `except OSError` into:

- `EBADF` — benign race where the dirfd that
  `os.listdir('/proc/self/fd')` itself opened ends up
  in `candidates`, then auto-closes before the loop
  reaches it. Demote to `log.debug()` + `continue`;
  prior `log.exception` drowned the post-fork log
  channel with stack traces every spawn.
- other errnos (EIO / EPERM / EINTR / ...) keep the
  loud `log.exception` surface — those ARE genuinely
  unexpected.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 8c730193f9)
2026-06-29 19:20:21 -04:00
Gud Boi f9689d4e68 Guard `subint_forkserver` stub against re-alias
Add `test_subint_forkserver_key_errors_cleanly` — a tn-tier
regression guard that pins down the variant-2 reservation
contract: the `'subint_forkserver'` key in
`_spawn._methods` MUST raise `NotImplementedError` today,
not silently dispatch to `main_thread_forkserver_proc`.

The transient alias-state existed briefly during the rename
(commit `57dae0e4`'s "Split forkserver backend into variant
1/2 mods" landed the alias; `5e83881f` flipped it to the
stub). Without a guard, a future refactor could easily
re-collapse the two keys back to a single coro and silently
break the variant-1 / variant-2 contract.

Also asserts the stub's error msg surfaces the two pointers
an operator hitting it actually needs:

- `'main_thread_forkserver'` — the working backend they
  prolly meant,
- `'msgspec#1026'` — the upstream blocker that has to land
  before variant-2 can ship.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit cbdf1eb6db)
2026-06-29 19:20:21 -04:00
Gud Boi 4569e800ab Migrate test/smoketest imports + rename test file
Rename `tests/spawn/test_subint_forkserver.py` →
`test_main_thread_forkserver.py` and migrate its imports +
internal refs to the new canonical names:

- `fork_from_worker_thread`, `wait_child` → from
  `tractor.spawn._main_thread_forkserver`.
- `run_subint_in_worker_thread` → still from `_subint_forkserver`
  (variant-2 primitive).
- Module docstring + tier-3 fixture + the `*_spawn_basic` test fn
  renamed for variant-1-honesty.
- Orphan-harness subprocess argv flipped from `'subint_forkserver'`
  → `'main_thread_forkserver'`.

`ai/conc-anal/subint_fork_from_main_thread_smoketest.py` imports split
the same way.

`tractor/spawn/_subint_forkserver.py` drops the backward- compat
re-exports of the fork primitives — the only consumers (test file
+ smoketest) now import from `_main_thread_forkserver` directly.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 9f0709eee2)
2026-06-29 19:20:21 -04:00
Gud Boi e924b6952c Add `subint_forkserver_proc` stub, flip dispatch, prune
Reduce `_subint_forkserver.py` to its variant-2 placeholder shape:

- Add `subint_forkserver_proc` async stub raising `NotImplementedError`
  with a redirect msg pointing at the working variant-1 backend
  (`main_thread_forkserver`), jcrist/msgspec#1026 (upstream PEP 684
  blocker), and #379 (subint umbrella).

- `tractor.spawn._spawn._methods['subint_forkserver']` now dispatches to
  the stub instead of aliasing the variant-1 coroutine
  — `--spawn-backend=subint_forkserver` errors cleanly.

- Drop now-dead module-scope: `ChildSigintMode`
  / `_DEFAULT_CHILD_SIGINT` defs, `_has_subints` try/except (replaced
  with import from `._subint`), unused imports (`partial`, `Literal`,
  `sys`, msgtypes/pretty_struct, `current_actor`,
  `cancel_on_completion`/`soft_kill`, `_server` TYPE_CHECKING).

- Backward-compat re-exports of fork primitives kept until the follow-up
  commit migrates external test imports.

- `tests/spawn/test_subint_forkserver.py::forkserver_spawn_method`
  fixture: flip hardcoded `'subint_forkserver'`
  → `'main_thread_forkserver'` so the test still exercises the working
  backend (full file rename comes in the test-import migration commit).

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 5e83881f10)
2026-06-29 19:20:21 -04:00
Gud Boi 4d6d613c30 Split forkserver backend into variant 1/2 mods
The `subint_forkserver` name was always aspirational —
today's impl forks from a regular main-interp worker
thread and the child runs trio on its own main interp;
NO subinterp anywhere in parent or child. Splitting the
backend into two clearly-named variants drops the lie:

- **variant 1** — `main_thread_forkserver` (the working
  impl). New `SpawnMethodKey` literal + `_methods`
  dispatch entry + `_runtime.Actor._from_parent()`
  match-arm. The spawn-coro `subint_forkserver_proc`
  moves to `_main_thread_forkserver` and is renamed
  `main_thread_forkserver_proc()`.

- **variant 2** — `subint_forkserver` (future, reserved).
  Module shrinks to a placeholder describing the
  variant-2 design (subint-isolated child runtime, gated
  on jcrist/msgspec#1026 + PEP 684). Today the legacy
  `'subint_forkserver'` key aliases to
  `main_thread_forkserver_proc` so existing
  `--spawn-backend=subint_forkserver` invocations keep
  working; flipped to a `NotImplementedError` stub in a
  follow-up.

Deats,
- `Actor._from_parent()` spawn-method gate now accepts
  both `'main_thread_forkserver'` and
  `'subint_forkserver'` (both go through the
  IPC-`SpawnSpec` path).
- the variant-1 spawn-coro stamps its own `SpawnSpec` /
  log lines with `spawn_method='main_thread_forkserver'`
  so subactor renders reflect the actual mechanism.
- docstring reorg: trio×fork hazard breakdown, POSIX
  fork-survival semantics, in-process-vs-stdlib
  forkserver design notes, and the TODO/cleanup section
  all move from `_subint_forkserver` to
  `_main_thread_forkserver` (lives with the working
  code). `_subint_forkserver` keeps a tight forward-
  looking doc that motivates the reserved key.
- `run_subint_in_worker_thread()` stays in
  `_subint_forkserver` as the companion primitive — it's
  the subint counterpart to `fork_from_worker_thread()`
  and will plug into the future variant-2 spawn-coro.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 57dae0e4a6)
2026-06-29 19:20:21 -04:00
Gud Boi d944eb5370 Extract fork primitives into `_main_thread_forkserver`
Move the truly-generic main-interp-worker-thread fork primitives
(`fork_from_worker_thread`, `_close_inherited_fds`, `_ForkedProc`,
`wait_child`, `_format_child_exit`) out of `_subint_forkserver.py` into
a sibling `_main_thread_forkserver.py` module so the primitive layer is
honestly named — none of these helpers touch a subint, they just fork
from a main-interp worker thread.

`_subint_forkserver.py` keeps its public surface intact via re-export so
any existing `from tractor.spawn._subint_forkserver import ...` callsite
still resolves.

Net: zero behavior change, preps the way for the upcoming spawn-method
key split where `main_thread_forkserver` ships as the working backend
and `subint_forkserver` becomes reserved for the future
subint-isolated-child variant (gated on jcrist/msgspec#1026).

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 99dade0fb3)
2026-06-29 19:20:21 -04:00
Gud Boi f9aeba4701 Doc future-subint payoffs for `_subint_forkserver`
Adds a "Future arch — what subints would buy us" section to
the module docstring, complementing the prior commit's
current-state rationale. Code is unchanged.

Frames the `subint` prefix as family-naming today (no actual
subinterp is created yet), then lays out the three concrete
wins that land once jcrist/msgspec#1026 unblocks PEP 684
isolated-mode subints:

- Cheaper forks — moving the parent's `trio.run()` into a
  subint shrinks the main-interp COW image the child inherits.
  The main interp becomes the literal forkserver: an
  intentionally-empty execution ctx whose only job is to call
  `os.fork()` cleanly.

- True parallelism — per-interp GIL means the forkserver
  thread on main and the trio thread on subint actually run in
  parallel. Spawn latency stops stalling the trio loop.

- Multi-actor-per-process — the architectural payoff. With
  per-interp-GIL subints, one process can host main + N
  subint-resident actor `trio.run()`s, and `os.fork()` reverts
  to the last-resort spawn (only when OS-level isolation is
  actually needed). Joins the story with the in-thread
  `_subint.py` backend: `subint` → in-process spawn,
  `subint_forkserver` → cross-process when a real OS boundary
  is required.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 4b5176e2c3)
2026-06-29 19:20:21 -04:00
Gud Boi 17bb7906d2 Doc `_subint_forkserver` design + fork semantics
Major expansion of the module docstring. Code is
unchanged; this lands the architectural reasoning that
was previously implicit, plus the POSIX/trio fork
mechanics the design relies on.

New sections:
- "Design rationale" — answers two implicit questions:
  (1) why a forkserver pattern at all (vs. forking
  directly from a trio task), (2) why in-process (vs.
  stdlib `mp.forkserver`'s sidecar process). Documents
  the three costs the in-process design avoids
  (sidecar lifecycle, per-spawn IPC, cold-start child)
  and the tradeoffs we accept in exchange (3.14-only,
  heavier than `to_thread.run_sync`).
- "Implementation status" — clarifies what's actually
  landed today vs. the envisioned arch: parent's
  `trio.run()` still lives on main interp (subint-
  hosted root gated on jcrist/msgspec#1026). Names
  why the "subint" prefix is correct anyway — same PR
  series as `_subint.py` / `_subint_fork.py`.
- "What survives the fork? — POSIX semantics" — POSIX
  preserves only the calling thread, so the
  `trio.run()` thread is gone in the child. Includes
  a small parent/child thread-survival table and
  covers the four artifact classes that DO cross the
  fork boundary (inherited fds, COW memory, Python
  thread state, user-level locks) and how each is
  handled.
- "FYI: how this dodges the `trio.run()` × `fork()`
  hazards" — itemizes each class of trio process-
  global state (wakeup-fd, `epoll`/`kqueue`,
  threadpool, cancel scopes / nurseries, `atexit`,
  foreign-language I/O) and explains how the
  forkserver-thread design avoids each.

Also,
- bump the gated msgspec issue link from
  `jcrist/msgspec#563` to `jcrist/msgspec#1026` (the
  PEP 684 isolated-mode tracker).

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 3ab99d557a)
2026-06-29 19:20:21 -04:00
Gud Boi 95b66a91c1 Log subint bootstrap excs + cancel-leak state
Two diagnostic gaps in `tractor.spawn._subint.subint_proc()` that hid
otherwise-silent failures, plus tracking-issue links on the two open
`subint_forkserver` follow-ups.

Deats,
- bootstrap-exc visibility: wrap the call to
  `_interpreters.exec(interp_id, bootstrap)` with
  `try/except BaseException` + `log.exception(...)`.
  * Without it, an `ImportError` / `SyntaxError` raised inside the
    dedicated driver thread goes only to Python's default thread
    excepthook — invisible to the parent, which then waits forever on
    `subint_exited.wait()`.
  * `?TODO` notes `anyio`'s `to_interpreter._interp_call` +
    `(retval, is_exception)` pattern as the next step for re-raising;
    skipped now bc it must coordinate with the `trio.Cancelled` paths
    around the existing `.wait()` calls.

- cancel-leak disambiguation: when the driver thread doesn't exit within
  `_HARD_KILL_TIMEOUT`, also log `_interpreters.is_running(interp_id)`
  as `subint_still_running=...` so the operator can tell "thread leaked,
  subint already done" apart from "thread alive bc subint is wedged".
  * pattern borrowed from `trio-parallel`'s `_sint.SintWorker.is_alive()`.

- `?TODO` near the `bootstrap` literal: future switch to
  `_interpreters.set___main___attrs()` — same API `anyio`
  uses in `to_interpreter._Worker.call()` — for passing
  non-`repr()`-roundtrippable values (`SpawnSpec` struct, callables,
  etc).
  * add cross-refs tracking issue `#379`.

Also,
- `Tracked at: [#449]` link on
  `subint_forkserver_test_cancellation_leak_issue.md`.
- `Tracked at: [#450]` link on
  `subint_forkserver_thread_constraints_on_pep684_issue.md`.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 54561959e6)
2026-06-29 19:20:21 -04:00
Gud Boi 88a90eaa4f Tighten orphan-SIGINT xfail to `strict=True`
Re-classify `test_orphaned_subactor_sigint_cleanup_DRAFT` from
flakey-env-sensitive (`strict=False` w/ "passes in isolation, flakey in
full suite") to a hard known-gap (`strict=True`) with the orphan-SIGINT
hang as the documented cause. The previous framing ("env pollution") let
the test silently pass when ordering happened to favor it; the new
framing forces an XPASS-as-FAIL the moment the underlying gap is
actually closed, so we can drop the mark intentionally instead of
accidentally.

Reason text + leading `# Known-gap test —` comment both point at
`ai/conc-anal/subint_forkserver_orphan_sigint_hang_issue.md`
for the full diagnosis.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 44bdb1697c)
2026-06-29 19:20:21 -04:00
Gud Boi 4d18eb6044 Pin forkserver hang to pytest `--capture=fd`
Sixth and final diagnostic pass — after all 4
cascade fixes landed (FD hygiene, pidfd wait,
`_parent_chan_cs` wiring, bounded peer-clear), the
actual last gate on
`test_nested_multierrors[subint_forkserver]`
turned out to be **pytest's default
`--capture=fd` stdout/stderr capture**, not
anything in the runtime cascade.

Empirical result: `pytest -s` → test PASSES in
6.20s. Default `--capture=fd` → hangs forever.

Mechanism: pytest replaces the parent's fds 1,2
with pipe write-ends it reads from. Fork children
inherit those pipes (since `_close_inherited_fds`
correctly preserves stdio). The error-propagation
cascade in a multi-level cancel test generates
7+ actors each logging multiple `RemoteActorError`
/ `ExceptionGroup` tracebacks — enough output to
fill Linux's 64KB pipe buffer. Writes block,
subactors can't progress, processes don't exit,
`_ForkedProc.wait` hangs.

Self-critical aside: I earlier tested w/ and w/o
`-s` and both hung, concluding "capture-pipe
ruled out". That was wrong — at that time fixes
1-4 weren't all in place, so the test was
failing at deeper levels long before reaching
the "produce lots of output" phase. Once the
cascade could actually tear down cleanly, enough
output flowed to hit the pipe limit. Order-of-
operations mistake: ruling something out based
on a test that was failing for a different
reason.

Deats,
- `subint_forkserver_test_cancellation_leak_issue
  .md`: new section "Update — VERY late: pytest
  capture pipe IS the final gate" w/ DIAG timeline
  showing `trio.run` fully returns, diagnosis of
  pipe-fill mechanism, retrospective on the
  earlier wrong ruling-out, and fix direction
  (redirect subactor stdout/stderr to `/dev/null`
  in fork-child prelude, conditional on
  pytest-detection or opt-in flag)
- `tests/test_cancellation.py`: skip-mark reason
  rewritten to describe the capture-pipe gate
  specifically; cross-refs the new doc section
- `tests/spawn/test_subint_forkserver.py`: the
  orphan-SIGINT test regresses back to xfail.
  Previously passed after the FD-hygiene fix,
  but the new `wait_for_no_more_peers(
  move_on_after=3.0)` bound in `async_main`'s
  teardown added up to 3s latency, pushing
  orphan-subactor exit past the test's 10s poll
  window. Real fix: faster orphan-side teardown
  OR extend poll window to 15s

No runtime code changes in this commit — just
test-mark adjustments + doc wrap-up.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit eceed29d4a)
(MTF-only portion: kept ai/conc-anal/subint_forkserver_test_cancellation_leak_issue.md tests/spawn/test_subint_forkserver.py)
2026-06-29 19:20:21 -04:00
Gud Boi 483a00ef18 Narrow forkserver hang to `async_main` outer tn
Fourth diagnostic pass — instrument `_worker`'s
fork-child branch (`pre child_target()` / `child_
target RETURNED rc=N` / `about to os._exit(rc)`)
and `_trio_main` boundaries (`about to trio.run` /
`trio.run RETURNED NORMALLY` / `FINALLY`). Test
config: depth=1/breadth=2 = 1 root + 14 forked =
15 actors total.

Fresh-run results,
- **9 processes complete the full flow**:
  `trio.run RETURNED NORMALLY` → `child_target
  RETURNED rc=0` → `os._exit(0)`. These are tree
  LEAVES (errorers) plus their direct parents
  (depth-0 spawners) — they actually exit
- **5 processes stuck INSIDE `trio.run(trio_
  main)`**: hit "about to trio.run" but never
  see "trio.run RETURNED NORMALLY". These are
  root + top-level spawners + one intermediate

The deadlock is in `async_main` itself, NOT the
peer-channel loops. Specifically, the outer
`async with root_tn:` in `async_main` never exits
for the 5 stuck actors, so the cascade wedges:

    trio.run never returns
      → _trio_main finally never runs
        → _worker never reaches os._exit(rc)
          → process never dies
            → parent's _ForkedProc.wait() blocks
              → parent's nursery hangs
                → parent's async_main hangs
                  → (recurse up)

The precise new question: **what task in the 5
stuck actors' `async_main` never completes?**
Candidates:
1. shielded parent-chan `process_messages` task
   in `root_tn` — but we cancel it via
   `_parent_chan_cs.cancel()` in `Actor.cancel()`,
   which only runs during
   `open_root_actor.__aexit__`, which itself runs
   only after `async_main`'s outer unwind — which
   doesn't happen. So the shield isn't broken in
   this path.
2. `actor_nursery._join_procs.wait()` or similar
   inline in the backend `*_proc` flow.
3. `_ForkedProc.wait()` on a grandchild that DID
   exit — but pidfd_open watch didn't fire (race
   between `pidfd_open` and the child exiting?).

Most specific next probe: add DIAG around
`_ForkedProc.wait()` enter/exit to see whether
pidfd-based wait returns for every grandchild
exit. If a stuck parent's `_ForkedProc.wait()`
never returns despite its child exiting → pidfd
mechanism has a race bug under nested forkserver.

Asymmetry observed in the cascade tree: some d=0
spawners exit cleanly, others stick, even though
they started identically. Not purely depth-
determined — some race condition in nursery
teardown when multiple siblings error
simultaneously.

No code changes — diagnosis-only.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 4d0555435b)
2026-06-29 19:20:21 -04:00
Gud Boi 245d563a91 Refine `subint_forkserver` cancel-cascade diag
Third diagnostic pass on
`test_nested_multierrors[subint_forkserver]` hang.
Two prior hypotheses ruled out + a new, more
specific deadlock shape identified.

Ruled out,
- **capture-pipe fill** (`-s` flag changes test):
  retested explicitly — `test_nested_multierrors`
  hangs identically with and without `-s`. The
  earlier observation was likely a competing
  pytest process I had running in another session
  holding registry state
- **stuck peer-chan recv that cancel can't
  break**: pivot from the prior pass. With
  `handle_stream_from_peer` instrumented at ENTER
  / `except trio.Cancelled:` / finally: 40
  ENTERs, ZERO `trio.Cancelled` hits. Cancel never
  reaches those tasks at all — the recvs are
  fine, nothing is telling them to stop

Actual deadlock shape: multi-level mutual wait.

    root              blocks on spawner.wait()
      spawner         blocks on grandchild.wait()
        grandchild    blocks on errorer.wait()
          errorer     Actor.cancel() ran, but proc
                      never exits

`Actor.cancel()` fired in 12 PIDs — but NOT in
root + 2 direct spawners. Those 3 have peer
handlers stuck because their own `Actor.cancel()`
never runs, which only runs when the enclosing
`tractor.open_nursery()` exits, which waits on
`_ForkedProc.wait()` for the child pidfd to
signal, which only signals when the child
process fully exits.

Refined question: **why does an errorer process
not exit after its `Actor.cancel()` completes?**
Three hypotheses (unverified):
1. `_parent_chan_cs.cancel()` fires but the
   shielded loop's recv is stuck in a way cancel
   still can't break
2. `async_main`'s post-cancel unwind has other
   tasks in `root_tn` awaiting something that
   never arrives (e.g. outbound IPC reply)
3. `os._exit(rc)` in `_worker` never runs because
   `_child_target` never returns

Next-session probes (priority order):
1. instrument `_worker`'s fork-child branch —
   confirm whether `child_target()` returns /
   `os._exit(rc)` is reached for errorer PIDs
2. instrument `async_main`'s final unwind — see
   which await in teardown doesn't complete
3. compare under `trio_proc` backend at the
   equivalent level to spot divergence

No code changes — diagnosis-only.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit ab86f7613d)
2026-06-29 19:20:21 -04:00
Gud Boi c4def0220f Surface silent failures in `_subint_forkserver`
Three places that previously swallowed exceptions silently now log via
`log.exception()` so they surface in the runtime log when something
weird happens — easier to track down sneaky failures in the
fork-from-worker-thread / subint-bootstrap primitives.

Deats,
- `_close_inherited_fds()`: post-fork child's per-fd `os.close()`
  swallow now logs the fd that failed to close. The comment notes the
  expected failure modes (already-closed-via-listdir-race,
  otherwise-unclosable) — both still fine to ignore semantically, but
  worth flagging in the log.
- `fork_from_worker_thread()` parent-side timeout branch: the
  `os.close(rfd)` + `os.close(wfd)` cleanup now logs each pipe-fd close
  failure separately before raising the `worker thread didn't return`
  RuntimeError.
- `run_subint_in_worker_thread._drive()`: when
  `_interpreters.exec(interp_id, bootstrap)` raises a `BaseException`,
  log the full call signature (interp_id + bootstrap) along with the
  captured exception, before stashing into `err` for the outer caller.

Behavior unchanged — only adds observability.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 458a35cf09)
2026-06-29 19:20:21 -04:00
Gud Boi 59d268b9d4 Doc ruled-out fix + capture-pipe aside
Two new sections in
`subint_forkserver_test_cancellation_leak_issue.md`
documenting continued investigation of the
`test_nested_multierrors[subint_forkserver]` peer-
channel-loop hang:

1. **"Attempted fix (DID NOT work) — hypothesis
   (3)"**: tried sync-closing peer channels' raw
   socket fds from `_serve_ipc_eps`'s finally block
   (iterate `server._peers`, `_chan._transport.
   stream.socket.close()`). Theory was that sync
   close would propagate as `EBADF` /
   `ClosedResourceError` into the stuck
   `recv_some()` and unblock it. Result: identical
   hang. Either trio holds an internal fd
   reference that survives external close, or the
   stuck recv isn't even the root blocker. Either
   way: ruled out, experiment reverted, skip-mark
   restored.
2. **"Aside: `-s` flag changes behavior for peer-
   intensive tests"**: noticed
   `test_context_stream_semantics.py` under
   `subint_forkserver` hangs with default
   `--capture=fd` but passes with `-s`
   (`--capture=no`). Working hypothesis: subactors
   inherit pytest's capture pipe (fds 1,2 — which
   `_close_inherited_fds` deliberately preserves);
   verbose subactor logging fills the buffer,
   writes block, deadlock. Fix direction (if
   confirmed): redirect subactor stdout/stderr to
   `/dev/null` or a file in `_actor_child_main`.
   Not a blocker on the main investigation;
   deserves its own mini-tracker.

Both sections are diagnosis-only — no code changes
in this commit.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 7cd47ef7fb)
2026-06-29 19:20:21 -04:00
Gud Boi 43ee4a47ff Use `pidfd` for cancellable `_ForkedProc.wait`
Two coordinated improvements to the `subint_forkserver` backend:

1. Replace `trio.to_thread.run_sync(os.waitpid, ...,
   abandon_on_cancel=False)` in `_ForkedProc.wait()`
   with `trio.lowlevel.wait_readable(pidfd)`. The
   prior version blocked a trio cache thread on a
   sync syscall — outer cancel scopes couldn't
   unwedge it when something downstream got stuck.
   Same pattern `trio.Process.wait()` and
   `proc_waiter` (the mp backend) already use.

2. Drop the `@pytest.mark.xfail(strict=True)` from
   `test_orphaned_subactor_sigint_cleanup_DRAFT` —
   the test now PASSES after 0cd0b633 (fork-child
   FD scrub). Same root cause as the nested-cancel
   hang: inherited IPC/trio FDs were poisoning the
   child's event loop. Closing them lets SIGINT
   propagation work as designed.

Deats,
- `_ForkedProc.__init__` opens a pidfd via
  `os.pidfd_open(pid)` (Linux 5.3+, Python 3.9+)
- `wait()` parks on `trio.lowlevel.wait_readable()`,
  then non-blocking `waitpid(WNOHANG)` to collect
  the exit status (correct since the pidfd signal
  IS the child-exit notification)
- `ChildProcessError` swallow handles the rare race
  where someone else reaps first
- pidfd closed after `wait()` completes (one-shot
  semantics) + `__del__` belt-and-braces for
  unexpected-teardown paths
- test docstring's `@xfail` block replaced with a
  `# NOTE` comment explaining the historical
  context + cross-ref to the conc-anal doc; test
  remains in place as a regression guard

The two changes are interdependent — the
cancellable `wait()` matters for the same nested-
cancel scenarios the FD scrub fixes, since the
original deadlock had trio cache workers wedged in
`os.waitpid` swallowing the outer cancel.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit c20b05e181)
2026-06-29 19:20:21 -04:00
Gud Boi f7438b2968 Scrub inherited FDs in fork-child prelude
Implements fix-direction (1)/blunt-close-all-FDs from
b71705bd (`subint_forkserver` nested-cancel hang
diag), targeting the multi-level cancel-cascade
deadlock in
`test_nested_multierrors[subint_forkserver]`.

The diagnosis doc voted for surgical FD cleanup via
`actor.ipc_server` handle as the cleanest approach,
but going blunt is actually the right call: after
`os.fork()`, the child immediately enters
`_actor_child_main()` which opens its OWN IPC
sockets / wakeup-fd / epoll-fd / etc. — none of the
parent's FDs are needed. Closing everything except
stdio is safe AND defends against future
listener/IPC additions to the parent inheriting
silently into children.

Deats,
- new `_close_inherited_fds(keep={0,1,2}) -> int`
  helper. Linux fast-path enumerates `/proc/self/fd`;
  POSIX fallback uses `RLIMIT_NOFILE` range. Matches
  the stdlib `subprocess._posixsubprocess.close_fds`
  strategy. Returns close-count for sanity logging
- wire into `fork_from_worker_thread._worker()`'s
  post-fork child prelude — runs immediately after
  the pid-pipe `os.close(rfd/wfd)`, before the user
  `child_target` callable executes
- docstring cross-refs the diagnosis doc + spells
  out the FD-inheritance-cascade mechanism and why
  the close-all approach is safe for our spawn shape

Validation pending: re-run `test_nested_multierrors[subint_forkserver]`
to confirm the deadlock is gone.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 9993db0193)
2026-06-29 19:20:21 -04:00
Gud Boi 80aa465cf6 Refine `subint_forkserver` nested-cancel hang diagnosis
Major rewrite of
`subint_forkserver_test_cancellation_leak_issue.md`
after empirical investigation revealed the earlier
"descendant-leak + missing tree-kill" diagnosis
conflated two unrelated symptoms:

1. **5-zombie leak holding `:1616`** — turned out to
   be a self-inflicted cleanup bug: `pkill`-ing a bg
   pytest task (SIGTERM/SIGKILL, no SIGINT) skipped
   the SC graceful cancel cascade entirely. Codified
   the real fix — SIGINT-first ladder w/ bounded
   wait before SIGKILL — in e5e2afb5 (`run-tests`
   SKILL) and
   `feedback_sc_graceful_cancel_first.md`.
2. **`test_nested_multierrors[subint_forkserver]`
   hangs indefinitely** — the actual backend bug,
   and it's a deadlock not a leak.

Deats,
- new diagnosis: all 5 procs are kernel-`S` in
  `do_epoll_wait`; pytest-main's trio-cache workers
  are in `os.waitpid` waiting for children that are
  themselves waiting on IPC that never arrives —
  graceful `Portal.cancel_actor` cascade never
  reaches its targets
- tree-structure evidence: asymmetric depth across
  two identical `run_in_actor` calls — child 1
  (3 threads) spawns both its grandchildren; child 2
  (1 thread) never completes its first nursery
  `run_in_actor`. Smells like a race on fork-
  inherited state landing differently per spawn
  ordering
- new hypothesis: `os.fork()` from a subactor
  inherits the ROOT parent's IPC listener FDs
  transitively. Grandchildren end up with three
  overlapping FD sets (own + direct-parent + root),
  so IPC routing becomes ambiguous. Predicts bug
  scales with fork depth — matches reality: single-
  level spawn works, multi-level hangs
- ruled out: `_ForkedProc.kill()` tree-kill (never
  reaches hard-kill path), `:1616` contention (fixed
  by `reg_addr` fixture wiring), GIL starvation
  (each subactor has its own OS process+GIL),
  child-side KBI absorption (`_trio_main` only
  catches KBI at `trio.run()` callsite, reached
  only on trio-loop exit)
- four fix directions ranked: (1) blanket post-fork
  `closerange()`, (2) `FD_CLOEXEC` + audit,
  (3) targeted FD cleanup via `actor.ipc_server`
  handle, (4) `os.posix_spawn` w/ `file_actions`.
  Vote: (3) — surgical, doesn't break the "no exec"
  design of `subint_forkserver`
- standalone repro added (`spawn_and_error(breadth=
  2, depth=1)` under `trio.fail_after(20)`)
- stopgap: skip `test_nested_multierrors` + multi-
  level-spawn tests under the backend via
  `@pytest.mark.skipon_spawn_backend(...)` until
  fix lands

Killing the "tree-kill descendants" fix-direction
section: it addressed a bug that didn't exist.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 35da808905)
2026-06-29 19:20:21 -04:00
Gud Boi 212d7161d0 Add `subint_forkserver` test-cancellation leak doc
New `ai/conc-anal/
subint_forkserver_test_cancellation_leak_issue.md`
captures a descendant-leak surfaced while wiring
`subint_forkserver` into the full test matrix:
running `tests/test_cancellation.py` under
`--spawn-backend=subint_forkserver` reproducibly
leaks **exactly 5** `subint-forkserv` comm-named
child processes that survive session exit, each
holding a `LISTEN` on `:1616` (the tractor default
registry addr) — and therefore poisons every
subsequent test session that defaults to that addr.

Deats,
- TL;DR + ruled-out checks confirming the procs are
  ours (not piker / other tractor-embedding apps) —
  `/proc/$pid/cmdline` + cwd both resolve to this
  repo's `py314/` venv
- root cause: `_ForkedProc.kill()` is PID-scoped
  (plain `os.kill(SIGKILL)` to the direct child),
  not tree-scoped — grandchildren spawned during a
  multi-level cancel test get reparented to init and
  inherit the registry listen socket
- proposed fix directions ranked: (1) put each
  forkserver-spawned subactor in its own process-
  group (`os.setpgrp()` in fork-child) + tree-kill
  via `os.killpg(pgid, SIGKILL)` on teardown,
  (2) `PR_SET_CHILD_SUBREAPER` on root, (3) explicit
  `/proc/<pid>/task/*/children` walk. Vote: (1) —
  POSIX-standard, aligns w/ `start_new_session=True`
  semantics in `subprocess.Popen` / trio's
  `open_process`
- inline reproducer + cleanup recipe scoped to
  `$(pwd)/py314/bin/python.*pytest.*spawn-backend=
  subint_forkserver` so cleanup doesn't false-flag
  unrelated tractor procs (consistent w/
  `run-tests` skill's zombie-check guidance)

Stopgap hygiene fix (wiring `reg_addr` through the 5
leaky tests in `test_cancellation.py`) is incoming as
a follow-up — that one stops the blast radius, but
zombies still accumulate per-run until the real
tree-kill fix lands.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit e3f4f5a387)
2026-06-29 19:20:21 -04:00
Gud Boi 7c9e2260a2 Mv `test_subint_cancellation.py` to `tests/spawn/` subpkg
Also, some slight touchups in `.spawn._subint`.

(cherry picked from commit 1e357dcf08)
2026-06-29 19:20:21 -04:00
Gud Boi 2a5157aa14 Label forkserver child as `subint_forkserver`
Follow-up to 72d1b901 (was prev commit adding `debug_mode` for
`subint_forkserver`): that commit wired the runtime-side
`subint_forkserver` SpawnSpec-recv gate in `Actor._from_parent`, but the
`subint_forkserver_proc` child-target was still passing
`spawn_method='trio'` to `_trio_main` — so `Actor.pformat()` / log lines
would report the subactor as plain `'trio'` instead of the actual
parent-side spawn mechanism. Flip the label to `'subint_forkserver'`.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit e31eb8d7c9)
2026-06-29 19:20:21 -04:00
Gud Boi ded6278ea3 Drop unneeded f-str prefixes
(cherry picked from commit 5e85f184e0)
2026-06-29 19:20:21 -04:00
Gud Boi 4e8fed1262 Shorten some timeouts in `subint_forkserver` suites
(cherry picked from commit f5f37b69e6)
2026-06-29 19:20:21 -04:00
Gud Boi 4ef9222d12 Refine `subint_forkserver` orphan-SIGINT diagnosis
Empirical follow-up to the xfail'd orphan-SIGINT test:
the hang is **not** "trio can't install a handler on a
non-main thread" (the original hypothesis from the
`child_sigint` scaffold commit). On py3.14:

- `threading.current_thread() is threading.main_thread()`
  IS True post-fork — CPython re-designates the
  fork-inheriting thread as "main" correctly
- trio's `KIManager` SIGINT handler IS installed in the
  subactor (`signal.getsignal(SIGINT)` confirms)
- the kernel DOES deliver SIGINT to the thread

But `faulthandler` dumps show the subactor wedged in
`trio/_core/_io_epoll.py::get_events` — trio's
wakeup-fd mechanism (which turns SIGINT into an epoll-wake)
isn't firing. So the `except KeyboardInterrupt` at
`tractor/spawn/_entry.py::_trio_main:164` — the runtime's
intentional "KBI-as-OS-cancel" path — never fires.

Deats,
- new `ai/conc-anal/subint_forkserver_orphan_sigint_hang_issue.md`
  (+385 LOC): full writeup — TL;DR, symptom reproducer,
  the "intentional cancel path" the bug defeats,
  diagnostic evidence (`faulthandler` output +
  `getsignal` probe), ruled-out hypotheses
  (non-main-thread issue, wakeup-fd inheritance,
  KBI-as-trio-check-exception), and fix directions
- `test_orphaned_subactor_sigint_cleanup_DRAFT` xfail
  `reason` + test docstring rewritten to match the
  refined understanding — old wording blamed the
  non-main-thread path, new wording points at the
  `epoll_wait` wedge + cross-refs the new conc-anal doc
- `_subint_forkserver` module docstring's
  `child_sigint='trio'` bullet updated: now notes trio's
  handler is already correctly installed, so the flag may
  end up a no-op / doc-only mode once the real root cause
  is fixed

Closing the gap aligns with existing design intent (make
the already-designed "KBI-as-OS-cancel" behavior actually
fire), not a new feature.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit a72deef709)
2026-06-29 19:20:21 -04:00
Gud Boi e7740d131f Scaffold `child_sigint` modes for forkserver
Add configuration surface for future child-side SIGINT
plumbing in `subint_forkserver_proc` without wiring up the
actual trio-native SIGINT bridge — lifting one entry-guard
clause will flip the `'trio'` branch live once the
underlying fork-prelude plumbing is implemented.

Deats,
- new `ChildSigintMode = Literal['ipc', 'trio']` type +
  `_DEFAULT_CHILD_SIGINT = 'ipc'` module-level default.
  Docstring block enumerates both:
  - `'ipc'` (default, currently the only implemented mode):
    no child-side SIGINT handler — `trio.run()` is on the
    fork-inherited non-main thread where
    `signal.set_wakeup_fd()` is main-thread-only, so
    cancellation flows exclusively via the parent's
    `Portal.cancel_actor()` IPC path. Known gap: orphan
    children don't respond to SIGINT
    (`test_orphaned_subactor_sigint_cleanup_DRAFT`)
  - `'trio'` (scaffolded only): manual SIGINT → trio-cancel
    bridge in the fork-child prelude so external Ctrl-C
    reaches stuck grandchildren even w/ a dead parent
- `subint_forkserver_proc` pulls `child_sigint` out of
  `proc_kwargs` (matches how `trio_proc` threads config to
  `open_process`, keeps `start_actor(proc_kwargs=...)` as
  the ergonomic entry point); validates membership + raises
  `NotImplementedError` for `'trio'` at the backend-entry
  guard
- `_child_target` grows a `match child_sigint:` arm that
  slots in the future `'trio'` impl without restructuring
  — today only the `'ipc'` case is reachable
- module docstring "Still-open work" list grows a bullet
  pointing at this config + the xfail'd orphan-SIGINT test

No behavioral change on the default path — `'ipc'` is the
existing flow. Scaffolding only.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit dcd5c1ff40)
2026-06-29 19:20:21 -04:00
Gud Boi 0595ae7cb8 Add DRAFT `subint_forkserver` orphan-SIGINT test
Tier-4 test `test_orphaned_subactor_sigint_cleanup_DRAFT`
documents an empirical SIGINT-delivery gap in the
`subint_forkserver` backend: when the parent dies via
`SIGKILL` (no IPC `Portal.cancel_actor()` possible) and
`SIGINT` is sent to the orphan child, the child DOES NOT
unwind — CPython's default `KeyboardInterrupt` is delivered
to `threading.main_thread()`, whose tstate is dead in the
post-fork child bc fork inherited the worker thread, not
main. Trio running on the fork-inherited worker thread
therefore never observes the signal. Marked
`xfail(strict=True)` so the mark flips to XPASS→fail once
the backend grows explicit SIGINT plumbing.

Deats,
- harness runs the failure-mode sequence out-of-process:
  1. harness subprocess runs a fresh Python script
     that calls `try_set_start_method('subint_forkserver')`
     then opens a root actor + one `sleep_forever` subactor
  2. parse `PARENT_READY=<pid>` + `CHILD_PID=<pid>` markers
     off harness `stdout` to confirm IPC handshake
     completed
  3. `SIGKILL` the parent, `proc.wait()` to reap the
     zombie (otherwise `os.kill(pid, 0)` keeps reporting
     it alive)
  4. assert the child survived the parent-reap (i.e. was
     actually orphaned, not reaped too) before moving on
  5. `SIGINT` the orphan child, poll `os.kill(child_pid, 0)`
     every 100ms for up to 10s
- supporting helpers: `_read_marker()` with per-proc
  bytes-buffer to carry partial lines across calls,
  `_process_alive()` liveness probe via `kill(pid, 0)`
- Linux-only via `platform.system() != 'Linux'` skip —
  orphan-reparenting semantics don't generalize to
  other platforms
- port offset (`reg_addr[1] + 17`) so the harness listener
  doesn't race concurrently-running backend tests
- best-effort `finally:` cleanup: `SIGKILL` any still-alive
  pids + `proc.kill()` + bounded `proc.wait()` to avoid
  leaking orphans across the session

Also, tier-4 header comment documents the cross-backend
generalization path: applicable to any multi-process
backend (`trio`, `mp_spawn`, `mp_forkserver`,
`subint_forkserver`), NOT to plain `subint` (in-process
subints have no orphan OS-child). Move path: lift
harness into `tests/_orphan_harness.py`, parametrize on
session `_spawn_method`, add
`skipif _spawn_method == 'subint'`.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 76605d5609)
2026-06-29 19:20:21 -04:00
Gud Boi 95e4a27c4c Reset post-fork `_state` in forkserver child
`os.fork()` inherits the parent's entire memory image,
including `tractor.runtime._state` globals that encode
"this process is the root actor" — `_runtime_vars`'s
`_is_root=True`, pre-populated `_root_mailbox` +
`_registry_addrs`, and the parent's `_current_actor`
singleton.

A fresh `exec`-based child starts with those globals at
their module-level defaults (all falsey/empty). The
forkserver child needs to match that shape BEFORE calling
`_actor_child_main()`, otherwise `Actor.__init__()` takes
the `is_root_process() == True` branch and pre-populates
`self.enable_modules`, which then trips
`assert not self.enable_modules` at the top of
`Actor._from_parent()` on the subsequent parent→child
`SpawnSpec` handshake.

Fix: at the start of `_child_target`, null
`_state._current_actor` and overwrite `_runtime_vars` with
a cold-root blank (`_is_root=False`, empty mailbox/addrs,
`_debug_mode=False`) before `_actor_child_main()` runs.

Found-via: `test_subint_forkserver_spawn_basic` hitting
the `enable_modules` assert on child-side runtime boot.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 63ab7c986b)
2026-06-29 19:20:21 -04:00
Gud Boi 7f1fa2c3a5 Wire `subint_forkserver` as first-class backend
Promote `_subint_forkserver` from primitives-only into a
registered spawn backend: `'subint_forkserver'` is now a
`SpawnMethodKey` literal, dispatched via `_methods` to
the new `subint_forkserver_proc()` target, feature-gated
under the existing `subint`-family py3.14+ case, and
selectable via `--spawn-backend=subint_forkserver`.

Deats,
- new `subint_forkserver_proc()` spawn target in
  `_subint_forkserver`:
  - mirrors `trio_proc()`'s supervision model — real OS
    subprocess so `Portal.cancel_actor()` + `soft_kill()`
    on graceful teardown, `os.kill(SIGKILL)` on hard-reap
    (no `_interpreters.destroy()` race to fuss over bc the
    child lives in its own process)
  - only real diff from `trio_proc` is the spawn mechanism:
    fork from a main-interp worker thread via
    `fork_from_worker_thread()` (off-loaded to trio's
    thread pool) instead of `trio.lowlevel.open_process()`
  - child-side `_child_target` closure runs
    `tractor._child._actor_child_main()` with
    `spawn_method='trio'` — the child is a regular trio
    actor, "subint_forkserver" names how the parent
    spawned, not what the child runs
- new `_ForkedProc` class — thin `trio.Process`-compatible
  shim around a raw OS pid: `.poll()` via
  `waitpid(WNOHANG)`, async `.wait()` off-loaded to a trio
  cache thread, `.kill()` via `SIGKILL`, `.returncode`
  cached for repeat calls. `.stdin`/`.stdout`/`.stderr`
  are `None` (fork-w/o-exec inherits parent FDs; we don't
  marshal them) which matches `soft_kill()`'s `is not None`
  guards

Also, new backend-tier test
`test_subint_forkserver_spawn_basic` drives the registered
backend end-to-end via `open_root_actor` + `open_nursery` +
`run_in_actor` w/ a trivial portal-RPC round-trip. Uses a
`forkserver_spawn_method` fixture to flip
`_spawn_method`/`_ctx` for the test's duration + restore on
teardown (so other session-level tests don't observe the
global flip). Test module docstring reworked to describe
the three tiers now covered: (1) primitive-level, (2)
parent-trio-driven primitives, (3) full registered backend.

Status: still-open work (tracked on `tractor#379`) doc'd
inline in the module docstring — no cancel/hard-kill stress
coverage yet, child-side subint-hosted root runtime still
future (gated on `msgspec#563`), thread-hygiene audit
pending the same unblock.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 26914fde75)
2026-06-29 19:20:21 -04:00
Gud Boi ba6fda84ff Add `subint_forkserver` PEP 684 audit-plan doc
Follow-up tracker companion to the module-docstring TODO
added in `372a0f32`. Catalogs why `_subint_forkserver`'s
two "non-trio thread" constraints
(`fork_from_worker_thread()` +
`run_subint_in_worker_thread()` both allocating dedicated
`threading.Thread`s; test helper named
`run_fork_in_non_trio_thread`) exist today, and which of
them would dissolve once msgspec PEP 684 support ships
(`msgspec#563`) and tractor flips to isolated-mode subints.

Deats,
- three reasons enumerated for the current constraints:
  - class-A GIL-starvation — **fixed** by isolated mode:
    subints don't share main's GIL so abandoned-thread
    contention disappears
  - destroy race / tstate-recycling from `subint_proc` —
    **unclear**: `_PyXI_Enter` + `_PyXI_Exit` are
    cross-mode, so isolated doesn't obviously fix it;
    needs empirical retest on py3.14 + isolated API
  - fork-from-main-interp-tstate (the CPython-level
    `_PyInterpreterState_DeleteExceptMain` gate) — the
    narrow reason for using a dedicated thread; **probably
    fixed** IF the destroy-race also resolves (bc trio's
    cache threads never drove subints → clean main-interp
    tstate)
- TL;DR table of which constraints unwind under each
  resolution branch
- four-step audit plan for when `msgspec#563` lands:
  - flip `_subint` to isolated mode
  - empirical destroy-race retest
  - audit `_subint_forkserver.py` — drop `non_trio`
    qualifier / maybe inline primitives
  - doc fallout — close the three `subint_*_issue.md`
    siblings w/ post-mortem notes

Also, cross-refs the three sibling `conc-anal/` docs, PEPs
684 + 734, `msgspec#563`, and `tractor#379` (the overall
subint spawn-backend tracking issue).

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit cf2e71d87f)
2026-06-29 19:20:21 -04:00
Gud Boi c17579ecea Add trio-parent tests for `_subint_forkserver`
New pytest module `tests/spawn/test_subint_forkserver.py`
drives the forkserver primitives from inside a real
`trio.run()` in the parent — the runtime shape tractor will
actually use when we wire up a `subint_forkserver` spawn
backend proper. Complements the standalone no-trio-in-parent
`ai/conc-anal/subint_fork_from_main_thread_smoketest.py`.

Deats,
- new test pkg `tests/spawn/` (+ empty `__init__.py`)
- two tests, both `@pytest.mark.timeout(30, method='thread')`
  for the GIL-hostage safety reason doc'd in
  `ai/conc-anal/subint_sigint_starvation_issue.md`:
  - `test_fork_from_worker_thread_via_trio` — parent-side
    plumbing baseline. `trio.run()` off-loads forkserver
    prims via `trio.to_thread.run_sync()` + asserts the
    child reaps cleanly
  - `test_fork_and_run_trio_in_child` — end-to-end: forked
    child calls `run_subint_in_worker_thread()` with a
    bootstrap str that does `trio.run()` in a fresh subint
- both tests wrap the inner `trio.run()` in a
  `dump_on_hang()` for post-mortem if the outer
  `pytest-timeout` fires
- intentionally NOT using `--spawn-backend` — the tests
  drive the primitives directly rather than going through
  tractor's spawn-method registry (which the forkserver
  isn't plugged into yet)

Also, rename `run_trio_in_subint()` →
`run_subint_in_worker_thread()` for naming consistency with
the sibling `fork_from_worker_thread()`. The action is really
"host a subint on a worker thread", not specifically "run
trio" — trio just happens to be the typical payload.
Propagate the rename to the smoketest.

Further, add a "TODO — cleanup gated on msgspec PEP 684
support" section to the `_subint_forkserver` module
docstring: flags the dedicated-`threading.Thread` design as
potentially-revisable once isolated-mode subints are viable
in tractor. Cross-refs `msgspec#563` + `tractor#379` and
points at an audit-plan conc-anal doc we'll add next.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 25e400d526)
2026-06-29 19:20:21 -04:00
Gud Boi 8bf8bea860 Lift fork prims into `_subint_forkserver` mod
The smoketest (prior commit) empirically validated the
"fork-from-main-interp-worker-thread" arch on py3.14. Promote
the validated primitives out of the `ai/conc-anal/` smoketest
into `tractor.spawn._subint_forkserver` so they can eventually
be wired into a real "subint forkserver" spawn backend.

Deats,
- new module `tractor/spawn/_subint_forkserver.py` (337 LOC):
  - `fork_from_worker_thread(child_target, thread_name)` —
    spawn a main-interp `threading.Thread`, call `os.fork()`
    from it, shuttle the child pid back to main via a pipe
  - `run_trio_in_subint(bootstrap, ...)` — post-fork helper:
    create a fresh subint + drive `_interpreters.exec()` on
    a dedicated worker thread running the `bootstrap` str
    (typically imports `trio`, defines an async entry, calls
    `trio.run()`)
  - `wait_child(pid, expect_exit_ok)` — `os.waitpid()` +
    pass/fail classification reusable from harness AND the
    eventual real spawn path
- feature-gated py3.14+ via the public
  `concurrent.interpreters` presence check; matches the gate
  in `tractor.spawn._subint`
- module docstring doc's the CPython-block context
  (cross-refs `_subint_fork` stub + the two `conc-anal/`
  docs) and status: EXPERIMENTAL, not yet registered in
  `_spawn._methods`

Also, refactor the smoketest
`ai/conc-anal/subint_fork_from_main_thread_smoketest.py` to
import the primitives from the new module rather than inline
its own copies. Keeps the smoketest and the tractor-side
impl in sync as the forkserver design evolves; the smoketest
remains a zero-`tractor`-runtime CPython-level check
(imports ONLY the three primitives, no runtime bring-up).

Status: next step is to drive these from a parent-side
`trio.run()` and hook the returned child pid into the normal
actor-nursery/IPC flow — then register `subint_forkserver`
as a `SpawnMethodKey` in `_spawn.py`.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 82332fbceb)
2026-06-29 19:20:21 -04:00
Gud Boi d9def89f0f Add CPython-level `subint_fork` workaround smoketest
Standalone script to validate the "main-interp worker-thread
forkserver + subint-hosted trio" arch proposed as a workaround
to the CPython-level refusal doc'd in
`ai/conc-anal/subint_fork_blocked_by_cpython_post_fork_issue.md`.

Deliberately NOT a `tractor` test — zero `tractor` imports.
Uses `_interpreters` (private stdlib) + `os.fork()` directly so
pass/fail is a property of CPython alone, independent of our
runtime. Requires py3.14+.

Deats,
- four scenarios via `--scenario`:
  - `control_subint_thread_fork` — the KNOWN-BROKEN case as a
    harness sanity; if the child DOESN'T abort, our analysis
    is wrong
  - `main_thread_fork` — baseline sanity, must always succeed
  - `worker_thread_fork` — architectural assertion: regular
    `threading.Thread` attached to main interp calls
    `os.fork()`; child should survive post-fork cleanup
  - `full_architecture` — end-to-end: fork from a main-interp
    worker thread, then in child create a subint driving a
    worker thread running `trio.run()`
- exit code 0 on EXPECTED outcome (for `control_*` that means
  "child aborted", not "child succeeded")
- each scenario prints a self-contained pass/fail banner; use
  `os.waitpid()` of the parent + per-scenario status prints to
  observe the child's fate

Also, log NLNet provenance for this session's three-sub-phase
work (py3.13 gate tightening, `pytest-timeout` + marker
refactor, `subint_fork` prototype → CPython-block finding).

Prompt-IO: ai/prompt-io/claude/20260422T200723Z_797f57c_prompt_io.md

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit de4f470b6c)
2026-06-29 19:20:21 -04:00
Gud Boi 38e7e4ced6 Doc `subint_fork` as blocked by CPython post-fork
Empirical finding: the WIP `subint_fork_proc` scaffold
landed in `cf0e3e6f` does *not* work on current CPython.
The `fork()` syscall succeeds in the parent, but the
CHILD aborts immediately during
`PyOS_AfterFork_Child()` →
`_PyInterpreterState_DeleteExceptMain()`, which gates
on the current tstate belonging to the main interp —
the child dies with `Fatal Python error: not main
interpreter`.

CPython devs acknowledge the fragility with an in-source
comment (`// Ideally we could guarantee tstate is running
main.`) but expose no user-facing hook to satisfy the
precondition — so the strategy is structurally dead until
upstream changes.

Rather than delete the scaffold, reshape it into a
documented dead-end so the next person with this idea
lands on the reason rather than rediscovering the same
CPython-level refusal.

Deats,
- Move `subint_fork_proc` out of `tractor.spawn._subint`
  into a new `tractor.spawn._subint_fork` dedicated
  module (153 LOC). Module + fn docstrings now describe
  the blockage directly; the fn body is trimmed to a
  `NotImplementedError` pointing at the analysis doc —
  no more dead-code `bootstrap` sketch bloating
  `_subint.py`.
- `_spawn.py`: keep `'subint_fork'` in `SpawnMethodKey`
  + the `_methods` dispatch so
  `--spawn-backend=subint_fork` routes to a clean
  `NotImplementedError` rather than "invalid backend";
  comment calls out the blockage. Collapse the duplicate
  py3.14 feature-gate in `try_set_start_method()` into a
  combined `case 'subint' | 'subint_fork':` arm.
- New 337-line analysis:
  `ai/conc-anal/subint_fork_blocked_by_cpython_post_fork_issue.md`.
  Annotated walkthrough from the user-visible fatal
  error down to the specific `Modules/posixmodule.c` +
  `Python/pystate.c` source lines enforcing the refusal,
  plus an upstream-report draft.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 0f48ed2eb9)
2026-06-29 19:20:21 -04:00
Gud Boi d257b00c74 Add WIP `subint_fork_proc` backend scaffold
Experimental third spawn backend: use a fresh
sub-interpreter purely as a trio-free launchpad from
which to `os.fork()` + exec back into
`python -m tractor._child`. Per issue #379's
"fork()-workaround/hacks" thread.

Intent is to sidestep both,
- the trio+fork hazards hitting `trio_proc` (python- trio/trio#1614 et
  al.), since the forking interp is guaranteed trio-free.

- the shared-GIL abandoned-thread hazards hitting `subint_proc`
  (`ai/conc-anal/subint_sigint_starvation_issue.md`), since we don't
  *stay* in the subint — it only lives long enough to call `os.fork()`

Downstream of the fork+exec, all the existing `trio_proc` plumbing is
reused verbatim: `ipc_server.wait_for_peer()`, `SpawnSpec`, `Portal`
yield, soft-kill.

Status: NOT wired up beyond scaffolding. The fn raises
`NotImplementedError` immediately; the `bootstrap` fork/exec string
builder and the `# TODO: orchestrate driver thread` block are kept
in-tree as deliberate dead code so the next iteration starts from
a concrete shape rather than a blank page.

Docstring calls out three open questions that need
empirical validation before wiring this up:
1. Does CPython permit `os.fork()` from a non-main
   legacy subint?
2. Can the child stay fork-without-exec and
   `trio.run()` directly from within the launchpad
   subint?
3. How do `signal.set_wakeup_fd()` handlers and other
   process-global state interact when the forking
   thread is inside a subint?

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit eee79a0357)
2026-06-29 19:20:21 -04:00
Gud Boi 80b6d013c4 Expand `subint` sigint-starvation hang catalog
Add two more tests to the catalog in
`conc-anal/subint_sigint_starvation_issue.md` — same
signal-wakeup-fd-saturation fingerprint (abandoned legacy-subint driver
threads → shared-GIL starvation → `write() = EAGAIN` on the wakeup pipe
→ silent SIGINT drop), different load patterns.

Deats,
- `test_cancel_while_childs_child_in_sync_sleep[subint-False]`: nested
  actor-tree + sync-sleeping grandchild. Under `trio`/`mp_*` the "zombie
  reaper" is a subproc `SIGKILL`; no equivalent exists under subint, so
  the grandchild persists in its abandoned driver thread. Often only
  manifests under full-suite runs (earlier tests seed the
  abandoned-thread pool).

- `test_multierror_fast_nursery[subint-25-0.5]`: 25 concurrent subactors
  all go through teardown on the multierror. Bounded hard-kills run in
  parallel — so the total budget is ~3s, not 3s × 25. Leaves 25
  abandoned driver threads simultaneously alive, an extreme pressure
  multiplier. `strace` shows several successful `write(16, "\2", 1) = 1`
  (GIL round-robin IS giving main brief slices) before finally
  saturating with `EAGAIN`.

Also include a `pstree -snapt <pid>` capture showing
16+ live `{subint-driver[<interp_id>}` threads at the
moment of hang — the direct GIL-contender population.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit f3cea714bc)
2026-06-29 19:20:21 -04:00
Gud Boi 9c4f409e04 Wall-cap `subint` audit tests via `pytest-timeout`
Add a hard process-level wall-clock bound on the two
known-hanging subint-backend tests so an unattended
suite run can't wedge indefinitely in either of the
hang classes doc'd in `ai/conc-anal/`.

Deats,
- New `testing` dep: `pytest-timeout>=2.3`.
- `test_stale_entry_is_deleted`:
  `@pytest.mark.timeout(3, method='thread')`. The
  `method='thread'` choice is deliberate —
  `method='signal'` routes via `SIGALRM` which is
  starved by the same GIL-hostage path that drops
  `SIGINT` (see `subint_sigint_starvation_issue.md`),
  so it'd never actually fire in the starvation case.
- `test_subint_non_checkpointing_child`: same
  decorator, same reasoning (defense-in-depth over
  the inner `trio.fail_after(15)`).

At timeout, `pytest-timeout` hard-kills the pytest
process itself — that's the intended behavior here;
the alternative is the suite never returning.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 189f4e3ffc)
(MTF-only portion: kept tests/test_subint_cancellation.py)
2026-06-29 19:20:21 -04:00
Gud Boi aeff8b85b0 Add prompt-io log for `subint` hang-class docs
Log the `claude-opus-4-7` collab that produced `e92e3cd2` ("Doc `subint`
backend hang classes + arm `dump_on_hang`"). Substantive bc the two new
`ai/conc-anal/` docs were jointly authored — user framed the two-class
split + set candidate-fix ordering for the class-2 (Ctrl-C-able) hang;
claude drafted the prose and the test-side cross-linking comments.

`.raw.md` is in diff-ref mode — per-file pointers via `git diff
e92e3cd2~1..e92e3cd2 -- <path>` rather than re-embedding content that
already lives in `git log -p`.

Prompt-IO: ai/prompt-io/claude/20260420T192739Z_5e8cd8b2_prompt_io.md

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit a65fded4c6)
2026-06-29 19:20:21 -04:00
Gud Boi 47cc61378d Doc `subint` backend hang classes + arm `dump_on_hang`
Classify and write up the two distinct hang modes hit during Phase
B subint bringup (issue #379) so future triage doesn't re-derive them
from scratch.

Deats, two new `ai/conc-anal/` docs,
- `subint_sigint_starvation_issue.md`: abandoned legacy-subint thread
  + shared GIL → main trio loop starves → signal-wakeup-fd pipe fills
  → `SIGINT` silently dropped (`strace` shows `write() = EAGAIN` on the
  wakeup-fd). Un- Ctrl-C-able. Structurally a CPython limit; blocked on
  `msgspec` PEP 684 (jcrist/msgspec#563)

- `subint_cancel_delivery_hang_issue.md`: parent-side trio task parks on
  an orphaned IPC channel after subint teardown — no clean EOF delivered
  to the waiting receive. Ctrl-C-able (main loop iterates fine); OUR bug
  to fix. Candidate fix: explicit parent-side channel abort in
  `subint_proc`'s hard-kill teardown

Cross-link the docs from their test reproducers,
- `test_stale_entry_is_deleted` (→ starvation class): wrap
  `trio.run(main)` in `dump_on_hang(seconds=20)` so a future regression
  captures a stack dump. Kept un- skipped so the dump file is
  inspectable

- `test_subint_non_checkpointing_child` (→ delivery class): extend
  docstring with a "KNOWN ISSUE" block pointing at the analysis

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 4a3254583b)
(MTF-only portion: kept ai/conc-anal/subint_cancel_delivery_hang_issue.md ai/conc-anal/subint_sigint_starvation_issue.md tests/test_subint_cancellation.py)
2026-06-29 19:20:21 -04:00
Gud Boi d4d2531bbd Add `subint` cancellation + hard-kill test audit
Lock in the escape-hatch machinery added to `tractor.spawn._subint`
during the Phase B.2/B.3 bringup (issue #379) so future stdlib
regressions or our own refactors don't silently re-introduce the
mid-suite hangs.

Deats,
- `test_subint_happy_teardown`: baseline — spawn a subactor, one portal
  RPC, clean teardown. If this breaks, something's wrong unrelated to
  the hard-kill shields.
- `test_subint_non_checkpointing_child`: cancel a subactor stuck in
  a non-checkpointing Python loop (`threading.Event.wait()` releases the
  GIL but never inserts a trio checkpoint). Validates the bounded-shield
  + daemon-driver-thread combo abandons the thread after
    `_HARD_KILL_TIMEOUT`.

Every test is wrapped in `trio.fail_after()` for a deterministic
per-test wall-clock ceiling (an unbounded audit would defeat itself) and
arms `tractor.devx.dump_on_hang()` so a hang captures a stack dump
— pytest's stderr capture swallows `faulthandler` output by default.

Gated via `pytest.importorskip('concurrent.interpreters')` and
a module-level skip when `--spawn-backend` isn't `'subint'`.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 2ed5e6a6e8)
2026-06-29 19:20:21 -04:00
Gud Boi 1f9a1a967e Raise `subint` floor to py3.14 and split dep-groups
The private `_interpreters` C module ships since 3.13, but that vintage
wedges under our `threading.Thread` + multi-trio usage pattern
—> `_interpreters.exec()` silently never makes progress. 3.14 fixes it.
So gate on the presence of the public `concurrent.interpreters` wrapper
(3.14+ only) even tho we still call into the private module at runtime.

Deats,
- `try_set_start_method('subint')` error msg + `_subint` module
  docstring/comments rewritten to document the 3.14 floor and why 3.13
  can't work.
- `_subint._has_subints` gate now imports `concurrent.interpreters` (not
  `_interpreters`) as the version sentinel.

Also, reshuffle `pyproject.toml` deps into
per-python-version `[tool.uv.dependency-groups]`:
- `subints` group: `msgspec>=0.21.0`, py>=3.14
- `eventfd` group: `cffi>=1.17.1`, py>=3.13,<3.14
- `sync_pause` group: `greenback`, py>=3.13,<3.14
  (was in `devx`; moved out bc no 3.14 yet)

Bump top-level `msgspec>=0.20.0` too.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 34d9d482e4)
(MTF-only portion: kept tractor/spawn/_spawn.py tractor/spawn/_subint.py)
2026-06-29 19:20:21 -04:00
Gud Boi e3d8a83101 Bound subint teardown shields with hard-kill timeout
Unbounded `trio.CancelScope(shield=True)` at the
soft-kill and thread-join sites can wedge the parent
trio loop indefinitely when a stuck subint ignores
portal-cancel (e.g. bc the IPC channel is already
broken).

Deats,
- add `_HARD_KILL_TIMEOUT` (3s) module-level const
- wrap both shield sites with
  `trio.move_on_after()` so we abandon a stuck
  subint after the deadline
- flip driver thread to `daemon=True` so proc-exit
  also isn't blocked by a wedged subint
- pass `abandon_on_cancel=True` to
  `trio.to_thread.run_sync(driver_thread.join)`
  — load-bearing for `move_on_after` to actually
  fire
- log warnings when either timeout triggers
- improve `InterpreterError` log msg to explain
  the abandoned-thread scenario

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 99541feec7)
2026-06-29 19:20:21 -04:00
Gud Boi 1d70b308bc Add prompt-IO log for subint destroy-race fix
Log the `claude-opus-4-7` session that produced
the `_subint.py` dedicated-thread fix (`26fb8206`).
Substantive bc the patch was entirely AI-generated;
raw log also preserves the CPython-internals
research informing Phase B.3 hard-kill work.

Prompt-IO: ai/prompt-io/claude/20260418T042526Z_26fb820_prompt_io.md

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit c041518bdb)
2026-06-29 19:20:21 -04:00
Gud Boi 01a986467c Fix subint destroy race via dedicated OS thread
`trio.to_thread.run_sync(_interpreters.exec, ...)` runs `exec()` on
a cached worker thread — and when that thread is returned to the
cache after the subint's `trio.run()` exits, CPython still keeps
the subint's tstate attached to the (now idle) worker. Result: the
teardown `_interpreters.destroy(interp_id)` in the `finally` block
can block the parent's trio loop indefinitely, waiting for a tstate
release that only happens when the worker either picks up a new job
or exits.

Manifested as intermittent mid-suite hangs under
`--spawn-backend=subint` — caught by a
`faulthandler.dump_traceback_later()` showing the main thread stuck
in `_interpreters.destroy()` at `_subint.py:293` with only an idle
trio-cache worker as the other live thread.

Deats,
- drive the subint on a plain `threading.Thread` (not
  `trio.to_thread`) so the OS thread truly exits after
  `_interpreters.exec()` returns, releasing tstate and unblocking
  destroy
- signal `subint_exited.set()` back to the parent trio loop from
  the driver thread via `trio.from_thread.run_sync(...,
  trio_token=...)` — capture the token at `subint_proc` entry
- swallow `trio.RunFinishedError` in that signal path for the case
  where parent trio has already exited (proc teardown)
- in the teardown `finally`, off-load the sync
  `driver_thread.join()` to `trio.to_thread.run_sync` (cache thread
  w/ no subint tstate → safe) so we actually wait for the driver to
  exit before `_interpreters.destroy()`

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 31cbd11a5b)
2026-06-29 19:20:21 -04:00
Gud Boi 1a35dd2cba Doc the `_interpreters` private-API choice in `_subint`
Expand the comment block above the `_interpreters`
import explaining *why* we use the private C mod
over `concurrent.interpreters`: the public API only
exposes PEP 734's `'isolated'` config which breaks
`msgspec` (missing PEP 684 slot). Add reference
links to PEP 734, PEP 684, cpython sources, and
the msgspec upstream tracker (jcrist/msgspec#563).

Also,
- update error msgs in both `_spawn.py` and
  `_subint.py` to say "3.13+" (matching the actual
  `_interpreters` availability) instead of "3.14+".
- tweak the mod docstring to reflect py3.13+
  availability via the private C module.

Review: PR #444 (copilot-pull-request-reviewer)
https://github.com/goodboy/tractor/pull/444

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 8a8d01e076)
2026-06-29 19:20:21 -04:00
Gud Boi 84b697371d Impl min-viable `subint` spawn backend (B.2)
Replace the B.1 scaffold stub w/ a working spawn
flow driving PEP 734 sub-interpreters on dedicated
OS threads.

Deats,
- use private `_interpreters` C mod (not the public
  `concurrent.interpreters` API) to get `'legacy'`
  subint config — avoids PEP 684 C-ext compat
  issues w/ `msgspec` and other deps missing the
  `Py_mod_multiple_interpreters` slot
- bootstrap subint via code-string calling new
  `_actor_child_main()` from `_child.py` (shared
  entry for both CLI and subint backends)
- drive subint lifetime on an OS thread using
  `trio.to_thread.run_sync(_interpreters.exec, ..)`
- full supervision lifecycle mirrors `trio_proc`:
  `ipc_server.wait_for_peer()` → send `SpawnSpec`
  → yield `Portal` via `task_status.started()`
- graceful shutdown awaits the subint's inner
  `trio.run()` completing; cancel path sends
  `portal.cancel_actor()` then waits for thread
  join before `_interpreters.destroy()`

Also,
- extract `_actor_child_main()` from `_child.py`
  `__main__` block as callable entry shape bc the
  subint needs it for code-string bootstrap
- add `"subint"` to the `_runtime.py` spawn-method
  check so child accepts `SpawnSpec` over IPC

Prompt-IO: ai/prompt-io/claude/20260417T124437Z_5cd6df5_prompt_io.md

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit b8f243e98d)
(MTF-only portion: kept ai/prompt-io/claude/20260417T124437Z_5cd6df5_prompt_io.md ai/prompt-io/claude/20260417T124437Z_5cd6df5_prompt_io.raw.md tractor/spawn/_subint.py)
2026-06-29 19:20:21 -04:00
Gud Boi cd113f27b0 Add `'subint'` spawn backend scaffold (#379)
Land the scaffolding for a future sub-interpreter (PEP 734
`concurrent.interpreters`) actor spawn backend per issue #379. The
spawn flow itself is not yet implemented; `subint_proc()` raises a
placeholder `NotImplementedError` pointing at the tracking issue —
this commit only wires up the registry, the py-version gate, and
the harness.

Deats,
- bump `pyproject.toml` `requires-python` to `>=3.12, <3.15` and
  list the `3.14` classifier — the new stdlib
  `concurrent.interpreters` module only ships on 3.14
- extend `SpawnMethodKey = Literal[..., 'subint']`
- `try_set_start_method('subint')` grows a new `match` arm that
  feature-detects the stdlib module and raises `RuntimeError` with
  a clear banner on py<3.14
- `_methods` registers the new `subint_proc()` via the same
  bottom-of-module late-import pattern used for `._trio` / `._mp`

Also,
- new `tractor/spawn/_subint.py` — top-level `try: from concurrent
  import interpreters` guards `_has_subints: bool`; `subint_proc()`
  signature mirrors `trio_proc`/`mp_proc` so the Phase B.2 impl can
  drop in without touching the registry
- re-add `import sys` to `_spawn.py` (needed for the py-version msg
  in the gate-error)
- `_testing.pytest.pytest_configure` wraps `try_set_start_method()`
  in a `pytest.UsageError` handler so `--spawn-backend=subint` on
  py<3.14 prints a clean banner instead of a traceback

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit d318f1f8f4)
(MTF-only portion: kept tractor/spawn/_spawn.py tractor/spawn/_subint.py)
2026-06-29 19:20:21 -04:00
Bd 65bf9df5ba
Merge pull request #468 from goodboy/test_cpu_throttling
Test cpu throttling
2026-06-29 19:18:34 -04:00
Gud Boi 51326f4b7a Fix uds `get_random` reaper-regex break (+2 nits)
Copilot's 2nd-pass review (PR #468) caught a regression I landed in
the uds fix: `UDSAddress.get_random()` put the per-call token AFTER
`@{pid}` (`no_runtime_*@{pid}.{token}.sock`), which breaks the
`tractor._testing._reap` matcher
`^(?P<name>.+)@(?P<pid>\d+)\.sock$` — so no-runtime orphan socks
stopped matching and never got reaped/attributed. Move the token
INTO the name (`{prefix}.{token}@{pid}.sock`) so the canonical
`@{pid}.sock` suffix stays intact for both that regex and the
`spawn._reap` reconstruction; also bump the token to 8 hex chars.

Also two robustness nits from the same review,
- `tests.conftest._measure_sustained_headroom()`: guard `frac <= 0`
  before `1./frac` — a 0/parked-core freq read would
  `ZeroDivisionError`, get swallowed by the broad `except` into a
  1.0 (no-throttle), defeating the probe on the exact broken box it
  should flag; read 0 as max throttle.
- `scripts/cpu-perf-check`: mark the burn procs `daemon=True` and
  wrap sampling in `try/finally` so a Ctrl-C / error reaps them
  instead of leaving stray CPU hogs.

Regressed-by: 09c50f49 (uds no-runtime token placed after `@pid`)
Found-via: Copilot review #4595803812 (`_testing._reap` regex)
Review: PR #468 (Copilot)
https://github.com/goodboy/tractor/pull/468#pullrequestreview-4595803812

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-29 18:51:26 -04:00
Gud Boi 63498a80c2 Use `cpu_perf_headroom()` at remaining deadlines
Finish TODO #2 from PR #468: the last raw `cpu_scaling_factor()`
call-sites still used the static-only check (blind to the
sustained-load power-cap). Point them at the `cpu_perf_headroom()`
SUPERSET — it calls `cpu_scaling_factor()` internally + the
session-cached throttle probe, so it's always >= the old factor
(never LESS headroom, so CI stays green).

Migrated,
- `test_spawning`: the `fail_after(1 * ...)` smoke deadline
  (#465-added).
- `test_inter_peer_cancellation`: `this_fast` budget.
- `test_docs_examples`: example-run `timeout`.
- `test_resource_cache`: fan-out `timeout`.

`test_cancellation` already used `cpu_perf_headroom()`; only its
explanatory comments still name the static helper.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-29 18:13:47 -04:00
Gud Boi 09c50f495d Uniquify no-runtime UDS `get_random()` paths
W/o a live runtime `get_random()` named UDS socks purely by
`(prefix, pid)`, so two calls in one proc returned the SAME
`no_runtime_*@{pid}.sock` — the 2nd `.bind()` then tripped
`EADDRINUSE`. Append a per-call `uuid4().hex[:6]` token so
each call yields a distinct sockpath.

This fixes the 3 `tests.discovery.test_tpt_bind_addrs` uds
failures (one registrar + disjoint-bind, two non-registrar
binds) where `reg_addr` and a "random" bind addr aliased —
the uds CI job's only red, surfaced when this branch rebased
onto the newer base that carries those tests.

Scoped to the no-runtime branch ON PURPOSE: the runtime
`{name}@{pid}` convention stays deterministic so
`spawn._reap.unlink_uds_bind_addrs()` can still reconstruct
+ unlink a SIGKILL'd subactor's sock (#454).

Deats,
- `_uds.py`: add `uuid4` import + token in the no-runtime
  branch
- `_testing.addr`: tighten `get_rando_addr()` docstring re the
  intra-proc per-call token (not just pid-keyed namespacing)

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-29 18:08:50 -04:00
Gud Boi 5b1a9edeb3 Fix `_burn` bigint blowup, ctx-mgr `_read_mhz`
Both `_burn()` impls (the `cpu-perf-check` script + the
`_measure_sustained_headroom()` probe in `tests.conftest`) grew `x`
~x**2 per iter, so the loop quickly went bigint alloc/mul-bound — a
noisy CPU load + needless memory across N procs. Mask each step to
64-bit for a steady, fixed-width ALU burn (still pegs every core,
which is all the freq probe needs).

Also, `_read_mhz()` opened the sysfs freq files without a ctx-mgr;
`with open(...)` so the FD closes deterministically (matches the
script's own `_read()`).

Review: PR #468 (Copilot)
https://github.com/goodboy/tractor/pull/468

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-29 18:08:50 -04:00
Gud Boi 707caf63d1 Use `cpu_perf_headroom()` for timing-test deadlines
`cpu_perf_headroom()` is a strict SUPERSET of `cpu_scaling_factor()`
— it folds in static cpu-freq scaling + the slow-CI bump AND the
sustained-load power-cap throttle probe. Point the timing-deadline
call sites at it so they're robust to sustained throttling too (the
gremlin behind mass `trio` deadline-miss flakes),

- `test_legacy_one_way_streaming`: `time_quad_ex` cancel-deadline +
  `test_a_quadruple_example`'s `this_fast` smoke bound.
- `test_cancellation`: `test_cancel_via_SIGINT_other_task` +
  `test_fast_graceful_cancel_*` deadlines.

`cpu_perf_headroom >= cpu_scaling_factor` always, so never less
headroom (CI stays green); `cpu_scaling_factor()` remains the
internal static component.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-29 18:08:50 -04:00
Gud Boi 0ad5261905 Add `cpu-perf-check` sustained-throttle gate script
Standalone CLI companion to `cpu_perf_headroom()` (20cb99ec): idle
freq snapshots LIE — every static knob (`governor`, EPP,
`platform_profile`, `scaling_max_freq`) can read "performance"
while a firmware/EC power cap (AMD PPT/STAPM + friends) clamps the
package to ~30% the moment a sustained multi-core load lands,
masquerading as a `trio`-backend deadline-miss "regression" on
byte-identical code.

Deats,
- burns every core for `CPU_PERF_SECS` (default 4s) and samples the
  ACHIEVED `scaling_cur_freq` steady-state (post boost-ramp) vs the
  package max ceiling,
- exits 0 when the sustained fraction clears
  `CPU_PERF_HEALTHY_FRAC` (default 0.45), 1 when throttled — so it
  gates a suite run: `scripts/cpu-perf-check && pytest tests/ ...`,
- prints the static knobs first (to show they all read fine) then
  the remediation list on failure (`platform_profile` bounce, USB-C
  PD replug, `ryzenadj`, reboot) w/ the key reminder: do NOT bump
  test budgets — the box is slow, not the code.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-29 18:08:50 -04:00
Gud Boi 333af7debd Add `cpu_perf_headroom()` for throttle-aware deadlines
Mass `trio` deadline-miss failures on byte-identical code turned
out to be a firmware/EC power-cap (AMD PPT/STAPM) clamping the
all-core sustained clock while every static knob (`governor`,
`scaling_max_freq`, EPP, platform-profile) still read "performance"
— invisible to the existing `cpu_scaling_factor()` check. See
`scripts/cpu-perf-check` + the
`ai/conc-anal/trio_033_cancel_cascade_slowdown_depth3_issue.md`
notes.

Deats,
- add `_measure_sustained_headroom()` to `tests/conftest.py`: a
  one-shot ~0.9s all-core burn (explicit `fork`-ctx `mp` procs)
  sampling achieved-vs-max freq AFTER the boost window; under a 0.6
  gate it returns the full inverse fraction (capped 4x), else 1.0;
  best-effort 1.0 on non-linux or any error,
- add `cpu_perf_headroom()`: `max()` of the static scaling factor
  and the (session-cached) sustained probe,
- inflate deadline budgets by it in `test_dynamic_pub_sub`, both
  `test_clustering` cases, the
  `test_multi_nested_subactors_error_through_nurseries` pexpect
  waits + `test_nested_multierrors`,
- `xfail(strict=False)` `test_nested_multierrors` depth=3 under
  throttle: the deep tree trips tractor's INTERNAL reap deadlines
  (`soft_kill`/`hard_kill` `terminate_after=1.6`) minting a
  `Cancelled` inside the runtime — not fixable by test-budget
  inflation; auto-clears once the box un-throttles.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-29 18:08:50 -04:00
Bd f8b7233b3e
Merge pull request #460 from goodboy/docs_vibed_for_serious
Docs (vibed) for serious: real documentation with a modern theme
2026-06-29 16:22:31 -04:00
Gud Boi ca78eb2bed Skip the UDS example on macOS CI
`uds_transport_actor_tree.py` (new in 759d46dd) reliably fails on
macOS CI with a bare `returncode 1`. The example forces
`enable_transports=['uds']`, and UDS-on-macOS is otherwise
un-exercised by the matrix (there's no `tpt_proto=uds` macOS job),
so it's the first thing to surface a macOS UDS-path issue.

It passes cleanly on Linux, and the peer-cred paths are already
platform-guarded + graceful (`get_peer_info`/SO_PEERCRED on linux
vs `get_peer_pid`/LOCAL_PEERPID elsewhere) - so the crash is deeper
in the macOS UDS stack. The test swallows the subproc stderr (only
re-raises when the last line has "Error"), so the exact failure
can't be pinned from CI logs; needs a macOS box to root-cause.

Skip it on `_friggin_macos and ci_env` (same shape as the existing
`full_fledged_streaming_service` skip) so #460 lands green; the
macOS UDS root-cause is a follow-up.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-29 13:55:25 -04:00
Gud Boi 43216c8d4a Show error/cancel teardowns in the `Context` diagram
The `context_handshake` sequence diagram only showed the graceful
`Return` ending - underselling the whole SC pitch (errors + cancels
propagate *both* ways across the wire). Add a "... or, instead of a
graceful Return" group with the non-graceful endings,

- parent-initiated: `ctx.cancel()` (or a parent-side error) ->
  child relays `Error: ContextCancelled`,
- child-initiated: child ships `Error: <raised>` (or
  `ContextCancelled`).

Labelled `Error: ...` since `ContextCancelled` is an exception that
rides inside an `Error` wire msg, not its own msg type - keeping
the diagram's wire-level labels honest. Re-render the committed
`_diagrams/context_handshake.svg` fallback (CI has no `d2` bin) to
match.

Also cache-bust d2 images in `d2diagrams.py`: sphinx tags css/js
with `?v=<hash>` but not images, so an edited diagram kept serving
the browser's stale copy at the stable `_images/<stem>.svg` url
(cost us a manual hard-refresh). Hash each rendered svg + rewrite
only the d2 `<img src>` to `...svg?v=<hash8>` via the
`html-page-context` hook - content-addressed, so unchanged diagrams
keep their query.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-28 21:55:39 -04:00
Gud Boi 23edf03563 Slim README to lean on the hosted docs
Continuing the README slim-down: stop inlining content that now
lives (and renders better) on the GitHub Pages site, and just
point readers there.

- "Where do i start" drops the SC reading-list links in favor of
  a single nudge to the docs quickstart.
- "Example codez" drops the full `we_are_processes.py` listing -
  keep the one-minute pitch prose + the docs/examples links.
- "What's on the TODO" collapses the inline bullet list down to a
  link to the docs roadmap ("what the future holds").

Also,
- prune the now-dead RST link targets (`blog post`, `trio docs`,
  `SC`, `libdill-docs`, `async sandwich`, `messages`, ..).
- swap the `|docs|` badge from readthedocs over to the GitHub
  Pages `docs.yml` action badge + `goodboy.github.io` target.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-28 21:14:12 -04:00
Gud Boi 3753c671be Slim README to lean on the hosted docs
Continuing the README slim-down: stop inlining content that now
lives (and renders better) on the GitHub Pages site, and just
point readers there.

- "Where do i start" drops the SC reading-list links in favor of
  a single nudge to the docs quickstart.
- "Example codez" drops the full `we_are_processes.py` listing -
  keep the one-minute pitch prose + the docs/examples links.
- "What's on the TODO" collapses the inline bullet list down to a
  link to the docs roadmap ("what the future holds").

Also,
- prune the now-dead RST link targets (`blog post`, `trio docs`,
  `SC`, `libdill-docs`, `async sandwich`, `messages`, ..).
- swap the `|docs|` badge from readthedocs over to the GitHub
  Pages `docs.yml` action badge + `goodboy.github.io` target.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-28 14:20:50 -04:00
Gud Boi c028e03d32 Update quickstart `we_are_processes` walkthrough
The reworked `we_are_processes.py` (now `Context`-API based) prints
different stdout, so the hardcoded expected-output block went
stale: swap the old `Yo, i'm 'worker_N'...` / "self-destruct in 1
sec" lines for the real run - "self-destruct in 2s.." then the
per-worker `Started ep-task in subactor,` / `N::'worker_N'@<pid>`
blocks.

Also retell the prose to match: subs spawn concurrently from bg
`trio` tasks, each runs a `@tractor.context` `endpoint()` that
`ctx.started()`-hands its name + pid back through
`Portal.open_context()`, then parks in `trio.sleep_forever()`
before the root crashes + the tree is reaped.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-28 13:29:02 -04:00
Gud Boi 274cab948c Slim the README to one `Context` showcase
The README inlined six example scripts; per our own "point at
`examples/`, don't duplicate" philosophy that's a lot of rot
surface. Drop all but the showcase and rework it onto the modern
`Context` API to mirror the docs landing + `we_are_processes.py`:
spawn a subactor per core, `open_context()` each, crash the root,
reap the tree.

Replace the rest with a short "want more?" pointer to the docs site
+ the `examples/` dir (where the debugger, streaming, cancellation,
`asyncio`, msging + cluster demos all live, CI-run).

Also tidy a few nits while here,
- fix the title typo "structurred" -> "structured",
- close the unterminated quote in the `UV_PROJECT_ENVIRONMENT`
  install snippet,
- add the blank line a nested `trionics` bullet list needs so the
  PyPI long-desc (this file) renders as valid RST.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-28 13:29:02 -04:00
Gud Boi c3e5ed1bd7 Rework landing example onto the `Context` API
`run_in_actor()` is slated to become a hilevel wrapper
(`runtime/_supervise.py` "TODO: DEPRECATE THIS"), so the showcase
`we_are_processes.py` shouldn't lead with it. Move it to the modern
API: each `worker_<i>` subactor runs a `@tractor.context`
`endpoint()` that `started()`-hands its name + pid back over
`Portal.open_context()` and parks; the root sleeps then raises on
purpose so the runtime reaps the whole tree (zero zombies).

The subs spawn concurrently from bg `trio.Task`s so each child's
cold `import tractor` (~0.4s, see #470) overlaps instead of
stacking; a comment flags the coming `main_thread_forkserver`
backend (#463) which'll make serial spawns cheap enough to just
loop.

Match the landing prose to the snippet — name the `Context` +
`started()` handshake it now leads with.

Also, document `--watch examples` on the `sphinx-autobuild` cmds so
edits to `literalinclude`-d example scripts (which live outside
`docs/`) live-reload too.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-28 13:29:02 -04:00
Gud Boi d688eaba84 Prime the docs for GitHub Pages serving
The gh-pages deploy workflow already exists (build on PR + main,
deploy on main push via `actions/deploy-pages`); wire up the sphinx
side,

- `sphinx.ext.githubpages`: emits a `.nojekyll` so Pages serves the
  `_static/` + `_images/` dirs (Jekyll otherwise drops `_`-prefixed
  paths, breaking every asset),
- `html_baseurl`: the canonical site root
  (`https://goodboy.github.io/tractor/`) for the `<link
  rel="canonical">` + `og:url` tags.

The one remaining (one-time, manual) step is the repo setting:
Settings -> Pages -> Source = "GitHub Actions".

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-28 13:29:02 -04:00
Gud Boi 46197f00c3 Redesign the landing hero + nav branding
Rework the landing top into a left-anchored hero row and match the
navbar branding to it.

- `index.rst` + new `tractor_hero.html`: an inline-svg hero
  (`[logo] distributed structured concurrency`) with a full-width
  sub-tagline, plus a single screen-reader-only `<h1>` ("tractor")
  so the section headings become proper `<h2>` (was 4x `<h1>`), and
  the intro prose reflowed into 3 bullets,
- `conf.py`: a `logo.text` ("tractor") beside the navbar wireframe,
  polars-style,
- `custom.css`: the hero-row layout (logo + tagline + sub-line,
  single row on desktop / stack on mobile), the navbar brand styled
  like the centered nav tabs with `display:inline-block` to kill
  the parent link's propagated hover-underline, and smaller landing
  section headings w/ their `tl;dr` margin-note pulled level.

The hero linework stays `fill: currentColor` so the whole thing
flips with the light/dark toggle.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-28 13:29:02 -04:00
Gud Boi 4fdbc98438 Document logo tweaking + add `svgtool` helper
Capture the (easy-to-forget) logo recolour + theme-preview flow,

- new `notes_to_self/svgtool.py`: a headless-firefox svg helper
  (`colors`/`recolor`/`preview`/`page`); its `page --theme
  dark|light` injects a `localStorage` pin so a pydata `auto`-mode
  build renders its dark variant in a screenshot — the only way to
  verify theme-adaptive bits headlessly,
- `howtodocs.md`: a "Tweaking the logo" table (which file +
  `color:` knob per context) and an `svgtool` section that breaks
  down the `--theme` trick for the sphinx-rusty.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-28 13:29:02 -04:00
Gud Boi 1f84d5f529 Adopt the wireframe logo in navbar + README
Carry the landing hero's transparent-faces wireframe look to the
two other logo contexts. An `<img>`-embedded svg can't read the
page CSS (no `currentColor`), so the colours are baked per target,

- navbar: 2 variants (`tractor_logo_nav_light.svg` near-black
  lines, `…_nav_dark.svg` near-white) wired via the pydata
  `logo.image_light`/`image_dark` opts so it swaps with the theme
  toggle, matching the hero,
- README: `tractor_logo_wire.svg` — one neutral-grey, with INLINE
  fills (no `<style>`) so it reads on both GitHub light + dark and
  survives GitHub's svg sanitiser.

Faces are `fill: none` throughout; the filled `tractor_logo_side`
stays the recolour source + favicon.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-28 13:29:02 -04:00
Gud Boi eb87235835 Make the hero logo a theme-adaptive wireframe
Swap the landing-page hero from a static `<img>` (white-faced —
which glared as a light box on the dark theme) to an INLINE svg
whose linework uses `fill: currentColor` and whose faces are
transparent. The wireframe now inherits the theme text colour
(`svg.hero-logo { color: var(--pst-color-text-base); }`) and the
page background shows through, so it flips correctly with the
light/dark toggle.

- `tractor_logo_hero.html`: the inline svg partial (`.st0` ->
  `currentColor`, `.st1` -> `none`), `.. raw:: html :file:`-d into
  `index.rst`,
- the shared `tractor_logo_side.svg` is untouched here (navbar +
  favicon still ride it).

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-28 13:29:02 -04:00
Gud Boi 3072686c49 Document serving the docs over a LAN
Add a "Share on your LAN" blurb to both docs how-to homes
(`notes_to_self/howtodocs.md` + the dev-tips "Building these docs"
section): bind the server to all ifaces — via `sphinx-autobuild
--host 0.0.0.0` (live-reload) or `python -m http.server --bind
0.0.0.0` (static) — so a peer on your subnet can hit
`http://<lan-ip>:8000` (`hostname -I` prints the ip).

Verified a LAN-ip `curl` returns 200 + the socket binds `0.0.0.0`;
includes a trusted-LAN-only caveat since it's an unauthenticated
server on every interface.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-28 13:29:02 -04:00
Gud Boi c351671c16 Document the docs build/preview flow
Collect the sphinx build + live-preview one-liners (incl. the nix
`.#docs` opt-in shell and the `d2` diagram specifics) so
contributors don't have to reverse-engineer them,

- new `notes_to_self/howtodocs.md` terse cheat-sheet (force-added
  past the `notes_to_self/` ignore, same as `howtorelease.md`),
- a rendered "Building these docs" section in the dev-tips guide
  (`docs/project/dev-tips.rst`),
- a README pointer to the note.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-28 13:29:02 -04:00
Gud Boi 32d22abd93 Relock `uv.lock` for the `docs` dependency-group
The `docs` group (sphinx 9 + `pydata-sphinx-theme` stack) was added
to `pyproject.toml` but its lockfile half never landed — the
committed `uv.lock` still matched `main` and `uv lock --check`
flagged it out-of-sync. Relock to add the (additions-only) docs
deps so `uv sync --locked` and any CI lock-check pass.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-28 13:29:02 -04:00
Gud Boi be0a10b0d2 Add prompt-io provenance for d2-ext + docstring patches 2026-06-28 13:29:02 -04:00
Gud Boi 888162ea0c Fix informal-RST in docstrings for clean autodoc
The new `api/` reference pages surfaced 22 docutils warnings from
informal reST in public docstrings; fix the markup so the docs
build is warning-free (24 -> 0), clearing the path to a future
`-W`/`nitpicky` flip in CI.

Deats (docstring content only; no code/signature changes),
- give bullet lists a blank line + base-column indent (`Context`,
  `Context.cancel_called`/`.cancelled_caught`/ `.outcome`,
  `ActorNursery.cancel_called`, `query_actor`,
  `open_crash_handler`),
- demote the under-short `Behaviour:` underline in `Context.cancel`
  to a `**bold**` label,
- close an unbalanced backtick in the `wait_for_actor` summary and
  use the `` `role`\ s `` escaped-plural idiom where a role was
  pluralized (`gather_contexts`, `mk_pdb`, `MsgCodec`, msg `Error`,
  `open_context_from_portal`),
- make the `|_` method-tree in `ContextCancelled.canceller` a
  literal block (the bare `|` was read as a substitution ref),
- same blank-line fix for the `#318` entry in `NEWS.rst`.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-28 13:29:01 -04:00
Gud Boi 5bc15497b1 Harden the `.. d2::` directive render path
Three robustness fixes surfaced by the PR #460 self-review,

- a render that's *attempted and fails* (a `d2` bin is present but
  errors on the source) now raises a docutils error instead of
  silently serving the stale committed SVG — so `sphinx-build -W`
  fails on a broken `.d2` edit. The no-binary case stays a quiet
  committed-SVG fallback.
- render into a sibling temp-file then `os.replace()` so a
  failed/torn render can never clobber a good committed SVG.
- guard against 2 distinct `.d2` sources colliding on a single
  `_diagrams/<stem>.svg` within a build.

`render_svg()` now returns a `RenderResult` tristate that `run()`
maps to error / raw-source-literal / image.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-28 13:29:01 -04:00
Gud Boi 1c2048aefb Add opt-in `docs` dev-shell with `d2`
Factor the shared dev-shell toolchain into `basePkgs` + `baseHook`
so extra-tooling shells can extend it, then add a `docs` shell =
base + `pkgs.d2` (the diagram renderer our `.. d2::` sphinx
directive shells out to). Keeps `d2` OUT of the `default` shell so
casual dev envs stay lean.

Enter with `nix develop .#docs`, then build via `uv run --group
docs make -C docs html`.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-28 13:29:01 -04:00
Gud Boi b2eac01b37 Fix review nits from PR #460 self-review
Address the actionable findings from the `/code-review` pass
(#1-6); the d2-ext + docstring nits (#7-10) are left for a
follow-up.

Deats,
- `uds_transport_actor_tree.py`: `portal.chan.raddr.sockpath` is
  the *shared listener* socket (named for the root registrar), NOT
  the child's path — relabel it + lead with the per-child peer pid,
  and stop claiming it's the child addr in the docstring,
- `docs.yml`: scope the `pages` `concurrency` group to the `deploy`
  job w/ `cancel-in-progress: false` so a PR build can't cancel an
  in-flight production deploy,
- `architecture.rst`: `'subint'` is not a selectable `start_method`
  on this branch (`SpawnMethodKey` lacks it) — reframe as
  in-development/roadmap,
- `context.rst`: `StreamOverrun` isn't re-exported from `tractor`;
  point at `tractor._exceptions`,
- `debugging/`: sweep the 3 literalinclude'd examples off the
  deprecated `.result()` -> `.wait_for_result()`,
- `quickstart.rst`: proc-title is `name@pid` (per `Aid.reprol`),
  not `@uuid`.

NOTE, the `debugging/` examples are pexpect-tested; re-run
`tests/devx/test_debugger.py` for them.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-28 13:29:01 -04:00
Gud Boi f3ab10b103 Add gh-pages docs deploy workflow
`docs.yml`: sphinx build on PRs + main pushes (via `uv sync
--no-dev --group docs`), deploying to gh-pages only from main
pushes using the official
`actions/{upload-pages-artifact,deploy-pages}` pair - the
`msgspec`-style flow wished for in #123. No `d2` bin in CI (yet) so
the committed SVGs serve as-is.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-28 13:29:01 -04:00
Gud Boi bbb41b69c5 Write the big boi docs content tree
Replace the ancient `docs/index.rst` (still teaching
`tractor.run()`, `@stream` + arbiters..) with a full ~32 page tree
teaching ONLY the current api (`.wait_for_result()`, registrar
naming, `@context` + `open_context()` as the core model),

- landing: hero example, feature cards + canon links,
- `start/`: install + a 4-example quickstart on-ramp,
- `explain/`: an "SC across processes" essay distilling the essence
  per #157's orig ask + a runtime architecture tour,
- `guide/`: 12 task-focused pages incl the flagship multi-process
  debugging walkthrough, `Context` + `MsgStream` deep-dives,
  cancellation semantics (self-vs-cross cancel rules), discovery,
  infected `asyncio`, typed msging + the #126 testing-tips page,
- `api/`: 10 curated autodoc pages (all targets import-verified vs
  the reorg'd subpkg tree),
- `project/`: changelog include, ported dev-tips (drops old
  `docs/dev_tips.rst`) + roadmap.

Every code block is a `literalinclude` from `examples/`
- zero duplication, all CI-run - w/ `d2` figs floated
into the RHS margin per the 3-col design. Build is green; the 24
remaining warnings all source from lib docstring rst-isms or legacy
`NEWS.rst` content.

Substantially resolves #157 (refine round pending); chips at #175 +
#126.

Prompt-IO: ai/prompt-io/claude/20260611T175152Z_8526985c_prompt_io.md

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-28 13:29:01 -04:00
Gud Boi 4a1f633f8c Add `d2` diagram sources + rendered SVGs
7 hand-authored `docs/diagrams/*.d2` sources (sketch-mode,
grayscale theme, `elk` layout) w/ their pre-rendered fallback SVGs
committed under `docs/_diagrams/` (served as-is when no `d2` bin is
found, eg. in CI),

- `actor_tree`: the hero supervision-tree,
- `context_handshake`: seq diagram of the
  `Start`/`StartAck`/`Started`/`Yield`/`Stop`/`Return` ctx dialog,
- `streaming_pipeline`: 4-actor fan-in topology,
- `runtime_stack`: the per-actor layer cake,
- `debug_lock`: root-tty-lock REPL serialization,
- `error_propagation`: one-cancels-all boxed-err flow,
- `infected_aio`: guest-mode loop nesting.

Rendered w/ `d2` v0.7.1 via `nix run nixpkgs#d2`.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-28 13:29:01 -04:00
Gud Boi 6dafe85f54 Rebuild sphinx scaffold on `pydata` theme
Full `docs/conf.py` rewrite targeting `pydata-sphinx-theme` 0.18
(3-col layout: nav-index left, content middle, page-toc right)
skinned to a minimal b&w look via `_static/css/custom.css`, keeping
the `algol_nu` pygments style.

Also add 2 local sphinx exts under `docs/_ext/`,

- `d2diagrams.py`: a `.. d2::` directive rendering `d2lang` sources
  at build time when a `d2` bin is found (`D2_BIN` env supports
  argful values eg. `'nix run nixpkgs#d2 --'`), falling back to any
  git-committed SVG, else emitting the raw source as a literal
  block; no pypi ext was usable (all stubs or returncode-swallowing
  per the #157 research),
- `marginalia.py`: a theme-agnostic `.. margin::` directive
  (`sphinx-book-theme` style `Sidebar` subclass) for prose-anchored
  RHS asides; the custom css floats these (and `:margin:` d2 figs)
  into the right gutter on wide screens.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-28 13:29:01 -04:00
Gud Boi e8ec1bcc57 Add `docs` dependency-group for sphinx 9 stack
Fill in the long-TODO'd `[dependency-groups]` entry: `sphinx>=9.1`
+ `pydata-sphinx-theme>=0.18` +
design/copybutton/opengraph/togglebutton exts; relock.

Build via, `uv run --group docs make -C docs html`

Theme choice per the #157 research: `numpy`, `ray` + `polars` all
ride `pydata-sphinx-theme` these days (`ray` migrated off
`sphinx-book-theme`, which now hard-pins a stale pydata 0.16.1
anyway).

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-28 13:29:01 -04:00
Gud Boi 759d46dd17 Add 5 examples filling doc-tutorial gaps
New runnable (and so CI-auto-tested) scripts backing the new docs
guides, each following the `test_docs_examples.py` runner
conventions,

- `typed_payloads.py`: `@tractor.context(pld_spec=)` typed
  `started()`/stream roundtrip + a deliberate `MsgTypeError` catch
  demoing send-side validation,
- `nested_actor_tree.py`: 3-level tree w/ fan-out rpc through a
  mid-tier supervisor actor,
- `service_daemon_discovery.py`: registered daemon located via
  `find_actor()`/`wait_for_actor()` sans any spawn-portal ref,
- `uds_transport_actor_tree.py`: `enable_transports=['uds']` tree
  printing the filesystem sockaddr + kernel peer-pid creds,
- `streaming_broadcast_fanout.py`: one ipc stream fanned out to N
  local tasks via `MsgStream.subscribe()`.

All gaps were mined from in-code TODOs + the docs recon pass; see
#175 for the orig tutorial wishlist.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-28 13:29:01 -04:00
Gud Boi f11a74c15d Un-hide stdlib primes example, add `trio` shim
Rename `parallelism/_concurrent_futures_primes.py` ->
`concurrent_futures_primes.py` so the example-runner
(`test_docs_examples.py`) stops skipping it (leading `_` =
excluded) and CI finally exercises the `concurrent.futures`
baseline we compare against in the new parallelism guide.

Deats,
- keep the original executor code verbatim in a sync
  `check_primes()` fn for clean docs excerpting,
- add module docstring + zero-arg `async def main()` +
  `trio.run(main)` guard per the runner conventions.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-28 13:29:01 -04:00
Gud Boi fb8a6b2473 Use `.wait_for_result()` in dated examples
Swap the deprecated `portal.result()` calls for the modern
`.wait_for_result()` spelling in,
- `a_trynamic_first_scene.py` (x2)
- `actor_spawning_and_causality.py`
- `parallelism/single_func.py`

These 3 are literalinclude'd by the new docs tree so the rendered
code must teach the current api; the `debugging/` set still calls
`.result()` (pexpect pattern-matched tests, left for a follow-up
sweep).

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-28 13:29:01 -04:00
Bd 2e2d988bf7
Merge pull request #469 from goodboy/wkt/rm_test_warnings
Rm all the test suite warnings
2026-06-28 13:17:38 -04:00
Gud Boi bffeba952a Use `aid.uid` over manual `(name, uuid)` tuples
The `.uid`->`.aid` migration (prior commits) re-assembled the
legacy uid pair by hand as `(x.aid.name, x.aid.uuid)`.
`Aid.uid` (`msg/types.py`) is the non-deprecated canonical
property returning exactly that pair, so swap to `x.aid.uid`
everywhere — byte-identical, just DRY + consistent.

Covers the 5 review-flagged source sites (`msg._ops`,
`devx.debug._sync`, `experimental._pubsub`, `_exceptions`)
plus the ~22 unflagged identical sites (`_streaming` f-string
logs + 6 test modules); also tightens the `_exceptions`
`our_uid` annotation to `tuple[str, str]`.

Review: PR #469 (copilot-pull-request-reviewer)
https://github.com/goodboy/tractor/pull/469#pullrequestreview-4575979601

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-27 21:38:30 -04:00
Gud Boi 3e04968e79 Floor `get_rando_addr` port at 1024, not 1000
Ports below 1024 are privileged on linux
(`net.ipv4.ip_unprivileged_port_start = 1024`), so a non-root
`.bind()` on one raises `PermissionError`. The old `1000 +` floor
could roll into the 1000-1023 range and trip flaky `[Errno 13]
Permission denied` registry-listener binds; bump the base to 1024
so every generated port is non-privileged.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-27 20:35:58 -04:00
Gud Boi bc91a1bdfa Inline `send_yield` body to drop its deprecation
`experimental._pubsub`'s `fan_out_to_ctxs()` pushed each packet
via the now-deprecated `Context.send_yield()`, tripping its own
`DeprecationWarning` on every published msg.

Inline that method's exact body —
`await ctx.chan.send(Yield(cid=ctx.cid, pld=payload))` — so the
wire msg stays byte-identical (still a `Yield`) yet no warning
fires. The "proper" fix (swap to `MsgStream.send()` via
`Context.open_stream()`) needs BOTH sides migrated: the
subscriber still uses the legacy `Portal.open_stream_from()`
which never calls `.started()`, so a publisher-side
`open_stream()` raises a `RuntimeError` ("`started()` must be
called before opening a stream") — verified. Left as the
module's standing rework TODO.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-25 20:32:05 -04:00
Gud Boi d55c971644 Filter 2 unfixable-at-source test warnings via ini
Two test-suite warnings can't be fixed at source, so register
documented `[tool.pytest.ini_options] filterwarnings` ignores
(real library users still see both):

- `forkpty()` multi-threaded `DeprecationWarning` — emitted by
  `pexpect` (stdlib `pty.py`) for the `devx` debugger REPL tests;
  we never call it. UPSTREAM fix = pexpect avoiding `forkpty()`
  under multithreading.
- `@tractor.stream` `DeprecationWarning` — the legacy
  `test_legacy_one_way_streaming.py` exists to test that
  DEPRECATED API; the warning fires at import (module-level
  decorators) so a per-test mark can't catch it, hence the ini
  filter.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-25 19:09:42 -04:00
Gud Boi dc8562cf14 Adapt SIGINT test to `trio` strict exception-groups
`test_cancel_via_SIGINT_other_task` opened its nursery with the
now-deprecated `strict_exception_groups=False`. Under strict EGs
(trio's default since 0.25) the SIGINT-induced `KeyboardInterrupt`
surfaces wrapped in a `BaseExceptionGroup` — alongside the child
tasks' `Cancelled`s, so it's MULTI-exc and `collapse_eg()` can't
fold it back to a bare KI (the `# why no work!?` the author hit).

Drop the deprecated flag, use a default (strict) nursery, and
assert the result is a bare `KeyboardInterrupt` OR a
`BaseExceptionGroup` whose `.subgroup(KeyboardInterrupt)` is
non-empty.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-25 19:09:42 -04:00
Gud Boi 4607090a5d Use `.aid` fields over deprecated `.uid` in tests
Mirror the core swap on the test side: every `Channel`/`Actor`
`.uid` access now reads the non-deprecated `.aid` struct's
`(.name, .uuid)` (or `.aid.name` for a `.uid[0]`), keeping the
compared / printed value byte-identical while silencing the
self-inflicted `DeprecationWarning`. Touches
`test_context_stream_semantics`, `test_spawning`, `test_local`,
`test_advanced_streaming`, `msg.test_ext_types_msgspec`,
`discovery.test_multi_program`, `test_infected_asyncio`.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-25 17:44:22 -04:00
Gud Boi 727aee0f84 Use `.aid` fields over deprecated `.uid` in core
`tractor`'s own `Channel.uid`/`Actor.uid` now warn + redirect to
the `.aid: msg.Aid` struct, yet the runtime kept calling the
deprecated tuple property internally — tripping its own
`DeprecationWarning` across many tests.

Swap each internal `.uid` for the behavior-preserving
`(x.aid.name, x.aid.uuid)` (and `.uid[0]` -> `.aid.name`) so the
emitted value / log output stays byte-identical, just sourced
from the non-deprecated struct. Touches `msg._ops`,
`_streaming`, `_exceptions`, `devx.debug._sync`,
`experimental._pubsub`.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-25 17:37:29 -04:00
Gud Boi 725c4ef212 Use `.cancel_called` over deprecated `ActorNursery.cancelled`
`tests.test_cancellation` asserted on `ActorNursery.cancelled`,
which now warns + redirects to `.cancel_called`. Swap to the
non-deprecated property so the suite stops tripping its own
`DeprecationWarning`.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-25 17:32:54 -04:00
Gud Boi c4dee6ab4b Raw-string pexpect patterns to kill `SyntaxWarning`s
`tests.devx.test_tooling`'s end-of-tree `expect()` patterns use
a literal `\(` (pexpect regex), which Python 3.12+ flags as an
invalid escape sequence. Make them raw strings so the `\(` is
preserved verbatim and no `SyntaxWarning` fires.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-25 17:32:54 -04:00
Gud Boi e1039b4d86 Register `has_nested_actors`/`trio` pytest marks
`config.addinivalue_line('markers', ...)` the two custom marks
the suite uses but never registered, silencing
`PytestUnknownMarkWarning` (e.g. `test_debugger.py`'s
`has_nested_actors`). Registering in-code (vs a `pyproject`
`markers=` block) keeps the mark + its doc next to the rest of
the harness config.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-25 17:32:54 -04:00
Gud Boi 3335865d5d Add `hung-dump.xsh` hang-triage tool to `scripts`
Snapshot diagnostic state for a hung `pytest`/`tractor`
process tree: for each pid (+ `pgrep -P` descendants) dump
the `ps` forest, `/proc/<pid>/wchan` + `stack` (kernel-side
blocked syscall), and `py-spy dump` (python-side stack).

Salvaged from the retired `wkt/warnings_cleanup` branch
(untracked, not upstream); homed here as standalone tooling.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-25 15:07:15 -04:00
Bd d95cd94e20
Merge pull request #465 from goodboy/trio_033_upgrade
Add `supervise_run_process()` to `trionics`
2026-06-24 23:30:18 -04:00
Gud Boi 6764251e6b Clarify eg-wrap contract + tighten `stdout` typing
Doc/type accuracy in `supervise_run_process()`:

- the docstring claimed the rc!=0 `CalledProcessError` had
  "no nursery-eg-wrapped CPE to catch/`collapse_eg`", but it
  IS raised as a task into the parent `tn`, so under
  `trio>=0.33` it surfaces `ExceptionGroup`-wrapped like any
  `tn.start()`-task raise. Say so + point at `except*` /
  `collapse_eg()` (matching the inline rc-check comment).
- widen `stdout: int` to `int|None` — the param accepts an
  explicit `None` (inherit) per its own docstring.
- note that stdout is NOT captured, so the CPE note carries
  only stderr (a cmd logging errors to stdout should
  `relay_stdout=True`).

Review: PR #465 (copilot-pull-request-reviewer)
https://github.com/goodboy/tractor/pull/465#pullrequestreview-4548191641

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-24 19:09:45 -04:00
Gud Boi 02cb789901 Bound retained `stderr` to a tail + de-dup decode
`_relay_stream_lines()` kept ALL captured `stderr` for the
rc!=0 `CalledProcessError` note, so a long-lived `check=True`
child could buffer its entire stderr history in parent mem.
Add an `accum_cap` (the new `_STDERR_NOTE_TAIL_BYTES`=16KiB)
that trims the buffer to its TAIL — the bytes nearest the
failure, which are what the note wants anyway.

While here, factor the per-line decode/`rstrip` into a
`_decode_line()` helper so the in-loop + trailing-flush
emits no longer duplicate it.

Review: PR #465 (copilot-pull-request-reviewer)
https://github.com/goodboy/tractor/pull/465#pullrequestreview-4548191641

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-24 19:08:19 -04:00
Gud Boi a3f8765264 Resolve relay `emit` only when relaying
`supervise_run_process()` bound the relay `emit` method
(`getattr(log, relay_level)`) unconditionally at entry, so a
bad/typo'd `relay_level` raised `AttributeError` even for a
call that requested NO relay (the common silent path). Bind
it lazily behind `relay_stdout or relay_stderr` so only an
actual relay request validates the level.

Review: PR #465 (copilot-pull-request-reviewer)
https://github.com/goodboy/tractor/pull/465#pullrequestreview-4548191641

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-24 19:05:55 -04:00
Gud Boi e076296cda Guard `stdout=PIPE`/`capture_*` footguns
Two `supervise_run_process()` caller-kwarg footguns that
previously failed obscurely now raise a clear, early
`ValueError`:

- a bare `stdout=subprocess.PIPE` override spawned NO drain
  reader (only `relay_stdout` does), so a child emitting
  >64KiB would deadlock on a full ~64KiB pipe buffer.
- `trio`'s `capture_stdout`/`capture_stderr` alias our
  MANAGED `stdout`/`stderr` keys; forwarding them made
  `trio.run_process` raise an opaque "can't specify both"
  error, so reject them up-front pointing at `relay_*` /
  the `stdout=` override.

Review: PR #465 (copilot-pull-request-reviewer)
https://github.com/goodboy/tractor/pull/465#pullrequestreview-4548191641

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-24 19:04:35 -04:00
Gud Boi 7f6782b9be Fix `test_simple_context` for `trio>=0.33` groups
Under `trio>=0.33`'s strict exception-groups a lone
`error_parent` arrives wrapped in a 1-exc `ExceptionGroup`
instead of collapsing to the bare exc, so it skips the
`except error_parent` arm and lands in the group arm — where
`is_multi_cancelled()` is (correctly) `False`, tripping the
assert. Also accept `beg.subgroup(error_parent) is not None`.

Review: PR #465 (copilot + self CI-triage)
https://github.com/goodboy/tractor/pull/465#pullrequestreview-4548191641

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-24 18:19:16 -04:00
Gud Boi 35414a42a9 Scale `test_most_beautiful_word` deadline for CI
Flat `trio.fail_after(1)` wraps a whole actor spawn + IPC
round-trip in `test_most_beautiful_word` — sub-second on a
warm box, but slow/noisy CI runners (esp. macOS) blow the
1s deadline with a `trio.TooSlowError`. Scale the budget by
`cpu_scaling_factor()`: a strict `1s` locally (factor
`1.0`), CI/CPU-throttle headroom (~3x) on macOS.

Review: PR #465 (copilot + self CI-triage)
https://github.com/goodboy/tractor/pull/465#pullrequestreview-4548191641

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-24 18:19:16 -04:00
Gud Boi 8fcb9eb7d6 Skip `test_parent_tty_isolated` off-Linux (`/proc`-only) 2026-06-24 18:19:16 -04:00
Gud Boi 14329ec1d6 Strip ANSI + accept `_create(...)` in devx tests
Two version-compat fixes for the `devx` debugger
test-suite, both about matching upstream output that
got more verbose w/ recent lib releases.

ANSI stripping (`tests/devx/conftest.py`),
- Add `ansi_strip(text)` helper + `_ansi_re` pattern
  (regex per https://stackoverflow.com/a/14693789).
- Apply inside `in_prompt_msg()` + `assert_before()` so
  substring matches against REPL/traceback output stay
  robust to color leakage.
- Motivated by py3.13's colored tracebacks +
  `pdbp`/pygments highlighting leaking ANSI even when
  `PYTHON_COLORS=0` is set in the `spawn` fixture (not
  every renderer in the spawned subproc honors it).
- Replaces the longstanding inline TODO that linked
  the SO answer w/o impl'ing.

trio 0.30+ `Cancelled._create(` match (`test_debugger`),
- In `test_shield_pause` swap the two
  `"raise Cancelled._create()"` assertion patterns →
  `"raise Cancelled._create("` (open-paren form, no
  closing).
- trio >=0.30 raises a multi-line
  `raise Cancelled._create(source=.., reason=..,
  source_task=..)` w/ cancel-reason metadata, so the
  legacy bare-`()` form no longer matches. Inline
  comment documents the trio-version pivot.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 3854cf5ecb)
(cherry picked from commit c07cf2546b)
2026-06-24 18:19:16 -04:00
Gud Boi 008824d650 Add `supervise_run_process` to `trionics._subproc`
A `trio.Nursery.start()`-style wrapper around
`trio.run_process()` that surfaces rc!=0 errors
deterministically, ALWAYS isolates the parent
controlling-tty, and optionally live-relays the child's
std-streams to `log.<level>` per-line. Suits both
short-lived test-runners + long-lived daemons.

`supervise_run_process()`,
- Deterministic rc!=0: pass `check=False` to `trio`
  and do our OWN post-drain rc-check from the
  supervisor coro body AFTER `own_tn.__aexit__` — NOT
  inside the internal nursery, since that would
  race-cancel the still-draining relay reader and lose
  stderr lines. (Re)build + raise a BARE
  `subprocess.CalledProcessError`: `.stderr=` for
  programmatic callers + an `add_note()`'d
  `|_.stderr:` block for human teardown logs. No
  nursery-eg-wrapped CPE to `collapse_eg` around.
- Parent controlling-tty isolation: `stdin=DEVNULL`
  always, `stdout=DEVNULL` unless relayed/overridden
  (via `stdout=` kwarg w/ `_UNSET` sentinel so explicit
  `None` = inherit still works). Prevents a spawned
  program from clobbering the launching tty's scrollback
  w/ control-seqs.
- Live per-line relay: `relay_stdout=True`/
  `relay_stderr=True` → relayed to `log.<relay_level>`
  (default `'io'`, our custom level 21). Picked to sort
  just above stdlib `INFO`=20 so it shows at usual
  `info`/`devx` levels yet stays separately filterable;
  `runtime`=15 was REJECTED as a default since it'd be
  silently filtered at usual verbosity — footgun for
  daemon supervisors whose whole point is visibility.
  STREAMED, not buffered-until-exit.
- Non-blocking `tn.start()` semantics: live
  `trio.Process` handed up via
  `task_status.started()` immediately (else
  `tn.start()` would block till child exit, losing
  the long-lived-daemon use case). Supervise/relay bg
  tasks run to completion in this coro.
- `**run_process_kwargs` forwarded verbatim (env, shell,
  cwd, start_new_session, executable, ...); MANAGED keys
  (`stdin`/`stdout`/`stderr`/`check`) win on conflict.
- Crash-handling layer intentionally NOT baked in —
  compose `maybe_open_crash_handler()` ON TOP at the
  call-site.

`_relay_stream_lines()` helper,
- Concurrent pipe-drain reader. MANDATORY whenever piping
  w/o `capture_*` since nothing else drains the OS pipe —
  child blocks on `write()` once kernel buf (~64KiB) fills
  → deadlock.
- Modes (combine freely): `emit`-only live relay,
  `accum`-only silent drain+capture (for the CPE note),
  or both. Per-line splitting handles cross-chunk
  residuals + flushes any trailing un-newline-term'd line
  at EOF.

`_add_stderr_note()` helper,
- Attaches an indented `|_.stderr:` note to a CPE via
  `add_note()` for legible rc!=0 reporting at teardown.

Tests (`tests/trionics/test_subproc.py`),
- Hermetic `trio`-only (no actor-runtime).
- `test_stdout_relayed_per_line`: per-line stdout relay.
- `test_parent_tty_isolated`: child fd1 is OUR pipe (no
  `/dev/pts/*`), fd0 pinned to `/dev/null`.
- `test_no_deadlock_on_big_unnewlined_output`: 200KiB
  no-newline output completes under `fail_after(2)` —
  exercises the concurrent drain (without it, the child
  blocks at ~64KiB).
- `test_stderr_relay_and_cpe_rebuild`: rc!=0 w/
  `relay_stderr=True` → bare `CalledProcessError` w/ the
  `.stderr` note + per-line live relay.
- `test_nonrelay_cpe_note`: rc!=0 w/o relay → same
  deterministic post-drain CPE w/ `.stderr` note (silent
  drain+capture path).

Re-export `supervise_run_process` from `tractor.trionics`.

Prompt-IO: ai/prompt-io/claude/20260601T231429Z_0e3e008b_prompt_io.md

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit f595acc76c)
(cherry picked from commit 6df9ee11bc)
2026-06-24 18:19:16 -04:00
Gud Boi 6255c17db9 Use `is not None` check for peer-connect `event`
Matches the explicit `dict.pop(uid, None)` contract one
line above; same semantics as the prior truthy check.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 0e3e008b0c)
(cherry picked from commit 13ed668512)
2026-06-24 18:19:16 -04:00
Bd 6a1149d401
Merge pull request #467 from goodboy/custom_log_levels_api
Add `add_log_level()` custom log-level API
2026-06-24 18:18:37 -04:00
Gud Boi 88cc68edd3 Fix `add_log_level()` re-registration value drift
The bound `StackLevelAdapter.<name>` emit method captured its
level via a `_level: int = value` default-arg at bind time, so
a later `add_log_level(name, <new-value>)` re-registration left
the method emitting at the *stale* original level. Read
`CUSTOM_LEVELS[name_up]` at call time instead — the method binds
once but always tracks the current registered value.

Also,
- correct the `add_log_level()` docstring's "Idempotent" note
  to describe the call-time value lookup (was the misleading
  "no-op-ish refresh (won't clobber an already-bound method)").
- sync stale `_reap.py` doc/comment markers `tractor[` ->
  `_subactor[` to match the actual `_proctitle._def_prefix`
  proc-title prefix (doc-only drift; the code markers already
  referenced `_proctitle._def_prefix`).

Review: PR #467 (copilot-pull-request-reviewer)
https://github.com/goodboy/tractor/pull/467#pullrequestreview-4562945515

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-24 18:03:28 -04:00
Gud Boi 3727ce9e6f Bump lockfile for `xonsh>=0.23.8` release 2026-06-24 16:20:50 -04:00
Gud Boi 90c316f90c Code-style, couple newline/ws tweaks
(cherry picked from commit 8526985c97)
(cherry picked from commit 0952b33a9e)
2026-06-24 16:20:50 -04:00
Gud Boi 6db796ed83 Pin to latest `xonsh` release
(cherry picked from commit c4cad921b9)
(cherry picked from commit ade15b4204)
2026-06-24 16:20:50 -04:00
Gud Boi eb7ef3f0bf Hoist proc-title prefix to `_def_prefix` const
Make the sub-actor proc-title prefix a single
authoritative constant (`_proctitle._def_prefix`) so
the reap-recognition markers and `xontrib` banner pick
it up automatically — one place to flip the prefix
shape going fwd.

Deats,
- `_proctitle._def_prefix: str = '_subactor'`. New
  module-level const consumed by everything that needs
  to know the prefix.
- `set_actor_proctitle(actor, prefix=_def_prefix)`:
  takes an explicit `prefix` arg (default = the const)
  so callers can override per-spawn if they want.
- Default proc-title format:
  `'tractor[<reprol>]'` → `f'{prefix}[<reprol>]'`
  i.e. `_subactor[<reprol>]` by default.
- `_testing/_reap.py`: cmdline + comm markers source
  the prefix from `_proctitle._def_prefix` instead of
  the hardcoded `'tractor['`. So
  `_is_tractor_subactor()` tracks the const
  automatically.
- `xontrib/tractor_diag.xsh`: `acli.reap` orphan-mode
  banner now interpolates the
  `_TRACTOR_PROC_CMDLINE_MARKERS` tuple directly so
  the human-readable mode line stays in sync if the
  prefix shape changes again.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 3a45dbd503)
(cherry picked from commit fd8d39c0ce)
2026-06-24 16:20:50 -04:00
Gud Boi bb0b2ffa3c Add `add_log_level()` factory + register `IO`=21
Follow-up to f595acc7 (`supervise_run_process`) which
called `log.io(...)` for std-stream relay assuming an
`IO=21` level existed. Add the registration via a new
factory + tests covering both the factory and the new
level.

`add_log_level()` factory,
- One call wires the four (otherwise hand-synced) pieces:
  - `CUSTOM_LEVELS[NAME]` — drives the `stacklevel` bump
    in `StackLevelAdapter.log()` + `get_logger()`'s
    per-level audit.
  - `logging.addLevelName()` — stdlib name registration.
  - `STD_PALETTE[NAME]` + `BOLD_PALETTE['bold'][NAME]` —
    color entries consumed by `get_console_log()`'s
    `ColoredFormatter` build.
  - Same-named (lowercase) emit method bound on
    `StackLevelAdapter` so `log.<name>('msg')` works +
    `get_logger()`'s per-level method audit passes.
- Idempotent: re-registering an existing name is a
  no-op-ish refresh that won't clobber an already-bound
  method.
- Method binding uses a default-arg `_level=value` so
  the level int is captured (not late-bound across
  multiple registrations).

`IO=21` level (first user),
- Purple. Used by `tractor.trionics._subproc`'s
  std-stream relay (see f595acc7).
- Value 21 picked to sit just ABOVE stdlib `INFO`=20 so
  it's SHOWN BY DEFAULT at usual `info`/`devx` console
  levels — a `runtime`=15 relay would be silently
  filtered (footgun for daemon supervisors whose whole
  point is visibility). Still distinctly labeled +
  filterable.

Tests (`tests/test_log_sys.py`),
- `test_io_custom_level_registered`: validates the IO
  level is fully wired (`CUSTOM_LEVELS`, `addLevelName`,
  both palettes, `StackLevelAdapter.io()` callable);
  emits a record + sanity-asserts `21 >= INFO(20)`.
- `test_add_log_level_pluggable`: registers a fresh
  `XLVL=19` (cyan) via `add_log_level()`, asserts all
  four wires + the bound `xlog.xlvl()` emit, then
  try/finally cleans up the module-global mutations so
  later `get_logger()` audits don't trip on a
  half-removed level.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 7bd7dd50c7)
(cherry picked from commit 93558fe3c9)
2026-06-24 16:20:50 -04:00
Bd fd41f88b2b
Merge pull request #464 from goodboy/trionics_start_or_cancel
Add `start_or_cancel()` to `trionics._taskc`
2026-06-24 15:08:20 -04:00
Gud Boi 3fbc77c812 Tighten `start_or_cancel`'s startup-RTE match
The `'started' in rte.args[0]` check was too loose: it
matched a child task's OWN `RuntimeError(...started...)`
(not just trio's "child exited without calling
task_status.started()"), and would `TypeError` on a
non-`str` `args[0]`. Guard with `isinstance(..., str)` and
match trio's exact wording so a real child error can't be
demoted to `Cancelled` under cancellation.

Review: PR #464 (copilot-pull-request-reviewer)
https://github.com/goodboy/tractor/pull/464#pullrequestreview-4548203343

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-24 13:34:37 -04:00
Gud Boi a722d535d9 Fix dropped `for/else` re-raise in masking CM
`30e15925` ("Add `start_or_cancel()` to `trionics._taskc`")
inserted `async def start_or_cancel()` — whose body opens its
own col-4 `try:` — immediately before the trailing `else:
raise`. Because the edit was a pure insertion (0 deletions),
the *same* `else: raise` lines were silently REPARENTED: they
used to be the `for exc_match in matching: ... else: raise`
of `maybe_raise_from_masking_exc`, but now bind to
`start_or_cancel`'s `try/except` where they're unreachable
dead code.

Net effect: `maybe_raise_from_masking_exc` lost the `for/else`
re-raise of the un-masked exception, so a masked child
cancellation gets swallowed instead of surfaced.

- restore the `for/else: raise` to `maybe_raise_from_masking_exc`
- drop the now-dead `else: raise` from `start_or_cancel`

Surfaced as 2 deterministic failures in
`test_sigint_closes_lifetime_stack[wait_for_ctx-bg_aio_task-
send_SIGINT_to=child-*]` (the SIGINT-to-child "silent-abandon"
regime). Bisected with `trio` held at `0.29.0`: clean at
`9c36363b` (0/8), broken at `30e15925` (8/8), fixed (0/8).
NOT a `trio` (0.29↔0.33 identical) nor logging-plugin
regression.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 325574cc07)
(cherry picked from commit 0b8033fdaa)
2026-06-22 16:28:56 -04:00
Gud Boi 90466ec016 Add `start_or_cancel()` to `trionics._taskc`
Wrapper around `trio.Nursery.start()` that DOESN'T mask
out-of-band cancellation as a lossy startup failure.
Picks the right re-raise: ambient `Cancelled` when
present, the genuine startup-protocol `RuntimeError`
otherwise.

The problem,
- `trio.Nursery.start()` raises a generic
  `RuntimeError("child exited without calling
  task_status.started()")` whenever the started task
  exits BEFORE calling `task_status.started()` —
  INCLUDING the common case where the child was
  cancelled out-of-band by an *ancestor* cancel-scope
  erroring/cancelling.
- In that case the original `trio.Cancelled` is
  swallowed and the caller is left w/ an opaque,
  root-cause-detached `RuntimeError`.

The fix,
- Catch the "...started" RTE.
- `await trio.lowlevel.checkpoint_if_cancelled()` —
  re-raises the in-flight `Cancelled` IFF we're under
  effective cancellation (ancestor-inclusive), carrying
  trio's auto-generated reason which points at the true
  root exc.
- If we're NOT cancelled the `checkpoint_if_cancelled()`
  is a cheap no-op and we fall through to re-raise the
  genuine startup-protocol RTE.

Re-export from `tractor.trionics` so callers don't have
to reach into `_taskc`.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 30e15925ba)
(cherry picked from commit 2b4589b1ee)
2026-06-22 16:28:56 -04:00
Bd 2fdf3901cb
Merge pull request #462 from goodboy/subint_era_tooling
Subint era tooling
2026-06-19 08:58:12 -04:00
Gud Boi 084f0fc404 Give macOS CI extra headroom for cancel-cascade tests
The `cpu_scaling_factor()` CI bump (2x) wasn't enough for the
macOS runners — slower + noisier than linux for our multi-actor
cancel-cascade timing — so `test_nested_multierrors[depth=3]` and
`test_fast_graceful_cancel_*` flaked there (linux + sdist green),

- make the CI headroom platform-aware: 3x on macOS, 2x on linux
  (keeps the proven linux budget; depth=3 inner 12*3=36s still
  fits under the 40s outer wall).
- give `test_fast_graceful_cancel_*` the `cpu_scaling_factor()`
  headroom it was missing entirely (was a bare `timeout=2.9`).

Verified locally w/ `CI=1` (2x linux path; 3x macOS path confirmed
via forced `_non_linux`).

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-18 14:54:07 -04:00
Gud Boi c797bcb783 Address Copilot review nits on PR #462
All 5 flagged items were valid (4 real bugs + 1 dead assert),

- fix an inverted `sys.version_info < (3, 14)` guard in
  `ipc._linux` — the "`cffi` has no 3.14 support" import note now
  fires on 3.14+ (where it applies) instead of on older pys.
- use `os.environ.get('PYTHON_COLORS')` in the `sync_bp` example
  so it doesn't `KeyError` when run outside the test harness.
- correct `dump_task_tree()`'s docstring: the `/tmp` + `/dev/tty`
  tee is gated on `write_file`/`write_tty`, not "unconditional".
- tidy the `ActorTooSlowError` message spacing in `cancel_actor`.
- replace a tautological `applied is True or applied is False` in
  `test_patches` with `isinstance(applied, bool)` (the value is
  order-dependent across the module).

Review: PR #462 (copilot-pull-request-reviewer)
https://github.com/goodboy/tractor/pull/462#pullrequestreview-4527179852

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-18 13:37:55 -04:00
Gud Boi 7b518fe4e1 Make `cpu_scaling_factor()` CI-aware for timing tests
GH Actions (and most shared) CI runners are slow + noisy and —
unlike a throttled local box — don't expose CPU-freq scaling via
sysfs, so `cpu_scaling_factor()` read `1.0` and the timing-
sensitive deadlines/asserts that key off it got NO headroom on
CI (a class of `TooSlowError` / `assert diff < this_fast` flakes),

- add a flat `_ci_env` x2 bump inside `cpu_scaling_factor()` so
  every test already using it (quad streaming, SIGINT-cancel,
  docs examples, ...) gets CI headroom for free — compounds with
  any local-throttle factor.
- route the `time_quad_ex` cancel-deadline through it instead of
  a bespoke per-test `ci_env` bump.
- fix a real bug in `test_nested_multierrors`: its OUTER
  `@tractor_test(timeout=10)` was *smaller* than the inner
  `fail_after_w_trace` budget (trio depth=3 = 12s), so the outer
  wall fired first and pre-empted the snapshot-capturing inner
  deadline -> `FAILED` instead of dumping. Bump the outer to `40`
  (> max inner budget) and scale the inner budgets by
  `cpu_scaling_factor()` too.

Verified locally with `CI=1` (quad + both `test_nested_multierrors`
depths + `test_cancel_via_SIGINT_other_task` green).

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-18 12:03:30 -04:00
Gud Boi b0ac681245 Fix dead-code env-var override notice in `open_root_actor`
The `TRACTOR_LOGLEVEL`/`TRACTOR_SPAWN_METHOD` override-notice
branches were unreachable: `loglevel`/`start_method` were
reassigned to the env value BEFORE the `!=` compare, so the
"OVERRIDES caller-passed" message never fired. Capture the
caller value first, then compare. Rel. `208e7c09`/`d4eac06d`
"Honor env-vars" (`trionics.start_or_cancel`); surfaced by
`/code-review high` on #462.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-17 19:46:43 -04:00
Gud Boi fdac157d3d Harden cancel-ack hard-kill escalation
Two defensive fixes around the `Portal.cancel_actor()` +
`_try_cancel_then_kill()` escalation from `34f333a0`
"Escalate cancel-ack timeouts to `proc.terminate()`" (the
`trionics.start_or_cancel` follow-up); surfaced by
`/code-review high` on #462,

- guard `proc.terminate()` for backends whose `proc` slot
  isn't a `Process` — the future `subint` backend stores an
  `int` interp-id, so escalation would `AttributeError`
  instead of hard-killing; now it logs + no-ops.
- swap `assert cs.cancelled_caught` for an
  `if cs.cancelled_caught and raise_on_timeout:` guard so an
  unexpected shielded-scope exit returns a soft `False`
  rather than crashing `cancel_actor()` mid-teardown.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-17 19:46:04 -04:00
Gud Boi d28173f4c0 Make SIGUSR1 `stackscope` dumps actually work
Two fixes to the hang-debug SIGUSR1 task-tree dump path,
surfaced by `/code-review high` on #462,

- re-add `_debug_mode` to the sub-actor handler-install gate
  in `_runtime.py`. Dropping it (rel. `3a386ba5`/`3d9c75b6`
  "Drop debug_mode gate", from the `custom_log_levels_api`
  follow-up) was meant to *also* enable non-pdb runs, but
  nothing sets `use_stackscope` from `debug_mode`, so
  debug-mode subs were left with NO handler — and the default
  SIGUSR1 disposition then *kills* them. Now additive:
  `_debug_mode OR use_stackscope OR env`.
- pass `write_file=True` at both `dump_task_tree()` SIGUSR1
  call sites so the advertised `/tmp/tractor-stackscope-<pid>`
  `.log` tee is actually written (was dead under
  `--capture=fd`). Matches `1b1ef10a` "Re-enable writing
  `stackscope` to file by default"; param from `0df90500`.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-17 19:45:21 -04:00
Gud Boi f08a7d52b5 Drop stray `breakpoint()` in `RuntimeVars.__setattr__`
Left-over debug trap from the `_runtime_vars` pure get/set
refactor — it fired on *every* struct-form rt-var write (e.g.
via `.update()`), hanging any non-tty / CI / forked actor on
`pdb` stdin.

Surfaced by a `/code-review high` pass on #462.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-17 19:44:34 -04:00
Gud Boi 41b5371473 Relock `uv.lock` after rebase onto `main`
Regenerate the lockfile so it's consistent with the
post-rebase `pyproject.toml` — which now carries both #461's
landed tooling (`pytest>=9.0.3`, …) and this branch's
tractor deps (`setproctitle`, `pytest-timeout`, `psutil`),

- `uv lock` resolves the merged dep set against the landed
  `main` baseline.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-17 17:39:44 -04:00
Gud Boi 6d45581910 Add autouse fixture to reset `_runtime_vars` per-test
`open_root_actor()` writes `_enable_tpts` (and friends) into
the process-global `_state._runtime_vars` dict but nothing
resets it on actor teardown. Under the in-proc `pytest`
launchpad a uds-using test leaks `_enable_tpts=['uds']` into
a sibling tcp test, tripping the
`registry_addrs`×`enable_transports` proto-guard in
`open_root_actor()` with a `ValueError`.

New `_reset_runtime_vars` fixture snapshots + restores the
dict around every test so no runtime-var state crosses a
test boundary.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-17 17:39:44 -04:00
Gud Boi 6a7dea45bb Bump trio `echoserver` cancel timeout 1→4s
Same trio 0.29 → 0.33 cancel-cascade slowdown that hit
`test_nested_multierrors` (ea67f1b6) — bumps the
`trio`-backend (non-debug, non-forking) budget in
`test_echoserver_detailed_mechanics` from 1s → 4s.

- The 1s budget raced the ~1s teardown deadline. On a
  deadline-fire trio 0.33 injects
  `Cancelled(source='deadline')` (cancel-reason
  metadata) that wraps the mid-stream KBI in a
  `BaseExceptionGroup`, breaking the bare
  `pytest.raises(KeyboardInterrupt)` below.
- Bump matches the forking-spawner branch (4s).
- Inline NOTE references the tracking issue
  `ai/conc-anal/trio_033_cancel_cascade_slowdown_depth3_issue.md`.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit d7da502d93)
(cherry picked from commit d0144e52cb)
2026-06-17 17:39:44 -04:00
Gud Boi fa0208e65f Bump trio depth=3 cancel timeout 6→12s
trio 0.29 → 0.33 lock bump (c7741bba) slowed the
depth=3 cancel-cascade in `test_nested_multierrors`
from <6s to ~7-8s; the 6s deadline was firing and its
`Cancelled(source='deadline')` (trio 0.33's new
cancel-reason metadata) collapsed a BEG branch,
breaking the `RemoteActorError` assertion downstream.

- Split the `('trio', _)` case-match into per-depth
  arms: `('trio', 1)` keeps 6s (still finishes in
  ~3s); `('trio', 3)` → 12s.
- Updated inline NOTE explains the version pivot +
  links the tracking issue
  `ai/conc-anal/trio_033_cancel_cascade_slowdown_depth3_issue.md`.
- Existing MTF/`subint_forkserver` budgets unchanged.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit ea67f1b67b)
(cherry picked from commit 57b3ea59ea)
2026-06-17 17:39:44 -04:00
Gud Boi b6bf865f5e Fix `get_logger()` collapse of nested sub-pkgs
Strip the trailing `pkg_path` token ONLY when it duplicates the
caller's leaf-*module* name (which the console header already
shows via `{filename}`), instead of blindly dropping the last
token. This keeps genuine, possibly-*nested* sub-PACKAGE parts
addressable as their own sub-loggers.

- detect a true leaf-mod by comparing the caller's `__name__`
  vs `__package__` (a pkg `__init__` has them equal -> its
  trailing token is a real sub-pkg, NOT a leaf to strip).
- `name='devx.debug'` now -> `tractor.devx.debug`, DISTINCT
  from a bare `devx` -> `tractor.devx`; the old unconditional
  `pkg_path = subpkg_path` collapsed both to `tractor.devx` and
  silently broke per-sub-pkg level control via the logging-spec.
- `get_logger(__name__)` leaf-strip still works (cosmetic, bc
  the leaf-mod is in the `{filename}` header field).

Also,
- update the `LogSpec` caveat: sub-PACKAGE granularity now
  addressable at ANY depth; leaf *modules* intentionally aren't
  (they're the `{filename}`); top-level mods (eg. `to_asyncio`)
  still emit on the root logger.
- adjust `test_root_pkg_not_duplicated_in_logger_name` to the
  new literal explicit-`name` contract (no leaf-collapse).

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 9c36363b01)
2026-06-17 17:39:44 -04:00
Gud Boi 8bb9df9e06 Lift `--ll`/`--tl` to plugin + `LogSpec` API
Two coupled changes that let downstream projects (eg. `modden`) inherit
the test-harness loglevel plumbing for free via
`tractor._testing.pytest`:

Plugin lift (`tests/conftest.py` → `_testing/pytest.py`),
- mv `pytest_addoption(--ll)`, the `loglevel` autouse
  fixture, and `test_log` fixture out of the test-suite-
  local conftest into the reusable plugin.
- add `--tl`/`--tractor-loglevel` as a DISTINCT flag from
  `--ll`: `--ll` is the consuming-project's OWN app
  loglevel (scoped to its pkg-hierarchy), `--tl` is the
  `tractor.*` runtime loglevel. `--tl` falls back to
  `--ll` when unset (preserves current `tractor`-suite
  behavior).
- add `testing_pkg_name` session fixture (default
  `'tractor'`) — downstream projects override to e.g.
  `'modden'` so `--ll` scopes to their own hierarchy
  instead of `tractor.*`.
- `loglevel` fixture now yields the resolved
  tractor-runtime level (passed to
  `open_root_actor(loglevel=<.>)` by `@tractor_test`)
  AND separately applies `--ll` to the
  `testing_pkg_name` hierarchy when that isn't
  `tractor`. `test_log` scopes the per-test logger to
  `testing_pkg_name`.

`tractor.log` "logging-spec" mini-DSL,
- `LogSpec = str|bool`. Accepted forms:
  - `True` → enable `pkg_name` root at `default_level`
    (fallback `'cancel'`).
  - `False` → no-op.
  - bare level eg. `'info'` → root-logger at that level.
  - `'sub:info,x:cancel'` → per-sub-logger filter-spec;
    each `<name>` is RELATIVE to `pkg_name` (must NOT
    include the pkg-token).
- `parse_logspec()` → `{sublog|None: level}` mapping.
  `None` key = root-logger. Mixed bare-level + filters
  in one spec is rejected w/ a helpful err msg; so is
  embedding the `pkg_name` token in a sub-name.
- `apply_logspec()` → `(primary_level, {name: log})`:
  parses then enables a `colorlog` stderr handler per
  named (sub)logger. Authoritative sub-logger filters
  get `propagate=False` so they don't double-emit
  through a parallel root-level handler.
- !GRANULARITY CAVEAT! sub-logger names match at
  sub-pkg granularity, not leaf-module — so `devx.debug`
  collapses to the same `tractor.devx` logger as a bare
  `devx`, and top-level lib modules (eg.
  `tractor.to_asyncio`) emit under the *root* logger
  rather than a phantom `to_asyncio` child. Documented
  inline on `LogSpec`.

Other,
- `tests/conftest.py` keeps a NOTE pointing to the
  plugin for future-debugging clarity (don't remove
  silently — the lift is the relevant signal).

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 19a77708ba)
2026-06-17 17:39:44 -04:00
Gud Boi 944262d8a6 Add `maybe_signal_aio_task()` + cause-chain guard
Factor the "deliver an exc to a running aio task" pattern out of
`translate_aio_errors()` + `open_channel_from()` into a shared
`maybe_signal_aio_task()` helper. Add a cause-chain matrix comment
+ relay-echo guard so the final-raise block can't cycle
  `trio_err.__cause__` back onto its own derivative relay.

`maybe_signal_aio_task()`,
- Delivers `exc` via `aio_task._fut_waiter.set_exception()` — NOT
  `aio_task.set_exception()` which on py3.13+ ALWAYS raises
  `RuntimeError("Task does not support set_exception")` (dead code as
  a relay mechanism).
- Returns `(delivered: bool, report: str)`. Caller uses `delivered` to
  flip `wait_on_aio_task` when delivery failed (avoids hanging on
  `_aio_task_complete.wait()`).
- `pre_captured_fut=`: required when the caller crosses a trio
  checkpoint between capturing `_fut_waiter` and invoking the helper.
  `Task._wakeup` clears `_fut_waiter = None` so re-reading
  post-checkpoint loses the ref even though the exc is still in-flight
  on the (now-`done()`) original fut.
- `cause=`: sets `exc.__cause__ = cause` so the relay carries
  a "trio_err -> caused -> relay" chain through `set_exception()`
  → `Task._wakeup` → coro raise → `wait_on_coro_final_result`
  → `signal_trio_when_done` → `task.result()`-raise.
- `allow_cancel_fallback=True`: opt-in `aio_task.cancel()` for the
  narrow case where `_fut_waiter is None` AND task is runnable (sitting
  in asyncio's ready queue, not parked on a poke-able future). NEVER
  cancels when `_fut_waiter` carries an in-flight exc — that would race
  + mask the real terminating exc.

`translate_aio_errors()`,
- Replace the two ad-hoc `_fut_waiter.set_exception()`
  / `aio_task.set_exception()` call sites w/ the helper.
- Capture `pre_cp_fut = aio_task._fut_waiter` BEFORE the post-shutdown
  `trio.lowlevel.checkpoint()` (critical: `_wakeup` clears the ref).
- New "cross-loop cause-chain matrix" comment block on the final-raise
  — tabulates every `(trio_err, aio_err, trio_to_raise)` combo into
  exactly one terminal `raise X [from Y]` or early `return`. Covers the
  sibling `signal_trio_when_done()` resolution + the relay-echo
  INVARIANT.
- New relay-echo guard: if `aio_err` is one of OUR OWN signals
  (`TrioTaskExited`/`TrioCancelled`) AND `aio_err.__cause__ is
  trio_err`, raise the bare `trio_err` instead of `trio_err from
  aio_err` (which would CYCLE the cause chain since the relay was itself
  caused-by `trio_err`).
- Drop the stale "the `task.set_exception(aio_taskc)` call MUST NOT
  EXCEPT or this WILL HANG" warning — the helper handles the failure
  path explicitly via `delivered=False` → `wait_on_aio_task = False`.
- Carry `cause=trio_err` on both the cancel-relay (`TrioCancelled`) and
  the graceful-exit relay (`TrioTaskExited`) so the aio-side traceback
  shows the real root.

`open_channel_from()`,
- Adopt the same helper; drop the dead "SHOULD NEVER GET HERE !?!?"
  + `tractor.pause(shield=True)` panic branch.
- Capture in-flight trio-side exc via `sys.exc_info()[1]` and pass as
  `cause=` — non-`None` only when the `try` body raised (graceful exit
  → None).

Other,
- Top-level import: `sys` (for `sys.exc_info()`).
- `run_as_asyncio_guest()`: add commented-out alt `out: Outcome = await
  trio_done_fute` next to the shielded version — exploratory note for
  the longstanding "why is `.shield()` needed?" TODO.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit acd1cbeec4)
2026-06-17 17:39:44 -04:00
Gud Boi b57d095204 Drop `debug_mode` gate on stackscope SIGUSR1
SIGUSR1 task-tree dumps via `stackscope` should work in
plain (non-pdb) runs too — esp. in infected-`asyncio`
processes where the kernel-default SIGUSR1 disposition is
`Term` (proc dies on `kill -USR1` w/o an installed
handler). Ungate the install path from `_debug_mode` in
both root and sub-actor init; the `use_stackscope` rt-var
+ `TRACTOR_ENABLE_STACKSCOPE` env-var checks remain as
the actual opt-in (e.g. via `--enable-stackscope`).

Deats,
- `_root.open_root_actor`: drop the `debug_mode and ...`
  conjunction around the `enable_stack_on_sig()` call;
  now gated only on the `enable_stack_on_sig` arg itself.
- `_runtime.Actor` sub-actor init: lift the
  `use_stackscope`/`TRACTOR_ENABLE_STACKSCOPE` branch out
  of the `if rvs['_debug_mode']:` block to peer-level.
  The `use_greenback` branch stays inside `_debug_mode`
  (pdb-specific).
- Refresh inline comments on both sites to call out the
  infected-`asyncio` "default SIGUSR1 = terminate proc"
  rationale.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 3d9c75b6ed)
2026-06-17 17:39:44 -04:00
Gud Boi 4b70b564c8 Use trace CM helpers in `test_infected_asyncio`
Adopt the `_testing.trace` CM helpers in two MTF-hang-prone
tests so on-timeout we get a fresh
`ptree`/`wchan`/`py-spy` diag snapshot on disk instead of
opaque pytest timeout-kills. Same shape as bd07a95d for
`test_dynamic_pub_sub`.

Deats,
- `test_echoserver_detailed_mechanics`:
  * inner `trio.fail_after` → `fail_after_w_trace`. Adds
    `fail_after_w_trace: FailAfterWTraceFactory` fixture
    param.
  * mv per-backend `timeout` calc to top of test body (was
    interleaved w/ helper defs).
  * factor deep
    `open_nursery`/`open_context`/`open_stream` body into
    `_body()` so the wrapping `main()` stays a 2-liner —
    keeps the nested-CM block at its natural indent level
    instead of pushing it under yet another `async with`.
  * drop `with_timeout: bool` knob + `fa_main()` helper
    (knob was hard-coded `True`).
- `test_sigint_closes_lifetime_stack`:
  * outer `signal.alarm`/`try`/`finally` → single
    `afk_alarm_w_trace(10)` CM. Adds
    `afk_alarm_w_trace: AfkAlarmWTraceFactory` fixture
    param.
  * drop `_AFK_CAP_S` + `armed_alarm` vars (CM owns both).
  * explanatory comment refreshed to mention
    `AFKAlarmTimeout` + the disk-snapshot side effect.

Other,
- Drop debug `return 1e3` short-circuit from `delay()`
  fixture — snuck in as a scratch line, was clobbering the
  proper `debug_mode`-branched return.
- Top-level import: `FailAfterWTraceFactory`,
  `AfkAlarmWTraceFactory` from `tractor._testing.trace`.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 1cafaecf52)
2026-06-17 17:39:44 -04:00
Gud Boi da582a4d1b Add `acli.watch` flicker-free alias-loop
Per-terminal optimized `watch`-like xonsh alias that
runs an arbitrary callable alias in a loop inside the
alt-screen buffer with flicker-free repaint. Supersedes
the inline `acli.ptree` polling .xsh snippet (removed
from `_ptree` docstr in favor of
`acli.watch acli.ptree pytest`).

Deats,
- alt-screen entry/exit (`\033[?1049h/l`) + cursor-hide
  (`\033[?25l/h`) wrapped in try/finally so Ctrl-C always
  returns to a pristine shell.
- per-frame draw uses cursor-home (`\033[H`) + per-line
  EL (`\033[K` before each `\n`) + post-draw erase-down
  (`\033[J`) → stale tail chars from a longer prior
  frame are obvi cleared; no full-screen flash.
- SIGWINCH-aware: terminal resize sets a flag, next
  frame does a full clear (`\033[H\033[2J`) instead of
  the cheap cursor-home path.
- Ctrl-C handling: install `signal.default_int_handler`
  so `KeyboardInterrupt` lands cleanly; prior handler
  restored on exit.
- Output capture: redirect the alias's stdout to
  `StringIO` per frame so we can post-process the EL
  fix. Aliases writing directly to `sys.stdout.buffer`
  / `os.write(1)` bypass capture — EL-fix won't apply
  but loop still works.
- Alias unwrap: xonsh stores callables as either a bare
  callable OR `[fn, *preset_args]`. Both forms handled;
  subprocess-style aliases rejected w/ a friendly err
  msg.
- `argparse` w/ `-n`/`--interval` (default 0.3s); rest
  of argv forwarded as alias args.
- Reg `'acli.watch': watch` in `_TCLI_ALIASES`.

Other,
- Tn `_ptree` `args: list[str]` param.
- Mod-header `Provides:` block updated w/ `acli.watch`
  entry.
- Top-level imports: `os`, `sys`, `signal`, `time`,
  `typing.Callable`.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit bb239e847f)
2026-06-17 17:39:44 -04:00
Gud Boi 30242d03fb Add `acli.ptree` poll .xsh snippet to docstr
(cherry picked from commit f617c8cb73)
2026-06-17 17:39:44 -04:00
Gud Boi fad1227d7c Filter `_find_tractor_strays` by ppid disposition
Only flag `tractor._child` procs as cross-test ghosts of
THIS run if `ppid==1` (init-adopted real leak) or `ppid`
is in the walk's `seen` set (descendant we missed via
race).

Previously, procs whose `ppid` points to some OTHER live non-`pytest`
(in the use of `acli.ptree pytest`) process belong to a different
tractor app (`piker`, another `pytest` shell, a long-running tractor
daemon) and were being falsely flagged as cross-test ghosts.

Deats,
- post-cmdline-match check via `_ppid_from_proc(pid)`,
  short-circuit on `None` (proc died in-flight).
- expand module docstring to spell out the ownership
  filter rule + its rationale.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit a6d4ac3aac)
2026-06-17 17:39:44 -04:00
Gud Boi c8a77fb92b Use trace CM helpers in `test_dynamic_pub_sub`
Replace inline `trio.fail_after` + manual `signal.alarm` guard with the
`_testing.trace` CM helpers that auto-capture a full ptree/wchan/py-spy
diag snapshot to disk on timeout.

Deats,
- inner guard: `trio.fail_after` → `fail_after_w_trace` (async CM,
  captures on `TooSlowError`).
- outer AFK guard: raw `signal.alarm` → `afk_alarm_w_trace` (sync
  CM, captures on `SIGALRM`), only armed under fork backends.
  Extracts `_run_and_match()` helper to keep branching clean.
- bump `fail_after_s` from 4/12 → 8/20 to stop borderline flakes
  while diag harness accumulates evidence.
- drop `_DIAG_CAP_S` var + manual signal import (now internal to
  `afk_alarm_w_trace`).

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit bd07a95d80)
2026-06-17 17:39:44 -04:00
Gud Boi 68698afac7 Harden `test_cancellation` for fork-spawner backends
Deats,
- `pytestmark`: enrich `skipon_spawn_backend('subint')` reason with
  conc-anal doc refs + GH#379 link, add `reap_subactors_per_test`,
  `track_orphaned_uds_per_test`,
  `detect_runaway_subactors_per_test` fixtures
- `test_nested_multierrors`: parametrize over `depth` `{1, 3}`, add
  MTF `xfail(strict=False)` with detailed race-window comment
  explaining the BEG shape mismatch, wrap body in
  `fail_after_w_trace` with per-backend timeout budget, bump
  `@tractor_test(timeout=10)`, drop old multiprocessing depth
  special-casing
- `test_multierror_fast_nursery`: wrap in
  `fail_after_w_trace(30.0)`, accept `TooSlowError` in
  `pytest.raises`, surface explicit `pytest.fail` on hang
- `test_cancel_while_childs_child_in_sync_sleep`: swap
  `spawn_backend` param for `is_forking_spawner`, widen
  `fail_after` delay for fork-based spawners
- `test_remote_error`, `test_multierror`,
  `test_cancel_infinite_streamer`, `test_some_cancels_all`: add
  `set_fork_aware_capture` fixture param
- Drop commented-out per-test `skipon_spawn_backend` blocks (now
  covered by module-level `pytestmark`)

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 32955db02e)
2026-06-17 17:39:44 -04:00
Gud Boi 98522661d6 Add init-adopted orphan reap to `reap_subactors_per_test`
Post-yield now also reaps init-adopted (`ppid==1`) tractor procs
that appeared during the test — leaked subactors whose mid-tier
parent died during cascade teardown, reparenting them to init.
Pre-yield snapshot of existing orphans scopes reap to THIS test's
leaks only, avoiding reap of unrelated tractor uses (piker, etc.)
on the box.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 01ce2857ea)
2026-06-17 17:39:44 -04:00
Gud Boi dfe853e159 Add subtree-walk to `reap()` for full actor-tree teardown
`reap(include_descendants=True)` now expands each orphan-root pid
into its full psutil subtree before delivering SIGINT, so a
multi-level leaked actor-tree gets torn down in a single pass
instead of requiring repeated calls (each pass kills the current
`ppid==1` level, the level below becomes init-adopted, etc.).

Falls back to the original flat `pids` list when `psutil` is
unavailable. Emits a log line when expansion adds descendant pids.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 8de684f5de)
2026-06-17 17:39:44 -04:00
Gud Boi f0a1971814 Add hang-snapshot session index to pytest summary
- `_testing/trace.py`: add `_SNAPSHOT_INDEX` session- scoped list
  populated by `_do_capture_snapshot()` on each successful dump;
  add TODO for future `TRACTOR_TRACE_HOLD=1` pause-on-hang mode
- `_testing/pytest.py`: add `pytest_terminal_summary` hook that
  prints all captured snapshot dirs at end-of-session so paths
  don't get buried in scrollback

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit fb87c36263)
2026-06-17 17:39:44 -04:00
Gud Boi 46c1147f6e Add stray-proc scan + refine `_testing.trace` capture
Deats,
- `_find_tractor_strays()`: scan `/proc/*/cmdline` for
  `tractor._child` procs NOT in the walk's `seen` set — surfaces
  ghost subactor trees from prior test runs (cross-test launchpad
  contamination).
- `dump_proc_tree(include_strays=True)`: refactor classification
  into `_classify_walk()` closure, walk stray roots as additional
  trees, emit stray-root summary in header. Also: `tractor._child`
  procs reparented to init are now always classified as orphans
  regardless of cgroup-slice (leaked subactor ≠ desktop-launched
  app).
- `_do_capture_snapshot()`: use `sys.__stderr__` to bypass pytest
  `--capture=sys` redirection so snapshot paths always land on the
  real terminal
- `fail_after_w_trace()`: capture diag snapshot on
  non-`TooSlowError` exceptions when the `fail_after` scope's
  cancel had already fired (e.g. nursery wraps `Cancelled` into a
  `BaseExceptionGroup` that escapes before `TooSlowError` can be
  raised).

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 3a243a1fd4)
2026-06-17 17:39:44 -04:00
Gud Boi da2d5bb3d2 Mv core impl `tractor_diag.xsh` to `_testing.trace`
Extract all pure-Python diagnostic helpers (`dump_proc_tree`,
`dump_hung_state`, `scan_bindspace`, `dump_all`, `resolve_pids`,
`ensure_sudo_cached`, etc.) from the xonsh xontrib into a new
`tractor/_testing/trace.py` module so the same logic is callable
from both the `acli.*` terminal aliases AND in-test capture-on-hang
fixtures.

Deats,
- `_testing/trace.py`: new module (1171 lines) — proc-tree walker,
  hung-state dumper, bindspace scanner, `dump_all()` snapshot
  archiver, `AFKAlarmTimeout` exc, `fail_after_w_trace()` async CM
  (trio `fail_after` + auto-snapshot on `TooSlowError`),
  `afk_alarm_w_trace()` sync CM (`signal.alarm` + snapshot on
  `SIGALRM`), plus pytest fixture wrappers for both.
- `_testing/pytest.py`: re-export the two fixtures via `from .trace
  import` so pytest plugin-discovery picks them up.
- `tractor_diag.xsh`: thin terminal wrappers that import from
  `_testing.trace` — drops ~627 lines of inline impl. Add
  `acli.dump_all` alias for full snapshot-bundle CLI access.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 7509e313ff)
2026-06-17 17:39:44 -04:00
Gud Boi 36ad56843e Harden `test_infected_asyncio` for fork spawners
Deats,
- `test_echoserver_detailed_mechanics`: add `is_forking_spawner`
  param, wrap `main()` in `fa_main()` with per-backend
  `trio.fail_after` (4s fork / 1s trio) to cap cancel-cascade
  teardown that compounds under forkserver.
- `test_sigint_closes_lifetime_stack`: swap `start_method` param
  for `is_forking_spawner`, pre-init `tmp_file`/`ctx` to `None` so
  KBI firing before `open_context` body doesn't `UnboundLocalError`,
  add `pytest.fail` guard for the spawn-time IPC race case, arm
  `signal.alarm` AFK-safety cap (10s) under fork backends

Also,
- `pytestmark`: add `track_orphaned_uds_per_test` +
  `detect_runaway_subactors_per_test` fixtures.
- `delay()`: hardcode `return 1e3` at top (debug override still in
  place).

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 7ee0dc2e8f)
2026-06-17 17:39:44 -04:00
Gud Boi acb042ec77 Adjust `test_streaming_to_actor_cluster` timeout
For forking spawner backends that is.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit b10011a36e)
2026-06-17 17:39:44 -04:00
Gud Boi 75c87831cb Enrich `pytestmark` in `test_inter_peer_cancellation`
- `skipon_spawn_backend('subint')`: expand reason with specific
  analysis doc refs + GH issue #379 umbrella link.
- add `track_orphaned_uds_per_test` fixture via `usefixtures` to
  blame-attribute UDS sock-file orphans left by SIGKILL cancel
  cascades.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 7d0a53d205)
2026-06-17 17:39:44 -04:00
Gud Boi 105f0c2944 Adjust `test_simple_context` timeout for forking spawner
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 75d5b4cf7b)
2026-06-17 17:39:44 -04:00
Gud Boi 48a0f2fc49 Add `set_fork_aware_capture`, timeout to msg tests
- `test_ext_types_over_ipc`: wrap `main()` in `fa_main()` with
  `trio.fail_after(2)` + commented `capfd.disabled()` investigation
  (pytest#14444).
- `test_basic_payload_spec`: add fixture param with note on fork-spawner
  hang prevention.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 8aa07a7932)
2026-06-17 17:39:44 -04:00
Gud Boi d70e003184 Add signal-alarm guard to `test_dynamic_pub_sub`
Outer `signal.alarm` cap that fires even when trio's
`fail_after` is blocked by a shielded-await deadlock
(the bug-class-3 hang under MTF backends). Only armed
for fork-based spawners where the bug lives.

Deats,
- `_DIAG_CAP_S = fail_after_s + 5` — slightly larger than the
  trio-native guard so it always loses when the in-band path works.
- `test_log.cancel()` breadcrumbs at each cancel-scope boundary so the
  last-fired breadcrumb names the swallow point on hang.
- try/finally wrapping around each scope level for deterministic
  breadcrumb emission.
- add `is_forking_spawner`, `set_fork_aware_capture` fixture params.
- rework `fail_after_s`: 4s for fork, 12s for trio (was 30/12).

Also,
- `test_sigint_both_stream_types`: `assert 0` -> `pytest.fail()`, add
  TODO re `pytest.raises()`.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 10db117864)
2026-06-17 17:39:44 -04:00
Gud Boi aced458350 Fix `is_forking_spawner` fixture to call helper fn
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 83b6a3373a)
2026-06-17 17:39:44 -04:00
Gud Boi 42881b3d38 Add ppid-aware liveness buckets to `bindspace_scan`
Split the old `live`/`orphans` sock classification
into three ppid-aware buckets: `live-active` (PID
alive, parent owns it), `orphaned-alive` (PID alive
but `ppid==1`, init-adopted — `acli.reap` candidate),
and `orphaned-dead` (PID gone, sock stale).

Deats,
- new `_ppid()` helper reads `/proc/<pid>/stat` field [3] for parent
  PID, handles the tricky `(comm)` field (can contain spaces/parens) by
  splitting from last `)`.
- live-active rows now show `(ppid=<N>)` for ctx.
- orphaned-alive rows flagged `(adopted by init)`.
- cleanup suggestion: `acli.reap --uds` for both
  alive-orphan graceful cancel + dead-sock cleanup
  in one shot; manual `rm` kept as fallback.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 9bbb6f796b)
2026-06-17 17:39:44 -04:00
Gud Boi f5940de5c0 Add boot-race conc-anal, widen `xfail` to `n_dups=8`
New `ai/conc-anal/spawn_time_boot_death_dup_name_issue.md`
documenting the spawn-time rc=2 race under rapid
same-name spawning against a forkserver + registrar
— the `wait_for_peer_or_proc_death` helper now surfaces
the death instead of parking forever on the handshake
wait.

Also,
- extract inline `xfail` into module-level
  `_DOGGY_BOOT_RACE_XFAIL` marker.
- apply it to `n_dups=8` too (previously bare) bc
  larger N widens the race window enough to fire
  occasionally.
- link to tracking issue #456.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 92443dc4ef)
2026-06-17 17:39:44 -04:00
Gud Boi 9f9a536fa9 Adjust legacy streaming test timeouts for fork+UDS
Forking spawner + UDS transport has different timing
vs `trio_proc` — streaming example completes faster
in some cases, slower in others depending on fork
overhead + sock setup.

Deats,
- add `expect_cancel` param to `cancel_after()`, raise
  `ActorTooSlowError` when cancel scope fires unexpectedly instead of
  silently returning `None`.
- `time_quad_ex` fixture: bump timeout +1 for forking+UDS, explicit
  `ActorTooSlowError` on `None` result instead of bare `assert results`.
- `test_not_fast_enough_quad`: `xfail` for forking+UDS being "too fast"
  (cancel doesn't fire bc streaming finishes before delay).
- add `is_forking_spawner`, `tpt_proto` fixture params throughout.

Also,
- `_testing/pytest.py`: widen `start_method` parametrize and
  `is_forking_spawner` fixture to `scope='session'`.
- `"""` -> `'''` docstring style throughout.
- hoist `_non_linux` to module scope (was redefined locally in two
  places).
- type hints, kwarg-style `partial()` calls.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit d3cbc92751)
2026-06-17 17:39:44 -04:00
Gud Boi f0b944f8db Add bare-name arg, `ss` hints to `bindspace_scan`
`acli.bindspace_scan piker` now resolves `<name>` to
`$XDG_RUNTIME_DIR/<name>` — useful for projects like
`piker` that bind sibling sub-dirs alongside tractor's
default. Full paths still work as-is.

Also,
- rename "unparseable" section to "non-tractor" with
  clearer desc (filename lacks `@<pid>` suffix)
- print per-sock `ss -lpx 'src = <path>'` cmds for
  non-tractor socks so callers can manually resolve
  listener-PID liveness

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 099104e0af)
2026-06-17 17:39:44 -04:00
Gud Boi 5f2e89ed1f Harden `test_registrar` with reap fixtures, timeouts
Add module-level `pytestmark` applying per-test
`reap_subactors_per_test`, `track_orphaned_uds_per_test`, and
`detect_runaway_subactors_per_test` fixtures — registrar tests stress
discovery roundtrips that historically left orphaned UDS sock-files.

Deats,
- drop unused `say_hello()` fn, keep only `say_hello_use_wait`;
  rename param `func` -> `ria_fn`.
- use `@tractor_test(timeout=7)` instead of separate
  `@pytest.mark.timeout(7, method='thread')` decorator.
- add `with_timeout()` helper, wire into
  `test_subactors_unregister_on_cancel_remote_daemon`.
- uncomment `_timeout_main()` in `test_stale_entry_is_deleted`, use
  configurable `timeout` var + `debug_mode` guard for `tractor.pause()`
  on cancel.
- `dump_on_hang(seconds=timeout*2)` instead of hardcoded `20`.
- fix typo "oustanding" -> "outstanding".

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit abd3950ba6)
2026-06-17 17:39:44 -04:00
Gud Boi 3c6ea8009b Add `_is_tractor_subactor()`, cgroup-aware `ptree`
Rework reap/diag tooling to identify tractor sub-actors via
intrinsic proc signals — cmdline/comm markers from `setproctitle` —
instead of env-var or cwd matching.

Deats,
- new `_is_tractor_subactor()` checks cmdline for `tractor[` /
  `tractor._child` markers, falls back to `/proc/<pid>/comm` for
  zombie-resilient detection (kernel preserves `comm` past exit
  until reap)
- `_read_comm()` reads kernel per-task name set by `setproctitle()`
  — the zombie-safe ID signal
- `_read_status_state()` reads single-letter proc state from
  `/proc/<pid>/status` (`Z` = zombie)
- `find_orphans()` drops `repo_root` requirement, uses
  `_is_tractor_subactor()` for intrinsic sub-actor ID instead of
  cwd coincidence-matching
- new `find_zombies()` with optional `parent_pid` filter for
  zombie-state sub-actors

Also,
- rename `pytree` -> `ptree` throughout xontrib
- add `_which_cgroup_slice()` — reads `/proc/<pid>/cgroup` to
  distinguish `system.slice` services vs `user.slice` desktop apps
  from genuinely leaked orphans
- `_ptree` classifies `ppid==1` procs into `system-slice`,
  `user-slice`, and `orphans` buckets with per-section output
- `_tractor_reap` drops `git rev-parse` / `sys.path` hack — assumes
  tractor importable from active venv

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 522b57570b)
2026-06-17 17:39:44 -04:00
Gud Boi 338f0a1463 Add per-actor `setproctitle` via `devx._proctitle`
New `tractor.devx._proctitle` mod sets each
sub-actor's `argv[0]` (and kernel `comm`) to
`tractor[<aid.reprol()>]` — e.g.
`tractor[doggy@1027301b]` — so `ps`/`top`/`htop`
and `acli.pytree`/reaper tooling can identify
actors at a glance without parsing full cmdlines.

Deats,
- `set_actor_proctitle()` wraps the `setproctitle`
  pkg with `ImportError` guard; optional at runtime
  but listed in `pyproject.toml` so default installs
  benefit.
- called early in `_child._actor_child_main()` after
  `Actor` construction, before `_trio_main()` entry.
- tests in `tests/devx/test_proctitle.py`: format
  unit test, `/proc/{cmdline,comm}` integration
  test, negative detection test.

Resolves #457

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit d60245777e)
2026-06-17 17:39:44 -04:00
Gud Boi 6b14c3dbe7 Add dup-name cancel-cascade escalation test
Extend `test_register_duplicate_name` w/ cancel-level log
breadcrumbs and `try/finally` for better diag on the cancel-cascade
hang.

Add `test_dup_name_cancel_cascade_escalates_to_hard_kill` as a
regression test for the TCP+MTF duplicate-name cancel-cascade
deadlock. Spawns N same-name actors, calls `an.cancel()`, and
asserts teardown completes within a `trio.fail_after()` budget that
scales w/ `n_dups`.

Deats,
- parametrize `n_dups` (2, 4, 8) to widen the race window for
  concurrent `register_actor` RPCs.
- `n_dups=4` xfail'd — exposes a separate boot-race bug (doggy
  `rc=2` under rapid same-name spawn), tracked in #456.
- post-teardown asserts all `Portal` chans disconnect, verifying
  hard-kill escalation worked.

Relates to https://github.com/goodboy/tractor/issues/456

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit caebf60f4e)
2026-06-17 17:39:44 -04:00
Gud Boi 2cd668dcc8 Add `acli.reap`, namespace `tractor_diag` cmds
Group all xontrib aliases under an `acli.` prefix
so xonsh prefix-completion treats them as a sub-cmd
group — `acli.<TAB>` lists the full set. No parent
`acli` cmd exists; the dot is purely naming.

Renames (incl `-` -> `_` in suffixes for shell-
identifier-friendliness):

  - `pytree`         -> `acli.pytree`
  - `hung-dump`      -> `acli.hung_dump`
  - `bindspace-scan` -> `acli.bindspace_scan`

Add new `acli.reap` wrapping `scripts/tractor-reap`:

Deats,
- 3 opt-in phases via flags:

  1. process reap — `find_orphans()` (default,
     PPid=1 + cwd=repo + cmdline `python`) or
     `find_descendants(--parent PID)`. SIGINT
     first, SIGKILL after `--grace` (def 3.0s).

  2. `/dev/shm` sweep (`--shm`/`--shm-only`) —
     `find_orphaned_shm()` + `reap_shm()`. needed
     bc `tractor` disables `mp.resource_tracker`.

  3. UDS sock-file sweep (`--uds`/`--uds-only`) —
     `find_orphaned_uds()` + `reap_uds()` for stale
     `${XDG_RUNTIME_DIR}/tractor/<name>@<pid>.sock`
     entries. See #452.

- `--dry-run` lists matches without signalling/
  unlinking; survivor pids or sweep errors flip
  the alias rc to `1`.
- lazy-imports `tractor._testing._reap` after
  `git rev-parse --show-toplevel` (with
  `Path(__file__).parent.parent` fallback) so the
  contrib is loadable before the venv is on
  `sys.path`.
- `argparse.SystemExit` on `-h`/bad-args is
  caught + returned as the alias rc instead of
  killing xonsh.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit cec6cc2a56)
2026-06-17 17:39:44 -04:00
Gud Boi 9426fea5bd Escalate cancel-ack timeouts to `proc.terminate()`
Wires SC-discipline cancel-then-escalate into
`ActorNursery.cancel()`:

  graceful cancel-req -> bounded wait -> hard-kill

Deats,
- add `raise_on_timeout: bool = False` kwarg to `Portal.cancel_actor()`.
  When `True`, bounded- wait expiry raises `ActorTooSlowError` instead
  of the legacy DEBUG-log + return-`False` path. Default stays `False`
  for callers that handle their own escalation (e.g.
  `_spawn.soft_kill()` polling `proc.poll()`).

- add `_try_cancel_then_kill()` helper in `_supervise` used by per-child
  cancel tasks. On `ActorTooSlowError`, escalates via `proc.terminate()`
  (SIGTERM) so a non-acking sub doesn't park `soft_kill()` forever
  waiting on `proc.poll()`.

- replace `tn.start_soon(portal.cancel_actor)` in
  `ActorNursery.cancel()` with the helper.

Debug-mode bypass:
-----------------
skip escalation (fall back to legacy fire-and-forget cancel) when ANY
of:
- `Lock.ctx_in_debug is not None` (some actor is currently
  REPL-locked)
- `_runtime_vars['_debug_mode']` (root opened with `debug_mode=True`).
- `ActorNursery._at_least_one_child_in_debug` (per-child `debug_mode=`
  opt-in).

ORing covers root-debug, child-debug, and active- REPL-lock cases
without false-positively SIGTERM- ing a sub-tree proxying stdio for
a REPL session.

Motivated by the `subint_forkserver` dup-name hang where a same-named
sibling subactor's cancel-RPC failed to ack within
`Portal.cancel_timeout` (TCP+ forkserver register-RPC contention) and
the nursery `__aexit__` deadlocked.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 34f333a026)
2026-06-17 17:39:44 -04:00
Gud Boi e9b63e8569 Add `ActorTooSlowError` for cancel-cascade timeouts
Distinct from `trio.TooSlowError` so that existing
`except trio.TooSlowError:` blocks don't silently
mask actor-cancel timeouts — these must propagate
to let a supervisor escalate to
`proc.terminate()` per SC-discipline:

  graceful cancel-req -> bounded wait -> hard-kill

Motivated by #subint_forkserver dup-name hang
where `Portal.cancel_actor()` silently swallowed
the timeout and the supervisor never escalated,
leaving a same-named sibling subactor parked
forever.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 38ffb875bd)
2026-06-17 17:39:44 -04:00
Gud Boi 4c600dc528 Tidy proto-guard `ValueError` fmt in `open_root_actor()`
Pre-compute `mismatch_lines` str instead of `+`-concat
inside the f-string raise site; slightly easier to read
and avoids the `+ '\n\n'` continuation.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 5cd06810db)
2026-06-17 17:39:44 -04:00
Gud Boi 254a46c345 Mk `--capture` guard CI-aware w/ local warn
Refactor `pytest_load_initial_conftests()` to split
the fork-spawn × capture-mode check into two policies:

- CI (`CI` env-var set): `pytest.exit(rc=2)` on
  mismatch — forces every matrix-row to declare
  `--capture=sys` explicitly.
- local: `warnings.warn()` + continue — lets devs
  experiment with `--capture=fd` to validate fixes.

Deats,
- drop `_cap_fd_set` global; add
  `_CAPSYS_REQUIRED_SPAWNERS` frozenset for the
  spawner-name lookup
- move inline comment wall → proper docstring w/
  Background, Trade-off, Validation-policy sections
- `maybe_xfail_for_spawner()` now takes
  `request: pytest.FixtureRequest` and reads
  `request.config.option.capture` instead of the
  `_cap_sys_passed_as_flag` global
- recognize `tee-sys` as fork-safe (only `fd`-level
  capture deadlocks)
- `set_fork_aware_capture()` returns the actual
  capture mode str from config, not a hardcoded
  `'sys'`
- lift `import warnings` to module level (was duped
  inside `pytest_configure`)

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 255c9c3a7c)
2026-06-17 17:39:44 -04:00
Gud Boi f500bebbe6 Add `--tree` flag and cross-bucket parent annos to `pytree`
Extend `pytree` with two usability improvements:

- `--tree`/`-t` opt-in flag emits a flat walk-order `## tree` section at
  the top preserving contiguous parent-child shape (no
  severity-grouping), so the full tree structure is visible without
  cross-ref'ing between severity buckets.

- Cross-bucket parent annotation: when a row's parent (by ppid) lives in
  a *different* severity bucket, suffix with `[parent: <pid> (in
  `<bucket>`)]` so the `└─` marker resolves even when bucketing scatters
  parent/child into separate sections.

Also,
- split arg parsing into flag vs positional args.
- add `pid_to_bucket` dict + `walk_order` list to back both features
- rename inner `ppid` shadow to `ppid_str` to avoid collision with the
  outer `ppid` variable.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 0f4e671862)
2026-06-17 17:39:44 -04:00
Gud Boi a0456fece4 Add `enable_transports`/`registry_addrs` proto guard
Raise `ValueError` from `open_root_actor()` when any
`registry_addrs` entry uses a transport proto not in
`enable_transports` — historically this caused a
silent indefinite hang during the registrar handshake
(the actor could never connect to register/discover).

Also,
- update `test_root_passes_tpt_to_sub` to detect a
  proto mismatch between parametrized `tpt_proto_key`
  and CLI `tpt_proto`, asserting the new guard raises
  `ValueError` with expected msg content.
- replace old commented-out notes with a clearer
  explanation of the mismatch foot-gun.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit d036ef7d7f)
2026-06-17 17:39:44 -04:00
Gud Boi 0c64e76bf9 Fix shutdown deadlock on UDS unlink race
Wrap `os.unlink()` in `close_listener()` with a `FileNotFoundError`
guard — under concurrent pytest sessions the sock-file can already be
reaped. Without this the raise aborts `_serve_ipc_eps`'s finally before
`_shutdown.set()`, deadlocking `wait_for_shutdown()` on
`actor.cancel()`.

Also,
- close each endpoint independently in the finally so one raise doesn't
  strand the rest.
- always signal `_shutdown.set()` regardless of remaining ep count.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 2ee44a6fdd)
2026-06-17 17:39:44 -04:00
Gud Boi 84d52835ab Add `tractor_diag`(nosis) xontrib with aliases
Xonsh xontrib providing three diagnostic commands
for tractor development / hang investigation:

- `pytree <pid|pat>` — psutil-backed proc tree with severity-bucketed
  output (zombies > orphans > live), tree-depth markers, zombie-safe
  rendering.
- `hung-dump <pid|pat>` — kernel `wchan`/`stack` + `py-spy dump
  --locals` per descendant, sudo-cred caching upfront, pgrep fallback
  when psutil absent.
- `bindspace-scan [<dir>]` — scan UDS bindspace for orphaned
  `<name>@<pid>.sock` files whose binder pid is dead, emit `rm`
  one-liner for cleanup.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 7b14fdcd96)
2026-06-17 17:39:44 -04:00
Gud Boi cc5be294b1 Mk per-test reap fixtures opt-in
Rename `_track_orphaned_uds_per_test` and
`_detect_runaway_subactors_per_test` to public names (drop `_` prefix),
drop `autouse=True`. Tests that need per-test reap blame now opt in via
`pytestmark = pytest.mark.usefixtures(...)`.

Also,
- reduce `sample_interval` from 0.5 -> 0.05s so the CPU probe is cheaper
  per pid.
- add empty-`only_pids` fast-path in `find_runaway_subactors` to skip
  psutil import when no descendants were spawned.
- extract `new_pids` intermediate var for clarity.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit e4953851de)
2026-06-17 17:39:44 -04:00
Gud Boi d8a398f6f5 Mv `daemon` + `test_multi_program` to `discovery/`
All `daemon` fixture consumers are discovery-
protocol tests now living under `tests/discovery/`.
Move the fixture, its `_wait_for_daemon_ready`
helper, and `test_multi_program.py` into that subdir
so scope matches usage.

Also,
- add `pytestmark` for `track_orphaned_uds_per_test`
  + `detect_runaway_subactors_per_test` to `test_multi_program` as
    regression net.
- drop now-unused `_PROC_SPAWN_WAIT` + `socket` import from root
  conftest.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit c4082be876)
2026-06-17 17:39:44 -04:00
Gud Boi e873bb6164 Replace sleep with active poll in `daemon` fixture
First draft at resolving,
https://github.com/goodboy/tractor/issues/424

`tests.conftest.py.daemon()` previously used a blind
`time.sleep(_PROC_SPAWN_WAIT + uds_bonus + ci_bonus)` to "wait for the
daemon to come up" before yielding the proc to the test.

Two problems:

1. **Racy under load** — sleep is fixed at design time; loaded boxes
   / cold starts / fork-spawn cost spikes blow past it, leading to
   `ConnectionRefusedError` /`OSError: connect failed` flakes in
   `test_register_duplicate_name`.

2. **Wasteful when daemon comes up fast** — happy-path pays the FULL
   sleep regardless. ~3s of dead time per fixture invocation, ~10-20s
   per full suite run.

Replace with `_wait_for_daemon_ready()` — active poll via stdlib
`socket.create_connection` (TCP) or `socket.connect` (UDS) on the
daemon's bind addr, with 50ms backoff and a 10s/15s deadline (CI gets
extra headroom). Daemon-died-during-startup early-exit catches the case
where `_PROC_SPAWN_WAIT` was silently masking daemon startup crashes.

Why stdlib `socket` (Option 2 from the conc-anal doc) instead of
`tractor`'s own `_root.ping_tpt_socket` closure or trio?

- `tractor.run_daemon()` doesn't return from bootstrap until the runtime
  is fully ready to handle IPC, so probing listen-side acceptance is
  sufficient.
- no need to do the full IPC handshake just to validate readiness.
  Sidesteps the `trio.run()` bootstrap cost (~50ms) per fixture too.

`claude`'s verification: 10/10 runs of `tests/test_multi_program.py`
pass on both `--tpt-proto=tcp` and `--tpt-proto=uds`. Per-test wall-time
`test_register_duplicate_name`: 4.31s → 1.10s. Full file: ~12s → 3.27s
per transport.

Doc-tracked at:
`ai/conc-anal/test_register_duplicate_name_daemon_connect_race_issue.md`

Future work — session-scoped trio runtime in a bg thread to share
fixture-side trio operations across many fixtures (currently overkill
for the one fixture that needs it).

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit ec8c4659c4)
2026-06-17 17:39:44 -04:00
Gud Boi 63c6da9e82 Use single f-string per pid in runaway warning
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 086e9f2c07)
2026-06-17 17:39:44 -04:00
Gud Boi 272b6686ae Harden `test_debugger` for forkserver spawners
Use `is_forking_spawner` fixture + gate spawner-
specific expect patterns in nested-error and daemon
tests. Add `set_fork_aware_capture` to multi-sub
tests that need capture-mode awareness.

Deats,
- replace `start_method` param with `is_forking_spawner` bool fixture.
- bump inter-send delay to 0.1s for IPC stability under fork backends.
- gate `bdb.BdbQuit` + relay-uid patterns behind `not
  is_forking_spawner` (not visible under capsys).
- add `expect(child, EOF)` to confirm clean exit.
- switch caught exc from `AssertionError` to `ValueError` in daemon
  test.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 9031605807)
2026-06-17 17:39:44 -04:00
Gud Boi 797d6cf4fa Drop global mutation of `_PROC_SPAWN_WAIT`
In top level `daemon`-fixture that is..

Use a local `bg_daemon_spawn_delay` instead of
mutating the module-level `_PROC_SPAWN_WAIT` —
previously each `daemon` fixture invocation would
permanently add 1.6s (UDS) or 1s (CI) to the
global, inflating delays across the session.

Also, emit a `test_log.warning()` when verbose
loglevel is silently reduced to `'info'`.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit c4885f9d99)
2026-06-17 17:39:44 -04:00
Gud Boi 98fdaf343d Add `tractor.trionics.patches` subpkg + first fix
With a seminal patch fixing `trio`'s `WakeupSocketpair.drain()` which
can busy-loop due to lack of handling `EOF`.

New `tractor.trionics.patches` subpkg housing defensive monkey-patches
for upstream `trio` bugs we've encountered while running `tractor`
— particularly as of recent, fork-survival edge cases that haven't been
filed/fixed upstream yet. Each patch is idempotent, version-gated via
`is_needed()`, and carries a `# REMOVE WHEN:` marker pointing at the
upstream release whose adoption allows deletion.

Subpkg layout + per-patch contract documented in
`tractor/trionics/patches/README.md` — `apply()` / `is_needed()`
/ `repro()` API, registry pattern via `_PATCHES` in `__init__.py`,
single-call entry point `apply_all()`.

First patch, `_wakeup_socketpair`:
- `trio`'s `WakeupSocketpair.drain()` loops on `recv(64KB)` and exits
  ONLY on `BlockingIOError`, NEVER on `recv() == b''` (peer-closed FIN).
- under `fork()`-spawning backends the COW-inherited socketpair fds
  & `_close_inherited_fds()` teardown can leave a `WakeupSocketpair`
  instance whose write-end is closed, and `drain()` then **spins forever
  in C with no Python checkpoints**,
- this obviously burns 100% CPU and no signal delivery.

Standalone repro:

    from trio._core._wakeup_socketpair import WakeupSocketpair
    ws = WakeupSocketpair()
    ws.write_sock.close()
    ws.drain()  # spins forever

Patch is one-line — break the drain loop on b'' EOF.

Manifested as two distinct test failures:

- `tests/test_multi_program.py::test_register_duplicate_name` hung at
  100% CPU on the busy-loop directly (fork child's worker thread)
- `tests/test_infected_asyncio.py::test_aio_simple_error` Mode-A
  deadlock — busy-loop wedged trio's scheduler inside `start_guest_run`,
  both threads parked in `epoll_wait`, no TCP connect-back to parent
  ever happened.

Same patch fixes both. Restored 99.7% pass rate on full
suite under `--spawn-backend=main_thread_forkserver`
(was hanging indefinitely before).

Wired into `tractor._child._actor_child_main` via `apply_all()` BEFORE
any trio runtime init. Harmless on non-fork backends.

Conc-anal write-ups, including strace + py-spy evidence:

- `ai/conc-anal/trio_wakeup_socketpair_busy_loop_under_fork_issue.md`
- `ai/conc-anal/infected_asyncio_under_main_thread_forkserver_hang_issue.md`

Regression tests in `tests/trionics/test_patches.py`: each test asserts
(a) the bug exists pre-patch (or is fixed upstream — skip cleanly), (b)
the patch fixes it with a SIGALRM wall-clock cap so a regression hangs
loud instead of silently.

TODO:
- [ ] file the upstream `python-trio/trio` issue + PR.
- [ ] use the `repro()` callable in `_wakeup_socketpair.py` IS the issue
      body's evidence section.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 0ef549fadb)
(factored: dropped spawn-backend-only paths: ai/conc-anal/infected_asyncio_under_main_thread_forkserver_hang_issue.md)
2026-06-17 17:39:44 -04:00
Gud Boi 73c9b9ec93 Add `tractor.spawn._reap.unlink_uds_bind_addrs()`
Inside a new new `tractor.spawn._reap` submod which kicks off providing
post-mortem subactor cleanup primitives, parent-side; consider it the
"sibling" of `tractor._testing._reap` which is the test-harness-oriented
brother mod.

Today: `unlink_uds_bind_addrs()` provides a starter bug-fix for #454
where `hard_kill()`'s `SIGKILL` bypasses the subactor's
`_serve_ipc_eps`-`finally:` `os.unlink(addr.sockpath)`, leaking
`${XDG_RUNTIME_DIR}/tractor/<name>@<pid>.sock` files..

This adds 2 cleanup paths:
- explicit `bind_addrs` (when set at spawn time),
OR
- convention-based reconstruction from `subactor.aid.name + proc.pid`
  for the random-self-assign case.

`.spawn.hard_kill()` now invokes the cleanup unconditionally
post-`SIGKILL`; graceful-exit case is a no-op via `FileNotFoundError`
skip.

Future work — authoritative tracking via a per-process
UDS bind-addr registry — documented in module docstring,
deferred to a follow-up PR.

Co-fix: `tractor/spawn/_trio.py::new_proc` already passes
`bind_addrs` + `subactor` to `hard_kill` via prior work
on this branch.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit e9712dcaeb)
2026-06-17 17:39:44 -04:00
Gud Boi a4f6d893ed Add per-test runaway-subactor CPU detector to `_reap`
New `find_runaway_subactors()` helper + autouse
`_detect_runaway_subactors_per_test` fixture that
samples `psutil.cpu_percent()` on descendants to
catch tight-loop bugs (e.g. #452-class `recvfrom`
on a closed socket). Checks both at setup
(leftovers from a prior hung test) and teardown
(spawned by this test).

Intentionally does NOT kill the runaway — emits
a loud warning with diag commands (`strace`,
`lsof`, `ss`, `kill`) so the pid stays alive for
hands-on investigation. Session-end reaper still
SIGINT/SIGKILL survivors on normal exit.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 5cf0312c78)
2026-06-17 17:39:44 -04:00
Gud Boi 8589a796c1 Fix `maybe_override_capture` to not get invalid capX fixture names..
(cherry picked from commit 32e89c67ee)
2026-06-17 17:39:44 -04:00
Gud Boi afa14b01d2 Add fork-aware capture fixtures to `_testing.pytest`
Extend the pytest plugin with helpers that detect
and adapt to `--capture=sys` under fork-based
spawners (`main_thread_forkserver`, `mp_forkserver`)
where fd-capture causes hangs.

Deats,
- track `_cap_sys_passed_as_flag` + `_cap_fd_set`
  globals in `pytest_load_initial_conftests()`.
- add `@pytest.hookimpl(tryfirst=True)` + re-parse
  args after appending `--capture=sys`.
- `_is_forking_spawner()` predicate + fixture.
- `maybe_xfail_for_spawner()` — enalbes skipping tests that need capsys
  but weren't passed `--capture=sys`.
- `set_fork_aware_capture` fixture — returns the appropriate capture
  fixture per spawner backend based on `start_method: str` set via CLI.
- wire `set_fork_aware_capture` into `tractor_test`
  wrapper's fixture injection.

Also,
- add `alert_on_finish` session fixture (terminal
  bell on completion; tho not sure it works fully..)
- add `ids=` to `start_method` parametrize.
- restore `default=False` on `--enable-stackscope`.
- drop commented-out `--ll` option block; we will likely factor it to
  our plugin eventually however..

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit d549c72052)
2026-06-17 17:39:44 -04:00
Gud Boi 836270e995 Adjust `test_shield_pause` for capsys backends
Under `main_thread_forkserver` the bootstrapping
hook switches to `--capture=sys`, so subactor
fd-level output (tree dumps, zombie-reaper msgs)
isn't captured per-test by pexpect. Gate those
expects behind a `no_capfd` check so the test
passes on both capture modes.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 5a9926fc32)
2026-06-17 17:39:44 -04:00
Gud Boi 02177d96c4 Default `--ll` to `None` in test harness
Only override `tractor.log._default_loglevel` when
the flag is explicitly passed — lets per-spawn and
per-example `loglevel` kwargs take effect instead
of being clobbered by the hard-coded `'ERROR'`
default.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 72a0465c52)
2026-06-17 17:39:44 -04:00
Gud Boi b13d06af80 Update debug examples + harden `test_debugger`
Pass explicit `loglevel` to `spawn()` calls in
`test_debugger` tests — required for pexpect
pattern matching now that examples no longer
hard-code log levels.

Also,
- make `expect()` return the decoded `before` str.
- add `start_method` param + fork-backend timeout
  slack (+4s) in nested-error test.
- clean up debug examples: drop unused loglevels,
  rename `n` -> `an`, fix docstrings, add TODO
  comments for tpt parametrize via osenv.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 9431a81d37)
2026-06-17 17:39:44 -04:00
Gud Boi 99098c5a14 Update `sync_bp` + tighten `test_pause_from_sync`
Add `disable_pdbp_color()` to the `sync_bp` example
to suppress pygments prompt coloring when
`PYTHON_COLORS=0` — makes pexpect pattern matching
deterministic.

Deats,
- set `loglevel='pdb'` in both script + test spawn.
- disable `enable_stack_on_sig` in example, assert
  no `stackscope` output in test.
- update `attach_patts` keys/values with `|_<Task`
  / `|_<Thread` / `|_('subactor'` prefixes to match
  actual tree-dump format.
- add call-site patterns (`tractor.pause_from_sync()`
  `tractor.pause()`, `breakpoint(hide_tb=...)`).
- trim trailing `\n` from `Lock.repr()` output.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit fc2e298a29)
2026-06-17 17:39:44 -04:00
Gud Boi 158506d8c0 Add `use_stackscope` runtime var for subactor init
Track `stackscope` enablement in `RuntimeVars` so
the flag propagates to subactors via the standard
rtvar IPC path instead of relying solely on the
`TRACTOR_ENABLE_STACKSCOPE` env var.

Deats,
- add `use_stackscope: bool` to `RuntimeVars`
  struct + defaults dict
- `enable_stack_on_sig()` sets the rtvar on
  successful `stackscope` import, asserts unset
  on `ImportError`
- nest stackscope init under `_debug_mode` gate
  in `Actor.async_main`, check rtvar alongside
  env var
- defer `maybe_init_greenback` import to its own
  `use_greenback` branch

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 48523358cf)
2026-06-17 17:39:44 -04:00
Gud Boi e1da7be4d9 Fix `SIGUSR1` tree-dump ordering in `_stackscope`
Factor the sub-actor relay loop out of
`dump_tree_on_sig()` into `_relay_sig_to_subactors()`
and chain both dump + relay in a single
`run_sync_soon` callback (`_dump_then_relay`) so the
parent's task-tree flushes BEFORE any sub receives
the signal — fixes a hierarchical-ordering race
where subs could dump ahead of the parent in the
muxed pty stream.

Also,
- gate file/tty sink writes behind `write_file` +
  `write_tty` params on `dump_task_tree()`.
- use `actor.aid.uid` instead of deprecated `.uid`.
- update `test_shield_pause` expects to match the
  new sequential parent -> relay-log -> sub ordering.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit e2b790a70d)
2026-06-17 17:39:44 -04:00
Gud Boi af72192461 Add `pytest_load_initial_conftests()` for `--capture=`
Move `--capture=sys` enforcement from a static ini
flag to a `pytest_load_initial_conftests()` bootstrap
hook that dynamically flips capture mode only when a
fork-based spawner (like `main_thread_forkserver`) is
detected; non-fork backends keep `--capture=fd`.

Also,
- load `tractor._testing.pytest` via `-p` in ini
  (bc bootstrapping hooks must register before
  conftest `pytest_plugins` runs).
- register `_reap` as sub-plugin via `pytest_plugins`
  tuple in `._testing.pytest`.
- drop now-duplicate reap fixtures (already in `_reap`
  per 1cdc7fb3).
- rename `tractor_enable_stackscope` dest -> `enable_stackscope`
  and pop env var on disable.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 61d4525137)
2026-06-17 17:39:44 -04:00
Gud Boi 831cb6fcf0 Add `--uds`/`--uds-only` flags to `tractor-reap`
Wire up `find_orphaned_uds()` + `reap_uds()` from
`_reap` as a new phase-3 UDS sweep in the CLI
script. Opt-in via `--uds` (run after proc reap +
shm) or `--uds-only` (skip other phases).

Also,
- consolidate skip-proc-reap logic into a single
  `skip_proc_reap` bool covering both `--shm-only`
  and `--uds-only`
- extend header docstring + usage examples

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 0996a83655)
2026-06-17 17:39:44 -04:00
Gud Boi 80048ff6ad Add UDS orphan-sweep helpers + reap fixtures to `_reap`
Extend the `_testing._reap` mod with UDS sock-file leak detection +
cleanup, complementing the existing shm and subactor-process
reaping:

- `get_uds_dir()`, `_parse_uds_name()`, `find_orphaned_uds()`,
  `reap_uds()` — detect `<name>@<pid>.sock` files under
  `${XDG_RUNTIME_DIR}/tractor/` whose binder pid is dead (including
  the `1616` registry sentinel).
- `_reap_orphaned_subactors` session-scoped autouse fixture: SIGINT
  lingering subactors, wait, SIGKILL survivors, then sweep orphaned
  UDS files.
- `_track_orphaned_uds_per_test` fn-scoped autouse fixture:
  snapshot sock-file dir before/after each test, warn + reap new
  orphans to prevent cascade flakiness under `--tpt-proto=uds`.
- `reap_subactors_per_test` opt-in fn-scoped fixture for modules
  with known-leaky teardown.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 1cdc7fb302)
2026-06-17 17:39:44 -04:00
Gud Boi f9e64d0035 Allow per-call `start_method`/`loglevel` overrides
In `tests/devx/conftest.py::spawn`, refactor the
fixture-internal closures so consumer tests can pass
explicit `start_method`/`loglevel` to each `_spawn()`
invocation rather than only inheriting the fixture-
scoped parametrize values.

Deats,
- promote `set_spawn_method()` and `set_loglevel()`
  to take their respective values as fn params (vs
  closing over the fixture-scope vars).
- give `_spawn()` `start_method=start_method` and
  `loglevel: str|None = None` kwargs so callers
  override one-off without re-parametrizing the
  suite. NOTE: this drops the implicit fixture-
  scoped `loglevel` forward — `_spawn()` callers
  now must pass `loglevel=...` explicitly.
- TODO: figure out how `--ll <level>` should map to
  the default (currently `None` → uses env-var or
  tractor default).
- add a docstring to `_spawn()` so its role as the
  consumer-facing closure is obvious from `help()`.

Also,
- `assert_before()` now returns the `.before` output
  on success (was `None`); add a one-line docstring
  describing the new return contract.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 486249d74f)
2026-06-17 17:39:44 -04:00
Gud Boi 5d37acb5b8 Drop test-local timeouts, +`sync_pause` to dev
In `pyproject.toml`,
- include the `sync_pause` group from `dev`, so dev
  installs ship `greenback` for `pause_from_sync()`.

Comment out per-test `@pytest.mark.timeout(...)`
markers in,
- `tests/devx/test_debugger.py`
- `tests/discovery/test_registrar.py`
- `tests/spawn/test_main_thread_forkserver.py`
- `tests/spawn/test_subint_cancellation.py`
- `tests/test_advanced_streaming.py`
- `tests/test_cancellation.py`

The global cap was already dropped (3c366cac); these
were the leftover per-test caps which now block
interactive `pdb` flows under the new spawn backends.

In `uv.lock`,
- pull `greenback` into the resolved `dev` deps
  (per the `sync_pause` include above).
- catch up the prior `xonsh` editable→PyPI switch
  (from the `pyproject.toml` `tool.uv.sources` edit).

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit b7115fc875)
(factored: dropped spawn-backend-only paths under tests/spawn/)
2026-06-17 17:39:44 -04:00
Gud Boi d4eac06de2 Honor `TRACTOR_LOGLEVEL`+`TRACTOR_SPAWN_METHOD` env-vars
Add env-var overrides inside `._root.open_root_actor()` so
devs/test-runs can swap the actor-spawn backend or crank
console verbosity *without* touching application code.

In `._root.open_root_actor()`,
- read `TRACTOR_LOGLEVEL` early, overriding any caller-passed
  `loglevel` and stashing an `env_ll_report` to emit once the
  console log is set up.
- pull the `loglevel` fallback (`or _default_loglevel`) and
  `log.get_console_log()` init *up* so the env-var report
  routes through tractor's own logger.
- read `TRACTOR_SPAWN_METHOD`, overriding any caller-passed
  `start_method` and warn-logging when the env-var clobbers
  an explicit caller value.

Wire the same vars through `tests/devx/conftest.py::spawn`,
- request the `loglevel` fixture, set both `TRACTOR_LOGLEVEL`
  and `TRACTOR_SPAWN_METHOD` in `os.environ` before each
  `pexpect.spawn()` (inherited by the example subproc).
- expand `supported_spawners` to include
  `main_thread_forkserver` and `subint_forkserver` bc
  example scripts no longer need per-script CLI plumbing.
- pop both vars in fixture teardown so a leaked value can't
  re-route a later in-process tractor test's spawn-backend
  or loglevel.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 208e7c0926)
2026-06-17 17:39:44 -04:00
Gud Boi 950b6bc6b8 Add todo for running `test_debugger` suite on forkserver spawner
(cherry picked from commit 2917b74ba4)
2026-06-17 17:39:44 -04:00
Gud Boi c31b85587a Route `stackscope` SIGUSR1 onto trio loop
Signal handlers fire in a non-trio stack frame; calling
`stackscope.extract(recurse_child_tasks=True)` from there
only walks the `<init>` task and misses everything inside
`async_main`'s nurseries — exactly the part you want to
see during a hang.

Fix: capture `trio.lowlevel.current_trio_token()` at
`enable_stack_on_sig()` time and stash it as a module-
level `_trio_token`. The SIGUSR1 handler then dispatches
the dump *onto* the trio loop via
`_trio_token.run_sync_soon(_safe_dump_task_tree)`, so
`stackscope.extract` runs from a real trio-task context
and walks the full nursery tree.

Late-binding: pytest's `pytest_configure` calls
`enable_stack_on_sig()` outside any `trio.run`, so token
capture there is a `RuntimeError` — left at `None`. The
runtime re-calls `enable_stack_on_sig()` from inside
`async_main` (subactor side) where the token IS
available, so subactors get the full-tree path.
`dump_tree_on_sig` falls back to a direct call when
`_trio_token is None` (parent process pre-trio.run, or
signal delivered after `trio.run` returns).

`_safe_dump_task_tree()` is a `run_sync_soon`-friendly
wrapper that swallows any exception from
`dump_task_tree()` — trio prints + crashes on uncaught
exceptions in scheduled callbacks; better to log + keep
the run alive so the user can re-trigger.

Other,
- emit `capture-bypass tee: <fpath>` line + `tail -f`
  hint in the rendered dump header so users know where
  to find the artifact even when stdio is captured.
- swap the inline `f'     |_{actor}'` line for a
  `_pformat.nest_from_op` rendering of `actor_repr`
  (matches the rest of the runtime's nested-op style).
- log lines on handler install + already-installed
  branches now note `(trio_token captured: <bool>)`
  so it's obvious from the log whether the full-tree
  path is wired.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 2d4995e08d)
2026-06-17 17:39:44 -04:00
Gud Boi 1d185e4d94 Add `--enable-stackscope` pytest plugin flag
New `--enable-stackscope` CLI flag installs a SIGUSR1 →
trio-task-tree-dump handler in pytest itself + every
spawned subactor for live stack visibility during hang
investigations. Lighter than `--tpdb` (no pdb machinery
/ tty-lock contention) — pure stack-only triage.

Plumbing:
- `_testing.pytest.pytest_addoption()` adds the flag.
- `_testing.pytest.pytest_configure()` (when flag set):
  * exports `TRACTOR_ENABLE_STACKSCOPE=1` so fork-children
    inherit it via environ,
  * installs the handler in pytest itself via
    `enable_stack_on_sig()`.
- `runtime._runtime.Actor.async_main()` extends the
  existing `_debug_mode` gate to ALSO fire when
  `TRACTOR_ENABLE_STACKSCOPE` is in env — so subactors
  install the same handler at runtime startup.

Capture-bypass tee in `dump_task_tree()`:
Pytest's default `--capture=fd` swallows `log.devx()`
output, making SIGUSR1 dumps invisible right when you
need them. Render the dump once to a `full_dump` str,
then unconditionally tee to:

- `/tmp/tractor-stackscope-<pid>.log` (append-mode,
  always written) — guaranteed-readable artifact even
  under CI / `nohup` / no-tty. `tail -f` to follow.
- `/dev/tty` (best-effort) — pytest never captures the
  tty; ignored if device is missing.

Other,
- squelch the benign `RuntimeWarning` ("coroutine method
  'asend'/'athrow' was never awaited") from
  `stackscope._glue`'s import-time async-gen type
  introspection so `--enable-stackscope` setup stays
  quiet.
- log msg in the `_runtime` ImportError branch now
  mentions `--enable-stackscope` alongside debug-mode.

Usage,
  pytest --enable-stackscope -k <hang-test>
  # in another shell, find the pid + signal:
  kill -USR1 <pytest-or-subactor-pid>
  # tail the artifact:
  tail -f /tmp/tractor-stackscope-<pid>.log

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 5418f2dc3c)
2026-06-17 17:39:44 -04:00
Gud Boi dacd7a7c57 Backend-aware `fail_after` in pub/sub test
Mirror `060f7d24`'s pattern (backend-aware timeout in
`maybe_expect_raises`) for `test_dynamic_pub_sub`'s hard
`trio.fail_after` cap. Fork-based backends pay per-spawn
fork+IPC-handshake cost which stacks over `cpus - 1`
sequential `n.run_in_actor()` calls; empirically 12s
flakes on `main_thread_forkserver` under UDS
cross-pytest contention (#451 / #452).

Defaults:
- `main_thread_forkserver` → 30s
- everything else          → 12s (unchanged)

Hoist the timeout-pick out of the `main()` closure so the
dispatch happens once in the trio task rather than
re-evaluating per spawn.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 383b0fdd75)
2026-06-17 17:39:44 -04:00
Gud Boi c603a6e1a0 Backend-aware timeout in `maybe_expect_raises`
Default `timeout` from `int = 3` → `int|None = None`;
when unset, pick a backend-aware value. Fork-based
backends (`main_thread_forkserver`) need real headroom
bc actor spawn + IPC ctx-exit + msg-validation error
path is much heavier than under `trio` backend —
especially under cross-pytest-stream contention (#451).

Defaults:
- `main_thread_forkserver` → 30s
- everything else          → 3s (unchanged)

Empirical flake history that motivated 30s as the floor
on fork backends (all from `test_basic_payload_spec`):

- 3s  → all-valid variant flaked w/ `TooSlowError`
- 8s  → `invalid-return` variant flaked w/ `Cancelled`
        (surfaced instead of `MsgTypeError` bc the
        outer `fail_after` fired mid-error-path)
- 15s → flaked under cross-pytest-stream contention

30s gives plenty of headroom while still failing-loud
on a genuine hang. Callers can opt out by passing an
explicit `timeout=` kw.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 060f7d24c4)
2026-06-17 17:39:44 -04:00
Gud Boi 3954d9f527 Return parent `pid: int` from new `reap_subactors_per_test` fixture
(cherry picked from commit f8178df0fd)
2026-06-17 17:39:44 -04:00
Gud Boi 13ccbaff60 Use `trio.fail_after` cap in `test_dynamic_pub_sub`
Drop `@pytest.mark.timeout(...)` for the per-test wall-clock
cap on `test_dynamic_pub_sub`; rely on `trio.fail_after(12)`
inside `main()` instead.

Both pytest-timeout enforcement modes are incompatible with
trio under fork-based backends:

- `method='signal'` (SIGALRM) synchronously raises `Failed`
  in trio's main thread mid-`epoll.poll()`, leaving
  `GLOBAL_RUN_CONTEXT` half-installed ("Trio guest run got
  abandoned") so EVERY subsequent `trio.run()` in the same
  pytest process bails with
  `RuntimeError: Attempted to call run() from inside a run()`
  — full-session poison.
- `method='thread'` calls `_thread.interrupt_main()` which
  can let the KBI escape trio's `KIManager` under fork-
  cascade teardown races and bubble out of pytest entirely
  — kills the whole session.

`trio.fail_after()` keeps cancellation inside the trio loop:
- Raises `TooSlowError` cleanly through the open-nursery's
  cancel cascade.
- Doesn't disturb any out-of-band signal/thread state.
- Failure stays scoped to the single test — no cross-test
  global state corruption either way.

Verified empirically: 10 hammer-runs of `test_dynamic_pub_sub`
go from 5/10 fail (with global-state poison) to 3/10 fail
(no poison, all sibling tests still pass). The ~30%
remaining flake rate is a genuine fork-cancel-cascade
hang — separate from this fix but no longer contaminates.

Module-level NOTE comment explains the rationale so future
readers don't re-introduce the bug.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 530160fa69)
2026-06-17 17:39:44 -04:00
Gud Boi ec5141b720 Add opt-in `reap_subactors_per_test` fixture
Function-scoped, NON-autouse zombie-subactor reaper for
modules whose teardown is known-leaky enough to cascade-
fail every following test in a session.

Sibling to the autouse session-scoped `_reap_orphaned_subactors`. The
session-scoped one fires at session end — too late to save tests that
follow a hung/leaky test in the suite. The new fixture, opted into via
`pytestmark = pytest.mark.usefixtures(...)`, runs between tests in
a problem-module so a leftover subactor from test N can't squat on
registrar ports / UDS paths / shm segments needed by tests N+1,
N+2, ...

Intentionally NOT autouse — the fixture's presence on a module signals
"this module's teardown leaks; please root-cause instead of relying
forever on cleanup". A visibility-vs-convenience trade picked in favor
of the former.

Apply to `tests/test_infected_asyncio.py` since both recent full-suite
runs (parallel-tpt-proto + TCP-only) showed the cascade originating in
this file's KBI- and SIGINT-flavored tests under
`main_thread_forkserver`. Module-comment names the specific offenders so
future de-flake work has a starting point.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit b376eb0332)
2026-06-17 17:39:44 -04:00
Gud Boi d24ccaada1 Fix `_testing.addr.get_rando_addr` cross-process collisions
Previously the random port was a default-arg expression
(`_rando_port: str = random.randint(1000, 9999)`) — evaluated
ONCE at module import time, making it a per-process singleton.
Two parallel pytest sessions had a 1/9000 birthday-pair chance
of picking the same port; when it hit, every `reg_addr`-using
test in BOTH runs would cascade-fail with "Address already in
use".

Switch to per-call `random.randint()` salted with `os.getpid()`
so:

- within one session: two calls return distinct ports — e.g.
  `test_tpt_bind_addrs::bind-subset-reg` now actually gets two
  different reg addrs on the TCP backend (it was silently
  duplicating before),
- across parallel sessions: pid salt biases each process's
  port choices apart, making cross-run collisions
  vanishingly rare.

Drop the bogus `: str` annotation (was always `int`). UDS already gets
per-process isolation via `UDSAddress.get_random()`'s `@<pid>`
socket-path suffix, so no change needed there.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 7c5dd4d033)
2026-06-17 17:39:44 -04:00
Gud Boi f7a58b82fe Sweep `subint_forkserver` → `main_thread_forkserver` in code
After the variant-1 / variant-2 backend split, update remaining
string-match refs to the variant-1 backend so user-visible gates
+ skip-marks + comments name the working backend correctly:

- `tractor._root._DEBUG_COMPATIBLE_BACKENDS`: include
  `main_thread_forkserver`, drop the stub-only `subint_forkserver`
  entry.
- `tests/test_spawning.py::test_loglevel_propagated_to_subactor`:
  capfd-skip flips to `main_thread_forkserver`.
- `tests/test_infected_asyncio.py::test_sigint_closes_lifetime_stack`:
  xfail-condition flips to `main_thread_forkserver`.
- `tests/test_shm.py`: drop stale "broken on `main_thread_forkserver`"
  reason-text since the `mp.SharedMemory(track=False)`
  + resource-tracker monkey-patch in `.ipc._mp_bs` makes the tests pass;
  the skip-mark only fires on plain `subint` now.
- Comment / docstring sweep: `runtime._state`, `runtime._runtime`,
  `_testing.pytest`, `_subint.py`, `pyproject.toml`,
  `test_cancellation.py`, `test_registrar.py` — refs to variant-1
  backend updated.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 205382a39b)
(factored: dropped spawn-backend-only path: tractor/spawn/_subint.py)
2026-06-17 17:39:44 -04:00
Gud Boi 149e1e1993 Wire `reg_addr` into `test_context_stream_semantics`
Same wire-up pattern as the prior `test_dynamic_pub_sub`
commit: each test that already pulled in `debug_mode`
now also pulls in `reg_addr` and passes
`registry_addrs=[reg_addr]` into `tractor.open_nursery()`,
so the suite's standard registry-addr conventions apply.

Tests touched:
- `test_started_misuse`
- `test_simple_context`
- `test_parent_cancels`
- `test_one_end_stream_not_opened`
- `test_maybe_allow_overruns_stream`
- `test_ctx_with_self_actor`

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 66f1941f46)
2026-06-17 17:39:44 -04:00
Gud Boi a4554106fc Wire `test_dynamic_pub_sub` to standard fixtures
Pull in the `reg_addr`, `debug_mode`, and `test_log`
fixtures so this test follows the same conventions as
the rest of the suite:

- pass `registry_addrs=[reg_addr]` + `debug_mode` into
  `tractor.open_nursery()` (so `--tpdb` etc work).
- after the `pytest.raises` block, add `assert err` +
  `test_log.exception('Timed out AS EXPECTED')` so the
  expected timeout is logged explicitly instead of
  swallowed.

Also,
- drop whitespace-only blank lines around the
  `subs` param of `consumer()` and `ctx` param of
  `one_task_streams_and_one_handles_reqresp()`.
- promote `test_sigint_both_stream_types`'s one-line
  docstring to multi-line form.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 9b05f659b3)
2026-06-17 17:39:44 -04:00
Gud Boi d5a15b4d70 Bump `test_stale_entry_is_deleted`'s timeout to 30
Seems that when run in-suite it delays more then the so-measured "happy
path" timing; better to have no suite-global interruption then asserting
a fast single test's run.

(cherry picked from commit 65fcfbf224)
2026-06-17 17:39:44 -04:00
Gud Boi e5ca5bb017 Add `--shm` orphan sweep to `tractor-reap`
Since `tractor.ipc._mp_bs.disable_mantracker()` turns off
`mp.resource_tracker` entirely (see the conc-anal doc
`subint_forkserver_mp_shared_memory_issue.md`), a
hard-crashing actor can leave `/dev/shm/<key>` segments
that nothing else GCs. New `tractor-reap` phase 2 sweeps
them.

Deats,
- `tractor/_testing/_reap.py`: add `find_orphaned_shm()`
  + `reap_shm()` helpers. Match criteria: regular file
  under `/dev/shm`, owned by current uid, AND no live
  proc has it open (mmap'd or fd-held). In-use
  enumeration via `psutil.Process.memory_maps()` +
  `.open_files()` — xplatform, kernel-canonical (same
  answer `lsof` would give), no reliance on
  tractor-specific shm-key naming.
- `_ensure_shm_supported()` guard: helpers raise
  `NotImplementedError` outside Linux/FreeBSD bc macOS
  POSIX shm has no fs-visible path (`shm_open` only)
  and Windows is a different story.
- `scripts/tractor-reap`: new `--shm` (run after
  process reap) and `--shm-only` (skip process phase)
  flags. `-n` dry-runs both phases. Exit code is `1`
  if either phase had survivors/errors.
- `pyproject.toml` + `uv.lock`: add `psutil>=7.0.0` to
  the `testing` dep group; lazy-imported in `_reap.py`
  so the process-reap path stays import-clean without
  it.

Also,
- doc `--shm` in `.claude/skills/run-tests/SKILL.md`
  (new section 10c) — covers match criteria + the
  preservation guarantee for unrelated apps.
- flip mitigation status in
  `subint_forkserver_mp_shared_memory_issue.md` from
  "could extend `tractor-reap`" to "implemented", with
  a note that callers should still UUID-pin shm keys to
  avoid cross-session collisions.

Verified locally vs 81 in-use segments held by `piker`,
`lttng-ust-*`, `aja-shm-*` — all preserved; only the
genuinely-orphaned tractor segments got unlinked.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 4f12d69b41)
(factored: dropped subint_forkserver conc-anal doc update)
2026-06-17 17:39:44 -04:00
Gud Boi 0b6a7aa1a9 Fix `SharedMemory` under `subint_forkserver`
Implements the resolution described in c99d475d's
`subint_forkserver_mp_shared_memory_issue.md` (now
updated with the resolution post-mortem). Two-part
fix that side-steps `mp.resource_tracker` entirely
rather than try to make it fork-safe — turns out
that's both simpler AND more correct given tractor
already SC-manages allocation lifetimes.

Deats,
- `tractor/ipc/_mp_bs.py::disable_mantracker()`: drop the
  `platform.python_version_tuple()[:-1] >= ('3', '13')` branch — patches
  now run unconditionally:
  * monkey-patch `mp.resource_tracker. _resource_tracker` to a no-op
    `ManTracker` subclass (empty `register` / `unregister`
    / `ensure_running`).
  * return `partial(SharedMemory, track=False)` for the per-allocation
    opt-out.
  * belt + suspenders: even if something dodges the wrapper, the
    singleton can't talk to the inherited (broken) parent fd.

- `tractor/ipc/_shm.py::open_shm_list()`: drop the 3.13+ conditional
  skip of the unlink-callback; install a `try_unlink()` wrapper that
  swallows `FileNotFoundError` (sibling-already-cleaned race in
  shared-key setups). Without `mp.resource_tracker` doing it for us, we
  own the unlink — `actor. lifetime_stack` is the right place since
  tractor already controls actor lifecycle.

- `tests/test_shm.py`: uncomment-out `subint_forkserver` from the
  module-level skip- list (tests pass now). Inline comment cross-refs
  the two `_mp_bs` / `_shm` workarounds.

- `ai/conc-anal/subint_forkserver_mp_shared_memory_ issue.md`: heavy
  rewrite — flips status from "open / unresolvable in tractor" to
  "resolved, kept as decision record". Adds Resolution section, "Why
  this is the right call" rationale (mp tracker is widely criticized;
  tractor already owns lifecycle), trade-offs (crash-leaked segments,
  lost mp leak warning), verification (7 passed under both
  `subint_forkserver` and `trio` backends), and upstream issue links

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit aa3e230926)
(factored: dropped subint_forkserver conc-anal doc update)
2026-06-17 17:39:44 -04:00
Gud Boi 50392e6e78 Document `SharedMemory` × `subint_forkserver` incompat
New `ai/conc-anal/` doc: `mp.SharedMemory` is
fork-without-exec unsafe — child inherits parent's
`resource_tracker` fd → EBADF on first shm op;
leaked `/shm_list` cascades `FileExistsError`
across parametrize variants. Canonical CPython
issue class, NOT a tractor bug. Includes two
longer-term mitigation paths (reset inherited
tracker fd vs migrate off `mp.shared_memory`).

Also, update `tests/test_shm.py`:
- comment out `subint_forkserver` from skip list
- rewrite reason with precise failure-mode
  descriptions + link to the analysis doc

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit c99d475d03)
(factored: dropped spawn-backend-only paths: ai/conc-anal/subint_forkserver_mp_shared_memory_issue.md)
2026-06-17 17:39:44 -04:00
Gud Boi f633ebf5c6 Add `tractor-reap` CLI + document auto-reap
New `scripts/tractor-reap` CLI wraps the
`_testing._reap` mod for manual zombie-subactor
cleanup after crashed pytest sessions. Two modes:

- orphan-mode (default): finds PPid==1 procs
  with cwd matching repo root + `python` in
  cmdline.
- descendant-mode (`--parent <pid>`): scoped
  sweep under a still-live supervisor.

SC-polite: SIGINT with bounded grace window
(default 3s) before escalating to SIGKILL.
Exit code signals whether escalation was needed
(useful for CI health-checks).

Also, document both the auto-reap fixture and
the CLI in `/run-tests` SKILL.md (section 10).

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 6d76b60404)
2026-06-17 17:39:44 -04:00
Gud Boi 48ace6dd82 Add `_testing._reap` + auto-reap fixture
Zombie-subactor cleanup for the test suite, SC-polite discipline
(`SIGINT` first, bounded grace, `SIGKILL` only on survivors). Two parts:
a shared reaper module + an autouse session-end fixture that runs it.

Deats,
- new `tractor/_testing/_reap.py` (+230 LOC) — Linux- only reaper using
  `/proc/<pid>/{status,cwd,cmdline}` inspection. Two detection modes:
  - `find_descendants(parent_pid)` for the in-session case
    (PPid-direct-match while pytest is still alive).
  - `find_orphans(repo_root)` for the CLI / post- mortem case (`PPid==1`
    reparented to init + `cwd` filter to repo root + `python` cmdline
    filter).
- `reap(pids, *, grace=3.0, poll=0.25)` does the signal ladder: SIGINT
  all, poll up to `grace` for exit, SIGKILL any survivors. Returns
  `(signalled, killed)` for caller-side reporting.
- new `_reap_orphaned_subactors` session-scoped autouse fixture in
  `tractor/_testing/pytest.py` — after `yield`, runs
  `find_descendants(os.getpid())` + `reap(...)` so each pytest session
  leaves no surviving forks.
- companion CLI scaffolding lives at `scripts/tractor-reap` (separate
  commit) for the pytest-died-mid-session case where the in-session
  fixture didn't get to run.

Also,
- promote `from tractor.spawn._spawn import SpawnMethodKey` to
  module-top in `pytest.py` (was inline-imported inside
  `pytest_generate_tests`), and reuse it in
  `pytest_collection_modifyitems` to assert each `skipon_spawn_backend`
  mark arg is a valid spawn-method literal — catches typos at collection
  time.
- inline `# ?TODO` flags running these through the `try_set_backend`
  checker for stronger validation.

Cross-refs `feedback_sc_graceful_cancel_first.md` for the
SIGINT-before-SIGKILL discipline rationale.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit eae478f3d5)
2026-06-17 17:39:44 -04:00
Gud Boi 04ffde40f3 Skip `test_loglevel_propagated_to_subactor` on subint forkserver too
(cherry picked from commit 2ca0f41e61)
2026-06-17 17:39:44 -04:00
Gud Boi 18754f2570 Wire `reg_addr` through infected-asyncio tests
Continues the hygiene pattern from de601676 (cancel tests) into
`tests/test_infected_asyncio.py`: many tests here were calling
`tractor.open_nursery()` w/o `registry_addrs=[reg_addr]` and thus racing
on the default `:1616` registry across sessions. Thread the
session-unique `reg_addr` through so leaked or slow-to-teardown
subactors from a prior test can't cross-pollute.

Deats,
- add `registry_addrs=[reg_addr]` to `open_nursery()`
  calls in suite where missing.
- `test_sigint_closes_lifetime_stack`:
  - add `reg_addr`, `debug_mode`, `start_method`
    fixture params
  - `delay` now reads the `debug_mode` param directly
    instead of calling `tractor.debug_mode()` (fires
    slightly earlier in the test lifecycle)
  - sanity assert `if debug_mode: assert
    tractor.debug_mode()` after nursery open
  - new print showing SIGINT target
    (`send_sigint_to` + resolved pid)
  - catch `trio.TooSlowError` around
    `ctx.wait_for_result()` and conditionally
    `pytest.xfail` when `send_sigint_to == 'child'
    and start_method == 'subint_forkserver'` — the
    known orphan-SIGINT limitation tracked in
    `ai/conc-anal/subint_forkserver_orphan_sigint_hang_issue.md`
- parametrize id typo fix: `'just_trio_slee'` → `'just_trio_sleep'`

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit b350aa09ee)
2026-06-17 17:39:44 -04:00
Gud Boi e70ef50a0c Default `pytest` to use `--capture=sys`
Lands the capture-pipe workaround from the prior cluster of diagnosis
commits: switch pytest's `--capture` mode from the default `fd`
(redirects fd 1,2 to temp files, which fork children inherit and can
deadlock writing into) to `sys` (only `sys.stdout` / `sys.stderr` — fd
1,2 left alone).

Trade-off documented inline in `pyproject.toml`:
- LOST: per-test attribution of raw-fd output (C-ext writes,
  `os.write(2, ...)`, subproc stdout). Still goes to terminal / CI
  capture, just not per-test-scoped in the failure report.
- KEPT: `print()` + `logging` capture per-test (tractor's logger uses
  `sys.stderr`).
- KEPT: `pytest -s` debugging behavior.

This allows us to re-enable `test_nested_multierrors` without
skip-marking + clears the class of pytest-capture-induced hangs for any
future fork-based backend tests.

Deats,
- `pyproject.toml`: `'--capture=sys'` added to `addopts` w/ ~20 lines of
  rationale comment cross-ref'ing the post-mortem doc

- `test_cancellation`: drop `skipon_spawn_backend('subint_forkserver')`
  from `test_nested_ multierrors` — no longer needed.
  * file-level `pytestmark` covers any residual.

- `tests/spawn/test_subint_forkserver.py`: orphan-SIGINT test's xfail
  mark loosened from `strict=True` to `strict=False` + reason rewritten.
  * it passes in isolation but is session-env-pollution sensitive
    (leftover subactor PIDs competing for ports / inheriting harness
    FDs).
  * tolerate both outcomes until suite isolation improves.

- `test_shm`: extend the existing
  `skipon_spawn_backend('subint', ...)` to also skip
  `'subint_forkserver'`.
  * Different root cause from the cancel-cascade class:
    `multiprocessing.SharedMemory`'s `resource_tracker` + internals
    assume fresh- process state, don't survive fork-without-exec cleanly

- `tests/discovery/test_registrar.py`: bump timeout 3→7s on one test
  (unrelated to forkserver; just a flaky-under-load bump).

- `tractor.spawn._subint_forkserver`: inline comment-only future-work
  marker right before `_actor_child_main()` describing the planned
  conditional stdout/stderr-to-`/dev/null` redirect for cases where
  `--capture=sys` isn't enough (no code change — the redirect logic
  itself is deferred).

EXTRA NOTEs
-----------
The `--capture=sys` approach is the minimum- invasive fix: just a pytest
ini change, no runtime code change, works for all fork-based backends,
trade-offs well-understood (terminal-level capture still happens, just
not pytest's per-test attribution of raw-fd output).

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 4c133ab541)
(factored: dropped spawn-backend-only paths: tests/spawn/test_subint_forkserver.py + tractor/spawn/_subint_forkserver.py; the xfail-loosening bullet above no longer applies)
2026-06-17 17:39:44 -04:00
Gud Boi f68ae75444 Update `subint_forkserver` skip reason: capture-pipe
Refresh the `test_nested_multierrors` skip-mark
reason to the final diagnosis: the hang is pytest's
default `--capture=fd` pipe filling from high-volume
subactor traceback output inherited via fds 1,2 in
fork children — `pytest -s` passes cleanly. Records
the fix direction (redirect child stdio to
`/dev/null` in the fork-child prelude) for whoever
lands the backend.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit eceed29d4a)
(factored: kept only the tests/test_cancellation.py skip-reason update of
 "Pin forkserver hang to pytest `--capture=fd`"; dropped the subint
 conc-anal doc + tests/spawn/test_subint_forkserver.py)
2026-06-17 17:39:44 -04:00
Gud Boi af0b1fb2dc Bound peer-clear wait in `async_main` finally
Fifth diagnostic pass pinpointed the hang to
`async_main`'s finally block — every stuck actor
reaches `FINALLY ENTER` but never `RETURNING`.
Specifically `await ipc_server.wait_for_no_more_
peers()` never returns when a peer-channel handler
is stuck: the `_no_more_peers` Event is set only
when `server._peers` empties, and stuck handlers
keep their channels registered.

Wrap the call in `trio.move_on_after(3.0)` + a
warning-log on timeout that records the still-
connected peer count. 3s is enough for any
graceful cancel-ack round-trip; beyond that we're
in bug territory and need to proceed with local
teardown so the parent's `_ForkedProc.wait()` can
unblock. Defensive-in-depth regardless of the
underlying bug — a local finally shouldn't block
on remote cooperation forever.

Verified: with this fix, ALL 15 actors reach
`async_main: RETURNING` (up from 10/15 before).

Test still hangs past 45s though — there's at
least one MORE unbounded wait downstream of
`async_main`. Candidates enumerated in the doc
update (`open_root_actor` finally /
`actor.cancel()` internals / trio.run bg tasks /
`_serve_ipc_eps` finally). Skip-mark stays on
`test_nested_multierrors[subint_forkserver]`.

Also updates
`subint_forkserver_test_cancellation_leak_issue.md`
with the new pinpoint + summary of the 6-item
investigation win list:
1. FD hygiene fix (`_close_inherited_fds`) —
   orphan-SIGINT closed
2. pidfd-based `_ForkedProc.wait` — cancellable
3. `_parent_chan_cs` wiring — shielded parent-chan
   loop now breakable
4. `wait_for_no_more_peers` bound — THIS commit
5. Ruled-out hypotheses: tree-kill missing, stuck
   socket recv, capture-pipe fill (all wrong)
6. Remaining unknown: at least one more unbounded
   wait in the teardown cascade above `async_main`

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit e312a68d8a)
(factored: dropped subint_forkserver conc-anal doc update)
2026-06-17 17:39:44 -04:00
Gud Boi 6bf8364c16 Skip-mark `subint_forkserver` nested-multierror hang
Skip-mark the still-hanging
`test_nested_multierrors[subint_forkserver]` via
`@pytest.mark.skipon_spawn_backend('subint_forkserver',
reason=...)` so it stops blocking the test matrix
while the remaining bug is being chased. The mark is
an inert no-op until that (in-dev) backend lands.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 506617c695)
(factored: kept only the tests/test_cancellation.py skip-mark; dropped
 the subint_forkserver conc-anal doc update)
2026-06-17 17:39:44 -04:00
Gud Boi d328177873 Break parent-chan shield during teardown
Completes the nested-cancel deadlock fix started in
0cd0b633 (fork-child FD scrub) and fe540d02 (pidfd-
cancellable wait). The remaining piece: the parent-
channel `process_messages` loop runs under
`shield=True` (so normal cancel cascades don't kill
it prematurely), and relies on EOF arriving when the
parent closes the socket to exit naturally.

Under exec-spawn backends (`trio_proc`, mp) that EOF
arrival is reliable — parent's teardown closes the
handler-task socket deterministically. But fork-
based backends like `subint_forkserver` share enough
process-image state that EOF delivery becomes racy:
the loop parks waiting for an EOF that only arrives
after the parent finishes its own teardown, but the
parent is itself blocked on `os.waitpid()` for THIS
actor's exit. Mutual wait → deadlock.

Deats,
- `async_main` stashes the cancel-scope returned by
  `root_tn.start(...)` for the parent-chan
  `process_messages` task onto the actor as
  `_parent_chan_cs`
- `Actor.cancel()`'s teardown path (after
  `ipc_server.cancel()` + `wait_for_shutdown()`)
  calls `self._parent_chan_cs.cancel()` to
  explicitly break the shield — no more waiting for
  EOF delivery, unwinding proceeds deterministically
  regardless of backend
- inline comments on both sites explain the mutual-
  wait deadlock + why the explicit cancel is
  backend-agnostic rather than a forkserver-specific
  workaround

With this + the prior two fixes, the
`subint_forkserver` nested-cancel cascade unwinds
cleanly end-to-end.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 8ac3dfeb85)
2026-06-17 17:39:44 -04:00
Gud Boi 662a34b994 Wire `reg_addr` through leaky cancel tests
Stopgap companion to d0121960 (`subint_forkserver`
test-cancellation leak doc): five tests in
`tests/test_cancellation.py` were running against the
default `:1616` registry, so any leaked
`subint-forkserv` descendant from a prior test holds
the port and blows up every subsequent run with
`TooSlowError` / "address in use". Thread the
session-unique `reg_addr` fixture through so each run
picks its own port — zombies can no longer poison
other tests (they'll only cross-contaminate whatever
happens to share their port, which is now nothing).

Deats,
- add `reg_addr: tuple` fixture param to:
  - `test_cancel_infinite_streamer`
  - `test_some_cancels_all`
  - `test_nested_multierrors`
  - `test_cancel_via_SIGINT`
  - `test_cancel_via_SIGINT_other_task`
- explicitly pass `registry_addrs=[reg_addr]` to the
  two `open_nursery()` calls that previously had no
  kwargs at all (in `test_cancel_via_SIGINT` and
  `test_cancel_via_SIGINT_other_task`)
- add bounded `@pytest.mark.timeout(7, method='thread')`
  to `test_nested_multierrors` so a hung run doesn't
  wedge the whole session

Still doesn't close the real leak — the
`subint_forkserver` backend's `_ForkedProc.kill()` is
PID-scoped not tree-scoped, so grandchildren survive
teardown regardless of registry port. This commit is
just blast-radius containment until that fix lands.
See `ai/conc-anal/
subint_forkserver_test_cancellation_leak_issue.md`.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 1af2121057)
2026-06-17 17:39:44 -04:00
Gud Boi 353fb82c36 Enable `debug_mode` for `subint_forkserver`
The `subint_forkserver` backend's child runtime is trio-native (uses
`_trio_main` + receives `SpawnSpec` over IPC just like `trio`/`subint`),
so `tractor.devx.debug._tty_lock` works in those subactors. Wire the
runtime gates that historically hard-coded `_spawn_method == 'trio'` to
recognize this third backend.

Deats,
- new `_DEBUG_COMPATIBLE_BACKENDS` module-const in `tractor._root`
  listing the spawn backends whose subactor runtime is trio-native
  (`'trio'`, `'subint_forkserver'`). Both the enable-site
  (`_runtime_vars['_debug_mode'] = True`) and the cleanup-site reset
  key.
  off the same tuple — keep them in lockstep when adding backends
- `open_root_actor`'s `RuntimeError` for unsupported backends now
  reports the full compatible-set + the rejected method instead of the
  stale "only `trio`" msg.
- `runtime._runtime.Actor._from_parent`'s SpawnSpec-recv gate adds
  `'subint_forkserver'` to the existing `('trio', 'subint')` tuple
  — fork child-side runtime receives the same SpawnSpec IPC handshake as
  the others.
- `subint_forkserver_proc` child-target now passes
  `spawn_method='subint_forkserver'` (was hard-coded `'trio'`) so
  `Actor.pformat()` / log lines reflect the actual parent-side spawn
  mechanism rather than masquerading as plain `trio`.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 8bcbe730bf)
2026-06-17 17:39:44 -04:00
Gud Boi 21b17b6a40 Refactor `_runtime_vars` into pure get/set API
Resetting `_runtime_vars` post-(forking-)spawn was
previously only possible via direct mutation of
`_state._runtime_vars` from an external module + an
inline default dict duplicating the
`_state.py`-internal defaults. Split the access
surface into a pure getter + explicit setter so such
a reset call site becomes a one-liner composition:
`set_runtime_vars(get_runtime_vars(clear_values=True))`.

Deats `tractor/runtime/_state.py`,
- extract initial values into a module-level
  `_RUNTIME_VARS_DEFAULTS: dict[str, Any]` constant; the
  live `_runtime_vars` is now initialised from
  `dict(_RUNTIME_VARS_DEFAULTS)`
- `get_runtime_vars()` grows a `clear_values: bool = False`
  kwarg. When True, returns a fresh copy of
  `_RUNTIME_VARS_DEFAULTS` instead of the live dict —
  still a **pure read**, never mutates anything
- new `set_runtime_vars(rtvars: dict | RuntimeVars)` —
  atomic replacement of the live dict's contents via
  `.clear()` + `.update()`, so existing references to the
  same dict object remain valid. Accepts either the
  historical dict form or the `RuntimeVars` struct

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 7804a9fe57693dd5e15bee6a08e7d2fa14b6a98a)
(factored: kept only the tractor/runtime/_state.py part; dropped
 tractor/spawn/_subint_forkserver.py call-site rewire)
2026-06-17 17:39:44 -04:00
Gud Boi d5c549a3c3 Mark `subint`-hanging tests with `skipon_spawn_backend`
Adopt the `@pytest.mark.skipon_spawn_backend('subint',
reason=...)` marker (a617b521) across the suites
reproducing the `subint` GIL-contention / starvation
hang classes doc'd in `ai/conc-anal/subint_*_issue.md`.

Deats,
- Module-level `pytestmark` on full-file-hanging suites:
  - `tests/test_cancellation.py`
  - `tests/test_inter_peer_cancellation.py`
  - `tests/test_pubsub.py`
  - `tests/test_shm.py`
- Per-test decorator where only one test in the file
  hangs:
  - `tests/discovery/test_registrar.py
    ::test_stale_entry_is_deleted` — replaces the
    inline `if start_method == 'subint': pytest.skip`
    branch with a declarative skip.
  - `tests/test_subint_cancellation.py
    ::test_subint_non_checkpointing_child`.
- A few per-test decorators are left commented-in-
  place as breadcrumbs for later finer-grained unskips.

Also, some nearby tidying in the affected files:
- Annotate loose fixture / test params
  (`pytest.FixtureRequest`, `str`, `tuple`, `bool`) in
  `tests/conftest.py`, `tests/devx/conftest.py`, and
  `tests/test_cancellation.py`.
- Normalize `"""..."""` → `'''...'''` docstrings per
  repo convention on a few touched tests.
- Add `timeout=6` / `timeout=10` to
  `@tractor_test(...)` on `test_cancel_infinite_streamer`
  and `test_some_cancels_all`.
- Drop redundant `spawn_backend` param from
  `test_cancel_via_SIGINT`; use `start_method` in the
  `'mp' in ...` check instead.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 4b2a0886c3)
(factored: dropped spawn-backend-only path: tests/test_subint_cancellation.py)
2026-06-17 17:39:44 -04:00
Gud Boi 239c3fb14d Add `skipon_spawn_backend` pytest marker
A reusable `@pytest.mark.skipon_spawn_backend( '<backend>' [, ...],
reason='...')` marker for backend-specific known-hang / -borked cases
— avoids scattering `@pytest.mark.skipif(lambda ...)` branches across
tests that misbehave under a particular `--spawn-backend`.

Deats,
- `pytest_configure()` registers the marker via
  `addinivalue_line('markers', ...)`.
- New `pytest_collection_modifyitems()` hook walks
  each collected item with `item.iter_markers(
  name='skipon_spawn_backend')`, checks whether the
  active `--spawn-backend` appears in `mark.args`, and
  if so injects a concrete `pytest.mark.skip(
  reason=...)`. `iter_markers()` makes the decorator
  work at function, class, or module (`pytestmark =
  [...]`) scope transparently.
- First matching mark wins; default reason is
  `f'Borked on --spawn-backend={backend!r}'` if the
  caller doesn't supply one.

Also, tighten type annotations on nearby `pytest`
integration points — `pytest_configure`, `debug_mode`,
`spawn_backend`, `tpt_protos`, `tpt_proto` — now taking
typed `pytest.Config` / `pytest.FixtureRequest` params.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 3b26b59dad)
2026-06-17 17:39:44 -04:00
Gud Boi 6afeb6b6ef Skip `test_stale_entry_is_deleted` hanger with `subint`s
(cherry picked from commit 985ea76de5)
2026-06-17 17:39:44 -04:00
Gud Boi b2cc4f502c Wall-cap `test_stale_entry_is_deleted` via `pytest-timeout`
Add a hard process-level wall-clock bound on a test
known to wedge un-Ctrl-C-ably under an in-dev spawn
backend, so an unattended suite run can't hang
indefinitely.

Deats,
- New `testing` dep: `pytest-timeout>=2.3`.
- `test_stale_entry_is_deleted`:
  `@pytest.mark.timeout(3, method='thread')`. The
  `method='thread'` choice is deliberate —
  `method='signal'` routes via `SIGALRM` which can be
  starved by the same GIL-hostage path that drops
  `SIGINT`, so it'd never actually fire in the
  starvation case.

At timeout, `pytest-timeout` hard-kills the pytest
process itself — that's the intended behavior here;
the alternative is the suite never returning.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 189f4e3f72e9f1eda5d24bcbab5743f7e35bd913)
(factored: kept pyproject + tests/discovery/test_registrar.py parts of
 "Wall-cap `subint` audit tests via `pytest-timeout`"; dropped
 tests/test_subint_cancellation.py)
2026-06-17 17:39:44 -04:00
Gud Boi fd4caa1903 Arm `dump_on_hang` on `test_stale_entry_is_deleted`
Wrap the test's `trio.run(main)` in
`dump_on_hang(seconds=20)` so any future hang
regression captures a stack dump for triage instead
of wedging CI silently; under the default backends
it's a no-op safety net.

Includes a "KNOWN ISSUE" comment block documenting
the (future) `subint` backend hang classes observed
against this test during Phase B bringup (#379).

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 4a3254583b)
(factored: kept only the tests/discovery/test_registrar.py part of
 "Doc `subint` backend hang classes + arm `dump_on_hang`"; dropped
 subint conc-anal docs + tests/test_subint_cancellation.py)
2026-06-17 17:39:44 -04:00
Gud Boi 253b210bcd Add `._debug_hangs` to `.devx` for hang triage
Bottle up the diagnostic primitives that actually cracked the
silent mid-suite hangs in the `subint` spawn-backend bringup (issue
there" session has them on the shelf instead of reinventing from
scratch.

Deats,
- `dump_on_hang(seconds, *, path)` — context manager wrapping
  `faulthandler.dump_traceback_later()`. Critical gotcha baked in:
  dumps go to a *file*, not `sys.stderr`, bc pytest's stderr
  capture silently eats the output and you can spend an hour
  convinced you're looking at the wrong thing
- `track_resource_deltas(label, *, writer)` — context manager
  logging per-block `(threading.active_count(),
  len(_interpreters.list_all()))` deltas; quickly rules out
  leak-accumulation theories when a suite progressively worsens (if
  counts don't grow, it's not a leak, look for a race on shared
  cleanup instead)
- `resource_delta_fixture(*, autouse, writer)` — factory returning
  a `pytest` fixture wrapping `track_resource_deltas` per-test; opt
  in by importing into a `conftest.py`. Kept as a factory (not a
  bare fixture) so callers own `autouse` / `writer` wiring

Also,
- export the three names from `tractor.devx`
- dep-free on py<3.13 (swallows `ImportError` for `_interpreters`)
- link back to the provenance in the module docstring (issue #379 /
  commit `26fb820`)

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 09466a1e9d)
2026-06-17 17:39:44 -04:00
Gud Boi f7dfd37df4 Extract `_actor_child_main()` as shared child entry
Pull the `_child.py` `__main__` block body out into
a callable `_actor_child_main()` so alternate spawn
backends can bootstrap a subactor without going
through the CLI entrypoint.

Deats,
- new `_actor_child_main(uid, loglevel, parent_addr,
  infect_asyncio, spawn_method='trio')` holds the
  full child-side runtime startup previously inlined
  under `if __name__ == '__main__':`
- `__main__` block reduces to arg-parsing + a call
  into the new func
- add `"subint"` to the `_runtime.py` spawn-method
  check so a child accepts `SpawnSpec` from that
  (future) backend; inert str-compare w/o it

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit b8f243e98d)
(factored: kept only the `_child.py`/`_runtime.py` entry-extraction parts of
 "Impl min-viable `subint` spawn backend (B.2)"; dropped
 tractor/spawn/_subint.py + subint prompt-io logs)
2026-06-17 17:39:44 -04:00
Gud Boi 4b63b7f1cc Handle py3.14+ incompats as test skips
Since we're devving subints we require the 3.14+ stdlib API
and a couple compiled libs don't support it yet, namely:
- `cffi`, which we're only using for the `.ipc._linux` eventfd
  stuff (now factored into `hotbaud` anyway).
- `greenback`, which requires `greenlet` which doesn't seem to be
  wheeled yet
  * on nixos the sdist build was failing due to lack of `g++` which
    i don't care to figure out rn since we don't need `.devx` stuff
    immediately for this subints prototype.
  * [ ] we still need to adjust any dependent suites to skip.

Adjust `test_ringbuf` to skip on import failure.

Also project wide,
- pin us to py 3.13+ in prep for last-2-minor-version policy.
- drop `msgspec>=0.20.0`, the first release with py3.14 support.

(cherry picked from commit d2ea8aa2de)
2026-06-17 17:39:44 -04:00
Gud Boi 54a694d452 Open py-version range + harness gate for py3.14 backends (#379)
Prep for a future sub-interpreter (PEP 734
`concurrent.interpreters`) spawn backend per issue
test-harness error-gating; the backend itself comes
later.

Deats,
- bump `pyproject.toml` `requires-python` to
  `>=3.12, <3.15` and list the `3.14` classifier —
  the new stdlib `concurrent.interpreters` module
  only ships on 3.14
- `_testing.pytest.pytest_configure` wraps
  `try_set_start_method()` in a `pytest.UsageError`
  handler so an unsupported `--spawn-backend` on the
  running py-version prints a clean banner instead
  of a traceback

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit d318f1f8f4)
(factored: kept only the pyproject + `_testing/pytest.py` parts of
 "Add `'subint'` spawn backend scaffold (#379)"; dropped
 tractor/spawn/_spawn.py + tractor/spawn/_subint.py)
2026-06-17 17:39:44 -04:00
Bd eb3e2b3574
Merge pull request #461 from goodboy/tooling_skills_n_config_from_mtf_dev
Tooling skills n config from mtf dev
2026-06-17 17:33:28 -04:00
Gud Boi 26f2b23da2 Address Copilot review nits on PR #461 (round 2)
Apply the genuinely-valid items from the latest GH Copilot
pass; the rest are house-style/local-convenience and get a
reply instead,

- drop the stray space before the colon in
  `test_no_runtime`'s `pytest.raises(...)` (one-off typo).
- drop the unused `cffi` binding in `test_ringbuf` — the
  `importorskip` side-effect is all we need.
- declare the `run-tests` skill's process-cleanup commands
  in `allowed-tools` (`ss`/`pgrep`/`pkill`/`sleep` + a
  scoped `rm -f /tmp/registry@*.sock`).

Review: PR #461 (copilot-pull-request-reviewer)
https://github.com/goodboy/tractor/pull/461#pullrequestreview-4519551472

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-17 17:16:36 -04:00
Gud Boi c0fdfa4c7b Pin sdist-install step to py3.13
The `sdist` CI job's install step ran bare `python -m pip
install`, picking up the runner's default py3.12 — which our
bumped `requires-python = ">=3.13"` now rejects
(`3.12.3 not in '<3.15,>=3.13'`).

- install into a `uv venv --python 3.13` so it matches the
  build step's `--python=3.13`.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-17 16:25:41 -04:00
Gud Boi c5feeac4ce Pin `pytest>=9.0.3` for CVE-2025-71176 floor
Tighten the test-dep floor from `>=9.0` to `>=9.0.3` so a
fresh `uv` relock can't resolve back into the vulnerable
9.0.0–9.0.2 window; 9.0.3 is where upstream patched the
insecure-tmpdir advisory (CVE-2025-71176).

- annotate the constraint w/ the CVE id for future readers.
- update the existing bump-rationale comment to name the
  precise patched version.

(this commit-msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-17 16:05:41 -04:00
Gud Boi 9f1a64fcf7 Relock `uv.lock` for py3.13+ & `pytest` CVE
Regenerate the lockfile to match the bumped
`requires-python = ">=3.13, <3.15"` so the shipped venv
state lines up with `pyproject.toml`,

- drop the now-unsupported `cp312` wheels.
- add `cp314` wheels + py3.14 `resolution-markers`.
- bump the lock's own `requires-python` envelope.

The relock also pulls `pytest` 8.3.5 -> 9.1.0 which
resolves the insecure-tmpdir advisory (CVE-2025-71176,
patched in 9.0.3); this closes dependabot alert #3 and
supersedes the bot's standalone bump in #442.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-17 15:56:19 -04:00
Gud Boi 6b0cb17c04 Address Copilot review nits on PR #461
Per the GH Copilot review, clean a few metadata/comment
inconsistencies that would otherwise live on `main`,

- drop the `3.12` Trove classifier since `requires-python`
  is now `>=3.13`.
- remove the `ci.yml` comment referencing the not-yet-landed
  `--spawn-method=main_thread_forkserver` (misleading on
  `main`, and leaks the MTF name early).
- fix the `//tmp` double-slash typo in the local `Read()`
  permission pattern.

Review: PR #461 (copilot-pull-request-reviewer)
https://github.com/goodboy/tractor/pull/461#pullrequestreview-4510034938

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-06-17 15:47:07 -04:00
Gud Boi dd651943d1 Add `logspec` leaf-mod Route B follow-up doc
Follow-up note documenting why the deeper "Route B" fix
for `LogSpec`/`apply_logspec()` true per-leaf-MODULE level
control was NOT taken — in favor of the smaller
sub-PACKAGE fix that shipped in 9c36363b.

Doc covers,
- Status: what 9c36363b already gives (per-sub-pkg
  control at any nesting depth, `devx.debug` ≠ `devx`)
  vs. what remains unaddressed (per-leaf-mod levels,
  top-level lib mods like `tractor.to_asyncio` on the
  root logger).
- "Route B" sketch: make logger *identity* the full
  dotted module path; mv the cosmetic leaf-trim out of
  logger-naming into the *formatter's* `{name}`
  rendering.
- 6 breaking-change costs: every logger name changes,
  formatter rewrite, propagation/double-emit surface
  grows, level-inheritance semantics shift,
  `modden`/`piker` contract churn, `get_logger()`
  refactor risk.
- Migration plan if pursued: extract a pure
  `_mk_logger_name()` helper w/ an exhaustive name-shape
  test matrix, swap `get_logger()` to use it for
  identity, swap formatter to use the display string,
  golden-diff rendered headers, coordinate w/
  downstreams.
- "Route A" alternative: a `logging.Filter` keyed on
  `record.module`/`pathname` for per-leaf control w/o
  name churn — lower risk, narrower power.
- Recommendation: defer Route B; prefer Route A if
  per-leaf is needed soon; the shipped sub-PKG fix
  covers the common ask.

Lives under `ai/tooling-todos/` since it's a deferred-
work decision record, not a triage/conc-anal doc.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 5b3c2e3762)
(cherry picked from commit 232d6ccfbf)
2026-06-17 15:47:07 -04:00
Gud Boi cec0731253 Mk `test_no_runtime()` not require `pytest-trio`
(cherry picked from commit 79dda4cb4a)
(cherry picked from commit f2871ae3ff)
2026-06-17 15:47:07 -04:00
Gud Boi 9defec4922 Bump to latest `pytest` release!
(cherry picked from commit e329c3108c)
(cherry picked from commit 8d917cbf87)
2026-06-17 13:45:31 -04:00
Gud Boi 0a3772c0c1 Add `RuntimeVars` env-var lift design plan
Draft plan for consolidating pytest CLI flags,
ad-hoc env vars, and hardcoded fixture defaults
into the existing (but unused) `RuntimeVars`
struct as the single source of truth.

Deats,
- `_rtvars.py` leaf mod w/ `dump`/`load`/`get`/
  `update` helpers using `str(dict)` +
  `ast.literal_eval` encoding
- phased migration: test infra first, then
  runtime callers, then per-session bindspace
- addresses concurrent pytest session collisions
  and subproc env propagation for `devx/` scripts

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 7882c37ce0)
(cherry picked from commit 79c189bddd)
2026-06-17 13:45:31 -04:00
Gud Boi 3f18caaaac Add `test_register_duplicate_name` race analysis
Document the intermittent connect-refused failure in the registrar
daemon test — root cause is the `daemon` fixture's blind `time.sleep()`
readiness gate racing against the subproc's `bind()`/ `listen()`
completion. Distinct from the cancel- cascade `TooSlowError` flake
class.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 29f9928524)
(cherry picked from commit 15c3b670e1)
2026-06-17 13:45:31 -04:00
Gud Boi 3dc3f8401c Flip back to default `pytest` capture for CI
(cherry picked from commit 22cdf15b73)
(cherry picked from commit 781abf7558)
2026-06-17 13:45:31 -04:00
Gud Boi 8c6b7fdb31 Add posix-multithreaded-`fork()` explainer doc
(cherry picked from commit 532a9834f3)
(cherry picked from commit 23b8a80e15)
2026-06-17 13:45:31 -04:00
Gud Boi 4e9f392ca3 Drop global `pytest-timeout` cap from `pyproject.toml`
`timeout = 200` was firing via SIGALRM (the default
`method='signal'`) which synchronously raises `Failed` in
trio's main thread mid-`epoll.poll()`, abandoning trio's
runner mid-flight and leaving `GLOBAL_RUN_CONTEXT` half-
installed. EVERY subsequent `trio.run()` in the same pytest
session then bails with
`RuntimeError: Attempted to call run() from inside a run()`.

Empirical impact: a session that hits a single 200s hang
cascades into 30-40 false-positive failures across every
downstream test file that uses `trio.run`. Recent UDS run
saw 1 real timeout (`test_unregistered_err_still_relayed`)
poison 38 sibling tests with cascade-fails — a debugging
nightmare.

Same architectural bug we already documented in
`tests/test_advanced_streaming.py::test_dynamic_pub_sub`
(see its module-level NOTE) — both `pytest-timeout`
enforcement modes are incompatible with trio under fork-
based spawn backends. Now scoped session-wide.

For tests that legitimately need a wall-clock cap, the
canonical pattern is `with trio.fail_after(N):` INSIDE the
test — trio's own `Cancelled` machinery cleanly unwinds
the actor nursery without disturbing global state.

For CI: rely on job-level wall-clock timeouts (e.g. GitHub
Actions `timeout-minutes`) to abort genuinely-stuck suites.

`pyproject.toml` comment block spells this all out so a
future contributor doesn't reach back for `timeout =` and
re-introduce the bug.

ALSO, bump `xonsh` to at least `0.23.0` release.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 3c366cac13)
(cherry picked from commit bbf4fe66e3)
2026-06-17 13:45:31 -04:00
Gud Boi cda94235f4 Codify capture-pipe hang lesson in skills
Encode the hard-won lesson from the forkserver
cancel-cascade investigation into two skill docs
so future sessions grep-find it before spelunking
into trio internals.

Deats,
- `.claude/skills/conc-anal/SKILL.md`:
  - new "Unbounded waits in cleanup paths"
    section — rule: bound every `await X.wait()`
    in cleanup paths with `trio.move_on_after()`
    unless the setter is unconditionally
    reachable. Recent example:
    `ipc_server.wait_for_no_more_peers()` in
    `async_main`'s finally (was unbounded,
    deadlocked when any peer handler stuck)
  - new "The capture-pipe-fill hang pattern"
    section — mechanism, grep-pointers to the
    existing `conftest.py` guards (`tests/conftest
    .py:258`, `:316`), cross-ref to the full
    post-mortem doc, and the grep-note: "if a
    multi-subproc tractor test hangs, `pytest -s`
    first, conc-anal second"
- `.claude/skills/run-tests/SKILL.md`: new
  "Section 9: The pytest-capture hang pattern
  (CHECK THIS FIRST)" with symptom / cause /
  pre-existing guards to grep / three-step debug
  recipe (try `-s`, lower loglevel, redirect
  stdout/stderr) / signature of this bug vs. a
  real code hang / historical reference

Cost several investigation sessions before the
capture-pipe issue surfaced — it was masked by
deeper cascade deadlocks. Once the cascades were
fixed, the tree tore down enough to generate
pipe-filling log volume. Lesson: **grep this
pattern first when any multi-subproc tractor test
hangs under default pytest but passes with `-s`.**

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 4106ba73ea)
(cherry picked from commit 45c442060b)
2026-06-17 13:45:31 -04:00
Gud Boi a26619dfea Claude-perms: ensure /commit-msg files can be written!
(cherry picked from commit 76d12060aa)
(cherry picked from commit 1d70d33d9a)
2026-06-17 13:45:31 -04:00
Gud Boi 974cb159e8 Use SIGINT-first ladder in `run-tests` cleanup
The previous cleanup recipe went straight to
SIGTERM+SIGKILL, which hides bugs: tractor is
structured concurrent — `_trio_main` catches SIGINT
as an OS-cancel and cascades `Portal.cancel_actor`
over IPC to every descendant. So a graceful SIGINT
exercises the actual SC teardown path; if it hangs,
that's a real bug to file (the forkserver `:1616`
zombie was originally suspected to be one of these
but turned out to be a teardown gap in
`_ForkedProc.kill()` instead).

Deats,
- step 1: `pkill -INT` scoped to `$(pwd)/py*` — no
  sleep yet, just send the signal
- step 2: bounded wait loop (10 × 0.3s = ~3s) using
  `pgrep` to poll for exit. Loop breaks early on
  clean exit
- step 3: `pkill -9` only if graceful timed out, w/
  a logged escalation msg so it's obvious when SC
  teardown didn't complete
- step 4: same SIGINT-first ladder for the rare
  `:1616`-holding zombie that doesn't match the
  cmdline pattern (find PID via `ss -tlnp`, then
  `kill -INT NNNN; sleep 1; kill -9 NNNN`)
- steps 5-6: UDS-socket `rm -f` + re-verify
  unchanged

Goal: surface real teardown bugs through the test-
cleanup workflow instead of papering over them with
`-9`.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 70d58c4bd2)
(cherry picked from commit 222784ccc8)
2026-06-17 13:45:31 -04:00
Gud Boi c170007ddc Add zombie-actor check to `run-tests` skill
Fork-based backends (esp. `subint_forkserver`) can
leak child actor processes on cancelled / SIGINT'd
test runs; the zombies keep the tractor default
registry (`127.0.0.1:1616` / `/tmp/registry@1616.sock`)
bound, so every subsequent session can't bind and
50+ unrelated tests fail with the same
`TooSlowError` / "address in use" signature. Document
the pre-flight + post-cancel check as a mandatory
step 4.

Deats,
- **primary signal**: `ss -tlnp | grep ':1616'` for a
  bound TCP registry listener — the authoritative
  check since :1616 is unique to our runtime
- `pgrep -af` scoped to `$(pwd)/py[0-9]*/bin/python.*
  _actor_child_main|subint-forkserv` for leftover
  actor/forkserver procs — scoped deliberately so we
  don't false-flag legit long-running tractor-
  embedding apps like `piker`
- `ls /tmp/registry@*.sock` for stale UDS sockets
- scoped cleanup recipe (SIGTERM + SIGKILL sweep
  using the same `$(pwd)/py*` pattern, UDS `rm -f`,
  re-verify) plus a fallback for when a zombie holds
  :1616 but doesn't match the pattern: `ss -tlnp` →
  kill by PID
- explicit false-positive warning calling out the
  `piker` case (`~/repos/piker/py*/bin/python3 -m
  tractor._child ...`) so a bare `pgrep` doesn't lead
  to nuking unrelated apps

Goal: short-circuit the "spelunking into test code"
rabbit-hole when the real cause is just a leaked PID
from a prior session, without collateral damage to
other tractor-embedding projects on the same box.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit d093c31979)
(cherry picked from commit 1ebe15db3b)
2026-06-17 13:45:31 -04:00
Gud Boi fc3ab9561b Add global 200s `pytest-timeout`
(cherry picked from commit 5998774535)
(cherry picked from commit 19dd6fc739)
2026-06-17 13:45:31 -04:00
Gud Boi e5d4d9494b Import-or-skip `.devx.` tests requiring `greenback`
Which is for sure true on py3.14+ rn since `greenlet` didn't want to
build for us (yet).

(cherry picked from commit d6e70e9de4)
(cherry picked from commit ba2e474d9d)
2026-06-17 11:53:12 -04:00
Gud Boi 47ac8c0fef Avoid skip `.ipc._ringbuf` import when no `cffi`
(cherry picked from commit 03bf2b931e)
(cherry picked from commit 9157f58c15)
2026-06-17 11:03:12 -04:00
Gud Boi ba111b3042 Split py-version-gated uv dependency-groups
Reshuffle `pyproject.toml` deps into per-python-version
`[tool.uv.dependency-groups]`:
- `subints` group: `msgspec>=0.21.0`, py>=3.14
- `eventfd` group: `cffi>=1.17.1`, py>=3.13,<3.14
- `sync_pause` group: `greenback`, py>=3.13,<3.14
  (was in `devx`; moved out bc no 3.14 yet)

Bump top-level `msgspec>=0.20.0` too.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 34d9d482e4)
(factored: kept only the pyproject dep-group parts of
 "Raise `subint` floor to py3.14 and split dep-groups"; dropped
 tractor/spawn/_spawn.py + tractor/spawn/_subint.py)
(cherry picked from commit ab6796dd45)
2026-06-16 14:21:08 -04:00
Gud Boi e8e4657e6d Pin `xonsh` to GH `main` in editable mode
(cherry picked from commit 64ddc42ad8)
(cherry picked from commit c4951c86ec)
2026-06-16 14:21:08 -04:00
Gud Boi c72dc8c235 Bump `xonsh` to latest pre `0.23` release
(cherry picked from commit b524ee4633)
(cherry picked from commit 83dab8e24e)
2026-06-16 14:21:08 -04:00
Gud Boi fae2525461 Expand `/run-tests` venv pre-flight to cover all cases
Rework section 3 from a worktree-only check into a
structured 3-step flow: detect active venv, interpret
results (Case A: active, B: none, C: worktree), then
run import + collection checks.

Deats,
- Case B prompts via `AskUserQuestion` when no venv
  is detected, offering `uv sync` or manual activate
- add `uv run` fallback section for envs where venv
  activation isn't practical
- new allowed-tools: `uv run python`, `uv run pytest`,
  `uv pip show`, `AskUserQuestion`

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit b1a0753a3f)
(cherry picked from commit 2dab0fdcc8)
2026-06-16 14:21:08 -04:00
Gud Boi dca79670aa Add `lastfailed` cache inspection to `/run-tests` skill
New "Inspect last failures" section reads the pytest
`lastfailed` cache JSON directly — instant, no
collection overhead, and filters to `tests/`-prefixed
entries to avoid stale junk paths.

Also,
- add `jq` tool permission for `.pytest_cache/` files

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit ba86d482e3)
(cherry picked from commit 2cad99f0bd)
2026-06-16 14:21:08 -04:00
Gud Boi bca7f10915 Reorganize `.gitignore` by skill/purpose
Group `.claude/` ignores per-skill instead of a
flat list: `ai.skillz` symlinks, `/open-wkt`,
`/code-review-changes`, `/pr-msg`, `/commit-msg`.
Add missing symlink entries (`yt-url-lookup` ->
`resolve-conflicts`, `inter-skill-review`). Drop
stale `Claude worktrees` section (already covered
by `.claude/wkts/`).

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit d3d6f646f9)
(cherry picked from commit 82aa0cca73)
2026-06-16 14:21:08 -04:00
Gud Boi 61f9b62d04 Ignore notes & snippets subdirs in `git`
(cherry picked from commit 9cf3d588e7)
(cherry picked from commit 9a7b595029)
2026-06-16 14:21:08 -04:00
Bd e75e29b1dc
Merge pull request #444 from goodboy/spawn_modularize
Spawner modules: split up subactor spawning  backends
2026-04-23 18:42:33 -04:00
Gud Boi a7b1ee34ef Restore fn-arg `_runtime_vars` in `trio_proc` teardown
During the Phase A extraction of `trio_proc()` out of
`spawn._spawn` into its own submod, the
`debug.maybe_wait_for_debugger(child_in_debug=...)` call site in
the hard-reap `finally` got refactored from the original
`_runtime_vars.get('_debug_mode', ...)` (the fn parameter — the
dict that was constructed by the *parent* for the *child*'s
`SpawnSpec`) to `get_runtime_vars().get(...)` (a global getter that
returns the *parent's* live `_state`). Those are semantically
different — the first asks "is the child we just spawned in debug
mode?", the second asks "are *we* in debug mode?". Under
mixed-debug-mode trees the swap can incorrectly skip (or
unnecessarily delay) the debugger-lock wait during teardown.

Revert to the fn-parameter lookup and add an inline `NOTE` comment
calling out the distinction so it's harder to regress again.

Deats,
- `spawn/_trio.py`: `child_in_debug=get_runtime_vars().get(...)` →
  `child_in_debug=_runtime_vars.get(...)` at the
  `debug.maybe_wait_for_debugger(...)` call in the hard-reap block;
  add 4-line `NOTE` explaining the parent-vs-child distinction.
- `spawn/__init__.py`: drop trailing whitespace after the
  `'mp_forkserver'` docstring bullet.
- `ai/prompt-io/prompts/subints_spawner.md`: drop duplicated `with`
  in `"as with with subprocs"` prose (copilot grammar catch).

Review: PR #444 (Copilot)
https://github.com/goodboy/tractor/pull/444#pullrequestreview-4165928469

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-23 18:30:11 -04:00
Gud Boi ae5b63c0bc Bump to `msgspec>=0.21.0` in lock file 2026-04-17 19:28:11 -04:00
Gud Boi f75865fb2e Tidy `spawn/` subpkg docstrings and imports
Drop unused `TYPE_CHECKING` imports (`Channel`,
`_server`), remove commented-out `import os` in
`_entry.py`, and use `get_runtime_vars()` accessor
instead of bare `_runtime_vars` in `_trio.py`.

Also,
- freshen `__init__.py` layout docstring for the
  new per-backend submod structure
- update `_spawn.py` + `_trio.py` module docstrings

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-17 19:03:00 -04:00
Gud Boi e0b8f23cbc Add prompt-io files for "phase-A", fix typos caught by copilot 2026-04-17 18:26:41 -04:00
Gud Boi 8d662999a4 Bump to `msgspec>=0.21` for py314 support 2026-04-17 16:54:07 -04:00
Gud Boi d7ca68cf61 Mv `trio_proc`/`mp_proc` to per-backend submods
Split the monolithic `spawn._spawn` into a slim
"core" + per-backend submodules so a future
`._subint` backend (per issue #379) can drop in
without piling more onto `_spawn.py`.

`._spawn` retains the cross-backend supervisor
machinery: `SpawnMethodKey`, `_methods` registry,
`_spawn_method`/`_ctx` state, `try_set_start_method()`,
the `new_proc()` dispatcher, and the shared helpers
`exhaust_portal()`, `cancel_on_completion()`,
`hard_kill()`, `soft_kill()`, `proc_waiter()`.

Deats,
- mv `trio_proc()` → new `spawn._trio`
- mv `mp_proc()` → new `spawn._mp`, reads `_ctx` and
  `_spawn_method` via `from . import _spawn` for
  late binding bc both get mutated by
  `try_set_start_method()`
- `_methods` wires up the new submods via late
  bottom-of-module imports to side-step circular
  dep (both backend mods pull shared helpers from
  `._spawn`)
- prune now-unused imports from `_spawn.py` — `sys`,
  `is_root_process`, `current_actor`,
  `is_main_process`, `_mp_main`, `ActorFailure`,
  `pretty_struct`, `_pformat`

Also,
- `_testing.pytest.pytest_generate_tests()` now
  drives the valid-backend set from
  `typing.get_args(SpawnMethodKey)` so adding a
  new backend (e.g. `'subint'`) doesn't require
  touching the harness
- refresh `spawn/__init__.py` docstring for the
  new layout

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-17 16:48:22 -04:00
Gud Boi b5b0504918 Add prompt-IO log for subint spawner design kickoff
Log the `claude-opus-4-7` design session that produced the phased plan
(A: modularize `_spawn`, B: `_subint` backend, C: harness) and concrete
Phase A file-split for #379. Substantive bc the plan directly drives
upcoming impl.

Prompt-IO: ai/prompt-io/claude/20260417T034918Z_9703210_prompt_io.md

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-17 16:48:22 -04:00
Gud Boi de78a6445b Initial prompt to vibe subint support Bo 2026-04-17 16:48:18 -04:00
Bd 5c98ab1fb6
Merge pull request #429 from goodboy/multiaddr_support
Multiaddresses: a novel `libp2p` peep's idea worth embracing
2026-04-16 23:59:11 -04:00
Gud Boi 3867403fab Scale `test_open_local_sub_to_stream` timeout by CPU factor
Import and apply `cpu_scaling_factor()` from
`conftest`; bump base from 3.6 -> 4 and multiply
through so CI boxes with slow CPUs don't flake.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-16 20:03:32 -04:00
Gud Boi 7c8e5a6732 Drop `snippets/multiaddr_ex.py` scratch script
Since we no longer need the example after integrating `multiaddr` into
the `.discovery` subsys.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-16 17:45:38 -04:00
Gud Boi 3152f423d8 Condense `.raw.md` prompt-IO logs, add `diff_cmd` refs
Replace verbose inline code dumps in `.raw.md`
entries with terse summaries and `git diff`
cmd references. Add `diff_cmd` metadata to each
entry's YAML frontmatter so readers can reproduce
the actual output diff.

Also,
- rename `multiaddr_declare_eps.md_` -> `.md`
  (drop trailing `_` suffix)

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-16 17:44:14 -04:00
Gud Boi ed65301d32 Fix misc bugs caught by Copilot review
Deats,
- use `proc.poll() is None` in `sig_prog()` to
  distinguish "still running" from exit code 0;
  drop stale `breakpoint()` from fallback kill
  path (would hang CI).
- add missing `raise` on the `RuntimeError` in
  `async_main()` when no tpt bind addrs given.
- clean up stale uid entries from the registrar
  `_registry` when addr eviction empties the
  addr list.
- update `discovery.__init__` docstring to match
  the new eager `._multiaddr` import.
- fix `registar` -> `registrar` typo in teardown
  report log msg.

Review: PR #429 (Copilot)
https://github.com/goodboy/tractor/pull/429

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-14 19:54:15 -04:00
Gud Boi 8817032c90 Prefer fresh conn for unreg, fallback to `_parent_chan`
The prior approach eagerly reused `_parent_chan` when
parent IS the registrar, but that channel may still
carry ctx/stream teardown protocol traffic —
concurrent `unregister_actor` RPC causes protocol
conflicts. Now try a fresh `get_registry()` conn
first; only fall back to the parent channel on
`OSError` (listener already closed/unlinked).

Deats,
- fresh `get_registry()` is the primary path for
  all addrs regardless of `parent_is_reg`
- `OSError` handler checks `parent_is_reg` +
  `rent_chan.connected()` before fallback
- fallback catches `OSError` and
  `trio.ClosedResourceError` separately
- drop unused `reg_addr: Address` annotation

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-14 19:54:15 -04:00
Gud Boi 70dc60a199 Bump UDS `listen()` backlog 1 -> 128 for multi-actor unreg
A backlog of 1 caused `ECONNREFUSED` when multiple
sub-actors simultaneously connect to deregister from
a remote-daemon registrar. Now matches the TCP
transport's default backlog (~128).

Also,
- add cross-ref comments between
  `_uds.close_listener()` and `async_main()`'s
  `parent_is_reg` deregistration path explaining
  the UDS socket-file lifecycle

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-14 19:54:15 -04:00
Gud Boi cd287c7e93 Fix `test_registrar_merge_binds_union` for UDS collision
`get_random()` can produce the same UDS filename for a given
pid+actor-state, so the "disjoint addrs" premise doesn't always hold.
Gate the `len(bound) >= 2` assertion on whether the registry and bind
addrs actually differ via `expect_disjoint`.

Also,
- drop unused `partial` import

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-14 19:54:15 -04:00
Gud Boi 7b04b2cdfc Reuse `_parent_chan` to unregister from parent-registrar
When the parent actor IS the registrar, reuse the existing parent
channel for `unregister_actor` RPC instead of opening a new connection
via `get_registry()`. This avoids failures when the registrar's listener
socket is already closed during teardown (e.g. UDS transport unlinks the
socket file rapidly).

Deats,
- detect `parent_is_reg` by comparing `_parent_chan.raddr` against
  `reg_addrs` and if matched, create a `Portal(rent_chan)` directly
  instead of `async with get_registry()`.
- rename `failed` -> `failed_unreg` for clarity.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-14 19:54:14 -04:00
Gud Boi 75b07c4b7c Show trailing bindspace-path-div in `repr(UDSAddress)` 2026-04-14 19:54:14 -04:00
Gud Boi 86d4e0d3ed Harden `sig_prog()` retries, adjust debugger test timeouts
Retry signal delivery in `sig_prog()` up to `tries`
times (default 3) w/ `canc_timeout` sleep between
attempts; only fall back to `_KILL_SIGNAL` after all
retries exhaust. Bump default timeout 0.1 -> 0.2.

Also,
- `test_multi_nested_subactors_error_through_nurseries`
  gives the first prompt iteration a 5s timeout even
  on linux bc the initial crash sequence can be slow
  to arrive at a `pdb` prompt

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-14 19:54:14 -04:00
Gud Boi ccb013a615 Add `prefer_addr()` transport selection to `_api`
New locality-aware addr preference for multihomed
actors: UDS > local TCP > remote TCP. Uses
`ipaddress` + `socket.getaddrinfo()` to detect
whether a `TCPAddress` is on the local host.

Deats,
- `_is_local_addr()` checks loopback or
  same-host IPs via interface enumeration
- `prefer_addr()` classifies an addr list into
  three tiers and picks the latest entry from
  the highest-priority non-empty tier
- `query_actor()` and `wait_for_actor()` now
  call `prefer_addr()` instead of grabbing
  `addrs[-1]` or a single pre-selected addr

Also,
- `Registrar.find_actor()` returns full
  `list[UnwrappedAddress]|None` so callers can
  apply transport preference

Prompt-IO: ai/prompt-io/claude/20260414T163300Z_befedc49_prompt_io.md

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-14 19:54:14 -04:00
Gud Boi c3d6cc9007 Rename `discovery._discovery` to `._api`
Adjust all imports to match.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-14 19:54:14 -04:00
Gud Boi cb7b76c44f Use multi-addr `dict` registry, drop `bidict`
Replace `Registrar._registry: bidict[uid, addr]`
with `dict[uid, list[UnwrappedAddress]]` to
support actors binding on multiple transports
simultaneously (multi-homed).

Deats,
- `find_actor_addr()` returns first addr from
  the uid's list
- `get_registry()` now returns per-uid addr
  lists
- `find_actor_addrs()` uses `.extend()` to
  collect all addrs for a given actor name
- `register_actor_addr()` appends to the uid's
  list (dedup'd) and evicts stale entries where
  a different uid claims the same addr
- `delete_actor_addr()` does a linear scan +
  `.remove()` instead of `bidict.inverse.pop()`;
  deletes the uid entry entirely when no addrs
  remain

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-14 19:54:14 -04:00
Gud Boi 23677f8a3c Use distinct startup report for registrar vs client
Set `header` to "Contacting existing registry"
for non-registrar actors and "Opening new
registry" for registrars, so the boot log
reflects the actual role.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-14 19:54:14 -04:00
Gud Boi 06ff2dd5f2 Permit the `prompt-io` skill by default 2026-04-14 19:54:14 -04:00
Gud Boi a891e003b2 Expose `_multiaddr` API from `tractor.discovery`
Re-export `parse_endpoints`, `parse_maddr`, and
`mk_maddr` in `discovery.__init__` so downstream
(piker) can import directly from the pkg ns.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-14 19:54:14 -04:00
Gud Boi e90241baaa Add `parse_endpoints()` to `_multiaddr`
Provide a service-table parsing API for downstream projects (like
`piker`) to declare per-actor transport bind addresses as a config map
of actor-name -> multiaddr strings (e.g. from a TOML `[network]`
section).

Deats,
- `EndpointsTable` type alias: input `dict[str, list[str|tuple]]`.
- `ParsedEndpoints` type alias: output `dict[str, list[Address]]`.
- `parse_endpoints()` iterates the table and delegates each entry to the
  existing `tractor.discovery._discovery.wrap_address()` helper, which
  handles maddr strings, raw `(host, port)` tuples, and pre-wrapped
  `Address` objs.
- UDS maddrs use the multiaddr spec name `/unix/...` (not tractor's
  internal `/uds/` proto_key)

Also add new tests,
- 7 new pure unit tests (no trio runtime): TCP-only, mixed tpts,
  unwrapped tuples, mixed str+tuple, unsupported proto (`/udp/`),
  empty table, empty actor list
- all 22 multiaddr tests pass rn.

Prompt-IO:
ai/prompt-io/claude/20260413T205048Z_269d939c_prompt_io.md

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-14 19:54:14 -04:00
Gud Boi 7079a597c5 Add `test_tpt_bind_addrs.py` + fix type-mixing bug
Add 9 test variants (6 fns) covering all three
`tpt_bind_addrs` code paths in `open_root_actor()`:
- registrar w/ explicit bind (eq, subset, disjoint)
- non-registrar w/ explicit bind (same/diff
  bindspace) using `daemon` fixture
- non-registrar default random bind (baseline)
- maddr string input parsing
- registrar merge produces union
- `open_nursery()` forwards `tpt_bind_addrs`

Fix type-mixing bug at `_root.py:446` where the
registrar merge path did `set(Address + tuple)`,
preventing dedup and causing double-bind `OSError`.
Wrap `uw_reg_addrs` before the set union so both
sides are `Address` objs.

Also,
- add prompt-io output log for this session
- stage original prompt input for tracking

Prompt-IO: ai/prompt-io/claude/20260413T192116Z_f851f28_prompt_io.md

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-14 19:54:14 -04:00
Gud Boi bc60aa1ec5 Add `tpt_bind_addrs` param to `open_root_actor()`
Allow callers to explicitly declare transport
bind addrs instead of always auto-generating
random ones from ponged registrar addresses.

Deats,
- new `tpt_bind_addrs` kwarg wraps each input
  addr via `wrap_address()` at init time.
- non-registrar path only auto-generates random
  bind addrs when `tpt_bind_addrs` is empty.
- registrar path merges user-provided bind addrs
  with `uw_reg_addrs` via `set()` union.
- drop the deprecated `arbiter_addr` param and
  its `DeprecationWarning` shim entirely.

Also,
- expand `registry_addrs` type annotation to
  `Address|UnwrappedAddress`.
- replace bare `assert accept_addrs` in
  `async_main()` with a descriptive
  `RuntimeError` msg.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-14 19:54:14 -04:00
Gud Boi f881683c97 Tweak timeouts and rm `arbiter_addr` in tests
Use `cpu_scaling_factor()` headroom in
`test_peer_spawns_and_cancels_service_subactor`'s `fail_after` to avoid
flaky timeouts on throttled CI runners. Rename `arbiter_addr=` ->
`registry_addrs=[..]` throughout `test_spawning` and
`test_task_broadcasting` suites to match the current `open_root_actor()`
/ `open_nursery()` API.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-14 19:54:14 -04:00
Gud Boi 490fac432c Preserve absolute UDS paths in `parse_maddr()`
Drop the `.lstrip('/')` on the unix protocol value
so the lib-prepended `/` restores the absolute-path
semantics that `mk_maddr()` strips when encoding.
Pass `Path` components (not `str`) to `UDSAddress`.

Also, update all UDS test params to use absolute
paths (`/tmp/tractor_test/...`, `/tmp/tractor_rt/...`)
matching real runtime sockpath behavior; tighten
`test_parse_maddr_uds` to assert exact `filedir`.

Review: PR #429 (copilot-pull-request-reviewer[bot])
https://github.com/goodboy/tractor/pull/429#pullrequestreview-4018448152

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-14 19:54:14 -04:00
Gud Boi 5f6e45e1d4 Fix `mk_maddr()` crash on absolute UDS paths
Strip leading `/` from `filepath` before building
the `/unix/{path}` multiaddr string; OW absolute
sockpaths like `/run/user/1000/tractor/foo.sock`
produce `/unix//run/..` which `py-multiaddr`
rejects as "empty protocol path".

Woops, missed this in the initial `mk_maddr()` impl
bc the unit tests only used relative `filedir` values
(which was even noted in a comment..). The bug only
surfaces when the `.maddr` property on `UDSTransport`
is hit during logging/repr with real runtime addrs.

Found-via: cross-suite `pytest tests/ipc/ tests/msg/`
where `tpt_proto='uds'` leaks into msg tests

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-14 19:54:14 -04:00
Gud Boi 5c4438bacc Add `parse_maddr()` tests + registrar maddr integ test
Cover `parse_maddr()` with unit tests for tcp/ipv4,
tcp/ipv6, uds, and unsupported-protocol error paths,
plus full `addr -> mk_maddr -> str -> parse_maddr`
roundtrip verification.

Adds,
- a `_maddr_to_tpt_proto` inverse-mapping assertion.
- an `wrap_address()` maddr-string acceptance test.
- a `test_reg_then_unreg_maddr` end-to-end suite which audits passing
  the registry addr as multiaddr str through the entire runtime.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-14 19:54:14 -04:00
Gud Boi 90ba0e3658 Add `parse_maddr()` + `str` arm in `wrap_address()`
Inverse of `mk_maddr()`: parse a multiaddr string like
`/ip4/127.0.0.1/tcp/1234` back into a tractor `Address`.

Deats,
- add `_maddr_to_tpt_proto` reverse mapping dict
- add `parse_maddr()` fn dispatching on protocol
  combo: `[ip4|ip6, tcp]` -> `TCPAddress`,
  `[unix]` -> `UDSAddress`
- strip leading `/` the multiaddr lib prepends to
  unix protocol values for correct round-trip
- add `str` match case in `wrap_address()` for
  `/`-prefixed multiaddr strings, broaden type hint
  to `UnwrappedAddress|str`

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-14 19:54:14 -04:00
Gud Boi 1f1e09a786 Move `test_discovery` to `tests/discovery/test_registrar`
All tests are registrar-actor integration scenarios
sharing intertwined helpers + `enable_modules=[__name__]`
task fns, so keep as one mod but rename to reflect
content. Now lives alongside `test_multiaddr.py` in
the new `tests/discovery/` subpkg.

Also,
- update 5 refs in `/run-tests` SKILL.md to match
  the new path
- add `discovery/` subdir to the test directory
  layout tree

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-14 19:54:14 -04:00
Gud Boi 7cf3b5d00d Add `test_multiaddr.py` suite for `mk_maddr()`
Cover `_tpt_proto_to_maddr` mapping, TCP (ipv4/ipv6),
UDS, unsupported `proto_key` error, and round-trip
re-parse for both transport types.

Deats,
- new `tests/discovery/` subpkg w/ empty `__init__.py`
- `test_tpt_proto_to_maddr_mapping`: verify `tcp` and
  `uds` entries
- `test_mk_maddr_tcp_ipv4`: full assertion on
  `/ip4/127.0.0.1/tcp/1234` incl protocol iteration
- `test_mk_maddr_tcp_ipv6`: verify `/ip6/::1/tcp/5678`
- `test_mk_maddr_uds`: relative `filedir` bc the
  multiaddr parser rejects double-slash from abs paths
- `test_mk_maddr_unsupported_proto_key`: `ValueError`
  on `proto_key='quic'` via `SimpleNamespace` mock
- `test_mk_maddr_roundtrip`: parametrized over tcp +
  uds, re-parse `str(maddr)` back through `Multiaddr`

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-14 19:54:14 -04:00
Gud Boi c72d495d68 Use `_tpt_proto_to_maddr` lookup in `mk_maddr()`
Address Copilot review: the mapping table was
defined but never referenced. Now `mk_maddr()`
resolves `proto_key` -> maddr protocol name via
the table and rejects unknown keys upfront.

Also add missing `Path` import to the `multiaddr`
usage snippet.

Review: PR #429 (Copilot)
https://github.com/goodboy/tractor/pull/429#pullrequestreview-4010456884

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-14 19:54:14 -04:00
Gud Boi 926e861f52 Use upstream `py-multiaddr` for `._multiaddr`
Drop the NIH (notinventedhere) custom parser (`parse_maddr()`,
`iter_prot_layers()`, `prots`/`prot_params` tables) which was never
called anywhere in the codebase.

Replace with a thin `mk_maddr()` factory that wraps the upstream
`multiaddr.Multiaddr` type, dispatching on `Address.proto_key` to build
spec-compliant paths.

Deats,
- `'tcp'` addrs detect ipv4 vs ipv6 via stdlib
  `ipaddress` (resolves existing TODO)
- `'uds'` addrs map to `/unix/{path}` per the
  multiformats protocol registry (code 400)
- fix UDS `.maddr` to include full sockpath
  (previously only used `filedir`, dropped filename)
- standardize protocol names: `ipv4`->`ip4`,
  `uds`->`unix`
- `.maddr` properties now return `Multiaddr` objs
  (`__str__()` gives the canonical path form so all
  existing f-string/log consumers work unchanged)
- update `MsgTransport` protocol hint accordingly

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-14 19:54:14 -04:00
Gud Boi d9cb38372f Add `multiaddr` dep to `pyproject.toml`
Bump lock file to match obvi.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-14 19:54:14 -04:00
Gud Boi b4c975b48b Add `multiaddr` usage snippet for IP4 and UDS
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-14 19:54:14 -04:00
Gud Boi 8344537aa6 Add `uds` to `._multiaddr`, tweak typing 2026-04-14 19:54:14 -04:00
Bd 3a9f4ea383
Merge pull request #434 from mahmoudhas/add-replay-parent-main-opt-out
Add per-actor parent-main inheritance opt-out
2026-04-10 21:29:42 -04:00
mahmoud ca1b01f926 mpi integration test 2026-04-10 20:58:54 -04:00
Gud Boi 570c975f14 Fix test typo + use `wait_for_result()` API
- "boostrap" → "bootstrap" in mod docstring
- replace deprecated `portal.result()` with
  `portal.wait_for_result()` + value assertion
  inside the nursery block

Review: PR #1 (Copilot)
https://github.com/mahmoudhas/tractor/pull/1#pullrequestreview-4091096072

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-10 20:58:54 -04:00
Gud Boi a0a7668670 Fix typos + typing in `_mp_fixup_main`
- "spawing" → "spawning", close unbalanced
  backtick on `` `start_method='trio'` ``
- "uneeded" → "unneeded", "deats" → "details"
- Remove double `d` annotation; filter
  `get_preparation_data()` result into only
  `ParentMainData` keys before returning
- Use `pop('authkey', None)` for safety

Review: PR #1 (Copilot)
https://github.com/mahmoudhas/tractor/pull/1#pullrequestreview-4091096072

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-10 20:58:54 -04:00
Gud Boi 27bf566d75 Guard `_mp_fixup_main` on non-empty parent data
Use walrus `:=` to combine the assignment and
truthiness check for `_parent_main_data` into the
`if` condition, cleanly skipping the fixup block
when `inherit_parent_main=False` yields `{}`.

Review: PR #438 (Copilot)
https://github.com/goodboy/tractor/pull/438

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-10 20:58:54 -04:00
Gud Boi e8f1eca8d2 Tighten `test_spawning` types, parametrize loglevel
Parametrize `test_loglevel_propagated_to_subactor`
across `'debug'`, `'cancel'`, `'critical'` levels
(was hardcoded to just `'critical'`) and move it
above the parent-main tests for logical grouping.

Also,
- add `start_method: str` annotations throughout
- use `portal.wait_for_result()` in
  `test_most_beautiful_word` (replaces `.result()`)
- expand mod docstring to describe test coverage
- reformat `check_parent_main_inheritance` docstr

Review: PR #438 (Copilot)
https://github.com/goodboy/tractor/pull/438

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-10 20:58:54 -04:00
Gud Boi 656c6c30d1 Delegate `_mp_fixup_main` to stdlib `mp.spawn`
Drop hand-copied `_fixup_main_from_name()` and `_fixup_main_from_path()`
in favor of direct re-exports from `multiprocessing.spawn`. Simplify
`_mp_figure_out_main()` to call stdlib's `get_preparation_data()`
instead of reimplementing `__main__` module inspection inline.

Also,
- drop `ORIGINAL_DIR` global and `os`, `sys`, `platform`, `types`,
  `runpy` imports.
- pop `authkey` from prep data (unserializable and unneeded by our spawn
  path).
- update mod docstring to reflect delegation.

Review: PR #438 (Copilot)
https://github.com/goodboy/tractor/pull/438

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-10 20:58:54 -04:00
Gud Boi acf6568275 Clarify `inherit_parent_main` docstring scope
Note the opt-out only applies to the trio spawn
backend; `multiprocessing` `spawn`/`forkserver`
reconstruct `__main__` via stdlib bootstrap.

Review: PR #438 (Copilot)
https://github.com/goodboy/tractor/pull/438

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-10 20:58:54 -04:00
Gud Boi c6c591e61a Drop `spawn_test_support` pkg, inline parent-main tests
Replace the subproc-based test harness with inline
`tractor.open_nursery()` calls that directly check
`actor._parent_main_data` instead of comparing
`__main__.__name__` across a process boundary
(which is a no-op under pytest bc the parent
`__main__` is `pytest.__main__`).

Deats,
- delete `tests/spawn_test_support/` pkg (3 files)
- add `check_parent_main_inheritance()` helper fn
  that asserts on `_parent_main_data` emptiness
- rewrite both `run_in_actor` and `start_actor`
  parent-main tests as inline async fns
- drop `tmp_path` fixture and unused imports

Review: PR #434 (goodboy, Copilot)
https://github.com/goodboy/tractor/pull/434

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-10 20:58:54 -04:00
mahmoud b883b27646 Exercise parent-main inheritance through spawn test support
Move the subprocess probe into dedicated spawn test support files so the inheritance tests cover the real __main__ replay path without monkeypatching or inline script strings.
2026-04-10 20:58:54 -04:00
mahmoud 00637764d9 Address review follow-ups for parent-main inheritance opt-out
Clean up mutable defaults, give parent-main bootstrap data a named type, and add direct start_actor coverage so the opt-out change is clearer to review.
2026-04-10 20:58:54 -04:00
mahmoud ea971d25aa Rename parent-main inheritance flag.
Use `inherit_parent_main` across the actor APIs and helper to better describe the behavior, and restore the reviewer note at child bootstrap where the inherited `__main__` data is copied from `SpawnSpec`.
2026-04-10 20:58:54 -04:00
mahmoud 83b6c4270a Simplify parent-main replay opt-out.
Keep actor-owned parent-main capture and let `_mp_figure_out_main()` decide whether to return `__main__` bootstrap data, avoiding the extra SpawnSpec plumbing while preserving the per-actor flag.
2026-04-10 20:58:54 -04:00
mahmoud 6309c2e6fc Route parent-main replay through SpawnSpec
Keep trio child bootstrap data in the spawn handshake instead of stashing it on Actor state so the replay opt-out stays explicit and avoids stale-looking runtime fields.
2026-04-10 20:58:54 -04:00
mahmoud f5301d3fb0 Add per-actor parent-main replay opt-out
Let actor callers skip replaying the parent __main__ during child startup so downstream integrations can avoid inheriting incompatible bootstrap state without changing the default spawn behavior.
2026-04-10 20:58:54 -04:00
Bd 9f8e9eb739
Merge pull request #440 from goodboy/moar_skillz_squashed
Moar skillz (squashed): refinements, factorings and extensions
2026-04-10 18:31:45 -04:00
Gud Boi 6b04650187 Widen `allowed-tools` and dedup `settings.local`
Expand `run-tests` skill `allowed-tools` to cover
the documented pre-flight workflow: `git rev-parse`
for worktree detection, `python --version`, and
`UV_PROJECT_ENVIRONMENT=py* uv sync` for venv
setup. Also dedup `gh api`/`gh pr` entries in
`settings.local.json` and widen `py313` → `py*`
so non-3.13 setups aren't blocked.

Review: PR #440 (copilot-pull-request-reviewer)
https://github.com/goodboy/tractor/pull/440

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-10 18:21:45 -04:00
Gud Boi a0607f363e Allow open/close-wkt skills for model 2026-04-10 16:37:34 -04:00
Gud Boi 2ee86ddb1a Migrate shared skills to `ai.skillz` symlinks
Drop inline `commit-msg/SKILL.md` — now deployed
as a symlink from the central `ai.skillz` repo via
`deploy-skill.sh`.

Gitignore all symlinked skill dirs so they stay
machine-local:
- fully-symlinked: `py-codestyle`, `close-wkt`,
  `open-wkt`, `plan-io`, `prompt-io`,
  `code-review-changes`, `resolve-conflicts`,
  `inter-skill-review`, `yt-url-lookup`
- hybrid (symlinked SKILL.md + references):
  `commit-msg/SKILL.md`, `pr-msg/SKILL.md`,
  `pr-msg/references`

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-10 16:37:34 -04:00
Gud Boi 0286d36ed7 Add repo-local `claude` skills + settings + gitignore
Add `/run-tests`, `/conc-anal` skill definitions and
`/pr-msg` `format-reference.md` that live in-repo
(not symlinked from `ai.skillz`).

- `/run-tests`: `pytest` suite runner with
  dev-workflow helpers, never-auto-commit rule.
- `/conc-anal`: concurrency analysis skill.
- `/pr-msg` `format-reference.md`: canonical PR
  description structure + cross-service ref-links.
- `ai_notes/docs_todos.md`: `literalinclude` idea.
- `settings.local.json`: permission rules for `gh`,
  `git`, `python3`, `cat`, skill invocations.
- `.gitignore`: ignore commit-msg/pr-msg `msgs/`,
  `LATEST` files, review ctx, session conf, claude
  worktrees.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-10 16:37:34 -04:00
Bd 2db6f97130
Merge pull request #439 from goodboy/revert_wrapt_tractor_test_deco
Drop `wrapt` for `tractor_test`, revert to `functools`
2026-04-10 16:36:03 -04:00
Gud Boi 9af6adc181 Fix runtime kwarg leaking in `tractor_test`
The `functools` rewrite forwarded all `kwargs`
through `_main(**kwargs)` to `wrapped(**kwargs)`
unchanged — the Windows `start_method` default
could leak to test fns that don't declare it.
The pre-wrapt code guarded against this with
named wrapper params.

Extract runtime settings (`reg_addr`, `loglevel`,
`debug_mode`, `start_method`) as closure locals
in `wrapper`; `_main` uses them directly for
`open_root_actor()` while `kwargs` passes to
`wrapped()` unmodified.

Review: PR #439 (Copilot)
https://github.com/goodboy/tractor/pull/439#pullrequestreview-4091005202

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-10 16:20:01 -04:00
Gud Boi 452a32fb23 Drop `wrapt` for `tractor_test`, revert to `functools`
Realized a bit late that (pretty sure) i already tried this using
`wrapt` idea and waay back and found the same "issue" XD

The `wrapt.decorator` transparently proxies `__code__` from the async
test fn, fooling `pytest`'s coroutine detection into skipping wrapped
tests as "unhandled coroutines". `functools.wraps` preserves the sig for
fixture injection via `__wrapped__` without leaking the async nature.

So i let `claude` rework the latest code to go back to using the old
stdlib wrapping again..

Deats,
- `functools.partial` replaces `wrapt.PartialCallableObjectProxy`.
- wrapper takes plain `**kwargs`; runtime settings extracted via
  `kwargs.get()` in `_main()`.
- `iscoroutinefunction()` guard moved before wrapper definition.
- drop all `*args` passing (fixture kwargs only).

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-10 12:08:46 -04:00
Bd f47010d7e9
Merge pull request #436 from goodboy/moc_revert_to_fix
Moc revert to fix: reversion fix for bug in #435
2026-04-09 17:54:41 -04:00
Gud Boi 3f198bc86c Drop commented-out `tractor.pause()` debug hooks
Remove 3 leftover `# await tractor.pause(shield=True)`
/ `# await tractor.pause()` calls in
`maybe_open_context()` that were used during the
`_Cache.run_ctx` teardown race diagnostic session
(PR #436). These are dead commented-out code with no
runtime effect — just noise.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-09 17:41:28 -04:00
Gud Boi 391c8d3566 Address Copilot review fixes on `maybe_open_context()`
Deats,
- drop unused `import tractor` (F401)
- fix `_Cache.locks` annotation to `trio.StrictFIFOLock`
- fix typos: "mabye-value", "Acquir lock"
- add `resources.pop()` cleanup in the caller if
  `service_tn.start()` fails — prevents a
  permanent `_Cache.resources` leak on
  `__aenter__` failure (note: Copilot's suggested
  outer `try/finally` in `run_ctx` would
  re-introduce the atomicity gap)
- add `user_registered` flag so `users -= 1` only
  runs when the task actually incremented
- move lock pop into the `users <= 0` teardown
  block so the last exiting user always cleans up,
  regardless of who created the lock; drop
  now-dead `lock_registered` var

Also,
- swap `fid` for `ctx_key` in debug log msgs
- remove stale commented-out `# fid` refs

Review: PR #436 (copilot-pull-request-reviewer)
https://github.com/goodboy/tractor/pull/436

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-09 15:10:07 -04:00
Gud Boi 4fc477cfd6 Revert `resources.pop()` back inside `run_ctx` inner finally
Reverts the `_Cache.run_ctx` change from 93aa39db which
moved `resources.pop(ctx_key)` to an outer `finally`
*after* the acm's `__aexit__()`. That introduced an
atomicity gap: `values` was already popped in the inner
finally but `resources` survived through the acm teardown
checkpoints. A re-entering task that creates a fresh lock
(the old one having been popped by the exiting caller)
could then acquire immediately and find stale `resources`
(for which now we raise a `RuntimeError('Caching resources ALREADY
exist?!')`).

Deats,
- the orig 93aa39db rationale was a preemptive guard
  against acm `__aexit__()` code accessing `_Cache`
  mid-teardown, but no `@acm` in `tractor` (or `piker`) ever
  does that; the scenario never materialized.
- by popping both `values` AND `resources` atomically
  (no checkpoint between them) in the inner finally,
  the re-entry race window is closed: either the new
  task sees both entries (cache hit) or neither
  (clean cache miss).
- `test_moc_reentry_during_teardown` now passes
  without `xfail`! (:party:)

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-09 14:42:42 -04:00
Gud Boi cd1cd03725 Add prompt-io log for `run_ctx` teardown analysis
Documents the diagnostic session tracing why
per-`ctx_key` locking alone doesn't close the
`_Cache.run_ctx` teardown race — the lock pops
in the exiting caller's task but resource cleanup
runs in the `run_ctx` task inside `service_tn`.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-09 14:42:42 -04:00
Gud Boi febe587c6c Drop `xfail` from `test_moc_reentry_during_teardown`
The per-`ctx_key` locking fix in f086222d intended to resolve the
teardown race reproduced by the new test suite, so the test SHOULD now
pass. TLDR, it doesn't Bp

Also add `collapse_eg()` to the test's ctx-manager stack so that when
run with `pytest <...> --tpdb` we'll actually `pdb`-REPL the RTE when it
hits (previously an assert-error).

(this commit-msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-09 14:42:42 -04:00
Gud Boi 4d3c5b9163 Use per-key locking+user tracking in `maybe_open_context()`
(Hopefully!) solving a long-run bug with the `brokerd.kraken` backend in
`piker`..

- Track `_Cache.users` per `ctx_key` via a `defaultdict[..., int]`
  instead of a single global counter; fix premature teardown when
  multiple ctx keys are active simultaneously.
- Key `_Cache.locks` on `ctx_key` (not bare `fid`) so different kwarg
  sets for the same `acm_func` get independent `StrictFIFOLock`s.
- Add `_UnresolvedCtx` sentinel class to replace bare `None` check;
  avoid false-positive teardown when a wrapped acm legitimately yields
  `None`.
- Swap resource-exists `assert` for detailed `RuntimeError`.

Also,
- fix "whih" typo.
- add debug logging for lock acquire/release lifecycle.

(this commit-msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-09 14:42:42 -04:00
Bd 8b106b9144
Merge pull request #435 from goodboy/moc_coverage_test_by_claude
`.trionics.maybe_open_context()` race-edge-case coverage
2026-04-09 14:41:59 -04:00
Gud Boi d6ece8eab3 Only run CI on pushes to `main`, ow just on PR/dev branches 2026-04-07 14:17:08 -04:00
Gud Boi 8494eb9b8a Run CI workflow on PRs even from forks 2026-04-07 14:02:43 -04:00
Gud Boi cab366cd65 Add xfail test for `_Cache.run_ctx` teardown race
Reproduce the piker `open_cached_client('kraken')` scenario: identical
`ctx_key` callers share one cached resource, and a new task re-enters
during `__aexit__` — hitting `assert not resources.get()` bc `values`
was popped but `resources` wasn't yet.

Deats,
- `test_moc_reentry_during_teardown` uses an `in_aexit` event to
  deterministically land in the teardown window.
- marked `xfail(raises=AssertionError)` against unpatched code (fix in
  `9e49eddd` or wtv lands on the `maybe_open_ctx_locking` or thereafter
  patch branch).

Also, add prompt-io log for the session.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

Prompt-IO: ai/prompt-io/claude/20260406T193125Z_85f9c5d_prompt_io.md
2026-04-06 18:17:04 -04:00
Gud Boi 85f9c5df6f Add per-`ctx_key` isolation tests for `maybe_open_context()`
Add `test_per_ctx_key_resource_lifecycle` to verify that per-key user
tracking correctly tears down resources independently - exercises the
fix from 02b2ef18 where a global `_Cache.users` counter caused stale
cache hits when the same `acm_func` was called with different kwargs.

Also, add a paired `acm_with_resource()` helper `@acm` that yields its
`resource_id` for per-key testing in the above suite.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

Prompt-IO: ai/prompt-io/claude/20260406T172848Z_02b2ef1_prompt_io.md
2026-04-06 14:37:47 -04:00
Gud Boi ebe9d5e4b5 Parametrize `test_resource_cache.test_open_local_sub_to_stream`
Namely with multiple pre-sleep `delay`-parametrizations before either,

- parent-scope cancel-calling (as originally) or,
- depending on the  new `cancel_by_cs: bool` suite parameter, optionally
  just immediately exiting from (the newly named)
  `maybe_cancel_outer_cs()` a checkpoint.

In the latter case we ensure we **don't** inf sleep to avoid leaking
those tasks into the `Actor._service_tn` (though we should really have
a better soln for this)..

Deats,
- make `cs` args optional and adjust internal logic to match.
- add some notes around various edge cases and issues with using the
  actor-service-tn as the scope by default.
2026-04-06 14:37:47 -04:00
Bd bbf01d5161
Merge pull request #430 from goodboy/dependabot/uv/pygments-2.20.0
Bump pygments from 2.19.2 to 2.20.0
2026-04-05 13:33:33 -04:00
Bd ec8e8a2786
Merge pull request #426 from goodboy/remote_exc_type_registry
Fix remote exc relay + add `reg_err_types()` tests
2026-04-02 22:44:36 -04:00
Gud Boi c3d1ec22eb Fix `Type[BaseException]` annots, guard `.src_type` resolve
- Use `Type[BaseException]` (not bare `BaseException`)
  for all err-type references: `get_err_type()` return,
  `._src_type`, `boxed_type` in `unpack_error()`.
- Add `|None` where types can be unresolvable
  (`get_err_type()`, `.boxed_type` property).
- Add `._src_type_resolved` flag to prevent repeated
  lookups and guard against `._ipc_msg is None`.
- Fix `recevier` and `exeptions` typos.

Review: PR #426 (Copilot)
https://github.com/goodboy/tractor/pull/426

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-02 18:21:19 -04:00
Gud Boi 8f44efa327 Drop stale `.cancel()`, fix docstring typo in tests
- Remove leftover `await an.cancel()` in
  `test_registered_custom_err_relayed`; the
  nursery already cancels on scope exit.
- Fix `This document` -> `This documents` typo in
  `test_unregistered_err_still_relayed` docstring.

Review: PR #426 (Copilot)
https://github.com/goodboy/tractor/pull/426

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-02 18:21:19 -04:00
Gud Boi 5968a3c773 Use `'<unknown>'` for unresolvable `.boxed_type_str`
Add a teensie unit test to match.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-02 18:21:19 -04:00
Gud Boi 80597b80bf Add passing test for unregistered err relay
Drop the `xfail` test and instead add a new one that ensures the
`tractor._exceptions` fixes enable graceful relay of
remote-but-unregistered error types via the unboxing of just the
`rae.src_type_str/boxed_type_str` content. The test also ensures
a warning is included with remote error content indicating the user
should register their error type for effective cross-actor re-raising.

Deats,
- add `test_unregistered_err_still_relayed`: verify the
  `RemoteActorError` IS raised with `.boxed_type`
  as `None` but `.src_type_str`, `.boxed_type_str`,
  and `.tb_str` all preserved from the IPC msg.
- drop `test_unregistered_boxed_type_resolution_xfail`
  since the new above case covers it and we don't need to have
  an effectively entirely repeated test just with an inverse assert
  as it's last line..

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-02 18:21:19 -04:00
Gud Boi a41c6d5c70 Fix unregistered-remote-error-type relay crash
Make `RemoteActorError` resilient to unresolved
custom error types so that errors from remote actors
always relay back to the caller - even when the user
hasn't called `reg_err_types()` to register the exc type.

Deats,
- `.src_type`: log warning + return `None` instead
  of raising `TypeError` which was crashing the
  entire `_deliver_msg()` -> `pformat()` chain
  before the error could be relayed.
- `.boxed_type_str`: fallback to `_ipc_msg.boxed_type_str`
  when the type obj can't be resolved so the type *name* is always
  available.
- `unwrap_src_err()`: fallback to `RuntimeError` preserving
  original type name + traceback.
- `unpack_error()`: log warning when `get_err_type()` returns
  `None` telling the user to call `reg_err_types()`.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-02 18:21:19 -04:00
Gud Boi 9c37b3f956 Add `reg_err_types()` test suite for remote exc relay
Verify registered custom error types round-trip correctly over IPC via
`reg_err_types()` + `get_err_type()`.

Deats,
- `TestRegErrTypesPlumbing`: 5 unit tests for the type-registry plumbing
  (register, lookup, builtins, tractor-native types, unregistered
  returns `None`)
- `test_registered_custom_err_relayed`: IPC end-to-end for a registered
  `CustomAppError` checking `.boxed_type`, `.src_type`, and `.tb_str`
- `test_registered_another_err_relayed`: same for `AnotherAppError`
  (multi-type coverage)
- `test_unregistered_custom_err_fails_lookup`: `xfail` documenting that
  `.boxed_type` can't resolve without `reg_err_types()` registration

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-02 18:21:19 -04:00
Bd 8f6bc56174
Merge pull request #427 from goodboy/subsys_reorg
Mv core mods to `runtime/`, `spawn/`, `discovery/` subpkgs
2026-04-02 18:21:00 -04:00
Gud Boi b14dbde77b Skip `test_empty_mngrs_input_raises` on UDS tpt
The `open_actor_cluster()` teardown hangs
intermittently on UDS when `gather_contexts(mngrs=())`
raises `ValueError` mid-setup; likely a race in the
actor-nursery cleanup vs UDS socket shutdown. TCP
passes reliably (5/5 runs).

- Add `tpt_proto` fixture param to the test
- `pytest.skip()` on UDS with a TODO for deeper
  investigation of `._clustering`/`._supervise`
  teardown paths

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-02 17:59:13 -04:00
Gud Boi cd6509b724 Fix `tractor_test` kwarg check and Windows `start_method` default
- Use `kw in kwargs` membership test instead of
  `kwargs[kw]` to avoid `KeyError` on missing params.
- Restructure Windows `start_method` logic to properly
  default to `'trio'` when unset; only raise on an
  explicit non-trio value.

Review: PR #427 (Copilot)
https://github.com/goodboy/tractor/pull/427#pullrequestreview-4009934142

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-02 17:59:13 -04:00
Gud Boi 93d99ed2eb Move `get_cpu_state()` to `conftest` as shared latency headroom
Factor the CPU-freq-scaling helper out of
`test_legacy_one_way_streaming` into `conftest.py`
alongside a new `cpu_scaling_factor()` convenience fn
that returns a latency-headroom multiplier (>= 1.0).

Apply it to the two other flaky-timeout tests,
- `test_cancel_via_SIGINT_other_task`: 2s -> scaled
- `test_example[we_are_processes.py]`: 16s -> scaled

Deats,
- add `get_cpu_state()` + `cpu_scaling_factor()` to
  `conftest.py` so all test mods can share the logic.
- catch `IndexError` (empty glob) in addition to
  `FileNotFoundError`.
- rename `factor` var -> `headroom` at call sites for
  clarity on intent.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-02 17:59:13 -04:00
Gud Boi 6215e3b2dd Adjust `test_a_quadruple_example` time-limit for CPU scaling
Add `get_cpu_state()` helper to read CPU freq settings
from `/sys/devices/system/cpu/` and use it to compensate
the perf time-limit when `auto-cpufreq` (or similar)
scales down the max frequency.

Deats,
- read `*_pstate_max_freq` and `scaling_max_freq`
  to compute a `cpu_scaled` ratio.
- when `cpu_scaled != 1.`, increase `this_fast` limit
  proportionally (factoring dual-threaded cores).
- log a warning via `test_log` when compensating.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-02 17:59:13 -04:00
Gud Boi be5d8da8c0 Just alias `Arbiter` via assignment 2026-04-02 17:59:13 -04:00
Gud Boi 21ed181835 Filter `__pycache__` from example discovery in tests
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-02 17:59:13 -04:00
Gud Boi 9ec2749ab7 Rename `Arbiter` -> `Registrar`, mv to `discovery._registry`
Move the `Arbiter` class out of `runtime._runtime` into its
logical home at `discovery._registry` as `Registrar(Actor)`.
This completes the long-standing terminology migration from
"arbiter" to "registrar/registry" throughout the codebase.

Deats,
- add new `discovery/_registry.py` mod with `Registrar`
  class + backward-compat `Arbiter = Registrar` alias.
- rename `Actor.is_arbiter` attr -> `.is_registrar`;
  old attr now a `@property` with `DeprecationWarning`.
- `_root.py` imports `Registrar` directly for
  root-actor instantiation.
- export `Registrar` + `Arbiter` from `tractor.__init__`.
- `_runtime.py` re-imports from `discovery._registry`
  for backward compat.

Also,
- update all test files to use `.is_registrar`
  (`test_local`, `test_rpc`, `test_spawning`,
  `test_discovery`, `test_multi_program`).
- update "arbiter" -> "registrar" in comments/docstrings
  across `_discovery.py`, `_server.py`, `_transport.py`,
  `_testing/pytest.py`, and examples.
- drop resolved TODOs from `_runtime.py` and `_root.py`.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-02 17:59:13 -04:00
Gud Boi f3441a6790 Update tests+examples imports for new subpkgs
Adjust all `tractor._state`, `tractor._addr`,
`tractor._supervise`, etc. refs in tests and examples
to use the new `runtime/`, `discovery/`, `spawn/` paths.

Also,
- use `tractor.debug_mode()` pub API instead of
  `tractor._state.debug_mode()` in a few test mods
- add explicit `timeout=20` to `test_respawn_consumer_task`
  `@tractor_test` deco call

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-02 17:59:13 -04:00
Gud Boi cc42d38284 Mv core mods to `runtime/`, `spawn/`, `discovery/` subpkgs
Restructure the flat `tractor/` top-level private mods
into (more nested) subpackages:

- `runtime/`: `_runtime`, `_portal`, `_rpc`, `_state`,
  `_supervise`
- `spawn/`: `_spawn`, `_entry`, `_forkserver_override`,
  `_mp_fixup_main`
- `discovery/`: `_addr`, `_discovery`, `_multiaddr`

Each subpkg `__init__.py` is kept lazy (no eager
imports) to avoid circular import issues.

Also,
- update all intra-pkg imports across ~35 mods to use
  the new subpkg paths (e.g. `from .runtime._state`
  instead of `from ._state`)

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-02 17:59:13 -04:00
Gud Boi 6827ceba12 Use `wrapt` for `tractor_test()` decorator
Refactor the test-fn deco to use `wrapt.decorator`
instead of `functools.wraps` for better fn-sig
preservation and optional-args support via
`PartialCallableObjectProxy`.

Deats,
- add `timeout` and `hide_tb` deco params
- wrap test-fn body with `trio.fail_after(timeout)`
- consolidate per-fixture `if` checks into a loop
- add `iscoroutinefunction()` type-check on wrapped fn
- set `__tracebackhide__` at each wrapper level

Also,
- update imports for new subpkg paths:
  `tractor.spawn._spawn`, `tractor.discovery._addr`,
  `tractor.runtime._state`
  (see upcoming, likely large patch commit ;)

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-02 17:59:13 -04:00
Gud Boi 94458807ce Expose `RuntimeVars` + `get_runtime_vars()` from pkg
Export the new `RuntimeVars` struct and `get_runtime_vars()`
from `tractor.__init__` and improve the accessor to
optionally return the struct form.

Deats,
- add `RuntimeVars` and `get_runtime_vars` to
  `__init__.py` exports; alphabetize `_state` imports.
- move `get_runtime_vars()` up in `_state.py` to sit
  right below `_runtime_vars` dict definition.
- add `as_dict: bool = True` param so callers can get
  either the legacy `dict` or the new `RuntimeVars`
  struct.
- drop the old stub fn at bottom of `_state.py`.
- rm stale `from .msg.pretty_struct import Struct` comment.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-02 17:59:13 -04:00
Gud Boi be5e7e446b Proto a `._state.RuntimeVars` struct
So we can start transition from runtime-vars `dict` to a typed struct
for better clarity and wire-ready monitoring potential, as well as
better traceability when .

Deats,
- add a new `RuntimeVars(Struct)` with all fields from `_runtime_vars`
  dict typed out
- include `__setattr__()` with `breakpoint()` for debugging
  any unexpected mutations.
- add `.update()` method for batch-updating compat with `dict`.
- keep old `_runtime_vars: dict` in place (we need to port a ton of
  stuff to adjust..).

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-02 17:59:13 -04:00
Gud Boi 571b2b320e Add `reg_err_types()` for custom remote exc lookup
Allow external app code to register custom exception types
on `._exceptions` so they can be re-raised on the receiver
side of an IPC dialog via `get_err_type()`.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-02 17:59:13 -04:00
Gud Boi c7b5d00f19 Add `get_runtime_vars()` accessor to `._state`
Expose a copy of the current actor's `_runtime_vars` dict
via a public fn; TODO to convert to `RuntimeVars` struct.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-04-02 17:59:13 -04:00
dependabot[bot] 1049f7bf38
Bump pygments from 2.19.2 to 2.20.0
Bumps [pygments](https://github.com/pygments/pygments) from 2.19.2 to 2.20.0.
- [Release notes](https://github.com/pygments/pygments/releases)
- [Changelog](https://github.com/pygments/pygments/blob/master/CHANGES)
- [Commits](https://github.com/pygments/pygments/compare/2.19.2...2.20.0)

---
updated-dependencies:
- dependency-name: pygments
  dependency-version: 2.20.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-03-30 20:28:44 +00:00
Bd cc3bfac741
Merge pull request #366 from goodboy/dereg_on_oserror
Make `find_actor()` delete stale sockaddr entries from registrar on `OSError`
2026-03-25 03:27:27 -04:00
Gud Boi e71eec07de Refine type annots in `_discovery` and `_runtime`
- Add `LocalPortal` union to `query_actor()` return
  type and `reg_portal` var annotation since the
  registrar yields a `LocalPortal` instance.
- Update docstring to note the `LocalPortal` case.
- Widen `.delete_addr()` `addr` param to accept
  `list[str|int]` bc msgpack deserializes tuples as
  lists over IPC.
- Tighten `uid` annotation to `tuple[str, str]|None`.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-25 02:16:48 -04:00
Gud Boi b557ec20a7 Coerce IPC `addr` to `tuple` in `.delete_addr()`
`msgpack` deserializes tuples as lists over IPC so
the `bidict.inverse.pop()` needs a `tuple`-cast to
match registry keys.

Regressed-by: 85457cb (`registry_addrs` change)
Found-via: `/run-tests` test_stale_entry_is_deleted

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-25 01:36:58 -04:00
Gud Boi 85457cb839 Address Copilot review suggestions on PR #366
- Use `bidict.forceput()` in `register_actor()` to handle
  duplicate addr values from stale entries or actor restarts.
- Fix `uid` annotation to `tuple[str, str]|None` in
  `maybe_open_portal()` and handle the `None` return from
  `delete_addr()` in log output.
- Pass explicit `registry_addrs=[reg_addr]` to `open_nursery()`
  and `find_actor()` in `test_stale_entry_is_deleted` to ensure
  the test uses the remote registrar.
- Update `query_actor()` docstring to document the new
  `(addr, reg_portal)` yield shape.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-25 00:21:09 -04:00
Gud Boi 850219f60c Guard `reg_portal` for `None` in `maybe_open_portal()`
Fix potential `AttributeError` when `query_actor()` yields
a `None` portal (peer-found-locally path) and an `OSError`
is raised during transport connect.

Also,
- fix `Arbiter.delete_addr()` return type to
  `tuple[str, str]|None` bc it can return `None`.
- fix "registar" typo -> "registrar" in comment.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-17 17:26:57 -04:00
Tyler Goodlet d929fb75b5 Rename `.delete_sockaddr()` -> `.delete_addr()` 2026-03-13 21:51:15 -04:00
Tyler Goodlet 403c2174a1 Always no-raise try-to-pop registry addrs 2026-03-13 21:51:15 -04:00
Tyler Goodlet 528012f35f Add stale entry deleted from registrar test
By spawning an actor task that immediately shuts down the transport
server and then sleeps, verify that attempting to connect via the
`._discovery.find_actor()` helper delivers `None` for the `Portal`
value.

Relates to #184 and #216
2026-03-13 21:51:15 -04:00
Tyler Goodlet 0dfa6f4a8a Don't unwrap and unwrapped addr, just warn on delete XD 2026-03-13 21:51:15 -04:00
Tyler Goodlet a0d3741fac Ensure `._registry` values are hashable, since `bidict`! 2026-03-13 21:51:15 -04:00
Tyler Goodlet 149b800c9f Handle stale registrar entries; detect and delete
In cases where an actor's transport server task (by default handling new
TCP connections) terminates early but does not de-register from the
pertaining registry (aka the registrar) actor's address table, the
trying-to-connect client actor will get a connection error on that
address. In the case where client handles a (local) `OSError` (meaning
the target actor address is likely being contacted over `localhost`)
exception, make a further call to the registrar to delete the stale
entry and `yield None` gracefully indicating to calling code that no
`Portal` can be delivered to the target address.

This issue was originally discovered in `piker` where the `emsd`
(clearing engine) actor would sometimes crash on rapid client
re-connects and then leave a `pikerd` stale entry. With this fix new
clients will attempt connect via an endpoint which will re-spawn the
`emsd` when a `None` portal is delivered (via `maybe_spawn_em()`).
2026-03-13 21:51:15 -04:00
Tyler Goodlet 03f458a45c Add `Arbiter.delete_sockaddr()` to remove addrs
Since stale addrs can be leaked where the actor transport server task
crashes but doesn't (successfully) unregister from the registrar, we
need a remote way to remove such entries; hence this new (registrar)
method.

To implement this make use of the `bidict` lib for the `._registry`
table thus making it super simple to do reverse uuid lookups from an
input socket-address.
2026-03-13 21:51:15 -04:00
Bd e77198bb64
Merge pull request #422 from goodboy/global_uds_in_test_harness
Run (some) test suites in CI with `--tpt-proto uds`
2026-03-13 21:50:45 -04:00
Gud Boi 5b8f6cf4c7 Use `.aid.uid` to avoid deprecation warns in tests
- `test_inter_peer_cancellation`: swap all `.uid` refs
  on `Actor`, `Channel`, and `Portal` to `.aid.uid`
- `test_legacy_one_way_streaming`: same + fix `print()`
  to multiline style

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-13 21:10:52 -04:00
Gud Boi 8868ff19f3 Flip to `ActorNursery.cancel_called` API
Avoid deprecation warnings, prepping for property removal.
2026-03-13 21:10:52 -04:00
Gud Boi 066011b83d Bump `fail_after` delay on non-linux for sync-sleep test
Use 6s timeout on non-linux (vs 4s) in
`test_cancel_while_childs_child_in_sync_sleep()` to avoid
flaky `TooSlowError` on slower CI runners.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-13 21:10:52 -04:00
Gud Boi b1d003d850 Add `--tpt-proto` CI matrix and wire to `pytest`
- add `tpt_proto: ['tcp', 'uds']` matrix dimension
  to the `testing` job.
- exclude `uds` on `macos-latest` for now.
- pass `--tpt-proto=${{ matrix.tpt_proto }}` to the
  `pytest` invocation.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-13 21:10:52 -04:00
Gud Boi 8991ec2bf5 Fix warns and de-reg race in `test_discovery`
Removes the `pytest` deprecation warns and attempts to avoid
some de-registration raciness, though i'm starting to think the
real issue is due to not having the fixes from #366 (which handle
the new dereg on `OSError` case from UDS)?

- use `.channel.aid.uid` over deprecated `.channel.uid`
  throughout `test_discovery.py`.
- add polling loop (up to 5s) for subactor de-reg check
  in `spawn_and_check_registry()` to handle slower
  transports like UDS where teardown takes longer.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-13 21:10:52 -04:00
Gud Boi dfc153c228 'Bump `daemon` pre-wait for 'uds' parametrization' 2026-03-13 21:10:52 -04:00
Gud Boi 52e8fb43ee Tighten UDS addr validation and sockname prefixes
- add `is_valid` and `sockpath.resolve()` asserts in
  `get_rando_addr()` for the `'uds'` case plus an
  explicit `UDSAddress` type annotation.
- rename no-runtime sockname prefixes from
  `'<unknown-actor>'`/`'root'` to
  `'no_runtime_root'`/`'no_runtime_actor'` with a proper
  if/else branch in `UDSAddress.get_random()`.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-13 21:10:52 -04:00
Gud Boi 99577b719a Skip cluster test on UDS, wire `tpt_proto` fixture
Add UDS skip-guard to `test_streaming_to_actor_cluster()`
and plumb `tpt_proto` through the `@tractor_test` wrapper
so transport-parametrized tests can receive it.

Deats,
- skip cluster test when `tpt_proto == 'uds'` with
  descriptive msg, add TODO about `@pytest.mark.no_tpt`.
- add `tpt_proto: str|None` param to inner wrapper in
  `tractor_test()`, forward to decorated fn when its sig
  accepts it.
- register custom `no_tpt` marker via `pytest_configure()`
  to avoid unknown-marker warnings.
- add masked todo for `no_tpt` marker-check code in `tpt_proto` fixture
  (needs fn-scope to work, left as TODO).
- add `request` param to `tpt_proto` fixture for future
  marker inspection.

Also,
- add doc-string to `test_streaming_to_actor_cluster()`.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-13 21:10:52 -04:00
Gud Boi 4092db60b2 Revert advanced-fault UDS edge case handling
Namely the workaround expected exc branches added in ef7ed7a for the UDS
parametrization. With the new boxing of the underlying CREs as
tpt-closed, we can expect the same exc outcomes as in the TCP cases.

Also this tweaks some error report logging content used while debugging
this,
- properly `repr()` the `TransportClosed.src_exc`-type from
  the maybe emit in `.report_n_maybe_raise()`.
- remove the redudant `chan.raddr` from the "closed abruptly"
  header in the tpt-closed handler of `._rpc.process_messages()`,
  the `Channel.__repr__()` now contains it by default.
2026-03-13 21:10:52 -04:00
Gud Boi 4f333dee05 Pass `enable_transports` in `daemon` fixture
Forward the `tpt_proto` fixture val into spawned daemon
subprocesses via `run_daemon(enable_transports=..)` and
sync `_runtime_vars['_enable_tpts']` in the `tpt_proto`
fixture so sub-actors inherit the transport setting.

Deats,
- add `enable_transports={enable_tpts}` to the daemon
  spawn-cmd template in `tests/conftest.py`.
- set `_state._runtime_vars['_enable_tpts']` in the
  `tpt_proto` fixture in `_testing/pytest.py`.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-13 21:10:52 -04:00
Gud Boi 8a2f74da2c Bump `_PROC_SPAWN_WAIT` and use `test_log` in `daemon`
For more reliability with the oob registrar using tests
via the `daemon` fixture,
- increase spawn-wait to `2` in CI, `1` OW; drop
  the old py<3.7 branch.
- move `_ci_env` to module-level (above `_non_linux`)
  so `_PROC_SPAWN_WAIT` can reference it at parse time.
- add `test_log` fixture param to `daemon()`, use it
  for the error-on-exit log line instead of bare `log`.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-13 21:10:52 -04:00
Gud Boi 2bf155131d Make `spawn()` `expect_timeout` configurable
Add `expect_timeout: float` param to `_spawn()`
so individual tests can tune `pexpect` timeouts
instead of relying on the hard-coded 3/10 split.

Deats,
- default to 4s, bump by +6 on non-linux CI.
- use walrus `:=` to capture resolved timeout and  assert
  `spawned.timeout == timeout` for sanity.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-13 21:10:52 -04:00
Gud Boi 0f29f5717a `ci.yml` add todos for mp-backend runs and for supporting subints soon! 2026-03-13 21:10:52 -04:00
Gud Boi 5ea721683b Use `.aid.uid` to avoid deprecation warns
I started getting annoyed by all the warnings from `pytest` during work
on macos suport in CI, so this replaces all `Actor.uid`/`Channel.uid`
accesses with `.aid.uid` (or `.aid.reprol()` for log msgs) across the
core runtime and IPC subsystems to avoid the noise.

This also provides incentive to start the adjustment to all
`.uid`-holding/tracking internal `dict`-tables/data-structures to
instead use `.msg.types.Aid`. Hopefully that will come a (vibed?) follow
up shortly B)

Deats,
- `._context`: swap all `self._actor.uid`, `self.chan.uid`,
  and `portal.actor.uid` refs to `.aid.uid`; use
  `.aid.reprol()` for log/error formatting.
- `._rpc`: same treatment for `actor.uid`, `chan.uid` in
  log msgs and cancel-scope handling; fix `str(err)` typo
  in `ContextCancelled` log.
- `._runtime`: update `chan.uid` -> `chan.aid.uid` in ctx
  cache lookups, RPC `Start` msg, registration and
  cancel-request handling; improve ctxc log formatting.
- `._spawn`: replace all `subactor.uid` with
  `.aid.uid` for child-proc tracking, IPC peer waiting,
  debug-lock acquisition, and nursery child dict ops.
- `._supervise`: same for `subactor.uid` in cancel and
  portal-wait paths; use `actor.aid.uid` for error dict.
- `._state`: fix `last.uid` -> `last.aid.uid` in
  `current_actor()` error msg.

Also,
- `._chan`: make `Channel.aid` a proper `@property` backed
  by `._aid` so we can add validation/typing later.
- `.log`: use `current_actor().aid.uuid` instead of
  `.uid[1]` for actor-uid log field.
- `.msg.types`: add TODO comment for `Start.aid` field
  conversion.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-13 21:10:52 -04:00
Gud Boi f84ef44992 Repair lifetime-stack suite's flakiness
Event on linux i was noticing lotsa false negatives based on sub
teardown race conditions, so this tries to both make way for
(eventually?) expanding the set of suite cases and ensure the current
ones are more reliable on every run.

The main change is to hange the `error_in_child=False` case to use
parent-side-cancellation via a new `trio.move_on_after(timeout)` instead
of `actor.cancel_soon()` (which is now toggled by a new `self_cancel:
bool` but unused rn), and add better teardown assertions.

Low level deats,
- add `rent_cancel`/`self_cancel` params to
  `crash_and_clean_tmpdir()` for different cancel paths;
  default to `rent_cancel=True` which just sleeps forever
  letting the parent's timeout do the work.
- use `trio.move_on_after()` with longer timeouts per
  case: 1.6s for error, 1s for cancel.
- use the `.move_on_after()` cancel-scope to assert `.cancel_called`
  pnly when `error_in_child=False`, indicating we
  parent-graceful-cancelled the sub.
- add `loglevel` fixture, pass to `open_nursery()`.
- log caught `RemoteActorError` via console logger.
- add `ids=` to parametrize for readable test names.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-13 21:10:52 -04:00
Gud Boi 1e0c57c6c5 Wrap cluster test in `trio.fail_after()`
Add a 6s timeout guard around `test_streaming_to_actor_cluster()`
to catch hangs, and nest the `async with` block inside it.
Found this when running `pytest tests/ --tpt-proto uds`.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-13 21:10:52 -04:00
Gud Boi 65660c77c7 Add note about `--tpt-proto` controlling `reg_addr`-type 2026-03-13 21:10:52 -04:00
Bd c9b415475f
Merge pull request #413 from goodboy/to_asyncio_channel_iface
Extend the `to_asyncio` inter-loop-task channel iface
2026-03-13 21:09:13 -04:00
Gud Boi 359bcf691f Update `docs/README.rst` to use `chan` API style
Sync the inline "infected asyncio" echo-server example
with the new `LinkedTaskChannel` iface from prior commits.

- `to_trio`/`from_trio` params -> `chan: LinkedTaskChannel`
- use `chan.started_nowait()`, `.send_nowait()`, `.get()`
- swap yield order to `(chan, first)`
- update blurb to describe the new unified channel API

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-13 20:54:49 -04:00
Gud Boi b3ce5ab4f6 Swap `open_channel_from()` to yield `(chan, first)`
Deliver `(LinkedTaskChannel, Any)` instead of the prior `(first, chan)`
order from `open_channel_from()` to match the type annotation and be
consistent with `trio.open_*_channel()` style where the channel obj
comes first.

- flip `yield first, chan` -> `yield chan, first`
- update type annotation + docstring to match
- swap all unpack sites in tests and examples

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-13 19:28:57 -04:00
Gud Boi e89fe03da7 Fix `LinkedTaskChannel` docstrings from GH bot review
Address valid findings from copilot's PR #413 review
(https://github.com/goodboy/tractor/pull/413
 #pullrequestreview-3925876037):

- `.get()` docstring referenced non-existent
  `._from_trio` attr, correct to `._to_aio`.
- `.send()` docstring falsely claimed error-raising
  on missing `from_trio` arg; reword to describe the
  actual `.put_nowait()` enqueue behaviour.
- `.open_channel_from()` return type annotation had
  `tuple[LinkedTaskChannel, Any]` but `yield` order
  is `(first, chan)`; fix annotation + docstring to
  match actual `tuple[Any, LinkedTaskChannel]`.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-13 19:28:57 -04:00
Gud Boi 417b796169 Use `chan: LinkedTaskChannel` API in all aio-task fns
Convert every remaining `to_trio`/`from_trio` fn-sig style
to the new unified `chan: LinkedTaskChannel` iface added in
prior commit (c46e9ee8).

Deats,
- `to_trio.send_nowait(val)` (1st call) -> `chan.started_nowait(val)`
- `to_trio.send_nowait(val)` (subsequent) -> `chan.send_nowait(val)`
- `await from_trio.get()` -> `await chan.get()`

Converted fns,
- `sleep_and_err()`, `push_from_aio_task()` in
  `tests/test_infected_asyncio.py`
- `sync_and_err()` in `tests/test_root_infect_asyncio.py`
- `aio_streamer()` in
  `tests/test_child_manages_service_nursery.py`
- `aio_echo_server()` in
  `examples/infected_asyncio_echo_server.py`
- `bp_then_error()` in `examples/debugging/asyncio_bp.py`

Also,
- drop stale comments referencing old param names.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-13 19:28:57 -04:00
Gud Boi 36cbc07602 Tried out an alt approach for `.to_asyncio` crashes
This change is masked out now BUT i'm leaving it in for reference.

I was debugging a multi-actor fault where the primary source actor was
an infected-aio-subactor (`brokerd.ib`) and it seemed like the REPL was only
entering on the `trio` side (at a `.open_channel_from()`) and not
eventually breaking in the `asyncio.Task`. But, since (changing
something?) it seems to be working now, it's just that the `trio` side
seems to sometimes handle before the (source/causing and more
child-ish) `asyncio`-task, which is a bit odd and not expected..
We could likely refine (maybe with an inter-loop-task REPL lock?) this
at some point and ensure a child-`asyncio` task which errors always
grabs the REPL **first**?

Lowlevel deats/further-todos,
- add (masked) `maybe_open_crash_handler()` block around
  `asyncio.Task` execution with notes about weird parent-addr
  delivery bug in `test_sync_pause_from_aio_task`
  * yeah dunno what that's about but made a bug; seems to be IPC
    serialization of the `TCPAddress` struct somewhere??
- add inter-loop lock TODO for avoiding aio-task clobbering
  trio-tasks when both crash in debug-mode

Also,
- change import from `tractor.devx.debug` to `tractor.devx`
- adjust `get_logger()` call to use new implicit mod-name detection
  added to `.log.get_logger()`, i.e. sin `name=__name__`.
- some teensie refinements to `open_channel_from()`:
  * swap return type annotation for  to `tuple[LinkedTaskChannel, Any]`
    (was `Any`).
  * update doc-string to clarify started-value delivery
  * add err-log before `.pause()` in what should be an unreachable path.
  * add todo to swap the `(first, chan)` pair to match that of ctx..

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])

[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-13 19:28:57 -04:00
Tyler Goodlet 1f2fad22ee Extend `.to_asyncio.LinkedTaskChannel` for aio side
With methods to comms similar to those that exist for the `trio` side,
- `.get()` which proxies verbatim to the `._to_aio: asyncio.Queue`,
- `.send_nowait()` which thin-wraps to `._to_trio: trio.MemorySendChannel`.

Obviously the more correct design is to break up the channel type into
a pair of handle types, one for each "side's" task in each event-loop,
that's hopefully coming shortly in a follow up patch B)

Also,
- fill in some missing doc strings, tweak some explanation comments and
  update todos.
- adjust the `test_aio_errors_and_channel_propagates_and_closes()` suite
  to use the new `chan` fn-sig-API with `.open_channel_from()` including
  the new methods for msg comms; ensures everything added here works e2e.
2026-03-13 19:28:57 -04:00
Tyler Goodlet ca5f6f50a8 Explain the `infect_asyncio: bool` param to pass in RTE msg 2026-03-13 19:28:57 -04:00
Bd a7ff1387c7
Merge pull request #414 from goodboy/struct_field_filtering
Hide private fields in `Struct.pformat()` output
2026-03-13 19:22:22 -04:00
Gud Boi abbb4a79c8 Drop unused import noticed by `copilot` 2026-03-13 11:52:18 -04:00
Gud Boi 1529095c32 Add `tests/msg/` sub-pkg, audit `pformat()` filtering
Reorganize existing msg-related test suites under
a new `tests/msg/` subdir (matching `tests/devx/`
and `tests/ipc/` convention) and add unit tests for
the `_`-prefixed field filtering in `pformat()`.

Deats,
- `git mv` `test_ext_types_msgspec` and `test_pldrx_limiting` into
  `tests/msg/`.
- add `__init__.py` + `conftest.py` for the new test sub-pkg.
- add new `test_pretty_struct.py` suite with 8 unit tests:
  - parametrized field visibility (public shown, `_`-private hidden,
    mixed)
  - direct `iter_struct_ppfmt_lines()` assertion
  - nested struct recursion filtering
  - empty struct edge case
  - real `MsgDec` via `mk_dec()` hiding `_dec`
  - `repr()` integration via `Struct.__repr__()`

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-12 18:32:31 -04:00
Gud Boi 8215a7ba34 Hide private fields in `Struct.pformat()` output
Skip fields starting with `_` in pretty-printed struct output
to avoid cluttering displays with internal/private state (and/or accessing
private properties which have errors Bp).

Deats,
- add `if k[0] == '_': continue` check to skip private fields
- change nested `if isinstance(v, Struct)` to `elif` since we
  now have early-continue for private fields
- mv `else:` comment to clarify it handles top-level fields
- fix indentation of `yield` statement to only output
  non-private, non-nested fields

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-10 00:33:32 -04:00
Bd c1c4d85958
Merge pull request #406 from goodboy/macos_support
macOS support (for the fancy peeps with nice furniture)
2026-03-10 00:28:54 -04:00
Gud Boi 88b084802f Merge `testing-macos` into unified `testing` matrix
Drop the separate `testing-macos` job and add
`macos-latest` to the existing OS matrix; bump
timeout to 16 min to accommodate macOS runs.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-09 23:28:58 -04:00
Gud Boi bf1dcea9d1 Announce macOS support in `pyproject` and README
- add `"Operating System :: MacOS"` classifier.
- add macOS bullet to README's TODO/status section.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-09 23:23:58 -04:00
Bd 5c270b89d5
Merge pull request #342 from goodboy/macos_in_ci
Macos in ci
2026-03-09 20:33:38 -04:00
Gud Boi 6ee0149e8d Another cancellation test timeout bump for non-linux 2026-03-09 19:46:42 -04:00
Gud Boi 9c4cd869fb OK-FINE, skip streaming docs example on macos!
It seems something is up with their VM-img or wtv bc i keep increasing
the subproc timeout and nothing is changing. Since i can't try
a `-xlarge` one without paying i'm just muting this test for now.
2026-03-09 19:46:42 -04:00
Gud Boi afd66ce3b7 Final try, drop logging level in streaming example to see if macos can cope.. 2026-03-09 19:46:42 -04:00
Gud Boi f9bdb1b35d Try one more timeout bumps for flaky docs streaming ex.. 2026-03-09 19:46:42 -04:00
Gud Boi d135ce94af Restyle `test_legacy_one_way_streaming` mod
- convert all doc-strings to `'''` multiline style.
- rename `nursery` -> `an`, `n` -> `tn` to match
  project-wide conventions.
- add type annotations to fn params (fixtures, test
  helpers).
- break long lines into multiline style for fn calls,
  assertions, and `parametrize` decorator lists.
- add `ids=` to `@pytest.mark.parametrize`.
- use `'` over `"` for string literals.
- add `from typing import Callable` import.
- drop spurious blank lines inside generators.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-09 19:46:25 -04:00
Gud Boi fb94aa0095 Tidy a typing-typo, add explicit `ids=` for paramed suites 2026-03-09 19:35:47 -04:00
Gud Boi b71e8575e5 Skip a couple more `ctlc` flaking suites 2026-03-09 19:31:16 -04:00
Gud Boi bbc028e84c Increase macos job timeout to 16s 2026-03-09 19:31:16 -04:00
Gud Boi 016306adf5 Allow `ctlcs_bish(<condition-args>)` skipping
Via ensuring `all(mark.args)` on wtv expressions are arg-passed to the
mark decorator; use it to skip the `test_subactor_breakpoint` suite when
`ctlc=True` since it seems too unreliable in CI.
2026-03-09 19:31:15 -04:00
Gud Boi 712c009790 Hike `testdir.spawn()` timeout on non-linux in CI 2026-03-09 19:30:41 -04:00
Gud Boi 79396b4a26 2x the ctl-c loop prompt-timeout for non-linux in CI 2026-03-09 19:30:41 -04:00
Gud Boi 5b2905b702 Xplatform tweaks for `daemon` fixture
There's a very sloppy registrar-actor-bootup syncing approach used in
this fixture (basically just guessing how long to sleep to wait for it
to init and bind the registry socket) using a `global _PROC_SPAWN_WAIT`
that needs to be made more reliable. But, for now i'm just playing along
with what's there to try and make less CI runs flaky by,

- sleeping *another* 1s when run from non-linux CI.
- reporting stdout (if any) alongside stderr on teardown.
- not strictly requiring a `proc.returncode == -2` indicating successful
  graceful cancellation via SIGINT; instead we now error-log and only
  raise the RTE on `< 0` exit code.
  * though i can't think of why this would happen other then an
    underlying crash which should propagate.. but i don't think any test
    suite does this intentionally rn?
  * though i don't think it should ever happen, having a CI run
    "error"-fail bc of this isn't all that illuminating, if there is
    some weird `.returncode == 0` termination case it's likely not
    a failure?

For later, see the new todo list; we should sync to some kind of "ping"
polling of the tpt address if possible which is already easy enough for
TCP reusing an internal closure from `._root.open_root_actor()`.
2026-03-09 19:30:41 -04:00
Gud Boi 776af3fce6 Register our `ctlcs_bish` marker to avoid warnings 2026-03-09 19:30:41 -04:00
Gud Boi 4639685770 Fill out types in `test_discovery` mod 2026-03-09 19:30:41 -04:00
Gud Boi 98a7d69341 Always pre-sleep in `daemon` fixture when in non-linux CI.. 2026-03-09 19:30:41 -04:00
Gud Boi ab6c955949 Lol fine! bump it a bit more XD 2026-03-09 19:30:41 -04:00
Gud Boi a72bb9321e Bleh, bump timeout again for docs-exs suite when in CI 2026-03-09 19:30:41 -04:00
Gud Boi 0e2949ea59 Bump docs-exs subproc timeout, exception log any timeouts 2026-03-09 19:30:41 -04:00
Gud Boi fb73935dbc Add a `test_log` fixture for emitting from *within* test bodies or fixtures 2026-03-09 19:30:41 -04:00
Gud Boi 94dfeb1441 Add delay before root-actor open, macos in CI.. 2026-03-09 19:30:41 -04:00
Gud Boi 9c1bcb23af Skip legacy-one-way suites on non-linux in CI 2026-03-09 19:30:41 -04:00
Gud Boi a1ea373f34 Ok.. try a longer prompt timeout? 2026-03-09 19:30:41 -04:00
Gud Boi e8f3d64e71 Increase prompt timeout for macos in CI 2026-03-09 19:30:41 -04:00
Gud Boi b30faaca82 Adjust debugger test suites for macos
Namely, after trying to get `test_multi_daemon_subactors` to work for
the `ctlc=True` case (for way too long), give up on that (see
todo/comments) and skip it; the normal case works just fine. Also tweak
the `test_ctxep_pauses_n_maybe_ipc_breaks` pattern matching for
non-`'UDS'` per the previous script commit; we can't use UDS alongside
`pytest`'s tmp dir generation, mega lulz.
2026-03-09 19:30:40 -04:00
Gud Boi 51701fc8dc Ok just skip `test_shield_pause` for macos..
Something something the SIGINT handler isn't being swapped correctly?
2026-03-09 19:29:47 -04:00
Gud Boi 7b89204afd Tweak `do_ctlc()`'s `delay` default
To be a null default and set to `0.1` when not passed by the caller so
as to avoid having to pass `0.1` if you wanted the
param-defined-default.

Also,
- in the `spawn()` fixtures's `unset_colors()` closure, add in a masked
  `os.environ['NO_COLOR'] = '1'` since i found it while trying to debug
  debugger tests.
- always return the `child.before` content from `assert_before()`
  helper; again it comes in handy when debugging console matching tests.
2026-03-09 19:29:18 -04:00
Gud Boi 82d02ef404 Lul, never use `'uds'` tpt for macos test-scripts
It's explained in the comment and i really think it's getting more
hilarious the more i learn about the arbitrary limitations of user space
with this tina platform.
2026-03-09 19:29:18 -04:00
Gud Boi b7546fd221 Longer timeout for `test_one_end_stream_not_opened`
On non-linux that is.
2026-03-09 19:29:18 -04:00
Gud Boi 86c95539ca Loosen shml test assert for key shortening on macos 2026-03-09 19:29:18 -04:00
Gud Boi 706a4b761b Add 6sec timeout around `test_simple_rpc` suite for macos 2026-03-09 19:29:18 -04:00
Gud Boi c5af2fa778 Add a `@no_macos` skipif deco 2026-03-09 19:29:18 -04:00
Gud Boi 86489cc453 Use py version in job `name`, consider macos in linux matrix? 2026-03-09 19:29:18 -04:00
Gud Boi 2631fb4ff3 Only run CI on <py3.14 2026-03-09 19:29:18 -04:00
Gud Boi aee86f2544 Run macos job on `uv` and newer `actions@v4` 2026-03-09 19:29:18 -04:00
Tyler Goodlet 83c8a8ad78 Add macos run using only the `trio` spawner 2026-03-09 19:29:18 -04:00
Gud Boi daae196048 Warn if `.ipc._uds.get_peer_pid()` returns null 2026-03-08 19:17:16 -04:00
Gud Boi 70efcb09a0 Slight refinements to `._state.get_rt_dir()`
Per the `copilot` review,
https://github.com/goodboy/tractor/pull/406#pullrequestreview-3893270953

now we also,
- pass `exists_ok=True` to `.mkdir()` to avoid conc races.
- expose `appname: str` param for caller override.
- normalize `subdir` to avoid escaping the base rt-dir location.
2026-03-08 19:17:16 -04:00
Gud Boi a7e74acdff Doc `getsockopt()` args (for macOS)
Per the questionable `copilot` review which is detailed for follow up in
https://github.com/goodboy/tractor/issues/418. These constants are
directly linked from the kernel sources fwiw.
2026-03-08 19:17:16 -04:00
Gud Boi 9c3d3bcec1 Add prompt flush hack for `bash` on macos as well.. 2026-03-08 19:17:16 -04:00
Gud Boi 521fb97fe9 Support UDS on macos (for realz)
Though it was a good (vibed) try by @dnks, the previous "fix" was not
actually adding unix socket support but merely sidestepping a crash due
to `get_peer_info()`'s impl never going to work on MacOS (and it was
never intended to).

This patch instead solves the underlying issue by implementing a new
`get_peer_pid()` helper which does in fact retrieve the peer's PID in
a more generic/cross-platform way (:fingers_crossed:); much thanks to
the linked SO answer for this solution!

Impl deats,
- add `get_peer_pid()` and call it from
  `MsgpackUDSStream.get_stream_addrs()` when we detect a non-'linux'
  platform, OW use the original soln: `get_stream_addrs()`.
- add a new case for the `match (peername, sockname)` with a
  `case (str(), str()):` which seems to at least work on macos.
- drop all the `LOCAL_PEERCRED` dynamic import branching since it was
  never needed and was never going to work.
2026-03-08 19:17:16 -04:00
Gud Boi d8a3969048 Also shorten shm-key for `ShmList` on macos
Same problem as for the `ShmArray` tokens, so tweak and reuse
the `_shorten_key_for_macos()` helper and call it from
`open_shm_list()` similarly.

Some tweaks/updates to the various helpers,
- support `prefix/suffix` inputs and if provided take their lengths and
  subtract them from the known *macOS shm_open() has a 31 char limit
  (PSHMNAMLEN)* when generating and using the `hashlib.sha256()` value
  which overrides (for now..) wtv `key` is passed by the caller.
- pass the appropriate `suffix='_first/_last'` values for the `ShmArray`
  token generators case.
- add a `prefix: str = 'shml_'` param to `open_shm_list()`.
- better log formatting with `!r` to report any key shortening.
2026-03-08 19:17:16 -04:00
Gud Boi 01c0db651a Port macOS shm 31-char name limit hack from `piker`
Adapt the `PSHMNAMLEN` fix from `piker.data._sharedmem` (orig commit
96fb79ec thx @dnks!) to `tractor.ipc._shm` accounting for the
module-local differences:

- Add `hashlib` import for sha256 key hashing
- Add `key: str|None` field to `NDToken` for storing
  the original descriptive key separate from the
  (possibly shortened) OS-level `shm_name`
- Add `__eq__()`/`__hash__()` to `NDToken` excluding
  the `key` field from identity comparison
- Add `_shorten_key_for_macos()` using `t_` prefix
  (vs piker's `p_`) with 16 hex chars of sha256
- Use `platform.system() == 'Darwin'` in `_make_token()`
  (tractor already imports the `platform` module vs
  piker's `sys.platform`)
- Wrap `shm_unlink()` in `ShmArray.destroy()` with
  `try/except FileNotFoundError` for teardown races
  (was already done in `SharedInt.destroy()`)
- Move token creation before `SharedMemory()` alloc in
  `open_shm_ndarray()` so `token.shm_name` is used
  as the OS-level name
- Use `lookup_key` pattern in `attach_shm_ndarray()`
  to decouple `_known_tokens` dict key from OS name

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-08 19:17:16 -04:00
Gud Boi 7bcd7aca2b Reorg `socket` conditional imports a bit
Move the multi-platorm-supporting conditional/dynamic `socket` constant
imports to *after* the main cross-platform ones.
Also add constant typing and reformat comments a bit for the macOS case.
2026-03-08 19:17:16 -04:00
Gud Boi 920d0043b4 Force parent subdirs for macos 2026-03-08 19:17:16 -04:00
wygud 93b9a6cd97 Add macOS compatibility for Unix socket credential passing
Make socket credential imports platform-conditional in `.ipc._uds`.
- Linux: use `SO_PASSCRED`/`SO_PEERCRED` from socket module
- macOS: use `LOCAL_PEERCRED` (0x0001) instead, no need for `SO_PASSCRED`
- Conditionally call `setsockopt(SO_PASSCRED)` only on Linux

Fixes AttributeError on macOS where SO_PASSCRED doesn't exist.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-03-08 19:17:16 -04:00
Tyler Goodlet e7cefba67f Use `platformdirs` for `.config.get_rt_dir()`
Thanks to the `tox`-dev community for such a lovely pkg which seems to
solves all the current cross-platform user-dir problems B)

Also this,
- now passes `platformdirs.user_runtime_dir(appname='tractor')`
  and allows caller to pass an optional `subdir` under `tractor/`
  if desired.
- drops the `.config._rtdir: Path` mod var.
- bumps the lock file with the new dep.
2026-03-08 19:16:49 -04:00
Bd 683476cc96
Merge pull request #421 from goodboy/py_pkging_update
Py pkging and version support update, go 3.12 and 3.13
2026-03-08 19:14:39 -04:00
Gud Boi ad24df0ed7 Drop `pytest.ini`, now covered in `pyproject.toml` 2026-03-08 18:58:30 -04:00
Gud Boi a1622c0b94 Bump `ruff.toml` to target py313 2026-03-08 18:39:08 -04:00
Gud Boi a385d20810 Disable the `xonsh` autoloaded `pytest` plugin 2026-03-08 18:38:31 -04:00
Gud Boi 7f9044c1ef Bump pkg classifiers to match py versions range 2026-03-08 18:16:30 -04:00
Gud Boi d0618e3cb4 Pin to py<3.14 (particularly for macos) 2026-03-08 18:16:30 -04:00
Gud Boi a5bebf76d5 Pin to py-3.12+ and pin-up some deps
Namely to get a fix (i patched in) to `pdbp` and the latest
prompt-injection feats from `xonsh` B)

Also leave in a (masked) `.uv.sources.pdbp` section for easy
patch-test-submit in the future from my own fork and bump the lock file
to match!
2026-03-08 18:16:21 -04:00
Bd 814b2e7e62
Merge pull request #416 from goodboy/claudy_skillz
Claudy skillz: kicking off some `claude` code skill-files
2026-03-04 21:36:49 -05:00
Gud Boi 1704f73504 Add local `claude` settings for commit-msg perms
Auto-allow the tool calls used by the `/commit-msg` skill
so the workflow requires minimal user prompting.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-04 16:09:37 -05:00
Gud Boi c735fc8544 Update `.gitignore` for `gish` local files
Rename `gitea/` comment to `gish`, add `gh/` ignore, and
expand TODOs about syncing with git hosting services.
Also mute any `.claude/*_commit*.txt` files.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-04 16:06:27 -05:00
Gud Boi c5ea6040bf Improve `/commit-msg` skill fmting + consistency
Align `SKILL.md` and `style-guide-reference.md` with the
claude-code skills docs and actual usage conventions.

Deats,
- add missing frontmatter fields: `argument-hint`,
  `disable-model-invocation`, and scope `Bash` tool to
  `Bash(git *)` prefix pattern per docs.
- add `Grep`/`Glob` to `allowed-tools`.
- restructure `!`backtick`` usage for proper dynamic
  context injection (not mixed with instructional text).
- use markdown link for `style-guide-reference.md` ref
  per docs' supporting-files convention.
- switch timestamp format to cross-OS-safe
  `date -u +%Y%m%dT%H%M%SZ`.

Also,
- rm errant blank lines between footer attribution and
  reference-def lines in both files.
- fix double space in style guide subject length desc.
- reflow long lines for 67 char limit.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-04 15:05:17 -05:00
Gud Boi d4f2fa547a Extend `.gitignore` for dev/gen artifacts
Add ignore rules for commit-msg gen tmp files, nix dev
profiles, vim sessions, macOS metadata, gitea local docs,
and private LLM conversation logs.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-01 17:46:03 -05:00
Gud Boi 20896bfbab Add `commit-msg` skill + style guide reference
Add a `claude-code` skill for generating commit msgs
matching `tractor`'s style, plus a companion style guide
derived from analysis of 500 repo commits.

Deats,
- `SKILL.md` defines the `commit-msg` skill with YAML
  frontmatter, outlining the generation process, formatting
  rules, and footer conventions.
- `style-guide-reference.md` documents verb frequencies,
  backtick usage patterns, section markers, abbreviations,
  and tone from historical commit analysis.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-01 15:46:27 -05:00
Bd 70bb77280e
Merge pull request #411 from goodboy/tpt_tolerance
Tpt-tolerance: more lowlevel `trio` CRE/BRE -> `TransportClosed` translations
2026-02-19 16:40:17 -05:00
Gud Boi 916f88a070 Less newlines in `._rpc` log msg 2026-02-19 16:31:54 -05:00
Gud Boi 91f2f3ec10 Use test-harness `loglevel` in inter-peer suite 2026-02-19 16:29:20 -05:00
Tyler Goodlet 3e5124e184 Hide `._rpc._invoke()` frame, again.. 2026-02-19 16:28:22 -05:00
Gud Boi fa86269e30 Stuff from auto-review in https://github.com/goodboy/tractor/pull/412 .. 2026-02-19 16:20:21 -05:00
Gud Boi d0b92bbeba Clean up `._transport` error-case comment
Expand and clarify the comment for the default `case _`
block in the `.send()` error matcher, noting that we
console-error and raise-thru for unexpected disconnect
conditions.

(this patch was suggested by copilot in,
 https://github.com/goodboy/tractor/pull/411)

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-02-19 16:18:39 -05:00
Gud Boi 9470815f5a Fix `spawn` fixture cleanup + test assertions
Improve the `spawn` fixture teardown logic in
`tests/devx/conftest.py` fixing the while-else bug, and fix
`test_advanced_faults` genexp for `TransportClosed` exc type
checking.

Deats,
- replace broken `while-else` pattern with direct
  `if ptyproc.isalive()` check after the SIGINT loop.
- fix undefined `spawned` ref -> `ptyproc.isalive()` in
  while condition.
- improve walrus expr formatting in timeout check (multiline
  style).

Also fix `test_ipc_channel_break_during_stream()` assertion,
- wrap genexp in `all()` call so it actually checks all excs
  are `TransportClosed` instead of just creating an unused
  generator.

(this patch was suggested by copilot in,
 https://github.com/goodboy/tractor/pull/411)

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-02-19 16:14:11 -05:00
Gud Boi 592d918394 Tweak `test_inter_peer_cancellation` for races
Adjust `basic_echo_server()` default sequence len to avoid the race
where the 'tell_little_bro()` finished streaming **before** the
echo-server sub is cancelled by its peer subactor (which is the whole
thing we're testing!).

Deats,
- bump `rng_seed` default from 50 -> 100 to ensure peer
  cancel req arrives before echo dialog completes on fast hw.
- add `trio.sleep(0.001)` between send/receive in msg loop on the
  "client" streamer side to give cancel request transit more time to
  arrive.

Also,
- add more native `tractor`-type hints.
- reflow `basic_echo_server()` doc-string for 67 char limit
- add masked `pause()` call with comment about unreachable
  code path
- alphabetize imports: mv `current_actor` and `open_nursery`
  below typed imports

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-02-19 15:24:42 -05:00
Gud Boi 0cddc67bdb Add doc-strs to `get_root()` + `maybe_open_portal()`
Brief descriptions for both fns in `._discovery` clarifying
what each delivers and under what conditions.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-02-19 13:55:02 -05:00
Gud Boi 052fe2435f Improve `Channel` doc-strs + minor cleanups
Flesh out missing method doc-strings, improve log msg formatting and
assert -> `RuntimeError` for un-inited tpt layer.

Deats,
- add doc-string to `.send()` noting `TransportClosed` raise
  on comms failures.
- add doc-string to `.recv()`.
- expand `._aiter_msgs()` doc-string, line-len reflow.
- add doc-string to `.connected()`.
- convert `assert self._transport` -> `RuntimeError` raise
  in `._aiter_msgs()` for more explicit crashing.
- expand `_connect_chan()` doc-string, note it's lowlevel
  and suggest `.open_portal()` to user instead.
- factor out `src_exc_str` in `TransportClosed` log handler
  to avoid double-call
- use multiline style for `.connected()` return expr.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-02-19 13:55:02 -05:00
Gud Boi 28819bf5d3 Add `Actor.is_root()` convenience predicate meth 2026-02-19 13:55:02 -05:00
Gud Boi 07c2ba5c0d Drop `trio`-exc-catching if tpt-closed covers them
Remove the `trio.ClosedResourceError` and `trio.BrokenResourceError`
handling that should now be subsumed by `TransportClosed` re-raising out
of the `.ipc` stack.

Deats,
- drop CRE and BRE from `._streaming.MsgStream.aclose()/.send()` blocks.
- similarly rm from `._context.open_context_from_portal()`.
- also from `._portal.Portal.cancel_actor()` and drop the
  (now-completed-todo) comment about this exact thing.

Also add comment in `._rpc.try_ship_error_to_remote()` noting the
remaining `trio` catches there are bc the `.ipc` layers *should* be
wrapping them; thus `log.critical()` use is warranted.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-02-19 13:55:02 -05:00
Gud Boi 50f40f427b Include `TransportClosed` in tpt-layer err handling
Add `TransportClosed` to except clauses where `trio`'s own
resource-closed errors are already caught, ensuring our
higher-level tpt exc is also tolerated in those same spots.
Likely i will follow up with a removal of the `trio` variants since most
*should be* caught and re-raised as tpt-closed out of the `.ipc` stack
now?

Add `TransportClosed` to various handler blocks,
- `._streaming.MsgStream.aclose()/.send()` except blocks.
- the broken-channel except in `._context.open_context_from_portal()`.
- obvi import it where necessary in those ^ mods.

Adjust `test_advanced_faults` suite + exs-script to match,
- update `ipc_failure_during_stream.py` example to catch
  `TransportClosed` alongside `trio.ClosedResourceError`
  in both the break and send-check paths.
- shield the `trio.sleep(0.01)` after tpt close in example to avoid
  taskc-raise/masking on that checkpoint since we want to simulate
  waiting for a user to send a KBI.
- loosen `ExceptionGroup` assertion to `len(excs) <= 2` and ensure all
  excs are `TransportClosed`.
- improve multi-line formatting, minor style/formatting fixes in
  condition expressions.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-02-19 13:55:02 -05:00
Gud Boi bf6de55865 Improve tpt-closed msg-fmt/content and CRE case matching
Refine tpt-error reporting to include closure attribution (`'locally'`
vs `'by peer'`), tighten match conditions and reduce needless newlines
in exc reprs.

Deats,
- factor out `trans_err_msg: str` and `by_whom: str` into a `dict`
  lookup before the `match:` block to pair specific err msgs to closure
  attribution strings.
- use `by_whom` directly as `CRE` case guard condition
  (truthy when msg matches known underlying CRE msg content).
- conveniently include `by_whom!r` in `TransportClosed` message.
- fix `'locally ?'` -> `'locally?'` in send-side `CRE`
  handler (drop errant space).
- add masked `maybe_pause_bp()` calls at both `CRE` sites (from when
  i was tracing a test harness issue where the UDS socket path wasn't
  being cleaned up on teardown).
- drop trailing `\n` from `body=` args to `TransportClosed`.
- reuse `trans_err_msg` for the `BRE`/broken-pipe guard.

Also adjust testing, namely `test_ctxep_pauses_n_maybe_ipc_breaks`'s
expected patts-set for new msg formats to be raised out of
`.ipc._transport`.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-02-19 13:55:02 -05:00
Gud Boi 5ded99a886 Add a `._trace.maybe_pause_bp()` for tpt-broken cases
Internal helper which falls back to sync `pdb` when the
child actor can't reach root to acquire the TTY lock.

Useful when debugging tpt layer failures (intentional or
otherwise) where a sub-actor can no longer IPC-contact the
root to coordinate REPL access; root uses `.pause()` as
normal while non-root falls back to `mk_pdb().set_trace()`.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-02-19 13:55:02 -05:00
Gud Boi 7145fa364f Add `SIGINT` cleanup to `spawn` fixture in `devx/conftest`
Convert `spawn` fixture to a generator and add post-test graceful
subproc cleanup via `SIGINT`/`SIGKILL` to avoid leaving stale `pexpect`
child procs around between test runs as well as any UDS-tpt socket files
under the system runtime-dir.

Deats,
- convert `return _spawn` -> `yield _spawn` to enable
  post-yield teardown logic.
- add a new `nonlocal spawned` ref so teardown logic can access the last
  spawned child from outside the delivered spawner fn-closure.
- add `SIGINT`-loop after yield with 5s timeout, then
  `SIGKILL` if proc still alive.
- add masked `breakpoint()` and TODO about UDS path cleanup

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-02-19 13:55:02 -05:00
Gud Boi f8e25688c7 Unmask `ClosedResourceError` handling in `._transport`
Unmask the CRE case block for peer-closed socket errors which already
had a TODO about reproducing the condition. It appears this case can
happen during inter-actor comms teardowns in `piker`, but i haven't been
able to figure out exactly what reproduces it yet..

So activate the block again for that 'socket already closed'-msg case,
and add a TODO questioning how to reproduce it.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-02-12 00:51:50 -05:00
Tyler Goodlet c3f455a8ec Mask tpt-closed handling of `chan.send(return_msg)`
A partial revert of commit c05d08e426 since it seem we already
suppress tpt-closed errors lower down in `.ipc.Channel.send()`; given
that i'm pretty sure this new handler code should basically never run?

Left in a todo to remove the masked content once i'm done more
thoroughly testing under `piker`.
2026-02-12 00:51:50 -05:00
Tyler Goodlet f78e842fba More `TransportClosed`-handling around IPC-IO
For IPC-disconnects-during-teardown edge cases, augment some `._rpc`
machinery,
- in `._invoke()` around the `await chan.send(return_msg)` where we
  suppress if the underlying `Channel` already disconnected.
- add a disjoint handler in `_errors_relayed_via_ipc()` which just
  reports-n-reraises the exc (same as prior behaviour).
  * originally i thought it needed to be handled specially (to avoid
    being crash handled) but turns out that isn't necessary?
  * hence the also-added-bu-masked-out `debug_filter` / guard expression
    around the `await debug._maybe_enter_pm()` line.
- show the `._invoke()` frame for the moment.
2026-02-12 00:51:50 -05:00
Bd 3638b80c9d
Merge pull request #412 from goodboy/root_actor_raddrs_fix
Non-registrar, root actor `_root_addrs` runtime-vars fix
2026-02-12 00:49:40 -05:00
Gud Boi 2ed9e65530 Clear rtvs state on root shutdown..
Fixes the bug discovered in last test update, not sure how this wasn't
caught already XD
2026-02-11 22:17:26 -05:00
Gud Boi 6cab363c51 Catch-n-fail on stale `_root_addrs` state..
Turns out we aren't clearing the `._state._runtime_vars` entries in
between `open_root_actor` calls.. This test refinement catches that by
adding runtime-vars asserts on the expected root-addrs value; ensure
`_runtime_vars['_root_addrs'] ONLY match the values provided by the
test's CURRENT root actor.

This causes a failure when the (just added)
`test_non_registrar_spawns_child` is run as part of the module suite,
it's fine when run standalone.
2026-02-11 22:17:26 -05:00
Gud Boi 8aee24e83f Fix when root-actor addrs is set as rtvs
Move `_root_addrs` assignment to after `async_main()` unblocks (via
`.started()`) which now delivers the bind addrs , ensuring correct
`UnwrappedAddress` propagation into `._state._runtime_vars` for
non-registar root actors..

Previously for non-registrar root actors the `._state._runtime_vars`
entries were being set as `Address` values which ofc IPC serialize
incorrectly rn vs. the unwrapped versions, (well until we add a msgspec
for their structs anyway) and thus are passed in incorrect form to
children/subactors during spawning..

This fixes the issue by waiting for the `.ipc.*` stack to
bind-and-resolve any randomly allocated addrs (by the OS) until after
the initial `Actor` startup is complete.

Deats,
- primarily, mv `_root_addrs` assignment from before `root_tn.start()`
  to after, using started(-ed) `accept_addrs` now delivered from
  `._runtime.async_main()`..
- update `task_status` type hints to match.
- unpack and set the `(accept_addrs, reg_addrs)` tuple from
  `root_tn.start()` call into `._state._runtime_vars` entries.
- improve and embolden comments distinguishing registrar vs non-registrar
  init paths, ensure typing reflects wrapped vs. unwrapped addrs.

Also,
- add a masked `mk_pdb().set_trace()` for debugging `raddrs` values
  being "off".
- add TODO about using UDS on linux for root mailbox
- rename `trans_bind_addrs` -> `tpt_bind_addrs` for clarity.
- expand comment about random port allocation for
  non-registrar case

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-02-11 22:17:26 -05:00
Gud Boi cdcc1b42fc Add test for non-registrar root sub-spawning
Ensure non-registrar root actors can spawn children and that
those children receive correct parent contact info. This test
catches the bug reported in,

https://github.com/goodboy/tractor/issues/410

Add new `test_non_registrar_spawns_child()` which spawns a sub-actor
from a non-registrar root and verifies the child can manually connect
back to its parent using `get_root()` API, auditing
`._state._runtime_vars` addr propagation from rent to child.

Also,
- improve type hints throughout test suites
  (`subprocess.Popen`, `UnwrappedAddress`, `Aid` etc.)
- rename `n` -> `an` for actor nursery vars
- use multiline style for function signatures

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-02-11 22:17:26 -05:00
Bd 51ac0c623e
Merge pull request #402 from goodboy/log_sys_testing
Log sys testing, starting to get "serious" about it..
2026-02-11 22:13:17 -05:00
Gud Boi 3f0bde1bf8 Use bare `get_logger()` in `.to_asyncio` 2026-02-11 22:02:41 -05:00
Gud Boi fa1a15dce8 Cleaups per copilot PR review 2026-02-11 21:51:40 -05:00
Gud Boi 5850844297 Mk `test_implicit_mod_name_applied_for_child()` check init-mods
Test pkg-level init module and sub-pkg module logger naming
to better validate auto-naming logic.

Deats,
- create `pkg_init_mod` and write `mod_code` to it for
  testing pkg-level `__init__.py` logger instance creation.
  * assert `snakelib.__init__` logger name is `proj_name`.
- write `mod_code` to `subpkg/__init__.py`` as well and check the same.

Also,
- rename some vars,
  * `pkg_mod` -> `pkg_submod`,
  * `pkgmod` -> `subpkgmod`
- add `ModuleType` import for type hints
- improve comments explaining pkg init vs first-level
  sub-module naming expectations.
- drop trailing whitespace and unused TODO comment
- remove masked `breakpoint()` call

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-02-11 21:43:37 -05:00
Gud Boi ff02939213 Toss in some `colorlog` alts to try 2026-02-11 21:05:16 -05:00
Gud Boi d61e8caab2 Improve `test_log_sys` for new auto-naming logic
Add assertions and comments to better test the reworked
implicit module-name detection in `get_logger()`.

Deats,
- add `assert not tractor.current_actor()` check to verify
  no runtime is active during test.
- import `.log` submod directly for use.
- add masked `breakpoint()` for debugging mod loading.
- add comment about using `ranger` to inspect `testdir` layout
  of auto-generated py pkg + module-files.
- improve comments explaining pkg-root-log creation.
- add TODO for testing `get_logger()` call from pkg
  `__init__.py`
- add comment about first-pkg-level module naming.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-02-11 21:05:07 -05:00
Gud Boi 0b0c83e9da Drop `name=__name__` from all `get_logger()` calls
Use new implicit module-name detection throughout codebase to simplify
logger creation and leverage auto-naming from caller mod .

Main changes,
- drop `name=__name__` arg from all `get_logger()` calls
  (across 29 modules).
- update `get_console_log()` calls to include `name='tractor'` for
  enabling root logger in test harness and entry points; this ensures
  logic in `get_logger()` triggers so that **all** `tractor`-internal
  logging emits to console.
- add info log msg in test `conftest.py` showing test-harness
  log level

Also,
- fix `.actor.uid` ref to `.actor.aid.uid` in `._trace`.
- adjust a `._context` log msg formatting for clarity.
- add TODO comments in `._addr`, `._uds` for when we mv to
  using `multiaddr`.
- add todo for `RuntimeVars` type hint TODO in `.msg.types` (once we
  eventually get that all going obvi!)

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-02-11 21:04:49 -05:00
Gud Boi 5e7c0f264d Rework `.get_logger()`, better autonaming, deduping
Overhaul of the automatic-calling-module-name detection and sub-log
creation logic to avoid (at least warn) on duplication(s) and still
handle the common usage of a call with `name=__name__` from a mod's top
level scope.

Main `get_logger()` changes,
- refactor auto-naming logic for implicit `name=None` case such that we
  handle at least `tractor` internal "bare" calls from internal submods.
- factor out the `get_caller_mod()` closure (still inside
  `get_logger()`)for introspecting caller's module with configurable
  frame depth.
- use `.removeprefix()` instead of `.lstrip()` for stripping pkg-name
  from mod paths
- mv root-logger creation before sub-logger name processing
- improve duplicate detection for `pkg_name` in `name`
- add `_strict_debug=True`-only-emitted warnings for duplicate
  pkg/leaf-mod names.
- use `print()` fallback for warnings when no actor runtime is up at
  call time.

Surrounding tweaks,
- add `.level` property to `StackLevelAdapter` for getting
  current emit level as lowercase `str`.
- mv `_proj_name` def to just above `get_logger()`
- use `_curr_actor_no_exc` partial in `_conc_name_getters`
  to avoid runtime errors
- improve comments/doc-strings throughout
- keep some masked `breakpoint()` calls for future debugging

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-02-11 21:04:29 -05:00
Gud Boi edf1189fe0 Multi-line styling in `test.devx.conftest` 2026-02-11 21:04:22 -05:00
Tyler Goodlet de24bfe052 Mv `load_module_from_path()` to a new `._code_load` submod 2026-02-11 21:03:29 -05:00
Tyler Goodlet e235b96894 Use new `pkg_name` in log-sys test suites 2026-02-11 21:03:07 -05:00
Tyler Goodlet dea4b9fd93 Implicitly name sub-logs by caller's mod
That is when no `name` is passed to `get_logger()`, try to introspect
the caller's `module.__name__` and use it to infer/get the "namespace
path" to that module the same as if using `name=__name__` as in the most
common usage.

Further, change the `_root_name` to be `pkg_name: str`, a public and
more obvious param name, and deprecate the former. This obviously adds
the necessary impl to make the new
`test_sys_log::test_implicit_mod_name_applied_for_child` test pass.

Impl detalles for `get_logger()`,
- add `pkg_name` and deprecate `_root_name`, include failover logic
  and a warning.
- implement calling module introspection using
  `inspect.stack()/getmodule()` to get both the `.__name__` and
  `.__package__` info alongside adjusted logic to set the `name`
  when not provided but only when a new `mk_sublog: bool` is set.
- tweak the `name` processing for implicitly set case,
  - rename `sub_name` -> `pkg_path: str` which is the path
    to the calling module minus that module's name component.
  - only partition `name` if `pkg_name` is `in` it.
  - use the `_root_log` for `pkg_name` duplication warnings.

Other/related,
- add types to various public mod vars missing them.
- rename `.log.log` -> `.log._root_log`.
2026-02-11 21:03:07 -05:00
Tyler Goodlet 557e2cec6a Add an implicit-pkg-path-as-logger-name test
A bit of test driven dev to anticipate support  of `.log.get_logger()`
usage such that it can be called from arbitrary sub-modules, themselves
embedded in arbitrary sub-pkgs, of some project; the when not provided,
the `sub_name` passed to the `Logger.getChild(<sub_name>)` will be set
as the sub-pkg path "down to" the calling module.

IOW if you call something like,

`log = tractor.log.get_logger(pkg_name='mypylib')`

from some `submod.py` in a project-dir that looks like,

mypylib/
  mod.py
  subpkg/
    submod.py  <- calling module

the `log: StackLevelAdapter` child-`Logger` instance will have a
`.name: str = 'mypylib.subpkg'`, discluding the `submod` part since this
already rendered as the `{filename}` header in `log.LOG_FORMAT`.

Previously similar behaviour would be obtained by passing
`get_logger(name=__name__)` in the calling module and so much so it
motivated me to make this the default, presuming we can introspect for
the info.

Impl deats,
- duplicated a `load_module_from_path()` from `modden` to load the
  `testdir` rendered py project dir from its path.
 |_should prolly factor it down to this lib anyway bc we're going to
   need it for hot code reload? (well that and `watchfiles` Bp)
- in each of `mod.py` and `submod.py` render the `get_logger()` code
  sin `name`, expecting the (coming shortly) implicit introspection
  feat to do this.
- do `.name` and `.parent` checks against expected sub-logger values
  from `StackLevelAdapter.logger.getChildren()`.
2026-02-11 21:03:07 -05:00
Tyler Goodlet 0e3229f16d Start a logging-sys unit-test module
To start ensuring that when `name=__name__` is passed we try to
de-duplicate the `_root_name` and any `leaf_mod: str` since it's already
included in the headers as `{filename}`.

Deats,
- heavily document the de-duplication `str.partition()`s in
  `.log.get_logger()` and provide the end fix by changing the predicate,
  `if rname == 'tractor':` -> `if rname == _root_name`.
  * also toss in some warnings for when we still detect duplicates.
- add todo comments around logging "filters" (vs. our "adapter").
- create the new `test_log_sys.test_root_pkg_not_duplicated()` which
  runs green with the fixes from ^.
- add a ton of test-suite todos both for existing and anticipated
  logging sys feats in the new mod.
2026-02-11 21:03:07 -05:00
Bd 448d25aef4
Merge pull request #409 from goodboy/nixos_flake
Nixos flake, for the *too-hip-for-arch-ers*
2026-02-11 21:02:37 -05:00
Gud Boi 343c9e0034 Tweaks per the `copilot` PR review 2026-02-11 20:55:08 -05:00
Gud Boi 1dc27c5161 Add a dev-overlay nix flake
Based on the impure template from `pyproject.nix` and providing
a dev-shell for easy bypass-n-hack on nix(os) using `uv`.

Deats,
- include bash completion pkgs for devx/happiness.
- pull `ruff` from <nixpkgs> to avoid wheel (build) issues.
- pin to py313 `cpython` for now.
2026-01-23 16:27:19 -05:00
Gud Boi 14aefa4b11 Reorg dev deps into nested groups
Namely,
- `devx` for console debugging extras used in `tractor.devx`.
- `repl` for @goodboy's `xonsh` hackin utils.
- `testing` for harness stuffs.
- `lint` for whenever we start doing that; it requires special
  separation on nixos in order to pull `ruff` from pkgs.

Oh and bump the lock file.
2026-01-23 16:24:24 -05:00
281 changed files with 41340 additions and 5132 deletions

View File

@ -0,0 +1,38 @@
# Docs TODOs
## Auto-sync README code examples with source
The `docs/README.rst` has inline code blocks that
duplicate actual example files (e.g.
`examples/infected_asyncio_echo_server.py`). Every time
the public API changes we have to manually sync both.
Sphinx's `literalinclude` directive can pull code directly
from source files:
```rst
.. literalinclude:: ../examples/infected_asyncio_echo_server.py
:language: python
:caption: examples/infected_asyncio_echo_server.py
```
Or to include only a specific function/section:
```rst
.. literalinclude:: ../examples/infected_asyncio_echo_server.py
:language: python
:pyobject: aio_echo_server
```
This way the docs always reflect the actual code without
manual syncing.
### Considerations
- `README.rst` is also rendered on GitHub/PyPI which do
NOT support `literalinclude` - so we'd need a build
step or a separate `_sphinx_readme.rst` (which already
exists at `docs/github_readme/_sphinx_readme.rst`).
- Could use a pre-commit hook or CI step to extract code
from examples into the README for GitHub rendering.
- Another option: `sphinx-autodoc` style approach where
docstrings from the actual module are pulled in.

View File

@ -0,0 +1,125 @@
# `RuntimeVars` env-var lift — design plan
Status: **draft, awaiting user edits**
## Goal
Consolidate the sprawl of pytest CLI flags + ad-hoc env vars +
hardcoded fixture defaults into a *single* env-var-encoded
runtime-vars envelope, with a typed in-memory representation
(`tractor.runtime._state.RuntimeVars`) as the sole source of
truth.
## Why now
- `--tpt-proto`, `--spawn-backend`, `--diag-on-hang`,
`--diag-capture-delay` and (soon) `TRACTOR_REG_ADDR` etc. are
proliferating. Each adds a parsing seam.
- `tests/devx/test_debugger.py` invokes example scripts as
separate subprocesses; they currently can't see the
fixture-allocated `reg_addr` at all (root cause of why
parametrizing devx scripts on `reg_addr` is on your TODO).
- Concurrent pytest sessions on the same host collide on
shared defaults (the `registry@1616` race we just fixed is
one symptom; per-session unique addr is the structural
fix).
- `tractor.runtime._state.RuntimeVars: Struct` is already
defined and **unused** — its docstring even says it
"should be utilized as possible for future calls."
## Design
### Module: `tractor/_testing/_rtvars.py`
Lifted from `modden.runtime.env`, ~50 LOC, no new deps.
```python
_TRACTOR_RT_VARS_OSENV: str = '_TRACTOR_RT_VARS'
def dump_rtvars(rtvars: RuntimeVars|dict) -> tuple[str, str]:
'''str-serialize via `str(dict)` — ast.literal_eval-able'''
def load_rtvars(env: dict) -> RuntimeVars:
'''ast.literal_eval the env-var value, hydrate to struct'''
def get_rtvars(proc: psutil.Process|None = None) -> RuntimeVars:
'''read the var from a target proc's env (or current)'''
def update_rtvars(
rtvars: RuntimeVars|dict|None = None,
update_osenv: bool|dict = True,
) -> tuple[str, str]:
'''mutate + re-encode + (optionally) write to os.environ'''
```
### Encoding choice: `str(dict)` + `ast.literal_eval`
Pros:
- stdlib only
- handles all the types tractor's tests need: `str`, `int`,
`float`, `bool`, `None`, `list`, `tuple`, `dict`
- human-readable in the env (greppable, inspectable via
`cat /proc/<pid>/environ | tr '\0' '\n'`)
Cons:
- non-stdlib types (msgspec Structs, `Path`, custom classes)
must be lowered first — fine for the test fixture set
- not stable across Python versions for esoteric repr cases
(we don't hit any)
Alternatives considered:
- **msgpack**: adds a dep + binary form is ungreppable
- **json**: doesn't preserve tuples (becomes lists), which is
a common type for `reg_addr`
- **toml/yaml**: heavier deps, no real benefit
### `RuntimeVars` becomes the single source of truth
The legacy `_runtime_vars: dict[str, Any]` global in
`runtime/_state.py` becomes a *cached view* of a
`RuntimeVars` singleton instance:
- `get_runtime_vars()` returns either the struct or a
`.to_dict()` view depending on caller's preference
- `set_runtime_vars(...)` validates against the struct schema
- spawn-time SpawnSpec sends the struct (already does
conceptually — just gets typed)
- `__setattr__` `breakpoint()` debug instrumentation gets
removed (unrelated cleanup, mentioned in conversation)
### Migration path
**Phase 0** *(prep)*: strip the stray `breakpoint()` from
`RuntimeVars.__setattr__`.
**Phase 1**: land `_rtvars.py` as a leaf module, used only by
test infra. Subprocess-spawned scripts in `tests/devx/`
read `_TRACTOR_RT_VARS` on startup → reconstruct
`RuntimeVars` → call `tractor.open_root_actor(**rtvars.as_kwargs())`.
Concurrent runs become deterministic-isolated because each
session writes a unique `_registry_addrs` into the env.
**Phase 2**: migrate runtime callers (`_state.get_runtime_vars`,
spawn `SpawnSpec`, `Actor.async_main`) to operate on the
struct directly, with the dict as a compat view that gets
deprecated.
**Phase 3** *(structural)*: per-session bindspace subdir
`/run/user/<uid>/tractor/<session_uuid>/` — encoded in the
rt-vars envelope, picked up by every subactor automatically.
Obsoletes the entire bindspace-leak warning class.
## Open design questions (user input wanted)
- (placeholder for your edits)
- (placeholder)
- (placeholder)
## Out-of-scope for this lift
- Anything in `modden.runtime.env` related to `Spawn`,
`WmCtl`, `Wks` — that's a workspace orchestration layer,
not an env-var helper. We only lift the four utility
functions + the var name constant.
- Switching to msgpack/json — explicitly chosen against
above.

View File

@ -0,0 +1,42 @@
{
"permissions": {
"allow": [
"Bash(cp .claude/*)",
"Read(.claude/**)",
"Read(.claude/skills/run-tests/**)",
"Write(.claude/**/*commit_msg*)",
"Write(.claude/git_commit_msg_LATEST.md)",
"Skill(run-tests)",
"Skill(close-wkt)",
"Skill(open-wkt)",
"Skill(prompt-io)",
"Bash(date *)",
"Bash(git diff *)",
"Bash(git log *)",
"Bash(git status)",
"Bash(git remote:*)",
"Bash(git stash:*)",
"Bash(git mv:*)",
"Bash(git rev-parse:*)",
"Bash(test:*)",
"Bash(ls:*)",
"Bash(grep:*)",
"Bash(find:*)",
"Bash(ln:*)",
"Bash(cat:*)",
"Bash(mkdir:*)",
"Bash(gh pr:*)",
"Bash(gh api:*)",
"Bash(gh issue:*)",
"Bash(UV_PROJECT_ENVIRONMENT=py* uv sync:*)",
"Bash(UV_PROJECT_ENVIRONMENT=py* uv run:*)",
"Bash(echo EXIT:$?:*)",
"Bash(echo \"EXIT=$?\")",
"Read(/tmp/**)"
],
"deny": [],
"ask": []
},
"prefersReducedMotion": false,
"outputStyle": "default"
}

View File

@ -0,0 +1,225 @@
# Commit Message Style Guide for `tractor`
Analysis based on 500 recent commits from the `tractor` repository.
## Core Principles
Write commit messages that are technically precise yet casual in
tone. Use abbreviations and informal language while maintaining
clarity about what changed and why.
## Subject Line Format
### Length and Structure
- Target: ~50 chars with a hard-max of 67.
- Use backticks around code elements (72.2% of commits)
- Rarely use colons (5.2%), except for file prefixes
- End with '?' for uncertain changes (rare: 0.8%)
- End with '!' for important changes (rare: 2.0%)
### Opening Verbs (Present Tense)
Most common verbs from analysis:
- `Add` (14.4%) - wholly new features/functionality
- `Use` (4.4%) - adopt new approach/tool
- `Drop` (3.6%) - remove code/feature
- `Fix` (2.4%) - bug fixes
- `Move`/`Mv` (3.6%) - relocate code
- `Adjust` (2.0%) - minor tweaks
- `Update` (1.6%) - enhance existing feature
- `Bump` (1.2%) - dependency updates
- `Rename` (1.2%) - identifier changes
- `Set` (1.2%) - configuration changes
- `Handle` (1.0%) - add handling logic
- `Raise` (1.0%) - add error raising
- `Pass` (0.8%) - pass parameters/values
- `Support` (0.8%) - add support for something
- `Hide` (1.4%) - make private/internal
- `Always` (1.4%) - enforce consistent behavior
- `Mk` (1.4%) - make/create (abbreviated)
- `Start` (1.0%) - begin implementation
Other frequent verbs: `More`, `Change`, `Extend`, `Disable`, `Log`,
`Enable`, `Ensure`, `Expose`, `Allow`
### Backtick Usage
Always use backticks for:
- Module names: `trio`, `asyncio`, `msgspec`, `greenback`, `stackscope`
- Class names: `Context`, `Actor`, `Address`, `PldRx`, `SpawnSpec`
- Method names: `.pause_from_sync()`, `._pause()`, `.cancel()`
- Function names: `breakpoint()`, `collapse_eg()`, `open_root_actor()`
- Decorators: `@acm`, `@context`
- Exceptions: `Cancelled`, `TransportClosed`, `MsgTypeError`
- Keywords: `finally`, `None`, `False`
- Variable names: `tn`, `debug_mode`
- Complex expressions: `trio.Cancelled`, `asyncio.Task`
Most backticked terms in tractor:
`trio`, `asyncio`, `Context`, `.pause_from_sync()`, `tn`,
`._pause()`, `breakpoint()`, `collapse_eg()`, `Actor`, `@acm`,
`.cancel()`, `Cancelled`, `open_root_actor()`, `greenback`
### Examples
Good subject lines:
```
Add `uds` to `._multiaddr`, tweak typing
Drop `DebugStatus.shield` attr, add `.req_finished`
Use `stackscope` for all actor-tree rendered "views"
Fix `.to_asyncio` inter-task-cancellation!
Bump `ruff.toml` to target py313
Mv `load_module_from_path()` to new `._code_load` submod
Always use `tuple`-cast for singleton parent addrs
```
## Body Format
### General Structure
- 43.2% of commits have no body (simple changes)
- Use blank line after subject
- Max line length: 67 chars
- Use `-` bullets for lists (28.0% of commits)
- Rarely use `*` bullets (2.4%)
### Section Markers
Use these markers to organize longer commit bodies:
- `Also,` (most common: 26 occurrences)
- `Other,` (13 occurrences)
- `Deats,` (11 occurrences) - for implementation details
- `Further,` (7 occurrences)
- `TODO,` (3 occurrences)
- `Impl details,` (2 occurrences)
- `Notes,` (1 occurrence)
### Common Abbreviations
Use these freely (sorted by frequency):
- `msg` (63) - message
- `bg` (37) - background
- `ctx` (30) - context
- `impl` (27) - implementation
- `mod` (26) - module
- `obvi` (17) - obviously
- `tn` (16) - task name
- `fn` (15) - function
- `vs` (15) - versus
- `bc` (14) - because
- `var` (14) - variable
- `prolly` (9) - probably
- `ep` (6) - entry point
- `OW` (5) - otherwise
- `rn` (4) - right now
- `sig` (4) - signal/signature
- `deps` (3) - dependencies
- `iface` (2) - interface
- `subproc` (2) - subprocess
- `tho` (2) - though
- `ofc` (2) - of course
### Tone and Style
- Casual but technical (use `XD` for humor: 23 times)
- Use `..` for trailing thoughts (108 occurrences)
- Use `Woops,` to acknowledge mistakes (4 subject lines)
- Don't be afraid to show personality while being precise
### Example Bodies
Simple with bullets:
```
Add `multiaddr` and bump up some deps
Since we're planning to use it for (discovery)
addressing, allowing replacement of the hacky (pretend)
attempt in `tractor._multiaddr` Bp
Also pin some deps,
- make us py312+
- use `pdbp` with my frame indexing fix.
- mv to latest `xonsh` for fancy cmd/suggestion injections.
Bump lock file to match obvi!
```
With section markers:
```
Use `stackscope` for all actor-tree rendered "views"
Instead of the (much more) limited and hacky `.devx._code`
impls, move to using the new `.devx._stackscope` API which
wraps the `stackscope` project.
Deats,
- make new `stackscope.extract_stack()` wrapper
- port over frame-descing to `_stackscope.pformat_stack()`
- move `PdbREPL` to use `stackscope` render approach
- update tests for new stack output format
Also,
- tweak log formatting for consistency
- add typing hints throughout
```
## Special Patterns
### WIP Commits
Rare (0.2%) - avoid committing WIP if possible
### Merge Commits
Auto-generated (4.4%), don't worry about style
### File References
- Use `module.py` or `.submodule` style
- Rarely use `file.py:line` references (0 in analysis)
### Links
- GitHub links used sparingly (3 total)
- Prefer code references over external links
## Footer
The default footer should credit `claude` (you) for helping generate
the commit msg content:
```
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
```
Further, if the patch was solely or in part written
by `claude`, instead add:
```
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
```
## Summary Checklist
Before committing, verify:
- [ ] Subject line uses present tense verb
- [ ] Subject line ~50 chars (hard max 67)
- [ ] Code elements wrapped in backticks
- [ ] Body lines ≤67 chars
- [ ] Abbreviations used where natural
- [ ] Casual yet precise tone
- [ ] Section markers if body >3 paragraphs
- [ ] Technical accuracy maintained
## Analysis Metadata
```
Source: tractor repository
Commits analyzed: 500
Date range: 2019-2025
Analysis date: 2026-02-08
```
---
(this style guide was generated by [`claude-code`][claude-code-gh]
analyzing commit history)
[claude-code-gh]: https://github.com/anthropics/claude-code

View File

@ -0,0 +1,297 @@
---
name: conc-anal
description: >
Concurrency analysis for tractor's trio-based
async primitives. Trace task scheduling across
checkpoint boundaries, identify race windows in
shared mutable state, and verify synchronization
correctness. Invoke on code segments the user
points at, OR proactively when reviewing/writing
concurrent cache, lock, or multi-task acm code.
argument-hint: "[file:line-range or function name]"
allowed-tools:
- Read
- Grep
- Glob
- Task
---
Perform a structured concurrency analysis on the
target code. This skill should be invoked:
- **On demand**: user points at a code segment
(file:lines, function name, or pastes a snippet)
- **Proactively**: when writing or reviewing code
that touches shared mutable state across trio
tasks — especially `_Cache`, locks, events, or
multi-task `@acm` lifecycle management
## 0. Identify the target
If the user provides a file:line-range or function
name, read that code. If not explicitly provided,
identify the relevant concurrent code from context
(e.g. the current diff, a failing test, or the
function under discussion).
## 1. Inventory shared mutable state
List every piece of state that is accessed by
multiple tasks. For each, note:
- **What**: the variable/dict/attr (e.g.
`_Cache.values`, `_Cache.resources`,
`_Cache.users`)
- **Scope**: class-level, module-level, or
closure-captured
- **Writers**: which tasks/code-paths mutate it
- **Readers**: which tasks/code-paths read it
- **Guarded by**: which lock/event/ordering
protects it (or "UNGUARDED" if none)
Format as a table:
```
| State | Writers | Readers | Guard |
|---------------------|-----------------|-----------------|----------------|
| _Cache.values | run_ctx, moc¹ | moc | ctx_key lock |
| _Cache.resources | run_ctx, moc | moc, run_ctx | UNGUARDED |
```
¹ `moc` = `maybe_open_context`
## 2. Map checkpoint boundaries
For each code path through the target, mark every
**checkpoint** — any `await` expression where trio
can switch to another task. Use line numbers:
```
L325: await lock.acquire() ← CHECKPOINT
L395: await service_tn.start(...) ← CHECKPOINT
L411: lock.release() ← (not a checkpoint, but changes lock state)
L414: yield (False, yielded) ← SUSPEND (caller runs)
L485: no_more_users.set() ← (wakes run_ctx, no switch yet)
```
**Key trio scheduling rules to apply:**
- `Event.set()` makes waiters *ready* but does NOT
switch immediately
- `lock.release()` is not a checkpoint
- `await sleep(0)` IS a checkpoint
- Code in `finally` blocks CAN have checkpoints
(unlike asyncio)
- `await` inside `except` blocks can be
`trio.Cancelled`-masked
## 3. Trace concurrent task schedules
Write out the **interleaved execution trace** for
the problematic scenario. Number each step and tag
which task executes it:
```
[Task A] 1. acquires lock
[Task A] 2. cache miss → allocates resources
[Task A] 3. releases lock
[Task A] 4. yields to caller
[Task A] 5. caller exits → finally runs
[Task A] 6. users-- → 0, sets no_more_users
[Task A] 7. pops lock from _Cache.locks
[run_ctx] 8. wakes from no_more_users.wait()
[run_ctx] 9. values.pop(ctx_key)
[run_ctx] 10. acm __aexit__ → CHECKPOINT
[Task B] 11. creates NEW lock (old one popped)
[Task B] 12. acquires immediately
[Task B] 13. values[ctx_key] → KeyError
[Task B] 14. resources[ctx_key] → STILL EXISTS
[Task B] 15. 💥 RuntimeError
```
Identify the **race window**: the range of steps
where state is inconsistent. In the example above,
steps 910 are the window (values gone, resources
still alive).
## 4. Classify the bug
Categorize what kind of concurrency issue this is:
- **TOCTOU** (time-of-check-to-time-of-use): state
changes between a check and the action based on it
- **Stale reference**: a task holds a reference to
state that another task has invalidated
- **Lifetime mismatch**: a synchronization primitive
(lock, event) has a shorter lifetime than the
state it's supposed to protect
- **Missing guard**: shared state is accessed
without any synchronization
- **Atomicity gap**: two operations that should be
atomic have a checkpoint between them
## 5. Propose fixes
For each proposed fix, provide:
- **Sketch**: pseudocode or diff showing the change
- **How it closes the window**: which step(s) from
the trace it eliminates or reorders
- **Tradeoffs**: complexity, perf, new edge cases,
impact on other code paths
- **Risk**: what could go wrong (deadlocks, new
races, cancellation issues)
Rate each fix: `[simple|moderate|complex]` impl
effort.
## 6. Output format
Structure the full analysis as:
```markdown
## Concurrency analysis: `<target>`
### Shared state
<table from step 1>
### Checkpoints
<list from step 2>
### Race trace
<interleaved trace from step 3>
### Classification
<bug type from step 4>
### Fixes
<proposals from step 5>
```
## Tractor-specific patterns to watch
These are known problem areas in tractor's
concurrency model. Flag them when encountered:
### `_Cache` lock vs `run_ctx` lifetime
The `_Cache.locks` entry is managed by
`maybe_open_context` callers, but `run_ctx` runs
in `service_tn` — a different task tree. Lock
pop/release in the caller's `finally` does NOT
wait for `run_ctx` to finish tearing down. Any
state that `run_ctx` cleans up in its `finally`
(e.g. `resources.pop()`) is vulnerable to
re-entry races after the lock is popped.
### `values.pop()` → acm `__aexit__``resources.pop()` gap
In `_Cache.run_ctx`, the inner `finally` pops
`values`, then the acm's `__aexit__` runs (which
has checkpoints), then the outer `finally` pops
`resources`. This creates a window where `values`
is gone but `resources` still exists — a classic
atomicity gap.
### Global vs per-key counters
`_Cache.users` as a single `int` (pre-fix) meant
that users of different `ctx_key`s inflated each
other's counts, preventing teardown when one key's
users hit zero. Always verify that per-key state
(`users`, `locks`) is actually keyed on `ctx_key`
and not on `fid` or some broader key.
### `Event.set()` wakes but doesn't switch
`trio.Event.set()` makes waiting tasks *ready* but
the current task continues executing until its next
checkpoint. Code between `.set()` and the next
`await` runs atomically from the scheduler's
perspective. Use this to your advantage (or watch
for bugs where code assumes the woken task runs
immediately).
### `except` block checkpoint masking
`await` expressions inside `except` handlers can
be masked by `trio.Cancelled`. If a `finally`
block runs from an `except` and contains
`lock.release()`, the release happens — but any
`await` after it in the same `except` may be
swallowed. This is why `maybe_open_context`'s
cache-miss path does `lock.release()` in a
`finally` inside the `except KeyError`.
### Cancellation in `finally`
Unlike asyncio, trio allows checkpoints in
`finally` blocks. This means `finally` cleanup
that does `await` can itself be cancelled (e.g.
by nursery shutdown). Watch for cleanup code that
assumes it will run to completion.
### Unbounded waits in cleanup paths
Any `await <event>.wait()` in a teardown path is
a latent deadlock unless the event's setter is
GUARANTEED to fire. If the setter depends on
external state (peer disconnects, child process
exit, subsequent task completion) that itself
depends on the current task's progress, you have
a mutual wait.
Rule: **bound every `await X.wait()` in cleanup
paths with `trio.move_on_after()`** unless you
can prove the setter is unconditionally reachable
from the state at the await site. Concrete recent
example: `ipc_server.wait_for_no_more_peers()` in
`async_main`'s finally (see
`ai/conc-anal/subint_forkserver_test_cancellation_leak_issue.md`
"probe iteration 3") — it was unbounded, and when
one peer-handler was stuck the wait-for-no-more-
peers event never fired, deadlocking the whole
actor-tree teardown cascade.
### The capture-pipe-fill hang pattern (grep this first)
When investigating any hang in the test suite
**especially under fork-based backends**, first
check whether the hang reproduces under `pytest
-s` (`--capture=no`). If `-s` makes it go away
you're not looking at a trio concurrency bug —
you're looking at a Linux pipe-buffer fill.
Mechanism: pytest replaces fds 1,2 with pipe
write-ends. Fork-child subactors inherit those
fds. High-volume error-log tracebacks (cancel
cascade spew) fill the 64KB pipe buffer. Child
`write()` blocks. Child can't exit. Parent's
`waitpid`/pidfd wait blocks. Deadlock cascades up
the tree.
Pre-existing guards in `tests/conftest.py` encode
this knowledge — grep these BEFORE blaming
concurrency:
```python
# tests/conftest.py:258
if loglevel in ('trace', 'debug'):
# XXX: too much logging will lock up the subproc (smh)
loglevel: str = 'info'
# tests/conftest.py:316
# can lock up on the `_io.BufferedReader` and hang..
stderr: str = proc.stderr.read().decode()
```
Full post-mortem +
`ai/conc-anal/subint_forkserver_test_cancellation_leak_issue.md`
for the canonical reproduction. Cost several
investigation sessions before catching it —
because the capture-pipe symptom was masked by
deeper cascade-deadlocks. Once the cascades were
fixed, the tree tore down enough to generate
pipe-filling log volume → capture-pipe finally
surfaced. Grep-note for future-self: **if a
multi-subproc tractor test hangs, `pytest -s`
first, conc-anal second.**

View File

@ -0,0 +1,241 @@
# PR/Patch-Request Description Format Reference
Canonical structure for `tractor` patch-request
descriptions, designed to work across GitHub,
Gitea, SourceHut, and GitLab markdown renderers.
**Line length: wrap at 72 chars** for all prose
content (Summary bullets, Motivation paragraphs,
Scopes bullets, etc.). Fill lines *to* 72 — don't
stop short at 50-65. Only raw URLs in
reference-link definitions may exceed this.
## Template
```markdown
<!-- pr-msg-meta
branch: <branch-name>
base: <base-branch>
submitted:
github: ___
gitea: ___
srht: ___
-->
## <Title: present-tense verb + backticked code>
### Summary
- [<hash>][<hash>] Description of change ending
with period.
- [<hash>][<hash>] Another change description
ending with period.
- [<hash>][<hash>] [<hash>][<hash>] Multi-commit
change description.
### Motivation
<1-2 paragraphs: problem/limitation first,
then solution. Hard-wrap at 72 chars.>
### Scopes changed
- [<hash>][<hash>] `pkg.mod.func()` — what
changed.
* [<hash>][<hash>] Also adjusts
`.related_thing()` in same module.
- [<hash>][<hash>] `tests.test_mod` — new/changed
test coverage.
<!--
### Cross-references
Also submitted as
[github-pr][] | [gitea-pr][] | [srht-patch][].
### Links
- [relevant-issue-or-discussion](url)
- [design-doc-or-screenshot](url)
-->
(this pr content was generated in some part by
[`claude-code`][claude-code-gh])
[<hash>]: https://<service>/<owner>/<repo>/commit/<hash>
[claude-code-gh]: https://github.com/anthropics/claude-code
<!-- cross-service pr refs (fill after submit):
[github-pr]: https://github.com/<owner>/<repo>/pull/___
[gitea-pr]: https://<host>/<owner>/<repo>/pulls/___
[srht-patch]: https://git.sr.ht/~<owner>/<repo>/patches/___
-->
```
## Markdown Reference-Link Strategy
Use reference-style links for ALL commit hashes
and cross-service PR refs to ensure cross-service
compatibility:
**Inline usage** (in bullets):
```markdown
- [f3726cf9][f3726cf9] Add `reg_err_types()`
for custom exc lookup.
```
**Definition** (bottom of document):
```markdown
[f3726cf9]: https://github.com/goodboy/tractor/commit/f3726cf9
```
### Why reference-style?
- Keeps prose readable without long inline URLs.
- All URLs in one place — trivially swappable
per-service.
- Most git services auto-link bare SHAs anyway,
but explicit refs guarantee it works in *any*
md renderer.
- The `[hash][hash]` form is self-documenting —
display text matches the ref ID.
- Cross-service PR refs use the same mechanism:
`[github-pr][]` resolves via a ref-link def
at the bottom, trivially fillable post-submit.
## Cross-Service PR Placeholder Mechanism
The generated description includes three layers
of cross-service support, all using native md
reference-links:
### 1. Metadata comment (top of file)
```markdown
<!-- pr-msg-meta
branch: remote_exc_type_registry
base: main
submitted:
github: ___
gitea: ___
srht: ___
-->
```
A YAML-ish HTML comment block. The `___`
placeholders get filled with PR/patch numbers
after submission. Machine-parseable for tooling
(e.g. `gish`) but invisible in rendered md.
### 2. Cross-references section (in body)
```markdown
<!--
### Cross-references
Also submitted as
[github-pr][] | [gitea-pr][] | [srht-patch][].
-->
```
Commented out at generation time. After submitting
to multiple services, uncomment and the ref-links
resolve via the stubs at the bottom.
### 3. Ref-link stubs (bottom of file)
```markdown
<!-- cross-service pr refs (fill after submit):
[github-pr]: https://github.com/goodboy/tractor/pull/___
[gitea-pr]: https://pikers.dev/goodboy/tractor/pulls/___
[srht-patch]: https://git.sr.ht/~goodboy/tractor/patches/___
-->
```
Commented out with `___` number placeholders.
After submission: uncomment, replace `___` with
the actual number. Each service-specific copy
fills in all services' numbers so any copy can
cross-reference the others.
### Post-submission file layout
```
pr_msg_LATEST.md # latest draft (skill root)
msgs/
20260325T002027Z_mybranch_pr_msg.md # timestamped
github/
42_pr_msg.md # github PR #42
gitea/
17_pr_msg.md # gitea PR #17
srht/
5_pr_msg.md # srht patch #5
```
Each `<service>/<num>_pr_msg.md` is a copy with:
- metadata `submitted:` fields filled in
- cross-references section uncommented
- ref-link stubs uncommented with real numbers
- all services cross-linked in each copy
This mirrors the `gish` skill's
`<backend>/<num>.md` pattern.
## Commit-Link URL Patterns by Service
| Service | Pattern |
|-----------|-------------------------------------|
| GitHub | `https://github.com/<o>/<r>/commit/<h>` |
| Gitea | `https://<host>/<o>/<r>/commit/<h>` |
| SourceHut | `https://git.sr.ht/~<o>/<r>/commit/<h>` |
| GitLab | `https://gitlab.com/<o>/<r>/-/commit/<h>` |
## PR/Patch URL Patterns by Service
| Service | Pattern |
|-----------|-------------------------------------|
| GitHub | `https://github.com/<o>/<r>/pull/<n>` |
| Gitea | `https://<host>/<o>/<r>/pulls/<n>` |
| SourceHut | `https://git.sr.ht/~<o>/<r>/patches/<n>` |
| GitLab | `https://gitlab.com/<o>/<r>/-/merge_requests/<n>` |
## Scope Naming Convention
Use Python namespace-resolution syntax for
referencing changed code scopes:
| File path | Scope reference |
|---------------------------|-------------------------------|
| `tractor/_exceptions.py` | `tractor._exceptions` |
| `tractor/_state.py` | `tractor._state` |
| `tests/test_foo.py` | `tests.test_foo` |
| Function in module | `tractor._exceptions.func()` |
| Method on class | `.RemoteActorError.src_type` |
| Class | `tractor._exceptions.RAE` |
Prefix with the package path for top-level refs;
use leading-dot shorthand (`.ClassName.method()`)
for sub-bullets where the parent module is already
established.
## Title Conventions
Same verb vocabulary as commit messages:
- `Add` — wholly new feature/API
- `Fix` — bug fix
- `Drop` — removal
- `Use` — adopt new approach
- `Move`/`Mv` — relocate code
- `Adjust` — minor tweak
- `Update` — enhance existing feature
- `Support` — add support for something
Target 50 chars, hard max 70. Always backtick
code elements.
## Tone
Casual yet technically precise — matching the
project's commit-msg style. Terse but every bullet
carries signal. Use project abbreviations freely
(msg, bg, ctx, impl, mod, obvi, fn, bc, var,
prolly, ep, etc.).
---
(this format reference was generated by
[`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

View File

@ -0,0 +1,632 @@
---
name: run-tests
description: >
Run tractor test suite (or subsets). Use when the user wants
to run tests, verify changes, or check for regressions.
argument-hint: "[test-path-or-pattern] [--opts]"
allowed-tools:
- Bash(python -m pytest *)
- Bash(python -c *)
- Bash(python --version *)
- Bash(UV_PROJECT_ENVIRONMENT=py* uv run python *)
- Bash(UV_PROJECT_ENVIRONMENT=py* uv run pytest *)
- Bash(UV_PROJECT_ENVIRONMENT=py* uv sync *)
- Bash(UV_PROJECT_ENVIRONMENT=py* uv pip show *)
- Bash(git rev-parse *)
- Bash(ls *)
- Bash(cat *)
- Bash(jq * .pytest_cache/*)
# process inspection + SIGINT-first cleanup ladder (see
# the zombie-actor pre-flight / teardown steps below).
- Bash(ss *)
- Bash(pgrep *)
- Bash(pkill *)
- Bash(sleep *)
- Bash(rm -f /tmp/registry@*.sock)
- Read
- Grep
- Glob
- Task
- AskUserQuestion
---
Run the `tractor` test suite using `pytest`. Follow this
process:
## 1. Parse user intent
From the user's message and any arguments, determine:
- **scope**: full suite, specific file(s), specific
test(s), or a keyword pattern (`-k`).
- **transport**: which IPC transport protocol to test
against (default: `tcp`, also: `uds`).
- **options**: any extra pytest flags the user wants
(e.g. `--ll debug`, `--tpdb`, `-x`, `-v`).
If the user provides a bare path or pattern as argument,
treat it as the test target. Examples:
- `/run-tests` → full suite
- `/run-tests test_local.py` → single file
- `/run-tests test_registrar -v` → file + verbose
- `/run-tests -k cancel` → keyword filter
- `/run-tests tests/ipc/ --tpt-proto uds` → subdir + UDS
## 2. Construct the pytest command
Base command:
```
python -m pytest
```
### Default flags (always include unless user overrides):
- `-x` (stop on first failure)
- `--tb=short` (concise tracebacks)
- `--no-header` (reduce noise)
### Path resolution:
- If the user gives a bare filename like `test_local.py`,
resolve it under `tests/`.
- If the user gives a subdirectory like `ipc/`, resolve
under `tests/ipc/`.
- Glob if needed: `tests/**/test_*<pattern>*.py`
### Key pytest options for this project:
| Flag | Purpose |
|---|---|
| `--ll <level>` | Set tractor log level (e.g. `debug`, `info`, `runtime`) |
| `--tpdb` / `--debug-mode` | Enable tractor's multi-proc debugger |
| `--tpt-proto <key>` | IPC transport: `tcp` (default) or `uds` |
| `--spawn-backend <be>` | Spawn method: `trio` (default), `mp_spawn`, `mp_forkserver` |
| `-k <expr>` | pytest keyword filter |
| `-v` / `-vv` | Verbosity |
| `-s` | No output capture (useful with `--tpdb`) |
### Common combos:
```sh
# quick smoke test of core modules
python -m pytest tests/test_local.py tests/test_rpc.py -x --tb=short --no-header
# full suite, stop on first failure
python -m pytest tests/ -x --tb=short --no-header
# specific test with debug
python -m pytest tests/discovery/test_registrar.py::test_reg_then_unreg -x -s --tpdb --ll debug
# run with UDS transport
python -m pytest tests/ -x --tb=short --no-header --tpt-proto uds
# keyword filter
python -m pytest tests/ -x --tb=short --no-header -k "cancel and not slow"
```
## 3. Pre-flight: venv detection (MANDATORY)
**Always verify a `uv` venv is active before running
`python` or `pytest`.** This project uses
`UV_PROJECT_ENVIRONMENT=py<MINOR>` naming (e.g.
`py313`) — never `.venv`.
### Step 1: detect active venv
Run this check first:
```sh
python -c "
import sys, os
venv = os.environ.get('VIRTUAL_ENV', '')
prefix = sys.prefix
print(f'VIRTUAL_ENV={venv}')
print(f'sys.prefix={prefix}')
print(f'executable={sys.executable}')
"
```
### Step 2: interpret results
**Case A — venv is active** (`VIRTUAL_ENV` is set
and points to a `py<MINOR>/` dir under the project
root or worktree):
Use bare `python` / `python -m pytest` for all
commands. This is the normal, fast path.
**Case B — no venv active** (`VIRTUAL_ENV` is empty
or `sys.prefix` points to a system Python):
Use `AskUserQuestion` to ask the user:
> "No uv venv is active. Should I activate one
> via `UV_PROJECT_ENVIRONMENT=py<MINOR> uv sync`,
> or would you prefer to activate your shell venv
> first?"
Options:
1. **"Create/sync venv"** — run
`UV_PROJECT_ENVIRONMENT=py<MINOR> uv sync` where
`<MINOR>` is detected from `python --version`
(e.g. `313` for 3.13). Then use
`py<MINOR>/bin/python` for all subsequent
commands in this session.
2. **"I'll activate it myself"** — stop and let the
user `source py<MINOR>/bin/activate` or similar.
**Case C — inside a git worktree** (`git rev-parse
--git-common-dir` differs from `--git-dir`):
Verify Python resolves from the **worktree's own
venv**, not the main repo's:
```sh
python -c "import tractor; print(tractor.__file__)"
```
If the path points outside the worktree, create a
worktree-local venv:
```sh
UV_PROJECT_ENVIRONMENT=py<MINOR> uv sync
```
Then use `py<MINOR>/bin/python` for all commands.
**Why this matters**: without the correct venv,
subprocesses spawned by tractor resolve modules
from the wrong editable install, causing spurious
`AttributeError` / `ModuleNotFoundError`.
### Fallback: `uv run`
If the user can't or won't activate a venv, all
`python` and `pytest` commands can be prefixed
with `UV_PROJECT_ENVIRONMENT=py<MINOR> uv run`:
```sh
# instead of: python -m pytest tests/ -x
UV_PROJECT_ENVIRONMENT=py313 uv run pytest tests/ -x
# instead of: python -c 'import tractor'
UV_PROJECT_ENVIRONMENT=py313 uv run python -c 'import tractor'
```
`uv run` auto-discovers the project and venv,
but is slower than a pre-activated venv due to
lock-file resolution on each invocation. Prefer
activating the venv when possible.
### Step 3: import + collection checks
After venv is confirmed, always run these
(especially after refactors or module moves):
```sh
# 1. package import smoke check
python -c 'import tractor; print(tractor)'
# 2. verify all tests collect (no import errors)
python -m pytest tests/ -x -q --co 2>&1 | tail -5
```
If either fails, fix the import error before running
any actual tests.
### Step 4: zombie-actor / stale-registry check (MANDATORY)
The tractor runtime's default registry address is
**`127.0.0.1:1616`** (TCP) / `/tmp/registry@1616.sock`
(UDS). Whenever any prior test run — especially one
using a fork-based backend like `subint_forkserver`
leaks a child actor process, that zombie keeps the
registry port bound and **every subsequent test
session fails to bind**, often presenting as 50+
unrelated failures ("all tests broken"!) across
backends.
**This has to be checked before the first run AND
after any cancelled/SIGINT'd run** — signal failures
in the middle of a test can leave orphan children.
```sh
# 1. TCP registry — any listener on :1616? (primary signal)
ss -tlnp 2>/dev/null | grep ':1616' || echo 'TCP :1616 free'
# 2. leftover actor/forkserver procs — scoped to THIS
# repo's python path, so we don't false-flag legit
# long-running tractor-using apps (e.g. `piker`,
# downstream projects that embed tractor).
pgrep -af "$(pwd)/py[0-9]*/bin/python.*_actor_child_main|subint-forkserv" \
| grep -v 'grep\|pgrep' \
|| echo 'no leaked actor procs from this repo'
# 3. stale UDS registry sockets
ls -la /tmp/registry@*.sock 2>/dev/null \
|| echo 'no leaked UDS registry sockets'
```
**Interpretation:**
- **TCP :1616 free AND no stale sockets** → clean,
proceed. The actor-procs probe is secondary — false
positives are common (piker, any other tractor-
embedding app); only cleanup if `:1616` is bound or
sockets linger.
- **TCP :1616 bound OR stale sockets present**
surface PIDs + cmdlines to the user, offer cleanup:
```sh
# 1. GRACEFUL FIRST (tractor is structured concurrent — it
# catches SIGINT as an OS-cancel in `_trio_main` and
# cascades Portal.cancel_actor via IPC to every descendant.
# So always try SIGINT first with a bounded timeout; only
# escalate to SIGKILL if graceful cleanup doesn't complete).
pkill -INT -f "$(pwd)/py[0-9]*/bin/python.*_actor_child_main|subint-forkserv"
# 2. bounded wait for graceful teardown (usually sub-second).
# Loop until the processes exit, or timeout. Keep the
# bound tight — hung/abrupt-killed descendants usually
# hang forever, so don't wait more than a few seconds.
for i in $(seq 1 10); do
pgrep -f "$(pwd)/py[0-9]*/bin/python.*_actor_child_main|subint-forkserv" >/dev/null || break
sleep 0.3
done
# 3. ESCALATE TO SIGKILL only if graceful didn't finish.
if pgrep -f "$(pwd)/py[0-9]*/bin/python.*_actor_child_main|subint-forkserv" >/dev/null; then
echo 'graceful teardown timed out — escalating to SIGKILL'
pkill -9 -f "$(pwd)/py[0-9]*/bin/python.*_actor_child_main|subint-forkserv"
fi
# 4. if a test zombie holds :1616 specifically and doesn't
# match the above pattern, find its PID the hard way:
ss -tlnp 2>/dev/null | grep ':1616' # prints `users:(("<name>",pid=NNNN,...))`
# then (same SIGINT-first ladder):
# kill -INT <NNNN>; sleep 1; kill -9 <NNNN> 2>/dev/null
# 5. remove stale UDS sockets
rm -f /tmp/registry@*.sock
# 6. re-verify
ss -tlnp 2>/dev/null | grep ':1616' || echo 'TCP :1616 now free'
```
**Never ignore stale registry state.** If you see the
"all tests failing" pattern — especially
`trio.TooSlowError` / connection refused / address in
use on many unrelated tests — check registry **before**
spelunking into test code. The failure signature will
be identical across backends because they're all
fighting for the same port.
**False-positive warning for step 2:** a plain
`pgrep -af '_actor_child_main'` will also match
legit long-running tractor-embedding apps (e.g.
`piker` at `~/repos/piker/py*/bin/python3 -m
tractor._child ...`). Always scope to the current
repo's python path, or only use step 1 (`:1616`) as
the authoritative signal.
## 4. Run and report
- Run the constructed command.
- Use a timeout of **600000ms** (10min) for full suite
runs, **120000ms** (2min) for single-file runs.
- If the suite is large (full `tests/`), consider running
in the background and checking output when done.
- Use `--lf` (last-failed) to re-run only previously
failing tests when iterating on a fix.
### On failure:
- Show the failing test name(s) and short traceback.
- If the failure looks related to recent changes, point
out the likely cause and suggest a fix.
- **Check the known-flaky list** (section 8) before
investigating — don't waste time on pre-existing
timeout issues.
- **NEVER auto-commit fixes.** If you apply a code fix
during test iteration, leave it unstaged. Tell the
user what changed and suggest they review the
worktree state, stage files manually, and use
`/commit-msg` (inline or in a separate session) to
generate the commit message. The human drives all
`git add` and `git commit` operations.
### On success:
- Report the pass/fail/skip counts concisely.
## 5. Test directory layout (reference)
```
tests/
├── conftest.py # root fixtures, daemon, signals
├── devx/ # debugger/tooling tests
├── ipc/ # transport protocol tests
├── msg/ # messaging layer tests
├── discovery/ # discovery subsystem tests
│ ├── test_multiaddr.py # multiaddr construction
│ └── test_registrar.py # registry/discovery protocol
├── test_local.py # registrar + local actor basics
├── test_rpc.py # RPC error handling
├── test_spawning.py # subprocess spawning
├── test_multi_program.py # multi-process tree tests
├── test_cancellation.py # cancellation semantics
├── test_context_stream_semantics.py # ctx streaming
├── test_inter_peer_cancellation.py # peer cancel
├── test_infected_asyncio.py # trio-in-asyncio
└── ...
```
## 6. Change-type → test mapping
After modifying specific modules, run the corresponding
test subset first for fast feedback:
| Changed module(s) | Run these tests first |
|---|---|
| `runtime/_runtime.py`, `runtime/_state.py` | `test_local.py test_rpc.py test_spawning.py test_root_runtime.py` |
| `discovery/` (`_registry`, `_discovery`, `_addr`) | `tests/discovery/ test_multi_program.py test_local.py` |
| `_context.py`, `_streaming.py` | `test_context_stream_semantics.py test_advanced_streaming.py` |
| `ipc/` (`_chan`, `_server`, `_transport`) | `tests/ipc/ test_2way.py` |
| `runtime/_portal.py`, `runtime/_rpc.py` | `test_rpc.py test_cancellation.py` |
| `spawn/` (`_spawn`, `_entry`) | `test_spawning.py test_multi_program.py` |
| `devx/debug/` | `tests/devx/test_debugger.py` (slow!) |
| `to_asyncio.py` | `test_infected_asyncio.py test_root_infect_asyncio.py` |
| `msg/` | `tests/msg/` |
| `_exceptions.py` | `test_remote_exc_relay.py test_inter_peer_cancellation.py` |
| `runtime/_supervise.py` | `test_cancellation.py test_spawning.py` |
## 7. Quick-check shortcuts
### After refactors (fastest first-pass):
```sh
# import + collect check
python -c 'import tractor' && python -m pytest tests/ -x -q --co 2>&1 | tail -3
# core subset (~10s)
python -m pytest tests/test_local.py tests/test_rpc.py tests/test_spawning.py tests/discovery/test_registrar.py -x --tb=short --no-header
```
### Inspect last failures (without re-running):
When the user asks "what failed?", "show failures",
or wants to check the last-failed set before
re-running — read the pytest cache directly. This
is instant and avoids test collection overhead.
```sh
python -c "
import json, pathlib, sys
p = pathlib.Path('.pytest_cache/v/cache/lastfailed')
if not p.exists():
print('No lastfailed cache found.'); sys.exit()
data = json.loads(p.read_text())
# filter to real test node IDs (ignore junk
# entries that can accumulate from system paths)
tests = sorted(k for k in data if k.startswith('tests/'))
if not tests:
print('No failures recorded.')
else:
print(f'{len(tests)} last-failed test(s):')
for t in tests:
print(f' {t}')
"
```
**Why not `--cache-show` or `--co --lf`?**
- `pytest --cache-show 'cache/lastfailed'` works
but dumps raw dict repr including junk entries
(stale system paths that leak into the cache).
- `pytest --co --lf` actually *collects* tests which
triggers import resolution and is slow (~0.5s+).
Worse, when cached node IDs don't exactly match
current parametrize IDs (e.g. param names changed
between runs), pytest falls back to collecting
the *entire file*, giving false positives.
- Reading the JSON directly is instant, filterable
to `tests/`-prefixed entries, and shows exactly
what pytest recorded — no interpretation.
**After inspecting**, re-run the failures:
```sh
python -m pytest --lf -x --tb=short --no-header
```
### Full suite in background:
When core tests pass and you want full coverage while
continuing other work, run in background:
```sh
python -m pytest tests/ -x --tb=short --no-header -q
```
(use `run_in_background=true` on the Bash tool)
## 8. Known flaky tests
These tests have **pre-existing** timing/environment
sensitivity. If they fail with `TooSlowError` or
pexpect `TIMEOUT`, they are almost certainly NOT caused
by your changes — note them and move on.
| Test | Typical error | Notes |
|---|---|---|
| `devx/test_debugger.py::test_multi_nested_subactors_error_through_nurseries` | pexpect TIMEOUT | Debugger pexpect timing |
| `test_cancellation.py::test_cancel_via_SIGINT_other_task` | TooSlowError | Signal handling race |
| `test_inter_peer_cancellation.py::test_peer_spawns_and_cancels_service_subactor` | TooSlowError | Async timing (both param variants) |
| `test_docs_examples.py::test_example[we_are_processes.py]` | `assert None == 0` | `__main__` missing `__file__` in subproc |
**Rule of thumb**: if a test fails with `TooSlowError`,
`trio.TooSlowError`, or `pexpect.TIMEOUT` and you didn't
touch the relevant code path, it's flaky — skip it.
## 9. The pytest-capture hang pattern (CHECK THIS FIRST)
**Symptom:** a tractor test hangs indefinitely under
default `pytest` but passes instantly when you add
`-s` (`--capture=no`).
**Cause:** tractor subactors (especially under fork-
based backends) inherit pytest's stdout/stderr
capture pipes via fds 1,2. Under high-volume error
logging (e.g. multi-level cancel cascade, nested
`run_in_actor` failures, anything triggering
`RemoteActorError` + `ExceptionGroup` traceback
spew), the **64KB Linux pipe buffer fills** faster
than pytest drains it. Subactor writes block → can't
finish exit → parent's `waitpid`/pidfd wait blocks →
deadlock cascades up the tree.
**Pre-existing guards in the tractor harness** that
encode this same knowledge — grep these FIRST
before spelunking:
- `tests/conftest.py:258-260` (in the `daemon`
fixture): `# XXX: too much logging will lock up
the subproc (smh)` — downgrades `trace`/`debug`
loglevel to `info` to prevent the hang.
- `tests/conftest.py:316`: `# can lock up on the
_io.BufferedReader and hang..` — noted on the
`proc.stderr.read()` post-SIGINT.
**Debug recipe (in priority order):**
1. **Try `-s` first.** If the hang disappears with
`pytest -s`, you've confirmed it's capture-pipe
fill. Skip spelunking.
2. **Lower the loglevel.** Default `--ll=error` on
this project; if you've bumped it to `debug` /
`info`, try dropping back. Each log level
multiplies pipe-pressure under fault cascades.
3. **If you MUST use default capture + high log
volume**, redirect subactor stdout/stderr in the
child prelude (e.g.
`tractor.spawn._subint_forkserver._child_target`
post-`_close_inherited_fds`) to `/dev/null` or a
file.
**Signature tells you it's THIS bug (vs. a real
code hang):**
- Multi-actor test under fork-based backend
(`subint_forkserver`, eventually `trio_proc` too
under enough log volume).
- Multiple `RemoteActorError` / `ExceptionGroup`
tracebacks in the error path.
- Test passes with `-s` in the 5-10s range, hangs
past pytest-timeout (usually 30+ s) without `-s`.
- Subactor processes visible via `pgrep -af
subint-forkserv` or similar after the hang —
they're alive but blocked on `write()` to an
inherited stdout fd.
**Historical reference:** this deadlock cost a
multi-session investigation (4 genuine cascade
fixes landed along the way) that only surfaced the
capture-pipe issue AFTER the deeper fixes let the
tree actually tear down enough to produce pipe-
filling log volume. Full post-mortem in
`ai/conc-anal/subint_forkserver_test_cancellation_leak_issue.md`.
Lesson codified here so future-me grep-finds the
workaround before digging.
## 10. Reaping zombie subactors (`tractor-reap`)
**Symptom:** after a `pytest` run crashes, times out,
or is `Ctrl+C`'d, subactor forks (esp. under
`subint_forkserver`) can be reparented to `init`
(PPid==1) and linger. They hold onto ports, inherit
pytest's capture-pipe fds, and flakify later
sessions.
**Two layers of defense:**
### a) Session-scoped auto-fixture (always on)
`tractor/_testing/pytest.py::_reap_orphaned_subactors`
runs at pytest session teardown. It walks `/proc` for
direct descendants of the pytest pid, SIGINTs them,
waits up to 3s, then SIGKILLs survivors. SC-polite:
gives the subactor runtime a chance to run its trio
cancel shield + IPC teardown before escalation.
This is *autouse* and session-scoped — you don't need
to do anything. It just runs.
### b) `scripts/tractor-reap` CLI (manual reap)
For the **pytest-died-mid-session** case (Ctrl+C, OOM
kill, hung process you had to `kill -9`), the fixture
never ran. Reach for the CLI:
```sh
# default: orphans (PPid==1, cwd==repo, cmd contains python)
scripts/tractor-reap
# descendant-mode: from a still-live supervisor
scripts/tractor-reap --parent <pytest-pid>
# see what would be reaped, don't signal
scripts/tractor-reap -n
# tune the SIGINT → SIGKILL grace window
scripts/tractor-reap --grace 5
```
Exit code: `0` if everyone exited on SIGINT, `1` if
SIGKILL had to escalate — so you can chain it in CI
health-checks (`scripts/tractor-reap || <alert>`).
**What it matches** (orphan-mode):
- `PPid == 1` (reparented to init → definitely
orphaned, not just a currently-running child)
- `cwd == <repo-root>` (keeps the sweep scoped; won't
touch unrelated init-children elsewhere)
- `python` in cmdline
**What it does not do:** kill anything whose PPid is
still a live tractor parent. If the parent is alive
it's not an orphan; use `--parent <pid>` if you need
to force-reap under a still-live supervisor.
**When NOT to run it:** while a pytest session is
active in another terminal. It's safe (won't touch
that session's live children in orphan-mode) but can
race if the target session is mid-teardown.
### c) `--shm` / `--shm-only`: orphan-segment sweep
Because `tractor.ipc._mp_bs.disable_mantracker()`
turns off `mp.resource_tracker` (see
`ai/conc-anal/subint_forkserver_mp_shared_memory_issue.md`),
a hard-crashing actor can leave `/dev/shm/<key>`
segments behind that nothing else GCs.
```sh
# process reap THEN shm sweep
scripts/tractor-reap --shm
# shm sweep only (skip process phase)
scripts/tractor-reap --shm-only
# dry-run: list candidates, don't unlink
scripts/tractor-reap --shm -n
```
**Match criteria** (very conservative — this is a
shared-system path, can't be wrong):
- segment is a regular file under `/dev/shm`,
- owned by the **current uid** (`stat.st_uid`),
- AND **no live process holds it open**
enumerated by walking every readable
`/proc/<pid>/maps` (post-mmap mappings) AND
`/proc/<pid>/fd/*` (pre-mmap shm-opened fds).
The "nobody has it open" check is the
kernel-canonical "is this leaked?" test — same
answer `lsof /dev/shm/<key>` would give. No
reliance on tractor-specific naming, so it works
for any tractor app. Critically, it WILL NOT touch
segments held by other apps you have running
(e.g. `piker`, `lttng-ust-*`, `aja-shm-*`
verified locally with 81 in-use segments correctly
preserved).

View File

@ -1,10 +1,18 @@
name: CI
# NOTE distilled from,
# https://github.com/orgs/community/discussions/26276
on:
# any time someone pushes a new branch to origin
# any time a new update to 'main'
push:
branches:
- main
# Allows you to run this workflow manually from the Actions tab
# for on all (forked) PRs to repo
# NOTE, use a draft PR if you just want CI triggered..
pull_request:
# to run workflow manually from the "Actions" tab
workflow_dispatch:
jobs:
@ -29,7 +37,12 @@ jobs:
run: uv build --sdist --python=3.13
- name: Install sdist from .tar.gz
run: python -m pip install dist/*.tar.gz
# XXX must install under py3.13 (matching the build's
# `--python=3.13`); the runner's default `python` is 3.12
# which our `requires-python = ">=3.13"` now rejects.
run: |
uv venv --python 3.13
uv pip install dist/*.tar.gz
# ------ type-check ------
# mypy:
@ -74,24 +87,81 @@ jobs:
# run: mypy tractor/ --ignore-missing-imports --show-traceback
testing-linux:
name: '${{ matrix.os }} Python ${{ matrix.python }} - ${{ matrix.spawn_backend }}'
timeout-minutes: 10
testing:
name: '${{ matrix.os }} Python${{ matrix.python-version }} spawn_backend=${{ matrix.spawn_backend }} tpt_proto=${{ matrix.tpt_proto }} capture=${{ matrix.capture }}'
timeout-minutes: 20
runs-on: ${{ matrix.os }}
# NOTE on the matrix shape — the `capture=` mode follows
# `spawn_backend`:
#
# - `trio` / `mp_*` backends use `--capture=fd` (default)
# for per-test attribution of subactor *raw-fd* output
# in failure reports.
# - Fork-based backends (`main_thread_forkserver`,
# `subint_forkserver`) REQUIRE `--capture=sys` because
# fork-child × `--capture=fd` is a known deadlock
# pattern. See the long NOTE in `tractor._testing.pytest`'s
# `pytest_load_initial_conftests` for the mechanism +
# tradeoff write-up.
#
# If a future matrix row adds a fork-spawn backend
# WITHOUT setting `capture: 'sys'`, the
# `pytest_load_initial_conftests` hook fail-fasts on `CI=1`
# with a clear error msg. So the matrix is self-policing.
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest]
python-version: ['3.13']
os: [
ubuntu-latest,
macos-latest,
]
python-version: [
'3.13',
# '3.14',
]
spawn_backend: [
'trio',
# 'mp_spawn',
# 'mp_forkserver',
# ?TODO^ is it worth it to get these running again?
#
# - [ ] next-gen backends, on 3.13+
# https://github.com/goodboy/tractor/issues/379
# 'subinterpreter',
# 'subint',
]
tpt_proto: [
'tcp',
'uds',
]
capture: [
'fd', # default for non-fork backends
]
steps:
# Fork-based backends — added via `include:` so each
# cell carries its REQUIRED `capture: 'sys'` mode.
# Linux-only for now; macOS coverage TBD pending
# local validation.
include:
- os: ubuntu-latest
python-version: '3.13'
spawn_backend: 'main_thread_forkserver'
tpt_proto: 'tcp'
capture: 'sys'
- os: ubuntu-latest
python-version: '3.13'
spawn_backend: 'main_thread_forkserver'
tpt_proto: 'uds'
capture: 'sys'
# https://github.com/orgs/community/discussions/26253#discussioncomment-3250989
exclude:
# don't do UDS run on macOS (for now)
- os: macos-latest
tpt_proto: 'uds'
steps:
- uses: actions/checkout@v4
- name: 'Install uv + py-${{ matrix.python-version }}'
@ -118,7 +188,18 @@ jobs:
run: uv tree
- name: Run tests
run: uv run pytest tests/ --spawn-backend=${{ matrix.spawn_backend }} -rsx
run: >
uv run
pytest
tests/
-rsx
--spawn-backend=${{ matrix.spawn_backend }}
--tpt-proto=${{ matrix.tpt_proto }}
--capture=${{ matrix.capture }}
# NOTE: capture mode is matrix-driven — `fd` for
# non-fork backends (per-test fd attribution),
# `sys` for fork-based (avoids fork-child x
# capture-fd deadlock). See matrix-NOTE above.
# XXX legacy NOTE XXX
#

63
.github/workflows/docs.yml vendored 100644
View File

@ -0,0 +1,63 @@
name: docs
# build sphinx docs on every PR + push to main;
# deploy to gh-pages only from main pushes.
# (see goodboy/tractor#123 for the original ask)
on:
push:
branches:
- main
pull_request:
# to run workflow manually from the "Actions" tab
workflow_dispatch:
# needed by actions/deploy-pages
permissions:
contents: read
pages: write
id-token: write
jobs:
build:
name: 'sphinx build'
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Install latest uv
uses: astral-sh/setup-uv@v6
# NOTE, no `d2` bin is installed in CI (yet) so
# the pre-rendered + committed SVGs under
# `docs/_diagrams/` are used as-is; see
# `docs/_ext/d2diagrams.py` for the fallback
# policy.
- name: Build html docs
run: |
uv sync --no-dev --group docs
uv run --no-dev --group docs make -C docs html
- name: Upload pages artifact
uses: actions/upload-pages-artifact@v3
with:
path: docs/_build/html
deploy:
name: 'deploy to gh-pages'
if: ${{ github.ref == 'refs/heads/main' && github.event_name == 'push' }}
needs: build
# serialize deploys but NEVER cancel an in-flight one
# mid-upload; queue the next instead (scoped to this job so
# PR builds can't cancel a production deploy).
concurrency:
group: 'pages'
cancel-in-progress: false
runs-on: ubuntu-latest
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
steps:
- name: Deploy
id: deployment
uses: actions/deploy-pages@v4

66
.gitignore vendored
View File

@ -102,3 +102,69 @@ venv.bak/
# mypy
.mypy_cache/
# all files under
.git/
# require very explicit staging for anything we **really**
# want put/kept in repo.
notes_to_self/
snippets/
# ------- AI shiz -------
# `ai.skillz` symlinks,
# (machine-local, deploy via deploy-skill.sh)
.claude/skills/py-codestyle
.claude/skills/close-wkt
.claude/skills/plan-io
.claude/skills/prompt-io
.claude/skills/resolve-conflicts
.claude/skills/inter-skill-review
# /open-wkt specifics
.claude/skills/open-wkt
.claude/wkts/
claude_wkts
# /code-review-changes specifics
.claude/skills/code-review-changes
# review-skill ephemeral ctx (per-PR, single-use)
.claude/review_context.md
.claude/review_regression.md
# /pr-msg specifics
.claude/skills/pr-msg/*
# repo-specific
!.claude/skills/pr-msg/format-reference.md
# XXX, so u can nvim-telescope this file.
# !.claude/skills/pr-msg/pr_msg_LATEST.md
# /commit-msg specifics
# - any commit-msg gen tmp files
.claude/*_commit_*.md
.claude/*_commit*.txt
.claude/skills/commit-msg/*
!.claude/skills/commit-msg/style-duie-reference.md
# use prompt-io instead?
.claude/plans
# nix develop --profile .nixdev
.nixdev*
# :Obsession .
Session.vim
# `gish` local `.md`-files
# TODO? better all around automation!
# -[ ] it'd be handy to also commit and sync with wtv git service?
# -[ ] everything should be put under a `.gish/` no?
gitea/
gh/
# ------ macOS ------
# Finder metadata
**/.DS_Store
# LLM conversations that should remain private
docs/conversations/

View File

@ -78,6 +78,7 @@ Bug Fixes
for ``asyncio``-side errors to not propagate due to a race condition.
The implementation fix summary is:
- add state to signal the end of the ``trio`` side task to be
read by the ``asyncio`` side and always cancel any ongoing
task in such cases.

View File

@ -0,0 +1,314 @@
# Cancel-cascade `trio.TooSlowError` flakes under `main_thread_forkserver`
## Symptom
Running the full test suite under
```bash
./py313/bin/python -m pytest tests/ \
--tpt-proto=tcp \
--spawn-backend=main_thread_forkserver
```
surfaces a single, **rotating** `trio.TooSlowError`
failure each run. The failure isn't deterministic on
test identity — different test each run — but it
ALWAYS looks like:
```
FAILED tests/<file>::test_<name> - trio.TooSlowError
==== 1 failed, 373 passed, 17 skipped, 1112 xfailed,
01 xpassed, ~550 warnings in ~6min ====
```
Pass rate: **~99.7%** (373 of 374 non-skip tests).
Wall-clock per full run: 56 min.
## Tests observed flaking so far
Each row was the SOLE failure in a separate run:
| run # | test |
|---|---|
| 1 | `tests/test_advanced_streaming.py::test_dynamic_pub_sub[KeyboardInterrupt]` |
| 2 | `tests/test_infected_asyncio.py::test_context_spawns_aio_task_that_errors[parent_actor_cancels_child=False]` |
Both share the same shape:
- **Cancel cascade** of N subactors back to a parent root actor.
- N ≥ `multiprocessing.cpu_count()` for `test_dynamic_pub_sub`
(it spawns `cpus - 1` consumers + publisher + dynamic-consumer).
- N ≈ 2 for `test_context_spawns_aio_task_that_errors`
but each subactor is `infect_asyncio=True`, so each
cancel involves the trio↔asyncio guest-run unwind
which is structurally heavier than pure-trio.
- Test wraps the cascade in `trio.fail_after(N seconds)`
and the cap fires before the cascade completes.
The exact failing test rotates because each test is
independently close to the cap; whichever happens to
be unlucky in scheduling/CPU-contention on a given run
is the one that times out.
## Root-cause family
`hard_kill` (`tractor/spawn/_spawn.py:hard_kill`) runs
the SC-graceful teardown ladder per subactor:
1. `Portal.cancel_actor()` — graceful IPC cancel-req.
2. Wait `terminate_after=1.6s` for sub to exit.
3. If still alive: `proc.kill()` (SIGKILL).
4. (NEW) `_unlink_uds_bind_addrs()` — post-mortem
sock-file cleanup for UDS leaks (issue #452 fix).
For a cascade of N subactors, each pays steps 14. If
graceful-cancel doesn't complete within 1.6s for ANY
sub, that sub eats a full 1.6s of `move_on_after` plus
the `proc.wait()` post-SIGKILL.
Worst case under fork backend with N=cpus subs:
- N × 1.6s = 16s+ on a 10-core box just for the
graceful timeout phase
- Plus per-spawn fork-IPC handshake cost compounds
during teardown (each sub's IPC cleanup goes through
the same forkserver coordinator)
- Plus the new autouse fixtures
(`_track_orphaned_uds_per_test`,
`_detect_runaway_subactors_per_test`,
`_reap_orphaned_subactors`) all run at test
teardown, adding small (10s of ms) but cumulative
overhead
Current cap: 30s (`fail_after_s = 30 if
is_forking_spawner else 12`). Empirically fits the
median run but the tail breaks ~0.3% of the time.
## NOT regressing
To confirm this is a flake and not a regression:
- Pre-`WakeupSocketpair`-patch baseline: tests
HUNG INDEFINITELY (busy-loop never released).
- Post-patch: pass-or-fail-fast, ~99.7% pass, the
occasional cap-hit fails in bounded time (<60s for
the offending test).
- Same test PASSES under `--spawn-backend=trio`
(no fork, no hard-kill compounding).
So the suite is dramatically better than before; the
remaining flake is a known-tolerable steady-state.
## Possible mitigations (ranked)
### A. Bump the cap further
Cheapest. Change the per-test `fail_after_s` from 30
to e.g. 60 for fork backends. Pros: trivial. Cons:
masks any genuine slowness regression we'd want to
catch.
### B. CPU-count-aware cap
For tests whose N scales with `cpu_count()`, scale
the cap too:
```python
fail_after_s = (
max(30, cpu_count() * 3) # 3s/actor floor
if is_forking_spawner
else 12
)
```
Pros: scales with the actual cancel-cascade work.
Cons: still arbitrary multiplier.
### C. `pytest-rerunfailures` for these tests only
Mark the known-flaky tests with
`@pytest.mark.flaky(reruns=1)` (needs
`pytest-rerunfailures` dep). Single retry hides
genuine ~0.3% transient flakes.
Pros: no cap change, surfaces persistent failures
loudly. Cons: adds a dep, retries can mask real bugs
if used widely.
### D. Reduce `hard_kill`'s `terminate_after`
Drop from 1.6s → 0.8s. Cuts the worst-case cascade
time roughly in half. Risks: fewer subs get a chance
to run their cleanup before SIGKILL → more orphaned
state for the autouse reapers to handle (ironically,
adds back overhead elsewhere).
### E. Profile + targeted fix
Add `log.devx()` markers in `hard_kill` to time each
phase. Identify if any subactor is consistently
hitting the 1.6s cap (vs. exiting in <0.1s). If so,
that sub has a teardown bug worth fixing at source.
Pros: actually fixes the underlying slowness. Cons:
real investigation work, deferred from this round.
## Recommendation
Land this issue-doc as the tracker. Apply **(B)** as
a small follow-up — cheap and proportional. If it
still flakes, escalate to **(E)** with a `log.devx()`
profile-pass.
`(C)` is a backstop if `(B)` doesn't quite get there
and we need green CI faster than (E) can deliver.
## Verification protocol
After applying any mitigation:
```bash
# Run the suite N times back-to-back, count failures.
# A persistent failure on the SAME test == real bug.
# Failures rotating across tests == still cap-related.
for i in $(seq 1 5); do
./py313/bin/python -m pytest tests/ \
--tpt-proto=tcp \
--spawn-backend=main_thread_forkserver \
-q 2>&1 | tail -2
done
```
Target: 0 failures across 5 runs ⇒ ship. 12 failures
still rotating ⇒ apply (C). Same test failing twice
⇒ escalate to (E).
## Snapshot evidence (2026-05-13)
After landing the `fail_after_w_trace` /
`afk_alarm_w_trace` capture-on-timeout helpers
(`tractor._testing.trace`), `test_nested_multierrors`
on the `main_thread_forkserver` backend produces
**reproducible diag snapshots** at
`$XDG_CACHE_HOME/tractor/hung-dumps/test_nested_multierrors_start_method_main_thread_forkserver__<iso-ts>/`.
### Reproduction
```bash
pytest \
-v --verbose --durations=10 \
--spawn-backend=main_thread_forkserver \
--tpt-proto=uds \
--capture=sys --show-capture=stderr -rxX \
tests/test_cancellation.py::test_nested_multierrors
```
The test is `xfail(strict=False)` for MTF — it RUNS
each invocation so snapshots accumulate, but doesn't
break `--lf` workflow.
### Consistent shape across runs
5+ snapshots taken back-to-back show the SAME pattern:
- **Timing:** ~10s wall-clock total. Inner
`fail_after_w_trace(10)` fires at exactly T=10s;
cascade's `nursery.__aexit__` takes ~0.6s more to
gather + propagate the resulting
`BaseExceptionGroup`. **Trio backend completes the
SAME test in <6s** so the MTF cascade is ~2x
slower at minimum.
- **`BaseExceptionGroup` shape:** mixed
`[RemoteActorError, Cancelled]`. The first
subactor's natural error-propagation (`assert 0`
raised → `RemoteActorError` portal-result)
arrives before T=10s; the OTHER subactor's
portal-wait is still in flight at T=10s, gets
cancelled by `fail_after_w_trace`'s scope-cancel
→ returns `Cancelled` instead.
- **Orphan-spawn skew:** snapshot's `orphans` bucket
(after the `_is_tractor_subactor` cgroup-slice
override fix) consistently shows 2-4 init-adopted
procs at `depth_3` and `depth_1` levels — these
are the leaves whose parent (`depth_2` spawner)
was killed mid-cascade but who hadn't yet seen
the cancel signal themselves.
- **UDS sock-leak:** 2-6 dead-orphan socks per run
(varies with cascade timing). The
`track_orphaned_uds_per_test` fixture reaps them
post-test → contamination is isolated per-invocation.
### Capture mechanism
`fail_after_w_trace` covers two firing paths:
1. **`trio.TooSlowError`** raised at scope-exit
(body returned cleanly past deadline) — direct
`except` handler captures.
2. **Scope-cancel + body raises non-`Cancelled` exc**
(e.g. `nursery.__aexit__` wraps timeout-induced
`Cancelled` into a `BaseExceptionGroup` that
escapes before `trio.fail_after`'s exit-check
could fire `TooSlowError`) — body-raise `except`
handler checks `scope.cancel_called` and
captures if True. This path catches the
`test_nested_multierrors` shape specifically (see
"BaseExceptionGroup shape" above).
The snapshot dir contains:
- `trace.txt``ptree` + `hung_state` (kernel
`wchan`/`stack` + `py-spy dump --locals` when
sudo cached), with `include_strays=True`
surfacing any cross-test ghost subactor trees in
the `orphans` bucket.
- `bindspace.txt` — UDS bindspace classification
(live-active / orphaned-alive / orphaned-dead).
- `meta.json``{pid, label, captured_at, sudo_cached}`.
The end-of-session `pytest_terminal_summary` hook
in `tractor._testing.pytest` lists every snapshot
dir from the run so you don't have to scroll back
through captured-stderr lines:
```
========================= tractor hang-snapshot index ==========================
N `fail_after_w_trace` / `afk_alarm_w_trace` snapshot(s) captured this session:
<test-id>
→ /home/.../.cache/tractor/hung-dumps/<label>__<ts>
```
### Caveats
The snapshot fires AFTER the body-raise (not at the
exact moment of scope-cancel), so the parent's
py-spy frames show `_do_capture_snapshot` itself
running, NOT the cancel-cascade hang frame. To see
the actual hang state, manual `acli.ptree` /
`acli.hung_dump` from a second terminal at T=10s
would be needed — **not currently possible**
because per-test reaper fixtures clean up ~0.6s
post-timeout. See follow-up TODO in
`tractor/_testing/trace.py` for a
`TRACTOR_TRACE_HOLD=1` env-var pause mode.
## See also
- [#452](https://github.com/goodboy/tractor/issues/452) —
UDS sock-file leak (related — `hard_kill`'s
cleanup phase contributes to cascade time)
- `ai/conc-anal/trio_wakeup_socketpair_busy_loop_under_fork_issue.md`
— the upstream-trio fix that turned this from a
100% hang into a 0.3% flake
- `ai/conc-anal/infected_asyncio_under_main_thread_forkserver_hang_issue.md`
— the asyncio variant which contributes to one of
the rotating failures
- `tractor/spawn/_spawn.py::hard_kill` — the SIGKILL
cascade source
- `tractor/_testing/_reap.py::_track_orphaned_uds_per_test`,
`_detect_runaway_subactors_per_test`,
`_reap_orphaned_subactors` — autouse cleanup
fixtures whose cumulative teardown overhead
contributes to the cascade time

View File

@ -0,0 +1,281 @@
# `fork()` in a multi-threaded program — execution-side vs. memory-side of the same coin
A reference doc for readers who've encountered one of two
opposite-sounding framings of POSIX `fork()` semantics in a
multi-threaded program and are confused by the other.
This is a sibling to
`subint_fork_blocked_by_cpython_post_fork_issue.md` — that
doc covers a CPython-level refusal of fork-from-subint;
this one covers the more general POSIX layer, since
tractor's main-thread forkserver design rests on it.
## TL;DR
POSIX `fork()` only preserves the *calling* thread as a
runnable thread in the child — every other thread in the
parent simply never executes another instruction in the
child. trio's docs call this "leaked"; tractor's
`_main_thread_forkserver.py` docstring calls it "gone".
Both are correct: "gone" is the *execution* side (no
scheduler entry, no instructions retired), "leaked" is the
*memory* side (the dead threads' stacks and per-thread
heap structures still ride into the child's address space
as orphaned COW pages with no owner and no cleanup hook).
Same POSIX reality, two halves of the same coin.
## The two framings
[python-trio/trio#1614][trio-1614] (the canonical "trio +
fork" hazards thread) puts it this way:
> If you use `fork()` in a process with multiple threads,
> all the other thread stacks are just leaked: there's
> nothing else you can reasonably do with them.
`tractor.spawn._main_thread_forkserver`'s module docstring
(specifically the "What survives the fork? — POSIX
semantics" section) puts it this way:
> POSIX `fork()` only preserves the *calling* thread as a
> runnable thread in the child. Every other thread in the
> parent — trio's runner thread, any `to_thread` cache
> threads, anything else — never executes another
> instruction post-fork.
A reader bouncing between the two can be forgiven for
asking: well, *which* is it — leaked or gone?
The answer is "yes". They're describing the same POSIX
behavior from two different angles:
- trio is talking about the **bytes** the dead threads
leave behind — stacks, TLS slots, per-thread arena
metadata — and the fact that nothing in the child can
drive them forward, free them, or even safely walk
them. That's a memory leak in the strict sense: held
but unreachable.
- tractor is talking about the **execution** side
relevant to the forkserver design: which threads
retire instructions in the child? Exactly one — the
one that called `fork()`. Everything else, regardless
of the bytes left behind, is dead in a scheduler
sense.
Neither framing is wrong; they're just answering
different questions.
## POSIX `fork()` in a multi-threaded program — what actually happens
Per POSIX (and concretely on Linux glibc), the contract
of `fork()` in a multi-threaded process is:
1. The kernel creates a new process whose virtual
address space is a COW copy of the parent's. *All*
pages map across — code, heap, every thread's stack,
every malloc arena, every mmap region.
2. Of the parent's N threads, exactly **one** is
reified in the child as a runnable kernel task: the
thread that called `fork()`. The other N-1 threads
have *no* corresponding task in the child kernel. They
were never scheduled, never `clone()`d for the child,
never exist as runnable entities.
3. Their **memory artifacts** — pthread stacks, TLS,
`pthread_t` structures, glibc per-thread arena
bookkeeping — are still mapped in the child's address
space, because (1) duplicates *everything* page-wise.
They sit there as inert COW bytes.
4. The kernel does not clean those bytes up. There is no
"phantom-thread cleanup" pass post-fork. The kernel
doesn't know which mapped pages "belonged to" which
thread — at the kernel level mappings are
process-scoped, not thread-scoped.
5. The surviving thread (the caller of `fork()`) cannot
safely access those leaked bytes either. Any state
they encoded — held mutexes, in-flight syscalls,
half-updated invariants — is frozen at whatever
instant the parent's fork-syscall observed it. Some
of those mutexes may even still be locked from the
child's POV (the canonical "fork-in-multithreaded-
program-deadlocks" hazard; see `man pthread_atfork`).
So: from the kernel's PoV, the child has one thread.
From the address-space's PoV, the child has all the
parent's bytes — including the corpses of the N-1 dead
threads' stacks. Both true simultaneously.
## Why trio says "leaked"
trio's framing makes sense from the parent's
PoV, looking at *what those threads were doing*. In a
running `trio.run()` process you typically have:
- The trio runner thread itself — owns the `selectors`
epoll fd, the signal-wakeup-fd, the run-queue.
- Threadpool worker threads (`trio.to_thread`'s cache)
— blocked in `wait()` on the threadpool's work
condvar.
- Whatever other ad-hoc threads the application
started.
Each of those threads owns *real work-state*: epoll
registrations, file descriptors held in
soon-to-be-completed reads, half-released locks, posted
but unconsumed wakeups. After fork, that state is still
encoded in the child's memory. None of it is invalid in
a well-formed-bytes sense. It's just that:
- The thread that was driving it is gone.
- Nothing else in the child knows the layout well
enough to take over.
- Even if it did, the kernel objects backing the work
(epoll fd, signalfd) have separate post-fork
semantics that don't compose with userland trio
state.
So the bytes are *held* (they're in the child's
address space, they count against RSS, they survive
until something clobbers them), and they're
*unreachable* in any meaningful sense — no thread can
safely drive them forward. That is the textbook
definition of a leak.
trio's quote is reminding the user that `fork()` from a
multi-threaded process is a one-way memory hazard:
whatever those threads were doing, that work-state is
now garbage you happen to still be carrying.
## Why tractor says "gone"
tractor's `_main_thread_forkserver` framing is concerned
with a different question: *which thread executes in the
child, and is it safe?*
The forkserver design rests on POSIX's "calling thread
is the sole survivor" guarantee. We pick that calling
thread very deliberately: a dedicated worker that has
provably never entered trio. So the thread that *does*
run in the child is one whose locals, TLS, and stack
contain nothing trio-related. Trio's runner thread —
the one that owned the epoll fd and the run-queue — is
*gone* from the child in the execution sense. It will
never run another instruction. The fact that its stack
bytes still exist in the child's address space (the
"leaked" view) is irrelevant to the forkserver, because
nothing in the child reads or writes those pages.
So when the docstring says "Every other thread … is
gone the instant `fork()` returns in the child", it's
being precise about the surface that matters for the
backend: scheduler-level liveness. Nothing schedules
those threads ever again. Whether their bytes are
hanging around is a separate (and, for the design,
non-load-bearing) fact.
## Cross-table
The same tabular layout the `_main_thread_forkserver`
docstring uses, expanded with a fourth "what handles
it" column:
| thread | parent | child (executing) | child (memory) | what handles it |
|---------------------|-----------|-------------------|------------------------------|-----------------------------|
| forkserver worker | continues | sole survivor | live stack | runs the child's bootstrap |
| `trio.run()` thread | continues | not running | leaked stack (zombie bytes) | overwritten by child's fresh `trio.run()` |
| any other thread | continues | not running | leaked stack (zombie bytes) | overwritten / GC'd / clobbered by `exec()` if used |
The "child (executing)" column is the *execution* side
of the coin — what tractor cares about. The "child
(memory)" column is the *memory* side — what trio
cares about.
The "what handles it" column is the deliberate punchline
of the design: nothing has to handle the leaked bytes
*explicitly*. They get clobbered by ordinary forward
progress in the child:
- The fresh `trio.run()` the child boots up allocates
its own stack, scheduler, and run-queue, which over
time overlaps and overwrites the inherited zombie
pages.
- Python's GC walks live objects only; the dead-thread
Python frames aren't reachable from any
`PyThreadState`, so they get freed at the next
collection cycle.
- If the child eventually `exec()`s, the entire address
space is replaced and the leak vanishes.
## What this means for the forkserver design
The crucial point is that **the design doesn't and
*can't* prevent the leak**. There is no userland fix
for COW thread stacks. The kernel hands the child a
duplicated address space; that's what `fork()` *is*. No
amount of pre-fork hookery, `pthread_atfork()`
gymnastics, or post-fork cleanup can un-COW the dead
threads' pages without unmapping them, and unmapping
arbitrary regions of a duplicated address space is
neither portable nor safe.
What the design *does* ensure is the orthogonal
property: the survivor thread is one that doesn't need
any of that leaked state to function. Concretely:
- Survivor is the forkserver worker thread.
- That worker has provably never imported, called into,
or held any reference to `trio`. (Enforced by keeping
the worker's lifecycle entirely in
`_main_thread_forkserver.py` and never letting trio
task-state cross into it.)
- So the leaked pages — trio runner stack, threadpool
caches, etc. — are inert relative to the survivor.
No code path in the child references them.
- The child then boots its own fresh `trio.run()`,
which allocates new state in new pages. Over the
child's lifetime the COW'd zombie pages get
overwritten, GC'd, or (if the child eventually
`exec()`s) discarded wholesale.
The "leak" is real but inert. It costs RSS until
clobbered; it doesn't cost correctness. That's exactly
the property the forkserver pattern is built on, and
it's also why the design needs the "calling thread is
trio-free" precondition to be airtight: if the survivor
were a trio thread, it *would* try to drive the leaked
trio state, and the leak would no longer be inert.
## See also
- `tractor/spawn/_main_thread_forkserver.py` — module
docstring's "What survives the fork? — POSIX
semantics" section is the in-tree, code-adjacent
prose this doc expands on. The cross-table here is a
fourth-column expansion of the table there.
- [python-trio/trio#1614][trio-1614] — the trio issue
with the "leaked" framing, and the canonical thread
for trio + `fork()` hazards more broadly.
- [`subint_fork_blocked_by_cpython_post_fork_issue.md`](./subint_fork_blocked_by_cpython_post_fork_issue.md)
— sibling analysis covering CPython's *post-fork*
hooks (`PyOS_AfterFork_Child`,
`_PyInterpreterState_DeleteExceptMain`) and why
fork-from-non-main-subint is a CPython-level hard
refusal. Complementary axis: this doc is about POSIX
semantics; that doc is about the CPython runtime
layer that runs *after* POSIX `fork()` returns in
the child.
- `man pthread_atfork(3)` — canonical "fork in a
multithreaded process is dangerous" reference.
Especially the rationale section, which is the
closest thing to a normative statement of "the
surviving thread cannot safely use anything the dead
threads were touching."
- `man fork(2)` (Linux) — "Other than [the calling
thread], … no other threads are replicated …"
paragraph is the kernel-side statement of the
execution-side framing this doc opens with.
[trio-1614]: https://github.com/python-trio/trio/issues/1614

View File

@ -0,0 +1,142 @@
# Spawn-time boot-death (`rc=2`) under rapid same-name spawn against a registrar
## Symptom
Spawning N (≥4) sub-actors with the **same name** in tight
succession against a daemon registrar surfaces as
`ActorFailure: Sub-actor (...) died during boot (rc=2)
before completing parent-handshake`.
```
tests/discovery/test_multi_program.py
::test_dup_name_cancel_cascade_escalates_to_hard_kill[n_dups=4]
```
```
tractor._exceptions.ActorFailure:
Sub-actor ('doggy', '<uuid>') died during boot (rc=2)
before completing parent-handshake.
proc: <_ForkedProc pid=<n> returncode=None>
```
The `proc` repr shows `returncode=None` because the repr is
captured before `proc.wait()` returns; the actual
`os.WEXITSTATUS == 2` is reported via `result['died']` in the
race-helper.
## When it surfaces
- N=2 (`n_dups=2`): **always passes**.
- N=4 (`n_dups=4`): **consistent fail** under both `tpt-proto=tcp`
and `tpt-proto=uds`, MTF backend.
- N=8 (`n_dups=8`): **passes** (counter-intuitive — see "racing
windows").
- Non-MTF backends: not yet exercised systematically.
## What previously masked it
Pre the spawn-time `wait_for_peer_or_proc_death` race-helper
(in `tractor.spawn._spawn`), the parent's `start_actor` flow
ended with a bare:
```python
event, chan = await ipc_server.wait_for_peer(uid)
```
That awaits an unsignalled `trio.Event` on `_peer_connected[uid]`.
If the sub-actor process **dies during boot** (before its
runtime executes the parent-callback handshake that sets the
event), the wait parks forever. The dead proc becomes a zombie
because no one ever calls `proc.wait()` to reap it.
In test contexts the failure presented as a hang or a much
later `trio.TooSlowError` from an outer `fail_after`. In
production it'd present as a parent that never makes progress
past `start_actor`. The death itself was silently masked.
## What surfaces it now
`tractor.spawn._spawn.wait_for_peer_or_proc_death` (used by
`_main_thread_forkserver_proc`) races the handshake-wait
against `proc.wait()`. The race-helper raises `ActorFailure`
on death-first instead of parking, exposing the rc=2.
## Hypothesis: registrar-side same-name contention
The test spawns N actors with name `doggy` sequentially:
```python
for i in range(n_dups):
p: Portal = await an.start_actor('doggy')
portals.append(p)
```
Each spawned doggy:
1. Forks via the forkserver.
2. Boots its runtime in `_actor_child_main`.
3. Connects back to the parent for handshake.
4. Connects to the daemon registrar to call `register_actor`.
5. Enters its RPC msg-loop.
Step (4) is where the same-name contention lives. The
registrar's `register_actor` (in
`tractor.discovery._registry`) accepts duplicate names
(stores `(name, uuid) -> addr`), but its internal bookkeeping
may have a non-trivial check (e.g. `wait_for_actor` resolution,
`_addrs2aids` map updates) that errors out under specific
ordering between the existing entry and the incoming one.
`rc=2 == os.WEXITSTATUS == 2` corresponds to `sys.exit(2)`
in the doggy process — typically reached via an unhandled
exception that's translated to exit code 2 by Python's top-
level (e.g. `argparse` errors use 2; `SystemExit(2)` etc.).
So the doggy is hitting an explicit exit path during
`register_actor` or just-after.
The non-monotonic shape (N=2 OK, N=4 BAD, N=8 OK) suggests a
specific timing window — likely "the 3rd register-RPC arrives
while the 1st-or-2nd is in some intermediate state". With
N=8, the additional procs widen the registration spread
enough that no two land in the conflicting window.
## Where to dig next
- Add per-actor logging in `_actor_child_main` and
`register_actor` to surface the actual exception that
triggers the rc=2 exit. Currently the doggy dies before
the parent ever sees its stderr (forkserver doesn't
marshal child stdio back).
- Race-test the registrar's `register_actor` /
`unregister_actor` / `wait_for_actor` against same-name
concurrent calls in isolation (no spawn).
- Consider whether `register_actor` should be idempotent
under same-name re-register or should explicitly reject
same-name (and ideally with a clear `RemoteActorError`,
not `sys.exit(2)`).
## Test-suite handling
Currently:
- `tests/discovery/test_multi_program.py
::test_dup_name_cancel_cascade_escalates_to_hard_kill[n_dups=4]`
is `pytest.mark.xfail(strict=False, reason=...)` to keep
the suite green while this issue is investigated.
- `n_dups=2` and `n_dups=8` continue to validate the
cancel-cascade hard-kill escalation.
Once the underlying race is understood + fixed, drop the
xfail.
## Related work
- The cancel-cascade fix that introduced this regression
test:
`tractor/_exceptions.py:ActorTooSlowError`,
`tractor/runtime/_supervise.py:_try_cancel_then_kill`,
`tractor/runtime/_portal.py:Portal.cancel_actor(
raise_on_timeout=...)`.
- The spawn-time death-detection that exposed this:
`tractor/spawn/_spawn.py:wait_for_peer_or_proc_death`,
used by `tractor/spawn/_main_thread_forkserver.py`.

View File

@ -0,0 +1,161 @@
# `subint` backend: parent trio loop parks after subint teardown (Ctrl-C works; not a CPython-level issue)
Follow-up to the Phase B subint spawn-backend PR (see
`tractor.spawn._subint`, issue #379). Distinct from the
`subint_sigint_starvation_issue.md` (SIGINT-unresponsive
starvation hang): this one is **Ctrl-C-able**, which means
it's *not* the shared-GIL-hostage class and is ours to fix
from inside tractor rather than waiting on upstream CPython
/ msgspec progress.
## TL;DR
After a stuck-subint subactor is torn down via the
hard-kill path, a parent-side trio task parks on an
*orphaned resource* (most likely a `chan.recv()` /
`process_messages` loop on the now-dead subint's IPC
channel) and waits forever for bytes that can't arrive —
because the channel was torn down without emitting a clean
EOF/`BrokenResourceError` to the waiting receiver.
Unlike `subint_sigint_starvation_issue.md`, the main trio
loop **is** iterating normally — SIGINT delivers cleanly
and the test unhangs. But absent Ctrl-C, the test suite
wedges indefinitely.
## Symptom
Running `test_subint_non_checkpointing_child` under
`--spawn-backend=subint` (in
`tests/test_subint_cancellation.py`):
1. Test spawns a subactor whose main task runs
`threading.Event.wait(1.0)` in a loop — releases the
GIL but never inserts a trio checkpoint.
2. Parent does `an.cancel_scope.cancel()`. Our
`subint_proc` cancel path fires: soft-kill sends
`Portal.cancel_actor()` over the live IPC channel →
subint's trio loop *should* process the cancel msg on
its IPC dispatcher task (since the GIL releases are
happening).
3. Expected: subint's `trio.run()` unwinds, driver thread
exits naturally, parent returns.
4. Actual: parent `trio.run()` never completes. Test
hangs past its `trio.fail_after()` deadline.
## Evidence
### `strace` on the hung pytest process during SIGINT
```
--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---
write(17, "\2", 1) = 1
```
Contrast with the SIGINT-starvation hang (see
`subint_sigint_starvation_issue.md`) where that same
`write()` returned `EAGAIN`. Here the SIGINT byte is
written successfully → Python's signal handler pipe is
being drained → main trio loop **is** iterating → SIGINT
gets turned into `trio.Cancelled` → the test unhangs (if
the operator happens to be there to hit Ctrl-C).
### Stack dump (via `tractor.devx.dump_on_hang`)
Single main thread visible, parked in
`trio._core._io_epoll.get_events` inside `trio.run` at the
test's `trio.run(...)` call site. No subint driver thread
(subint was destroyed successfully — this is *after* the
hard-kill path, not during it).
## Root cause hypothesis
Most consistent with the evidence: a parent-side trio
task is awaiting a `chan.recv()` / `process_messages` loop
on the dead subint's IPC channel. The sequence:
1. Soft-kill in `subint_proc` sends `Portal.cancel_actor()`
over the channel. The subint's trio dispatcher *may* or
may not have processed the cancel msg before the subint
was destroyed — timing-dependent.
2. Hard-kill timeout fires (because the subint's main
task was in `threading.Event.wait()` with no trio
checkpoint — cancel-msg processing couldn't race the
timeout).
3. Driver thread abandoned, `_interpreters.destroy()`
runs. Subint is gone.
4. But the parent-side trio task holding a
`chan.recv()` / `process_messages` loop against that
channel was **not** explicitly cancelled. The channel's
underlying socket got torn down, but without a clean
EOF delivered to the waiting recv, the task parks
forever on `trio.lowlevel.wait_readable` (or similar).
This matches the "main loop fine, task parked on
orphaned I/O" signature.
## Why this is ours to fix (not CPython's)
- Main trio loop iterates normally → GIL isn't starved.
- SIGINT is deliverable → not a signal-pipe-full /
wakeup-fd contention scenario.
- The hang is in *our* supervision code, specifically in
how `subint_proc` tears down its side of the IPC when
the subint is abandoned/destroyed.
## Possible fix directions
1. **Explicit parent-side channel abort on subint
abandon.** In `subint_proc`'s teardown block, after the
hard-kill timeout fires, explicitly close the parent's
end of the IPC channel to the subint. Any waiting
`chan.recv()` / `process_messages` task sees
`BrokenResourceError` (or `ClosedResourceError`) and
unwinds.
2. **Cancel parent-side RPC tasks tied to the dead
subint's channel.** The `Actor._rpc_tasks` / nursery
machinery should have a handle on any
`process_messages` loops bound to a specific peer
channel. Iterate those and cancel explicitly.
3. **Bound the top-level `await actor_nursery
._join_procs.wait()` shield in `subint_proc`** (same
pattern as the other bounded shields the hard-kill
patch added). If the nursery never sets `_join_procs`
because a child task is parked, the bound would at
least let the teardown proceed.
Of these, (1) is the most surgical and directly addresses
the root cause. (2) is a defense-in-depth companion. (3)
is a band-aid but cheap to add.
## Current workaround
None in-tree. The test's `trio.fail_after()` bound
currently fires and raises `TooSlowError`, so the test
visibly **fails** rather than hangs — which is
intentional (an unbounded cancellation-audit test would
defeat itself). But in interactive test runs the operator
has to hit Ctrl-C to move past the parked state before
pytest reports the failure.
## Reproducer
```
./py314/bin/python -m pytest \
tests/test_subint_cancellation.py::test_subint_non_checkpointing_child \
--spawn-backend=subint --tb=short --no-header -v
```
Expected: hangs until `trio.fail_after(15)` fires, or
Ctrl-C unwedges it manually.
## References
- `tractor.spawn._subint.subint_proc` — current subint
teardown code; see the `_HARD_KILL_TIMEOUT` bounded
shields + `daemon=True` driver-thread abandonment
(commit `b025c982`).
- `ai/conc-anal/subint_sigint_starvation_issue.md` — the
sibling CPython-level hang (GIL-starvation,
SIGINT-unresponsive) which is **not** this issue.
- Phase B tracking: issue #379.

View File

@ -0,0 +1,337 @@
# `os.fork()` from a non-main sub-interpreter aborts the child (CPython refuses post-fork cleanup)
Third `subint`-class analysis in this project. Unlike its
two siblings (`subint_sigint_starvation_issue.md`,
`subint_cancel_delivery_hang_issue.md`), this one is not a
hang — it's a **hard CPython-level refusal** of an
experimental spawn strategy we wanted to try.
## TL;DR
An in-process sub-interpreter cannot be used as a
"launchpad" for `os.fork()` on current CPython. The fork
syscall succeeds in the parent, but the forked CHILD
process is aborted immediately by CPython's post-fork
cleanup with:
```
Fatal Python error: _PyInterpreterState_DeleteExceptMain: not main interpreter
```
This is enforced by a hard `PyStatus_ERR` gate in
`Python/pystate.c`. The CPython devs acknowledge the
fragility with an in-source comment (`// Ideally we could
guarantee tstate is running main.`) but provide no
mechanism to satisfy the precondition from user code.
**Implication for tractor**: the `subint_fork` backend
sketched in `tractor.spawn._subint_fork` is structurally
dead on current CPython. The submodule is kept as
documentation of the attempt; `--spawn-backend=subint_fork`
raises `NotImplementedError` pointing here.
## Context — why we tried this
The motivation is issue #379's "Our own thoughts, ideas
for `fork()`-workaround/hacks..." section. The existing
trio-backend (`tractor.spawn._trio.trio_proc`) spawns
subactors via `trio.lowlevel.open_process()` → ultimately
`posix_spawn()` or `fork+exec`, from the parent's main
interpreter that is currently running `trio.run()`. This
brushes against a known-fragile interaction between
`trio` and `fork()` tracked in
[python-trio/trio#1614](https://github.com/python-trio/trio/issues/1614)
and siblings — mostly mitigated in `tractor`'s case only
incidentally (we `exec()` immediately post-fork).
The idea was:
1. Create a subint that has *never* imported `trio`.
2. From a worker thread in that subint, call `os.fork()`.
3. In the child, `execv()` back into
`python -m tractor._child` — same as `trio_proc` does.
4. The fork is from a trio-free context → trio+fork
hazards avoided regardless of downstream behavior.
The parent-side orchestration (`ipc_server.wait_for_peer`,
`SpawnSpec`, `Portal` yield) would reuse
`trio_proc`'s flow verbatim, with only the subproc-spawn
mechanics swapped.
## Symptom
Running the prototype (`tractor.spawn._subint_fork.subint_fork_proc`,
see git history prior to the stub revert) on py3.14:
```
Fatal Python error: _PyInterpreterState_DeleteExceptMain: not main interpreter
Python runtime state: initialized
Current thread 0x00007f6b71a456c0 [subint-fork-lau] (most recent call first):
File "<script>", line 2 in <module>
<script>:2: DeprecationWarning: This process (pid=802985) is multi-threaded, use of fork() may lead to deadlocks in the child.
```
Key clues:
- The **`DeprecationWarning`** fires in the parent (before
fork completes) — fork *is* executing, we get that far.
- The **`Fatal Python error`** comes from the child — it
aborts during CPython's post-fork C initialization
before any user Python runs in the child.
- The thread name `subint-fork-lau[nchpad]` is ours —
confirms the fork is being called from the launchpad
subint's driver thread.
## CPython source walkthrough
### Call site — `Modules/posixmodule.c:728-793`
The post-fork-child hook CPython runs in the child process:
```c
void
PyOS_AfterFork_Child(void)
{
PyStatus status;
_PyRuntimeState *runtime = &_PyRuntime;
// re-creates runtime->interpreters.mutex (HEAD_UNLOCK)
status = _PyRuntimeState_ReInitThreads(runtime);
...
PyThreadState *tstate = _PyThreadState_GET();
_Py_EnsureTstateNotNULL(tstate);
...
// Ideally we could guarantee tstate is running main. ← !!!
_PyInterpreterState_ReinitRunningMain(tstate);
status = _PyEval_ReInitThreads(tstate);
...
status = _PyInterpreterState_DeleteExceptMain(runtime);
if (_PyStatus_EXCEPTION(status)) {
goto fatal_error;
}
...
fatal_error:
Py_ExitStatusException(status);
}
```
The `// Ideally we could guarantee tstate is running
main.` comment is a flashing warning sign — the CPython
devs *know* this path is fragile when fork is called from
a non-main subint, but they've chosen to abort rather than
silently corrupt state. Arguably the right call.
### The refusal — `Python/pystate.c:1035-1075`
```c
/*
* Delete all interpreter states except the main interpreter. If there
* is a current interpreter state, it *must* be the main interpreter.
*/
PyStatus
_PyInterpreterState_DeleteExceptMain(_PyRuntimeState *runtime)
{
struct pyinterpreters *interpreters = &runtime->interpreters;
PyThreadState *tstate = _PyThreadState_Swap(runtime, NULL);
if (tstate != NULL && tstate->interp != interpreters->main) {
return _PyStatus_ERR("not main interpreter"); ← our error
}
HEAD_LOCK(runtime);
PyInterpreterState *interp = interpreters->head;
interpreters->head = NULL;
while (interp != NULL) {
if (interp == interpreters->main) {
interpreters->main->next = NULL;
interpreters->head = interp;
interp = interp->next;
continue;
}
// XXX Won't this fail since PyInterpreterState_Clear() requires
// the "current" tstate to be set?
PyInterpreterState_Clear(interp); // XXX must activate?
zapthreads(interp);
...
}
...
}
```
The comment in the docstring (`If there is a current
interpreter state, it *must* be the main interpreter.`) is
the formal API contract. The `XXX` comments further in
suggest the CPython team is already aware this function
has latent issues even in the happy path.
## Chain summary
1. Our launchpad subint's driver OS-thread calls
`os.fork()`.
2. `fork()` succeeds. Child wakes up with:
- The parent's full memory image (including all
subints).
- Only the *calling* thread alive (the driver thread).
- `_PyThreadState_GET()` on that thread returns the
**launchpad subint's tstate**, *not* main's.
3. CPython runs `PyOS_AfterFork_Child()`.
4. It reaches `_PyInterpreterState_DeleteExceptMain()`.
5. Gate check fails: `tstate->interp != interpreters->main`.
6. `PyStatus_ERR("not main interpreter")``fatal_error`
goto → `Py_ExitStatusException()` → child aborts.
Parent-side consequence: `os.fork()` in the subint
bootstrap returned successfully with the child's PID, but
the child died before connecting back. Our parent's
`ipc_server.wait_for_peer(uid)` would hang forever — the
child never gets to `_actor_child_main`.
## Definitive answer to "Open Question 1"
From the (now-stub) `subint_fork_proc` docstring:
> Does CPython allow `os.fork()` from a non-main
> sub-interpreter under the legacy config?
**No.** Not in a usable-by-user-code sense. The fork
syscall is not blocked, but the child cannot survive
CPython's post-fork initialization. This is enforced, not
accidental, and the CPython devs have acknowledged the
fragility in-source.
## What we'd need from CPython to unblock
Any one of these, from least-to-most invasive:
1. **A pre-fork hook mechanism** that lets user code (or
tractor itself via `os.register_at_fork(before=...)`)
swap the current tstate to main before fork runs. The
swap would need to work across the subint→main
boundary, which is the actual hard part —
`_PyThreadState_Swap()` exists but is internal.
2. **A `_PyInterpreterState_DeleteExceptFor(tstate->interp)`
variant** that cleans up all *other* subints while
preserving the calling subint's state. Lets the child
continue executing in the subint after fork; a
subsequent `execv()` clears everything at the OS
level anyway.
3. **A cleaner error** than `Fatal Python error` aborting
the child. Even without fixing the underlying
capability, a raised Python-level exception in the
parent's `fork()` call (rather than a silent child
abort) would at least make the failure mode
debuggable.
## Upstream-report draft (for CPython issue tracker)
### Title
> `os.fork()` from a non-main sub-interpreter aborts the
> child with a fatal error in `PyOS_AfterFork_Child`; can
> we at least make it a clean `RuntimeError` in the
> parent?
### Body
> **Version**: Python 3.14.x
>
> **Summary**: Calling `os.fork()` from a thread currently
> executing inside a sub-interpreter causes the forked
> child process to abort during CPython's post-fork
> cleanup, with the following output in the child:
>
> ```
> Fatal Python error: _PyInterpreterState_DeleteExceptMain: not main interpreter
> ```
>
> From the **parent's** point of view the fork succeeded
> (returned a valid child PID). The failure is completely
> opaque to parent-side Python code — unless the parent
> does `os.waitpid()` it won't even notice the child
> died.
>
> **Root cause** (as I understand it from reading sources):
> `Modules/posixmodule.c::PyOS_AfterFork_Child()` calls
> `_PyInterpreterState_DeleteExceptMain()` with a
> precondition that `_PyThreadState_GET()->interp` be the
> main interpreter. When `fork()` is called from a thread
> executing inside a subinterpreter, the child wakes up
> with its tstate still pointing at the subint, and the
> gate in `Python/pystate.c:1044-1047` fails.
>
> A comment in the source
> (`Modules/posixmodule.c:753` — `// Ideally we could
> guarantee tstate is running main.`) suggests this is a
> known-fragile path rather than an intentional
> invariant.
>
> **Use case**: I was experimenting with using a
> sub-interpreter as a "fork launchpad" — have a subint
> that has never imported `trio`, call `os.fork()` from
> that subint's thread, and in the child `execv()` back
> into a fresh Python interpreter process. The goal was
> to sidestep known issues with `trio` + `fork()`
> interaction (see
> [python-trio/trio#1614](https://github.com/python-trio/trio/issues/1614))
> by guaranteeing the forking context had never been
> "contaminated" by trio's imports or globals. This
> approach would allow `trio`-using applications to
> combine `fork`-based subprocess spawning with
> per-worker `trio.run()` runtimes — a fairly common
> pattern that currently requires workarounds.
>
> **Request**:
>
> Ideally: make fork-from-subint work (e.g., by swapping
> the caller's tstate to main in the pre-fork hook), or
> provide a `_PyInterpreterState_DeleteExceptFor(interp)`
> variant that permits the caller's subint to survive
> post-fork so user code can subsequently `execv()`.
>
> Minimally: convert the fatal child-side abort into a
> clean `RuntimeError` (or similar) raised in the
> parent's `fork()` call. Even if the capability isn't
> expanded, the failure mode should be debuggable by
> user-code in the parent — right now it's a silent
> child death with an error message buried in the
> child's stderr that parent code can't programmatically
> see.
>
> **Related**: PEP 684 (per-interpreter GIL), PEP 734
> (`concurrent.interpreters` public API). The private
> `_interpreters` module is what I used to create the
> launchpad — behavior is the same whether using
> `_interpreters.create('legacy')` or
> `concurrent.interpreters.create()` (the latter was not
> tested but the gate is identical).
>
> Happy to contribute a minimal reproducer + test case if
> this is something the team wants to pursue.
## References
- `Modules/posixmodule.c:728`
[`PyOS_AfterFork_Child`](https://github.com/python/cpython/blob/main/Modules/posixmodule.c#L728)
- `Python/pystate.c:1040`
[`_PyInterpreterState_DeleteExceptMain`](https://github.com/python/cpython/blob/main/Python/pystate.c#L1040)
- PEP 684 (per-interpreter GIL):
<https://peps.python.org/pep-0684/>
- PEP 734 (`concurrent.interpreters` public API):
<https://peps.python.org/pep-0734/>
- [python-trio/trio#1614](https://github.com/python-trio/trio/issues/1614)
— the original motivation for the launchpad idea.
- tractor issue #379 — "Our own thoughts, ideas for
`fork()`-workaround/hacks..." section where this was
first sketched.
- `tractor.spawn._subint_fork` — in-tree stub preserving
the attempted impl's shape in git history.

View File

@ -0,0 +1,375 @@
#!/usr/bin/env python3
'''
Standalone CPython-level feasibility check for the "main-interp
worker-thread forkserver + subint-hosted trio" architecture
proposed as a workaround to the CPython-level refusal
documented in
`ai/conc-anal/subint_fork_blocked_by_cpython_post_fork_issue.md`.
Purpose
-------
Deliberately NOT a `tractor` test. Zero `tractor` imports.
Uses `_interpreters` (private stdlib) + `os.fork()` directly so
the signal is unambiguous pass/fail here is a property of
CPython alone, independent of our runtime.
Run each scenario in isolation; the child's fate is observable
only via `os.waitpid()` of the parent and the scenario's own
status prints.
Scenarios (pick one with `--scenario <name>`)
---------------------------------------------
- `control_subint_thread_fork` the KNOWN-BROKEN case we
documented in `subint_fork_blocked_by_cpython_post_fork_issue.md`:
drive a subint from a thread, call `os.fork()` inside its
`_interpreters.exec()`, watch the child abort. **Included as
a control** if this scenario DOESN'T abort the child, our
analysis is wrong and we should re-check everything.
- `main_thread_fork` baseline sanity. Call `os.fork()` from
the process's main thread. Must always succeed; if this
fails something much bigger is broken.
- `worker_thread_fork` the architectural assertion. Spawn a
regular `threading.Thread` (attached to main interp, NOT a
subint), have IT call `os.fork()`. Child should survive
post-fork cleanup.
- `full_architecture` end-to-end: main-interp worker thread
forks. In the child, fork-thread (still main-interp) creates
a subint, drives a second worker thread inside it that runs
a trivial `trio.run()`. Validates the "root runtime lives in
a subint in the child" piece of the proposed arch.
All scenarios print a self-contained pass/fail banner. Exit
code 0 on expected outcome (which for `control_*` means "child
aborted", not "child succeeded"!).
Requires Python 3.14+.
Usage
-----
::
python subint_fork_from_main_thread_smoketest.py \\
--scenario main_thread_fork
python subint_fork_from_main_thread_smoketest.py \\
--scenario full_architecture
'''
from __future__ import annotations
import argparse
import os
import sys
import threading
import time
# Hard-require py3.14 for the public `concurrent.interpreters`
# API (we still drop to `_interpreters` internally, same as
# `tractor.spawn._subint`).
try:
from concurrent import interpreters as _public_interpreters # noqa: F401
import _interpreters # type: ignore
except ImportError:
print(
'FAIL (setup): requires Python 3.14+ '
'(missing `concurrent.interpreters`)',
file=sys.stderr,
)
sys.exit(2)
# The actual primitives this script exercises live in
# `tractor.spawn._subint_forkserver` — we re-import them here
# rather than inlining so the module and the validation stay
# in sync. (Early versions of this file had them inline for
# the "zero tractor imports" isolation guarantee; now that
# CPython-level feasibility is confirmed, the validated
# primitives have moved into tractor proper.)
from tractor.spawn._main_thread_forkserver import (
fork_from_worker_thread,
wait_child,
)
from tractor.spawn._subint_forkserver import (
run_subint_in_worker_thread,
)
# ----------------------------------------------------------------
# small observability helpers (test-harness only)
# ----------------------------------------------------------------
def _banner(title: str) -> None:
line = '=' * 60
print(f'\n{line}\n{title}\n{line}', flush=True)
def _report(
label: str,
*,
ok: bool,
status_str: str,
expect_exit_ok: bool,
) -> None:
verdict: str = 'PASS' if ok else 'FAIL'
expected_str: str = (
'normal exit (rc=0)'
if expect_exit_ok
else 'abnormal death (signal or nonzero exit)'
)
print(
f'[{verdict}] {label}: '
f'expected {expected_str}; observed {status_str}',
flush=True,
)
# ----------------------------------------------------------------
# scenario: `control_subint_thread_fork` (known-broken)
# ----------------------------------------------------------------
def scenario_control_subint_thread_fork() -> int:
_banner(
'[control] fork from INSIDE a subint (expected: child aborts)'
)
interp_id = _interpreters.create('legacy')
print(f' created subint {interp_id}', flush=True)
# Shared flag: child writes a sentinel file we can detect from
# the parent. If the child manages to write this, CPython's
# post-fork refusal is NOT happening → analysis is wrong.
sentinel = '/tmp/subint_fork_smoketest_control_child_ran'
try:
os.unlink(sentinel)
except FileNotFoundError:
pass
bootstrap = (
'import os\n'
'pid = os.fork()\n'
'if pid == 0:\n'
# child — if CPython's refusal fires this code never runs
f' with open({sentinel!r}, "w") as f:\n'
' f.write("ran")\n'
' os._exit(0)\n'
'else:\n'
# parent side (inside the launchpad subint) — stash the
# forked PID on a shareable dict so we can waitpid()
# from the outer main interp. We can't just return it;
# _interpreters.exec() returns nothing useful.
' import builtins\n'
' builtins._forked_child_pid = pid\n'
)
# NOTE, we can't easily pull state back from the subint.
# For the CONTROL scenario we just time-bound the fork +
# check the sentinel. If sentinel exists → child ran →
# analysis wrong. If not → child aborted → analysis
# confirmed.
done = threading.Event()
def _drive() -> None:
try:
_interpreters.exec(interp_id, bootstrap)
except Exception as err:
print(
f' subint bootstrap raised (expected on some '
f'CPython versions): {type(err).__name__}: {err}',
flush=True,
)
finally:
done.set()
t = threading.Thread(
target=_drive,
name='control-subint-fork-launchpad',
daemon=True,
)
t.start()
done.wait(timeout=5.0)
t.join(timeout=2.0)
# Give the (possibly-aborted) child a moment to die.
time.sleep(0.5)
sentinel_present = os.path.exists(sentinel)
verdict = (
# "PASS" for our analysis means sentinel NOT present.
'PASS' if not sentinel_present else 'FAIL (UNEXPECTED)'
)
print(
f'[{verdict}] control: sentinel present={sentinel_present} '
f'(analysis predicts False — child should abort before '
f'writing)',
flush=True,
)
if sentinel_present:
os.unlink(sentinel)
try:
_interpreters.destroy(interp_id)
except _interpreters.InterpreterError:
pass
return 0 if not sentinel_present else 1
# ----------------------------------------------------------------
# scenario: `main_thread_fork` (baseline sanity)
# ----------------------------------------------------------------
def scenario_main_thread_fork() -> int:
_banner(
'[baseline] fork from MAIN thread (expected: child exits normally)'
)
pid = os.fork()
if pid == 0:
os._exit(0)
return 0 if _wait_child(
pid,
label='main_thread_fork',
expect_exit_ok=True,
) else 1
# ----------------------------------------------------------------
# scenario: `worker_thread_fork` (architectural assertion)
# ----------------------------------------------------------------
def _run_worker_thread_fork_scenario(
label: str,
*,
child_target=None,
) -> int:
'''
Thin wrapper: delegate the actual fork to the
`tractor.spawn._subint_forkserver` primitive, then wait
on the child and render a pass/fail banner.
'''
try:
pid: int = fork_from_worker_thread(
child_target=child_target,
thread_name=f'worker-fork-thread[{label}]',
)
except RuntimeError as err:
print(f'[FAIL] {label}: {err}', flush=True)
return 1
print(f' forked child pid={pid}', flush=True)
ok, status_str = wait_child(pid, expect_exit_ok=True)
_report(
label,
ok=ok,
status_str=status_str,
expect_exit_ok=True,
)
return 0 if ok else 1
def scenario_worker_thread_fork() -> int:
_banner(
'[arch] fork from MAIN-INTERP WORKER thread '
'(expected: child exits normally — this is the one '
'that matters)'
)
return _run_worker_thread_fork_scenario(
'worker_thread_fork',
)
# ----------------------------------------------------------------
# scenario: `full_architecture`
# ----------------------------------------------------------------
_CHILD_TRIO_BOOTSTRAP: str = (
'import trio\n'
'async def _main():\n'
' await trio.sleep(0.05)\n'
' return 42\n'
'result = trio.run(_main)\n'
'assert result == 42, f"trio.run returned {result}"\n'
'print(" CHILD subint: trio.run OK, result=42", '
'flush=True)\n'
)
def _child_trio_in_subint() -> int:
'''
CHILD-side `child_target`: drive a trivial `trio.run()`
inside a fresh legacy-config subint on a worker thread,
using the `tractor.spawn._subint_forkserver.run_subint_in_worker_thread`
primitive. Returns 0 on success.
'''
try:
run_subint_in_worker_thread(
_CHILD_TRIO_BOOTSTRAP,
thread_name='child-subint-trio-thread',
)
except RuntimeError as err:
print(
f' CHILD: run_subint_in_worker_thread timed out / thread '
f'never returned: {err}',
flush=True,
)
return 3
except BaseException as err:
print(
f' CHILD: subint bootstrap raised: '
f'{type(err).__name__}: {err}',
flush=True,
)
return 4
return 0
def scenario_full_architecture() -> int:
_banner(
'[arch-full] worker-thread fork + child runs trio in a '
'subint (end-to-end proposed arch)'
)
return _run_worker_thread_fork_scenario(
'full_architecture',
child_target=_child_trio_in_subint,
)
# ----------------------------------------------------------------
# main
# ----------------------------------------------------------------
SCENARIOS: dict[str, Callable[[], int]] = {
'control_subint_thread_fork': scenario_control_subint_thread_fork,
'main_thread_fork': scenario_main_thread_fork,
'worker_thread_fork': scenario_worker_thread_fork,
'full_architecture': scenario_full_architecture,
}
def main() -> int:
ap = argparse.ArgumentParser(
description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
ap.add_argument(
'--scenario',
choices=sorted(SCENARIOS.keys()),
required=True,
)
args = ap.parse_args()
return SCENARIOS[args.scenario]()
if __name__ == '__main__':
sys.exit(main())

View File

@ -0,0 +1,385 @@
# `subint_forkserver` backend: orphaned-subactor SIGINT wedged in `epoll_wait`
Follow-up to the Phase C `subint_forkserver` spawn-backend
PR (see `tractor.spawn._subint_forkserver`, issue #379).
Surfaced by the xfail'd
`tests/spawn/test_subint_forkserver.py::test_orphaned_subactor_sigint_cleanup_DRAFT`.
Related-but-distinct from
`subint_cancel_delivery_hang_issue.md` (orphaned-channel
park AFTER subint teardown) and
`subint_sigint_starvation_issue.md` (GIL-starvation,
SIGINT never delivered): here the SIGINT IS delivered,
trio's handler IS installed, but trio's event loop never
wakes — so the KBI-at-checkpoint → `_trio_main` catch path
(which is the runtime's *intentional* OS-cancel design)
never fires.
## TL;DR
When a `subint_forkserver`-spawned subactor is orphaned
(parent `SIGKILL`'d, no IPC cancel path available) and then
externally `SIGINT`'d, the subactor hangs in
`trio/_core/_io_epoll.py::get_events` (epoll_wait)
indefinitely — even though:
1. `threading.current_thread() is threading.main_thread()`
post-fork (CPython 3.14 re-designates correctly).
2. Trio's SIGINT handler IS installed in the subactor
(`signal.getsignal(SIGINT)` returns
`<function KIManager.install.<locals>.handler at 0x...>`).
3. The kernel does deliver SIGINT — the signal arrives at
the only thread in the process (the fork-inherited
worker which IS now "main" per Python).
Yet `epoll_wait` does not return. Trio's wakeup-fd mechanism
— the machinery that turns SIGINT into an epoll-wake — is
somehow not firing the wakeup. Until that's fixed, the
intentional "KBI-as-OS-cancel" path in
`tractor/spawn/_entry.py::_trio_main:164` is unreachable
for forkserver-spawned subactors whose parent dies.
## Symptom
Test: `tests/spawn/test_subint_forkserver.py::test_orphaned_subactor_sigint_cleanup_DRAFT`
(currently marked `@pytest.mark.xfail(strict=True)`).
1. Harness subprocess brings up a tractor root actor +
one `run_in_actor(_sleep_forever)` subactor via
`try_set_start_method('subint_forkserver')`.
2. Harness prints `CHILD_PID` (subactor) and
`PARENT_READY` (root actor) markers to stdout.
3. Test `os.kill(parent_pid, SIGKILL)` + `proc.wait()`
to fully reap the root-actor harness.
4. Child (now reparented to pid 1) is still alive.
5. Test `os.kill(child_pid, SIGINT)` and polls
`os.kill(child_pid, 0)` for up to 10s.
6. **Observed**: the child is still alive at deadline —
SIGINT did not unwedge the trio loop.
## What the "intentional" cancel path IS
`tractor/spawn/_entry.py::_trio_main:157-186`
```python
try:
if infect_asyncio:
actor._infected_aio = True
run_as_asyncio_guest(trio_main)
else:
trio.run(trio_main)
except KeyboardInterrupt:
logmeth = log.cancel
exit_status: str = (
'Actor received KBI (aka an OS-cancel)\n'
...
)
```
The "KBI == OS-cancel" mapping IS the runtime's
deliberate, documented design. An OS-level SIGINT should
flow as: kernel → trio handler → KBI at trio checkpoint
→ unwinds `async_main` → surfaces at `_trio_main`'s
`except KeyboardInterrupt:``log.cancel` + clean `rc=0`.
**So fixing this hang is not "add a new SIGINT behavior" —
it's "make the existing designed behavior actually fire in
this backend config".** That's why option (B) ("fix root
cause") is aligned with existing design intent, not a
scope expansion.
## Evidence
### Positive control: standalone fork-from-worker + `trio.run(sleep_forever)` + SIGINT WORKS
```python
import os, signal, time, trio
from tractor.spawn._subint_forkserver import (
fork_from_worker_thread, wait_child,
)
def child_target() -> int:
async def _main():
try:
await trio.sleep_forever()
except KeyboardInterrupt:
print('CHILD: caught KBI — trio SIGINT works!')
return
trio.run(_main)
return 0
pid = fork_from_worker_thread(child_target, thread_name='trio-sigint-test')
time.sleep(1.0)
os.kill(pid, signal.SIGINT)
wait_child(pid)
```
Result: `CHILD: caught KBI — trio SIGINT works!` + clean
exit. So the fork-child + trio signal plumbing IS healthy
in isolation. The hang appears only with the full tractor
subactor runtime on top.
### Negative test: full tractor subactor + orphan-SIGINT
Equivalent to the xfail test. Traceback dump via
`faulthandler.register(SIGUSR1, all_threads=True)` at the
stuck moment:
```
Current thread 0x00007... [subint-forkserv] (most recent call first):
File ".../trio/_core/_io_epoll.py", line 245 in get_events
File ".../trio/_core/_run.py", line 2415 in run
File "tractor/spawn/_entry.py", line 162 in _trio_main
File "tractor/_child.py", line 72 in _actor_child_main
File "tractor/spawn/_subint_forkserver.py", line 650 in _child_target
File "tractor/spawn/_subint_forkserver.py", line 308 in _worker
File ".../threading.py", line 1024 in run
```
### Thread + signal-mask inventory of the stuck subactor
Single thread (`tid == pid`, comm `'subint-forkserv'`,
which IS `threading.main_thread()` post-fork):
```
SigBlk: 0000000000000000 # nothing blocked
SigIgn: 0000000001001000 # SIGPIPE etc (Python defaults)
SigCgt: 0000000108000202 # bit 1 = SIGINT caught
```
Bit 1 set in `SigCgt` → SIGINT handler IS installed. So
trio's handler IS in place at the kernel level — not a
"handler missing" situation.
### Handler identity
Inside the subactor's RPC body, `signal.getsignal(SIGINT)`
returns `<function KIManager.install.<locals>.handler at
0x...>` — trio's own `KIManager` handler. tractor's only
SIGINT touches are `signal.getsignal()` *reads* (to stash
into `debug.DebugStatus._trio_handler`); nothing writes
over trio's handler outside the debug-REPL shielding path
(`devx/debug/_tty_lock.py::shield_sigint`) which isn't
engaged here (no debug_mode).
## Ruled out
- **GIL starvation / signal-pipe-full** (class A,
`subint_sigint_starvation_issue.md`): subactor runs on
its own GIL (separate OS process), not sharing with the
parent → no cross-process GIL contention. And `strace`-
equivalent in the signal mask shows SIGINT IS caught,
not queued.
- **Orphaned channel park** (`subint_cancel_delivery_hang_issue.md`):
different failure mode — that one has trio iterating
normally and getting wedged on an orphaned
`chan.recv()` AFTER teardown. Here trio's event loop
itself never wakes.
- **Tractor explicitly catching + swallowing KBI**:
greppable — the one `except KeyboardInterrupt:` in the
runtime is the INTENTIONAL cancel-path catch at
`_trio_main:164`. `async_main` uses `except Exception`
(not BaseException), so KBI should propagate through
cleanly if it ever fires.
- **Missing `signal.set_wakeup_fd` (main-thread
restriction)**: post-fork, the fork-worker thread IS
`threading.main_thread()`, so trio's main-thread check
passes and its wakeup-fd install should succeed.
## Root cause hypothesis (unverified)
The SIGINT handler fires but trio's wakeup-fd write does
not wake `epoll_wait`. Candidate causes, ranked by
plausibility:
1. **Wakeup-fd lifecycle race around tractor IPC setup.**
`async_main` spins up an IPC server + `process_messages`
loops early. Somewhere in that path the wakeup-fd that
trio registered with its epoll instance may be
closed/replaced/clobbered, so subsequent SIGINT writes
land on an fd that's no longer in the epoll set.
Evidence needed: compare
`signal.set_wakeup_fd(-1)` return value inside a
post-tractor-bringup RPC body vs. a pre-bringup
equivalent. If they differ, that's it.
2. **Shielded cancel scope around `process_messages`.**
The RPC message loop is likely wrapped in a trio cancel
scope; if that scope is `shield=True` at any outer
layer, KBI scheduled at a checkpoint could be absorbed
by the shield and never bubble out to `_trio_main`.
3. **Pre-fork wakeup-fd inheritance.** trio in the PARENT
process registered a wakeup-fd with its own epoll. The
child inherits the fd number but not the parent's
epoll instance — if tractor/trio re-uses the parent's
stale fd number anywhere, writes would go to a no-op
fd. (This is the least likely — `trio.run()` on the
child calls `KIManager.install` which should install a
fresh wakeup-fd from scratch.)
## Cross-backend scope question
**Untested**: does the same orphan-SIGINT hang reproduce
against the `trio_proc` backend (stock subprocess + exec)?
If yes → pre-existing tractor bug, independent of
`subint_forkserver`. If no → something specific to the
fork-from-worker path (e.g. inherited fds, mid-epoll-setup
interference).
**Quick repro for trio_proc**:
```python
# save as /tmp/trio_proc_orphan_sigint_repro.py
import os, sys, signal, time, glob
import subprocess as sp
SCRIPT = '''
import os, sys, trio, tractor
async def _sleep_forever():
print(f"CHILD_PID={os.getpid()}", flush=True)
await trio.sleep_forever()
async def _main():
async with (
tractor.open_root_actor(registry_addrs=[("127.0.0.1", 12350)]),
tractor.open_nursery() as an,
):
await an.run_in_actor(_sleep_forever, name="sf-child")
print(f"PARENT_READY={os.getpid()}", flush=True)
await trio.sleep_forever()
trio.run(_main)
'''
proc = sp.Popen(
[sys.executable, '-c', SCRIPT],
stdout=sp.PIPE, stderr=sp.STDOUT,
)
# parse CHILD_PID + PARENT_READY off proc.stdout ...
# SIGKILL parent, SIGINT child, poll.
```
If that hangs too, open a broader issue; if not, this is
`subint_forkserver`-specific (likely fd-inheritance-related).
## Why this is ours to fix (not CPython's)
- Signal IS delivered (`SigCgt` bitmask confirms).
- Handler IS installed (trio's `KIManager`).
- Thread identity is correct post-fork.
- `_trio_main` already has the intentional KBI→clean-exit
path waiting to fire.
Every CPython-level precondition is met. Something in
tractor's runtime or trio's integration with it is
breaking the SIGINT→wakeup→event-loop-wake pipeline.
## Possible fix directions
1. **Audit the wakeup-fd across tractor's IPC bringup.**
Add a trio startup hook that captures
`signal.set_wakeup_fd(-1)` at `_trio_main` entry,
after `async_main` enters, and periodically — assert
it's unchanged. If it moves, track down the writer.
2. **Explicit `signal.set_wakeup_fd` reset after IPC
setup.** Brute force: re-install a fresh wakeup-fd
mid-bringup. Band-aid, but fast to try.
3. **Ensure no `shield=True` cancel scope envelopes the
RPC-message-loop / IPC-server task.** If one does,
KBI-at-checkpoint never escapes.
4. **Once fixed, the `child_sigint='trio'` mode on
`subint_forkserver_proc`** becomes effectively a no-op
or a doc-only mode — trio's natural handler already
does the right thing. Might end up removing the flag
entirely if there's no behavioral difference between
modes.
## Current workaround
None; `child_sigint` defaults to `'ipc'` (IPC cancel is
the only reliable cancel path today), and the xfail test
documents the gap. Operators hitting orphan-SIGINT get a
hung process that needs `SIGKILL`.
## Reproducer
Inline, standalone (no pytest):
```python
# save as /tmp/orphan_sigint_repro.py (py3.14+)
import os, sys, signal, time, glob, trio
import tractor
from tractor.spawn._subint_forkserver import (
fork_from_worker_thread,
)
async def _sleep_forever():
print(f'SUBACTOR[{os.getpid()}]', flush=True)
await trio.sleep_forever()
async def _main():
async with (
tractor.open_root_actor(
registry_addrs=[('127.0.0.1', 12349)],
),
tractor.open_nursery() as an,
):
await an.run_in_actor(_sleep_forever, name='sf-child')
await trio.sleep_forever()
def child_target() -> int:
from tractor.spawn._spawn import try_set_start_method
try_set_start_method('subint_forkserver')
trio.run(_main)
return 0
pid = fork_from_worker_thread(child_target, thread_name='repro')
time.sleep(3.0)
# find the subactor pid via /proc
children = []
for path in glob.glob(f'/proc/{pid}/task/*/children'):
with open(path) as f:
children.extend(int(x) for x in f.read().split() if x)
subactor_pid = children[0]
# SIGKILL root → orphan the subactor
os.kill(pid, signal.SIGKILL)
os.waitpid(pid, 0)
time.sleep(0.3)
# SIGINT the orphan — should cause clean trio exit
os.kill(subactor_pid, signal.SIGINT)
# poll for exit
for _ in range(100):
try:
os.kill(subactor_pid, 0)
time.sleep(0.1)
except ProcessLookupError:
print('HARNESS: subactor exited cleanly ✔')
sys.exit(0)
os.kill(subactor_pid, signal.SIGKILL)
print('HARNESS: subactor hung — reproduced')
sys.exit(1)
```
Expected (current): `HARNESS: subactor hung — reproduced`.
After fix: `HARNESS: subactor exited cleanly ✔`.
## References
- `tractor/spawn/_entry.py::_trio_main:157-186` — the
intentional KBI→clean-exit path this bug makes
unreachable.
- `tractor/spawn/_subint_forkserver` — the backend whose
orphan cancel-robustness this blocks.
- `tests/spawn/test_subint_forkserver.py::test_orphaned_subactor_sigint_cleanup_DRAFT`
— the xfail'd reproducer in the test suite.
- `ai/conc-anal/subint_cancel_delivery_hang_issue.md`
sibling "orphaned channel park" hang (different class).
- `ai/conc-anal/subint_sigint_starvation_issue.md`
sibling "GIL starvation SIGINT drop" hang (different
class).
- tractor issue #379 — subint backend tracking.

View File

@ -0,0 +1,851 @@
# `subint_forkserver` backend: `test_cancellation.py` multi-level cancel cascade hang
> **Tracked at:** [#449](https://github.com/goodboy/tractor/issues/449)
Follow-up tracker: surfaced while wiring the new
`subint_forkserver` spawn backend into the full tractor
test matrix (step 2 of the post-backend-lands plan).
See also
`ai/conc-anal/subint_forkserver_orphan_sigint_hang_issue.md`
— sibling tracker for a different forkserver-teardown
class which probably shares the same fundamental root
cause (fork-FD-inheritance across nested spawns).
## TL;DR
`tests/test_cancellation.py::test_nested_multierrors[subint_forkserver]`
hangs indefinitely under our new backend. The hang is
**inside the graceful IPC cancel cascade** — every actor
in the multi-level tree parks in `epoll_wait` waiting
for IPC messages that never arrive. Not a hard-kill /
tree-reap issue (we don't reach the hard-kill fallback
path at all).
Working hypothesis (unverified): **`os.fork()` from a
subactor inherits the root parent's IPC listener socket
FDs**. When a first-level subactor forkserver-spawns a
grandchild, that grandchild inherits both its direct
spawner's FDs AND the root's FDs — IPC message routing
becomes ambiguous (or silently sends to the wrong
channel), so the cancel cascade can't reach its target.
## Corrected diagnosis vs. earlier draft
An earlier version of this doc claimed the root cause
was **"forkserver teardown doesn't tree-kill
descendants"** (SIGKILL only reaches the direct child,
grandchildren survive and hold TCP `:1616`). That
diagnosis was **wrong**, caused by conflating two
observations:
1. *5-zombie leak holding :1616* — happened in my own
workflow when I aborted a bg pytest task with
`pkill` (SIGTERM/SIGKILL, not SIGINT). The abrupt
kill skipped the graceful `ActorNursery.__aexit__`
cancel cascade entirely, orphaning descendants to
init. **This was my cleanup bug, not a forkserver
teardown bug.** Codified the fix (SIGINT-first +
bounded wait before SIGKILL) in
`feedback_sc_graceful_cancel_first.md` +
`.claude/skills/run-tests/SKILL.md`.
2. *`test_nested_multierrors` hangs indefinitely*
the real, separate, forkserver-specific bug
captured by this doc.
The two symptoms are unrelated. The tree-kill / setpgrp
fix direction proposed earlier would not help (1) (SC-
graceful-cleanup is the right answer there) and would
not help (2) (the hang is in the cancel cascade, not
in the hard-kill fallback).
## Symptom
Reproducer (py3.14, clean env):
```sh
# preflight: ensure clean env
ss -tlnp 2>/dev/null | grep ':1616' && echo 'FOUL — cleanup first!' || echo 'clean'
./py314/bin/python -m pytest --spawn-backend=subint_forkserver \
'tests/test_cancellation.py::test_nested_multierrors[subint_forkserver]' \
--timeout=30 --timeout-method=thread --tb=short -v
```
Expected: `pytest-timeout` fires at 30s with a thread-
dump banner, but the process itself **remains alive
after timeout** and doesn't unwedge on subsequent
SIGINT. Requires SIGKILL to reap.
## Evidence (tree structure at hang point)
All 5 processes are kernel-level `S` (sleeping) in
`do_epoll_wait` (trio's event loop waiting on I/O):
```
PID PPID THREADS NAME ROLE
333986 1 2 subint-forkserv pytest main (the test body)
333993 333986 3 subint-forkserv "child 1" spawner subactor
334003 333993 1 subint-forkserv grandchild errorer under child-1
334014 333993 1 subint-forkserv grandchild errorer under child-1
333999 333986 1 subint-forkserv "child 2" spawner subactor (NO grandchildren!)
```
### Asymmetric tree depth
The test's `spawn_and_error(breadth=2, depth=3)` should
have BOTH direct children spawning 2 grandchildren
each, going 3 levels deep. Reality:
- Child 1 (333993, 3 threads) DID spawn its two
grandchildren as expected — fully booted trio
runtime.
- Child 2 (333999, 1 thread) did NOT spawn any
grandchildren — clearly never completed its
nursery's first `run_in_actor`. Its 1-thread state
suggests the runtime never fully booted (no trio
worker threads for `waitpid`/IPC).
This asymmetry is the key clue: the two direct
children started identically but diverged. Probably a
race around fork-inherited state (listener FDs,
subactor-nursery channel state) that happens to land
differently depending on spawn ordering.
### Parent-side state
Thread-dump of pytest main (333986) at the hang:
- Main trio thread — parked in
`trio._core._io_epoll.get_events` (epoll_wait on
its event loop). Waiting for IPC from children.
- Two trio-cache worker threads — each parked in
`outcome.capture(sync_fn)` calling
`os.waitpid(child_pid, 0)`. These are our
`_ForkedProc.wait()` off-loads. They're waiting for
the direct children to exit — but children are
stuck in their own epoll_wait waiting for IPC from
the parent.
**It's a deadlock, not a leak:** the parent is
correctly running `soft_kill(proc, _ForkedProc.wait,
portal)` (graceful IPC cancel via
`Portal.cancel_actor()`), but the children never
acknowledge the cancel message (or the message never
reaches them through the tangled post-fork IPC).
## What's NOT the cause (ruled out)
- **`_ForkedProc.kill()` only SIGKILLs direct pid /
missing tree-kill**: doesn't apply — we never reach
the hard-kill path. The deadlock is in the graceful
cancel cascade.
- **Port `:1616` contention**: ruled out after the
`reg_addr` fixture-wiring fix; each test session
gets a unique port now.
- **GIL starvation / SIGINT pipe filling** (class-A,
`subint_sigint_starvation_issue.md`): doesn't apply
— each subactor is its own OS process with its own
GIL (not legacy-config subint).
- **Child-side `_trio_main` absorbing KBI**: grep
confirmed; `_trio_main` only catches KBI at the
`trio.run()` callsite, which is reached only if the
trio loop exits normally. The children here never
exit trio.run() — they're wedged inside.
## Hypothesis: FD inheritance across nested forks
`subint_forkserver_proc` calls
`fork_from_worker_thread()` which ultimately does
`os.fork()` from a dedicated worker thread. Standard
Linux/POSIX fork semantics: **the child inherits ALL
open FDs from the parent**, including listener
sockets, epoll fds, trio wakeup pipes, and the
parent's IPC channel sockets.
At root-actor fork-spawn time, the root's IPC server
listener FDs are open in the parent. Those get
inherited by child 1. Child 1 then forkserver-spawns
its OWN subactor (grandchild). The grandchild
inherits FDs from child 1 — but child 1's address
space still contains **the root's IPC listener FDs
too** (inherited at first fork). So the grandchild
has THREE sets of FDs:
1. Its own (created after becoming a subactor).
2. Its direct parent child-1's.
3. The ROOT's (grandparent's) — inherited transitively.
IPC message routing may be ambiguous in this tangled
state. Or a listener socket that the root thinks it
owns is actually open in multiple processes, and
messages sent to it go to an arbitrary one. That
would exactly match the observed "graceful cancel
never propagates".
This hypothesis predicts the bug **scales with fork
depth**: single-level forkserver spawn
(`test_subint_forkserver_spawn_basic`) works
perfectly, but any test that spawns a second level
deadlocks. Matches observations so far.
## Fix directions (to validate)
### 1. `close_fds=True` equivalent in `fork_from_worker_thread()`
`subprocess.Popen` / `trio.lowlevel.open_process` have
`close_fds=True` by default on POSIX — they
enumerate open FDs in the child post-fork and close
everything except stdio + any explicitly-passed FDs.
Our raw `os.fork()` doesn't. Adding the equivalent to
our `_worker` prelude would isolate each fork
generation's FD set.
Implementation sketch in
`tractor.spawn._subint_forkserver.fork_from_worker_thread._worker`:
```python
def _worker() -> None:
pid: int = os.fork()
if pid == 0:
# CHILD: close inherited FDs except stdio + the
# pid-pipe we just opened.
keep: set[int] = {0, 1, 2, rfd, wfd}
import resource
soft, _ = resource.getrlimit(resource.RLIMIT_NOFILE)
os.closerange(3, soft) # blunt; or enumerate /proc/self/fd
# ... then child_target() as before
```
Problem: overly aggressive — closes FDs the
grandchild might legitimately need (e.g. its parent's
IPC channel for the spawn-spec handshake, if we rely
on that). Needs thought about which FDs are
"inheritable and safe" vs. "inherited by accident".
### 2. Cloexec on tractor's own FDs
Set `FD_CLOEXEC` on tractor-created sockets (listener
sockets, IPC channel sockets, pipes). This flag
causes automatic close on `execve`, but since we
`fork()` without `exec()`, this alone doesn't help.
BUT — combined with a child-side explicit close-
non-cloexec loop, it gives us a way to mark "my
private FDs" vs. "safe to inherit". Most robust, but
requires tractor-wide audit.
### 3. Explicit FD cleanup in `_ForkedProc`/`_child_target`
Have `subint_forkserver_proc`'s `_child_target`
closure explicitly close the parent-side IPC listener
FDs before calling `_actor_child_main`. Requires
being able to enumerate "the parent's listener FDs
that the child shouldn't keep" — plausible via
`Actor.ipc_server`'s socket objects.
### 4. Use `os.posix_spawn` with explicit `file_actions`
Instead of raw `os.fork()`, use `os.posix_spawn()`
which supports explicit file-action specifications
(close this FD, dup2 that FD). Cleaner semantics, but
probably incompatible with our "no exec" requirement
(subint_forkserver is a fork-without-exec design).
**Likely correct answer: (3) — targeted FD cleanup
via `actor.ipc_server` handle.** (1) is too blunt,
(2) is too wide-ranging, (4) changes the spawn
mechanism.
## Reproducer (standalone, no pytest)
```python
# save as /tmp/forkserver_nested_hang_repro.py (py3.14+)
import trio, tractor
async def assert_err():
assert 0
async def spawn_and_error(breadth: int = 2, depth: int = 1):
async with tractor.open_nursery() as n:
for i in range(breadth):
if depth > 0:
await n.run_in_actor(
spawn_and_error,
breadth=breadth,
depth=depth - 1,
name=f'spawner_{i}_{depth}',
)
else:
await n.run_in_actor(
assert_err,
name=f'errorer_{i}',
)
async def _main():
async with tractor.open_nursery() as n:
for i in range(2):
await n.run_in_actor(
spawn_and_error,
name=f'top_{i}',
breadth=2,
depth=1,
)
if __name__ == '__main__':
from tractor.spawn._spawn import try_set_start_method
try_set_start_method('subint_forkserver')
with trio.fail_after(20):
trio.run(_main)
```
Expected (current): hangs on `trio.fail_after(20)`
— children never ack the error-propagation cancel
cascade. Pattern: top 2 direct children, 4
grandchildren, 1 errorer deadlocks while trying to
unwind through its parent chain.
After fix: `trio.TooSlowError`-free completion; the
root's `open_nursery` receives the
`BaseExceptionGroup` containing the `AssertionError`
from the errorer and unwinds cleanly.
## Update — 2026-04-23: partial fix landed, deeper layer surfaced
Three improvements landed as separate commits in the
`subint_forkserver_backend` branch (see `git log`):
1. **`_close_inherited_fds()` in fork-child prelude**
(`tractor/spawn/_subint_forkserver.py`). POSIX
close-fds-equivalent enumeration via
`/proc/self/fd` (or `RLIMIT_NOFILE` fallback), keep
only stdio. This is fix-direction (1) from the list
above — went with the blunt form rather than the
targeted enum-via-`actor.ipc_server` form, turns
out the aggressive close is safe because every
inheritable resource the fresh child needs
(IPC-channel socket, etc.) is opened AFTER the
fork anyway.
2. **`_ForkedProc.wait()` via `os.pidfd_open()` +
`trio.lowlevel.wait_readable()`** — matches the
`trio.Process.wait` / `mp.Process.sentinel` pattern
used by `trio_proc` and `proc_waiter`. Gives us
fully trio-cancellable child-wait (prior impl
blocked a cache thread on a sync `os.waitpid` that
was NOT trio-cancellable due to
`abandon_on_cancel=False`).
3. **`_parent_chan_cs` wiring** in
`tractor/runtime/_runtime.py`: capture the shielded
`loop_cs` for the parent-channel `process_messages`
task in `async_main`; explicitly cancel it in
`Actor.cancel()` teardown. This breaks the shield
during teardown so the parent-chan loop exits when
cancel is issued, instead of parking on a parent-
socket EOF that might never arrive under fork
semantics.
**Concrete wins from (1):** the sibling
`subint_forkserver_orphan_sigint_hang_issue.md` class
is **now fixed**`test_orphaned_subactor_sigint_cleanup_DRAFT`
went from strict-xfail to pass. The xfail mark was
removed; the test remains as a regression guard.
**test_nested_multierrors STILL hangs** though.
### Updated diagnosis (narrowed)
DIAGDEBUG instrumentation of `process_messages` ENTER/
EXIT pairs + `_parent_chan_cs.cancel()` call sites
showed (captured during a 20s-timeout repro):
- 80 `process_messages` ENTERs, 75 EXITs → 5 stuck.
- **All 40 `shield=True` ENTERs matched EXIT** — every
shielded parent-chan loop exits cleanly. The
`_parent_chan_cs` wiring works as intended.
- **The 5 stuck loops are all `shield=False`** — peer-
channel handlers (inbound connections handled by
`handle_stream_from_peer` in stream_handler_tn).
- After our `_parent_chan_cs.cancel()` fires, NEW
shielded process_messages loops start (on the
session reg_addr port — probably discovery-layer
reconnection attempts). These don't block teardown
(they all exit) but indicate the cancel cascade has
more moving parts than expected.
### Remaining unknown
Why don't the 5 peer-channel loops exit when
`service_tn.cancel_scope.cancel()` fires? They're in
`stream_handler_tn` which IS `service_tn` in the
current configuration (`open_ipc_server(parent_tn=
service_tn, stream_handler_tn=service_tn)`). A
standard nursery-scope-cancel should propagate through
them — no shield, no special handler. Something
specific to the fork-spawned configuration keeps them
alive.
Candidate follow-up experiments:
- Dump the trio task tree at the hang point (via
`stackscope` or direct trio introspection) to see
what each stuck loop is awaiting. `chan.__anext__`
on a socket recv? An inner lock? A shielded sub-task?
- Compare peer-channel handler lifecycle under
`trio_proc` vs `subint_forkserver` with equivalent
logging to spot the divergence.
- Investigate whether the peer handler is caught in
the `except trio.Cancelled:` path at
`tractor/ipc/_server.py:448` that re-raises — but
re-raise means it should still exit. Unless
something higher up swallows it.
### Attempted fix (DID NOT work) — hypothesis (3)
Tried: in `_serve_ipc_eps` finally, after closing
listeners, also iterate `server._peers` and
sync-close each peer channel's underlying stream
socket fd:
```python
for _uid, _chans in list(server._peers.items()):
for _chan in _chans:
try:
_stream = _chan._transport.stream if _chan._transport else None
if _stream is not None:
_stream.socket.close() # sync fd close
except (AttributeError, OSError):
pass
```
Theory: closing the socket fd from outside the stuck
recv task would make the recv see EBADF /
ClosedResourceError and unblock.
Result: `test_nested_multierrors[subint_forkserver]`
still hangs identically. Either:
- The sync `socket.close()` doesn't propagate into
trio's in-flight `recv_some()` the way I expected
(trio may hold an internal reference that keeps the
fd open even after an external close), or
- The stuck recv isn't even the root blocker and the
peer handlers never reach the finally for some
reason I haven't understood yet.
Either way, the sync-close hypothesis is **ruled
out**. Reverted the experiment, restored the skip-
mark on the test.
### Aside: `-s` flag does NOT change `test_nested_multierrors` behavior
Tested explicitly: both with and without `-s`, the
test hangs identically. So the capture-pipe-fill
hypothesis is **ruled out** for this test.
The earlier `test_context_stream_semantics.py` `-s`
observation was most likely caused by a competing
pytest run in my session (confirmed via process list
— my leftover pytest was alive at that time and
could have been holding state on the default
registry port).
## Update — 2026-04-23 (late): cancel delivery ruled in, nursery-wait ruled BLOCKER
**New diagnostic run** instrumented
`handle_stream_from_peer` at ENTER / `except
trio.Cancelled:` / finally, plus `Actor.cancel()`
just before `self._parent_chan_cs.cancel()`. Result:
- **40 `handle_stream_from_peer` ENTERs**.
- **0 `except trio.Cancelled:` hits** — cancel
never fires on any peer-handler.
- **35 finally hits** — those handlers exit via
peer-initiated EOF (normal return), NOT cancel.
- **5 handlers never reach finally** — stuck forever.
- **`Actor.cancel()` fired in 12 PIDs** — but the
PIDs with peer handlers that DIDN'T fire
Actor.cancel are exactly **root + 2 direct
spawners**. These 3 actors have peer handlers
(for their own subactors) that stay stuck because
**`Actor.cancel()` at these levels never runs**.
### The actual deadlock shape
`Actor.cancel()` lives in
`open_root_actor.__aexit__` / `async_main` teardown.
That only runs when the enclosing `async with
tractor.open_nursery()` exits. The nursery's
`__aexit__` calls the backend `*_proc` spawn target's
teardown, which does `soft_kill() →
_ForkedProc.wait()` on its child PID. That wait is
trio-cancellable via pidfd now (good) — but nothing
CANCELS it because the outer scope only cancels when
`Actor.cancel()` runs, which only runs when the
nursery completes, which waits on the child.
It's a **multi-level mutual wait**:
```
root blocks on spawner.wait()
spawner blocks on grandchild.wait()
grandchild blocks on errorer.wait()
errorer Actor.cancel() ran, but process
may not have fully exited yet
(something in root_tn holding on?)
```
Each level waits for the level below. The bottom
level (errorer) reaches Actor.cancel(), but its
process may not fully exit — meaning its pidfd
doesn't go readable, meaning the grandchild's
waitpid doesn't return, meaning the grandchild's
nursery doesn't unwind, etc. all the way up.
### Refined question
**Why does an errorer process not exit after its
`Actor.cancel()` completes?**
Possibilities:
1. `_parent_chan_cs.cancel()` fires (shielded
parent-chan loop unshielded), but the task is
stuck INSIDE the shielded loop's recv in a way
that cancel still can't break.
2. After `Actor.cancel()` returns, `async_main`
still has other tasks in `root_tn` waiting for
something that never arrives (e.g. outbound
IPC reply delivery).
3. The `os._exit(rc)` in `_worker` (at
`_subint_forkserver.py`) doesn't run because
`_child_target` never returns.
Next-session candidate probes (in priority order):
1. **Instrument `_worker`'s fork-child branch** to
confirm whether `child_target()` returns (and
thus `os._exit(rc)` is reached) for errorer
PIDs. If yes → process should die; if no →
trace back into `_actor_child_main` /
`_trio_main` / `async_main` to find the stuck
spot.
2. **Instrument `async_main`'s final unwind** to
see which await in the teardown doesn't
complete.
3. **Compare under `trio_proc` backend** at the
same `_worker`-equivalent level to see where
the flows diverge.
### Rule-out: NOT a stuck peer-chan recv
Earlier hypothesis was that the 5 stuck peer-chan
loops were blocked on a socket recv that cancel
couldn't interrupt. This pass revealed the real
cause: cancel **never reaches those tasks** because
their owning actor's `Actor.cancel()` never runs.
The recvs are fine — they're just parked because
nothing is telling them to stop.
## Update — 2026-04-23 (very late): leaves exit, middle actors stuck in `trio.run`
Yet another instrumentation pass — this time
printing at:
- `_worker` child branch: `pre child_target()` /
`child_target RETURNED rc=N` / `about to
os._exit(rc)`
- `_trio_main`: `about to trio.run` /
`trio.run RETURNED NORMALLY` / `FINALLY`
**Fresh-run results** (`test_nested_multierrors[
subint_forkserver]`, depth=1/breadth=2, 1 root + 14
forked = 15 actors total):
- **9 processes completed the full flow**
`trio.run RETURNED NORMALLY` → `child_target
RETURNED rc=0` → `about to os._exit(0)`. These
are the LEAVES of the tree (errorer actors) plus
their direct parents (depth-0 spawners). They
actually exit their processes.
- **5 processes are stuck INSIDE `trio.run(trio_main)`**
— they hit "about to trio.run" but NEVER see
"trio.run RETURNED NORMALLY". These are root +
top-level spawners + one intermediate.
**What this means:** `async_main` itself is the
deadlock holder, not the peer-channel loops.
Specifically, the outer `async with root_tn:` in
`async_main` never exits for the 5 stuck actors.
Their `trio.run` never returns → `_trio_main`
catch/finally never runs → `_worker` never reaches
`os._exit(rc)` → the PROCESS never dies → its
parent's `_ForkedProc.wait()` blocks → parent's
nursery hangs → parent's `async_main` hangs → ...
### The new precise question
**What task in the 5 stuck actors' `async_main`
never completes?** Candidates:
1. The shielded parent-chan `process_messages`
task in `root_tn` — but we explicitly cancel it
via `_parent_chan_cs.cancel()` in `Actor.cancel()`.
However, `Actor.cancel()` only runs during
`open_root_actor.__aexit__`, which itself runs
only after `async_main`'s outer unwind — which
doesn't happen. So the shield isn't broken.
2. `await actor_nursery._join_procs.wait()` or
similar in the inline backend `*_proc` flow.
3. `_ForkedProc.wait()` on a grandchild that
actually DID exit — but the pidfd_open watch
didn't fire for some reason (race between
pidfd_open and the child exiting?).
The most specific next probe: **add DIAG around
`_ForkedProc.wait()` enter/exit** to see whether
the pidfd-based wait returns for every grandchild
exit. If a stuck parent's `_ForkedProc.wait()`
NEVER returns despite its child exiting, the
pidfd mechanism has a race bug under nested
forkserver.
Alternative probe: instrument `async_main`'s outer
nursery exits to find which nursery's `__aexit__`
is stuck, drilling down from `trio.run` to the
specific `async with` that never completes.
### Cascade summary (updated tree view)
```
ROOT (pytest) STUCK in trio.run
├── top_0 (spawner, d=1) STUCK in trio.run
│ ├── spawner_0_d1_0 (d=0) exited (os._exit 0)
│ │ ├── errorer_0_0 exited (os._exit 0)
│ │ └── errorer_0_1 exited (os._exit 0)
│ └── spawner_0_d1_1 (d=0) exited (os._exit 0)
│ ├── errorer_0_2 exited (os._exit 0)
│ └── errorer_0_3 exited (os._exit 0)
└── top_1 (spawner, d=1) STUCK in trio.run
├── spawner_1_d1_0 (d=0) STUCK in trio.run (sibling race?)
│ ├── errorer_1_0 exited
│ └── errorer_1_1 exited
└── spawner_1_d1_1 (d=0) STUCK in trio.run
├── errorer_1_2 exited
└── errorer_1_3 exited
```
Grandchildren (d=0 spawners) exit OR stick —
asymmetric. Not purely depth-determined. Some race
condition in nursery teardown when multiple
siblings error simultaneously.
## Update — 2026-04-23 (late, probe iteration 3): hang pinpointed to `wait_for_no_more_peers()`
Further DIAGDEBUG at every milestone in `async_main`
(runtime UP / EXITED service_tn / EXITED root_tn /
FINALLY ENTER / RETURNING) plus `_ForkedProc.wait`
ENTER/RETURNED per-pidfd. Result:
**Every stuck actor reaches `async_main: FINALLY
ENTER` but NOT `async_main: RETURNING`.**
That isolates the hang to a specific await in
`async_main`'s finally block at
`tractor/runtime/_runtime.py:1837+`. The suspect:
```python
# Ensure all peers (actors connected to us as clients) are finished
if ipc_server := actor.ipc_server and ipc_server.has_peers(check_chans=True):
...
await ipc_server.wait_for_no_more_peers() # ← UNBOUNDED, blocks forever
```
`_no_more_peers` is an `Event` set only when
`server._peers` empties (see
`ipc/_server.py:526-530`). If ANY peer-handler is
stuck (the 5 unclosed loops from the earlier pass),
it keeps its channel in `server._peers`, so the
event never fires, so the wait hangs.
### Applied fix (partial, landed as defensive-in-depth)
`tractor/runtime/_runtime.py:1981`
`wait_for_no_more_peers()` call now wrapped in
`trio.move_on_after(3.0)` + a warning log when the
timeout fires. Commented with the full rationale.
**Verified:** with this fix, ALL 15 actors reach
`async_main: RETURNING` cleanly (up from 10/15
reaching end before).
**Unfortunately:** the test still hangs past 45s
total — meaning there's YET ANOTHER unbounded wait
downstream of `async_main`. The bounded
`wait_for_no_more_peers` unblocks one level, but
the cascade has another level above it.
### Candidates for the remaining hang
1. `open_root_actor`'s own finally / post-
`async_main` flow in `_root.py` — specifically
`await actor.cancel(None)` which has its own
internal waits.
2. The `trio.run()` itself doesn't return even
after the root task completes because trio's
nursery still has background tasks running.
3. Maybe `_serve_ipc_eps`'s finally has an await
that blocks when peers aren't clearing.
### Current stance
- Defensive `wait_for_no_more_peers` bound landed
(good hygiene regardless). Revealing a real
deadlock-avoidance gap in tractor's cleanup.
- Test still hangs → skip-mark restored on
`test_nested_multierrors[subint_forkserver]`.
- The full chain of unbounded waits needs another
session of drilling, probably at
`open_root_actor` / `actor.cancel` level.
### Summary of this investigation's wins
1. **FD hygiene fix** (`_close_inherited_fds`) —
correct, closed orphan-SIGINT sibling issue.
2. **pidfd-based `_ForkedProc.wait`** — cancellable,
matches trio_proc pattern.
3. **`_parent_chan_cs` wiring** —
`Actor.cancel()` now breaks the shielded parent-
chan `process_messages` loop.
4. **`wait_for_no_more_peers` bounded** —
prevents the actor-level finally hang.
5. **Ruled-out hypotheses:** tree-kill missing
(wrong), stuck socket recv (wrong).
6. **Pinpointed remaining unknown:** at least one
more unbounded wait in the teardown cascade
above `async_main`. Concrete candidates
enumerated above.
## Update — 2026-04-23 (VERY late): pytest capture pipe IS the final gate
After landing fixes 1-4 and instrumenting every
layer down to `tractor_test`'s `trio.run(_main)`:
**Empirical result: with `pytest -s` the test PASSES
in 6.20s.** Without `-s` (default `--capture=fd`) it
hangs forever.
DIAG timeline for the root pytest PID (with `-s`
implied from later verification):
```
tractor_test: about to trio.run(_main)
open_root_actor: async_main task started, yielding to test body
_main: about to await wrapped test fn
_main: wrapped RETURNED cleanly ← test body completed!
open_root_actor: about to actor.cancel(None)
Actor.cancel ENTER req_chan=False
Actor.cancel RETURN
open_root_actor: actor.cancel RETURNED
open_root_actor: outer FINALLY
open_root_actor: finally END (returning from ctxmgr)
tractor_test: trio.run FINALLY (returned or raised) ← trio.run fully returned!
```
`trio.run()` fully returns. The test body itself
completes successfully (pytest.raises absorbed the
expected `BaseExceptionGroup`). What blocks is
**pytest's own stdout/stderr capture** — under
`--capture=fd` default, pytest replaces the parent
process's fd 1,2 with pipe write-ends it's reading
from. Fork children inherit those pipe fds
(because `_close_inherited_fds` correctly preserves
stdio). High-volume subactor error-log tracebacks
(7+ actors each logging multiple
`RemoteActorError`/`ExceptionGroup` tracebacks on
the error-propagation cascade) fill the 64KB Linux
pipe buffer. Subactor writes block. Subactor can't
progress. Process doesn't exit. Parent's
`_ForkedProc.wait` (now pidfd-based and
cancellable, but nothing's cancelling here since
the test body already completed) keeps the pipe
reader alive... but pytest isn't draining its end
fast enough because test-teardown/fixture-cleanup
is in progress.
**Actually** the exact mechanism is slightly
different: pytest's capture fixture MIGHT be
actively reading, but faster-than-writer subactors
overflow its internal buffer. Or pytest might be
blocked itself on the finalization step.
Either way, `-s` conclusively fixes it.
### Why I ruled this out earlier (and shouldn't have)
Earlier in this investigation I tested
`test_nested_multierrors` with/without `-s` and
both hung. That's because AT THAT TIME, fixes 1-4
weren't all in place yet. The test was hanging at
multiple deeper levels long before reaching the
"generate lots of error-log output" phase. Once
the cascade actually tore down cleanly, enough
output was produced to hit the capture-pipe limit.
**Classic order-of-operations mistake in
debugging:** ruling something out too early based
on a test that was actually failing for a
different reason.
### Fix direction (next session)
Redirect subactor stdout/stderr to `/dev/null` (or
a session-scoped log file) in the fork-child
prelude, right after `_close_inherited_fds()`. This
severs the inherited pytest-capture pipes and lets
subactor output flow elsewhere. Under normal
production use (non-pytest), stdout/stderr would
be the TTY — we'd want to keep that. So the
redirect should be conditional or opt-in via the
`child_sigint`/proc_kwargs flag family.
Alternative: document as a gotcha and recommend
`pytest -s` for any tests using the
`subint_forkserver` backend with multi-level actor
trees. Simpler, user-visible, no code change.
### Current state
- Skip-mark on `test_nested_multierrors[subint_forkserver]`
restored with reason pointing here.
- Test confirmed passing with `-s` after all 4
cascade fixes applied.
- The 4 cascade fixes are NOT wasted — they're
correct hardening regardless of the capture-pipe
issue, AND without them we'd never reach the
"actually produces enough output to fill the
pipe" state.
## Stopgap (landed)
`test_nested_multierrors` skip-marked under
`subint_forkserver` via
`@pytest.mark.skipon_spawn_backend('subint_forkserver',
reason='...')`, cross-referenced to this doc. Mark
should be dropped once the peer-channel-loop exit
issue is fixed.
## References
- `tractor/spawn/_subint_forkserver.py::fork_from_worker_thread`
— the primitive whose post-fork FD hygiene is
probably the culprit.
- `tractor/spawn/_subint_forkserver.py::subint_forkserver_proc`
— the backend function that orchestrates the
graceful cancel path hitting this bug.
- `tractor/spawn/_subint_forkserver.py::_ForkedProc`
— the `trio.Process`-compatible shim; NOT the
failing component (confirmed via thread-dump).
- `tests/test_cancellation.py::test_nested_multierrors`
— the test that surfaced the hang.
- `ai/conc-anal/subint_forkserver_orphan_sigint_hang_issue.md`
— sibling hang class; probably same underlying
fork-FD-inheritance root cause.
- tractor issue #379 — subint backend tracking.

View File

@ -0,0 +1,186 @@
# Revisit `subint_forkserver` thread-cache constraints once msgspec PEP 684 support lands
> **Tracked at:** [#450](https://github.com/goodboy/tractor/issues/450)
Follow-up tracker for cleanup work gated on the msgspec
PEP 684 adoption upstream ([jcrist/msgspec#563](https://github.com/jcrist/msgspec/issues/563)).
Context — why this exists
-------------------------
The `tractor.spawn._subint_forkserver` submodule currently
carries two "non-trio" thread-hygiene constraints whose
necessity is tangled with issues that *should* dissolve
under PEP 684 isolated-mode subinterpreters:
1. `fork_from_worker_thread()` / `run_subint_in_worker_thread()`
internally allocate a **dedicated `threading.Thread`**
rather than using `trio.to_thread.run_sync()`.
2. The test helper is named
`run_fork_in_non_trio_thread()` — the
`non_trio` qualifier is load-bearing today.
This doc catalogs *why* those constraints exist, which of
them isolated-mode would fix, and what the
audit-and-cleanup path looks like once msgspec #563 is
resolved.
The three reasons the constraints exist
---------------------------------------
### 1. GIL-starvation class → fixed by PEP 684 isolated mode
The class-A hang documented in
`subint_sigint_starvation_issue.md` is entirely about
legacy-config subints **sharing the main GIL**. Once
msgspec #563 lands and tractor flips
`tractor.spawn._subint` to
`concurrent.interpreters.create()` (isolated config), each
subint gets its own GIL. Abandoned subint threads can't
contend for main's GIL → can't starve the main trio loop
→ signal-wakeup-pipe drains normally → no SIGINT-drop.
This class of hazard **dissolves entirely**. The
non-trio-thread requirement for *this reason* disappears.
### 2. Destroy race / tstate-recycling → orthogonal; unclear
The `subint_proc` dedicated-thread fix (commit `26fb8206`)
addressed a different issue: `_interpreters.destroy(interp_id)`
was blocking on a trio-cache worker that had run an
earlier `interp.exec()` for that subint. Working
hypothesis at the time was "the cached thread retains the
subint's tstate".
But tstate-handling is **not specific to GIL mode**
`_PyXI_Enter` / `_PyXI_Exit` (the C-level machinery both
configs use to enter/leave a subint from a thread) should
restore the caller's tstate regardless of GIL config. So
isolated mode **doesn't obviously fix this**. It might be:
- A py3.13 bug fixed in later versions — we saw the race
first on 3.13 and never re-tested on 3.14 after moving
to dedicated threads.
- A genuine CPython quirk around cached threads that
exec'd into a subint, persisting across GIL modes.
- Something else we misdiagnosed — the empirical fix
(dedicated thread) worked but the analysis may have
been incomplete.
Only way to know: once we're on isolated mode, empirically
retry `trio.to_thread.run_sync(interp.exec, ...)` and see
if `destroy()` still blocks. If it does, keep the
dedicated thread; if not, one constraint relaxed.
### 3. Fork-from-main-interp-tstate (the constraint in this module's helper names)
The fork-from-main-interp-tstate invariant — CPython's
`PyOS_AfterFork_Child`
`_PyInterpreterState_DeleteExceptMain` gate documented in
`subint_fork_blocked_by_cpython_post_fork_issue.md` — is
about the calling thread's **current** tstate at the
moment `os.fork()` runs. If trio's cache threads never
enter subints at all, their tstate is plain main-interp,
and fork from them would be fine.
The reason the smoke test +
`run_fork_in_non_trio_thread` test helper
currently use a dedicated `threading.Thread` is narrow:
**we don't want to risk a trio cache thread that has
previously been used as a subint driver being the one that
picks up the fork job**. If cached tstate doesn't get
cleared (back to reason #2), the fork's child-side
post-init would see the wrong interp and abort.
In an isolated-mode world where msgspec works:
- `subint_proc` would use the public
`concurrent.interpreters.create()` + `Interpreter.exec()`
/ `Interpreter.close()` — which *should* handle tstate
cleanly (they're the "blessed" API).
- If so, trio's cache threads are safe to fork from
regardless of whether they've previously driven subints.
- → the `non_trio` qualifier in
`run_fork_in_non_trio_thread` becomes
*overcautious* rather than load-bearing, and the
dedicated-thread primitives in `_subint_forkserver.py`
can likely be replaced with straight
`trio.to_thread.run_sync()` wrappers.
TL;DR
-----
| constraint | fixed by isolated mode? |
|---|---|
| GIL-starvation (class A) | **yes** |
| destroy race on cached worker | unclear — empirical test on py3.14 + isolated API required |
| fork-from-main-tstate requirement on worker | **probably yes, conditional on the destroy-race question above** |
If #2 also resolves on py3.14+ with isolated mode,
tractor could drop the `non_trio` qualifier from the fork
helper's name and just use `trio.to_thread.run_sync(...)`
for everything. But **we shouldn't do that preemptively**
— the current cautious design is cheap (one dedicated
thread per fork / per subint-exec) and correct.
Audit plan when msgspec #563 lands
----------------------------------
Assuming msgspec grows `Py_mod_multiple_interpreters`
support:
1. **Flip `tractor.spawn._subint` to isolated mode.** Drop
the `_interpreters.create('legacy')` call in favor of
the public API (`concurrent.interpreters.create()` +
`Interpreter.exec()` / `Interpreter.close()`). Run the
three `ai/conc-anal/subint_*_issue.md` reproducers —
class-A (`test_stale_entry_is_deleted` etc.) should
pass without the `skipon_spawn_backend('subint')` marks
(revisit the marker inventory).
2. **Empirical destroy-race retest.** In `subint_proc`,
swap the dedicated `threading.Thread` back to
`trio.to_thread.run_sync(Interpreter.exec, ...,
abandon_on_cancel=False)` and run the full subint test
suite. If `Interpreter.close()` (or the backing
destroy) blocks the same way as the legacy version
did, revert and keep the dedicated thread.
3. **If #2 clean**, audit `_subint_forkserver.py`:
- Rename `run_fork_in_non_trio_thread` → drop the
`_non_trio_` qualifier (e.g. `run_fork_in_thread`) or
inline the two-line `trio.to_thread.run_sync` call at
the call sites and drop the helper entirely.
- Consider whether `fork_from_worker_thread` +
`run_subint_in_worker_thread` still warrant being
separate module-level primitives or whether they
collapse into a compound
`trio.to_thread.run_sync`-driven pattern inside the
(future) `subint_forkserver_proc` backend.
4. **Doc fallout.** `subint_sigint_starvation_issue.md`
and `subint_cancel_delivery_hang_issue.md` both cite
the legacy-GIL-sharing architecture as the root cause.
Close them with commit-refs to the isolated-mode
migration. This doc itself should get a closing
post-mortem section noting which of #1/#2/#3 actually
resolved vs persisted.
References
----------
- `tractor.spawn._subint_forkserver` — the in-tree module
whose constraints this doc catalogs.
- `ai/conc-anal/subint_sigint_starvation_issue.md` — the
GIL-starvation class.
- `ai/conc-anal/subint_cancel_delivery_hang_issue.md`
sibling Ctrl-C-able hang class.
- `ai/conc-anal/subint_fork_blocked_by_cpython_post_fork_issue.md`
— why fork-from-subint is blocked (this drives the
forkserver-via-non-subint-thread workaround).
- `ai/conc-anal/subint_fork_from_main_thread_smoketest.py`
— empirical validation for the workaround.
- [PEP 684 — per-interpreter GIL](https://peps.python.org/pep-0684/)
- [PEP 734 — `concurrent.interpreters` public API](https://peps.python.org/pep-0734/)
- [jcrist/msgspec#563 — PEP 684 support tracker](https://github.com/jcrist/msgspec/issues/563)
- tractor issue #379 — subint backend tracking.

View File

@ -0,0 +1,350 @@
# `subint` backend: abandoned-subint thread can wedge main trio event loop (Ctrl-C unresponsive)
Follow-up to the Phase B subint spawn-backend PR (see
`tractor.spawn._subint`, issue #379). The hard-kill escape
hatch we landed (`_HARD_KILL_TIMEOUT`, bounded shields,
`daemon=True` driver-thread abandonment) handles *most*
stuck-subint scenarios cleanly, but there's one class of
hang that can't be fully escaped from within tractor: a
still-running abandoned sub-interpreter can starve the
**parent's** trio event loop to the point where **SIGINT is
effectively dropped by the kernel ↔ Python boundary** —
making the pytest process un-Ctrl-C-able.
## Symptom
Running `test_stale_entry_is_deleted[subint]` under
`--spawn-backend=subint`:
1. Test spawns a subactor (`transport_fails_actor`) which
kills its own IPC server and then
`trio.sleep_forever()`.
2. Parent tries `Portal.cancel_actor()` → channel
disconnected → fast return.
3. Nursery teardown triggers our `subint_proc` cancel path.
Portal-cancel fails (dead channel),
`_HARD_KILL_TIMEOUT` fires, driver thread is abandoned
(`daemon=True`), `_interpreters.destroy(interp_id)`
raises `InterpreterError` (because the subint is still
running).
4. Test appears to hang indefinitely at the *outer*
`async with tractor.open_nursery() as an:` exit.
5. `Ctrl-C` at the terminal does nothing. The pytest
process is un-interruptable.
## Evidence
### `strace` on the hung pytest process
```
--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---
write(37, "\2", 1) = -1 EAGAIN (Resource temporarily unavailable)
rt_sigreturn({mask=[WINCH]}) = 140585542325792
```
Translated:
- Kernel delivers `SIGINT` to pytest.
- CPython's C-level signal handler fires and tries to
write the signal number byte (`0x02` = SIGINT) to fd 37
— the **Python signal-wakeup fd** (set via
`signal.set_wakeup_fd()`, which trio uses to wake its
event loop on signals).
- Write returns `EAGAIN`**the pipe is full**. Nothing
is draining it.
- `rt_sigreturn` with the signal masked off — signal is
"handled" from the kernel's perspective but the actual
Python-level handler (and therefore trio's
`KeyboardInterrupt` delivery) never runs.
### Stack dump (via `tractor.devx.dump_on_hang`)
At 20s into the hang, only the **main thread** is visible:
```
Thread 0x...7fdca0191780 [python] (most recent call first):
File ".../trio/_core/_io_epoll.py", line 245 in get_events
File ".../trio/_core/_run.py", line 2415 in run
File ".../tests/discovery/test_registrar.py", line 575 in test_stale_entry_is_deleted
...
```
No driver thread shows up. The abandoned-legacy-subint
thread still exists from the OS's POV (it's still running
inside `_interpreters.exec()` driving the subint's
`trio.run()` on `trio.sleep_forever()`) but the **main
interp's faulthandler can't see threads currently executing
inside a sub-interpreter's tstate**. Concretely: the thread
is alive, holding state we can't introspect from here.
## Root cause analysis
The most consistent explanation for both observations:
1. **Legacy-config subinterpreters share the main GIL.**
PEP 734's public `concurrent.interpreters.create()`
defaults to `'isolated'` (per-interp GIL), but tractor
uses `_interpreters.create('legacy')` as a workaround
for C extensions that don't yet support PEP 684
(notably `msgspec`, see
[jcrist/msgspec#563](https://github.com/jcrist/msgspec/issues/563)).
Legacy-mode subints share process-global state
including the GIL.
2. **Our abandoned subint thread never exits.** After our
hard-kill timeout, `driver_thread.join()` is abandoned
via `abandon_on_cancel=True` and the thread is
`daemon=True` so proc-exit won't block on it — but the
thread *itself* is still alive inside
`_interpreters.exec()`, driving a `trio.run()` that
will never return (the subint actor is in
`trio.sleep_forever()`).
3. **`_interpreters.destroy()` cannot force-stop a running
subint.** It raises `InterpreterError` on any
still-running subinterpreter; there is no public
CPython API to force-destroy one.
4. **Shared-GIL + non-terminating subint thread → main
trio loop starvation.** Under enough load (the subint's
trio event loop iterating in the background, IPC-layer
tasks still in the subint, etc.) the main trio event
loop can fail to iterate frequently enough to drain its
wakeup pipe. Once that pipe fills, `SIGINT` writes from
the C signal handler return `EAGAIN` and signals are
silently dropped — exactly what `strace` shows.
The shielded
`await actor_nursery._join_procs.wait()` at the top of
`subint_proc` (inherited unchanged from the `trio_proc`
pattern) is structurally involved too: if main trio *does*
get a schedule slice, it'd find the `subint_proc` task
parked on `_join_procs` under shield — which traps whatever
`Cancelled` arrives. But that's a second-order effect; the
signal-pipe-full condition is the primary "Ctrl-C doesn't
work" cause.
## Why we can't fix this from inside tractor
- **No force-destroy API.** CPython provides neither a
`_interpreters.force_destroy()` nor a thread-
cancellation primitive (`pthread_cancel` is actively
discouraged and unavailable on Windows). A subint stuck
in pure-Python loops (or worse, C code that doesn't poll
for signals) is structurally unreachable from outside.
- **Shared GIL is the root scheduling issue.** As long as
we're forced into legacy-mode subints for `msgspec`
compatibility, the abandoned-thread scenario is
fundamentally a process-global GIL-starvation window.
- **`signal.set_wakeup_fd()` is process-global.** Even if
we wanted to put our own drainer on the wakeup pipe,
only one party owns it at a time.
## Current workaround
- **Fixture-side SIGINT loop on the `daemon` subproc** (in
this test's `daemon: subprocess.Popen` fixture in
`tests/conftest.py`). The daemon dying closes its end of
the registry IPC, which unblocks a pending recv in main
trio's IPC-server task, which lets the event loop
iterate, which drains the wakeup pipe, which finally
delivers the test-harness SIGINT.
- **Module-level skip on py3.13**
(`pytest.importorskip('concurrent.interpreters')`) — the
private `_interpreters` C module exists on 3.13 but the
multi-trio-task interaction hangs silently there
independently of this issue.
## Path forward
1. **Primary**: upstream `msgspec` PEP 684 adoption
([jcrist/msgspec#563](https://github.com/jcrist/msgspec/issues/563)).
Unlocks `concurrent.interpreters.create()` isolated
mode → per-interp GIL → abandoned subint threads no
longer starve the parent's main trio loop. At that
point we can flip `_subint.py` back to the public API
(`create()` / `Interpreter.exec()` / `Interpreter.close()`)
and drop the private `_interpreters` path.
2. **Secondary**: watch CPython for a public
force-destroy primitive. If something like
`Interpreter.close(force=True)` lands, we can use it as
a hard-kill final stage and actually tear down
abandoned subints.
3. **Harness-level**: document the fixture-side SIGINT
loop pattern as the "known workaround" for subint-
backend tests that can leave background state holding
the main event loop hostage.
## References
- PEP 734 (`concurrent.interpreters`):
<https://peps.python.org/pep-0734/>
- PEP 684 (per-interpreter GIL):
<https://peps.python.org/pep-0684/>
- `msgspec` PEP 684 tracker:
<https://github.com/jcrist/msgspec/issues/563>
- CPython `_interpretersmodule.c` source:
<https://github.com/python/cpython/blob/main/Modules/_interpretersmodule.c>
- `tractor.spawn._subint` module docstring (in-tree
explanation of the legacy-mode choice and its
tradeoffs).
## Reproducer
```
./py314/bin/python -m pytest \
tests/discovery/test_registrar.py::test_stale_entry_is_deleted \
--spawn-backend=subint \
--tb=short --no-header -v
```
Hangs indefinitely without the fixture-side SIGINT loop;
with the loop, the test completes (albeit with the
abandoned-thread warning in logs).
## Additional known-hanging tests (same class)
All three tests below exhibit the same
signal-wakeup-fd-starvation fingerprint (`write() → EAGAIN`
on the wakeup pipe after enough SIGINT attempts) and
share the same structural cause — abandoned legacy-subint
driver threads contending with the main interpreter for
the shared GIL until the main trio loop can no longer
drain its wakeup pipe fast enough to deliver signals.
They're listed separately because each exposes the class
under a different load pattern worth documenting.
### `tests/discovery/test_registrar.py::test_stale_entry_is_deleted[subint]`
Original exemplar — see the **Symptom** and **Evidence**
sections above. One abandoned subint
(`transport_fails_actor`, stuck in `trio.sleep_forever()`
after self-cancelling its IPC server) is sufficient to
tip main into starvation once the harness's `daemon`
fixture subproc keeps its half of the registry IPC alive.
### `tests/test_cancellation.py::test_cancel_while_childs_child_in_sync_sleep[subint-False]`
Cancel a grandchild that's in sync Python sleep from 2
nurseries up. The test's own docstring declares the
dependency: "its parent should issue a 'zombie reaper' to
hard kill it after sufficient timeout" — which for
`trio`/`mp_*` is an OS-level `SIGKILL` of the grandchild
subproc. **Under `subint` there's no equivalent** (no
public CPython API to force-destroy a running
sub-interpreter), so the grandchild's sync-sleeping
`trio.run()` persists inside its abandoned driver thread
indefinitely. The nested actor-tree (parent → child →
grandchild, all subints) means a single cancel triggers
multiple concurrent hard-kill abandonments, each leaving
a live driver thread.
This test often only manifests the starvation under
**full-suite runs** rather than solo execution —
earlier-in-session subint tests also leave abandoned
driver threads behind, and the combined population is
what actually tips main trio into starvation. Solo runs
may stay Ctrl-C-able with fewer abandoned threads in the
mix.
### `tests/test_cancellation.py::test_multierror_fast_nursery[subint-25-0.5]`
Nursery-error-path throughput stress-test parametrized
for **25 concurrent subactors**. When the multierror
fires and the nursery cancels, every subactor goes
through our `subint_proc` teardown. The bounded
hard-kills run in parallel (all `subint_proc` tasks are
sibling trio tasks), so the timeout budget is ~3s total
rather than 3s × 25. After that, **25 abandoned
`daemon=True` driver threads are simultaneously alive** —
an extreme pressure multiplier on the same mechanism.
The `strace` fingerprint is striking under this load: six
or more **successful** `write(16, "\2", 1) = 1` calls
(main trio getting brief GIL slices, each long enough to
drain exactly one wakeup-pipe byte) before finally
saturating with `EAGAIN`:
```
--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---
write(16, "\2", 1) = 1
rt_sigreturn({mask=[WINCH]}) = 140141623162400
--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---
write(16, "\2", 1) = 1
rt_sigreturn({mask=[WINCH]}) = 140141623162400
--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---
write(16, "\2", 1) = 1
rt_sigreturn({mask=[WINCH]}) = 140141623162400
--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---
write(16, "\2", 1) = 1
rt_sigreturn({mask=[WINCH]}) = 140141623162400
--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---
write(16, "\2", 1) = 1
rt_sigreturn({mask=[WINCH]}) = 140141623162400
--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---
write(16, "\2", 1) = 1
rt_sigreturn({mask=[WINCH]}) = 140141623162400
--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---
write(16, "\2", 1) = -1 EAGAIN (Resource temporarily unavailable)
rt_sigreturn({mask=[WINCH]}) = 140141623162400
```
Those successful writes indicate CPython's
`sys.getswitchinterval()`-based GIL round-robin *is*
giving main brief slices — just never long enough to run
the Python-level signal handler through to the point
where trio converts the delivered SIGINT into a
`Cancelled` on the appropriate scope. Once the
accumulated write rate outpaces main's drain rate, the
pipe saturates and subsequent signals are silently
dropped.
The `pstree` below (pid `530060` = hung `pytest`) shows
the subint-driver thread population at the moment of
capture. Even with fewer than the full 25 shown (pstree
truncates thread names to `subint-driver[<interp_id>`
interpreters `3` and `4` visible across 16 thread
entries), the GIL-contender count is more than enough to
explain the starvation:
```
>>> pstree -snapt 530060
systemd,1 --switched-root --system --deserialize=40
└─login,1545 --
└─bash,1872
└─sway,2012
└─alacritty,70471 -e xonsh
└─xonsh,70487 .../bin/xonsh
└─uv,70955 run xonsh
└─xonsh,70959 .../py314/bin/xonsh
└─python,530060 .../py314/bin/pytest -v tests/test_cancellation.py --spawn-backend=subint
├─{subint-driver[3},531857
├─{subint-driver[3},531860
├─{subint-driver[3},531862
├─{subint-driver[3},531866
├─{subint-driver[3},531877
├─{subint-driver[3},531882
├─{subint-driver[3},531884
├─{subint-driver[3},531945
├─{subint-driver[3},531950
├─{subint-driver[3},531952
├─{subint-driver[4},531956
├─{subint-driver[4},531959
├─{subint-driver[4},531961
├─{subint-driver[4},531965
├─{subint-driver[4},531968
└─{subint-driver[4},531979
```
(`pstree` uses `{...}` to denote threads rather than
processes — these are all the **driver OS-threads** our
`subint_proc` creates with name
`f'subint-driver[{interp_id}]'`. Every one of them is
still alive, executing `_interpreters.exec()` inside a
sub-interpreter our hard-kill has abandoned. At 16+
abandoned driver threads competing for the main GIL, the
main-interpreter trio loop gets starved and signal
delivery stalls.)

View File

@ -0,0 +1,273 @@
# `test_register_duplicate_name` racy connect-failure on `daemon` fixture readiness
## Symptom
`tests/test_multi_program.py::test_register_duplicate_name`
fails intermittently under BOTH transports + ALL spawn
backends with connect-refused errors:
```
# under --tpt-proto=uds
FAILED tests/test_multi_program.py::test_register_duplicate_name
- ConnectionRefusedError: [Errno 111] Connection refused
( ^^^ this exc was collapsed from a group ^^^ )
# under --tpt-proto=tcp
FAILED tests/test_multi_program.py::test_register_duplicate_name
- OSError: all attempts to connect to 127.0.0.1:36003 failed
( ^^^ this exc was collapsed from a group ^^^ )
```
Distinct from the cancel-cascade `TooSlowError` flake
class — see
`cancel_cascade_too_slow_under_main_thread_forkserver_issue.md`.
This is a **connect-time race** before the daemon is
fully ready to `accept()`, not a teardown-cascade
slowness.
## Root cause: blind `time.sleep()` in `daemon` fixture
`tests/conftest.py::daemon` boots a sub-py-process via
`subprocess.Popen([python, '-c', 'tractor.run_daemon(...)'])`,
then **blindly sleeps** a fixed delay before yielding
`proc` to the test:
```python
# excerpt from tests/conftest.py::daemon
proc = subprocess.Popen([
sys.executable, '-c', code,
])
bg_daemon_spawn_delay: float = _PROC_SPAWN_WAIT # 0.6
if tpt_proto == 'uds':
bg_daemon_spawn_delay += 1.6
if _non_linux and ci_env:
bg_daemon_spawn_delay += 1
# XXX, allow time for the sub-py-proc to boot up.
# !TODO, see ping-polling ideas above!
time.sleep(bg_daemon_spawn_delay)
assert not proc.returncode
yield proc
```
Inherent fragility: the delay is "long enough on dev
boxes most of the time" but has no actual
synchronization with the daemon's `bind()` + `listen()`
completion. Under any of:
- Loaded box (CI parallelism, big rebuild in
background, low-cpu-freq)
- Cold first-run (`importlib` cache miss, JIT warmup)
- Higher-than-expected `tractor` import cost
- Filesystem latency (UDS sockfile create, slow
tmpfs)
...the sleep finishes BEFORE the daemon has bound its
listen socket → first test client call to
`tractor.find_actor()` / `wait_for_actor()` /
`open_nursery(registry_addrs=[reg_addr])`'s implicit
connect → `ConnectionRefusedError` (TCP) or
`FileNotFoundError`/`ConnectionRefusedError` (UDS).
## Reproducer
Easiest: run the suite under load.
```bash
# create CPU pressure on another core in parallel
stress-ng --cpu 2 --timeout 600s &
./py313/bin/python -m pytest \
tests/test_multi_program.py::test_register_duplicate_name \
--spawn-backend=main_thread_forkserver \
--tpt-proto=tcp -v
```
Reproduces ~30-50% of the time on a dev laptop. On a
quiet idle box, may need 5-10 runs to hit.
## Why the existing `_PROC_SPAWN_WAIT` tuning is
inadequate
Recent `bg_daemon_spawn_delay` rename
(de-monotonic-grow fix) just-shipped removed the
*accumulation* bug where each invocation made the
NEXT test's wait longer too. Net effect: every
invocation now uses the SAME `0.6 + 1.6` (UDS) or
`0.6` (TCP) sleep, no growth. Good — but does
NOTHING for the underlying race. Each individual
test still relies on a blind sleep that may or may
not be sufficient.
Bumping the constant higher pushes flake rate down
but never to zero AND adds dead time to every
non-flaking run. Not a fix, just a knob.
## Side effects
- **Inter-test cascade**: a single failure can cascade
via leaked subprocesses (the `daemon` fixture's
cleanup may not fully tear down a daemon that never
reached "ready"). The `_reap_orphaned_subactors`
session-end + `_track_orphaned_uds_per_test`
per-test fixtures handle most of this now, but the
affected test itself still fails.
- **Worsens under fork-spawn backends**: the daemon
has more init work
(`_main_thread_forkserver`-coordinator-thread
startup, etc.) so the sleep has to cover MORE.
## Fix design — replace blind sleep with active poll
The right primitive is **poll the daemon's bind
address until it accepts a connection or we time
out**, with the timeout being a hard ceiling rather
than a baseline. Two implementation paths:
### Path A — TCP/UDS connect-poll loop
Try `socket.connect(reg_addr)` in a tight loop with
short backoff (~50ms), succeed on the first non-error
return, fail-loud on a hard cap (e.g. 10s). Same
primitive works for both transports because both use
`socket.connect()` semantics.
Rough shape:
```python
def _wait_for_daemon_ready(
reg_addr,
tpt_proto: str,
timeout: float = 10.0,
poll_interval: float = 0.05,
) -> None:
deadline = time.monotonic() + timeout
while True:
if tpt_proto == 'tcp':
sock = socket.socket(socket.AF_INET)
target = reg_addr # (host, port)
else: # uds
sock = socket.socket(socket.AF_UNIX)
target = os.path.join(*reg_addr)
try:
sock.settimeout(poll_interval)
sock.connect(target)
except (
ConnectionRefusedError,
FileNotFoundError,
socket.timeout,
) as exc:
if time.monotonic() >= deadline:
raise TimeoutError(
f'Daemon never accepted on {target!r} '
f'within {timeout}s'
) from exc
time.sleep(poll_interval)
else:
sock.close()
return
```
Pros: trivial primitive, no tractor-runtime
dependency, works pre-yield in the fixture body,
fail-fast on truly-broken daemon.
Cons: doesn't actually do an IPC handshake, just
proves listen-side is up. A daemon that bound but
hasn't initialized its registrar table yet would
still race.
### Path B — `tractor.find_actor()` poll
Use the actual discovery API the test would call:
```python
async def _wait_for_daemon_ready_via_discovery(
reg_addr,
timeout: float = 10.0,
poll_interval: float = 0.05,
):
deadline = trio.current_time() + timeout
async with tractor.open_root_actor(
registry_addrs=[reg_addr],
# ephemeral root just for the probe
):
while True:
try:
async with tractor.find_actor(
'registrar', # daemon's own name
registry_addrs=[reg_addr],
) as portal:
if portal is not None:
return
except Exception:
pass
if trio.current_time() >= deadline:
raise TimeoutError(...)
await trio.sleep(poll_interval)
```
Pros: actually proves the discovery path works,
handles the "bound but not ready" case naturally.
Cons: requires booting an ephemeral root actor JUST
for the probe (overhead), more code, and runs in trio
which complicates the sync-fixture context. Need a
`trio.run()` wrapper.
### Recommended: Path A with optional handshake check
Path A is much simpler + handles 95% of the bug
class. If "bound-but-not-ready" turns out to still
race (it shouldn't — `tractor.run_daemon` doesn't
return from `bind()` until the registrar is
fully populated), escalate to Path B as a focused
follow-up.
## Workarounds (until fix lands)
1. **Bump `_PROC_SPAWN_WAIT`** higher (current: 0.6).
2.03.0 hides most flakes at the cost of adding
dead time to every test. Not a fix but reduces
blast radius while the proper poll lands.
2. **`pytest-rerunfailures`** with `reruns=1` on the
`daemon` fixture's tests specifically. Hides the
flake but doesn't address it.
3. **Mark known-affected tests as `xfail(strict=False)`**
under `--ci`. Lets CI go green at the cost of
silently hiding regressions.
(Recommend skipping all three — implement the active
poll instead.)
## Investigation next steps
1. Implement Path A as a `_wait_for_daemon_ready()`
helper in `tests/conftest.py`. Replace the
`time.sleep(bg_daemon_spawn_delay)` call with it.
2. Drop the `_PROC_SPAWN_WAIT` constant entirely
(active poll obsoletes blind sleep).
3. Run the suite 5-10 times to validate flake rate
drops to 0.
4. If flakes persist, profile whether the daemon
process exits with non-zero before the poll's
deadline hits — that'd be a different bug
(daemon startup crash) that the blind sleep was
masking.
5. Cross-check `tests/test_multi_program.py::test_*`
— multiple tests use the `daemon` fixture; all
should benefit from the same poll primitive.
## Related
- `tests/conftest.py::daemon` — the fixture under
fix
- `tests/conftest.py::_PROC_SPAWN_WAIT` — the
constant to drop
- `cancel_cascade_too_slow_under_main_thread_forkserver_issue.md`
— distinct flake class (cancel-cascade
`TooSlowError` at teardown, not connect-time race)
- `trio_wakeup_socketpair_busy_loop_under_fork_issue.md`
— different bug entirely; this race was masked
pre-WakeupSocketpair-patch by the busy-loop
hangs.

View File

@ -0,0 +1,102 @@
# `trio` 0.29 -> 0.33 slows the depth=3 cancel-cascade
## Symptom
After locking to `trio==0.33.0` (commit `c7741bba`, was
`0.29.0`), this test reliably trips its `fail_after`
deadline on the **`trio`** backend:
```
FAILED tests/test_cancellation.py::test_nested_multierrors[start_method=trio-depth=3]
- AssertionError: assert False
where False = isinstance(
Cancelled(source='deadline', source_task=None, reason=None),
tractor.RemoteActorError,
)
```
A `fail_after_w_trace` hang-snapshot is captured for the
test each run (deadline-injected `Cancelled` wrapped into
the actor-nursery `BaseExceptionGroup`).
## Root cause (immediate)
The test budgets `fail_after(6)` for the `trio` backend.
That 6s was chosen (commit `32955db0`, while `trio==0.29`)
with the assertion that trio finishes "well under" 6s.
The `trio` 0.29 -> 0.33 bump slowed the depth=3 cascade
past that budget, so the 6s deadline now fires mid-cascade.
trio 0.33 added **cancel-reason tracking** — every
`Cancelled` now carries `(source=, reason=, source_task=)`.
The injected exc is `Cancelled(source='deadline')`, i.e.
trio itself naming our `fail_after(6)` scope as the cancel
origin. When that `Cancelled` collapses one branch of the
nursery BEG, the test's `isinstance(subexc,
RemoteActorError)` assertion fails. The healthy outcome is
`BEG = [RemoteActorError, RemoteActorError]`; the
`Cancelled` is purely an artifact of the deadline cutting
the cascade short.
## Measurements (standalone, this machine)
```
depth=1 trio ~3.15s PASS (keeps 6s budget)
depth=3 trio ~6.8-8.2s FAIL @ 6s (now bumped to 12s)
```
depth=1 still fits comfortably; only depth=3 (deeper
recursive spawn-and-error tree => more actors to reap)
exceeds the old budget. The ~2s/depth-level cost looks
like serialized per-actor reap / `terminate_after` waits.
## Mitigation applied
`test_nested_multierrors` now splits the `trio` budget:
```python
case ('trio', 1):
timeout = 6
case ('trio', 3):
timeout = 12 # was 6; see this doc
```
This stops the deadline from firing so the cascade
completes naturally to `[RAE, RAE]`.
## Also affected — same root cause, different test
`test_echoserver_detailed_mechanics[trio-raise_error=KeyboardInterrupt]`
(`tests/test_infected_asyncio.py`) tripped the *same*
slowdown via its much tighter `trio` budget of `1s`. The
single-aio-subactor teardown now takes ~1s, so the `1s`
`fail_after` raced the deadline (PASS at 0.99s / FAIL at
1.03s across back-to-back standalone runs). On a deadline-
fire the injected `Cancelled(source='deadline')` wraps the
mid-stream `KeyboardInterrupt` into a `BaseExceptionGroup`,
which is NOT a `KeyboardInterrupt` so the bare
`pytest.raises(KeyboardInterrupt)` fails. (The sibling
`raise_error=Exception` variant only "passes" by accident:
an `ExceptionGroup` *is-a* `Exception`, so its
`pytest.raises(Exception)` still matches even when wrapped.)
Mitigation: bump that `trio` budget `1 -> 4s` (matching the
forking-spawner case). Without a deadline-fire the KBI
propagates bare and the assertion passes.
## Open follow-up (the actual regression)
The budget bump is a band-aid — the underlying question is
**why** the depth=3 `trio` cancel-cascade went from <6s to
~7-8s across `trio` 0.29 -> 0.33. Candidate avenues:
- which scope owns the per-actor `terminate_after` wait,
and are the tree's reaps concurrent or serialized?
- did trio 0.33's abort/reschedule or cancel-reason
bookkeeping change checkpoint timing on the cancel path?
If/when the cascade speeds back up under-budget, depth=3
will start completing well under 12s — at which point the
budget can be tightened back toward 6s as a regression
tripwire. Related (different backend, same cascade class):
`cancel_cascade_too_slow_under_main_thread_forkserver_issue.md`.

View File

@ -0,0 +1,221 @@
# trio `WakeupSocketpair.drain()` busy-loop in forked child (peer-closed missed-EOF)
## Reproducer
```bash
./py313/bin/python -m pytest \
tests/test_multi_program.py::test_register_duplicate_name \
--tpt-proto=tcp \
--spawn-backend=main_thread_forkserver \
-v --capture=sys
```
Subactor pegs a CPU core indefinitely; parent test
hangs waiting for the subactor.
## Empirical evidence (caught alive)
```
$ sudo strace -p <subactor-pid>
recvfrom(6, "", 65536, 0, NULL, NULL) = 0
recvfrom(6, "", 65536, 0, NULL, NULL) = 0
recvfrom(6, "", 65536, 0, NULL, NULL) = 0
... (no `epoll_wait`, no other syscalls, just this back-to-back)
```
Pattern: tight C-level `recvfrom` loop returning 0
each call. No `epoll_wait` between iterations →
**not trio's task scheduler**. Pure synchronous C
loop.
```
$ sudo readlink /proc/<subactor-pid>/fd/6
socket:[<inode>]
$ sudo lsof -p <subactor-pid> | grep ' 6u'
<cmd> <pid> goodboy 6u unix 0xffff... 0t0 <inode> type=STREAM (CONNECTED)
```
fd=6 is an **AF_UNIX socket** in CONNECTED state.
Even though the test uses `--tpt-proto=tcp`, this fd
is NOT a tractor IPC channel — it's an internal
trio socketpair.
## Root-cause: `WakeupSocketpair.drain()`
`/site-packages/trio/_core/_wakeup_socketpair.py`:
```python
class WakeupSocketpair:
def __init__(self) -> None:
self.wakeup_sock, self.write_sock = socket.socketpair()
self.wakeup_sock.setblocking(False)
self.write_sock.setblocking(False)
...
def drain(self) -> None:
try:
while True:
self.wakeup_sock.recv(2**16)
except BlockingIOError:
pass
```
`socket.socketpair()` on Linux defaults to AF_UNIX
SOCK_STREAM. Both ends non-blocking. Normal flow:
1. Signal/wake event → `write_sock.send(b'\x00')`
queues a byte.
2. `wakeup_sock` becomes readable → trio's epoll
triggers.
3. Trio calls `drain()` to flush the buffer.
4. drain loops on `wakeup_sock.recv(64KB)`.
5. Eventually buffer empty → non-blocking socket
raises `BlockingIOError` → except → break.
**Bug surface — peer-closed missed-EOF**:
Non-blocking socket semantics:
- buffer has data → `recv` returns N>0 bytes (loop continues)
- buffer empty → `recv` raises `BlockingIOError`
- **peer FIN'd → `recv` returns 0 bytes (NEITHER exception NOR
break — infinite tight loop)**
`drain()` does not handle the `b''` return-value
(EOF) case. If `write_sock` has been closed (or the
process holding it is gone), every iteration returns
0 → infinite loop → 100% CPU on a single core.
## Why this triggers under `main_thread_forkserver`
Under `os.fork()` from the forkserver-worker thread:
1. Parent has a `WakeupSocketpair` instance with
`wakeup_sock=fdN`, `write_sock=fdM`. Both fds
open in parent.
2. Fork → child inherits BOTH fds (kernel-level fd
table dup).
3. `_close_inherited_fds()` runs in child →
closes everything except stdio. `wakeup_sock` and
`write_sock` of the parent's `WakeupSocketpair`
ARE closed in child.
4. Child's trio (running fresh) creates its OWN
`WakeupSocketpair` → NEW fd numbers (e.g. fd 6, 7).
5. **In `infect_asyncio` mode** the asyncio loop is
the host; trio runs as guest via
`start_guest_run`. trio still creates its
`WakeupSocketpair` in the I/O manager but its
role is different.
The race window: somewhere between (3) and (5), if a
`WakeupSocketpair` Python object reference inherited
via COW (from parent's pre-fork heap) survives long
enough that `drain()` is called on it AFTER its fds
were closed but BEFORE the child's NEW socketpair
takes over the recycled fd numbers — the recycled fd
will be one of the child's NEW socketpair ends, whose
peer might be FIN-flagged (e.g. parent-process
peer-end is closed).
Or simpler: the `wait_for_actor`/`find_actor` discovery
flow in `test_register_duplicate_name` triggers an
unusual code path where a stale `WakeupSocketpair`
gets `drain()`-called on a fd whose peer has already
closed.
## Why `drain()` shouldn't loop indefinitely on EOF
(upstream trio bug)
Even WITHOUT fork, `drain()` should treat `b''` as
EOF and break. The current code is correct for the
"buffer drained on a healthy socketpair" scenario but
incorrect for the "peer is gone" scenario. It's a
defensive-programming gap in trio.
A one-line patch upstream:
```python
def drain(self) -> None:
try:
while True:
data = self.wakeup_sock.recv(2**16)
if not data:
break # peer-closed; nothing more to drain
except BlockingIOError:
pass
```
## Workarounds (until the underlying issue lands)
1. **Skip-mark on the fork backend**:
`tests/test_multi_program.py`
`pytest.mark.skipon_spawn_backend('main_thread_forkserver',
reason='trio WakeupSocketpair.drain busy-loop, see ai/conc-anal/trio_wakeup_socketpair_busy_loop_under_fork_issue.md')`.
2. **Defensive monkey-patch in tractor's
forkserver-child prelude** — wrap
`WakeupSocketpair.drain` to handle `b''`:
```python
# in `_actor_child_main` or `_close_inherited_fds`'s
# post-fork prelude:
from trio._core._wakeup_socketpair import WakeupSocketpair
_orig_drain = WakeupSocketpair.drain
def _safe_drain(self):
try:
while True:
data = self.wakeup_sock.recv(2**16)
if not data:
return # peer closed
except BlockingIOError:
pass
WakeupSocketpair.drain = _safe_drain
```
Tracks upstream — remove once trio fixes.
3. **Upstream the fix**: 1-line PR to `python-trio/trio`
adding `if not data: break` to `drain()`.
## Investigation next steps
1. **Confirm via py-spy**: when caught alive, detach
strace first then
`sudo py-spy dump --pid <subactor> --locals`. The
busy thread should show `drain` from `WakeupSocketpair`
in the call chain.
2. **Identify which write-end peer is closed**: from
the inode of fd 6, look up the matching peer
inode via `ss -xp` and see whose process it
was/is.
3. **Verify the missed-EOF hypothesis**: hand-craft a
minimal `WakeupSocketpair` repro:
```python
from trio._core._wakeup_socketpair import WakeupSocketpair
ws = WakeupSocketpair()
ws.write_sock.close() # simulate peer-gone
ws.drain() # should hang forever
```
## Sibling bug
`tests/test_infected_asyncio.py::test_aio_simple_error`
hangs under the same backend with a DIFFERENT
fingerprint (Mode-A deadlock, both parties in
`epoll_wait`, no busy-loop). Distinct root cause —
see `infected_asyncio_under_main_thread_forkserver_hang_issue.md`.
Both share the broader theme: **trio internal-state
initialization isn't fully fork-safe under
`main_thread_forkserver`** for the more exotic
dispatch paths.
## See also
- [#379](https://github.com/goodboy/tractor/issues/379) — subint umbrella
- python-trio/trio#1614 — trio + fork hazards
- `trio._core._wakeup_socketpair.WakeupSocketpair`
source (the smoking gun)
- `ai/conc-anal/fork_thread_semantics_execution_vs_memory.md`
- `ai/conc-anal/infected_asyncio_under_main_thread_forkserver_hang_issue.md`

View File

@ -0,0 +1,54 @@
---
model: claude-opus-4-6
service: claude
session: (ad-hoc, not tracked via conf.toml)
timestamp: 2026-04-06T17:28:48Z
git_ref: 02b2ef1
scope: tests
substantive: true
raw_file: 20260406T172848Z_02b2ef1_prompt_io.raw.md
---
## Prompt
User asked to extend `tests/test_resource_cache.py` with a test
that reproduces the edge case fixed in commit `02b2ef18` (per-key
locking+user tracking in `maybe_open_context()`). The bug was
originally triggered in piker's `brokerd.kraken` backend where the
same `acm_func` was called with different kwargs, and the old
global `_Cache.users` counter caused:
- teardown skipped for one `ctx_key` bc another key's users kept
the global count > 0
- re-entry hitting `assert not resources.get(ctx_key)` during the
teardown window
User requested a test that would fail under the old code and pass
with the fix.
## Response summary
Designed and implemented `test_per_ctx_key_resource_lifecycle`
which verifies per-`ctx_key` resource isolation by:
1. Holding resource `'a'` open in a bg task
2. Opening+closing resource `'b'` (same `acm_func`, different
kwargs) while `'a'` is still alive
3. Re-opening `'b'` and asserting cache MISS — proving `'b'` was
torn down independently despite `'a'` keeping its own user
count > 0
With the old global counter, phase 3 would produce a stale cache
HIT (leaked resource) or crash on the assert.
Also added a trivial `acm_with_resource(resource_id)` ACM helper
at module level.
## Files changed
- `tests/test_resource_cache.py` — add `acm_with_resource` ACM +
`test_per_ctx_key_resource_lifecycle` test fn
## Human edits
None — committed as generated (pending user review).

View File

@ -0,0 +1,57 @@
---
model: claude-opus-4-6
service: claude
session: (ad-hoc, not tracked via conf.toml)
timestamp: 2026-04-06T19:31:25Z
git_ref: 85f9c5d
scope: tests
substantive: true
raw_file: 20260406T193125Z_85f9c5d_prompt_io.raw.md
---
## Prompt
User asked to reproduce the exact `assert not resources.get(ctx_key)`
crash originally seen in piker's `brokerd.kraken` backend via
`open_cached_client('kraken')`. Key constraints from user:
- In piker, kwargs were the **same** (empty) so all callers
share one `ctx_key = (fid, ())`
- The root issue is `_Cache.locks` being indexed by `fid`
rather than `ctx_key`, plus the race window between
`values.pop()` and `resources.pop()` in `_Cache.run_ctx`
(the acm `__aexit__` has checkpoints in between)
- Mark the test `@pytest.mark.xfail` so it can be committed
against the unpatched code and later adjusted once the fix
from `9e49eddd` is applied
User pointed to piker's `open_cached_client('kraken')` usage
as the real-world pattern to model.
## Response summary
Added `test_moc_reentry_during_teardown` which deterministically
reproduces the teardown race:
1. A `cached_client()` acm (no params, like
`kraken.api.get_client()`) signals via `in_aexit` event
when its `__aexit__` starts, then sleeps to hold the
window open
2. `use_and_exit` task enters+exits the cached resource
3. `reenter_during_teardown` waits on `in_aexit` then
re-enters — landing exactly in the window where
`values` is popped but `resources` still exists
4. The old `assert not resources.get(ctx_key)` fires
5. `@pytest.mark.xfail(raises=AssertionError)` catches it
Test confirmed: `XFAIL` with the exact traceback at
`_mngrs.py:327`.
## Files changed
- `tests/test_resource_cache.py` — add
`test_moc_reentry_during_teardown` xfail test fn
## Human edits
None — committed as generated (pending user review).

View File

@ -0,0 +1,52 @@
---
model: claude-opus-4-6
service: claude
session: (ad-hoc, not tracked via conf.toml)
timestamp: 2026-04-06T22:26:48Z
git_ref: f086222
scope: code
substantive: true
raw_file: 20260406T222648Z_f086222_prompt_io.raw.md
---
## Prompt
User applied the per-key locking patch (`f086222d`),
removed the `@pytest.mark.xfail` from
`test_moc_reentry_during_teardown`, and found the
test STILL fails — now with `RuntimeError` instead
of `AssertionError`. Asked to diagnose why the fix
doesn't actually resolve this race.
## Response summary
Traced the exact scheduling sequence across three
concurrent tasks to show that the per-`ctx_key`
locking change doesn't close the fundamental gap:
- `maybe_open_context`'s finally pops the lock and
sets `no_more_users` in the **exiting caller's
task**
- But the actual resource cleanup (`values.pop`,
acm `__aexit__`, `resources.pop`) runs in the
**`run_ctx` task** inside `service_tn`
- A re-entering task creates a fresh lock (old one
popped), acquires immediately, and hits stale
`resources[ctx_key]` while `run_ctx` is still in
`__aexit__`
Identified three possible fix strategies:
1. Let `run_ctx` own lock release/pop after
`resources.pop()`
2. Add a "teardown complete" event that new callers
wait on
3. Move `resources.pop()` into `run_ctx`'s inner
finally (before acm `__aexit__`)
## Files changed
No code changes — analysis only.
## Human edits
N/A — diagnostic output, no code generated.

View File

@ -0,0 +1,32 @@
---
model: claude-opus-4-6
service: claude
timestamp: 2026-04-06T22:26:48Z
git_ref: f086222
diff_cmd: git diff f086222~1..f086222
---
Here's the problem. The per-key locking doesn't fix this race because the **lock lifetime is decoupled from `run_ctx`'s lifetime**. Trace through what happens:
**`maybe_open_context` finally** (lines 468-495):
1. `users[ctx_key] -= 1` → 0
2. `no_more_users.set()` → wakes `run_ctx` (but it doesn't run yet, just becomes ready)
3. `_Cache.locks.pop(ctx_key)` → **lock is gone**
4. `maybe_open_context` exits
**`run_ctx`** now gets scheduled (lines 237-241):
5. Inner finally: `values.pop(ctx_key)` → values gone
6. `mng.__aexit__()` runs → `in_aexit.set()`, `await trio.sleep(10)` → **CHECKPOINT**
**`reenter_during_teardown`** wakes:
7. Enters `maybe_open_context` → no lock exists for `ctx_key` → creates **fresh** lock, acquires immediately
8. `values[ctx_key]``KeyError`
9. `resources.get(ctx_key)`**still exists!** (step 6 hasn't reached `resources.pop()` in `run_ctx`'s outer finally yet)
10. `raise RuntimeError('Caching resources ALREADY exist?!')`
The core issue: `no_more_users.set()` (step 2) and `locks.pop()` (step 3) happen in the **exiting caller's task**, but the actual resource cleanup (`values.pop`, acm `__aexit__`, `resources.pop`) happens in the **`run_ctx` task** inside `service_tn`. There's no synchronization between "lock removed, new callers welcome" and "run_ctx has actually finished tearing down". The per-`ctx_key` lock change just made the lock more granular — it didn't close this gap.
The fix needs to ensure a new caller can't acquire a lock for `ctx_key` until `run_ctx` has fully exited (including `resources.pop()`). Either:
- **don't pop the lock in `maybe_open_context`** — let `run_ctx` release/pop it after `resources.pop()`
- **wait on a "teardown complete" event** in the cache-miss path when stale `resources` exist
- **pop `resources` before the acm `__aexit__`** (move `resources.pop()` into the inner finally alongside `values.pop()`)

View File

@ -0,0 +1,74 @@
---
model: claude-opus-4-6
service: claude
session: (ad-hoc, not tracked via conf.toml)
timestamp: 2026-04-13T19:21:16Z
git_ref: f851f28
scope: code
substantive: true
---
## Prompt
User provided a detailed implementation plan for
a new `test_tpt_bind_addrs.py` test suite covering
`open_root_actor(tpt_bind_addrs=...)` — the three
runtime code paths in `_root.py:385-450`:
1. Non-registrar, no explicit bind -> random addrs
2. Registrar, no explicit bind -> registry_addrs
3. Explicit bind given -> `wrap_address()` + merge
Plan specified 6 test functions (~10 parametrized
variants), predicted a type-mixing bug at line 446,
and asked for an in-flight fix if confirmed.
## Response summary
Created `tests/discovery/test_tpt_bind_addrs.py`
with 9 collected test variants across 6 functions:
- `test_registrar_root_tpt_bind_addrs` (3 variants:
`bind-eq-reg`, `bind-subset-reg`,
`bind-disjoint-reg`)
- `test_non_registrar_root_tpt_bind_addrs`
(2 variants: `bind-same-bindspace`,
`bind-disjoint`)
- `test_non_registrar_default_random_bind`
(baseline, no explicit bind)
- `test_tpt_bind_addrs_as_maddr_str`
(multiaddr string input)
- `test_registrar_merge_binds_union`
(registrar + disjoint bind -> union)
- `test_open_nursery_forwards_tpt_bind_addrs`
(`open_nursery(**kwargs)` forwarding)
Confirmed and fixed the predicted bug at
`_root.py:446`: the registrar merge path mixed
`Address` objects (`tpt_bind_addrs`) with raw tuples
(`uw_reg_addrs`) inside `set()`, preventing
deduplication and causing double-bind `OSError`.
Fix: wrap `uw_reg_addrs` before the set union:
```python
# before (broken)
tpt_bind_addrs = list(set(
tpt_bind_addrs + uw_reg_addrs
))
# after (fixed)
tpt_bind_addrs = list(set(
tpt_bind_addrs
+ [wrap_address(a) for a in uw_reg_addrs]
))
```
All 9 tests pass after the fix.
## Files changed
- `tests/discovery/test_tpt_bind_addrs.py` (new)
- `tractor/_root.py:446` (bug fix, 1 line)
## Human edits
N/A — pending review.

View File

@ -0,0 +1,50 @@
---
model: claude-opus-4-6
service: claude
session: 76154e65-d8e1-4b5f-9275-0ea45ba7e98a
timestamp: 2026-04-13T20:50:48Z
git_ref: 269d939c
scope: code
substantive: true
raw_file: 20260413T205048Z_269d939c_prompt_io.raw.md
---
## Prompt
Implement a `parse_endpoints()` API in
`tractor.discovery._multiaddr` that lets downstream
projects declare per-actor transport bind addresses
as a service table of actor-name -> multiaddr
strings (loaded from config, e.g. TOML `[network]`
section). Add type aliases `EndpointsTable` and
`ParsedEndpoints`, plus 7 unit tests covering TCP,
UDS, tuples, mixed input, unsupported protocols, and
edge cases.
A detailed implementation plan was provided
specifying insertion points, type signatures, test
names, and verification commands.
## Response summary
Add `parse_endpoints()` function with two supporting
type aliases to `_multiaddr.py`. The function
iterates a `dict[str, list[str|tuple]]` service
table and delegates each entry to the existing
`wrap_address()` helper, which already handles maddr
strings, raw tuples, and pre-wrapped `Address`
objects. Seven pure unit tests (no trio runtime)
validate all input variants and edge cases. All 22
multiaddr tests pass.
## Files changed
- `tractor/discovery/_multiaddr.py` — add
`EndpointsTable`, `ParsedEndpoints` type aliases
and `parse_endpoints()` function after line 133
- `tests/discovery/test_multiaddr.py` — add
`parse_endpoints` import and 7 new test functions
## Human edits
None — committed as generated.

View File

@ -0,0 +1,32 @@
---
model: claude-opus-4-6
service: claude
timestamp: 2026-04-13T20:50:48Z
git_ref: 269d939c
diff_cmd: git diff 269d939c~1..269d939c
---
## Implementation output
### `tractor/discovery/_multiaddr.py`
> `git diff 269d939c~1..269d939c -- tractor/discovery/_multiaddr.py`
Type aliases `EndpointsTable` and `ParsedEndpoints`
plus `parse_endpoints()` helper that wraps each
entry via `wrap_address()`.
### `tests/discovery/test_multiaddr.py`
> `git diff 269d939c~1..269d939c -- tests/discovery/test_multiaddr.py`
Add import of `parse_endpoints` to existing import
block, then 7 new test functions.
### Test run output
```
22 passed, 1 warning in 0.05s
```
All 22 tests pass (15 existing + 7 new).

View File

@ -0,0 +1,38 @@
---
model: claude-opus-4-6
service: claude
session: multiaddr-support-rename-prefer
timestamp: 2026-04-14T16:33:00Z
git_ref: befedc49
scope: code
substantive: true
raw_file: 20260414T163300Z_befedc49_prompt_io.raw.md
---
## Prompt
Create a helper function that determines the best transport given
actor locality (distributed vs same host). Use PID/hostname
comparison for locality detection, apply at registry addr selection
only (not spawn-time).
## Response summary
New `prefer_addr()` + `_is_local_addr()` helpers
in `_api.py` using `socket.getaddrinfo()` and
`ipaddress` for PID/hostname locality detection.
Preference: UDS > local TCP > remote TCP.
Integrated into `query_actor()` and
`wait_for_actor()`. Also changed
`Registrar.find_actor()` to return full addr list
so callers can apply preference.
## Files changed
- `tractor/discovery/_discovery.py``_api.py`
— renamed + added `prefer_addr()`,
`_is_local_addr()`; updated `query_actor()` and
`wait_for_actor()` call sites
- `tractor/discovery/_registry.py`
`Registrar.find_actor()` returns
`list[UnwrappedAddress]|None`

View File

@ -0,0 +1,62 @@
---
model: claude-opus-4-6
service: claude
timestamp: 2026-04-14T16:33:00Z
git_ref: befedc49
diff_cmd: git diff befedc49~1..befedc49
---
### `tractor/discovery/_api.py`
> `git diff befedc49~1..befedc49 -- tractor/discovery/_api.py`
Add `_is_local_addr()` and `prefer_addr()` transport
preference helpers.
#### `_is_local_addr(addr: Address) -> bool`
Determines whether an `Address` is reachable on the
local host:
- `UDSAddress`: always returns `True`
(filesystem-bound, inherently local)
- `TCPAddress`: checks if `._host` is a loopback IP
via `ipaddress.ip_address().is_loopback`, then
falls back to comparing against the machine's own
interface IPs via
`socket.getaddrinfo(socket.gethostname(), None)`
#### `prefer_addr(addrs: list[UnwrappedAddress]) -> UnwrappedAddress`
Selects the "best" transport address from a
multihomed actor's address list. Wraps each
candidate via `wrap_address()` to get typed
`Address` objects, then classifies into three tiers:
1. **UDS** (same-host guaranteed, lowest overhead)
2. **TCP loopback / same-host IP** (local network)
3. **TCP remote** (only option for distributed)
Within each tier, the last-registered (latest) entry
is preferred. Falls back to `addrs[-1]` if no
heuristic matches.
### `tractor/discovery/_registry.py`
> `git diff befedc49~1..befedc49 -- tractor/discovery/_registry.py`
`Registrar.find_actor()` return type broadened from
single addr to `list[UnwrappedAddress]|None` — full
addr list lets callers apply transport preference.
#### Integration
`query_actor()` and `wait_for_actor()` now call
`prefer_addr(addrs)` instead of `addrs[-1]`.
### Verification
All discovery tests pass (13/13 non-daemon).
`test_local.py` and `test_multi_program.py` also
pass (daemon fixture teardown failures are
pre-existing and unrelated).

View File

@ -0,0 +1,101 @@
---
model: claude-opus-4-7[1m]
service: claude
session: subints-spawner-design-kickoff
timestamp: 2026-04-17T03:49:18Z
git_ref: 9703210
scope: docs
substantive: true
raw_file: 20260417T034918Z_9703210_prompt_io.raw.md
---
## Prompt
Drive the "first big boi, from GH issue" task seeded by
`ai/prompt-io/prompts/subints_spawner.md`: design, plan
and implement sub-interpreter (subint) spawn-backend
support per issue #379, including (1) modularizing
`tractor.spawn._spawn` into per-backend submods, (2) a new
`._subint` backend, and (3) harness parametrization via the
existing `--spawn-backend` / `start_method` pytest fixture
in `tractor._testing.pytest`.
Follow-up clarifications from the user (this turn):
1. Pin `<3.15` on this dev branch and feature-gate subint
tests — chose option (a).
2. Split Phase A (modularization) into its own PR first.
3. Defer the `fork()`-via-subint hack to a follow-up.
4. Harness flag is `pytest --spawn-backend <key>` CLI →
`start_method` fixture (prompt file updated to match).
## Response summary
Produced a three-phase plan and a concrete Phase A (pure
modularization) file-split plan for user review; no code
written yet — the green-light to start Phase A was given
in this same turn conditional on logging this prompt-io
entry first.
Phases:
- **A — modularize** `tractor/spawn/_spawn.py` (847 LOC):
keep generic machinery in `_spawn.py`, extract
`trio_proc``spawn/_trio.py`, `mp_proc`
`spawn/_mp.py`. No pin bump.
- **B — `_subint` backend**: bump `pyproject.toml`
`requires-python` upper to `<3.15`; add `'subint'` to
`SpawnMethodKey`; reuse existing UDS transport; shm
escape-hatch deferred.
- **C — harness**: drive the valid-backend tuple in
`tractor/_testing/pytest.py:345-349` from
`typing.get_args(SpawnMethodKey)`; skip subint tests on
Python < 3.14.
Key findings surfaced to the user:
- `pyproject.toml:12` currently pins `<3.14`; PEP 734
`concurrent.interpreters` only ships in 3.14 — the
load-bearing constraint.
- `_testing/pytest.py:345-349` hardcodes valid backends
as a string tuple (`'mp_spawn'`, `'mp_forkserver'`,
`'trio'`) — should be `get_args(SpawnMethodKey)`.
- `_testing/pytest.py:228` already imports
`try_set_start_method` from `tractor.spawn._spawn`
keeping the `_spawn.py` path as the "core" module
avoids breaking external refs during Phase A.
Phase A commit breakdown (each green via `/run-tests`):
1. Drive `_testing/pytest.py` backend-tuple from
`get_args(SpawnMethodKey)` — prep.
2. Extract `trio_proc``spawn/_trio.py`; `_methods`
registry imports it.
3. Extract `mp_proc``spawn/_mp.py`; `_methods` imports
it.
4. Tidy `spawn/__init__.py` docstring for the new layout.
Workflow: Phase A runs in a new worktree via `/open-wkt
spawn_modularize` (branch `wkt/spawn_modularize`) and
lands as its own PR before Phase B/C returns to the
current `claude_bun_opus4.7_subint_spawner_backend`
branch.
## Files changed
None this turn — design/planning only. The user updated
`ai/prompt-io/prompts/subints_spawner.md` themselves to
complete the truncated sentence on line 32 (harness-flag
reference).
Upcoming (Phase A, next turn):
- `tractor/spawn/_spawn.py` — trimmed to "core"
- `tractor/spawn/_trio.py` — new; holds `trio_proc()`
- `tractor/spawn/_mp.py` — new; holds `mp_proc()`
- `tractor/spawn/__init__.py` — docstring refresh
- `tractor/_testing/pytest.py` — backend tuple from
`get_args(SpawnMethodKey)`
## Human edits
None — committed as generated. User completed the
truncated sentence on line 32 of
`ai/prompt-io/prompts/subints_spawner.md` themselves
*before* this prompt-io log was written; that edit was
the user's authoring, not an edit to AI output.

View File

@ -0,0 +1,117 @@
---
model: claude-opus-4-7[1m]
service: claude
timestamp: 2026-04-17T03:49:18Z
git_ref: 9703210
diff_cmd: git diff HEAD~1..HEAD
---
No code was produced in this turn — this is a design/planning
exchange for subint spawner support per issue #379. Captured
verbatim below are the two substantive AI responses, which
together fix the phased plan (A: modularize, B: subint, C:
harness) and the concrete Phase A file split before any
implementation begins.
## Turn 1 — initial triage + gating questions
> Read: issue #379, current `spawn/_spawn.py` (847 lines),
> and pyproject pins.
>
> ## Task summary
> Design/plan/implement subint backend per #379, plus
> modularize `spawn/_spawn.py` into per-backend submods, plus
> harness flag for test parametrization.
>
> ## Key finding
> `pyproject.toml:12` pins `requires-python = ">=3.12, <3.14"`
> — but `concurrent.interpreters` / PEP 734 **only lands in
> 3.14**. This is the load-bearing design question before
> coding.
>
> ## Proposed phasing (incremental commits)
> - **Phase A — modularize** `_spawn.py`: extract `trio_proc`
> → `spawn/_trio.py`, `mp_proc``spawn/_mp.py`, keep
> generic helpers (`hard_kill`/`soft_kill`/`new_proc`
> dispatcher/`SpawnMethodKey`) in a slim core. Tests green
> at each step.
> - **Phase B — `_subint` backend**: extend `SpawnMethodKey`
> with `'subint'`; reuse UDS transport by default; shm is
> escape-hatch follow-up.
> - **Phase C — harness**: add backend-parametrize flag, skip
> subint when Python < 3.14.
>
> ## Gating questions before I start
> 1. **Python pin**: bump `<3.15` and feature-gate subint
> tests (a), require 3.14 outright (b), or backport shim
> (c)? I'd recommend (a).
> 2. **PR strategy**: one branch, commits per phase (current
> `claude_bun_opus4.7_subint_spawner_backend`)? Or split
> Phase A as its own PR first?
> 3. **`fork()`-via-subint hack** from the issue's "Our own
> thoughts" section — v1 or follow-up? I'd defer to
> follow-up.
> 4. **Prompt file line 32 is truncated** — what's the
> existing harness flag name you had in mind ("using the
> existing ___")?
>
> Which of these are already decided vs. open?
## User answers (for the raw record)
1. bump `<3.15`, feature-gate — option (a).
2. split Phase A as its own PR first.
3. defer `fork()`-via-subint hack.
4. harness flag is `--spawn-backend` CLI → `start_method`
fixture, per `tractor._testing._pytest`. Prompt file
updated accordingly.
## Turn 2 — concrete Phase A plan
> Good — prompt file now ends cleanly; harness flag
> confirmed as `--spawn-backend` CLI → `start_method`
> fixture (via `_testing/pytest.py:193,355`). Also noted
> `pytest.py:345-349` hardcodes valid backends as a string
> tuple — `typing.get_args(SpawnMethodKey)` should drive
> that list.
>
> ## Phase A — concrete plan (separate PR)
>
> **File split** in `tractor/spawn/`:
> - `_spawn.py` stays as "core": keeps `SpawnMethodKey`,
> `_spawn_method`, `proc_waiter`, `try_set_start_method`,
> `exhaust_portal`, `cancel_on_completion`, `hard_kill`,
> `soft_kill`, `new_proc` dispatcher, `_methods`
> registry. Kept at this path so `_testing/pytest.py:228`
> and other external imports don't break.
> - `_trio.py``trio_proc()` (lines 448671, ~224 lines)
> - `_mp.py``mp_proc()` (lines 672842, ~171 lines)
> - `_methods` dict imports the two backend fns from the
> new modules.
>
> **Commits** (each green via `/run-tests`):
> 1. Drive `_testing/pytest.py` backend-tuple from
> `get_args(SpawnMethodKey)` — tiny prep.
> 2. Extract `trio_proc``spawn/_trio.py`; `_methods`
> imports it.
> 3. Extract `mp_proc``spawn/_mp.py`; `_methods`
> imports it.
> 4. Tidy `spawn/__init__.py` docstring to describe new
> layout.
>
> No pyproject pin bump in Phase A (that's Phase B).
>
> **Workflow**: I'd run this in a new worktree via
> `/open-wkt spawn_modularize` (branch
> `wkt/spawn_modularize`), then open a PR before returning
> to the current `claude_bun_opus4.7_subint_spawner_backend`
> branch for Phase B/C.
>
> Green-light to `/open-wkt` and start Phase A?
## User final response
> all this first then yes, green light.
— log this `/prompt-io` entry first, then proceed with
Phase A in a worktree.

View File

@ -0,0 +1,86 @@
---
model: claude-opus-4-7[1m]
service: claude
session: subints-spawner-phase-a-impl
timestamp: 2026-04-17T03:58:05Z
git_ref: 61a73ba
scope: code
substantive: true
raw_file: 20260417T035800Z_61a73ba_prompt_io.raw.md
---
## Prompt
Follow-up from the design session logged in
`20260417T034918Z_9703210_prompt_io.md`: the user
greenlit Phase A after `/prompt-io` tracking, so
execute the Phase A "modularize `spawn._spawn`" plan
end-to-end in a new worktree — four logical steps
(pytest harness prep, `trio_proc` extraction,
`mp_proc` extraction, docstring refresh), each green
via `/run-tests`.
User directives during execution:
- **Option 3** for the test cadence ("continue A.2A.4
first, then run the full suite once at the end of
Phase A").
- **One commit** for the whole phase ("can't we just
commit the whole patch in one commit?") instead of
the 3/4-commit split I initially proposed.
- **Don't pre-draft** commit messages — wait for the
user to invoke `/commit-msg` (captured as feedback
memory `feedback_no_auto_draft_commit_msgs.md`).
## Response summary
Produced the cohesive Phase A modularization patch,
landed as commit `61a73bae` (subject: `Mv
trio_proc`/`mp_proc` to per-backend submods`). Five
files changed, +565 / -418 lines.
Key pieces of the patch (generated by claude,
reviewed by the human before commit):
- `tractor/spawn/_trio.py`**new**; receives
`trio_proc()` verbatim from `_spawn.py`; imports
cross-backend helpers back from `._spawn`.
- `tractor/spawn/_mp.py`**new**; receives
`mp_proc()` verbatim; uses `from . import _spawn`
for late-binding access to the mutable `_ctx` /
`_spawn_method` globals (design decision made
during impl, not the original plan).
- `tractor/spawn/_spawn.py` — shrunk 847 → 448 LOC;
import pruning; bottom-of-module late imports for
`trio_proc` / `mp_proc` with a one-line comment
explaining the circular-dep reason.
- `tractor/spawn/__init__.py` — docstring refresh
describing the new layout.
- `tractor/_testing/pytest.py` — the valid-backend
set now comes from `typing.get_args(SpawnMethodKey)`
so future additions (`'subint'`) don't need harness
edits.
## Files changed
See `git diff 61a73ba~1..61a73ba --stat`:
```
tractor/_testing/pytest.py | 12 +-
tractor/spawn/__init__.py | 31 +++-
tractor/spawn/_mp.py | 235 ++++++++++++++++++++++++
tractor/spawn/_spawn.py | 413 +-------------------------------
tractor/spawn/_trio.py | 292 ++++++++++++++++++++++++++++
5 files changed, 565 insertions(+), 418 deletions(-)
```
Validation:
- import probe + `_methods` wiring check — OK
- spawn-relevant test subset — 37 passed, 1 skipped
- full suite — 350 passed, 14 skipped, 7 xfailed, 1
xpassed
## Human edits
None — committed as generated by claude (no diff
between `.claude/git_commit_msg_LATEST.md` and the
committed body, as far as the assistant could
observe).

View File

@ -0,0 +1,138 @@
---
model: claude-opus-4-7[1m]
service: claude
timestamp: 2026-04-17T03:58:05Z
git_ref: 61a73ba
diff_cmd: git diff 61a73ba~1..61a73ba
---
Code generated in this turn was committed verbatim as
`61a73bae` ("Mv `trio_proc`/`mp_proc` to per-backend
submods"). Per diff-ref mode, per-file code is captured
via the pointers below, each followed by a prose
summary of what the AI generated. Non-code output
(sanity-check results, design rationale) is included
verbatim.
## Per-file generated content
### `tractor/spawn/_trio.py` (new, 292 lines)
> `git diff 61a73ba~1..61a73ba -- tractor/spawn/_trio.py`
Pure lift-and-shift of `trio_proc()` out of
`tractor/spawn/_spawn.py` (previously lines 448670).
Added AGPL header + module docstring describing the
backend; imports include local `from ._spawn import
cancel_on_completion, hard_kill, soft_kill` which
creates the bottom-of-module late-import pattern in
the core file to avoid a cycle. All call sites,
log-format strings, and body logic are byte-identical
to the originals — no semantic change.
### `tractor/spawn/_mp.py` (new, 235 lines)
> `git diff 61a73ba~1..61a73ba -- tractor/spawn/_mp.py`
Pure lift-and-shift of `mp_proc()` out of
`tractor/spawn/_spawn.py` (previously lines 672842).
Same AGPL header convention. Key difference from
`_trio.py`: uses `from . import _spawn` (module
import, not from-import) for `_ctx` and
`_spawn_method` references — these are mutated at
runtime by `try_set_start_method()`, so late binding
via `_spawn._ctx` / `_spawn._spawn_method` is required
for correctness. Also imports `cancel_on_completion`,
`soft_kill`, `proc_waiter` from `._spawn`.
### `tractor/spawn/_spawn.py` (modified, 847 → 448 LOC)
> `git diff 61a73ba~1..61a73ba -- tractor/spawn/_spawn.py`
- removed `trio_proc()` body (moved to `_trio.py`)
- removed `mp_proc()` body (moved to `_mp.py`)
- pruned imports now unused in core: `sys`,
`is_root_process`, `current_actor`,
`is_main_process`, `_mp_main`, `ActorFailure`,
`pretty_struct`, `_pformat`
- added bottom-of-file late imports
`from ._trio import trio_proc` and
`from ._mp import mp_proc` with a one-line
comment explaining why (circular dep)
- `_methods` dict unchanged structurally; still binds
`'trio' → trio_proc`, `'mp_spawn' → mp_proc`,
`'mp_forkserver' → mp_proc`
### `tractor/spawn/__init__.py` (modified)
> `git diff 61a73ba~1..61a73ba -- tractor/spawn/__init__.py`
Docstring rewrite only — no code. New text describes
the per-backend `Layout` with `- ._spawn` (core),
`- ._trio` (default), `- ._mp` (two mp variants), and
reiterates the "no eager import" NOTE.
### `tractor/_testing/pytest.py` (modified)
> `git diff 61a73ba~1..61a73ba -- tractor/_testing/pytest.py`
Two-line logical change (plus a comment):
- `from typing import` grows `get_args`
- `pytest_generate_tests()` replaces the hardcoded
`('mp_spawn','mp_forkserver','trio')` tuple with
`get_args(SpawnMethodKey)` via lazy import
`from tractor.spawn._spawn import SpawnMethodKey`
inside the function body (matching the project's
existing lazy-import convention for the spawn
module)
## Non-code output (verbatim)
### Design rationale for the file split
From my Phase A plan turn (already captured in
`20260417T034918Z_9703210_prompt_io.md`):
- `_spawn.py` stays as the "core" rather than renamed
to `_core.py` so external imports
(`_testing/pytest.py:228` imports
`try_set_start_method` from `tractor.spawn._spawn`)
keep working without churn.
- Per-backend extraction chosen over alternatives
(e.g. splitting generic helpers further) because
the immediate motivation is hosting a 3rd
`_subint.py` sibling cleanly in Phase B.
### Sanity-check output (verbatim terminal excerpts)
Post-extraction import probe:
```
extraction OK
_methods: {'trio': 'tractor.spawn._trio.trio_proc',
'mp_spawn': 'tractor.spawn._mp.mp_proc',
'mp_forkserver': 'tractor.spawn._mp.mp_proc'}
```
Spawn-relevant test subset (`tests/test_local.py
test_rpc.py test_spawning.py test_multi_program.py
test_discovery.py`):
```
37 passed, 1 skipped, 14 warnings in 55.37s
```
Full suite:
```
350 passed, 14 skipped, 7 xfailed, 1 xpassed,
151 warnings in 437.73s (0:07:17)
```
No regressions vs. `main`. One transient `-x`
early-stop `ERROR` on
`test_close_channel_explicit_remote_registrar[trio-True]`
was flaky (passed solo, passed without `-x`), not
caused by this refactor.
### Commit message
Also AI-drafted (via `/commit-msg`) — the 40-line
message on commit `61a73bae` itself. Not reproduced
here; see `git log -1 61a73bae`.

View File

@ -0,0 +1,97 @@
---
model: claude-opus-4-7[1m]
service: claude
session: subints-spawner-phase-b1-impl
timestamp: 2026-04-17T12:44:37Z
git_ref: 5cd6df5
scope: code
substantive: true
raw_file: 20260417T124437Z_5cd6df5_prompt_io.raw.md
---
## Prompt
Continuing the Phase B work from the design sessions
logged in `20260417T034918Z_9703210_prompt_io.md` and
the Phase A impl in `20260417T035800Z_61a73ba_...`.
User direction at this step:
- "ok now let's continue with the subints part of the
plan ya?" → kick off Phase B.
- After I proposed a 4-sub-phase breakdown (B.1
scaffolding, B.2 min-viable spawn, B.3 lifecycle,
B.4 test enablement), user: "ok can we continue
with B" → approving the phasing and design
defaults (thread-per-subint, UDS default,
code-string bootstrap via `interp.exec()`).
- User switched the worktree to
`subint_spawner_backend` (instead of opening a
fresh worktree as I'd proposed): "i already
switched branches in the wkt, you should work off
`subint_spawner_backend` now".
Scope of this turn: **B.1 scaffolding only**
wire the `'subint'` name through the spawn-method
registry + harness, add a feature-detected stub
module, bump pyproject. No actual spawn flow.
## Response summary
Produced the Phase B.1 scaffolding patch, landed as
commit `5cd6df58` (subject: `Add `'subint'` spawn
backend scaffold (#379)`). Four files changed,
+124 / -2 lines.
Key pieces (all generated by claude, reviewed by
human before commit):
- `tractor/spawn/_subint.py`**new**; feature-
detects `concurrent.interpreters`; `subint_proc()`
stub raises `RuntimeError` on py<3.14 or
`NotImplementedError` with issue-#379 URL on
py≥3.14. Signature mirrors `trio_proc`/`mp_proc`
so B.2 can drop the impl in without touching
`_methods`.
- `tractor/spawn/_spawn.py` — adds `'subint'` to
`SpawnMethodKey`, grows a `case 'subint'` arm in
`try_set_start_method()` with feature-gate, re-
imports `sys` for the gate-error msg, adds late
`from ._subint import subint_proc` import and
`_methods` entry.
- `tractor/_testing/pytest.py` — converts the
gate-error into `pytest.UsageError` via a
`try/except` around `try_set_start_method()` so
`--spawn-backend=subint` on py<3.14 prints a
clean banner instead of a traceback.
- `pyproject.toml` — pin `requires-python` `<3.14`
`<3.15`, add `3.14` trove classifier.
## Files changed
See `git diff 5cd6df5~1..5cd6df5 --stat`:
```
pyproject.toml | 3 +-
tractor/_testing/pytest.py | 8 +++-
tractor/spawn/_spawn.py | 15 ++++++
tractor/spawn/_subint.py | 100 +++++++++++++++++++++++++
4 files changed, 124 insertions(+), 2 deletions(-)
```
Validation on py3.13:
- registry/gate probe — OK; `_has_subints = False`,
`subint_proc()` raises `RuntimeError` as expected.
- `try_set_start_method('subint')` — raises cleanly.
- `pytest --spawn-backend=subint` — exits with
`pytest.UsageError` banner.
- Spawn-relevant test subset — 69 passed, 1 skipped.
- Full suite on py3.13 deferred (scaffolding is
no-op on <3.14).
- Full-suite on py3.14 pending user setting up the
venv.
## Human edits
None — committed as generated by claude. `uv.lock`
was intentionally left unstaged by the user (pin-
bump regenerated cp314 wheel entries but they were
deferred to a follow-up).

View File

@ -0,0 +1,168 @@
---
model: claude-opus-4-7[1m]
service: claude
timestamp: 2026-04-17T12:44:37Z
git_ref: 5cd6df5
diff_cmd: git diff 5cd6df5~1..5cd6df5
---
Code generated in this turn was committed verbatim as
`5cd6df58` ("Add `'subint'` spawn backend scaffold
(#379)"). Per diff-ref mode, per-file code is captured
via the pointers below, each followed by a prose
summary. Non-code output (sanity-check results,
design rationale) is included verbatim.
## Per-file generated content
### `tractor/spawn/_subint.py` (new, 100 lines)
> `git diff 5cd6df5~1..5cd6df5 -- tractor/spawn/_subint.py`
New scaffolding module for the PEP 734 subinterpreter
backend. Contents:
- AGPL header + module docstring (describes backend
intent, 3.14+ availability gate, and explicit
"SCAFFOLDING STUB" status pointing at issue #379).
- Top-level `try/except ImportError` wrapping
`from concurrent import interpreters as
_interpreters` → sets module-global
`_has_subints: bool`. This lets the registry stay
introspectable on py<3.14 while spawn-time still
fails cleanly.
- `subint_proc()` coroutine with signature matching
`trio_proc`/`mp_proc` exactly (same param names,
defaults, and `TaskStatus[Portal]` typing) —
intentional so Phase B.2 can drop the impl in
without touching `_methods` or changing call-site
binding.
- Body raises `RuntimeError` on py<3.14 (with
`sys.version` printed) or `NotImplementedError`
with issue-#379 URL on py≥3.14.
### `tractor/spawn/_spawn.py` (modified, +15 LOC)
> `git diff 5cd6df5~1..5cd6df5 -- tractor/spawn/_spawn.py`
- `import sys` re-added (pruned during Phase A, now
needed again for the py-version string in the
`'subint'` gate-error).
- `SpawnMethodKey = Literal[...]` grows `'subint'` as
the 4th member, with inline comment `# py3.14+ via
`concurrent.interpreters` (PEP 734)`.
- `try_set_start_method()` match-block grows a new
`case 'subint':` arm that imports
`from ._subint import _has_subints` lazily and
raises `RuntimeError` with a multi-line gate msg
if unavailable.
- Bottom-of-module late-import section grows
`from ._subint import subint_proc` alongside the
existing `_trio` / `_mp` imports.
- `_methods` dict grows `'subint': subint_proc`.
### `tractor/_testing/pytest.py` (modified, +8 LOC)
> `git diff 5cd6df5~1..5cd6df5 -- tractor/_testing/pytest.py`
`pytest_configure()` wraps the
`try_set_start_method(backend)` call in a
`try/except RuntimeError` that re-raises as
`pytest.UsageError(str(err))`. Rationale: the gate
error on py<3.14 is legitimately a configuration
problem, not a test failure, so pytest's UsageError
path (exit code 4) gives a clean single-line banner
instead of a traceback.
### `pyproject.toml` (modified, +2 / -1)
> `git diff 5cd6df5~1..5cd6df5 -- pyproject.toml`
- `requires-python` pin relaxed `>=3.12, <3.14`
`>=3.12, <3.15` to admit 3.14 as a valid target.
- Added `"Programming Language :: Python :: 3.14"`
to the trove classifiers.
## Non-code output (verbatim)
### Design choices captured in the prior turn
(Previously confirmed by the user on April 17 in this
session — relevant excerpts captured here for
provenance since they drove the shape of the impl):
1. **Python pin**: `<3.15` + feature-gate the subint
backend at spawn time — user answered option (a)
from the design triage.
2. **Deferred `fork()`-via-subint hack** from issue
#379's "Our own thoughts" section.
3. **Phase B phasing**: user approved the B.1 / B.2 /
B.3 / B.4 breakdown — this commit is strictly B.1
(scaffolding only, no spawn-flow impl).
4. **Option (B) worktree strategy**: new worktree
branched from `wkt/spawn_modularize`. *(Amended by
user at runtime: user switched the existing
`spawn_modularize` worktree to the
`subint_spawner_backend` branch instead.)*
### Sanity-check output (verbatim terminal excerpts)
Registry / feature-gate verification on py3.13:
```
SpawnMethodKey values: ('trio', 'mp_spawn',
'mp_forkserver', 'subint')
_methods keys: ['trio', 'mp_spawn',
'mp_forkserver', 'subint']
_has_subints: False (py version: (3, 13) )
[expected] RuntimeError: The 'subint' spawn backend
requires Python 3.14+ (stdlib
`concurrent.interpreters`, PEP 734).
```
`try_set_start_method('subint')` gate on py3.13:
```
[expected] RuntimeError: Spawn method 'subint'
requires Python 3.14+ (stdlib
`concurrent.interpreters`, PEP 734).
```
Pytest `--spawn-backend=subint` on py3.13 (the new
UsageError wrapper kicking in):
```
ERROR: Spawn method 'subint' requires Python 3.14+
(stdlib `concurrent.interpreters`, PEP 734).
Current runtime: 3.13.11 (main, Dec 5 2025,
16:06:33) [GCC 15.2.0]
```
Collection probe: `404 tests collected in 0.18s`
(no import errors from the new module).
Spawn-relevant test subset (`tests/test_local.py
test_rpc.py test_spawning.py test_multi_program.py
tests/discovery/`):
```
69 passed, 1 skipped, 10 warnings in 61.38s
```
Full suite was **not** run on py3.13 for this commit
— the scaffolding is no-op on <3.14 and full-suite
validation under py3.14 is pending that venv being
set up by the user.
### Commit message
Also AI-drafted (via `/commit-msg`, with the prose
rewrapped through `/home/goodboy/.claude/skills/pr-msg/
scripts/rewrap.py --width 67`) — the 33-line message
on commit `5cd6df58` itself. Not reproduced here; see
`git log -1 5cd6df58`.
### Known follow-ups flagged to user
- **`uv.lock` deferred**: pin-bump regenerated cp314
wheel entries in `uv.lock`, but the user chose to
not stage `uv.lock` for this commit. Warned
explicitly.
- **Phase B.2 needs py3.14 venv** — running the
actual subint impl requires it; user said they'd
set it up separately.

View File

@ -0,0 +1,117 @@
---
model: claude-opus-4-7[1m]
service: claude
session: subints-phase-b2-destroy-race-fix
timestamp: 2026-04-18T04:25:26Z
git_ref: 26fb820
scope: code
substantive: true
raw_file: 20260418T042526Z_26fb820_prompt_io.raw.md
---
## Prompt
Follow-up to Phase B.2 (`5cd6df58`) after the user
observed intermittent mid-suite hangs when running
the tractor test suite under `--spawn-backend=subint`
on py3.14. The specific sequence of prompts over
several turns:
1. User pointed at the `test_context_stream_semantics.py`
suite as the first thing to make run clean under
`--spawn-backend=subint`.
2. After a series of `timeout`-terminated runs that
gave no diagnostic info, user nudged me to stop
relying on `timeout` and get actual runtime
diagnostics ("the suite hangs indefinitely, so i
don't think this `timeout 30` is helping you at
all.."). Switched to
`faulthandler.dump_traceback_later(...)` and a
resource-tracker fixture to rule out leaks.
3. Captured a stack pinning the hang on
`_interpreters.destroy(interp_id)` in the subint
teardown finally block.
4. Proposed dedicated-OS-thread fix. User greenlit.
5. Implemented + verified on-worktree; user needed
to be pointed at the *worktree*'s `./py313` venv
because bare `pytest` was picking up the main
repo's venv (running un-patched `_subint.py`) and
still hanging.
Running theme over the whole exchange: this patch
only closes the *destroy race*. The user and I also
traced through the deeper cancellation story — SIGINT
can't reach subints, legacy-mode shares the GIL,
portal-cancel dies when the IPC channel is already
broken — and agreed the next step is a bounded
hard-kill in `subint_proc`'s teardown plus a
dedicated cancellation test suite. Those land as
separate commits.
## Response summary
Produced the `tractor/spawn/_subint.py` patch landed
as commit `26fb8206` ("Fix subint destroy race via
dedicated OS thread"). One file, +110/-84 LOC.
Mechanism: swap `trio.to_thread.run_sync(_interpreters
.exec, ...)` for a plain `threading.Thread(target=...
, daemon=False)`. The trio thread cache recycles
workers — so the OS thread that ran `_interpreters
.exec()` remained alive in the cache holding a
stale subint tstate, blocking
`_interpreters.destroy()` in the finally indefinitely.
A dedicated one-shot thread exits naturally after
the sync target returns, releasing tstate and
unblocking destroy.
Coordination across the trio↔thread boundary:
- `trio.lowlevel.current_trio_token()` captured at
`subint_proc` entry
- driver thread signals `subint_exited.set()` back
to parent trio via `trio.from_thread.run_sync(...,
trio_token=token)` (synchronous from the thread's
POV; the call returns after trio has run `.set()`)
- `trio.RunFinishedError` swallowed in that path for
the process-teardown case where parent trio already
exited
- teardown `finally` off-loads the sync
`driver_thread.join()` via `to_thread.run_sync` (a
cache thread carries no subint tstate — safe)
## Files changed
See `git diff 26fb820~1..26fb820 --stat`:
```
tractor/spawn/_subint.py | 194 +++++++++++++++++++------------
1 file changed, 110 insertions(+), 84 deletions(-)
```
Validation:
- `test_parent_cancels[chk_ctx_result_before_exit=True-
cancel_method=ctx-child_returns_early=False]`
(the specific test that was hanging for the user)
— passed in 1.06s.
- Full `tests/test_context_stream_semantics.py` under
subint — 61 passed in 100.35s (clean-cache re-run:
100.82s).
- Trio backend regression subset — 69 passed / 1
skipped / 89.19s — no regressions from this change.
## Files changed
Beyond the `_subint.py` patch, the raw log also
records the cancellation-semantics research that
spanned this conversation but did not ship as code
in *this* commit. Preserving it inline under "Non-
code output" because it directly informs the
Phase B.3 hard-kill impl that will follow (and any
upstream CPython bug reports we end up filing).
## Human edits
None — committed as generated. The commit message
itself was also AI-drafted via `/commit-msg` and
rewrapped via the project's `rewrap.py --width 67`
tooling; user landed it without edits.

View File

@ -0,0 +1,220 @@
---
model: claude-opus-4-7[1m]
service: claude
timestamp: 2026-04-18T04:25:26Z
git_ref: 26fb820
diff_cmd: git diff 26fb820~1..26fb820
---
Code generated in this turn was committed as `26fb8206`
("Fix subint destroy race via dedicated OS thread").
Per diff-ref mode, per-file code is captured via the
pointer below; the rest is verbatim non-code output
— test-session diagnostics, CPython-internals
reasoning, and the design synthesis that's worth
keeping alongside the code since it informs the
still-pending Phase B.3 hard-kill work.
## Per-file generated content
### `tractor/spawn/_subint.py` (modified, +110/-84 LOC)
> `git diff 26fb820~1..26fb820 -- tractor/spawn/_subint.py`
Rewrites the subint driver-thread strategy:
- replaces `trio.to_thread.run_sync(_interpreters.exec, ...)`
with a plain `threading.Thread(target=_subint_target,
daemon=False)` so the OS thread truly exits after
`_interpreters.exec()` returns
- captures a `trio.lowlevel.current_trio_token()` at
`subint_proc` entry; the driver thread signals
completion back via `trio.from_thread.run_sync(
subint_exited.set, trio_token=...)`
- swallows `trio.RunFinishedError` in the signal path
for the case where the parent trio loop has already
exited (process teardown)
- in the teardown `finally` off-loads the sync
`driver_thread.join()` call to `trio.to_thread.run_sync`
(a cache thread w/ no subint tstate — so no cache
conflict) to wait for the driver thread to fully
exit before calling `_interpreters.destroy()`
## Non-code output (verbatim) — the CPython-internals research
### What went wrong before this commit
Under `--spawn-backend=subint` on py3.14, most single
tests passed but longer runs hung intermittently. The
position of the hang moved between runs (test #22 on
one run, test #53 on another) suggesting a timing-
dependent race rather than a deterministic bug.
`faulthandler.dump_traceback_later()` eventually
caught a stack with the main thread blocked in
`_interpreters.destroy(interp_id)` at `_subint.py:293`.
Only 2 threads were alive:
- main thread waiting in `_interpreters.destroy()`
- one idle trio thread-cache worker in
`trio._core._thread_cache._work`
No subint was still running (`_interpreters.list_all()`
showed only the main interp). A resource-tracker
pytest fixture confirmed threads/subints did NOT
accumulate across tests — this was not a leak but a
specific "destroy blocks on cached thread w/ stale
tstate" race.
### Why the race exists
`trio.to_thread.run_sync` uses a thread *cache* to
avoid OS-thread creation overhead. When the sync
callable returns, the OS thread is NOT terminated —
it's parked in `_thread_cache._work` waiting for the
next job. CPython's subinterpreter implementation
attaches a **tstate** (thread-state object) to each
OS thread that ever entered a subint via
`_interpreters.exec()`. That tstate is released
lazily — either when the thread picks up a new job
(which re-attaches a new tstate, evicting the old
one) or when the thread truly exits.
`_interpreters.destroy(interp_id)` waits for *all*
tstates associated w/ that subint to be released
before it can proceed. If the cached worker is idle
holding the stale tstate, destroy blocks indefinitely.
Whether the race manifests depends on timing — if
the cached thread happens to pick up another job
quickly, destroy unblocks; if it sits idle, we hang.
### Why a dedicated `threading.Thread` fixes it
A plain `threading.Thread(target=_subint_target,
daemon=False)` runs its target once and exits. When
the target returns, OS-thread teardown (`_bootstrap_inner`
`_bootstrap`) fires and CPython releases the
tstate for that thread. `_interpreters.destroy()`
then has no blocker.
### Diagnostic tactics that actually helped
1. `faulthandler.dump_traceback_later(n, repeat=False,
file=open(path, 'w'))` for captured stack dumps on
hang. Critically, pipe to a `file=` not stderr —
pytest captures stderr weirdly and the dump is
easy to miss.
2. A resource-tracker autouse fixture printing
per-test `threading.active_count()` +
`len(_interpreters.list_all())` deltas → ruled out
leak-accumulation theories quickly.
3. Running the hanging test *solo* vs in-suite —
when solo passes but in-suite hangs, you know
it's a cross-test state-transfer bug rather than
a test-internal bug.
### Design synthesis — SIGINT + subints + SC
The user and I walked through the cancellation
semantics of PEP 684/734 subinterpreters in detail.
Key findings we want to preserve:
**Signal delivery in subints (stdlib limitation).**
CPython's signal machinery only delivers signals
(SIGINT included) to the *main thread of the main
interpreter*. Subints cannot install signal handlers
that will ever fire. This is an intentional design
choice in PEP 684 and not expected to change. For
tractor's subint actors, this means:
- Ctrl-C never reaches a subint directly.
- `trio.run()` running on a worker thread (as we do
for subints) already skips SIGINT handler install
because `signal.signal()` raises on non-main
threads.
- The only cancellation surface into a subint is
our IPC `Portal.cancel_actor()`.
**Legacy-mode subints share the main GIL** (which
our impl uses since `msgspec` lacks PEP 684 support
per `jcrist/msgspec#563`). This means a stuck subint
thread can starve the parent's trio loop during
cancellation — the parent can't even *start* its
teardown handling until the subint yields the GIL.
**Failure modes identified for Phase B.3 audit:**
1. Portal cancel lands cleanly → subint unwinds →
thread exits → destroy succeeds. (Happy path.)
2. IPC channel is already broken when we try to
send cancel (e.g., `test_ipc_channel_break_*`)
→ cancel raises `BrokenResourceError` → subint
keeps running unaware → parent hangs waiting for
`subint_exited`. This is what breaks
`test_advanced_faults.py` under subint.
3. Subint is stuck in non-checkpointing Python code
→ portal-cancel msg queued but never processed.
4. Subint is in a shielded cancel scope when cancel
arrives → delay until shield exits.
**Current teardown has a shield-bug too:**
`trio.CancelScope(shield=True)` wrapping the `finally`
block absorbs Ctrl-C, so even when the user tries
to break out they can't. This is the reason
`test_ipc_channel_break_during_stream[break_parent-...
no_msgstream_aclose]` locks up unkillable.
**B.3 hard-kill fix plan (next commit):**
1. Bound `driver_thread.join()` with
`trio.move_on_after(HARD_KILL_TIMEOUT)`.
2. If it times out, log a warning naming the
`interp_id` and switch the driver thread to
`daemon=True` mode (not actually possible after
start — so instead create as daemon=True upfront
and accept the tradeoff of proc-exit not waiting
for a stuck subint).
3. Best-effort `_interpreters.destroy()`; catch the
`InterpreterError` if the subint is still running.
4. Document that the leak is real and the only
escape hatch we have without upstream cooperation.
**Test plan for Phase B.3:**
New `tests/test_subint_cancellation.py` covering:
- SIGINT at spawn
- SIGINT mid-portal-RPC
- SIGINT during shielded section in subint
- Dead-channel cancel (mirror of `test_ipc_channel_
break_during_stream` minimized)
- Non-checkpointing subint (tight `while True` in
user code)
- Per-test `pytest-timeout`-style bounds so the
tests visibly fail instead of wedging the runner
### Sanity-check output (verbatim terminal excerpts)
Post-fix single-test validation:
```
1 passed, 1 warning in 1.06s
```
(same test that was hanging pre-fix:
`test_parent_cancels[...cancel_method=ctx-...False]`)
Full `tests/test_context_stream_semantics.py`
under subint:
```
61 passed, 1 warning in 100.35s (0:01:40)
```
and a clean-cache re-run:
```
61 passed, 1 warning in 100.82s (0:01:40)
```
No regressions on trio backend (same subset):
```
69 passed, 1 skipped, 3 warnings in 89.19s
```
### Commit msg
Also AI-drafted via `/commit-msg` + `rewrap.py
--width 67`. See `git log -1 26fb820`.

View File

@ -0,0 +1,111 @@
---
model: claude-opus-4-7[1m]
service: claude
session: subint-phase-b-hang-classification
timestamp: 2026-04-20T19:27:39Z
git_ref: HEAD (pre-commit; on branch `subint_spawner_backend`)
scope: docs
substantive: true
raw_file: 20260420T192739Z_5e8cd8b2_prompt_io.raw.md
---
## Prompt
Collab between user (`goodboy`) and `claude` to
disambiguate two distinct hang modes hit during
Phase B subint-spawn-backend bringup (issue #379).
The user ran the failing suites, captured `strace`
evidence on hung pytest pids, and set the framing:
> "These cannot be the same bug. Different fix
> paths. Write them up separately or we'll keep
> conflating them."
Follow-on asks:
- Cross-link each doc from its reproducer test so
a future triager lands on the analysis without
needing to grep `ai/conc-anal/`.
- On `test_stale_entry_is_deleted`: arm
`dump_on_hang(seconds=20, path=...)` so a
regression captures a stack dump (pytest stderr
capture otherwise eats `faulthandler` output).
- Keep `test_stale_entry_is_deleted` un-`skip`ped
so the dump file remains inspectable.
Defer the actual fix for the class-2 hang
(Ctrl-C-able, tractor-side bug) to a later commit
per the incremental-commits preference.
## Response summary
Wrote two new `ai/conc-anal/` classification docs
and cross-linked them from the reproducer tests:
1. **`subint_sigint_starvation_issue.md`** —
class 1: abandoned-legacy-subint thread +
shared GIL starves main trio loop →
signal-wakeup-fd pipe fills → SIGINT silently
dropped (`write() = EAGAIN`). Pytest process
un-Ctrl-C-able. Structurally a CPython limit;
blocked on `msgspec` PEP 684 support
(jcrist/msgspec#563). Reproducer:
`test_stale_entry_is_deleted[subint]`.
2. **`subint_cancel_delivery_hang_issue.md`** —
class 2: parent-side trio task parks on an
orphaned IPC channel after subint teardown;
no clean EOF delivered to waiting receiver.
Ctrl-C-able (main trio loop iterating fine).
OUR bug to fix. Candidate fix: explicit
parent-side channel abort in `subint_proc`'s
hard-kill teardown. Reproducer:
`test_subint_non_checkpointing_child`.
Test-side cross-links:
- `tests/discovery/test_registrar.py`:
`test_stale_entry_is_deleted``trio.run(main)`
wrapped in `dump_on_hang(seconds=20,
path=<per-method-tmp>)`; long inline comment
summarizes `strace` evidence + root-cause chain
and points at both docs.
- `tests/test_subint_cancellation.py`:
`test_subint_non_checkpointing_child` docstring
extended with "KNOWN ISSUE (Ctrl-C-able hang)"
section pointing at the class-2 doc + noting
the class-1 doc is NOT what this test hits.
## Files changed
- `ai/conc-anal/subint_sigint_starvation_issue.md`
— new, 205 LOC
- `ai/conc-anal/subint_cancel_delivery_hang_issue.md`
— new, 161 LOC
- `tests/discovery/test_registrar.py` — +52/-1
(arm `dump_on_hang`, inline-comment cross-link)
- `tests/test_subint_cancellation.py` — +26
(docstring "KNOWN ISSUE" block)
## Human edits
Substantive collab — prose was jointly iterated:
- User framed the two-doc split, set the
classification criteria (Ctrl-C-able vs not),
and provided the `strace` evidence.
- User decided to keep `test_stale_entry_is_deleted`
un-`skip`ped (my initial suggestion was
`pytestmark.skipif(spawn_backend=='subint')`).
- User chose the candidate fix ordering for
class 2 and marked "explicit parent-side channel
abort" as the surgical preferred fix.
- User picked the file naming convention
(`subint_<hang-shape>_issue.md`) over my initial
`hang_class_{1,2}.md`.
- Assistant drafted the prose, aggregated prior-
session root-cause findings from Phase B.2/B.3
bringup, and wrote the test-side cross-linking
comments.
No further mechanical edits expected before
commit; user may still rewrap via
`scripts/rewrap.py` if preferred.

View File

@ -0,0 +1,198 @@
---
model: claude-opus-4-7[1m]
service: claude
timestamp: 2026-04-20T19:27:39Z
git_ref: HEAD (pre-commit; will land on branch `subint_spawner_backend`)
diff_cmd: git diff HEAD~1..HEAD
---
Collab between `goodboy` (user) and `claude` (this
assistant) spanning multiple test-run iterations on
branch `subint_spawner_backend`. The user ran the
failing suites, captured `strace` evidence on the
hung pytest pids, and set the direction ("these are
two different hangs — write them up separately so
we don't re-confuse ourselves later"). The assistant
aggregated prior-session findings (Phase B.2/B.3
bringup) into two classification docs + test-side
cross-links. All prose was jointly iterated; the
user had final say on framing and decided which
candidate fix directions to list.
## Per-file generated content
### `ai/conc-anal/subint_sigint_starvation_issue.md` (new, 205 LOC)
> `git diff HEAD~1..HEAD -- ai/conc-anal/subint_sigint_starvation_issue.md`
Writes up the "abandoned-legacy-subint thread wedges
the parent trio loop" class. Key sections:
- **Symptom**`test_stale_entry_is_deleted[subint]`
hangs indefinitely AND is un-Ctrl-C-able.
- **Evidence** — annotated `strace` excerpt showing
SIGINT delivered to pytest, C-level signal handler
tries to write to the signal-wakeup-fd pipe, gets
`write() = -1 EAGAIN (Resource temporarily
unavailable)`. Pipe is full because main trio loop
isn't iterating often enough to drain it.
- **Root-cause chain** — our hard-kill abandons the
`daemon=True` driver OS thread after
`_HARD_KILL_TIMEOUT`; the subint *inside* that
thread is still running `trio.run()`;
`_interpreters.destroy()` cannot force-stop a
running subint (raises `InterpreterError`); legacy
subints share the main GIL → abandoned subint
starves main trio loop → wakeup-fd fills → SIGINT
silently dropped.
- **Why it's structurally a CPython limit** — no
public force-destroy primitive for a running
subint; the only escape is per-interpreter GIL
isolation, gated on msgspec PEP 684 adoption
(jcrist/msgspec#563).
- **Current escape hatch** — harness-side SIGINT
loop in the `daemon` fixture teardown that kills
the bg registrar subproc, eventually unblocking
a parent-side recv enough for the main loop to
drain the wakeup pipe.
### `ai/conc-anal/subint_cancel_delivery_hang_issue.md` (new, 161 LOC)
> `git diff HEAD~1..HEAD -- ai/conc-anal/subint_cancel_delivery_hang_issue.md`
Writes up the *sibling* hang class — same subint
backend, distinct root cause:
- **TL;DR** — Ctrl-C-able, so NOT the SIGINT-
starvation class; main trio loop iterates fine;
ours to fix.
- **Symptom**`test_subint_non_checkpointing_child`
hangs past the expected `_HARD_KILL_TIMEOUT`
budget even after the subint is torn down.
- **Diagnosis** — a parent-side trio task (likely
a `chan.recv()` in `process_messages`) parks on
an orphaned IPC channel; channel was torn down
without emitting a clean EOF /
`BrokenResourceError` to the waiting receiver.
- **Candidate fix directions** — listed in rough
order of preference:
1. Explicit parent-side channel abort in
`subint_proc`'s hard-kill teardown (surgical;
most likely).
2. Audit `process_messages` to add a timeout or
cancel-scope protection that catches the
orphaned-recv state.
3. Wrap subint IPC channel construction in a
sentinel that can force-close from the parent
side regardless of subint liveness.
### `tests/discovery/test_registrar.py` (modified, +52/-1 LOC)
> `git diff HEAD~1..HEAD -- tests/discovery/test_registrar.py`
Wraps the `trio.run(main)` call at the bottom of
`test_stale_entry_is_deleted` in
`dump_on_hang(seconds=20, path=<per-method-tmp>)`.
Adds a long inline comment that:
- Enumerates variant-by-variant status
(`[trio]`/`[mp_*]` = clean; `[subint]` = hangs
+ un-Ctrl-C-able)
- Summarizes the `strace` evidence and root-cause
chain inline (so a future reader hitting this
test doesn't need to cross-ref the doc to
understand the hang shape)
- Points at
`ai/conc-anal/subint_sigint_starvation_issue.md`
for full analysis
- Cross-links to the *sibling*
`subint_cancel_delivery_hang_issue.md` so
readers can tell the two classes apart
- Explains why it's kept un-`skip`ped: the dump
file is useful if the hang ever returns after
a refactor. pytest stderr capture would
otherwise eat `faulthandler` output, hence the
file path.
### `tests/test_subint_cancellation.py` (modified, +26 LOC)
> `git diff HEAD~1..HEAD -- tests/test_subint_cancellation.py`
Extends the docstring of
`test_subint_non_checkpointing_child` with a
"KNOWN ISSUE (Ctrl-C-able hang)" block:
- Describes the current hang: parent-side orphaned
IPC recv after hard-kill; distinct from the
SIGINT-starvation sibling class.
- Cites `strace` distinguishing signal: wakeup-fd
`write() = 1` (not `EAGAIN`) — i.e. main loop
iterating.
- Points at
`ai/conc-anal/subint_cancel_delivery_hang_issue.md`
for full analysis + candidate fix directions.
- Clarifies that the *other* sibling doc
(SIGINT-starvation) is NOT what this test hits.
## Non-code output
### Classification reasoning (why two docs, not one)
The user and I converged on the two-doc split after
running the suites and noticing two *qualitatively
different* hang symptoms:
1. `test_stale_entry_is_deleted[subint]` — pytest
process un-Ctrl-C-able. Ctrl-C at the terminal
does nothing. Must kill-9 from another shell.
2. `test_subint_non_checkpointing_child` — pytest
process Ctrl-C-able. One Ctrl-C at the prompt
unblocks cleanly and the test reports a hang
via pytest-timeout.
From the user: "These cannot be the same bug.
Different fix paths. Write them up separately or
we'll keep conflating them."
`strace` on the `[subint]` hang gave the decisive
signal for the first class:
```
--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---
write(5, "\2", 1) = -1 EAGAIN (Resource temporarily unavailable)
```
fd 5 is Python's signal-wakeup-fd pipe. `EAGAIN`
on a `write()` of 1 byte to a pipe means the pipe
buffer is full → reader side (main Python thread
inside `trio.run()`) isn't consuming. That's the
GIL-hostage signature.
The second class's `strace` showed `write(5, "\2",
1) = 1` — clean drain — so the main trio loop was
iterating and the hang had to be on the application
side of things, not the kernel-↔-Python signal
boundary.
### Why the candidate fix for class 2 is "explicit parent-side channel abort"
The second hang class has the trio loop alive. A
parked `chan.recv()` that will never get bytes is
fundamentally a tractor-side resource-lifetime bug
— the IPC channel was torn down (subint destroyed)
but no one explicitly raised
`BrokenResourceError` at the parent-side receiver.
The `subint_proc` hard-kill path is the natural
place to add that notification, because it already
knows the subint is unreachable at that point.
Alternative fix paths (blanket timeouts on
`process_messages`, sentinel-wrapped channels) are
less surgical and risk masking unrelated bugs —
hence the preference ordering in the doc.
### Why we're not just patching the code now
The user explicitly deferred the fix to a later
commit: "Document both classes now, land the fix
for class 2 separately so the diff reviews clean."
This matches the incremental-commits preference
from memory.

View File

@ -0,0 +1,155 @@
---
model: claude-opus-4-7[1m]
service: claude
session: subints-phase-b-hardening-and-fork-block
timestamp: 2026-04-22T20:07:23Z
git_ref: 797f57c
scope: code
substantive: true
raw_file: 20260422T200723Z_797f57c_prompt_io.raw.md
---
## Prompt
Session-spanning work on the Phase B `subint` spawn-backend.
Three distinct sub-phases in one log:
1. **Py3.13 gate tightening** — diagnose a reproducible hang
of subint spawn flow under py3.13 (works on py3.14), trace
to a private `_interpreters` module vintage issue, tighten
our feature gate from "`_interpreters` present" to "public
`concurrent.interpreters` present" (i.e. py3.14+).
2. **Test-harness hardening** — add `pytest-timeout` dep, put
`@pytest.mark.timeout(30, method='thread')` on the
three known-hanging subint tests cataloged in
`ai/conc-anal/subint_sigint_starvation_issue.md`. Separately,
code-review the user's in-flight `skipon_spawn_backend`
marker implementation; find four bugs; refactor to use
`item.iter_markers()`.
3. **`subint_fork` prototype → CPython-block finding** — draft
a WIP `subint_fork_proc` backend using a sub-interpreter as
a launchpad for `os.fork()` (to sidestep trio#1614). User
tests on py3.14, hits
`Fatal Python error: _PyInterpreterState_DeleteExceptMain:
not main interpreter`. Walk CPython sources (local clone at
`~/repos/cpython/`) to pinpoint the refusal
(`Modules/posixmodule.c:728` → `Python/pystate.c:1040`).
Revert implementation to a `NotImplementedError` stub in a
new `_subint_fork.py` submodule, document the finding in a
third `conc-anal/` doc with an upstream-report draft for
the CPython issue tracker. Finally, discuss user's proposed
workaround architecture (main-interp worker-thread
forkserver) and draft a standalone smoke-test script for
feasibility validation.
## Response summary
All three sub-phases landed concrete artifacts:
**Sub-phase 1** — `_subint.py` + `_spawn.py` gates + error
messages updated to require py3.14+ via the public
`concurrent.interpreters` module presence check. Module
docstring revised to explain the empirical reason
(py3.13's private `_interpreters` vintage wedges under
multi-trio-task usage even though minimal standalone
reproducers work fine there). Test-module
`pytest.importorskip` likewise switched.
**Sub-phase 2** — `pytest-timeout>=2.3` added to `testing`
dep group. `@pytest.mark.timeout(30, method='thread')`
applied on:
- `tests/discovery/test_registrar.py::test_stale_entry_is_deleted`
- `tests/test_cancellation.py::test_cancel_while_childs_child_in_sync_sleep`
- `tests/test_cancellation.py::test_multierror_fast_nursery`
- `tests/test_subint_cancellation.py::test_subint_non_checkpointing_child`
`method='thread'` documented inline as load-bearing — the
GIL-starvation path that drops `SIGINT` would equally drop
`SIGALRM`, so only a watchdog-thread timeout can reliably
escape.
`skipon_spawn_backend` plugin refactored into a single
`iter_markers`-driven loop in `pytest_collection_modifyitems`
(~30 LOC replacing ~30 LOC of nested conditionals). Four
bugs dissolved: wrong `.get()` key, module-level `pytestmark`
suppressing per-test marks, unhandled `pytestmark = [list]`
form, `pytest.Makr` typo. Marker help text updated to
document the variadic backend-list + `reason=` kwarg
surface.
**Sub-phase 3** — Prototype drafted (then reverted):
- `tractor/spawn/_subint_fork.py` — new dedicated submodule
housing the `subint_fork_proc` stub. Module docstring +
fn docstring explain the attempt, the CPython-level
block, and the reason for keeping the stub in-tree
(documentation of the attempt + starting point if CPython
ever lifts the restriction).
- `tractor/spawn/_spawn.py``'subint_fork'` registered as a
`SpawnMethodKey` literal + in `_methods`, so
`--spawn-backend=subint_fork` routes to a clean
`NotImplementedError` pointing at the analysis doc rather
than an "invalid backend" error.
- `ai/conc-anal/subint_fork_blocked_by_cpython_post_fork_issue.md`
third sibling conc-anal doc. Full annotated CPython
source walkthrough from user-visible
`Fatal Python error` → `Modules/posixmodule.c:728
PyOS_AfterFork_Child()` → `Python/pystate.c:1040
_PyInterpreterState_DeleteExceptMain()` gate. Includes a
copy-paste-ready upstream-report draft for the CPython
issue tracker with a two-tier ask (ideally "make it work",
minimally "cleaner error than `Fatal Python error`
aborting the child").
- `ai/conc-anal/subint_fork_from_main_thread_smoketest.py`
standalone zero-tractor-import CPython-level smoke test
for the user's proposed workaround architecture
(forkserver on a main-interp worker thread). Four
argparse-driven scenarios: `control_subint_thread_fork`
(reproduces the known-broken case as a test-harness
sanity), `main_thread_fork` (baseline), `worker_thread_fork`
(architectural assertion), `full_architecture`
(end-to-end trio-in-subint in forked child). User will
run on py3.14 next.
## Files changed
See `git log 26fb820..HEAD --stat` for the canonical list.
New files this session:
- `tractor/spawn/_subint_fork.py`
- `ai/conc-anal/subint_fork_blocked_by_cpython_post_fork_issue.md`
- `ai/conc-anal/subint_fork_from_main_thread_smoketest.py`
Modified (diff pointers in raw log):
- `tractor/spawn/_subint.py` (py3.14 gate)
- `tractor/spawn/_spawn.py` (`subint_fork` registration)
- `tractor/_testing/pytest.py` (`skipon_spawn_backend` refactor)
- `pyproject.toml` (`pytest-timeout` dep)
- `tests/discovery/test_registrar.py`,
`tests/test_cancellation.py`,
`tests/test_subint_cancellation.py` (timeout marks,
cross-refs to conc-anal docs)
## Human edits
Several back-and-forth iterations with user-driven
adjustments during the session:
- User corrected my initial mis-classification of
`test_cancel_while_childs_child_in_sync_sleep[subint-False]`
as Ctrl-C-able — second strace showed `EAGAIN`, putting
it squarely in class A (GIL-starvation). Re-analysis
preserved in the raw log.
- User independently fixed the `.get(reason)``.get('reason', reason)`
bug in the marker plugin before my review; preserved their
fix.
- User suggested moving the `subint_fork_proc` stub from
the bottom of `_subint.py` into its own
`_subint_fork.py` submodule — applied.
- User asked to keep the forkserver-architecture
discussion as background for the smoke-test rather than
committing to a tractor-side refactor until the smoke
test validates the CPython-level assumptions.
Commit messages in this range (b025c982 … 797f57c) were
drafted via `/commit-msg` + `rewrap.py --width 67`; user
landed them with the usual review.

View File

@ -0,0 +1,343 @@
---
model: claude-opus-4-7[1m]
service: claude
timestamp: 2026-04-22T20:07:23Z
git_ref: 797f57c
diff_cmd: git log 26fb820..HEAD # all session commits since the destroy-race fix log
---
Session-spanning conversation covering the Phase B hardening
of the `subint` spawn-backend and an investigation into a
proposed `subint_fork` follow-up which turned out to be
blocked at the CPython level. This log is a narrative capture
of the substantive turns (not every message) and references
the concrete code + docs the session produced. Per diff-ref
mode the actual code diffs are pointed at via `git log` on
each ref rather than duplicated inline.
## Narrative of the substantive turns
### Py3.13 hang / gate tightening
Diagnosed a reproducible hang of the `subint` backend under
py3.13 (test_spawning tests wedge after root-actor bringup).
Root cause: py3.13's vintage of the private `_interpreters` C
module has a latent thread/subint-interaction issue that
`_interpreters.exec()` silently fails to progress under
tractor's multi-trio usage pattern — even though a minimal
standalone `threading.Thread` + `_interpreters.exec()`
reproducer works fine on the same Python. Empirically
py3.14 fixes it.
Fix (from this session): tighten the `_has_subints` gate in
`tractor.spawn._subint` from "private module importable" to
"public `concurrent.interpreters` present" — which is 3.14+
only. This leaves `subint_proc()` unchanged in behavior (we
still call the *private* `_interpreters.create('legacy')`
etc. under the hood) but refuses to engage on 3.13.
Also tightened the matching gate in
`tractor.spawn._spawn.try_set_start_method('subint')` and
rev'd the corresponding error messages from "3.13+" to
"3.14+" with a sentence explaining why. Test-module
`pytest.importorskip` switched from `_interpreters`
`concurrent.interpreters` to match.
### `pytest-timeout` dep + `skipon_spawn_backend` marker plumbing
Added `pytest-timeout>=2.3` to the `testing` dep group with
an inline comment pointing at the `ai/conc-anal/*.md` docs.
Applied `@pytest.mark.timeout(30, method='thread')` (the
`method='thread'` is load-bearing — `signal`-method
`SIGALRM` suffers the same GIL-starvation path that drops
`SIGINT` in the class-A hang pattern) to the three known-
hanging subint tests cataloged in
`subint_sigint_starvation_issue.md`.
Separately code-reviewed the user's newly-staged
`skipon_spawn_backend` pytest marker implementation in
`tractor/_testing/pytest.py`. Found four bugs:
1. `modmark.kwargs.get(reason)` called `.get()` with the
*variable* `reason` as the dict key instead of the string
`'reason'` — user-supplied `reason=` was never picked up.
(User had already fixed this locally via `.get('reason',
reason)` by the time my review happened — preserved that
fix.)
2. The module-level `pytestmark` branch suppressed per-test
marker handling (the `else:` was an `else:` rather than
independent iteration).
3. `mod_pytestmark.mark` assumed a single
`MarkDecorator` — broke on the valid-pytest `pytestmark =
[mark, mark]` list form.
4. Typo: `pytest.Makr``pytest.Mark`.
Refactored the hook to use `item.iter_markers(name=...)`
which walks function + class + module scopes uniformly and
handles both `pytestmark` forms natively. ~30 LOC replaced
the original ~30 LOC of nested conditionals, all four bugs
dissolved. Also updated the marker help string to reflect
the variadic `*start_methods` + `reason=` surface.
### `subint_fork_proc` prototype attempt
User's hypothesis: the known trio+`fork()` issues
(python-trio/trio#1614) could be sidestepped by using a
sub-interpreter purely as a launchpad — `os.fork()` from a
subint that has never imported trio → child is in a
trio-free context. In the child `execv()` back into
`python -m tractor._child` and the downstream handshake
matches `trio_proc()` identically.
Drafted the prototype at `tractor/spawn/_subint.py`'s bottom
(originally — later moved to its own submod, see below):
launchpad-subint creation, bootstrap code-string with
`os.fork()` + `execv()`, driver-thread orchestration,
parent-side `ipc_server.wait_for_peer()` dance. Registered
`'subint_fork'` as a new `SpawnMethodKey` literal, added
`case 'subint' | 'subint_fork':` feature-gate arm in
`try_set_start_method()`, added entry in `_methods` dict.
### CPython-level block discovered
User tested on py3.14 and saw:
```
Fatal Python error: _PyInterpreterState_DeleteExceptMain: not main interpreter
Python runtime state: initialized
Current thread 0x00007f6b71a456c0 [subint-fork-lau] (most recent call first):
File "<script>", line 2 in <module>
<script>:2: DeprecationWarning: This process (pid=802985) is multi-threaded, use of fork() may lead to deadlocks in the child.
```
Walked CPython sources (local clone at `~/repos/cpython/`):
- **`Modules/posixmodule.c:728` `PyOS_AfterFork_Child()`** —
post-fork child-side cleanup. Calls
`_PyInterpreterState_DeleteExceptMain(runtime)` with
`goto fatal_error` on non-zero status. Has the
`// Ideally we could guarantee tstate is running main.`
self-acknowledging-fragile comment directly above.
- **`Python/pystate.c:1040`
`_PyInterpreterState_DeleteExceptMain()`** — the
refusal. Hard `PyStatus_ERR("not main interpreter")` gate
when `tstate->interp != interpreters->main`. Docstring
formally declares the precondition ("If there is a
current interpreter state, it *must* be the main
interpreter"). `XXX` comments acknowledge further latent
issues within.
Definitive answer to "Open Question 1" of the prototype
docstring: **no, CPython does not support `os.fork()` from
a non-main sub-interpreter**. Not because the fork syscall
is blocked (it isn't — the parent returns a valid pid),
but because the child cannot survive CPython's post-fork
initialization. This is an enforced invariant, not an
incidental limitation.
### Revert: move to stub submod + doc the finding
Per user request:
1. Reverted the working `subint_fork_proc` body to a
`NotImplementedError` stub, MOVED to its own submod
`tractor/spawn/_subint_fork.py` (keeps `_subint.py`
focused on the working `subint_proc` backend).
2. Updated `_spawn.py` to import the stub from the new
submod path; kept `'subint_fork'` in `SpawnMethodKey` +
`_methods` so `--spawn-backend=subint_fork` routes to a
clean `NotImplementedError` with pointer to the analysis
doc rather than an "invalid backend" error.
3. Wrote
`ai/conc-anal/subint_fork_blocked_by_cpython_post_fork_issue.md`
with the full annotated CPython walkthrough + an
upstream-report draft for the CPython issue tracker.
Draft has a two-tier ask: ideally "make it work"
(pre-fork tstate-swap hook or `DeleteExceptFor(interp)`
variant), minimally "give us a clean `RuntimeError` in
the parent instead of a `Fatal Python error` aborting
the child silently".
### Design discussion — main-interp-thread forkserver workaround
User proposed: set up a "subint forking server" that fork()s
on behalf of subint callers. Core insight: the CPython gate
is on `tstate->interp`, not thread identity, so **any thread
whose tstate is main-interp** can fork cleanly. A worker
thread attached to main-interp (never entering a subint)
satisfies the precondition.
Structurally this is `mp.forkserver` (which tractor already
has as `mp_forkserver`) but **in-process**: instead of a
separate Python subproc as the fork server, we'd put the
forkserver on a thread in the tractor parent process. Pros:
faster spawn (no IPC marshalling to external server + no
separate Python startup), inherits already-imported modules
for free. Cons: less crash isolation (forkserver failure
takes the whole process).
Required tractor-side refactor: move the root actor's
`trio.run()` off main-interp-main-thread (so main-thread can
run the forkserver loop). Nontrivial; approximately the same
magnitude as "Phase C".
The design would also not fully resolve the class-A
GIL-starvation issue because child actors' trio still runs
inside subints (legacy config, msgspec PEP 684 pending).
Would mitigate SIGINT-starvation specifically if signal
handling moves to the forkserver thread.
Recommended pre-commitment: a standalone CPython-only smoke
test validating the four assumptions the arch rests on,
before any tractor-side work.
### Smoke-test script drafted
Wrote `ai/conc-anal/subint_fork_from_main_thread_smoketest.py`:
argparse-driven, four scenarios (`control_subint_thread_fork`
reproducing the known-broken case, `main_thread_fork`
baseline, `worker_thread_fork` the architectural assertion,
`full_architecture` end-to-end with trio in a subint in the
forked child). No `tractor` imports; pure CPython + `_interpreters`
+ `trio`. Bails cleanly on py<3.14. Pass/fail banners per
scenario.
User will validate on their py3.14 env next.
## Per-code-artifact provenance
### `tractor/spawn/_subint_fork.py` (new submod)
> `git show 797f57c -- tractor/spawn/_subint_fork.py`
NotImplementedError stub for the subint-fork backend. Module
docstring + fn docstring explain the attempt, the CPython
block, and why the stub is kept in-tree. No runtime behavior
beyond raising with a pointer at the conc-anal doc.
### `tractor/spawn/_spawn.py` (modified)
> `git log 26fb820..HEAD -- tractor/spawn/_spawn.py`
- Added `'subint_fork'` to `SpawnMethodKey` literal with a
block comment explaining the CPython-level block.
- Generalized the `case 'subint':` arm to `case 'subint' |
'subint_fork':` since both use the same py3.14+ gate.
- Registered `subint_fork_proc` in `_methods` with a
pointer-comment at the analysis doc.
### `tractor/spawn/_subint.py` (modified across session)
> `git log 26fb820..HEAD -- tractor/spawn/_subint.py`
- Tightened `_has_subints` gate: dual-requires public
`concurrent.interpreters` + private `_interpreters`
(tests for py3.14-or-newer on the public-API presence,
then uses the private one for legacy-config subints
because `msgspec` still blocks the public isolated mode
per jcrist/msgspec#563).
- Updated module docstring, `subint_proc()` docstring, and
gate-error messages to reflect the 3.14+ requirement and
the reason (py3.13 wedges under multi-trio usage even
though the private module exists there).
### `tractor/_testing/pytest.py` (modified)
> `git log 26fb820..HEAD -- tractor/_testing/pytest.py`
- New `skipon_spawn_backend(*start_methods, reason=...)`
pytest marker expanded into `pytest.mark.skip(reason=...)`
at collection time via
`pytest_collection_modifyitems()`.
- Implementation uses `item.iter_markers(name=...)` which
walks function + class + module scopes uniformly and
handles both `pytestmark = <single Mark>` and
`pytestmark = [mark, ...]` forms natively. ~30-LOC
single-loop refactor replacing a prior nested
conditional that had four bugs (see "Review" narrative
above).
- Added `pytest.Config` / `pytest.Function` /
`pytest.FixtureRequest` type annotations on fixture
signatures while touching the file.
### `pyproject.toml` (modified)
> `git log 26fb820..HEAD -- pyproject.toml`
Added `pytest-timeout>=2.3` to `testing` dep group with
comment pointing at the `ai/conc-anal/` docs.
### `tests/discovery/test_registrar.py`,
`tests/test_subint_cancellation.py`,
`tests/test_cancellation.py` (modified)
> `git log 26fb820..HEAD -- tests/`
Applied `@pytest.mark.timeout(30, method='thread')` on
known-hanging subint tests. Extended comments to cross-
reference the `ai/conc-anal/*.md` docs. `method='thread'`
is documented inline as load-bearing (`signal`-method
SIGALRM suffers the same GIL-starvation path that drops
SIGINT).
### `ai/conc-anal/subint_fork_blocked_by_cpython_post_fork_issue.md` (new)
> `git show 797f57c -- ai/conc-anal/subint_fork_blocked_by_cpython_post_fork_issue.md`
Third sibling doc under `conc-anal/`. Structure: TL;DR,
context ("what we tried"), symptom (the user's exact
`Fatal Python error` output), CPython source walkthrough
with excerpted snippets from `posixmodule.c` +
`pystate.c`, chain summary, definitive answer to Open
Question 1, `## Upstream-report draft (for CPython issue
tracker)` section with a two-tier ask, references.
### `ai/conc-anal/subint_fork_from_main_thread_smoketest.py` (new, THIS turn)
Zero-tractor-import smoke test for the proposed workaround
architecture. Four argparse-driven scenarios covering the
control case + baseline + arch-critical case + end-to-end.
Pass/fail banners per scenario; clean `--help` output;
py3.13 early-exit.
## Non-code output (verbatim)
### The `strace` signature that kicked off the CPython
walkthrough
```
--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---
write(16, "\2", 1) = -1 EAGAIN (Resource temporarily unavailable)
rt_sigreturn({mask=[WINCH]}) = 139801964688928
```
### Key user quotes framing the direction
> ok actually we get this [fatal error] ... see if you can
> take a look at what's going on, in particular wrt to
> cpython's sources. pretty sure there's a local copy at
> ~/repos/cpython/
(Drove the CPython walkthrough that produced the
definitive refusal chain.)
> is there any reason we can't just sidestep this "must fork
> from main thread in main subint" issue by simply ensuring
> a "subint forking server" is always setup prior to
> invoking trio in a non-main-thread subint ...
(Drove the main-interp-thread-forkserver architectural
discussion + smoke-test script design.)
### CPython source tags for quick jump-back
```
Modules/posixmodule.c:728 PyOS_AfterFork_Child()
Modules/posixmodule.c:753 // Ideally we could guarantee tstate is running main.
Modules/posixmodule.c:778 status = _PyInterpreterState_DeleteExceptMain(runtime);
Python/pystate.c:1040 _PyInterpreterState_DeleteExceptMain()
Python/pystate.c:1044-1047 tstate->interp != main → PyStatus_ERR("not main interpreter")
```

View File

@ -0,0 +1,146 @@
---
model: claude-opus-4-7[1m]
service: claude
session: trio-0.33-subproc-supervisor-retroactive
timestamp: 2026-06-01T23:14:29Z
git_ref: 0e3e008b
scope: code
substantive: true
raw_file: 20260601T231429Z_0e3e008b_prompt_io.raw.md
---
## Prompt
**RETROACTIVE LOG** — original session prompts not
preserved; reconstructed from the staged work product.
The work designs a `trio.Nursery.start()`-style wrapper
around `trio.run_process()` for SC-friendly subprocess
supervision. From the resulting code shape, the
prompting intent was:
1. Surface rc!=0 `CalledProcessError` DETERMINISTICALLY,
without the nursery-eg-wrapping that complicates
`collapse_eg()` usage and races the relay reader on
trio's `check=True`-driven cancel cascade.
2. ALWAYS isolate the parent controlling-tty so a
spawned child can't emit terminal control-seqs onto
the launching tty (clobbering scrollback). Default
`stdin=DEVNULL`; default `stdout=DEVNULL` unless
explicitly relayed/overridden; distinguish "caller
passed nothing" from "caller passed `None` for
inherit".
3. Optional live per-line relay of child std-streams to
the `tractor` log — STREAMED (not
buffered-until-exit) so long-lived daemon output is
visible during the run. Pick a custom log level that
shows at usual `info`/`devx` console levels but is
separately filterable.
4. Concurrent pipe-drain reader MANDATORY when piping
without `capture_*` — without it the child blocks on
`write()` once the OS pipe buffer fills (~64KiB),
causing deadlocks on output bursts.
5. Non-blocking `tn.start()` semantics: hand the live
`trio.Process` to the parent immediately;
supervise/relay run to completion in the supervisor
coro.
6. Hermetic `trio`-only unit tests (no actor-runtime)
covering each of: per-line relay, tty isolation,
no-deadlock on >64KiB unnewlined output, CPE
rebuild w/ stderr relay, CPE rebuild on the silent
drain+capture path.
## Response summary
Adds `tractor/trionics/_subproc.py` (296 LOC) +
`tests/trionics/test_subproc.py` (230 LOC) + a
re-export in `tractor/trionics/__init__.py`.
**`supervise_run_process()`** (public, re-exported)
- `check=False` is forced to `trio.run_process`; the
rc-check runs in the supervisor coro AFTER `own_tn`
unwinds (both the child AND the relay readers have
hit EOF + fully drained). A BARE
`subprocess.CalledProcessError` is rebuilt + raised
from there, with `.stderr` bytes passed in the
constructor AND attached as an `add_note()`'d
`|_.stderr:` block for legible teardown logs.
- `stdin=DEVNULL` always. `stdout` default chosen via a
`_UNSET` sentinel: `relay_stdout=True` → PIPE,
explicit `stdout=...` → as given, else `DEVNULL`.
`stderr` defaults to PIPE whenever we relay OR need
the CPE note (when `check=True`), else `DEVNULL`.
- `relay_level='io'` (custom level 21; sorts just
above stdlib `INFO`=20 so it shows at usual
`info`/`devx` levels and stays separately
filterable). `runtime`=15 would silently filter at
default levels, so it's rejected as a default.
- `task_status.started(trio_proc)` delivers the live
process immediately. The internal `own_tn`
supervises `trio.run_process` + any relay readers to
completion.
- `**run_process_kwargs` forward verbatim;
`stdin/stdout/stderr/check` are MANAGED keys
(override on conflict).
- Crash-handling deliberately NOT baked in — compose
`maybe_open_crash_handler()` on top at the call-site.
**`_relay_stream_lines()`** (internal helper)
- Three modes (combinable): `emit`-only (live per-line
relay), `accum`-only (silent drain+capture for a CPE
note), or both (live relay AND capture).
- Per-line split handles cross-chunk residuals via a
rolling `residual` bytes buffer; flushes any trailing
un-newline-term'd line at EOF.
- `async with stream:` ensures aclose at EOF/cancel
(mirrors trio's internal `_subprocess` drain idiom).
**`_add_stderr_note()`** (internal helper)
- `add_note()`s a `textwrap.indent(...)`'d
`|_.stderr:` block onto a `CalledProcessError` for
teardown logs.
**Tests** (5 hermetic, trio-only) — `_capture_relay`
fixture monkeypatches `_subproc.log.<level>` to a list:
- `test_stdout_relayed_per_line`: per-line stdout
relay carries each `line=N` to the records.
- `test_parent_tty_isolated`: `readlink /proc/self/fd/0`
and `fd/1` from the child show `pipe:` (fd1) +
`/dev/null` (fd0); NO `/dev/pts/*`.
- `test_no_deadlock_on_big_unnewlined_output`: 200KiB
of `x` with no newlines completes inside
`fail_after(2)` — exercises the concurrent drain.
- `test_stderr_relay_and_cpe_rebuild`: rc=3 with
`relay_stderr=True` raises bare CPE
(via `collapse_eg()`) with `b'boom' in cpe.stderr`,
the note attached, AND per-line live relay.
- `test_nonrelay_cpe_note`: rc=7 with no relay still
produces CPE with `.stderr` + note via the silent
drain+capture path.
## Files changed
- `tractor/trionics/_subproc.py` — NEW. Public
`supervise_run_process()` + helpers
`_relay_stream_lines()` / `_add_stderr_note()` + the
`_UNSET` sentinel.
- `tests/trionics/test_subproc.py` — NEW. 5 hermetic
trio-only tests + `_capture_relay` monkeypatch
fixture.
- `tractor/trionics/__init__.py` — re-export
`supervise_run_process`.
## Human edits
**RETROACTIVE**: this log is being written from the
staged diff, not from a live session. The code as
staged is the canonical artifact; any human edits the
user made during the originating design session are
already integrated and cannot be separated post-hoc.
The `.raw.md` sibling is a diff-pointer placeholder,
NOT a pre-edit transcript.
Future prompt-io entries for in-flight work should be
written DURING the design session per the skill
contract so the pre-edit `.raw.md` captures the
unedited model output for genuine provenance.

View File

@ -0,0 +1,106 @@
---
model: claude-opus-4-7[1m]
service: claude
timestamp: 2026-06-01T23:14:29Z
git_ref: 0e3e008b
diff_cmd: git diff HEAD~1..HEAD
---
# RETROACTIVE — original model output not preserved
This `.raw.md` would normally contain the verbatim
pre-human-edit response from the design session that
produced the staged `_subproc.py` module + tests. That
session's transcript is not available, so this file
serves as a diff-pointer placeholder + transparency
note.
## Authoritative artifact
The committed code IS the artifact of record. Once the
companion commit lands, the unified diff is:
> `git diff HEAD~1..HEAD -- tractor/trionics/_subproc.py`
> `git diff HEAD~1..HEAD -- tests/trionics/test_subproc.py`
> `git diff HEAD~1..HEAD -- tractor/trionics/__init__.py`
Before committing, substitute `--cached` for the
pre-commit form.
## What is NOT here
Because this is retroactive:
- No verbatim chain-of-thought / discussion prose from
the design session.
- No rejected alternatives the model considered before
arriving at the final shape (e.g. whether the
rc-check should live inside `own_tn` vs after it; the
`_UNSET` sentinel vs a `None`-means-DEVNULL
convention; `io` vs `info` as the default relay
level).
- No pre-edit code blocks as the model first emitted
them, separable from any user cleanup applied before
the diff was staged.
## Inferred design choices visible in the final code
(Documented here because they're the kind of decision
detail an unedited raw transcript would have captured.)
1. **Post-drain rc-check in the supervisor coro body,
AFTER `own_tn.__aexit__`.** Placing the
`CalledProcessError` raise here (not inside
`own_tn`) means the EG-unwrap happens at the OUTER
`tn.start()` boundary — callers do `collapse_eg()`
if they want bare. Doing the raise INSIDE `own_tn`
would cancel the still-draining relay reader
mid-flight and lose stderr lines.
2. **`_UNSET` sentinel for `stdout`.** A plain default
of `None` couldn't distinguish "use the safe
`DEVNULL` default" from "caller explicitly passed
`None` (inherit, presumably knowingly)". The
sentinel keeps the SAFE default while letting power
users opt into inherit.
3. **`relay_level='io'` (custom level 21).** Chosen to
sort just above stdlib `INFO`=20 so a default
`--ll info` shows the relay, but it remains a
distinct level so users can filter
`tractor.trionics:io` separately. Picking
`runtime`=15 would have made the relay invisible at
default verbosity (a footgun for daemon supervisors
whose whole point is "I want to see this output").
4. **Reader is MANDATORY, not opt-in cosmetic.** With
`stdout=PIPE` / `stderr=PIPE` we OWN the drain
responsibility — there's no `trio.capture_*` running
under the hood here. The ~64KiB OS pipe buffer
means a child writing more than that without us
reading hangs at `write()` — a deadlock that won't
show up in small-output tests, which is why the
200KiB-no-newline test is in the suite.
5. **`task_status.started(trio_proc)` BEFORE the
`own_tn` exits.** Without this, `tn.start()` would
block until the child exits — losing the "start a
long-lived daemon and continue with parent work"
use case. With it, the parent gets the live process
handle immediately and the supervise+relay tasks
run in the supervisor coro until the child exits.
6. **`__notes__` via `add_note()` for the CPE
`.stderr`.** The `.stderr` attribute is what
`subprocess` callers expect; the `add_note()` is
what trio's exception-rendering shows. Both wired so
programmatic AND human consumers see the stderr at
teardown.
## Honesty statement
This file's content is RECONSTRUCTED from the staged
code, not extracted from a verbatim model transcript.
The prompt-io skill's intent is for the `.raw.md` to
be a pre-edit fossil; that's not possible here. Future
work should write the prompt-io entry DURING the
design session.

View File

@ -0,0 +1,93 @@
---
model: claude-fable-5[1m]
service: claude
session: 0780a862-e19a-4f0a-86cd-c8afc0997757
timestamp: 2026-06-11T17:51:52Z
git_ref: 8526985c
scope: docs+code+config
substantive: true
raw_file: 20260611T175152Z_8526985c_prompt_io.raw.md
---
## Prompt
> we need big boi docs; ours are way out of date and generally
> terrible. i'd like to use a sphinx theme in the vein and/or
> one-of/nearly-the-same-as used in one or all of the `msgspec`,
> `numpy`, `ray`, `xonsh`, `polars` projects.
>
> see the following oustanding but now very old issues to guide
> you: #175, #126, #123, #157
>
> we should try to target a 3 columned sphinx theme with,
> - document index-nav on left,
> - body content in middle,
> - diagrams (ideally in d2lang if possible) on RHS
>
> optimize for (if possible),
> - reusing examples code without duplication in docs,
> - generating new examples from todos throughout code base,
> - distilling the essence of SC across processes in the simplest
> and most friendly way possible.
>
> do this work in a /open-wkt and do you best without my guidance
> for the first major pass - we will refine asap once you are
> complete.
Session settings: `/effort max` + ultracode (multi-agent workflow
orchestration). Fully autonomous first pass; two orchestrated
agent fleets were used (6-agent recon survey, then 10-agent
content fan-out: 9 section writers + 1 examples smith), with the
orchestrator authoring the landing page, SC essay, sphinx
scaffold, d2 pipeline + diagram sources, CI workflow and all
integration/fix passes directly.
## Response summary
Complete sphinx docs revamp on branch `wkt/big_boi_docs`:
`pydata-sphinx-theme` 0.18 (3-column: left nav / content /
page-toc, with d2 diagrams + asides floated into the RHS margin
via custom CSS), sphinx 9.1, a local `.. d2::` directive
rendering `docs/diagrams/*.d2` sources with committed-SVG
fallback, a vendored `.. margin::` directive, ~25 new doc pages
(landing, start/, explain/, 12 guides, 10 api-ref pages,
project/), 5 new auto-tested examples + 3 modernized + 1
renamed, and a gh-pages deploy workflow (issue #123). All
example code is `literalinclude`d from `examples/` (zero
duplication, CI-verified). Build: green, 24 warnings all
pre-existing-docstring/NEWS sourced.
## Files changed
See the branch diff (uncommitted at entry-write time):
> `git diff test_cpu_throttling..wkt/big_boi_docs`
> `git -C <wkt> status --short` (pre-commit working tree)
- `docs/conf.py` — full rewrite for pydata theme + ext stack
- `docs/_ext/d2diagrams.py` — new `.. d2::` sphinx directive
- `docs/_ext/marginalia.py` — new `.. margin::` directive
- `docs/_static/css/custom.css` — b&w skin + RHS margin floats
- `docs/diagrams/*.d2` (7) — diagram sources (sketch/grayscale)
- `docs/_diagrams/*.svg` (7) — committed rendered fallbacks
- `docs/index.rst` — new landing (replaces dead-API doc)
- `docs/start/*.rst` (3), `docs/explain/*.rst` (3),
`docs/guide/*.rst` (13), `docs/api/*.rst` (10),
`docs/project/*.rst` (3) — new content tree
- `docs/dev_tips.rst` — removed (ported to project/dev-tips)
- `examples/{typed_payloads,nested_actor_tree,
service_daemon_discovery,uds_transport_actor_tree,
streaming_broadcast_fanout}.py` — new, smoke-tested
- `examples/{a_trynamic_first_scene,
actor_spawning_and_causality,parallelism/single_func}.py` —
`.result()` -> `.wait_for_result()` modernization
- `examples/parallelism/concurrent_futures_primes.py` — renamed
from leading-underscore + trio-runner shim added
- `pyproject.toml``docs` dependency-group filled in
- `uv.lock` — relock for docs group
- `.github/workflows/docs.yml` — build + gh-pages deploy
## Human edits
None yet — entry written pre-commit; the author reviews, stages
and commits manually (per repo workflow policy).

View File

@ -0,0 +1,49 @@
---
model: claude-fable-5[1m]
service: claude
timestamp: 2026-06-11T17:51:52Z
git_ref: 8526985c
diff_cmd: git diff test_cpu_throttling..wkt/big_boi_docs
---
# Raw output pointers (diff-ref mode)
All generated content is code/config/docs committed alongside
this entry on branch `wkt/big_boi_docs`; per the prompt-io
diff-ref decision rule each file's verbatim content is the diff
itself rather than a copy here:
> `git diff test_cpu_throttling..wkt/big_boi_docs -- docs/`
> `git diff test_cpu_throttling..wkt/big_boi_docs -- examples/`
> `git diff test_cpu_throttling..wkt/big_boi_docs -- pyproject.toml uv.lock .github/`
## Generation notes (non-code output summary)
- Theme research (web, agent-verified 2026-06-11): msgspec=furo,
xonsh=furo, numpy/ray/polars=pydata-sphinx-theme (ray migrated
off sphinx-book-theme); sphinx-book-theme 1.2.0 hard-pins
pydata 0.16.1 (stale) -> chose pydata 0.18 + sphinx 9.1.
- d2 ecosystem: no production-grade pypi extension exists
(sphinxcontrib-d2lang 0.0.5 ignores returncodes, uuid4 output
names; sphinx-d2 is an empty stub) -> wrote local
`docs/_ext/d2diagrams.py` (~230 LOC) with D2_BIN env
discovery, mtime caching, committed-SVG fallback and
literal-block last resort.
- Diagrams authored in d2 (theme-id 1 "Neutral Grey" + sketch
mode + ELK layout, validated by render + headless-firefox
screenshot loop): actor_tree, context_handshake (real
msg-spec names Start/StartAck/Started/Yield/Stop/Return),
streaming_pipeline, runtime_stack, debug_lock,
error_propagation, infected_aio.
- API truth enforced from a 6-agent recon pass over the
reorganized package tree (runtime/, discovery/, spawn/, ipc/,
msg/, devx/, trionics/): docs teach `.wait_for_result()`,
registrar (not arbiter) naming, `@tractor.context` +
`open_context()` as the core model, `run_in_actor()` as
convenience only.
- All ~30 literalincluded example scripts verified present; 9
touched/new example files smoke-run green (exit 0, <16s).
- Final build: `sphinx-build -b html` succeeded; 24 residual
warnings, every one sourced from pre-existing library
docstring rst-isms or legacy NEWS.rst content (left untouched
by design; flagged for a follow-up docstring lint pass).

View File

@ -0,0 +1,65 @@
---
model: claude-fable-5[1m]
service: claude
session: 638d5a98-abd2-4217-a447-19f2330eb334
timestamp: 2026-06-25T23:25:06Z
git_ref: 0a3c48fc
scope: code
substantive: true
raw_file: 20260625T232506Z_0a3c48fc_prompt_io.raw.md
---
## Prompt
Same `/code-review` follow-up session (PR #460). Finding #10:
the new `api/` autodoc reference pages surfaced 22 docutils
warnings sourced from informal reST in public docstrings. The
user opted to fix it on this branch,
> well we're on a clean main so why not just do these now too?
Goal: drive the docs build warning-free (24 -> 0) without any
code/behaviour change — docstring content only.
## Response summary
Surgical docstring-RST lint across 9 library modules + the one
offending `NEWS.rst` changelog entry,
- bullet lists given a blank line + base-column indent
(`Context`, `Context.cancel_called`/`.cancelled_caught`/
`.outcome`, `ActorNursery.cancel_called`, `query_actor`,
`open_crash_handler`, the `#318` `NEWS.rst` entry),
- the under-short `Behaviour:` underline in `Context.cancel`
demoted to a `**bold**` label,
- unbalanced/pluralized-role backticks fixed — closed the
`wait_for_actor` summary backtick; applied the `` `role`\ s ``
escaped-plural idiom in `gather_contexts`, `mk_pdb`,
`MsgCodec`, msg `Error`, `open_context_from_portal`,
- the `|_` method-tree in `ContextCancelled.canceller` made a
literal block (bare `|` was parsed as a substitution ref).
Verified: build `24 -> 0` warnings; `import tractor` clean under
`-W error::SyntaxWarning`; the `\ s` idiom renders as e.g.
"acms" (no backslash/space leak in HTML); `ruff` clean; diff
confirmed docstring-content-only.
Initial generation was delegated to a constrained subagent
(docstring-only edits + a rebuild-to-zero verification gate);
the orchestrator independently re-built, reviewed the full
diff, and added the `NEWS.rst` fix to reach zero.
## Files changed
- `tractor/_context.py`, `tractor/_exceptions.py`,
`tractor/devx/debug/_post_mortem.py`,
`tractor/devx/debug/_repl.py`, `tractor/discovery/_api.py`,
`tractor/msg/_codec.py`, `tractor/msg/types.py`,
`tractor/runtime/_supervise.py`,
`tractor/trionics/_mngrs.py` — docstring reST fixes,
- `NEWS.rst` — blank line before a bullet list in the `#318`
entry.
## Human edits
None — committed as generated (`0a3c48fc`).

View File

@ -0,0 +1,35 @@
---
model: claude-fable-5[1m]
service: claude
timestamp: 2026-06-25T23:25:06Z
git_ref: 0a3c48fc
diff_cmd: git diff 0a3c48fc~1..0a3c48fc
---
# Raw output pointers (diff-ref mode)
The generated patch is committed; the verbatim content is the
diff:
> `git diff 0a3c48fc~1..0a3c48fc`
## Generation notes (non-code, verbatim)
- Pure docstring-content edits; no signatures, logic, or
non-docstring lines touched (confirmed via `git diff` review).
- reST fix patterns applied: blank-line + base-column indent
for bullet lists ("Unexpected indentation" / "Definition
list ..." / "Block quote ..."); `**bold**` in place of an
under-length section underline ("Title underline too short");
closing/escaping backticks ("Inline interpreted text ...
without end-string"); literal-block for an ASCII tree whose
`|` chars tripped "Inline substitution_reference ...".
- The pluralized-role idiom is written `\\ s` in the Python
source (so the runtime docstring holds `` `role`\ s ``); a
bare `\ ` would raise a 3.13 invalid-escape SyntaxWarning.
Verified clean via `python -W error::SyntaxWarning -c
"import tractor"`.
- Verification gate: `sphinx-build` warning count 24 -> 2
(subagent, docstrings only) -> 0 after the orchestrator added
the `NEWS.rst` blank-line fix; rendered-HTML spot-check
confirmed no literal `\ s` leak; `ruff check` clean.

View File

@ -0,0 +1,64 @@
---
model: claude-fable-5[1m]
service: claude
session: 638d5a98-abd2-4217-a447-19f2330eb334
timestamp: 2026-06-25T23:25:06Z
git_ref: 3a5cfde0
scope: code
substantive: true
raw_file: 20260625T232506Z_3a5cfde0_prompt_io.raw.md
---
## Prompt
Follow-up to a `/code-review` of PR #460 (this branch) which
surfaced lower-severity findings in the `.. d2::` sphinx
directive. The user's driving instructions across the session,
> yup do both *at least*, then report back.
(re: "implement #7 (+ optionally #8) on the branch") and,
> well we're on a clean main so why not just do these now too?
(re: also doing #9). The three findings being addressed,
- #7: a `d2` render that is *attempted and fails* (binary
present but errors on the source) silently degraded to the
stale committed SVG and the build still exited 0,
- #8: `parallel_*_safe = True` was declared while `run()`
writes the shared output SVG during the read phase (torn
write under `sphinx-build -j`),
- #9: output keyed only by `src.stem` so two `.d2` sources
with the same stem collide.
## Response summary
Reworked `docs/_ext/d2diagrams.py`,
- `render_svg()` now returns a `RenderResult` tristate
(`OK` / `FELL_BACK` / `NO_OUTPUT` / `FAILED`) so `run()` can
distinguish a graceful no-binary fallback from a real render
failure; `FAILED` emits a `reporter.error` (build fails under
`-W`),
- the render writes into a sibling temp-file then `os.replace()`
(atomic) so a failed/torn render can never clobber a good
committed SVG,
- a per-build `_seen_outputs` map (reset on `builder-inited`)
errors on a same-stem output collision.
Verified: a corrupted `.d2` errors under `-W` (exit 1) with the
committed SVG byte-unchanged; the collision guard fires on a
duplicate stem; `nix run nixpkgs#d2` render path + atomic swap
leave no temp residue; `ruff` clean.
## Response summary (cont.) — Files changed
- `docs/_ext/d2diagrams.py` — tristate render result, atomic
temp-file render, output-collision guard, doc-string policy
update.
## Human edits
None — committed as generated (`3a5cfde0`). The user noted
follow-up refinements may come in later commits.

View File

@ -0,0 +1,35 @@
---
model: claude-fable-5[1m]
service: claude
timestamp: 2026-06-25T23:25:06Z
git_ref: 3a5cfde0
diff_cmd: git diff 3a5cfde0~1..3a5cfde0
---
# Raw output pointers (diff-ref mode)
The generated patch is committed; per the diff-ref decision
rule the verbatim content is the diff itself:
> `git diff 3a5cfde0~1..3a5cfde0 -- docs/_ext/d2diagrams.py`
## Generation notes (non-code, verbatim)
- `RenderResult` enum added; `render_svg()` return type changed
`bool -> RenderResult`. Mapping in `run()`: `FAILED` ->
`state_machine.reporter.error(...)`, `NO_OUTPUT` -> raw `.d2`
source as a `literal_block`, `OK`/`FELL_BACK` -> `image`.
- Atomicity via `tempfile.mkstemp(dir=out.parent, ...)` +
`os.replace(tmp, out)`; temp unlinked on any failure path.
- Collision guard: module-level `_seen_outputs: dict[str,str]`
keyed by output basename, cleared by a `builder-inited`
handler connected in `setup()`.
- `parallel_read_safe`/`parallel_write_safe` kept `True` but
now justified by the atomic swap (documented inline).
- Empirical verification performed before commit:
- broken `.d2` + `sphinx-build -W` -> exit 1, ERROR node
rendered, committed `actor_tree.svg` md5 unchanged,
- duplicate-stem orphan page -> "d2 output collision" error,
- `nix run nixpkgs#d2` forced re-render -> deterministic
(no git drift), no `.tmp` residue,
- `ruff check` clean.

View File

@ -0,0 +1,27 @@
# AI Prompt I/O Log — claude
This directory tracks prompt inputs and model
outputs for AI-assisted development using
`claude` (Claude Code).
## Policy
Prompt logging follows the
[NLNet generative AI policy][nlnet-ai].
All substantive AI contributions are logged
with:
- Model name and version
- Timestamps
- The prompts that produced the output
- Unedited model output (`.raw.md` files)
[nlnet-ai]: https://nlnet.nl/foundation/policies/generativeAI/
## Usage
Entries are created by the `/prompt-io` skill
or automatically via `/commit-msg` integration.
Human contributors remain accountable for all
code decisions. AI-generated content is never
presented as human-authored work.

View File

@ -0,0 +1,76 @@
ok now i want you to take a look at the most recent commit adding
a `tpt_bind_addrs` to `open_root_actor()` and extend the existing
tests/discovery/test_multiaddr* and friends to use this new param in
at least one suite with parametrizations over,
- `registry_addrs == tpt_bind_addrs`, as in both inputs are the same.
- `set(registry_addrs) >= set(tpt_bind_addrs)`, as in the registry
addrs include the bind set.
- `registry_addrs != tpt_bind_addrs`, where the reg set is disjoint from
the bind set in all possible combos you can imagine.
All of the ^above cases should further be parametrized over,
- the root being the registrar,
- a non-registrar root using our bg `daemon` fixture.
once we have a fairly thorough test suite and have flushed out all
bugs and edge cases we want to design a wrapping API which allows
declaring full tree's of actors tpt endpoints using multiaddrs such
that a `dict[str, list[str]]` of actor-name -> multiaddr can be used
to configure a tree of actors-as-services given such an input
"endpoints-table" can be matched with the number of appropriately
named subactore spawns in a `tractor` user-app.
Here is a small example from piker,
- in piker's root conf.toml we define a `[network]` section which can
define various actor-service-daemon names set to a maddr
(multiaddress str).
- each actor whether part of the `pikerd` tree (as a sub) or spawned
in other non-registrar rooted trees (such as `piker chart`) should
configurable in terms of its `tractor` tpt bind addresses via
a simple service lookup table,
```toml
[network]
pikerd = [
'/ip4/127.0.0.1/tcp/6116', # std localhost daemon-actor tree
'/uds/run/user/1000/piker/pikerd@6116.sock', # same but serving UDS
]
chart = [
'/ip4/127.0.0.1/tcp/3333', # std localhost daemon-actor tree
'/uds/run/user/1000/piker/chart@3333.sock',
]
```
We should take whatever common API is needed to support this and
distill it into a
```python
tractor.discovery.parse_endpoints(
) -> dict[
str,
list[Address]
|dict[str, list[Address]]
# ^recursive case, see below
]:
```
style API which can,
- be re-used easily across dependent projects.
- correctly raise tpt-backend support errors when a maddr specifying
a unsupport proto is passed.
- be used to handle "tunnelled" maddrs per
https://github.com/multiformats/py-multiaddr/#tunneling such that
for any such tunneled maddr-`str`-entry we deliver a data-structure
which can easily be passed to nested `@acm`s which consecutively
setup nested net bindspaces for binding the endpoint addrs using
a combo of our `.ipc.*` machinery and, say for example something like
https://github.com/svinota/pyroute2, more precisely say for
managing tunnelled wireguard eps within network-namespaces,
* https://docs.pyroute2.org/
* https://docs.pyroute2.org/netns.html
remember to include use of all default `.claude/skills` throughout
this work!

View File

@ -0,0 +1,34 @@
This is your first big boi, "from GH issue" design, plan and
implement task.
We need to try and add sub-interpreter (aka subint) support per the
issue,
https://github.com/goodboy/tractor/issues/379
Part of this work should include,
- modularizing and thus better organizing the `.spawn.*` subpkg by
breaking up various backends currently in `spawn._spawn` into
separate submods where it makes sense.
- add a new `._subint` backend which tries to keep as much of the
inter-process-isolation machinery in use as possible but with plans
to optimize for localhost only benefits as offered by python's
subints where possible.
* utilizing localhost-only tpts like UDS, shm-buffers for
performant IPC between subactors but also leveraging the benefits from
the traditional OS subprocs mem/storage-domain isolation, linux
namespaces where possible and as available/permitted by whatever
is happening under the hood with how cpython implements subints.
* default configuration should encourage state isolation as with
subprocs, but explicit public escape hatches to enable rigorously
managed shm channels for high performance apps.
- all tests should be (able to be) parameterized to use the new
`subints` backend and enabled by flag in the harness using the
existing `pytest --spawn-backend <spawn-backend>` support offered in
the `open_root_actor()` and `.testing._pytest` harness override
fixture.

View File

@ -0,0 +1,159 @@
# Logging-spec leaf-module granularity — "Route B" (decouple
# logger-*identity* from console-*display*)
Follow-up notes recording the breaking-changes / costs of the
deeper fix that would give the `tractor.log` logging-spec (see
`LogSpec`/`apply_logspec()`) true **per-leaf-MODULE** level
control — deliberately *not* taken (for now) in favour of the
smaller sub-PACKAGE fix already landed.
## Status / what already shipped
The cheap, contained fix is **done**: `get_logger()`'s "strip
#2" (`log.py`, the `pkg_path = subpkg_path` collapse) no longer
eats a real sub-package component. It now strips the trailing
token *only* when it duplicates the caller's leaf-*module*
filename (which the header already shows via `{filename}`).
Result:
- `devx.debug` resolves to `tractor.devx.debug`, **distinct**
from a bare `devx` -> `tractor.devx` (its parent). So the
logging-spec can dial sub-package levels at any nesting depth
(`devx.debug:runtime` ≠ `devx:cancel`).
- The `get_logger(__name__)` cosmetic ("don't repeat the leaf
module in `{name}` since `{filename}` shows it") is preserved.
What is **still NOT addressable** after that fix:
- **Per-leaf-MODULE** levels. Every module in a (sub-)pkg shares
that pkg's logger, because `get_logger()` drops the leaf
module-name from the logger key by design.
- **Top-level lib modules** (eg. `tractor.to_asyncio`,
`__package__ == 'tractor'`) emit on the *root* `tractor`
logger, so a `to_asyncio:<lvl>` spec entry hits a phantom
child -> no-op.
## What "Route B" is
Make the logger's *identity* the **full dotted module path**
(incl. the leaf module + top-level modules), eg.
`tractor.devx.debug._tty_lock` and `tractor.to_asyncio`, and
move the cosmetic leaf-trim out of logger-naming and into the
**formatter's `{name}` rendering**.
Net effect:
- Real per-module `Logger` nodes exist in the hierarchy ->
the spec can target ANY module; stdlib level-inheritance and
propagation "just work" top-down.
- Console headers stay clean because the formatter computes a
trimmed display string (drop the trailing token that equals
`{filename}`'s stem) instead of the logger doing it.
## Why it's "broad" — breaking changes / costs
The logger *name* is currently load-bearing well beyond
display; changing it ripples:
1. **Every logger name changes.**
Today (post sub-pkg fix) names collapse to the sub-package;
Route B = full module path. This touches:
- handler attachment points + the `getChild()` hierarchy,
- any `logging.getLogger('tractor.X')` string lookups,
- any name-based filtering,
- the dedup / `_strict_debug` warning logic *inside*
`get_logger()` itself — the `pkg_name in name`,
`leaf_mod in pkg_path`, "duplicate pkg-name" branches all
key off the *name shape* and would need re-derivation.
2. **Formatter rewrite.**
`LOG_FORMAT` uses `{name}` == `record.name` (the full logger
name). To keep headers clean we must compute a *display*
name and inject it as a record attr (eg. `record.pkg_ns`)
via a `logging.Filter` or a `colorlog.ColoredFormatter`
subclass overriding `.format()`, then point `LOG_FORMAT` at
that field. The `{filename}` vs `{name}` de-dup intent has
to be re-implemented per-record rather than per-logger.
3. **Propagation / double-emit surface grows.**
Full-depth loggers mean more intermediate nodes
(`...debug._tty_lock` -> `.debug` -> `.devx` -> `tractor`).
If more than one level carries a handler (spec sub-handlers
+ a root console), records double-emit. The
`propagate=False` trick we already use for filter-targeted
sub-loggers (`apply_logspec()`) must be applied carefully
across a deeper tree — more levels == more places to leak a
dup.
4. **Level-inheritance semantics shift.**
Today setting a level on `tractor.devx` gates *all* devx
emits (they share that logger). Post-Route-B,
`tractor.devx.debug._tty_lock` is its own `NOTSET` logger
that *inherits* the effective level from ancestors —
functionally similar via inheritance, BUT any code that does
`log.setLevel(...)` / reads `log.level` on a (previously
collapsed) logger now only affects that exact node. All
`setLevel`/`.level =` call sites need an audit (eg.
`get_logger()`'s own `log.level = rlog.level` line).
5. **Downstream contract churn.**
`modden` / `piker` call `get_logger()` / `get_console_log()`
and may depend on current names — including
`modden.runtime.daemon.setup_tractor_logging()` which
asserts `'tractor' not in name` on spec parts. The header
`{name}` field is user-visible in everyone's logs + CI
output. Changing the canonical names is a public-ish
behavior change -> needs a version note + downstream
coordination (or a formatter trim that keeps the *displayed*
string byte-identical to today).
6. **`get_logger()` refactor risk.**
The fn tangles two concerns: compute logger *identity* and
compute the *display* string. Route B forces splitting them
inside a ~300-line fn with multiple `_strict_debug`
branches, dup-warnings, and the `name=__name__` convenience.
High chance of subtle regressions without an exhaustive
name-derivation test matrix.
## Migration / test plan (if pursued)
- Extract a pure helper
`_mk_logger_name(pkg_name, mod_name, mod_pkg) -> (logger_name,
display_name)` and cover it with an exhaustive unit matrix:
auto vs explicit vs `__name__`; package-`__init__` vs leaf
module; nested vs flat; `pkg_name in name` vs not; top-level
module (`__package__ == pkg_name`).
- Switch `get_logger()` to use it for *identity*; switch the
formatter to use `display_name` (via a record attr).
- Re-run the full suite + golden-diff a sample of rendered log
headers to confirm zero cosmetic churn.
- Coordinate the name change with `modden`/`piker`; bump +
CHANGES note.
## Cheaper alternative — "Route A" (record-filter)
If per-leaf control is wanted *before* committing to Route B:
keep names collapsed, add a `logging.Filter` on the configured
handler keyed on `record.module` / `record.pathname` that maps
each record's source module -> its spec level. Set the base
logger to the *minimum* level in the spec (so records aren't
pre-dropped by the logger), and let the filter discriminate
up/down within that floor.
- Pros: no name churn, no formatter change, fully contained
next to `apply_logspec()`.
- Cons: a filter can only discriminate *within* what the logger
admits -> base must be permissive, so `at_least_level()`
expensive-work guards over-admit; matching dotted spec names
to a `pathname` is fiddly; doesn't clean up the hierarchy
itself.
## Recommendation
- Defer Route B unless true per-module loggers are wanted as a
first-class feature.
- If per-leaf control is needed soon, prefer **Route A**
(filter) — lower risk.
- The shipped sub-PACKAGE fix already covers the common ask
(`devx.debug` vs `devx`).

View File

@ -1,4 +1,4 @@
|logo| ``tractor``: distributed structurred concurrency
|logo| ``tractor``: distributed structured concurrency
``tractor`` is a `structured concurrency`_ (SC), multi-processing_ runtime built on trio_.
@ -23,16 +23,11 @@ model" looks like, and that's **intentional**.
Where do i start!?
------------------
The first step to grok ``tractor`` is to get an intermediate
knowledge of ``trio`` and **structured concurrency** B)
Some great places to start are,
- the seminal `blog post`_
- obviously the `trio docs`_
- wikipedia's nascent SC_ page
- the fancy diagrams @ libdill-docs_
New to ``trio`` and **structured concurrency**? Our docs collect the
best starting points and then walk you straight into a hands-on
quickstart:
https://goodboy.github.io/tractor/start/quickstart.html
Features
--------
@ -57,6 +52,7 @@ Features
`discovery`_ sys with plans to support multiple `modern protocol`_
approaches.
- Various ``trio`` extension APIs via ``tractor.trionics`` such as,
- task fan-out `broadcasting`_,
- multi-task-single-resource-caching and fan-out-to-multi
``__aenter__()`` APIs for ``@acm`` functions,
@ -98,7 +94,7 @@ the code base::
# but @goodboy prefers the more explicit (and shell agnostic)
# https://docs.astral.sh/uv/configuration/environment/#uv_project_environment
UV_PROJECT_ENVIRONMENT="tractor_py313
UV_PROJECT_ENVIRONMENT="tractor_py313"
# hint hint, enter @goodboy's fave shell B)
uv run --dev xonsh
@ -110,545 +106,53 @@ Alongside all this we ofc offer "releases" on PyPi::
Just note that YMMV since the main git branch is often much further
ahead then any latest release.
Hacking on the docs themselves? The build + live-preview one-liners
(incl. nix-shell specifics) are collected in `notes_to_self/howtodocs.md
<https://github.com/goodboy/tractor/blob/main/notes_to_self/howtodocs.md>`_,
and rendered as the "Building these docs" section of our dev-tips guide.
Example codez
-------------
In ``tractor``'s (very lacking) documention we prefer to point to
example scripts in the repo over duplicating them in docs, but with
that in mind here are some definitive snippets to try and hook you
into digging deeper.
Run a func in a process
***********************
Use ``trio``'s style of focussing on *tasks as functions*:
.. code:: python
"""
Run with a process monitor from a terminal using::
$TERM -e watch -n 0.1 "pstree -a $$" \
& python examples/parallelism/single_func.py \
&& kill $!
"""
import os
import tractor
import trio
async def burn_cpu():
pid = os.getpid()
# burn a core @ ~ 50kHz
for _ in range(50000):
await trio.sleep(1/50000/50)
return os.getpid()
async def main():
async with tractor.open_nursery() as n:
portal = await n.run_in_actor(burn_cpu)
# burn rubber in the parent too
await burn_cpu()
# wait on result from target function
pid = await portal.result()
# end of nursery block
print(f"Collected subproc {pid}")
if __name__ == '__main__':
trio.run(main)
This runs ``burn_cpu()`` in a new process and reaps it on completion
of the nursery block.
If you only need to run a sync function and retreive a single result, you
might want to check out `trio-parallel`_.
Zombie safe: self-destruct a process tree
*****************************************
``tractor`` tries to protect you from zombies, no matter what.
.. code:: python
"""
Run with a process monitor from a terminal using::
$TERM -e watch -n 0.1 "pstree -a $$" \
& python examples/parallelism/we_are_processes.py \
&& kill $!
"""
from multiprocessing import cpu_count
import os
import tractor
import trio
async def target():
print(
f"Yo, i'm '{tractor.current_actor().name}' "
f"running in pid {os.getpid()}"
)
await trio.sleep_forever()
async def main():
async with tractor.open_nursery() as n:
for i in range(cpu_count()):
await n.run_in_actor(target, name=f'worker_{i}')
print('This process tree will self-destruct in 1 sec...')
await trio.sleep(1)
# raise an error in root actor/process and trigger
# reaping of all minions
raise Exception('Self Destructed')
if __name__ == '__main__':
try:
trio.run(main)
except Exception:
print('Zombies Contained')
If you can create zombie child processes (without using a system signal)
it **is a bug**.
"Native" multi-process debugging
********************************
Using the magic of `pdbp`_ and our internal IPC, we've
been able to create a native feeling debugging experience for
any (sub-)process in your ``tractor`` tree.
.. code:: python
from os import getpid
import tractor
import trio
async def breakpoint_forever():
"Indefinitely re-enter debugger in child actor."
while True:
yield 'yo'
await tractor.breakpoint()
async def name_error():
"Raise a ``NameError``"
getattr(doggypants)
async def main():
"""Test breakpoint in a streaming actor.
"""
async with tractor.open_nursery(
debug_mode=True,
loglevel='error',
) as n:
p0 = await n.start_actor('bp_forever', enable_modules=[__name__])
p1 = await n.start_actor('name_error', enable_modules=[__name__])
# retreive results
stream = await p0.run(breakpoint_forever)
await p1.run(name_error)
if __name__ == '__main__':
trio.run(main)
You can run this with::
>>> python examples/debugging/multi_daemon_subactors.py
And, yes, there's a built-in crash handling mode B)
We're hoping to add a respawn-from-repl system soon!
SC compatible bi-directional streaming
**************************************
Yes, you saw it here first; we provide 2-way streams
with reliable, transitive setup/teardown semantics.
Our nascent api is remniscent of ``trio.Nursery.start()``
style invocation:
.. code:: python
import trio
import tractor
@tractor.context
async def simple_rpc(
ctx: tractor.Context,
data: int,
) -> None:
'''Test a small ping-pong 2-way streaming server.
'''
# signal to parent that we're up much like
# ``trio_typing.TaskStatus.started()``
await ctx.started(data + 1)
async with ctx.open_stream() as stream:
count = 0
async for msg in stream:
assert msg == 'ping'
await stream.send('pong')
count += 1
else:
assert count == 10
async def main() -> None:
async with tractor.open_nursery() as n:
portal = await n.start_actor(
'rpc_server',
enable_modules=[__name__],
)
# XXX: this syntax requires py3.9
async with (
portal.open_context(
simple_rpc,
data=10,
) as (ctx, sent),
ctx.open_stream() as stream,
):
assert sent == 11
count = 0
# receive msgs using async for style
await stream.send('ping')
async for msg in stream:
assert msg == 'pong'
await stream.send('ping')
count += 1
if count >= 9:
break
# explicitly teardown the daemon-actor
await portal.cancel_actor()
if __name__ == '__main__':
trio.run(main)
See original proposal and discussion in `#53`_ as well
as follow up improvements in `#223`_ that we'd love to
hear your thoughts on!
.. _#53: https://github.com/goodboy/tractor/issues/53
.. _#223: https://github.com/goodboy/tractor/issues/223
Worker poolz are easy peasy
***************************
The initial ask from most new users is *"how do I make a worker
pool thing?"*.
``tractor`` is built to handle any SC (structured concurrent) process
tree you can imagine; a "worker pool" pattern is a trivial special
case.
We have a `full worker pool re-implementation`_ of the std-lib's
``concurrent.futures.ProcessPoolExecutor`` example for reference.
You can run it like so (from this dir) to see the process tree in
real time::
$TERM -e watch -n 0.1 "pstree -a $$" \
& python examples/parallelism/concurrent_actors_primes.py \
&& kill $!
This uses no extra threads, fancy semaphores or futures; all we need
is ``tractor``'s IPC!
"Infected ``asyncio``" mode
***************************
Have a bunch of ``asyncio`` code you want to force to be SC at the process level?
Check out our experimental system for `guest`_-mode controlled
``asyncio`` actors:
.. code:: python
import asyncio
from statistics import mean
import time
import trio
import tractor
async def aio_echo_server(
to_trio: trio.MemorySendChannel,
from_trio: asyncio.Queue,
) -> None:
# a first message must be sent **from** this ``asyncio``
# task or the ``trio`` side will never unblock from
# ``tractor.to_asyncio.open_channel_from():``
to_trio.send_nowait('start')
# XXX: this uses an ``from_trio: asyncio.Queue`` currently but we
# should probably offer something better.
while True:
# echo the msg back
to_trio.send_nowait(await from_trio.get())
await asyncio.sleep(0)
@tractor.context
async def trio_to_aio_echo_server(
ctx: tractor.Context,
):
# this will block until the ``asyncio`` task sends a "first"
# message.
async with tractor.to_asyncio.open_channel_from(
aio_echo_server,
) as (first, chan):
assert first == 'start'
await ctx.started(first)
async with ctx.open_stream() as stream:
async for msg in stream:
await chan.send(msg)
out = await chan.receive()
# echo back to parent actor-task
await stream.send(out)
async def main():
async with tractor.open_nursery() as n:
p = await n.start_actor(
'aio_server',
enable_modules=[__name__],
infect_asyncio=True,
)
async with p.open_context(
trio_to_aio_echo_server,
) as (ctx, first):
assert first == 'start'
count = 0
async with ctx.open_stream() as stream:
delays = []
send = time.time()
await stream.send(count)
async for msg in stream:
recv = time.time()
delays.append(recv - send)
assert msg == count
count += 1
send = time.time()
await stream.send(count)
if count >= 1e3:
break
print(f'mean round trip rate (Hz): {1/mean(delays)}')
await p.cancel_actor()
if __name__ == '__main__':
trio.run(main)
Yes, we spawn a python process, run ``asyncio``, start ``trio`` on the
``asyncio`` loop, then send commands to the ``trio`` scheduled tasks to
tell ``asyncio`` tasks what to do XD
We need help refining the `asyncio`-side channel API to be more
`trio`-like. Feel free to sling your opinion in `#273`_!
.. _#273: https://github.com/goodboy/tractor/issues/273
Higher level "cluster" APIs
***************************
To be extra terse the ``tractor`` devs have started hacking some "higher
level" APIs for managing actor trees/clusters. These interfaces should
generally be condsidered provisional for now but we encourage you to try
them and provide feedback. Here's a new API that let's you quickly
spawn a flat cluster:
.. code:: python
import trio
import tractor
async def sleepy_jane():
uid = tractor.current_actor().uid
print(f'Yo i am actor {uid}')
await trio.sleep_forever()
async def main():
'''
Spawn a flat actor cluster, with one process per
detected core.
'''
portal_map: dict[str, tractor.Portal]
results: dict[str, str]
# look at this hip new syntax!
async with (
tractor.open_actor_cluster(
modules=[__name__]
) as portal_map,
trio.open_nursery() as n,
):
for (name, portal) in portal_map.items():
n.start_soon(portal.run, sleepy_jane)
await trio.sleep(0.5)
# kill the cluster with a cancel
raise KeyboardInterrupt
if __name__ == '__main__':
try:
trio.run(main)
except KeyboardInterrupt:
pass
.. _full worker pool re-implementation: https://github.com/goodboy/tractor/blob/master/examples/parallelism/concurrent_actors_primes.py
We prefer to point you at the runnable scripts under ``examples/``
- each is CI-run and ``literalinclude``-d straight into the docs, so
what you read there is what actually runs - rather than inline a
pile of them here. The one-minute pitch: spawn a subactor per core,
open a ``Context`` into each, then crash the root *on purpose* and
watch the runtime reap the whole tree - zero zombies, guaranteed (if
you can make a zombie child without a system signal, it **is a
bug**).
See it run - plus the full tour (the flagship multi-process
debugger, bidirectional streaming over a ``Context``, cancellation,
discovery, "infected ``asyncio``", typed messaging and worker-pool /
cluster patterns) - in the docs:
- docs: https://goodboy.github.io/tractor/
- examples: https://github.com/goodboy/tractor/tree/main/examples
Under the hood
--------------
``tractor`` is an attempt to pair trionic_ `structured concurrency`_ with
distributed Python. You can think of it as a ``trio``
*-across-processes* or simply as an opinionated replacement for the
stdlib's ``multiprocessing`` but built on async programming primitives
from the ground up.
``tractor`` is an attempt to pair trionic_ `structured
concurrency`_ with distributed Python - think of it as ``trio``
*-across-processes*, or as an opinionated replacement for the
stdlib's ``multiprocessing`` built on async primitives from the
ground up. But really it **is just** ``trio``: nurseries that
spawn *processes* and cancel-able streaming IPC between them. If
you can drive ``trio``, you can drive ``tractor``.
Don't be scared off by this description. ``tractor`` **is just** ``trio``
but with nurseries for process management and cancel-able streaming IPC.
If you understand how to work with ``trio``, ``tractor`` will give you
the parallelism you may have been needing.
"But wait - don't 'actors' have mailboxes and messages and
stuff?!" Well, we've got (well referenced) opinions on what an "actor
model" **actually is** (tl;dr: the `3 axioms`_, not the cultural
baggage) - that whole riff lives in our docs:
https://goodboy.github.io/tractor/explain/sc-distributed.html#hold-up-is-this-an-actor-model
Wait, huh?! I thought "actors" have messages, and mailboxes and stuff?!
***********************************************************************
Let's stop and ask how many canon actor model papers have you actually read ;)
From our experience many "actor systems" aren't really "actor models"
since they **don't adhere** to the `3 axioms`_ and pay even less
attention to the problem of *unbounded non-determinism* (which was the
whole point for creation of the model in the first place).
From the author's mouth, **the only thing required** is `adherance to`_
the `3 axioms`_, *and that's it*.
``tractor`` adheres to said base requirements of an "actor model"::
In response to a message, an actor may:
- send a finite number of new messages
- create a finite number of new actors
- designate a new behavior to process subsequent messages
**and** requires *no further api changes* to accomplish this.
If you want do debate this further please feel free to chime in on our
chat or discuss on one of the following issues *after you've read
everything in them*:
- https://github.com/goodboy/tractor/issues/210
- https://github.com/goodboy/tractor/issues/18
Let's clarify our parlance
**************************
Whether or not ``tractor`` has "actors" underneath should be mostly
irrelevant to users other then for referring to the interactions of our
primary runtime primitives: each Python process + ``trio.run()``
+ surrounding IPC machinery. These are our high level, base
*runtime-units-of-abstraction* which both *are* (as much as they can
be in Python) and will be referred to as our *"actors"*.
The main goal of ``tractor`` is is to allow for highly distributed
software that, through the adherence to *structured concurrency*,
results in systems which fail in predictable, recoverable and maybe even
understandable ways; being an "actor model" is just one way to describe
properties of the system.
What's on the TODO:
-------------------
Help us push toward the future of distributed `Python`.
- Erlang-style supervisors via composed context managers (see `#22
<https://github.com/goodboy/tractor/issues/22>`_)
- Typed messaging protocols (ex. via ``msgspec.Struct``, see `#36
<https://github.com/goodboy/tractor/issues/36>`_)
- Typed capability-based (dialog) protocols ( see `#196
<https://github.com/goodboy/tractor/issues/196>`_ with draft work
started in `#311 <https://github.com/goodboy/tractor/pull/311>`_)
- We **recently disabled CI-testing on windows** and need help getting
it running again! (see `#327
<https://github.com/goodboy/tractor/pull/327>`_). **We do have windows
support** (and have for quite a while) but since no active hacker
exists in the user-base to help test on that OS, for now we're not
actively maintaining testing due to the added hassle and general
latency..
What's on the TODO
------------------
The roadmap lives with our docs - see `what the future holds
<https://goodboy.github.io/tractor/project/index.html#what-the-future-holds>`_
for where ``tractor`` is headed.
Feel like saying hi?
--------------------
@ -659,36 +163,24 @@ say hi, please feel free to reach us in our `matrix channel`_. If
matrix seems too hip, we're also mostly all in the the `trio gitter
channel`_!
.. _structured concurrent: https://trio.discourse.group/t/concise-definition-of-structured-concurrency/228
.. _distributed: https://en.wikipedia.org/wiki/Distributed_computing
.. _multi-processing: https://en.wikipedia.org/wiki/Multiprocessing
.. _trio: https://github.com/python-trio/trio
.. _nurseries: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/#nurseries-a-structured-replacement-for-go-statements
.. _actor model: https://en.wikipedia.org/wiki/Actor_model
.. _trionic: https://trio.readthedocs.io/en/latest/design.html#high-level-design-principles
.. _async sandwich: https://trio.readthedocs.io/en/latest/tutorial.html#async-sandwich
.. _3 axioms: https://www.youtube.com/watch?v=7erJ1DV_Tlo&t=162s
.. .. _3 axioms: https://en.wikipedia.org/wiki/Actor_model#Fundamental_concepts
.. _adherance to: https://www.youtube.com/watch?v=7erJ1DV_Tlo&t=1821s
.. _trio gitter channel: https://gitter.im/python-trio/general
.. _matrix channel: https://matrix.to/#/!tractor:matrix.org
.. _broadcasting: https://github.com/goodboy/tractor/pull/229
.. _modern procotol: https://en.wikipedia.org/wiki/Rendezvous_protocol
.. _pdbp: https://github.com/mdmintz/pdbp
.. _pdb++: https://github.com/pdbpp/pdbpp
.. _cheap or nasty: https://zguide.zeromq.org/docs/chapter7/#The-Cheap-or-Nasty-Pattern
.. _(un)protocol: https://zguide.zeromq.org/docs/chapter7/#Unprotocols
.. _discovery: https://zguide.zeromq.org/docs/chapter8/#Discovery
.. _modern protocol: https://en.wikipedia.org/wiki/Rendezvous_protocol
.. _messages: https://en.wikipedia.org/wiki/Message_passing
.. _trio docs: https://trio.readthedocs.io/en/latest/
.. _blog post: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/
.. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency
.. _SC: https://en.wikipedia.org/wiki/Structured_concurrency
.. _libdill-docs: https://sustrik.github.io/libdill/structured-concurrency.html
.. _unrequirements: https://en.wikipedia.org/wiki/Actor_model#Direct_communication_and_asynchrony
.. _async generators: https://www.python.org/dev/peps/pep-0525/
.. _trio-parallel: https://github.com/richardsheridan/trio-parallel
.. _uv: https://docs.astral.sh/uv/
.. _msgspec: https://jcristharif.com/msgspec/
.. _guest: https://trio.readthedocs.io/en/stable/reference-lowlevel.html?highlight=guest%20mode#using-guest-mode-to-run-trio-on-top-of-other-event-loops
@ -699,10 +191,10 @@ channel`_!
.. |gh_actions| image:: https://github.com/goodboy/tractor/actions/workflows/ci.yml/badge.svg?branch=main
:target: https://github.com/goodboy/tractor/actions/workflows/ci.yml
.. |docs| image:: https://readthedocs.org/projects/tractor/badge/?version=latest
:target: https://tractor.readthedocs.io/en/latest/?badge=latest
:alt: Documentation Status
.. |docs| image:: https://github.com/goodboy/tractor/actions/workflows/docs.yml/badge.svg?branch=main
:target: https://goodboy.github.io/tractor/
:alt: Documentation
.. |logo| image:: _static/tractor_logo_side.svg
.. |logo| image:: _static/tractor_logo_wire.svg
:width: 250
:align: middle

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 98 KiB

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 93 KiB

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 80 KiB

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 80 KiB

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 98 KiB

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 61 KiB

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 90 KiB

View File

@ -0,0 +1,387 @@
# tractor: distributed structured concurrency.
'''
``.. d2::`` - embed d2lang_ diagrams in sphinx docs
with build-time rendering and a committed-SVG
fallback.
Rendering policy,
- when a ``d2`` binary is found (see discovery order
below) any out-of-date SVG is (re)rendered from its
``.d2`` source (normally kept in ``docs/diagrams/``)
into ``docs/_diagrams/``,
- otherwise any pre-rendered (and git committed) SVG
already in ``docs/_diagrams/`` is used as-is,
- when neither is possible the diagram *source* is
emitted as a literal block so no content is ever
silently dropped.
A render that is *attempted and fails* (a ``d2`` bin is
present but errors on the source) is NOT silently
degraded to the stale committed SVG: it raises a
docutils error (so ``sphinx-build -W`` fails the
build). The render is also atomic a failed/torn
render can never clobber a good committed SVG so the
last-good diagram survives a bad edit.
Binary discovery order,
- the ``D2_BIN`` env var; may contain args which are
split via `shlex`, eg.
``D2_BIN='nix run nixpkgs#d2 --'``,
- the ``d2_bin`` sphinx config value,
- ``shutil.which('d2')``.
Usage,
.. code:: rst
.. d2:: diagrams/actor_tree.d2
:caption: A tree of ``trio``-task-trees.
:margin:
.. _d2lang: https://d2lang.com
'''
from __future__ import annotations
import enum
import hashlib
import os
from pathlib import Path
import re
import shlex
import shutil
import subprocess as sp
import tempfile
from docutils import nodes
from docutils.parsers.rst import directives
from sphinx.application import Sphinx
from sphinx.util import logging
from sphinx.util.docutils import SphinxDirective
log = logging.getLogger(__name__)
# subdir (under the sphinx srcdir) holding rendered,
# git-committed, fallback SVG outputs.
_outdir: str = '_diagrams'
# per-build map of {output-svg-name: source-relpath} used
# to detect 2 distinct `.d2` sources colliding on a single
# output stem; reset on each `builder-inited` (see `setup`).
_seen_outputs: dict[str, str] = {}
class RenderResult(enum.Enum):
'''
Outcome of a `render_svg()` call.
'''
OK = 'ok' # fresh SVG exists on disk
FELL_BACK = 'fell_back' # no bin; serving committed SVG
NO_OUTPUT = 'no_output' # no bin AND no committed SVG
FAILED = 'failed' # bin present, render errored
def find_d2(
app: Sphinx,
) -> list[str]|None:
'''
Resolve the d2 render command as an argv list or
`None` when no binary can be found.
'''
if env_bin := os.environ.get('D2_BIN'):
return shlex.split(env_bin)
if cfg_bin := app.config.d2_bin:
return shlex.split(cfg_bin)
if path_bin := shutil.which('d2'):
return [path_bin]
return None
def render_svg(
app: Sphinx,
src: Path,
out: Path,
) -> RenderResult:
'''
Maybe (re)render `src` -> `out` and report the
outcome as a `RenderResult`.
A render is performed only when `out` is missing or
older than `src` AND a `d2` binary is available. The
write is atomic (temp-file + `os.replace`) so a
failed or torn render never clobbers an existing
(committed) SVG.
'''
fresh: bool = (
out.exists()
and
src.stat().st_mtime <= out.stat().st_mtime
)
if fresh:
return RenderResult.OK
d2cmd: list[str]|None = find_d2(app)
if d2cmd is None:
# no binary: fall back to whatever is committed,
# else signal the caller to emit the raw source.
if out.exists():
log.info(
f'no d2 binary; using committed svg '
f'for {src.name}'
)
return RenderResult.FELL_BACK
return RenderResult.NO_OUTPUT
out.parent.mkdir(
parents=True,
exist_ok=True,
)
# render into a sibling temp-file first so a bad
# `d2` exit (or a crash mid-write) leaves any prior
# committed SVG fully intact.
fd, tmpname = tempfile.mkstemp(
dir=str(out.parent),
prefix=f'.{out.stem}.',
suffix='.svg.tmp',
)
os.close(fd)
tmp = Path(tmpname)
argv: list[str] = (
d2cmd
+ list(app.config.d2_args)
+ [str(src), str(tmp)]
)
try:
proc = sp.run(
argv,
capture_output=True,
text=True,
timeout=120,
)
except (
OSError,
sp.TimeoutExpired,
) as err:
tmp.unlink(missing_ok=True)
log.warning(
f'd2 invocation failed for {src.name}: '
f'{err}'
)
return RenderResult.FAILED
if proc.returncode != 0:
tmp.unlink(missing_ok=True)
log.warning(
f'd2 render error for {src.name}:\n'
f'{proc.stderr}'
)
return RenderResult.FAILED
# atomic swap-in of the freshly rendered SVG.
os.replace(tmp, out)
return RenderResult.OK
class D2Diagram(SphinxDirective):
'''
Render a ``.d2`` source file (path relative to
the sphinx srcdir) as an SVG figure.
'''
required_arguments = 1
has_content = False
option_spec = {
'caption': directives.unchanged,
'alt': directives.unchanged,
'width': (
directives.length_or_percentage_or_unitless
),
'margin': directives.flag,
'class': directives.class_option,
'name': directives.unchanged,
}
def run(self) -> list[nodes.Node]:
relsrc: str = self.arguments[0]
srcdir = Path(self.env.srcdir)
src: Path = srcdir / relsrc
self.env.note_dependency(relsrc)
if not src.exists():
err = self.state_machine.reporter.error(
f'd2 source not found: {relsrc}',
line=self.lineno,
)
return [err]
out: Path = (
srcdir
/ _outdir
/ f'{src.stem}.svg'
)
# collision guard: two distinct sources must not
# map onto the same output stem within a build.
prior: str|None = _seen_outputs.get(out.name)
if (
prior is not None
and
prior != relsrc
):
err = self.state_machine.reporter.error(
f'd2 output collision: {relsrc!r} and '
f'{prior!r} both render to '
f'{_outdir}/{out.name}; rename one '
f'`.d2` stem',
line=self.lineno,
)
return [err]
_seen_outputs[out.name] = relsrc
result: RenderResult = render_svg(
self.env.app,
src,
out,
)
if result is RenderResult.FAILED:
# loud, build-failing (under `-W`) signal; the
# prior committed SVG, if any, is untouched.
err = self.state_machine.reporter.error(
f'd2 render failed for {relsrc} (see '
f'build log); the committed svg, if any, '
f'was left untouched',
line=self.lineno,
)
return [err]
if result is RenderResult.NO_OUTPUT:
# last resort: emit the raw d2 source so no
# content is ever silently dropped.
log.warning(
f'no svg available for {relsrc}; '
f'emitting d2 source as literal block'
)
src_text: str = src.read_text()
literal = nodes.literal_block(
src_text,
src_text,
)
literal['language'] = 'text'
return [literal]
# OK | FELL_BACK -> embed the SVG figure.
img = nodes.image(
uri=f'/{_outdir}/{out.name}',
alt=self.options.get(
'alt',
f'd2 diagram: {src.stem}',
),
)
if width := self.options.get('width'):
img['width'] = width
# content-hash the rendered svg so the served
# `<img src>` carries a `?v=<hash>` cache-buster
# (sphinx tags css/js this way but not images);
# without it an edited diagram keeps showing the
# browser's stale copy at the stable
# `_images/<stem>.svg` url on autobuild reload.
if out.exists():
img['d2_stem'] = out.stem
img['d2_cachebust'] = hashlib.sha256(
out.read_bytes()
).hexdigest()[:8]
fig = nodes.figure()
fig += img
classes: list[str] = (
['d2-diagram']
+ self.options.get('class', [])
)
if 'margin' in self.options:
# NB: the bare 'margin' class is what
# book-style themes key off for
# right-margin placement; our custom css
# uses 'd2-margin'.
classes += [
'margin',
'd2-margin',
]
fig['classes'] += classes
if caption_txt := self.options.get('caption'):
(
inline_nodes,
_msgs,
) = self.state.inline_text(
caption_txt,
self.lineno,
)
caption = nodes.caption(
caption_txt,
'',
*inline_nodes,
)
fig += caption
self.add_name(fig)
return [fig]
def _reset_seen(
app: Sphinx,
) -> None:
'''
Clear the per-build output-collision map at the
start of each build.
'''
_seen_outputs.clear()
def _cachebust_d2_images(
app: Sphinx,
pagename: str,
templatename: str,
context: dict,
doctree,
) -> None:
'''
Append a ``?v=<content-hash>`` query to every d2
diagram ``<img src>`` in the rendered page ``body``
so a browser re-fetches an *edited* SVG instead of
its stale cached copy at the stable
``_images/<stem>.svg`` url (sphinx cache-busts
css/js this way, but not images).
Scoped strictly to d2 images (those carrying the
`d2_cachebust` attr set in `D2Diagram.run()`); all
other markup is left untouched.
'''
if doctree is None or 'body' not in context:
return
busts: dict[str, str] = {}
for img in doctree.findall(nodes.image):
if ver := img.get('d2_cachebust'):
busts[img['d2_stem']] = ver
if not busts:
return
body: str = context['body']
for stem, ver in busts.items():
body = re.sub(
r'(src="[^"]*?' + re.escape(stem) + r'\.svg)(")',
rf'\1?v={ver}\2',
body,
)
context['body'] = body
def setup(app: Sphinx) -> dict:
app.add_config_value('d2_bin', None, 'env')
app.add_config_value('d2_args', [], 'env')
app.add_directive('d2', D2Diagram)
app.connect('builder-inited', _reset_seen)
app.connect('html-page-context', _cachebust_d2_images)
return {
'version': '0.1.0',
# NB: run() writes the rendered SVG during the
# READ phase; the temp-file + `os.replace` swap
# keeps that atomic so concurrent renders of the
# same diagram (under `sphinx-build -j`) can't
# tear the output file.
'parallel_read_safe': True,
'parallel_write_safe': True,
}

View File

@ -0,0 +1,51 @@
# tractor: distributed structured concurrency.
'''
``.. margin::`` - prose-anchored, right-margin asides.
A theme-agnostic vendoring of `sphinx_book_theme`'s
`Margin` directive: a `docutils` `Sidebar` subclass
which tags the node with a ``margin`` class; placement
is then pure CSS (see ``_static/css/custom.css``)
allowing use on any theme incl. our
`pydata_sphinx_theme`.
Usage,
.. code:: rst
.. margin:: An optional title
Aside content; text, figures, whatever.
'''
from docutils import nodes
from docutils.parsers.rst.directives.body import (
Sidebar,
)
from sphinx.application import Sphinx
class Margin(Sidebar):
'''
Notes/figures placed in the right margin, anchored
at the current point in the prose flow.
'''
required_arguments = 0
optional_arguments = 1
def run(self) -> list[nodes.Node]:
if not self.arguments:
self.arguments = ['']
out: list[nodes.Node] = super().run()
out[0].attributes['classes'].append('margin')
return out
def setup(app: Sphinx) -> dict:
app.add_directive('margin', Margin)
return {
'version': '0.1.0',
'parallel_read_safe': True,
'parallel_write_safe': True,
}

185
docs/_static/css/custom.css vendored 100644
View File

@ -0,0 +1,185 @@
/* tractor docs: a minimal black + white skin over
* `pydata-sphinx-theme` plus RHS-marginalia + d2
* diagram styling.
*/
html[data-theme="light"] {
--pst-color-primary: #000000;
--pst-color-secondary: #3d3d3d;
--pst-color-accent: #5a5a5a;
--pst-color-link: #000000;
--pst-color-link-hover: #5a5a5a;
--pst-color-inline-code: #1a1a1a;
--pst-color-inline-code-links: #000000;
}
html[data-theme="dark"] {
--pst-color-primary: #ffffff;
--pst-color-secondary: #c9c9c9;
--pst-color-accent: #a8a8a8;
--pst-color-link: #ffffff;
--pst-color-link-hover: #bdbdbd;
--pst-color-inline-code: #e8e8e8;
--pst-color-inline-code-links: #ffffff;
}
/* mono-chrome links: rely on underline for
* affordance instead of color.
*/
.bd-content a:not(.headerlink) {
text-decoration: underline;
text-underline-offset: 0.18em;
}
/* d2 diagram figures */
figure.d2-diagram {
text-align: center;
}
figure.d2-diagram img {
max-width: 100%;
height: auto;
}
/* keep light-rendered svgs legible in dark mode */
html[data-theme="dark"] figure.d2-diagram img {
background: #ffffff;
border-radius: 6px;
padding: 6px;
}
/* prose-anchored right-margin asides (tufte-ish):
* float right within the content column on wide
* screens, collapse inline on narrow ones.
*/
@media (min-width: 960px) {
aside.margin,
figure.margin,
div.margin,
figure.d2-margin {
float: right;
clear: right;
width: 44%;
margin: 0.2rem 0 1rem 1.4rem;
font-size: 0.85rem;
}
}
/* strip docutils sidebar chrome from margin asides */
aside.sidebar.margin {
border: none;
background: transparent;
padding: 0;
}
aside.sidebar.margin > p.sidebar-title {
font-weight: 600;
font-size: 0.9rem;
margin-bottom: 0.3rem;
}
/* landing-page hero logo (inline svg): the linework uses
* `fill: currentColor` so it inherits the theme text colour
* (auto-flips light<->dark), and the faces are transparent so
* the page background shows through. */
svg.hero-logo {
display: block;
max-width: 360px;
height: auto;
color: var(--pst-color-text-base);
}
/* 3-up landing hero row: name | logo | tagline, with a centered
* sub-tagline beneath. */
.hero-row {
display: flex;
align-items: center;
/* left-anchored single row; tighter gap so the logo sits over
* the sub-line and the tagline over the prose below it (stack
* only on mobile, via the media query). */
justify-content: flex-start;
flex-wrap: nowrap;
gap: clamp(0.5rem, 2vw, 1.5rem);
margin: 0 0 0.25rem;
}
.hero-row svg.hero-logo {
/* override the stacked-hero sizing for the inline row */
width: clamp(240px, 34vw, 360px);
margin: 0;
}
.hero-row .hero-tag {
margin: 0;
/* keep it on a single line beside the logo */
max-width: none;
white-space: nowrap;
font-size: clamp(1rem, 2.1vw, 1.4rem);
font-weight: 400;
line-height: 1.15;
color: var(--pst-color-text-base);
}
.hero-row .hero-tag {
min-width: 0;
}
.hero-sub {
/* wrap at the prose width (not the logo's), with a line of
* space before the prose section below. */
margin: 0 0 4rem;
max-width: none;
text-align: left;
font-size: 1.05rem;
color: var(--pst-color-text-muted);
}
/* the page carries a single rst <h1> ("tractor") for semantics
* + SEO, but the visual title is the hero so hide the doc
* title accessibly (still read by screen readers + search). */
#tractor > h1 {
position: absolute !important;
width: 1px;
height: 1px;
margin: -1px;
padding: 0;
overflow: hidden;
clip: rect(0 0 0 0);
clip-path: inset(50%);
white-space: nowrap;
border: 0;
}
/* smaller section headings on the landing only */
#sixty-seconds-of-why > :is(h1, h2),
#dig-in > :is(h1, h2),
#features > :is(h1, h2),
#where-do-i-start > :is(h1, h2) {
font-size: 1.75rem;
}
/* extra breathing room before the first landing section */
#sixty-seconds-of-why {
margin-top: 4rem;
}
/* pull the section's `tl;dr` margin-note up so its top edge is
* level with the heading (default floats it beside the body). */
#sixty-seconds-of-why aside.margin {
margin-top: -3.75rem;
}
/* stack the 3-up hero only on real mobile */
@media (max-width: 640px) {
.hero-row {
flex-direction: column;
}
.hero-row .hero-tag {
max-width: none;
}
}
/* navbar brand: the "distributed SC" tagline next to the logo
* (the logo `text` value), kept small + muted like a sub-line. */
.navbar-brand .title {
/* match the centered nav tabs (Bootstrap leaves the nav-link
* size/weight vars empty ~1rem / normal); `inline-block` is
* the real underline fix it stops the parent brand-link's
* :hover underline from propagating onto the text (a block
* child's `text-decoration:none` can't cancel an ancestor's
* underline). */
display: inline-block;
font-size: 1rem;
font-weight: 400;
text-decoration: none;
}
/* it's a brand mark, not a content link never underline the
* brand text (base / hover / focus). */
.navbar-brand,
.navbar-brand:hover,
.navbar-brand:focus,
.navbar-brand:visited:hover,
.navbar-brand .title,
.navbar-brand:hover .title {
text-decoration: none !important;
border-bottom: 0 !important;
}

462
docs/_static/tractor_hero.html vendored 100644
View File

@ -0,0 +1,462 @@
<div class="hero-row">
<svg class="hero-logo" role="img" aria-label="tractor" version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
viewBox="0 0 2670 1980" style="enable-background:new 0 0 2670 1980;" xml:space="preserve">
<style type="text/css">
.st0{fill:currentColor;}
.st1{fill:none;}
</style>
<g>
<path class="st0" d="M2275.3,1000.1c-0.1,2.3,0,4.7,0,7c0,189.3,0,378.7,0,568c0,6,0.1,12.1-0.8,17.9c-2.4,15.2-12.9,17.4-23.6,10
c-45.4-31-91.2-61.4-136.9-91.9l-30.1-20.3c-9.8-6.3-19.2-13.2-29.2-19.3c-6.7-4.1-9.8-9.7-9.9-17.2c-0.1-9.3-0.6-18.7-0.3-28
c0.2-7.3-1.7-13-6.5-19c-18.6-23.3-36.2-47.3-56.9-69c-18-18.8-36.7-36.8-56.4-53.8c-31.8-27.5-65.4-52.7-101-74.9
c-32.7-20.4-66.8-38.4-102.2-54c-50.7-22.4-103.2-39.2-157.1-51.5c-36.7-8.4-73.8-14.4-111.2-18.5c-34.5-3.8-69-6.5-103.7-5.8
c-23.1,0.5-46.2-1.3-69.3,0.4c-22.9,1.8-45.9,3-68.7,5.6c-72.7,8.4-143.7,24.6-212.1,50.6c-65.5,24.8-127.4,57-184.9,97.4
c-46.8,32.8-90.1,69.8-129.5,111.1c-18.1,19-35.1,39.2-50.5,60.6c-3.9,5.5-5.5,11-5.3,17.6c0.4,9.3,0,18.7,0,28
c0,9.3-3.1,16.4-11.8,21.4c-10.4,5.9-19.9,13.4-30.1,19.8l-25.4,17.2c-11.6,7.8-23.3,15.5-34.9,23.3c-34.3,23-68.6,46-102.9,69
c-14.6,9.7-23.6,5.5-25.2-12.1c-0.5-5.3-0.5-10.7-0.5-16c0-189.7,0-379.3,0-569l-0.1-26.1c0.2-190.3,0.1-380.7,0.2-571
c0-5.7,0.4-11.3,0.5-17c0.1-3.7,1.2-7.3,3-10.5c3.4-5.9,9.3-7.7,15.4-4.8c2.7,1.3,5.1,3.1,7.6,4.8c44.3,29.6,88.5,59.3,132.8,88.9
l29.7,20.1c10.8,6.6,20.9,14.3,31.7,20.9c6.7,4.1,9.7,9.8,9.8,17.3c0.2,11,0.4,22,0.2,33c-0.1,5.2,1.2,9.6,4.2,13.9
c15.4,22.2,33,42.8,51.6,62.2c53.9,56.4,114.1,105.1,181.4,144.6c38.4,22.6,78.3,42.4,119.9,58.6c40,15.6,80.9,28.3,122.8,38
c52.7,12.1,106,19.3,160,22.6c18.3,1.1,36.6,2.1,54.9,1.5c17.7-0.6,35.5,1.1,53.2-0.1c17.6-1.1,35.2-2.3,52.8-3.6
c35.9-2.5,71.4-8.5,106.7-15.1c53.4-9.9,105.5-25,156-45.1c53.8-21.5,105.1-48.2,153.2-80.9c43.4-29.5,83.6-62.7,121.1-99.3
c27.3-26.6,50.3-56.8,73.9-86.6c2.9-3.6,3.6-7.6,3.6-12.1c-0.1-10.7,0.2-21.3,0.4-32c0.1-7.5,3.2-13.1,9.9-17.2
c10.2-6.2,19.9-13.4,30-19.8l23.2-16.3c4.2-1.3,7.4-4.3,11-6.7c42.9-28.7,85.8-57.4,128.7-86.1c1.1-0.7,2.3-1.4,3.3-2.3
c10.9-9.1,21.4-4.7,23.8,10.7c0.8,5.6,0.6,11.3,0.6,17c0,189.3,0,378.7,0,568L2275.3,1000.1z M740,1203.1c-2-0.2-4-0.3-6-0.5
c-0.5,2.5-4.8,2-4.2,5.8c2.9-1,6.4-1.2,4.2-5.8C735.8,1206,738.1,1202.3,740,1203.1c-0.6-0.7-0.5-1.3,0.2-1.7
c-1.1-0.3-1.1-0.1-0.4,0.8C740,1202.3,739.9,1202.8,740,1203.1z M1935.5,780.5c-0.2,0.2-0.4,0.4-0.6,0.4c-0.3,0.1-0.6,0-0.8,0
c0.4-0.2,0.8-0.5,1.1-0.8c1.4-0.9,2-1.9,0.5-3.3c3.2,1.1,6.1,1.2,7.9-2.8c-3.3-0.7-4.6,3.8-7.7,2.9
C1935.7,778.2,1935.6,779.4,1935.5,780.5z M1934.8,1202.5c-0.1-0.2-0.1-0.4-0.2-0.6c0,0,0.1-0.1,0.1-0.1c0,0.2,0.1,0.3,0.1,0.5
c3.1,1.6,5,5.7,11.3,5.1C1941.3,1205,1938.9,1202,1934.8,1202.5z M2072.4,516.6c-1,2.3-3.4,3-5.3,4.3c-7.4,5.1-8.2,7.2-5.3,17.8
c4.2-6.5,8-12.6,11.9-18.6c1.1-1.7,0.9-3.1-1.3-3.6c0,1,0.3,1.8,1.4,2c0.2,0,0.4-0.4,0.6-0.7C2073.8,517.4,2073.1,517,2072.4,516.6
z M599.6,517.5c-0.4-0.8-1.1-1.7-1.8-0.8c-0.8,0.8,0.1,1.5,1,1.6c2.4,8.9,7.5,16.2,13.4,24.1c2.3-16.5,1.9-17.3-10.7-24.4
C600.9,517.6,600.2,517.6,599.6,517.5z M534.9,1353.5c-20,21.8-37.3,44.1-53.4,67.4c-16.1,23.4-26.8,49.5-41.8,73.5
c9-21,17.4-42.2,28.9-61.9c16.9-28.7,35.4-56.4,57.5-81.5c0.9-1,1.6-2.1,2.6-3.1c2.9-2.9,3.3-6,2.3-10
c-9.1-33.8-14.6-68.3-19.2-103c-5-38-8.3-76.1-10.6-114.3c-2.5-39.9-3.4-79.9-3.9-119.8c-0.1-5.1-1.4-6.8-6.7-6.9
c-21.6-0.2-43.3-0.8-64.9-1.3c-2,0-4-0.4-6,0.8c0,195.2,0,390.5,0,585.6c2.1,0.6,3.1-0.4,4.1-1.1c52.7-33.8,105.3-67.7,158-101.4
c3.6-2.3,5.7-5.4,7.3-9.1c6-13,12.5-25.8,20.8-37.4c4.2-5.9,3.9-11.9,2.5-19.1c-2.4,1.8-4,2.9-5.5,4.2
c-13.9,12.6-32.1,8.7-42.7-2.9c-7.3-8-12.4-17.2-17.1-26.8C542.3,1375.7,539.4,1365.1,534.9,1353.5z M534.6,627.2
c3.6-5,4.3-10.7,6.3-15.9c6.2-15.8,12.6-31.5,25.2-43.7c10.8-10.5,26.8-12.6,38.9-2.8c2.3,1.9,4.3,5,7.9,5
c1.6-8.3-0.2-15.3-5.1-21.9c-7.4-9.9-12.7-21-17.8-32.3c-2.4-5.3-5.8-9.2-10.8-12.4c-50.5-32.2-100.9-64.7-151.3-97.1
c-2.4-1.5-4.5-3.9-8.7-3.9c0,3.7,0,7.2,0,10.7c0,187.9,0,375.8,0,563.7c0,2,0.2,4,0,6c-0.6,4.9,1.4,6.3,6.2,6.1
c13.6-0.6,27.3-0.7,41-1c8.3-0.2,16.7-0.6,25-0.4c4.9,0.1,6.7-1.8,6.1-6.4c-0.1-1,0-2,0-3c1.2-75,4.3-149.8,13.7-224.3
c4.6-36.3,10-72.5,19.6-107.9c1.6-5.7,0.4-9.9-3.4-14.4c-10.7-12.8-21.7-25.4-31-39.1c-19-28.1-36.8-57-49.6-88.7
c-2.2-5.4-5.5-10.4-6.2-16.6C465,538.3,494.6,586,534.6,627.2z M2060.9,1411.1c-2.3,8.5-0.1,15.1,4,21.4
c5.9,9.3,13.4,18.1,17.1,28.2c3.9,10.7,10.9,16.6,19.9,22.4c47.8,30.3,95.3,61.1,143,91.7c2.7,1.7,5,4.3,9.5,4.5c0-3.4,0-6.6,0-9.8
c0-188.3,0-376.6,0-565c0-2-0.1-4,0-6c0.2-3.4-1.1-5-4.6-4.8c-2.3,0.1-4.7,0-7,0.1c-20,0.6-40,1.2-60,1.9c-6.7,0.2-6.7,0.4-7,7.4
c0,1.3,0,2.7-0.1,4c-0.4,60-3,119.9-8.9,179.7c-5,50.1-11.3,99.9-24,148.7c-1.4,5.2,0.1,8.3,3.5,12.3c15,17.8,29.6,36.1,42.4,55.5
c20,30.4,36.1,62.9,49.3,96.8c3.2,8.2,5.7,16.6,6.8,25.4c-33.9-90.8-82.6-145-107.2-169.7c-4.1,11-7.5,22.1-12.7,32.5
c-4.9,9.9-10.3,19.5-18.8,26.8c-7.1,6.1-15.1,8.9-24.8,7.5C2073.2,1421.4,2068.3,1415.5,2060.9,1411.1z M2244.8,454.3
c0.6,4.5-0.2,6.7-0.8,8.9c-6.5,24.2-17,47-28.2,69.2c-18.9,37.2-43,71.2-70.4,102.8c-2.7,3.2-3.6,5.8-2.5,9.8
c7.9,29.6,13.1,59.8,17.3,90.2c6.4,45.6,10.2,91.3,12.8,137.3c2,35.3,2.6,70.6,3.1,105.9c0.1,7.6,0.2,7.6,7.6,7.8
c17,0.5,34,1,51,1.4c5,0.1,10,0.2,15,0.5c3.2,0.1,5.1-0.9,4.9-4.5c-0.1-1.7,0-3.3,0-5c0-189.6,0-379.3-0.1-568.9
c0-2.4,1.1-5.2-1.2-7.3c-0.8,0.1-1.6,0-2.1,0.4c-52.7,33.8-105.3,67.7-158,101.5c-2,1.3-3.7,2.6-4.7,4.9c-6.9,16-16.9,30.1-25.8,45
c-3.1,5.2-3.4,10.2-2,15.7c2.6,0.1,3.5-1.6,4.6-2.6c15.7-13.9,33-10.6,45.6,3.8c6,6.8,10.4,14.6,14.4,22.7
c5.1,10.3,8.3,21.3,12.8,32.9C2185.2,576.1,2221.7,520.5,2244.8,454.3z M553.7,662.8c-0.3,1.3-0.7,2.8-1,4.3
c-7.1,31.2-11.7,62.8-15.4,94.5c-7.1,59.9-10.3,120.1-11.8,180.4c-0.3,13-0.2,26-0.5,39c-0.1,4.4,1.5,5.7,5.7,5.6
c25-0.5,50-0.7,74.9-1.1c7.4-0.1,7.6-0.2,7.6-7.8c0-85.6,0-171.2,0.1-256.8c0-5.3-1.7-8.5-5.8-11.7c-16.6-12.8-32.4-26.6-46.9-41.9
C559,665.6,557.6,663,553.7,662.8z M553.6,1319.7c2.1-1,2.8-1.2,3.2-1.6c16.4-17.1,34-32.9,52.7-47.4c3-2.4,3.8-5,3.8-8.6
c-0.1-13.3-0.1-26.7-0.1-40c0-72.3,0-144.7,0-217c0-10.3,0.4-9-9.2-9.2c-23.3-0.4-46.7-0.8-70-1.1c-2.8,0-5.9-1-9.3,1.5
C526.2,1104.4,531.1,1212.3,553.6,1319.7z M2119.8,1319.2c4.8-20.9,8.3-40.2,11.1-59.6c5.8-39.5,9.8-79.2,12.5-119.1
c3.1-45.5,4.7-91.1,5.1-136.7c0.1-7.6-0.1-7.7-7.6-7.7c-24.3,0.1-48.6,0.3-72.9,0.5c-7.9,0.1-8,0.1-8,8.5c0,84.6,0,169.1-0.1,253.7
c0,5.6,1.7,9.3,5.9,12.7c12.1,9.9,24.2,19.9,36,30.1C2107.7,1306.9,2113.1,1312.7,2119.8,1319.2z M2120.4,664.4
c-2.1-0.6-2.9-0.1-3.6,0.7c-15.2,16.5-33.1,29.9-50.2,44.2c-4.8,4-6.8,8-6.8,14.3c0.3,32,0.2,64,0.2,96c0,52.3,0,104.6,0,157
c0,8.1,0.1,8.2,8.2,8.2c18.3,0.2,36.7,0.2,55,0.4c6.7,0.1,13.3,0.1,20,0.3c3.7,0.1,5.7-1.1,5.4-5.1c-0.2-2.3,0-4.7-0.1-7
c-0.5-45-2.1-89.9-5.2-134.8C2139.2,780.1,2132.7,721.9,2120.4,664.4z M878.7,1138.5c-36,15-69.8,32.9-102.3,53.1
c-3.4,2.1-4.9,4.7-5.7,8.6c-2.8,14.4-5.7,28.8-12.2,42c-3.6,7.3-7.7,14.4-17.3,14.4c-9.6,0-13.4-7.4-17.4-14.4
c-1.9-3.2-2-7.6-5.4-9.8c-22.3,17.4-41.6,37.2-58.8,58.9c-5.8,7.3-10.2,15.3-12,24.8c-3.8,20.3-9.1,40.2-16.7,59.5
c-2.6,6.7-1.3,13.3-1.5,20.3c1.5-0.4,2-0.3,2.1-0.5c3.2-3.8,6.4-7.6,9.4-11.6c31.2-40.5,66-77.4,105.4-110.2
c36.9-30.8,75.6-59.1,117.2-83.1c54.3-31.2,110.9-57.4,170.8-76.4c38.4-12.2,77.4-21.8,117-29.2c9.5-1.8,19.1-3,28.6-4.6
c1.5-0.3,3.8,0.3,4.4-1.5c1.4-3.9,0.6-8-0.4-11.7c-1-3.6-4.5-1.7-6.9-1.5c-40.6,3.2-80.4,11-120.4,18c-7,1.2-10.4,4.1-10.9,11.2
c-0.3,3.8-0.5,9.1-5.8,9c-4.6-0.1-4.7-5-5.3-8.5c-0.7-4.5-2.4-6-7.3-4.9c-39.3,9.1-78,19.9-115.5,34.7c-4.6,1.8-7.1,4.2-7.8,8.9
c-0.9,5.9-2.2,11.8-5.2,17.1c-4.1,7.1-10.6,7.2-14.6,0C881.9,1147.7,880.7,1143.4,878.7,1138.5z M1032.8,891.9
c0.9-3.8,1.6-6.7,2.4-9.5c0.7-2.3,1.7-4.6,4.4-4.9c3.4-0.3,4.4,2.3,5.3,4.9c0.4,1.3,0.8,2.6,0.8,3.9c0.1,7.7,4.5,10.4,11.7,11.6
c32.5,5.5,64.8,11.9,97.5,15.7c7.9,0.9,15.9,1.7,23.8,2.4c1.9,0.2,4,0.7,5.3-1.6c3.5-6.5,0.3-13-7-14.2c-1-0.2-2-0.2-3-0.4
c-42.6-5.8-84.2-15.9-125.4-28.2c-41.8-12.4-82.5-28.2-121.6-47.1c-51-24.6-99.8-53.3-144.7-87.9c-20.6-15.8-40.9-32.1-59.9-49.7
c-29.1-26.9-56.4-55.7-80.1-87.7c-3.7-4.9-7.7-9.6-11.6-14.3c-0.8,7.6-2.4,14.7,0.5,21.8c7.4,18.2,12.4,37.2,16,56.5
c1.7,9.1,5,17.3,10.6,24.4c9.2,11.4,18.5,22.8,28.7,33.3c9.9,10.1,21.1,19,32.2,28.9c3.8-6.5,5.6-13,9.8-18.2
c7.7-9.4,17.7-9.5,25.3-0.1c3.6,4.5,5.7,9.7,7.8,15c4.3,10.9,6.8,22.3,8.9,33.7c0.9,5.2,3.3,8.1,7.5,10.7
c21.9,13.4,44.1,26,67.3,37.1c10.9,5.2,22.1,10,33.5,15.2c1.8-4.8,3.1-8.6,4.7-12.2c1.6-3.5,4.2-6.3,8.3-6.1
c3.6,0.2,5.8,2.9,7.3,6.1c2.4,5.2,3.8,10.6,4.8,16.2c0.8,4.6,2.9,7.2,7.7,9C950.6,871.5,990.8,883,1032.8,891.9z M1953.8,1233.7
c-2.9,5.8-4.9,11.5-8.8,16.2c-7.6,9.3-17.8,9.3-25.3,0c-3.4-4.2-5.5-9-7.5-14c-4.4-11.5-7.2-23.5-9.4-35.7
c-0.7-3.9-2.4-6.4-5.8-8.6c-30.4-19.4-62.4-35.6-95-50.8c-6.9-3.3-7-3-9.9,4.7c-0.7,1.9-1.5,3.7-2.3,5.5c-1.6,3.1-3.7,5.6-7.6,5.6
c-3.8,0-5.9-2.5-7.6-5.6c-2.9-5.4-4.3-11.3-5.1-17.2c-0.7-5-3-7.3-7.5-9.1c-37.8-15.2-77.3-24.6-116.7-34.3c-3.8-0.9-5.4,0.2-6.2,4
c-0.7,3.9-0.2,9.9-6.3,9.5c-5-0.3-5.1-6.1-5.3-9.6c-0.4-6.6-3.3-8.9-9.6-10c-32.8-5.6-65.4-12.2-98.6-15.7
c-8.3-0.9-16.5-2.1-24.8-2.7c-5.4-0.4-7.1,1.2-7.5,6.6c-0.3,3.9-1.3,8,5.2,8.4c6.3,0.4,12.4,2.1,18.7,3.1
c47.7,7.5,94.4,19.1,140.3,34.3c42.4,14,83.5,31.2,123,52c46.3,24.4,90.2,52.6,131,85.6c48.5,39.1,93.7,81.2,129.4,132.6
c1.9,2.8,4.8,7.5,7.3,6.7c4.5-1.5,1.4-6.8,2.1-10.4c0.6-3.4-0.5-6.6-1.8-9.7c-7.8-19.5-12.6-40-17.1-60.4c-1.1-5.1-3-9.4-6-13.5
c-17-22.8-36.7-43.1-57.6-62.2C1959.6,1237.1,1957.6,1234.7,1953.8,1233.7z M2043.6,585.1c-4.2,2.7-6.2,6.2-8.5,9.2
c-29.9,40.3-64.6,76.1-102.5,109c-33.8,29.3-69.5,56.2-107.6,79.4c-51.8,31.6-106.3,57.7-163.9,77.3
c-40.7,13.8-82.1,24.9-124.2,33.1c-13.7,2.7-27.6,4.7-41.3,7c-2.2,0.4-5.3-0.1-6.2,2.4c-1.4,3.8-1.1,8.2,0.3,11.7
c1.4,3.5,5.3,1.6,8.1,1.3c40.1-4.2,79.8-10.7,119.5-18.1c6.2-1.2,10.4-2.7,10.5-10.2c0-3.5,0.3-9.3,5.4-9.5c6-0.3,5.5,5.7,6.2,9.6
c0.7,3.9,2.3,4.9,6.1,3.9c10-2.7,20-5.1,30-7.5c28.5-6.9,56.4-15.6,83.9-25.7c6.8-2.5,10.2-6.3,11-13.3c0.5-5,2-9.8,4.5-14.2
c4.2-7.6,10.8-7.6,15.1,0.1c1.6,2.9,2.5,6.2,3.4,9.3c0.9,2.9,2.5,3.5,5.2,2.2c2.4-1.2,5-1.9,7.3-3.1c26.6-13.1,53.2-26,78.7-41.2
c11.9-7.1,19.6-14.7,21.3-29.3c1.4-12.4,5.5-24.8,12.8-35.4c8.1-11.7,19.5-11.7,27.5,0.1c3.1,4.6,5.4,9.7,8.2,14.9
c1.9-1.4,3.6-2.5,5.1-3.8c22.2-19.8,42.7-41.1,60.6-64.9c2.6-3.5,4.3-7.3,5.2-11.7c4.5-21.2,9.6-42.2,17.6-62.4
C2045.3,599.2,2044.5,592.6,2043.6,585.1z M777.2,1163.8c1.2,0.2,1.6,0.4,1.9,0.3c30.4-14.3,60.9-28.3,92.6-39.6
c4.2-1.5,4.4-3.7,3.9-7.5c-4.5-37.4-6.3-75-6.5-112.6c0-7.8-0.2-7.9-8.5-7.9c-22,0-44,0-65.9,0c-8.6,0-8.8,0.1-8.8,8.7
c-0.4,45.3-2.2,90.5-6.6,135.6C778.4,1148.4,776.3,1155.9,777.2,1163.8z M776.8,817.8c0.3,3.8,0.5,7.1,0.9,10.4
c2.5,21.2,4.1,42.4,5.5,63.7c1.8,28.6,2.3,57.2,2.7,85.9c0.1,7.1,0.3,7.3,7.2,7.3c22.6,0.1,45.3,0.1,67.9,0.1
c8.8,0,8.1-0.2,8.1-8.3c0.1-37,1.9-73.9,6.4-110.6c0.7-5.8-0.5-8.2-6-10.1c-29.5-10.3-58-23.3-86.2-36.8
C781.5,818.4,779.9,817.2,776.8,817.8z M1896.4,1163.9c1-2.7,0.2-5.3-0.1-7.9c-2.4-19.8-3.9-39.8-5.3-59.7
c-2.1-30.3-2.5-60.6-3.3-90.9c-0.2-8.7-0.2-8.8-8.6-8.8c-16.7-0.1-33.3,0-50-0.1c-6.3,0-12.7,0.2-19-0.1c-3.9-0.1-5.5,1.4-5.2,5.3
c0.1,1.7,0.1,3.3,0,5c-1.1,36-2,72-6.4,107.7c-0.9,7.7-1.1,7.9,6.3,10.6c28.5,10.4,56.1,22.9,83.3,36.1
C1890.6,1162.4,1893.1,1164.4,1896.4,1163.9z M1896.8,817.7c-2.5-0.3-4.3,0.6-6.1,1.5c-6,2.9-12,5.6-18,8.6
c-22.1,10.9-44.9,20.1-67.9,28.7c-7.5,2.8-7.3,3-6.5,10.6c1,8.9,1.9,17.9,2.7,26.8c2.4,28.2,3.2,56.5,3.6,84.7
c0.1,5.4,1.8,6.7,6.8,6.6c22.6-0.2,45.3-0.1,67.9-0.2c1,0,2-0.1,3,0c3.8,0.3,5.6-1.2,5.2-5.1c-0.2-1.6,0-3.3,0.1-5
c0.7-38.6,1.7-77.2,5.2-115.6C1894.1,845.4,1895.5,831.6,1896.8,817.7z M908.3,1109.6c1.4,0,2.4,0.2,3.2,0
c37.8-11.7,75.5-24,114.2-32.5c4.5-1,5.8-2.9,5.5-7.4c-1.3-22-2.5-43.9-2.3-65.9c0.1-7.3-0.2-7.4-8-7.4c-33.3,0-66.7,0.1-100,0.2
c-7.7,0-7.9,0.1-7.9,7.5c0,31-1.6,61.9-4,92.9C908.8,1101,907.3,1105.2,908.3,1109.6z M1765.3,1110.4c0-2.5,0.1-3.5,0-4.4
c-3.1-33.9-4.7-67.8-4.8-101.8c0-7.4-0.4-7.6-8.2-7.7c-33.3-0.1-66.7-0.2-100-0.2c-7.3,0-7.2,0.1-7.5,7.9c-0.7,21-1.1,42-2.3,62.9
c-0.5,9-1.3,8.3,7.6,10.4c29.8,7.1,59.4,14.9,88.5,24.5C1747.2,1104.8,1755.7,1107.4,1765.3,1110.4z M908.8,872.2
c-0.2,0.3-0.6,0.6-0.6,0.8c2.8,35.5,5.2,71,4.8,106.7c-0.1,4.6,1.9,5.6,6,5.6c34.7-0.1,69.3-0.1,104,0.1c4.9,0,6.2-1.8,6.1-6.4
c-0.5-22,0.8-44,2.1-65.9c0.3-5.2-0.9-7.7-6.4-8.8c-36.9-7.6-72.4-20.1-108.5-30.5C913.8,873,911.6,871.1,908.8,872.2z
M1765.6,872.4c-3.2-0.6-5.1,0.3-7,0.9c-22.2,6.7-44.2,14.5-66.7,20.3c-14.5,3.7-29,7.3-43.5,10.8c-4.4,1.1-6.5,2.6-6.2,8
c1.3,21.9,1.8,43.9,2.6,65.8c0.2,7,0.2,7.1,7.1,7.1c33.6,0,67.2,0,100.8-0.1c7.5,0,7.7-0.2,7.7-7.8c0.2-24.6,0.9-49.2,2.6-73.8
C1763.8,893.4,1764.7,883.1,1765.6,872.4z M725.8,1192.5c8.7-4.8,15.8-9.8,23.8-12.8c9.6-3.7,12.7-10.5,13-19.7
c0.1-1.3,0.2-2.7,0.4-4c3.4-27.1,5.6-54.3,6.7-81.5c1-23.6,1.5-47.2,2-70.8c0.2-7-0.1-7-7.3-7.1c-6.7-0.1-13.3,0-20,0
c-9,0-18,0.2-27,0c-4.2-0.1-6.2,1.3-5.9,5.7c0.2,2.3,0,4.7,0.1,7c0.8,27.6,1.2,55.2,3,82.8C717,1125.1,719.9,1158.2,725.8,1192.5z
M1946.2,1191.7c2.9-2.2,2.4-4.7,2.8-6.9c4.5-26.6,6.9-53.5,8.9-80.4c2.5-33.9,3.2-67.9,3.7-101.8c0.1-5-1.8-6.2-6.4-6.1
c-15.3,0.2-30.6,0-46,0.1c-7.6,0-7.8,0.2-7.7,7.8c0.7,44,2,87.9,6.7,131.6c1.1,10.3,2.2,20.5,3.5,30.7c0.4,2.8,0.2,6,3.5,7.8
C1925.8,1180.2,1936.2,1186.1,1946.2,1191.7z M1947.7,789.1c-9.1,5-16.9,10.1-25.3,13.8c-8.2,3.6-10.5,9.5-11.5,17.8
c-3,27.1-5.6,54.3-6.9,81.6c-1.2,25.3-1.5,50.6-2.5,75.9c-0.2,5.1,1.5,7,6.7,7c15-0.3,30-0.1,45-0.2c8.6,0,8.4,0.6,8.3-8.9
c-0.3-30.6-1-61.3-3.1-91.9C1956.3,852.9,1953.6,821.8,1947.7,789.1z M726,789.2c-3.3,16.4-5.1,31.3-6.7,46.2
c-4.6,42.4-6.3,85-7.3,127.7c-0.1,5.7,0,11.3-0.2,17c-0.1,3.4,1.4,4.8,4.7,4.8c17,0,34,0,51,0.1c3.6,0,4.6-1.9,4.4-5
c-0.1-2.3-0.1-4.7-0.1-7c-0.5-37.3-1.7-74.6-5-111.8c-1.4-15.9-2.8-31.9-5.5-47.6c-0.4-2.6-0.3-5.2-3.2-6.8
C747.6,801.1,737.2,795.4,726,789.2z M1625.1,910.5c-9.8,1.8-18,3.4-26.2,4.6c-23.3,3.5-46.2,9.9-70,10
c-10.9,0.1-21.8,2.4-32.7,3.7c-3.5,0.4-6.2,1.1-6,6c0.5,15,0.7,30,0.7,44.9c0,4.4,1.7,5.8,5.9,5.7c8.3-0.2,16.6,0,25-0.1
c30.6,0,61.3,0,91.9-0.1c9,0,9-0.1,9.2-9.4c0.2-13,0.2-26,0.6-38.9C1623.7,928.4,1624.5,919.8,1625.1,910.5z M1625.2,1071.1
c-2.2-23.8-2.3-46.1-2.4-68.4c0-4.5-1-6.6-6.1-6.6c-39.9,0.1-79.9,0.1-119.8-0.1c-5.1,0-6.1,2-6.1,6.5c0,14.6-0.1,29.3-0.6,43.9
c-0.1,4.5,1.6,5.6,5.6,6.1c12.6,1.4,25,3.8,37.7,4c8,0.1,15.9,1.1,23.8,2.5C1579.5,1063,1601.7,1066.9,1625.2,1071.1z
M1049.3,1070.4c2.6,1,4.9-0.1,7-0.6c23-5.5,46.3-8.9,69.7-12.3c16.7-2.5,33.7-3.8,50.6-5.2c5.1-0.4,6.8-1.9,6.6-7
c-0.4-14-0.3-27.9-0.5-41.9c-0.1-7-0.2-7.2-7.2-7.2c-39.3,0-78.5,0-117.8,0.1c-6.7,0-6.8,0.2-6.8,7.5c0,16.3-0.5,32.6-1.4,48.9
C1049.3,1058.6,1048.4,1064.5,1049.3,1070.4z M1049.8,910.2c-0.4,1.2-0.8,1.8-0.8,2.4c0.6,22.6,2.5,45.2,1.9,67.8
c-0.1,4.2,2.1,4.8,5.6,4.8c39.9,0,79.9-0.1,119.8,0.1c5.5,0,6.7-2.3,6.6-7.2c-0.2-11,0.1-22,0.2-32.9c0.1-15.9,0.1-15.5-16.3-17.4
c-16.5-1.9-33.1-2.4-49.6-5.2C1094.7,918.9,1072,915.9,1049.8,910.2z M1260.5,985.4c21.6,0,43.2,0,64.9,0c9.1,0,9.1-0.1,9.1-9.4
c0-10.3,0-20.6-0.1-30.9c-0.1-7.7-0.1-7.7-7.6-7.9c-2.3-0.1-4.7,0-7,0c-41.9,0-83.8-2.2-125.5-6.2c-4.6-0.4-5.9,0.9-6,5.1
c-0.2,13.6-0.6,27.3-0.8,40.9c-0.1,8.3,0.1,8.4,8.1,8.4C1217.3,985.4,1238.9,985.4,1260.5,985.4z M1485,931.5
c-3.1-1.4-5.4-0.6-7.7-0.4c-26.2,1.9-52.4,4.3-78.7,5.1c-17.3,0.5-34.6,1.3-51.9,1.1c-7.4-0.1-7.5,0.1-7.6,7.8
c-0.1,11.7,0,23.3-0.1,35c0,3.7,1,5.8,5,5.5c2-0.1,4,0,6,0c41.6,0,83.2,0,124.9,0c2,0,4-0.1,6,0c3.3,0.1,5-1.2,4.9-4.6
C1485.5,964.2,1485.3,947.6,1485,931.5z M1485,1051.1c0.3-17,0.5-33.6,0.9-50.1c0.1-4.2-2.2-4.8-5.7-4.8c-45,0.1-89.9,0.1-134.9,0
c-4.7,0-6.5,1.4-6.3,6.2c0.3,12,0.4,24,0,36c-0.2,5.2,2.1,6,6.5,6c11.6-0.1,23.3,0.1,34.9,0.4
C1415,1045.8,1449.5,1048.1,1485,1051.1z M1188.4,1050.6c47.5-3.4,93.2-6.8,139.2-6.4c0.7,0,1.3,0,2,0c3.2,0.2,4.9-0.9,4.9-4.5
c-0.1-13,0-25.9,0-38.9c0-3.4-1.6-4.8-4.9-4.6c-1.7,0.1-3.3,0-5,0c-42.6,0-85.1,0-127.7,0c-1.7,0-3.3,0.1-5,0
c-2.8-0.1-4.4,1.1-4.4,4.1C1187.9,1016.8,1188.1,1033.4,1188.4,1050.6z M2010.7,1230.1c0.9-7.3-0.9-13.8-1.5-20.3
c-6.8-66-10.3-132.2-10.7-198.6c-0.1-16.1-0.3-16.3-16.8-14.7c-2.9,0.3-4.2,1.1-4.1,4.1c0.1,3,0,6,0,9
c-0.3,41.3-2.1,82.6-6.1,123.8c-2,20.2-4.3,40.4-8.2,60.4c-0.9,4.8-0.2,7.8,4.5,10.4c10.2,5.7,19.9,12.2,29.8,18.3
C2001.8,1225,2005.6,1228.2,2010.7,1230.1z M2011.2,752.5c-2.7-0.8-3.6,0.3-4.7,1c-13.3,8.3-26.4,16.7-39.8,24.7
c-3.6,2.1-4.2,4.5-3.5,8.2c1.4,7.8,2.8,15.7,3.9,23.6c7.4,53.8,10.1,107.9,10.6,162.2c0.1,14.1-2.6,12.8,13.7,12.8
c7,0,7-0.2,7.1-7.3c0.4-27.3,0.6-54.6,1.8-81.8c1.4-30.9,3.1-61.8,6.1-92.6C2008,786.3,2009.6,769.5,2011.2,752.5z M662.1,1231.5
c3.7-2.4,5.6-3.7,7.5-4.9c11.6-7.7,22.9-16.2,35.5-22.2c5.3-2.5,6.2-5.5,5.1-11c-4.6-22.5-7-45.4-9-68.3
c-3.5-39.9-5-79.8-5.4-119.8c-0.1-8.6-0.2-8.6-8.8-8.7c-1.3,0-2.7,0-4,0c-7.9,0.1-8,0.1-7.9,8.4c0,24.7-0.6,49.3-1.6,74
c-1.3,32-3,63.9-6,95.8C665.8,1192.9,664.1,1211.1,662.1,1231.5z M662.9,752.1c-1,1.4-0.4,3-0.3,4.6c2,20.5,4.6,41.1,6,61.7
c3.8,52.5,6.5,105.1,6.4,157.8c0,8.8,0.1,8.9,8.5,8.9c14.5-0.1,12.1,1.3,12.3-12.1c1.1-62.7,3.5-125.2,14.8-187
c0.8-4.1-0.9-6-4.2-7.7c-13.4-6.8-25.6-15.5-38-23.9C666.8,753.1,665.5,751.1,662.9,752.1z M2044.3,728.6c-5.4,1.1-7.1,3.9-7.7,8.3
c-1.7,14.9-4,29.7-5.6,44.6c-4.6,42.1-7.2,84.2-8.9,126.5c-1,25.3-1.3,50.5-1.3,76.8c6.5,0,12.4-0.1,18.3,0
c3.8,0.1,5.5-1.4,5.2-5.3c-0.2-1.7,0-3.3,0-5c0-78.9,0-157.9,0-236.8C2044.3,734.9,2044.3,732,2044.3,728.6z M2044.3,1253
c0-3.7,0-6.4,0-9c0-78.6,0-157.2,0-235.8c0-13,1.3-11.5-11.9-11.5c-13,0-11.5-1.8-11.5,11.8c0,59.3,3.1,118.5,8.7,177.6
c1.9,19.5,4,39.1,7,58.5C2037.3,1248.9,2038.8,1251.8,2044.3,1253z M629.2,728.7c0,4.3,0,8,0,11.6c0,76.5,0,153.1,0,229.6
c0,15.8,0,15.8,16,15.3c7.6-0.2,7.6-0.2,7.7-7.7c0.1-3.3,0-6.7,0-10c-0.8-55.6-3.1-111.1-8.5-166.4c-2-21.2-5-42.3-7.3-63.4
C636.6,733.3,635,730.3,629.2,728.7z M629.2,1253.2c5.3-2.1,7.4-4.5,7.9-9.2c1.6-14.9,4-29.7,5.5-44.6
c6.5-62.4,9.5-124.9,10.3-187.6c0.2-15.6,0.1-15.6-15-15.6c-9,0-8.8-0.6-8.7,8.6c0,2,0,4,0,6c0,75.7,0,151.3,0,227
C629.2,1242.5,629.2,1247.2,629.2,1253.2z M2060,697.5c19-15.9,36.7-29.3,52.3-45.3c3.5-3.5,3.7-6.7,2.4-10.7
c-2.8-8.9-5.4-17.8-8.7-26.5c-3.5-9.3-7.4-18.5-13.7-26.4c-5.3-6.6-8-6.6-13.4-0.4c-1.3,1.5-2.7,3-3.6,4.7
c-8.9,15.7-17.3,31.5-15.3,50.5c0.4,3.6,0.1,7.3,0.1,11C2060,668,2060,681.5,2060,697.5z M613.2,1283.1
c-20.8,13.6-36.9,29.9-53.8,45.4c-2.8,2.6-2.3,5.1-1.5,8.1c4.1,14.7,8.6,29.3,15.3,43.2c2.6,5.4,5.4,10.7,9.8,15
c4.3,4.2,6.2,4.6,10,0.4c10.7-11.9,18.4-26.3,19.7-41.7C614.7,1330.6,613.2,1307.6,613.2,1283.1z M2060,1284.4
c0,19.5,0.7,36.8-0.2,54c-1.1,19.1,6.3,35.2,15.8,50.5c7.4,12.1,12.1,11.7,19.8-0.6c9.7-15.4,14.6-32.7,19.7-50
c1.1-3.7,0-6.2-2.6-8.8C2096.8,1313.6,2079.3,1299.9,2060,1284.4z M611.9,698.2c2.1-3.3,1.3-5.9,1.3-8.4c0.1-16-0.8-32,0.3-47.9
c1.2-18.4-6.2-33.8-15.1-48.6c-7.6-12.7-12.6-12.4-20.4,0.3c-10.1,16.3-15.1,34.6-20.4,52.8c-1,3.6,0.6,5.5,3,7.7
c8.8,8.1,17.3,16.7,26.3,24.5C594.8,685.4,603.3,691.5,611.9,698.2z M1788.2,864c-12.5,3.2-12.6,3.2-13.9,13.8
c-0.9,7.6-1.6,15.2-2.1,22.8c-1.8,26.2-2.9,52.5-3.2,78.8c0,4,0.8,6.3,5.4,6c4.6-0.3,9.3-0.2,14,0c5.1,0.1,6.9-2.1,6.7-7.3
c-0.5-11.3-0.2-22.6-0.7-33.9C1793.2,917.5,1792.3,890.9,1788.2,864z M885.3,1118.4c12.8-3,12.5-3,13.8-13.9
c3.5-29.5,4.6-59.1,5-88.8c0.3-23.5,5.2-18.7-19.8-19.4c-4.4-0.1-5.7,1.5-5.7,5.8C878.8,1040.8,880.3,1079.4,885.3,1118.4z
M885.7,863.5c-0.2,0.9-0.6,1.9-0.7,2.9c-4.3,33.7-6,67.5-6.1,101.4c0,5.5-2.5,12.6,1.3,16.2c3.8,3.6,10.8,1.1,16.3,1.2
c7.9,0.1,8.1-0.1,7.9-8.4c-0.6-33.2-1.5-66.4-5.3-99.4C897.9,865.6,898,865.6,885.7,863.5z M1788.4,1117.5
c4.4-34.4,5.9-68.6,6.5-102.9c0.1-5.6,2.4-12.7-0.9-16.5c-3.9-4.3-11.3-1.4-17.2-1.6c-7.6-0.2-7.7,0.1-7.7,7.6
c0,24.7,1.3,49.3,2.8,73.9c0.6,10.3,2.2,20.5,2.9,30.8c0.3,3.6,1.7,5.5,4.9,6.5C1782.2,1116,1784.4,1117.7,1788.4,1117.5z
M1481.1,1078.1c0-2.7,0.1-4.4,0-6c-0.3-7.8-0.3-8.1-8.2-8.7c-22.9-1.7-45.8-3.3-68.7-4.7c-19.9-1.2-39.9-0.9-59.9-1
c-4.6,0-5.5,2-5.4,6c0.1,3.7,0.3,6.7,5.2,6.2c1.3-0.1,2.7,0,4,0c34.3-0.3,68.4,1.9,102.5,5.6
C1460.6,1076.6,1470.4,1078.5,1481.1,1078.1z M1480,903c-15.3,1.6-30.8,3.6-46.3,4.8c-29.5,2.4-59.1,4.7-88.8,3.9
c-5.1-0.1-6.1,2-6,6.5c0.1,4.1,1.1,6.3,5.6,5.8c1.3-0.2,2.7,0,4,0c42.3,0.3,84.5-2.6,126.7-5.9c4.2-0.3,6.2-1.6,5.8-5.9
C1480.7,909.3,1482,906.3,1480,903z M1192.2,1078.2c3.1-0.2,5.4-0.3,7.6-0.5c37-4.6,74-7.6,111.3-7.8c6,0,12-0.1,18,0
c3.7,0.1,5.3-1.1,5.4-5.1c0.1-6.5,0-6.9-7.3-7c-40.6-0.8-81.1,1.7-121.6,4.6C1193.1,1063.3,1193.2,1063.6,1192.2,1078.2z
M1192.6,903.2c0.2,14.4,0.2,14.4,12.8,16c3,0.4,6,0.6,8.9,0.8c19.3,1.1,38.5,2,57.8,3.2c18.6,1.2,37.2,0.8,55.9,0.8
c6.4,0,6.3-0.5,6.6-6.8c0.2-5.8-3.3-5.5-7.1-5.4c-28.3,0.8-56.5-1.5-84.7-3.6C1226.3,907,1209.8,905.4,1192.6,903.2z M653.6,694.6
c2,12.3,3.8,22.4,5.1,32.6c0.5,3.7,1.7,6.1,4.8,8c14.2,8.8,27.5,18.8,42.2,26.6c2.7,1.4,6.2,6.3,8.9,2.4c2.7-3.7-2.6-6.2-5-8.5
c-19-17.3-36.7-35.8-51.8-56.7C657,698.1,656,697.3,653.6,694.6z M654,1284.6c0.8,0,1.3,0.1,1.5,0c1.4-1.4,2.9-2.7,4-4.3
c14.6-20.8,32.8-38.3,51.2-55.5c1.2-1.1,2.6-2.1,3.6-3.4c1.4-1.7,1.5-3.6-0.2-5.7c-2.9,0.3-5.3,2.1-7.8,3.6
c-10.2,6.3-19.8,13.5-30.5,18.8c-12.9,6.4-19.3,15.7-19.2,30.2C656.6,1273.7,654.9,1279.1,654,1284.6z M2019.1,698.8
c-0.5-0.3-1-0.6-1.5-0.9c-1,1.2-2,2.4-3,3.6c-6.8,9-13,18.4-21.4,26.1c-11,10.1-20.5,21.7-32.4,30.8c-1.9,1.5-3.6,3.2-2.1,5.9
c2,3.5,4.1,0.5,5.7-0.4c15.4-9.7,30.8-19.6,46.1-29.5c2.3-1.5,4-3.1,4.4-6.2C2016.2,718.5,2017.7,708.6,2019.1,698.8z
M2017.1,1283.9c0.7-0.2,1.4-0.5,2.1-0.7c-1.5-10.1-3-20.2-4.4-30.4c-0.4-2.6-1.9-4.1-3.9-5.4c-15.6-10-31.3-20.1-46.9-30
c-1.5-1-3.3-2.8-5-0.5c-1.7,2.4-0.3,4.2,1.7,5.9c4,3.4,8.1,6.9,11.8,10.6C1988.4,1249.3,2005.3,1264.3,2017.1,1283.9z M1044.6,1034
c0.2,0,0.3,0,0.5,0c0-11-0.1-21.9,0.1-32.9c0.1-4.2-2-4.8-5.5-4.9c-4.5-0.1-4.6,2.6-4.6,5.9c0.1,22.3,1.2,44.5,2.4,66.8
c0.1,1,0.2,2,0.4,3c0.2,1.2,1.2,1.8,2.1,1.7c0.8-0.1,1.9-1,2.2-1.7c0.5-1.5,0.7-3.2,0.8-4.9C1043.5,1055.9,1044,1045,1044.6,1034z
M1636.4,1033.8c0.7,0,1.4,0,2.1,0c0-10.6,0-21.2,0-31.8c0-2.7,0.2-5.3-3.6-5.8c-4.7-0.6-6,0.4-6,6.1c-0.1,21.9,0.2,43.7,2.1,65.6
c0.1,1.3,0.1,2.7,0.6,3.9c0.3,0.7,1.5,1.6,2.2,1.5c0.7,0,1.7-1.1,1.9-1.8c0.5-1.9,0.7-3.9,0.7-5.9
C1636.4,1055,1636.4,1044.4,1636.4,1033.8z M1638.5,948.1c-0.2,0-0.5,0-0.7,0c-0.5-11.6-1-23.3-1.6-34.9c-0.1-1.9,0.2-4.8-2.3-4.9
c-3.3-0.1-2.8,3-3,5.2c-0.3,3-0.4,6-0.7,9c-1.5,19-1.4,37.9-1.4,56.9c0,5.3,2,6.8,6.4,5.8c3.8-0.8,3.3-3.6,3.3-6.2
C1638.5,968.8,1638.5,958.4,1638.5,948.1z M1045.1,948c-0.1,0-0.3,0-0.4,0c-0.6-11.9-1.2-23.9-1.9-35.8c-0.1-1.9-0.4-4.3-3.2-3.9
c-2.2,0.3-1.9,2.4-2,4c-1.3,22.5-2.3,45.1-2.5,67.7c0,3.7,1.2,5.4,5.1,5.4c4.1,0.1,5-1.9,5-5.5C1045,969.3,1045.1,958.7,1045.1,948
z M741.9,740.1c-4.9,5.5-6.3,11-8.3,16.2c-0.8,2-0.8,4,1.2,5.3c6.2,4.1,11.6,9.4,19.7,13C751.6,762.1,748.8,750.9,741.9,740.1z
M1931.7,1241.3c11.7-16.6,11.5-18.2-3-28.3c-2.7-1.9-4.9-4.5-9.1-4.9C1921.9,1219.6,1924.7,1230.7,1931.7,1241.3z M1931.6,740
c-6.6,10.9-9.8,22-12.1,33.9c4-0.9,6.1-3.1,8.5-4.8C1943.5,758.3,1943.6,757.2,1931.6,740z M742.1,1241.3
c6.6-11.1,9.8-21.9,12.2-34.3c-8,3.6-13.2,8.8-19.2,12.6c-2,1.3-2.3,3.1-1.6,5.1C735.6,1230.2,736.7,1236.2,742.1,1241.3z
M597.6,1464.4c16.9-6.5,17.3-8.9,14-24.4C605.8,1447.2,601.7,1455,597.6,1464.4z M2061.5,1442.5c-5.1,14.3,5.6,17.3,12.5,22.8
C2072.1,1457,2066.2,1451,2061.5,1442.5z M1781.6,836.9c-1,5.2-3.6,8.5-1.8,12.6C1785.7,846.2,1785.7,846.2,1781.6,836.9z
M895.4,849.3c-1.5-4.5-1.6-7.9-3.8-11.1C887.9,846.3,887.9,846.3,895.4,849.3z M630.1,1284.6c-0.4-5.3,4.1-10.2,1.8-17
C627.4,1273.6,629.9,1279.2,630.1,1284.6z M629.9,697.1c0.2,5.4-3,11.4,2.4,16.6C633.8,707.4,630.4,702.5,629.9,697.1z
M2044.5,1284.5c-1-5.2,1.5-10.3-2.8-15C2040.3,1275.4,2042.4,1279.6,2044.5,1284.5z M2041.9,712.1c4.1-4.8,1.6-10,2.2-14.6
C2043,702.1,2039.8,706.2,2041.9,712.1z M892.2,1143.9c0.4-4.2,3.5-7.3,1.6-11.5C887.6,1136,887.6,1136,892.2,1143.9z
M1779.6,1132.2c-1,2.9-0.4,4.8,0.4,6.5c0.7,1.5,0.4,3.6,2.3,4.6C1785.7,1135.3,1785.7,1135.3,1779.6,1132.2z M729.5,774
c1.5,1.6,2.6,4,5.8,3.2C733.9,774.7,732.2,773.5,729.5,774z M737.9,778.4c-0.2,0.2-0.5,0.4-0.7,0.6c0.2,0.2,0.4,0.6,0.6,0.6
c0.2,0,0.5-0.3,0.7-0.5C738.4,778.8,738.1,778.6,737.9,778.4z"/>
<path class="st1" d="M534.9,1353.5c4.5,11.6,7.4,22.2,12.2,32c4.7,9.6,9.8,18.8,17.1,26.8c10.6,11.6,28.8,15.5,42.7,2.9
c1.4-1.3,3.1-2.4,5.5-4.2c1.4,7.2,1.7,13.2-2.5,19.1c-8.4,11.7-14.8,24.4-20.8,37.4c-1.7,3.7-3.7,6.7-7.3,9.1
c-52.7,33.7-105.3,67.6-158,101.4c-1.1,0.7-2.1,1.7-4.1,1.1c0-195.1,0-390.4,0-585.6c2.1-1.2,4.1-0.9,6-0.8
c21.6,0.5,43.3,1.1,64.9,1.3c5.3,0,6.7,1.8,6.7,6.9c0.5,40,1.4,79.9,3.9,119.8c2.4,38.2,5.6,76.4,10.6,114.3
c4.6,34.7,10.1,69.1,19.2,103c1.1,4,0.6,7.1-2.3,10c-0.9,0.9-1.7,2.1-2.6,3.1c-22.1,25.1-40.6,52.7-57.5,81.5
c-11.6,19.7-19.9,41-28.9,61.9c14.9-23.9,25.7-50.1,41.8-73.5C497.6,1397.6,514.9,1375.3,534.9,1353.5z"/>
<path class="st1" d="M534.6,627.2c-40-41.2-69.6-89-94.2-140.1c0.7,6.2,4,11.2,6.2,16.6c12.7,31.7,30.5,60.6,49.6,88.7
c9.3,13.7,20.4,26.3,31,39.1c3.8,4.5,5,8.7,3.4,14.4c-9.6,35.4-15,71.6-19.6,107.9c-9.4,74.5-12.5,149.3-13.7,224.3c0,1-0.1,2,0,3
c0.6,4.7-1.2,6.5-6.1,6.4c-8.3-0.2-16.7,0.2-25,0.4c-13.7,0.3-27.3,0.4-41,1c-4.8,0.2-6.8-1.2-6.2-6.1c0.3-2,0-4,0-6
c0-187.9,0-375.8,0-563.7c0-3.5,0-7.1,0-10.7c4.2-0.1,6.3,2.3,8.7,3.9c50.4,32.4,100.8,64.9,151.3,97.1c5,3.2,8.4,7.1,10.8,12.4
c5.1,11.2,10.4,22.4,17.8,32.3c4.9,6.6,6.7,13.5,5.1,21.9c-3.6-0.1-5.6-3.1-7.9-5c-12-9.8-28.1-7.7-38.9,2.8
c-12.6,12.2-19,27.9-25.2,43.7C538.8,616.5,538.2,622.2,534.6,627.2z"/>
<path class="st1" d="M2060.9,1411.1c7.4,4.3,12.3,10.3,20.6,11.4c9.8,1.4,17.7-1.4,24.8-7.5c8.5-7.3,13.9-16.9,18.8-26.8
c5.1-10.4,8.6-21.5,12.7-32.5c24.7,24.7,73.3,78.9,107.2,169.7c-1.1-8.8-3.5-17.2-6.8-25.4c-13.2-33.9-29.3-66.4-49.3-96.8
c-12.8-19.5-27.4-37.7-42.4-55.5c-3.4-4-4.8-7.1-3.5-12.3c12.8-48.8,19.1-98.6,24-148.7c5.9-59.7,8.4-119.6,8.9-179.7
c0-1.3,0-2.7,0.1-4c0.2-7,0.2-7.1,7-7.4c20-0.7,40-1.3,60-1.9c2.3-0.1,4.7,0,7-0.1c3.5-0.2,4.8,1.4,4.6,4.8c-0.1,2,0,4,0,6
c0,188.3,0,376.6,0,565c0,3.2,0,6.4,0,9.8c-4.5-0.2-6.9-2.8-9.5-4.5c-47.7-30.6-95.2-61.4-143-91.7c-9-5.7-16-11.6-19.9-22.4
c-3.7-10.1-11.2-18.9-17.1-28.2C2060.8,1426.2,2058.6,1419.6,2060.9,1411.1z"/>
<path class="st1" d="M2244.8,454.3c-23,66.2-59.6,121.8-106.6,172.4c-4.5-11.6-7.7-22.6-12.8-32.9c-4-8.1-8.4-15.8-14.4-22.7
c-12.6-14.4-29.9-17.7-45.6-3.8c-1.1,1-2,2.7-4.6,2.6c-1.4-5.5-1.1-10.5,2-15.7c8.8-14.9,18.9-29,25.8-45c1-2.3,2.7-3.7,4.7-4.9
c52.7-33.9,105.3-67.7,158-101.5c0.5-0.3,1.3-0.2,2.1-0.4c2.3,2.1,1.2,4.9,1.2,7.3c0.1,189.6,0.1,379.3,0.1,568.9
c0,1.7-0.1,3.3,0,5c0.2,3.6-1.7,4.6-4.9,4.5c-5-0.2-10-0.3-15-0.5c-17-0.5-34-1-51-1.4c-7.4-0.2-7.5-0.2-7.6-7.8
c-0.5-35.3-1.1-70.6-3.1-105.9c-2.6-45.9-6.4-91.7-12.8-137.3c-4.2-30.3-9.4-60.5-17.3-90.2c-1.1-4-0.2-6.6,2.5-9.8
c27.4-31.6,51.5-65.6,70.4-102.8c11.3-22.2,21.7-44.9,28.2-69.2C2244.6,460.9,2245.4,458.8,2244.8,454.3z"/>
<path class="st1" d="M553.7,662.8c3.8,0.2,5.2,2.8,6.9,4.6c14.5,15.2,30.3,29,46.9,41.9c4.1,3.2,5.8,6.4,5.8,11.7
c-0.2,85.6-0.1,171.2-0.1,256.8c0,7.5-0.1,7.7-7.6,7.8c-25,0.4-50,0.6-74.9,1.1c-4.3,0.1-5.9-1.2-5.7-5.6c0.4-13,0.2-26,0.5-39
c1.5-60.3,4.7-120.5,11.8-180.4c3.7-31.7,8.3-63.3,15.4-94.5C553,665.6,553.4,664.1,553.7,662.8z"/>
<path class="st1" d="M553.6,1319.7c-22.4-107.4-27.4-215.3-28.7-323.3c3.3-2.5,6.4-1.5,9.3-1.5c23.3,0.3,46.7,0.7,70,1.1
c9.5,0.1,9.2-1.2,9.2,9.2c0,72.3,0,144.7,0,217c0,13.3-0.1,26.7,0.1,40c0,3.6-0.7,6.2-3.8,8.6c-18.7,14.5-36.3,30.3-52.7,47.4
C556.3,1318.5,555.6,1318.7,553.6,1319.7z"/>
<path class="st1" d="M2119.8,1319.2c-6.7-6.5-12.1-12.3-18-17.5c-11.8-10.2-23.8-20.2-36-30.1c-4.2-3.4-5.9-7.1-5.9-12.7
c0.2-84.6,0.1-169.1,0.1-253.7c0-8.4,0.1-8.5,8-8.5c24.3-0.2,48.6-0.4,72.9-0.5c7.5,0,7.7,0,7.6,7.7c-0.3,45.6-1.9,91.2-5.1,136.7
c-2.7,39.9-6.7,79.6-12.5,119.1C2128.1,1279,2124.6,1298.3,2119.8,1319.2z"/>
<path class="st1" d="M2120.4,664.4c12.3,57.5,18.7,115.7,22.8,174.2c3.1,44.9,4.7,89.8,5.2,134.8c0,2.3-0.1,4.7,0.1,7
c0.3,4-1.7,5.2-5.4,5.1c-6.7-0.2-13.3-0.3-20-0.3c-18.3-0.1-36.7-0.2-55-0.4c-8-0.1-8.2-0.1-8.2-8.2c0-52.3,0-104.6,0-157
c0-32,0.2-64-0.2-96c-0.1-6.3,2-10.3,6.8-14.3c17.1-14.3,35.1-27.7,50.2-44.2C2117.5,664.3,2118.3,663.8,2120.4,664.4z"/>
<path class="st1" d="M878.7,1138.5c2,4.9,3.2,9.1,5.3,12.8c4,7.2,10.5,7.1,14.6,0c3-5.3,4.3-11.2,5.2-17.1c0.8-4.7,3.2-7.1,7.8-8.9
c37.6-14.8,76.3-25.6,115.5-34.7c5-1.1,6.6,0.4,7.3,4.9c0.6,3.5,0.7,8.4,5.3,8.5c5.3,0.1,5.5-5.2,5.8-9c0.5-7.1,3.9-10,10.9-11.2
c40-7,79.8-14.9,120.4-18c2.4-0.2,5.9-2.1,6.9,1.5c1,3.7,1.8,7.8,0.4,11.7c-0.7,1.8-2.9,1.2-4.4,1.5c-9.5,1.6-19.1,2.8-28.6,4.6
c-39.6,7.4-78.6,17-117,29.2c-59.9,19-116.5,45.2-170.8,76.4c-41.6,24-80.2,52.3-117.2,83.1c-39.4,32.8-74.3,69.6-105.4,110.2
c-3,3.9-6.2,7.7-9.4,11.6c-0.1,0.2-0.6,0.2-2.1,0.5c0.2-7-1.1-13.6,1.5-20.3c7.6-19.2,12.9-39.2,16.7-59.5
c1.8-9.6,6.2-17.5,12-24.8c17.2-21.7,36.5-41.5,58.8-58.9c3.4,2.2,3.5,6.6,5.4,9.8c4,7,7.8,14.4,17.4,14.4
c9.6,0,13.8-7.2,17.3-14.4c6.5-13.3,9.4-27.7,12.2-42c0.8-3.9,2.2-6.5,5.7-8.6C808.9,1171.4,842.7,1153.5,878.7,1138.5z"/>
<path class="st1" d="M1032.8,891.9c-42-9-82.2-20.4-121.3-35.7c-4.8-1.9-6.9-4.4-7.7-9c-0.9-5.6-2.3-11.1-4.8-16.2
c-1.5-3.2-3.7-5.9-7.3-6.1c-4.1-0.2-6.8,2.5-8.3,6.1c-1.6,3.6-2.8,7.4-4.7,12.2c-11.4-5.2-22.6-10-33.5-15.2
c-23.1-11.1-45.4-23.7-67.3-37.1c-4.3-2.6-6.6-5.5-7.5-10.7c-2.1-11.4-4.6-22.8-8.9-33.7c-2.1-5.3-4.2-10.5-7.8-15
c-7.6-9.4-17.5-9.4-25.3,0.1c-4.2,5.2-6,11.7-9.8,18.2c-11.1-9.9-22.3-18.8-32.2-28.9c-10.2-10.4-19.5-21.8-28.7-33.3
c-5.6-7-8.9-15.3-10.6-24.4c-3.7-19.3-8.6-38.3-16-56.5c-2.9-7.1-1.4-14.1-0.5-21.8c3.9,4.8,7.9,9.4,11.6,14.3
c23.7,32,51,60.7,80.1,87.7c19,17.6,39.4,33.9,59.9,49.7c45,34.7,93.8,63.3,144.7,87.9c39.2,18.9,79.8,34.7,121.6,47.1
c41.2,12.2,82.8,22.3,125.4,28.2c1,0.1,2,0.2,3,0.4c7.3,1.2,10.5,7.7,7,14.2c-1.3,2.3-3.4,1.7-5.3,1.6c-8-0.7-15.9-1.5-23.8-2.4
c-32.8-3.7-65.1-10.2-97.5-15.7c-7.1-1.2-11.5-3.9-11.7-11.6c0-1.3-0.4-2.6-0.8-3.9c-0.8-2.6-1.9-5.3-5.3-4.9
c-2.7,0.3-3.7,2.6-4.4,4.9C1034.4,885.3,1033.7,888.2,1032.8,891.9z"/>
<path class="st1" d="M1953.8,1233.7c3.8,1,5.8,3.4,8,5.3c21,19.1,40.7,39.4,57.6,62.2c3,4.1,4.9,8.4,6,13.5
c4.5,20.5,9.3,40.9,17.1,60.4c1.3,3.1,2.4,6.3,1.8,9.7c-0.6,3.6,2.4,8.9-2.1,10.4c-2.5,0.8-5.4-3.9-7.3-6.7
c-35.7-51.5-81-93.6-129.4-132.6c-40.8-32.9-84.7-61.2-131-85.6c-39.5-20.8-80.6-37.9-123-52c-45.9-15.2-92.6-26.8-140.3-34.3
c-6.2-1-12.4-2.7-18.7-3.1c-6.5-0.4-5.5-4.5-5.2-8.4c0.4-5.4,2.1-7.1,7.5-6.6c8.3,0.7,16.5,1.9,24.8,2.7
c33.2,3.4,65.8,10.1,98.6,15.7c6.3,1.1,9.2,3.4,9.6,10c0.2,3.6,0.3,9.3,5.3,9.6c6,0.4,5.5-5.6,6.3-9.5c0.7-3.9,2.4-5,6.2-4
c39.4,9.7,78.9,19.2,116.7,34.3c4.5,1.8,6.8,4.2,7.5,9.1c0.9,5.9,2.2,11.8,5.1,17.2c1.7,3.1,3.8,5.6,7.6,5.6c3.8,0,6-2.5,7.6-5.6
c0.9-1.8,1.7-3.6,2.3-5.5c2.8-7.7,2.9-7.9,9.9-4.7c32.6,15.3,64.6,31.4,95,50.8c3.4,2.2,5.1,4.7,5.8,8.6
c2.1,12.1,4.9,24.1,9.4,35.7c1.9,5,4.1,9.8,7.5,14c7.5,9.4,17.7,9.4,25.3,0C1948.9,1245.3,1950.9,1239.5,1953.8,1233.7z"/>
<path class="st1" d="M2043.6,585.1c0.9,7.5,1.7,14.1-0.8,20.4c-8,20.2-13.1,41.2-17.6,62.4c-0.9,4.4-2.6,8.3-5.2,11.7
c-17.8,23.8-38.4,45.2-60.6,64.9c-1.5,1.3-3.2,2.3-5.1,3.8c-2.9-5.3-5.1-10.4-8.2-14.9c-8-11.8-19.3-11.8-27.5-0.1
c-7.4,10.6-11.4,23-12.8,35.4c-1.7,14.6-9.4,22.2-21.3,29.3c-25.5,15.2-52.1,28.1-78.7,41.2c-2.4,1.2-5,2-7.3,3.1
c-2.7,1.3-4.3,0.7-5.2-2.2c-1-3.2-1.9-6.4-3.4-9.3c-4.2-7.7-10.8-7.7-15.1-0.1c-2.5,4.4-3.9,9.3-4.5,14.2c-0.8,7-4.2,10.7-11,13.3
c-27.5,10.2-55.4,18.9-83.9,25.7c-10,2.4-20.1,4.8-30,7.5c-3.8,1-5.4,0-6.1-3.9c-0.7-4-0.2-9.9-6.2-9.6c-5.1,0.2-5.3,6-5.4,9.5
c-0.1,7.5-4.3,9-10.5,10.2c-39.6,7.4-79.4,13.9-119.5,18.1c-2.8,0.3-6.7,2.3-8.1-1.3c-1.4-3.6-1.7-7.9-0.3-11.7
c0.9-2.4,3.9-2,6.2-2.4c13.8-2.4,27.6-4.4,41.3-7c42.1-8.2,83.5-19.3,124.2-33.1c57.6-19.6,112.2-45.7,163.9-77.3
c38.1-23.2,73.8-50.1,107.6-79.4c37.9-32.8,72.6-68.6,102.5-109C2037.5,591.3,2039.4,587.8,2043.6,585.1z"/>
<path class="st1" d="M777.2,1163.8c-0.9-7.9,1.2-15.4,2-23c4.4-45.1,6.3-90.3,6.6-135.6c0.1-8.6,0.2-8.7,8.8-8.7
c22-0.1,44-0.1,65.9,0c8.3,0,8.4,0.1,8.5,7.9c0.2,37.7,2,75.2,6.5,112.6c0.5,3.8,0.3,6-3.9,7.5c-31.7,11.3-62.2,25.3-92.6,39.6
C778.8,1164.3,778.4,1164.1,777.2,1163.8z"/>
<path class="st1" d="M776.8,817.8c3.1-0.6,4.7,0.6,6.5,1.4c28.2,13.5,56.6,26.5,86.2,36.8c5.4,1.9,6.7,4.3,6,10.1
c-4.5,36.7-6.3,73.6-6.4,110.6c0,8.1,0.7,8.3-8.1,8.3c-22.6-0.1-45.3,0-67.9-0.1c-7,0-7.1-0.2-7.2-7.3c-0.4-28.6-0.9-57.3-2.7-85.9
c-1.3-21.3-2.9-42.5-5.5-63.7C777.2,824.9,777.1,821.6,776.8,817.8z"/>
<path class="st1" d="M1896.4,1163.9c-3.3,0.5-5.7-1.5-8.4-2.8c-27.2-13.3-54.8-25.7-83.3-36.1c-7.4-2.7-7.2-2.9-6.3-10.6
c4.4-35.8,5.3-71.8,6.4-107.7c0.1-1.7,0.1-3.3,0-5c-0.3-3.8,1.3-5.4,5.2-5.3c6.3,0.2,12.7,0.1,19,0.1c16.7,0,33.3,0,50,0.1
c8.4,0,8.4,0.1,8.6,8.8c0.8,30.3,1.2,60.6,3.3,90.9c1.4,19.9,2.9,39.9,5.3,59.7C1896.5,1158.6,1897.4,1161.2,1896.4,1163.9z"/>
<path class="st1" d="M1896.8,817.7c-1.3,13.9-2.7,27.7-3.9,41.6c-3.5,38.5-4.5,77-5.2,115.6c0,1.7-0.2,3.3-0.1,5
c0.4,4-1.5,5.5-5.2,5.1c-1-0.1-2,0-3,0c-22.6,0-45.3-0.1-67.9,0.2c-5,0-6.8-1.2-6.8-6.6c-0.4-28.3-1.2-56.5-3.6-84.7
c-0.8-8.9-1.6-17.9-2.7-26.8c-0.9-7.6-1.1-7.8,6.5-10.6c23-8.7,45.8-17.8,67.9-28.7c6-2.9,12-5.7,18-8.6
C1892.5,818.3,1894.3,817.4,1896.8,817.7z"/>
<path class="st1" d="M908.3,1109.6c-1-4.4,0.4-8.6,0.8-12.8c2.5-30.9,4-61.8,4-92.9c0-7.4,0.2-7.5,7.9-7.5
c33.3-0.1,66.7-0.2,100-0.2c7.8,0,8.1,0.1,8,7.4c-0.3,22,1,44,2.3,65.9c0.3,4.5-1,6.5-5.5,7.4c-38.7,8.5-76.4,20.9-114.2,32.5
C910.7,1109.8,909.7,1109.6,908.3,1109.6z"/>
<path class="st1" d="M1765.3,1110.4c-9.6-3-18.1-5.6-26.6-8.4c-29.1-9.6-58.7-17.4-88.5-24.5c-8.9-2.1-8.1-1.4-7.6-10.4
c1.2-21,1.6-42,2.3-62.9c0.2-7.7,0.2-7.9,7.5-7.9c33.3,0,66.7,0.1,100,0.2c7.8,0,8.2,0.3,8.2,7.7c0.2,34,1.7,68,4.8,101.8
C1765.4,1106.9,1765.3,1107.9,1765.3,1110.4z"/>
<path class="st1" d="M908.8,872.2c2.8-1.1,5,0.8,7.5,1.5c36.1,10.3,71.6,22.9,108.5,30.5c5.5,1.1,6.7,3.6,6.4,8.8
c-1.3,21.9-2.6,43.9-2.1,65.9c0.1,4.7-1.2,6.5-6.1,6.4c-34.7-0.2-69.3-0.2-104-0.1c-4.1,0-6.1-1-6-5.6c0.4-35.6-2-71.2-4.8-106.7
C908.2,872.8,908.6,872.5,908.8,872.2z"/>
<path class="st1" d="M1765.6,872.4c-0.9,10.7-1.8,20.9-2.5,31.2c-1.7,24.6-2.4,49.2-2.6,73.8c-0.1,7.7-0.2,7.8-7.7,7.8
c-33.6,0.1-67.2,0.1-100.8,0.1c-6.9,0-6.9-0.2-7.1-7.1c-0.8-21.9-1.3-43.9-2.6-65.8c-0.3-5.4,1.8-7,6.2-8
c14.5-3.5,29-7.1,43.5-10.8c22.5-5.7,44.5-13.5,66.7-20.3C1760.5,872.7,1762.3,871.8,1765.6,872.4z"/>
<path class="st1" d="M725.8,1192.5c-5.9-34.3-8.8-67.4-11-100.6c-1.8-27.6-2.2-55.2-3-82.8c-0.1-2.3,0.1-4.7-0.1-7
c-0.4-4.4,1.6-5.8,5.9-5.7c9,0.2,18,0,27,0c6.7,0,13.3-0.1,20,0c7.2,0.1,7.4,0.2,7.3,7.1c-0.5,23.6-1,47.2-2,70.8
c-1.2,27.2-3.3,54.4-6.7,81.5c-0.2,1.3-0.3,2.6-0.4,4c-0.4,9.2-3.4,16-13,19.7C741.7,1182.7,734.5,1187.7,725.8,1192.5z"/>
<path class="st1" d="M1946.2,1191.7c-10.1-5.6-20.4-11.5-30.8-17.2c-3.3-1.8-3.1-5-3.5-7.8c-1.4-10.2-2.4-20.5-3.5-30.7
c-4.7-43.8-6-87.7-6.7-131.6c-0.1-7.7,0.1-7.8,7.7-7.8c15.3-0.1,30.6,0.1,46-0.1c4.6-0.1,6.5,1.1,6.4,6.1
c-0.5,34-1.2,67.9-3.7,101.8c-2,26.9-4.5,53.7-8.9,80.4C1948.7,1187,1949.1,1189.5,1946.2,1191.7z"/>
<path class="st1" d="M1947.7,789.1c5.8,32.7,8.6,63.8,10.8,95c2.1,30.6,2.8,61.2,3.1,91.9c0.1,9.6,0.3,8.9-8.3,8.9
c-15,0-30-0.1-45,0.2c-5.3,0.1-6.9-1.8-6.7-7c0.9-25.3,1.3-50.6,2.5-75.9c1.3-27.3,3.9-54.4,6.9-81.6c0.9-8.3,3.3-14.2,11.5-17.8
C1930.8,799.2,1938.6,794.1,1947.7,789.1z"/>
<path class="st1" d="M726,789.2c11.3,6.2,21.7,11.9,32.2,17.5c2.9,1.6,2.8,4.2,3.2,6.8c2.7,15.8,4.1,31.7,5.5,47.6
c3.3,37.2,4.5,74.5,5,111.8c0,2.3,0,4.7,0.1,7c0.1,3.1-0.8,5-4.4,5c-17-0.1-34-0.1-51-0.1c-3.3,0-4.8-1.5-4.7-4.8
c0.2-5.7,0-11.3,0.2-17c1-42.6,2.7-85.2,7.3-127.7C720.9,820.5,722.6,805.7,726,789.2z"/>
<path class="st1" d="M1625.1,910.5c-0.6,9.4-1.4,18-1.7,26.5c-0.4,13-0.4,26-0.6,38.9c-0.1,9.3-0.2,9.4-9.2,9.4
c-30.6,0.1-61.3,0.1-91.9,0.1c-8.3,0-16.7-0.1-25,0.1c-4.2,0.1-5.9-1.3-5.9-5.7c0-15-0.1-30-0.7-44.9c-0.2-4.8,2.5-5.5,6-6
c10.9-1.3,21.8-3.6,32.7-3.7c23.9-0.2,46.7-6.5,70-10C1607.1,913.9,1615.2,912.2,1625.1,910.5z"/>
<path class="st1" d="M1625.2,1071.1c-23.5-4.2-45.7-8.1-67.9-12.1c-7.9-1.4-15.8-2.4-23.8-2.5c-12.7-0.1-25.1-2.6-37.7-4
c-4-0.4-5.8-1.6-5.6-6.1c0.5-14.6,0.6-29.3,0.6-43.9c0-4.5,1-6.5,6.1-6.5c39.9,0.2,79.9,0.2,119.8,0.1c5,0,6.1,2.1,6.1,6.6
C1622.9,1025,1623,1047.3,1625.2,1071.1z"/>
<path class="st1" d="M1049.3,1070.4c-1-5.9-0.1-11.8,0.3-17.7c0.9-16.3,1.4-32.6,1.4-48.9c0-7.3,0.2-7.5,6.8-7.5
c39.3-0.1,78.5-0.1,117.8-0.1c7,0,7.1,0.2,7.2,7.2c0.2,14,0,27.9,0.5,41.9c0.2,5.2-1.5,6.6-6.6,7c-16.9,1.3-33.8,2.7-50.6,5.2
c-23.3,3.4-46.7,6.8-69.7,12.3C1054.2,1070.3,1051.9,1071.4,1049.3,1070.4z"/>
<path class="st1" d="M1049.8,910.2c22.1,5.7,44.8,8.7,67.4,12.5c16.4,2.8,33.1,3.3,49.6,5.2c16.4,1.9,16.4,1.5,16.3,17.4
c-0.1,11-0.3,22-0.2,32.9c0.1,4.9-1,7.2-6.6,7.2c-39.9-0.2-79.9-0.2-119.8-0.1c-3.4,0-5.7-0.6-5.6-4.8c0.6-22.6-1.3-45.2-1.9-67.8
C1049,912,1049.4,911.4,1049.8,910.2z"/>
<path class="st1" d="M1260.5,985.4c-21.6,0-43.2,0-64.9,0c-8,0-8.2-0.1-8.1-8.4c0.2-13.6,0.6-27.3,0.8-40.9c0.1-4.2,1.4-5.6,6-5.1
c41.7,4.1,83.5,6.2,125.5,6.2c2.3,0,4.7,0,7,0c7.5,0.2,7.5,0.2,7.6,7.9c0.1,10.3,0.1,20.6,0.1,30.9c0,9.3,0,9.4-9.1,9.4
C1303.8,985.4,1282.2,985.4,1260.5,985.4z"/>
<path class="st1" d="M1485,931.5c0.3,16.2,0.5,32.7,0.8,49.3c0.1,3.4-1.7,4.8-4.9,4.6c-2-0.1-4,0-6,0c-41.6,0-83.2,0-124.9,0
c-2,0-4-0.1-6,0c-4,0.3-5.1-1.8-5-5.5c0.1-11.7,0-23.3,0.1-35c0.1-7.6,0.1-7.8,7.6-7.8c17.3,0.2,34.6-0.6,51.9-1.1
c26.3-0.7,52.5-3.1,78.7-5.1C1479.6,930.9,1481.8,930,1485,931.5z"/>
<path class="st1" d="M1485,1051.1c-35.4-3-69.9-5.3-104.5-6.4c-11.6-0.4-23.3-0.5-34.9-0.4c-4.4,0-6.7-0.8-6.5-6
c0.4-12,0.3-24,0-36c-0.1-4.8,1.6-6.3,6.3-6.2c45,0.1,89.9,0.1,134.9,0c3.5,0,5.8,0.7,5.7,4.8
C1485.5,1017.5,1485.3,1034.1,1485,1051.1z"/>
<path class="st1" d="M1188.4,1050.6c-0.3-17.2-0.5-33.7-0.8-50.3c0-2.9,1.6-4.1,4.4-4.1c1.7,0,3.3,0,5,0c42.6,0,85.1,0,127.7,0
c1.7,0,3.3,0.1,5,0c3.3-0.2,4.9,1.2,4.9,4.6c-0.1,13-0.1,25.9,0,38.9c0,3.6-1.7,4.7-4.9,4.5c-0.7,0-1.3,0-2,0
C1281.6,1043.7,1235.9,1047.2,1188.4,1050.6z"/>
<path class="st1" d="M2010.7,1230.1c-5.1-1.9-9-5.1-13.1-7.7c-9.9-6.1-19.6-12.6-29.8-18.3c-4.8-2.6-5.5-5.6-4.5-10.4
c3.9-20,6.2-40.1,8.2-60.4c4-41.2,5.8-82.4,6.1-123.8c0-3,0.1-6,0-9c-0.1-3,1.1-3.8,4.1-4.1c16.5-1.5,16.7-1.4,16.8,14.7
c0.4,66.4,3.9,132.5,10.7,198.6C2009.9,1216.3,2011.6,1222.7,2010.7,1230.1z"/>
<path class="st1" d="M2011.2,752.5c-1.6,16.9-3.2,33.8-4.8,50.6c-3,30.8-4.6,61.7-6.1,92.6c-1.3,27.3-1.5,54.5-1.8,81.8
c-0.1,7.1-0.2,7.3-7.1,7.3c-16.3,0-13.6,1.3-13.7-12.8c-0.5-54.3-3.2-108.4-10.6-162.2c-1.1-7.9-2.5-15.8-3.9-23.6
c-0.7-3.6-0.1-6,3.5-8.2c13.4-8,26.6-16.4,39.8-24.7C2007.6,752.8,2008.5,751.7,2011.2,752.5z"/>
<path class="st1" d="M662.1,1231.5c1.9-20.4,3.7-38.6,5.4-56.8c3-31.9,4.7-63.8,6-95.8c1-24.6,1.6-49.3,1.6-74
c0-8.3,0.1-8.3,7.9-8.4c1.3,0,2.7,0,4,0c8.6,0.1,8.7,0.1,8.8,8.7c0.4,40,1.8,80,5.4,119.8c2,22.9,4.4,45.8,9,68.3
c1.1,5.5,0.2,8.5-5.1,11c-12.7,6-23.9,14.5-35.5,22.2C667.7,1227.9,665.8,1229.1,662.1,1231.5z"/>
<path class="st1" d="M662.9,752.1c2.6-0.9,3.9,1,5.4,2.1c12.4,8.4,24.6,17.1,38,23.9c3.3,1.7,5,3.6,4.2,7.7
c-11.3,61.8-13.7,124.4-14.8,187c-0.2,13.5,2.2,12.1-12.3,12.1c-8.4,0-8.5-0.1-8.5-8.9c0.1-52.7-2.6-105.3-6.4-157.8
c-1.5-20.6-4-41.1-6-61.7C662.5,755.1,661.9,753.4,662.9,752.1z"/>
<path class="st1" d="M2044.3,728.6c0,3.4,0,6.3,0,9.2c0,78.9,0,157.9,0,236.8c0,1.7-0.1,3.3,0,5c0.4,3.9-1.3,5.4-5.2,5.3
c-5.9-0.2-11.9,0-18.3,0c0-26.3,0.3-51.6,1.3-76.8c1.7-42.3,4.4-84.5,8.9-126.5c1.6-14.9,3.9-29.7,5.6-44.6
C2037.2,732.4,2038.9,729.7,2044.3,728.6z"/>
<path class="st1" d="M2044.3,1253c-5.6-1.2-7-4.1-7.7-8.6c-3.1-19.4-5.2-38.9-7-58.5c-5.6-59-8.7-118.2-8.7-177.6
c0-13.6-1.4-11.7,11.5-11.8c13.3,0,11.9-1.5,11.9,11.5c0,78.6,0,157.2,0,235.8C2044.3,1246.6,2044.3,1249.3,2044.3,1253z"/>
<path class="st1" d="M629.2,728.7c5.8,1.6,7.4,4.6,7.9,9c2.3,21.2,5.3,42.2,7.3,63.4c5.3,55.3,7.7,110.8,8.5,166.4
c0,3.3,0.1,6.7,0,10c-0.1,7.5-0.1,7.5-7.7,7.7c-16,0.5-16,0.5-16-15.3c0-76.5,0-153.1,0-229.6C629.2,736.7,629.2,733,629.2,728.7z"
/>
<path class="st1" d="M629.2,1253.2c0-6,0-10.6,0-15.3c0-75.7,0-151.3,0-227c0-2,0-4,0-6c-0.1-9.2-0.3-8.7,8.7-8.6
c15.1,0,15.2,0,15,15.6c-0.7,62.7-3.8,125.2-10.3,187.6c-1.5,14.9-3.9,29.7-5.5,44.6C636.6,1248.7,634.5,1251.1,629.2,1253.2z"/>
<path class="st1" d="M2060,697.5c0-15.9,0-29.5,0-43c0-3.7,0.3-7.3-0.1-11c-2-19,6.4-34.8,15.3-50.5c1-1.7,2.3-3.2,3.6-4.7
c5.4-6.2,8.1-6.2,13.4,0.4c6.3,7.8,10.3,17.1,13.7,26.4c3.2,8.7,5.9,17.6,8.7,26.5c1.3,4,1,7.1-2.4,10.7
C2096.7,668.2,2079,681.6,2060,697.5z"/>
<path class="st1" d="M613.2,1283.1c0,24.5,1.5,47.6-0.5,70.3c-1.4,15.4-9,29.8-19.7,41.7c-3.8,4.2-5.7,3.9-10-0.4
c-4.3-4.3-7.2-9.6-9.8-15c-6.7-13.8-11.2-28.4-15.3-43.2c-0.8-3-1.3-5.6,1.5-8.1C576.3,1313,592.4,1296.7,613.2,1283.1z"/>
<path class="st1" d="M2060,1284.4c19.3,15.5,36.8,29.3,52.5,45.2c2.6,2.6,3.7,5.1,2.6,8.8c-5.1,17.2-10,34.5-19.7,50
c-7.7,12.2-12.4,12.7-19.8,0.6c-9.5-15.4-16.9-31.4-15.8-50.5C2060.7,1321.2,2060,1303.9,2060,1284.4z"/>
<path class="st1" d="M611.9,698.2c-8.5-6.7-17.1-12.8-25-19.7c-9.1-7.8-17.5-16.4-26.3-24.5c-2.3-2.2-4-4.1-3-7.7
c5.3-18.2,10.3-36.5,20.4-52.8c7.8-12.7,12.8-13,20.4-0.3c8.9,14.9,16.3,30.2,15.1,48.6c-1,15.9-0.2,31.9-0.3,47.9
C613.2,692.2,614,694.9,611.9,698.2z"/>
<path class="st1" d="M1788.2,864c4.1,27,4.9,53.5,6.1,80.1c0.5,11.3,0.2,22.6,0.7,33.9c0.2,5.2-1.6,7.4-6.7,7.3
c-4.7-0.1-9.3-0.3-14,0c-4.6,0.3-5.5-2-5.4-6c0.2-26.3,1.3-52.5,3.2-78.8c0.5-7.6,1.2-15.2,2.1-22.8
C1775.6,867.2,1775.7,867.2,1788.2,864z"/>
<path class="st1" d="M885.3,1118.4c-5.1-39.1-6.6-77.6-6.6-116.3c0-4.2,1.3-5.9,5.7-5.8c25,0.7,20.1-4.1,19.8,19.4
c-0.4,29.6-1.5,59.3-5,88.8C897.9,1115.4,898.1,1115.4,885.3,1118.4z"/>
<path class="st1" d="M885.7,863.5c12.4,2.1,12.2,2.1,13.5,13.8c3.8,33,4.6,66.2,5.3,99.4c0.2,8.3,0,8.5-7.9,8.4
c-5.6-0.1-12.6,2.4-16.3-1.2c-3.8-3.6-1.3-10.7-1.3-16.2c0.1-33.9,1.8-67.7,6.1-101.4C885.1,865.4,885.4,864.5,885.7,863.5z"/>
<path class="st1" d="M1788.4,1117.5c-4,0.3-6.2-1.5-8.7-2.2c-3.2-0.9-4.6-2.9-4.9-6.5c-0.7-10.3-2.2-20.5-2.9-30.8
c-1.5-24.6-2.8-49.2-2.8-73.9c0-7.5,0.1-7.8,7.7-7.6c5.9,0.2,13.3-2.7,17.2,1.6c3.4,3.7,1,10.9,0.9,16.5
C1794.3,1048.8,1792.8,1083.1,1788.4,1117.5z"/>
<path class="st1" d="M1481.1,1078.1c-10.7,0.3-20.5-1.5-30.3-2.6c-34.1-3.7-68.2-5.9-102.5-5.6c-1.3,0-2.7-0.1-4,0
c-4.9,0.5-5.1-2.5-5.2-6.2c-0.1-4,0.7-6,5.4-6c20,0.1,39.9-0.2,59.9,1c22.9,1.4,45.8,3,68.7,4.7c7.9,0.6,7.8,0.9,8.2,8.7
C1481.2,1073.7,1481.1,1075.4,1481.1,1078.1z"/>
<path class="st1" d="M1480,903c2,3.3,0.7,6.3,1,9.2c0.4,4.3-1.6,5.6-5.8,5.9c-42.2,3.3-84.3,6.2-126.7,5.9c-1.3,0-2.7-0.1-4,0
c-4.5,0.5-5.5-1.7-5.6-5.8c-0.1-4.5,0.9-6.6,6-6.5c29.7,0.7,59.2-1.5,88.8-3.9C1449.3,906.5,1464.7,904.6,1480,903z"/>
<path class="st1" d="M1192.2,1078.2c1-14.7,0.9-15,13.5-15.8c40.5-2.9,80.9-5.4,121.6-4.6c7.3,0.1,7.4,0.5,7.3,7
c-0.1,4-1.7,5.2-5.4,5.1c-6-0.2-12-0.1-18,0c-37.3,0.2-74.3,3.2-111.3,7.8C1197.6,1078,1195.3,1078,1192.2,1078.2z"/>
<path class="st1" d="M1192.6,903.2c17.2,2.2,33.7,3.8,50.2,5c28.2,2.1,56.4,4.4,84.7,3.6c3.8-0.1,7.3-0.4,7.1,5.4
c-0.3,6.3-0.2,6.8-6.6,6.8c-18.6,0-37.2,0.4-55.9-0.8c-19.2-1.2-38.5-2.1-57.8-3.2c-3-0.2-6-0.4-8.9-0.8
C1192.8,917.6,1192.8,917.6,1192.6,903.2z"/>
<path class="st1" d="M653.6,694.6c2.5,2.7,3.4,3.5,4.2,4.6c15.1,20.9,32.8,39.4,51.8,56.7c2.5,2.3,7.7,4.8,5,8.5
c-2.7,3.9-6.2-1-8.9-2.4c-14.7-7.8-28.1-17.9-42.2-26.6c-3.1-1.9-4.3-4.4-4.8-8C657.4,717,655.5,706.9,653.6,694.6z"/>
<path class="st1" d="M654,1284.6c0.9-5.5,2.7-11,2.6-16.4c-0.1-14.4,6.3-23.8,19.2-30.2c10.6-5.3,20.3-12.5,30.5-18.8
c2.5-1.5,4.9-3.3,7.8-3.6c1.6,2.1,1.5,4,0.2,5.7c-1,1.3-2.4,2.3-3.6,3.4c-18.4,17.3-36.5,34.8-51.2,55.5c-1.1,1.6-2.6,2.9-4,4.3
C655.3,1284.8,654.8,1284.6,654,1284.6z"/>
<path class="st1" d="M2019.1,698.8c-1.4,9.8-3,19.6-4.2,29.5c-0.4,3-2.2,4.7-4.4,6.2c-15.3,9.9-30.7,19.7-46.1,29.5
c-1.5,1-3.7,4-5.7,0.4c-1.5-2.6,0.2-4.4,2.1-5.9c11.9-9.2,21.5-20.7,32.4-30.8c8.4-7.7,14.5-17.2,21.4-26.1c0.9-1.2,2-2.4,3-3.6
C2018.1,698.2,2018.6,698.5,2019.1,698.8z"/>
<path class="st1" d="M2017.1,1283.9c-11.8-19.6-28.7-34.6-44.6-50.5c-3.7-3.7-7.8-7.2-11.8-10.6c-1.9-1.7-3.4-3.4-1.7-5.9
c1.7-2.4,3.5-0.5,5,0.5c15.7,10,31.3,20,46.9,30c2,1.3,3.5,2.9,3.9,5.4c1.4,10.1,2.9,20.2,4.4,30.4
C2018.5,1283.4,2017.8,1283.7,2017.1,1283.9z"/>
<path class="st1" d="M1044.6,1034c-0.6,11-1.1,21.9-1.7,32.9c-0.1,1.6-0.3,3.3-0.8,4.9c-0.3,0.8-1.4,1.6-2.2,1.7
c-0.9,0.1-1.9-0.5-2.1-1.7c-0.1-1-0.3-2-0.4-3c-1.2-22.2-2.3-44.5-2.4-66.8c0-3.2,0-6,4.6-5.9c3.5,0.1,5.6,0.7,5.5,4.9
c-0.2,11-0.1,21.9-0.1,32.9C1044.9,1034,1044.8,1034,1044.6,1034z"/>
<path class="st1" d="M1636.4,1033.8c0,10.6,0,21.2,0,31.8c0,2-0.3,4-0.7,5.9c-0.2,0.8-1.2,1.8-1.9,1.8c-0.8,0-1.9-0.8-2.2-1.5
c-0.5-1.2-0.5-2.6-0.6-3.9c-1.9-21.8-2.2-43.7-2.1-65.6c0-5.7,1.3-6.7,6-6.1c3.9,0.5,3.6,3.1,3.6,5.8c0,10.6,0,21.2,0,31.8
C1637.8,1033.8,1637.1,1033.8,1636.4,1033.8z"/>
<path class="st1" d="M1638.5,948.1c0,10.3,0,20.6,0,31c0,2.6,0.5,5.4-3.3,6.2c-4.4,0.9-6.4-0.5-6.4-5.8c0-19,0-38,1.4-56.9
c0.2-3,0.4-6,0.7-9c0.2-2.2-0.3-5.4,3-5.2c2.5,0.1,2.2,3,2.3,4.9c0.6,11.6,1.1,23.3,1.6,34.9C1638,948.1,1638.2,948.1,1638.5,948.1
z"/>
<path class="st1" d="M1045.1,948c0,10.6-0.1,21.2,0,31.9c0,3.6-0.9,5.5-5,5.5c-3.9-0.1-5.1-1.7-5.1-5.4c0.1-22.6,1.2-45.1,2.5-67.7
c0.1-1.6-0.1-3.7,2-4c2.8-0.4,3.1,1.9,3.2,3.9c0.7,11.9,1.3,23.9,1.9,35.8C1044.8,948,1044.9,948,1045.1,948z"/>
<path class="st1" d="M741.9,740.1c7,10.8,9.7,22,12.6,34.5c-8.1-3.6-13.5-8.9-19.7-13c-2-1.3-2-3.3-1.2-5.3
C735.6,751.1,737,745.6,741.9,740.1z"/>
<path class="st1" d="M1931.7,1241.3c-6.9-10.6-9.7-21.7-12.1-33.3c4.2,0.4,6.4,3.1,9.1,4.9
C1943.2,1223.1,1943.4,1224.7,1931.7,1241.3z"/>
<path class="st1" d="M1931.6,740c12,17.2,11.9,18.3-3.6,29.1c-2.4,1.7-4.5,3.9-8.5,4.8C1921.8,761.9,1925,750.9,1931.6,740z"/>
<path class="st1" d="M742.1,1241.3c-5.3-5.2-6.5-11.1-8.6-16.5c-0.8-2-0.5-3.8,1.6-5.1c6-3.9,11.2-9.1,19.2-12.6
C751.8,1219.5,748.7,1230.3,742.1,1241.3z"/>
<path class="st1" d="M599.6,517.5c0.6,0.1,1.4,0.1,1.9,0.4c12.6,7.1,13,7.9,10.7,24.4c-5.9-7.9-11-15.2-13.4-24.1
C599,518,599.3,517.7,599.6,517.5z"/>
<path class="st1" d="M597.6,1464.4c4-9.4,8.1-17.2,14-24.4C614.9,1455.5,614.5,1457.9,597.6,1464.4z"/>
<path class="st1" d="M2072.4,516.4c2.2,0.6,2.3,2,1.3,3.7c-3.8,6.1-7.7,12.1-11.9,18.6c-2.8-10.6-2.1-12.7,5.3-17.8
c1.9-1.3,4.2-2,5.3-4.3L2072.4,516.4z"/>
<path class="st1" d="M2061.5,1442.5c4.7,8.5,10.5,14.5,12.5,22.8C2067.2,1459.8,2056.4,1456.8,2061.5,1442.5z"/>
<path class="st1" d="M1781.6,836.9c4.2,9.3,4.2,9.3-1.8,12.6C1777.9,845.4,1780.6,842.1,1781.6,836.9z"/>
<path class="st1" d="M895.4,849.3c-7.5-2.9-7.5-2.9-3.8-11.1C893.8,841.3,893.9,844.7,895.4,849.3z"/>
<path class="st1" d="M630.1,1284.6c-0.2-5.4-2.7-11,1.8-17C634.2,1274.4,629.7,1279.3,630.1,1284.6z"/>
<path class="st1" d="M629.9,697.1c0.5,5.4,3.9,10.3,2.4,16.6C626.9,708.5,630.1,702.5,629.9,697.1z"/>
<path class="st1" d="M2044.5,1284.5c-2.2-5-4.3-9.2-2.8-15C2046.1,1274.3,2043.5,1279.4,2044.5,1284.5z"/>
<path class="st1" d="M2041.9,712.1c-2.2-6,1.1-10.1,2.2-14.6C2043.5,702.2,2046,707.4,2041.9,712.1z"/>
<path class="st1" d="M892.2,1143.9c-4.6-7.9-4.6-7.9,1.6-11.5C895.7,1136.5,892.6,1139.7,892.2,1143.9z"/>
<path class="st1" d="M1779.6,1132.2c6.1,3.1,6.1,3.1,2.7,11.1c-1.9-1-1.7-3.1-2.3-4.6C1779.2,1137,1778.6,1135.1,1779.6,1132.2z"/>
<path class="st1" d="M1934.8,1202.5c4.1-0.6,6.5,2.5,11.4,4.9c-6.3,0.5-8.2-3.5-11.3-5.1L1934.8,1202.5z"/>
<path class="st1" d="M1935.8,777c3.1,0.9,4.5-3.5,7.8-2.9c-1.8,4-4.7,3.9-7.9,2.8L1935.8,777z"/>
<path class="st1" d="M729.5,774c2.7-0.5,4.4,0.6,5.8,3.2C732.1,778,731,775.7,729.5,774z"/>
<path class="st1" d="M734,1202.5c2.3,4.6-1.2,4.8-4.2,5.9c-0.6-3.7,3.7-3.3,4.2-5.7C734,1202.7,734,1202.5,734,1202.5z"/>
<path class="st1" d="M1935.2,780.3c-0.4,0.2-0.8,0.5-1.2,0.7c0.3,0,0.6,0.1,0.8,0c0.2-0.1,0.4-0.3,0.6-0.5
C1935.5,780.5,1935.2,780.3,1935.2,780.3z"/>
<path class="st1" d="M740,1203.1c-1.9-0.8-4.2,2.9-6-0.5c0,0,0,0.2,0,0.1C736,1202.8,738,1202.9,740,1203.1L740,1203.1z"/>
<path class="st1" d="M1935.7,777c1.4,1.3,0.9,2.4-0.5,3.3c0,0,0.3,0.3,0.3,0.3c0.1-1.2,0.3-2.3,0.4-3.5
C1935.8,777,1935.7,777,1935.7,777z"/>
<path class="st1" d="M737.9,778.4c0.2,0.2,0.4,0.5,0.6,0.7c-0.2,0.2-0.5,0.5-0.7,0.5c-0.2,0-0.4-0.4-0.6-0.6
C737.4,778.8,737.7,778.6,737.9,778.4z"/>
<path class="st1" d="M740,1203.1c0-0.3,0-0.8-0.2-1c-0.8-0.8-0.7-1.1,0.4-0.8C739.5,1201.9,739.4,1202.4,740,1203.1
C740,1203.1,740,1203.1,740,1203.1z"/>
<path class="st1" d="M1934.9,1202.3c0-0.2-0.1-0.3-0.1-0.5c0,0-0.1,0.1-0.1,0.1c0.1,0.2,0.1,0.4,0.2,0.6
C1934.8,1202.5,1934.9,1202.3,1934.9,1202.3z"/>
<path class="st1" d="M2072.4,516.6c0.7,0.4,1.4,0.8,2.1,1.2c-0.2,0.2-0.4,0.7-0.6,0.7c-1.1-0.2-1.5-1-1.4-2
C2072.4,516.4,2072.4,516.6,2072.4,516.6z"/>
<path class="st1" d="M598.7,518.3c-1-0.2-1.8-0.8-1-1.6c0.8-0.9,1.4,0,1.8,0.8C599.3,517.7,599,518,598.7,518.3z"/>
</g>
</svg>
<p class="hero-tag">distributed structured concurrency</p>
</div>
<p class="hero-sub">a multi-processing runtime built on (and shaped
entirely like) <a class="reference external"
href="https://github.com/python-trio/trio">trio</a>.</p>

View File

@ -0,0 +1,456 @@
<svg class="hero-logo" role="img" aria-label="tractor" version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
viewBox="0 0 2670 1980" style="enable-background:new 0 0 2670 1980;" xml:space="preserve">
<style type="text/css">
.st0{fill:currentColor;}
.st1{fill:none;}
</style>
<g>
<path class="st0" d="M2275.3,1000.1c-0.1,2.3,0,4.7,0,7c0,189.3,0,378.7,0,568c0,6,0.1,12.1-0.8,17.9c-2.4,15.2-12.9,17.4-23.6,10
c-45.4-31-91.2-61.4-136.9-91.9l-30.1-20.3c-9.8-6.3-19.2-13.2-29.2-19.3c-6.7-4.1-9.8-9.7-9.9-17.2c-0.1-9.3-0.6-18.7-0.3-28
c0.2-7.3-1.7-13-6.5-19c-18.6-23.3-36.2-47.3-56.9-69c-18-18.8-36.7-36.8-56.4-53.8c-31.8-27.5-65.4-52.7-101-74.9
c-32.7-20.4-66.8-38.4-102.2-54c-50.7-22.4-103.2-39.2-157.1-51.5c-36.7-8.4-73.8-14.4-111.2-18.5c-34.5-3.8-69-6.5-103.7-5.8
c-23.1,0.5-46.2-1.3-69.3,0.4c-22.9,1.8-45.9,3-68.7,5.6c-72.7,8.4-143.7,24.6-212.1,50.6c-65.5,24.8-127.4,57-184.9,97.4
c-46.8,32.8-90.1,69.8-129.5,111.1c-18.1,19-35.1,39.2-50.5,60.6c-3.9,5.5-5.5,11-5.3,17.6c0.4,9.3,0,18.7,0,28
c0,9.3-3.1,16.4-11.8,21.4c-10.4,5.9-19.9,13.4-30.1,19.8l-25.4,17.2c-11.6,7.8-23.3,15.5-34.9,23.3c-34.3,23-68.6,46-102.9,69
c-14.6,9.7-23.6,5.5-25.2-12.1c-0.5-5.3-0.5-10.7-0.5-16c0-189.7,0-379.3,0-569l-0.1-26.1c0.2-190.3,0.1-380.7,0.2-571
c0-5.7,0.4-11.3,0.5-17c0.1-3.7,1.2-7.3,3-10.5c3.4-5.9,9.3-7.7,15.4-4.8c2.7,1.3,5.1,3.1,7.6,4.8c44.3,29.6,88.5,59.3,132.8,88.9
l29.7,20.1c10.8,6.6,20.9,14.3,31.7,20.9c6.7,4.1,9.7,9.8,9.8,17.3c0.2,11,0.4,22,0.2,33c-0.1,5.2,1.2,9.6,4.2,13.9
c15.4,22.2,33,42.8,51.6,62.2c53.9,56.4,114.1,105.1,181.4,144.6c38.4,22.6,78.3,42.4,119.9,58.6c40,15.6,80.9,28.3,122.8,38
c52.7,12.1,106,19.3,160,22.6c18.3,1.1,36.6,2.1,54.9,1.5c17.7-0.6,35.5,1.1,53.2-0.1c17.6-1.1,35.2-2.3,52.8-3.6
c35.9-2.5,71.4-8.5,106.7-15.1c53.4-9.9,105.5-25,156-45.1c53.8-21.5,105.1-48.2,153.2-80.9c43.4-29.5,83.6-62.7,121.1-99.3
c27.3-26.6,50.3-56.8,73.9-86.6c2.9-3.6,3.6-7.6,3.6-12.1c-0.1-10.7,0.2-21.3,0.4-32c0.1-7.5,3.2-13.1,9.9-17.2
c10.2-6.2,19.9-13.4,30-19.8l23.2-16.3c4.2-1.3,7.4-4.3,11-6.7c42.9-28.7,85.8-57.4,128.7-86.1c1.1-0.7,2.3-1.4,3.3-2.3
c10.9-9.1,21.4-4.7,23.8,10.7c0.8,5.6,0.6,11.3,0.6,17c0,189.3,0,378.7,0,568L2275.3,1000.1z M740,1203.1c-2-0.2-4-0.3-6-0.5
c-0.5,2.5-4.8,2-4.2,5.8c2.9-1,6.4-1.2,4.2-5.8C735.8,1206,738.1,1202.3,740,1203.1c-0.6-0.7-0.5-1.3,0.2-1.7
c-1.1-0.3-1.1-0.1-0.4,0.8C740,1202.3,739.9,1202.8,740,1203.1z M1935.5,780.5c-0.2,0.2-0.4,0.4-0.6,0.4c-0.3,0.1-0.6,0-0.8,0
c0.4-0.2,0.8-0.5,1.1-0.8c1.4-0.9,2-1.9,0.5-3.3c3.2,1.1,6.1,1.2,7.9-2.8c-3.3-0.7-4.6,3.8-7.7,2.9
C1935.7,778.2,1935.6,779.4,1935.5,780.5z M1934.8,1202.5c-0.1-0.2-0.1-0.4-0.2-0.6c0,0,0.1-0.1,0.1-0.1c0,0.2,0.1,0.3,0.1,0.5
c3.1,1.6,5,5.7,11.3,5.1C1941.3,1205,1938.9,1202,1934.8,1202.5z M2072.4,516.6c-1,2.3-3.4,3-5.3,4.3c-7.4,5.1-8.2,7.2-5.3,17.8
c4.2-6.5,8-12.6,11.9-18.6c1.1-1.7,0.9-3.1-1.3-3.6c0,1,0.3,1.8,1.4,2c0.2,0,0.4-0.4,0.6-0.7C2073.8,517.4,2073.1,517,2072.4,516.6
z M599.6,517.5c-0.4-0.8-1.1-1.7-1.8-0.8c-0.8,0.8,0.1,1.5,1,1.6c2.4,8.9,7.5,16.2,13.4,24.1c2.3-16.5,1.9-17.3-10.7-24.4
C600.9,517.6,600.2,517.6,599.6,517.5z M534.9,1353.5c-20,21.8-37.3,44.1-53.4,67.4c-16.1,23.4-26.8,49.5-41.8,73.5
c9-21,17.4-42.2,28.9-61.9c16.9-28.7,35.4-56.4,57.5-81.5c0.9-1,1.6-2.1,2.6-3.1c2.9-2.9,3.3-6,2.3-10
c-9.1-33.8-14.6-68.3-19.2-103c-5-38-8.3-76.1-10.6-114.3c-2.5-39.9-3.4-79.9-3.9-119.8c-0.1-5.1-1.4-6.8-6.7-6.9
c-21.6-0.2-43.3-0.8-64.9-1.3c-2,0-4-0.4-6,0.8c0,195.2,0,390.5,0,585.6c2.1,0.6,3.1-0.4,4.1-1.1c52.7-33.8,105.3-67.7,158-101.4
c3.6-2.3,5.7-5.4,7.3-9.1c6-13,12.5-25.8,20.8-37.4c4.2-5.9,3.9-11.9,2.5-19.1c-2.4,1.8-4,2.9-5.5,4.2
c-13.9,12.6-32.1,8.7-42.7-2.9c-7.3-8-12.4-17.2-17.1-26.8C542.3,1375.7,539.4,1365.1,534.9,1353.5z M534.6,627.2
c3.6-5,4.3-10.7,6.3-15.9c6.2-15.8,12.6-31.5,25.2-43.7c10.8-10.5,26.8-12.6,38.9-2.8c2.3,1.9,4.3,5,7.9,5
c1.6-8.3-0.2-15.3-5.1-21.9c-7.4-9.9-12.7-21-17.8-32.3c-2.4-5.3-5.8-9.2-10.8-12.4c-50.5-32.2-100.9-64.7-151.3-97.1
c-2.4-1.5-4.5-3.9-8.7-3.9c0,3.7,0,7.2,0,10.7c0,187.9,0,375.8,0,563.7c0,2,0.2,4,0,6c-0.6,4.9,1.4,6.3,6.2,6.1
c13.6-0.6,27.3-0.7,41-1c8.3-0.2,16.7-0.6,25-0.4c4.9,0.1,6.7-1.8,6.1-6.4c-0.1-1,0-2,0-3c1.2-75,4.3-149.8,13.7-224.3
c4.6-36.3,10-72.5,19.6-107.9c1.6-5.7,0.4-9.9-3.4-14.4c-10.7-12.8-21.7-25.4-31-39.1c-19-28.1-36.8-57-49.6-88.7
c-2.2-5.4-5.5-10.4-6.2-16.6C465,538.3,494.6,586,534.6,627.2z M2060.9,1411.1c-2.3,8.5-0.1,15.1,4,21.4
c5.9,9.3,13.4,18.1,17.1,28.2c3.9,10.7,10.9,16.6,19.9,22.4c47.8,30.3,95.3,61.1,143,91.7c2.7,1.7,5,4.3,9.5,4.5c0-3.4,0-6.6,0-9.8
c0-188.3,0-376.6,0-565c0-2-0.1-4,0-6c0.2-3.4-1.1-5-4.6-4.8c-2.3,0.1-4.7,0-7,0.1c-20,0.6-40,1.2-60,1.9c-6.7,0.2-6.7,0.4-7,7.4
c0,1.3,0,2.7-0.1,4c-0.4,60-3,119.9-8.9,179.7c-5,50.1-11.3,99.9-24,148.7c-1.4,5.2,0.1,8.3,3.5,12.3c15,17.8,29.6,36.1,42.4,55.5
c20,30.4,36.1,62.9,49.3,96.8c3.2,8.2,5.7,16.6,6.8,25.4c-33.9-90.8-82.6-145-107.2-169.7c-4.1,11-7.5,22.1-12.7,32.5
c-4.9,9.9-10.3,19.5-18.8,26.8c-7.1,6.1-15.1,8.9-24.8,7.5C2073.2,1421.4,2068.3,1415.5,2060.9,1411.1z M2244.8,454.3
c0.6,4.5-0.2,6.7-0.8,8.9c-6.5,24.2-17,47-28.2,69.2c-18.9,37.2-43,71.2-70.4,102.8c-2.7,3.2-3.6,5.8-2.5,9.8
c7.9,29.6,13.1,59.8,17.3,90.2c6.4,45.6,10.2,91.3,12.8,137.3c2,35.3,2.6,70.6,3.1,105.9c0.1,7.6,0.2,7.6,7.6,7.8
c17,0.5,34,1,51,1.4c5,0.1,10,0.2,15,0.5c3.2,0.1,5.1-0.9,4.9-4.5c-0.1-1.7,0-3.3,0-5c0-189.6,0-379.3-0.1-568.9
c0-2.4,1.1-5.2-1.2-7.3c-0.8,0.1-1.6,0-2.1,0.4c-52.7,33.8-105.3,67.7-158,101.5c-2,1.3-3.7,2.6-4.7,4.9c-6.9,16-16.9,30.1-25.8,45
c-3.1,5.2-3.4,10.2-2,15.7c2.6,0.1,3.5-1.6,4.6-2.6c15.7-13.9,33-10.6,45.6,3.8c6,6.8,10.4,14.6,14.4,22.7
c5.1,10.3,8.3,21.3,12.8,32.9C2185.2,576.1,2221.7,520.5,2244.8,454.3z M553.7,662.8c-0.3,1.3-0.7,2.8-1,4.3
c-7.1,31.2-11.7,62.8-15.4,94.5c-7.1,59.9-10.3,120.1-11.8,180.4c-0.3,13-0.2,26-0.5,39c-0.1,4.4,1.5,5.7,5.7,5.6
c25-0.5,50-0.7,74.9-1.1c7.4-0.1,7.6-0.2,7.6-7.8c0-85.6,0-171.2,0.1-256.8c0-5.3-1.7-8.5-5.8-11.7c-16.6-12.8-32.4-26.6-46.9-41.9
C559,665.6,557.6,663,553.7,662.8z M553.6,1319.7c2.1-1,2.8-1.2,3.2-1.6c16.4-17.1,34-32.9,52.7-47.4c3-2.4,3.8-5,3.8-8.6
c-0.1-13.3-0.1-26.7-0.1-40c0-72.3,0-144.7,0-217c0-10.3,0.4-9-9.2-9.2c-23.3-0.4-46.7-0.8-70-1.1c-2.8,0-5.9-1-9.3,1.5
C526.2,1104.4,531.1,1212.3,553.6,1319.7z M2119.8,1319.2c4.8-20.9,8.3-40.2,11.1-59.6c5.8-39.5,9.8-79.2,12.5-119.1
c3.1-45.5,4.7-91.1,5.1-136.7c0.1-7.6-0.1-7.7-7.6-7.7c-24.3,0.1-48.6,0.3-72.9,0.5c-7.9,0.1-8,0.1-8,8.5c0,84.6,0,169.1-0.1,253.7
c0,5.6,1.7,9.3,5.9,12.7c12.1,9.9,24.2,19.9,36,30.1C2107.7,1306.9,2113.1,1312.7,2119.8,1319.2z M2120.4,664.4
c-2.1-0.6-2.9-0.1-3.6,0.7c-15.2,16.5-33.1,29.9-50.2,44.2c-4.8,4-6.8,8-6.8,14.3c0.3,32,0.2,64,0.2,96c0,52.3,0,104.6,0,157
c0,8.1,0.1,8.2,8.2,8.2c18.3,0.2,36.7,0.2,55,0.4c6.7,0.1,13.3,0.1,20,0.3c3.7,0.1,5.7-1.1,5.4-5.1c-0.2-2.3,0-4.7-0.1-7
c-0.5-45-2.1-89.9-5.2-134.8C2139.2,780.1,2132.7,721.9,2120.4,664.4z M878.7,1138.5c-36,15-69.8,32.9-102.3,53.1
c-3.4,2.1-4.9,4.7-5.7,8.6c-2.8,14.4-5.7,28.8-12.2,42c-3.6,7.3-7.7,14.4-17.3,14.4c-9.6,0-13.4-7.4-17.4-14.4
c-1.9-3.2-2-7.6-5.4-9.8c-22.3,17.4-41.6,37.2-58.8,58.9c-5.8,7.3-10.2,15.3-12,24.8c-3.8,20.3-9.1,40.2-16.7,59.5
c-2.6,6.7-1.3,13.3-1.5,20.3c1.5-0.4,2-0.3,2.1-0.5c3.2-3.8,6.4-7.6,9.4-11.6c31.2-40.5,66-77.4,105.4-110.2
c36.9-30.8,75.6-59.1,117.2-83.1c54.3-31.2,110.9-57.4,170.8-76.4c38.4-12.2,77.4-21.8,117-29.2c9.5-1.8,19.1-3,28.6-4.6
c1.5-0.3,3.8,0.3,4.4-1.5c1.4-3.9,0.6-8-0.4-11.7c-1-3.6-4.5-1.7-6.9-1.5c-40.6,3.2-80.4,11-120.4,18c-7,1.2-10.4,4.1-10.9,11.2
c-0.3,3.8-0.5,9.1-5.8,9c-4.6-0.1-4.7-5-5.3-8.5c-0.7-4.5-2.4-6-7.3-4.9c-39.3,9.1-78,19.9-115.5,34.7c-4.6,1.8-7.1,4.2-7.8,8.9
c-0.9,5.9-2.2,11.8-5.2,17.1c-4.1,7.1-10.6,7.2-14.6,0C881.9,1147.7,880.7,1143.4,878.7,1138.5z M1032.8,891.9
c0.9-3.8,1.6-6.7,2.4-9.5c0.7-2.3,1.7-4.6,4.4-4.9c3.4-0.3,4.4,2.3,5.3,4.9c0.4,1.3,0.8,2.6,0.8,3.9c0.1,7.7,4.5,10.4,11.7,11.6
c32.5,5.5,64.8,11.9,97.5,15.7c7.9,0.9,15.9,1.7,23.8,2.4c1.9,0.2,4,0.7,5.3-1.6c3.5-6.5,0.3-13-7-14.2c-1-0.2-2-0.2-3-0.4
c-42.6-5.8-84.2-15.9-125.4-28.2c-41.8-12.4-82.5-28.2-121.6-47.1c-51-24.6-99.8-53.3-144.7-87.9c-20.6-15.8-40.9-32.1-59.9-49.7
c-29.1-26.9-56.4-55.7-80.1-87.7c-3.7-4.9-7.7-9.6-11.6-14.3c-0.8,7.6-2.4,14.7,0.5,21.8c7.4,18.2,12.4,37.2,16,56.5
c1.7,9.1,5,17.3,10.6,24.4c9.2,11.4,18.5,22.8,28.7,33.3c9.9,10.1,21.1,19,32.2,28.9c3.8-6.5,5.6-13,9.8-18.2
c7.7-9.4,17.7-9.5,25.3-0.1c3.6,4.5,5.7,9.7,7.8,15c4.3,10.9,6.8,22.3,8.9,33.7c0.9,5.2,3.3,8.1,7.5,10.7
c21.9,13.4,44.1,26,67.3,37.1c10.9,5.2,22.1,10,33.5,15.2c1.8-4.8,3.1-8.6,4.7-12.2c1.6-3.5,4.2-6.3,8.3-6.1
c3.6,0.2,5.8,2.9,7.3,6.1c2.4,5.2,3.8,10.6,4.8,16.2c0.8,4.6,2.9,7.2,7.7,9C950.6,871.5,990.8,883,1032.8,891.9z M1953.8,1233.7
c-2.9,5.8-4.9,11.5-8.8,16.2c-7.6,9.3-17.8,9.3-25.3,0c-3.4-4.2-5.5-9-7.5-14c-4.4-11.5-7.2-23.5-9.4-35.7
c-0.7-3.9-2.4-6.4-5.8-8.6c-30.4-19.4-62.4-35.6-95-50.8c-6.9-3.3-7-3-9.9,4.7c-0.7,1.9-1.5,3.7-2.3,5.5c-1.6,3.1-3.7,5.6-7.6,5.6
c-3.8,0-5.9-2.5-7.6-5.6c-2.9-5.4-4.3-11.3-5.1-17.2c-0.7-5-3-7.3-7.5-9.1c-37.8-15.2-77.3-24.6-116.7-34.3c-3.8-0.9-5.4,0.2-6.2,4
c-0.7,3.9-0.2,9.9-6.3,9.5c-5-0.3-5.1-6.1-5.3-9.6c-0.4-6.6-3.3-8.9-9.6-10c-32.8-5.6-65.4-12.2-98.6-15.7
c-8.3-0.9-16.5-2.1-24.8-2.7c-5.4-0.4-7.1,1.2-7.5,6.6c-0.3,3.9-1.3,8,5.2,8.4c6.3,0.4,12.4,2.1,18.7,3.1
c47.7,7.5,94.4,19.1,140.3,34.3c42.4,14,83.5,31.2,123,52c46.3,24.4,90.2,52.6,131,85.6c48.5,39.1,93.7,81.2,129.4,132.6
c1.9,2.8,4.8,7.5,7.3,6.7c4.5-1.5,1.4-6.8,2.1-10.4c0.6-3.4-0.5-6.6-1.8-9.7c-7.8-19.5-12.6-40-17.1-60.4c-1.1-5.1-3-9.4-6-13.5
c-17-22.8-36.7-43.1-57.6-62.2C1959.6,1237.1,1957.6,1234.7,1953.8,1233.7z M2043.6,585.1c-4.2,2.7-6.2,6.2-8.5,9.2
c-29.9,40.3-64.6,76.1-102.5,109c-33.8,29.3-69.5,56.2-107.6,79.4c-51.8,31.6-106.3,57.7-163.9,77.3
c-40.7,13.8-82.1,24.9-124.2,33.1c-13.7,2.7-27.6,4.7-41.3,7c-2.2,0.4-5.3-0.1-6.2,2.4c-1.4,3.8-1.1,8.2,0.3,11.7
c1.4,3.5,5.3,1.6,8.1,1.3c40.1-4.2,79.8-10.7,119.5-18.1c6.2-1.2,10.4-2.7,10.5-10.2c0-3.5,0.3-9.3,5.4-9.5c6-0.3,5.5,5.7,6.2,9.6
c0.7,3.9,2.3,4.9,6.1,3.9c10-2.7,20-5.1,30-7.5c28.5-6.9,56.4-15.6,83.9-25.7c6.8-2.5,10.2-6.3,11-13.3c0.5-5,2-9.8,4.5-14.2
c4.2-7.6,10.8-7.6,15.1,0.1c1.6,2.9,2.5,6.2,3.4,9.3c0.9,2.9,2.5,3.5,5.2,2.2c2.4-1.2,5-1.9,7.3-3.1c26.6-13.1,53.2-26,78.7-41.2
c11.9-7.1,19.6-14.7,21.3-29.3c1.4-12.4,5.5-24.8,12.8-35.4c8.1-11.7,19.5-11.7,27.5,0.1c3.1,4.6,5.4,9.7,8.2,14.9
c1.9-1.4,3.6-2.5,5.1-3.8c22.2-19.8,42.7-41.1,60.6-64.9c2.6-3.5,4.3-7.3,5.2-11.7c4.5-21.2,9.6-42.2,17.6-62.4
C2045.3,599.2,2044.5,592.6,2043.6,585.1z M777.2,1163.8c1.2,0.2,1.6,0.4,1.9,0.3c30.4-14.3,60.9-28.3,92.6-39.6
c4.2-1.5,4.4-3.7,3.9-7.5c-4.5-37.4-6.3-75-6.5-112.6c0-7.8-0.2-7.9-8.5-7.9c-22,0-44,0-65.9,0c-8.6,0-8.8,0.1-8.8,8.7
c-0.4,45.3-2.2,90.5-6.6,135.6C778.4,1148.4,776.3,1155.9,777.2,1163.8z M776.8,817.8c0.3,3.8,0.5,7.1,0.9,10.4
c2.5,21.2,4.1,42.4,5.5,63.7c1.8,28.6,2.3,57.2,2.7,85.9c0.1,7.1,0.3,7.3,7.2,7.3c22.6,0.1,45.3,0.1,67.9,0.1
c8.8,0,8.1-0.2,8.1-8.3c0.1-37,1.9-73.9,6.4-110.6c0.7-5.8-0.5-8.2-6-10.1c-29.5-10.3-58-23.3-86.2-36.8
C781.5,818.4,779.9,817.2,776.8,817.8z M1896.4,1163.9c1-2.7,0.2-5.3-0.1-7.9c-2.4-19.8-3.9-39.8-5.3-59.7
c-2.1-30.3-2.5-60.6-3.3-90.9c-0.2-8.7-0.2-8.8-8.6-8.8c-16.7-0.1-33.3,0-50-0.1c-6.3,0-12.7,0.2-19-0.1c-3.9-0.1-5.5,1.4-5.2,5.3
c0.1,1.7,0.1,3.3,0,5c-1.1,36-2,72-6.4,107.7c-0.9,7.7-1.1,7.9,6.3,10.6c28.5,10.4,56.1,22.9,83.3,36.1
C1890.6,1162.4,1893.1,1164.4,1896.4,1163.9z M1896.8,817.7c-2.5-0.3-4.3,0.6-6.1,1.5c-6,2.9-12,5.6-18,8.6
c-22.1,10.9-44.9,20.1-67.9,28.7c-7.5,2.8-7.3,3-6.5,10.6c1,8.9,1.9,17.9,2.7,26.8c2.4,28.2,3.2,56.5,3.6,84.7
c0.1,5.4,1.8,6.7,6.8,6.6c22.6-0.2,45.3-0.1,67.9-0.2c1,0,2-0.1,3,0c3.8,0.3,5.6-1.2,5.2-5.1c-0.2-1.6,0-3.3,0.1-5
c0.7-38.6,1.7-77.2,5.2-115.6C1894.1,845.4,1895.5,831.6,1896.8,817.7z M908.3,1109.6c1.4,0,2.4,0.2,3.2,0
c37.8-11.7,75.5-24,114.2-32.5c4.5-1,5.8-2.9,5.5-7.4c-1.3-22-2.5-43.9-2.3-65.9c0.1-7.3-0.2-7.4-8-7.4c-33.3,0-66.7,0.1-100,0.2
c-7.7,0-7.9,0.1-7.9,7.5c0,31-1.6,61.9-4,92.9C908.8,1101,907.3,1105.2,908.3,1109.6z M1765.3,1110.4c0-2.5,0.1-3.5,0-4.4
c-3.1-33.9-4.7-67.8-4.8-101.8c0-7.4-0.4-7.6-8.2-7.7c-33.3-0.1-66.7-0.2-100-0.2c-7.3,0-7.2,0.1-7.5,7.9c-0.7,21-1.1,42-2.3,62.9
c-0.5,9-1.3,8.3,7.6,10.4c29.8,7.1,59.4,14.9,88.5,24.5C1747.2,1104.8,1755.7,1107.4,1765.3,1110.4z M908.8,872.2
c-0.2,0.3-0.6,0.6-0.6,0.8c2.8,35.5,5.2,71,4.8,106.7c-0.1,4.6,1.9,5.6,6,5.6c34.7-0.1,69.3-0.1,104,0.1c4.9,0,6.2-1.8,6.1-6.4
c-0.5-22,0.8-44,2.1-65.9c0.3-5.2-0.9-7.7-6.4-8.8c-36.9-7.6-72.4-20.1-108.5-30.5C913.8,873,911.6,871.1,908.8,872.2z
M1765.6,872.4c-3.2-0.6-5.1,0.3-7,0.9c-22.2,6.7-44.2,14.5-66.7,20.3c-14.5,3.7-29,7.3-43.5,10.8c-4.4,1.1-6.5,2.6-6.2,8
c1.3,21.9,1.8,43.9,2.6,65.8c0.2,7,0.2,7.1,7.1,7.1c33.6,0,67.2,0,100.8-0.1c7.5,0,7.7-0.2,7.7-7.8c0.2-24.6,0.9-49.2,2.6-73.8
C1763.8,893.4,1764.7,883.1,1765.6,872.4z M725.8,1192.5c8.7-4.8,15.8-9.8,23.8-12.8c9.6-3.7,12.7-10.5,13-19.7
c0.1-1.3,0.2-2.7,0.4-4c3.4-27.1,5.6-54.3,6.7-81.5c1-23.6,1.5-47.2,2-70.8c0.2-7-0.1-7-7.3-7.1c-6.7-0.1-13.3,0-20,0
c-9,0-18,0.2-27,0c-4.2-0.1-6.2,1.3-5.9,5.7c0.2,2.3,0,4.7,0.1,7c0.8,27.6,1.2,55.2,3,82.8C717,1125.1,719.9,1158.2,725.8,1192.5z
M1946.2,1191.7c2.9-2.2,2.4-4.7,2.8-6.9c4.5-26.6,6.9-53.5,8.9-80.4c2.5-33.9,3.2-67.9,3.7-101.8c0.1-5-1.8-6.2-6.4-6.1
c-15.3,0.2-30.6,0-46,0.1c-7.6,0-7.8,0.2-7.7,7.8c0.7,44,2,87.9,6.7,131.6c1.1,10.3,2.2,20.5,3.5,30.7c0.4,2.8,0.2,6,3.5,7.8
C1925.8,1180.2,1936.2,1186.1,1946.2,1191.7z M1947.7,789.1c-9.1,5-16.9,10.1-25.3,13.8c-8.2,3.6-10.5,9.5-11.5,17.8
c-3,27.1-5.6,54.3-6.9,81.6c-1.2,25.3-1.5,50.6-2.5,75.9c-0.2,5.1,1.5,7,6.7,7c15-0.3,30-0.1,45-0.2c8.6,0,8.4,0.6,8.3-8.9
c-0.3-30.6-1-61.3-3.1-91.9C1956.3,852.9,1953.6,821.8,1947.7,789.1z M726,789.2c-3.3,16.4-5.1,31.3-6.7,46.2
c-4.6,42.4-6.3,85-7.3,127.7c-0.1,5.7,0,11.3-0.2,17c-0.1,3.4,1.4,4.8,4.7,4.8c17,0,34,0,51,0.1c3.6,0,4.6-1.9,4.4-5
c-0.1-2.3-0.1-4.7-0.1-7c-0.5-37.3-1.7-74.6-5-111.8c-1.4-15.9-2.8-31.9-5.5-47.6c-0.4-2.6-0.3-5.2-3.2-6.8
C747.6,801.1,737.2,795.4,726,789.2z M1625.1,910.5c-9.8,1.8-18,3.4-26.2,4.6c-23.3,3.5-46.2,9.9-70,10
c-10.9,0.1-21.8,2.4-32.7,3.7c-3.5,0.4-6.2,1.1-6,6c0.5,15,0.7,30,0.7,44.9c0,4.4,1.7,5.8,5.9,5.7c8.3-0.2,16.6,0,25-0.1
c30.6,0,61.3,0,91.9-0.1c9,0,9-0.1,9.2-9.4c0.2-13,0.2-26,0.6-38.9C1623.7,928.4,1624.5,919.8,1625.1,910.5z M1625.2,1071.1
c-2.2-23.8-2.3-46.1-2.4-68.4c0-4.5-1-6.6-6.1-6.6c-39.9,0.1-79.9,0.1-119.8-0.1c-5.1,0-6.1,2-6.1,6.5c0,14.6-0.1,29.3-0.6,43.9
c-0.1,4.5,1.6,5.6,5.6,6.1c12.6,1.4,25,3.8,37.7,4c8,0.1,15.9,1.1,23.8,2.5C1579.5,1063,1601.7,1066.9,1625.2,1071.1z
M1049.3,1070.4c2.6,1,4.9-0.1,7-0.6c23-5.5,46.3-8.9,69.7-12.3c16.7-2.5,33.7-3.8,50.6-5.2c5.1-0.4,6.8-1.9,6.6-7
c-0.4-14-0.3-27.9-0.5-41.9c-0.1-7-0.2-7.2-7.2-7.2c-39.3,0-78.5,0-117.8,0.1c-6.7,0-6.8,0.2-6.8,7.5c0,16.3-0.5,32.6-1.4,48.9
C1049.3,1058.6,1048.4,1064.5,1049.3,1070.4z M1049.8,910.2c-0.4,1.2-0.8,1.8-0.8,2.4c0.6,22.6,2.5,45.2,1.9,67.8
c-0.1,4.2,2.1,4.8,5.6,4.8c39.9,0,79.9-0.1,119.8,0.1c5.5,0,6.7-2.3,6.6-7.2c-0.2-11,0.1-22,0.2-32.9c0.1-15.9,0.1-15.5-16.3-17.4
c-16.5-1.9-33.1-2.4-49.6-5.2C1094.7,918.9,1072,915.9,1049.8,910.2z M1260.5,985.4c21.6,0,43.2,0,64.9,0c9.1,0,9.1-0.1,9.1-9.4
c0-10.3,0-20.6-0.1-30.9c-0.1-7.7-0.1-7.7-7.6-7.9c-2.3-0.1-4.7,0-7,0c-41.9,0-83.8-2.2-125.5-6.2c-4.6-0.4-5.9,0.9-6,5.1
c-0.2,13.6-0.6,27.3-0.8,40.9c-0.1,8.3,0.1,8.4,8.1,8.4C1217.3,985.4,1238.9,985.4,1260.5,985.4z M1485,931.5
c-3.1-1.4-5.4-0.6-7.7-0.4c-26.2,1.9-52.4,4.3-78.7,5.1c-17.3,0.5-34.6,1.3-51.9,1.1c-7.4-0.1-7.5,0.1-7.6,7.8
c-0.1,11.7,0,23.3-0.1,35c0,3.7,1,5.8,5,5.5c2-0.1,4,0,6,0c41.6,0,83.2,0,124.9,0c2,0,4-0.1,6,0c3.3,0.1,5-1.2,4.9-4.6
C1485.5,964.2,1485.3,947.6,1485,931.5z M1485,1051.1c0.3-17,0.5-33.6,0.9-50.1c0.1-4.2-2.2-4.8-5.7-4.8c-45,0.1-89.9,0.1-134.9,0
c-4.7,0-6.5,1.4-6.3,6.2c0.3,12,0.4,24,0,36c-0.2,5.2,2.1,6,6.5,6c11.6-0.1,23.3,0.1,34.9,0.4
C1415,1045.8,1449.5,1048.1,1485,1051.1z M1188.4,1050.6c47.5-3.4,93.2-6.8,139.2-6.4c0.7,0,1.3,0,2,0c3.2,0.2,4.9-0.9,4.9-4.5
c-0.1-13,0-25.9,0-38.9c0-3.4-1.6-4.8-4.9-4.6c-1.7,0.1-3.3,0-5,0c-42.6,0-85.1,0-127.7,0c-1.7,0-3.3,0.1-5,0
c-2.8-0.1-4.4,1.1-4.4,4.1C1187.9,1016.8,1188.1,1033.4,1188.4,1050.6z M2010.7,1230.1c0.9-7.3-0.9-13.8-1.5-20.3
c-6.8-66-10.3-132.2-10.7-198.6c-0.1-16.1-0.3-16.3-16.8-14.7c-2.9,0.3-4.2,1.1-4.1,4.1c0.1,3,0,6,0,9
c-0.3,41.3-2.1,82.6-6.1,123.8c-2,20.2-4.3,40.4-8.2,60.4c-0.9,4.8-0.2,7.8,4.5,10.4c10.2,5.7,19.9,12.2,29.8,18.3
C2001.8,1225,2005.6,1228.2,2010.7,1230.1z M2011.2,752.5c-2.7-0.8-3.6,0.3-4.7,1c-13.3,8.3-26.4,16.7-39.8,24.7
c-3.6,2.1-4.2,4.5-3.5,8.2c1.4,7.8,2.8,15.7,3.9,23.6c7.4,53.8,10.1,107.9,10.6,162.2c0.1,14.1-2.6,12.8,13.7,12.8
c7,0,7-0.2,7.1-7.3c0.4-27.3,0.6-54.6,1.8-81.8c1.4-30.9,3.1-61.8,6.1-92.6C2008,786.3,2009.6,769.5,2011.2,752.5z M662.1,1231.5
c3.7-2.4,5.6-3.7,7.5-4.9c11.6-7.7,22.9-16.2,35.5-22.2c5.3-2.5,6.2-5.5,5.1-11c-4.6-22.5-7-45.4-9-68.3
c-3.5-39.9-5-79.8-5.4-119.8c-0.1-8.6-0.2-8.6-8.8-8.7c-1.3,0-2.7,0-4,0c-7.9,0.1-8,0.1-7.9,8.4c0,24.7-0.6,49.3-1.6,74
c-1.3,32-3,63.9-6,95.8C665.8,1192.9,664.1,1211.1,662.1,1231.5z M662.9,752.1c-1,1.4-0.4,3-0.3,4.6c2,20.5,4.6,41.1,6,61.7
c3.8,52.5,6.5,105.1,6.4,157.8c0,8.8,0.1,8.9,8.5,8.9c14.5-0.1,12.1,1.3,12.3-12.1c1.1-62.7,3.5-125.2,14.8-187
c0.8-4.1-0.9-6-4.2-7.7c-13.4-6.8-25.6-15.5-38-23.9C666.8,753.1,665.5,751.1,662.9,752.1z M2044.3,728.6c-5.4,1.1-7.1,3.9-7.7,8.3
c-1.7,14.9-4,29.7-5.6,44.6c-4.6,42.1-7.2,84.2-8.9,126.5c-1,25.3-1.3,50.5-1.3,76.8c6.5,0,12.4-0.1,18.3,0
c3.8,0.1,5.5-1.4,5.2-5.3c-0.2-1.7,0-3.3,0-5c0-78.9,0-157.9,0-236.8C2044.3,734.9,2044.3,732,2044.3,728.6z M2044.3,1253
c0-3.7,0-6.4,0-9c0-78.6,0-157.2,0-235.8c0-13,1.3-11.5-11.9-11.5c-13,0-11.5-1.8-11.5,11.8c0,59.3,3.1,118.5,8.7,177.6
c1.9,19.5,4,39.1,7,58.5C2037.3,1248.9,2038.8,1251.8,2044.3,1253z M629.2,728.7c0,4.3,0,8,0,11.6c0,76.5,0,153.1,0,229.6
c0,15.8,0,15.8,16,15.3c7.6-0.2,7.6-0.2,7.7-7.7c0.1-3.3,0-6.7,0-10c-0.8-55.6-3.1-111.1-8.5-166.4c-2-21.2-5-42.3-7.3-63.4
C636.6,733.3,635,730.3,629.2,728.7z M629.2,1253.2c5.3-2.1,7.4-4.5,7.9-9.2c1.6-14.9,4-29.7,5.5-44.6
c6.5-62.4,9.5-124.9,10.3-187.6c0.2-15.6,0.1-15.6-15-15.6c-9,0-8.8-0.6-8.7,8.6c0,2,0,4,0,6c0,75.7,0,151.3,0,227
C629.2,1242.5,629.2,1247.2,629.2,1253.2z M2060,697.5c19-15.9,36.7-29.3,52.3-45.3c3.5-3.5,3.7-6.7,2.4-10.7
c-2.8-8.9-5.4-17.8-8.7-26.5c-3.5-9.3-7.4-18.5-13.7-26.4c-5.3-6.6-8-6.6-13.4-0.4c-1.3,1.5-2.7,3-3.6,4.7
c-8.9,15.7-17.3,31.5-15.3,50.5c0.4,3.6,0.1,7.3,0.1,11C2060,668,2060,681.5,2060,697.5z M613.2,1283.1
c-20.8,13.6-36.9,29.9-53.8,45.4c-2.8,2.6-2.3,5.1-1.5,8.1c4.1,14.7,8.6,29.3,15.3,43.2c2.6,5.4,5.4,10.7,9.8,15
c4.3,4.2,6.2,4.6,10,0.4c10.7-11.9,18.4-26.3,19.7-41.7C614.7,1330.6,613.2,1307.6,613.2,1283.1z M2060,1284.4
c0,19.5,0.7,36.8-0.2,54c-1.1,19.1,6.3,35.2,15.8,50.5c7.4,12.1,12.1,11.7,19.8-0.6c9.7-15.4,14.6-32.7,19.7-50
c1.1-3.7,0-6.2-2.6-8.8C2096.8,1313.6,2079.3,1299.9,2060,1284.4z M611.9,698.2c2.1-3.3,1.3-5.9,1.3-8.4c0.1-16-0.8-32,0.3-47.9
c1.2-18.4-6.2-33.8-15.1-48.6c-7.6-12.7-12.6-12.4-20.4,0.3c-10.1,16.3-15.1,34.6-20.4,52.8c-1,3.6,0.6,5.5,3,7.7
c8.8,8.1,17.3,16.7,26.3,24.5C594.8,685.4,603.3,691.5,611.9,698.2z M1788.2,864c-12.5,3.2-12.6,3.2-13.9,13.8
c-0.9,7.6-1.6,15.2-2.1,22.8c-1.8,26.2-2.9,52.5-3.2,78.8c0,4,0.8,6.3,5.4,6c4.6-0.3,9.3-0.2,14,0c5.1,0.1,6.9-2.1,6.7-7.3
c-0.5-11.3-0.2-22.6-0.7-33.9C1793.2,917.5,1792.3,890.9,1788.2,864z M885.3,1118.4c12.8-3,12.5-3,13.8-13.9
c3.5-29.5,4.6-59.1,5-88.8c0.3-23.5,5.2-18.7-19.8-19.4c-4.4-0.1-5.7,1.5-5.7,5.8C878.8,1040.8,880.3,1079.4,885.3,1118.4z
M885.7,863.5c-0.2,0.9-0.6,1.9-0.7,2.9c-4.3,33.7-6,67.5-6.1,101.4c0,5.5-2.5,12.6,1.3,16.2c3.8,3.6,10.8,1.1,16.3,1.2
c7.9,0.1,8.1-0.1,7.9-8.4c-0.6-33.2-1.5-66.4-5.3-99.4C897.9,865.6,898,865.6,885.7,863.5z M1788.4,1117.5
c4.4-34.4,5.9-68.6,6.5-102.9c0.1-5.6,2.4-12.7-0.9-16.5c-3.9-4.3-11.3-1.4-17.2-1.6c-7.6-0.2-7.7,0.1-7.7,7.6
c0,24.7,1.3,49.3,2.8,73.9c0.6,10.3,2.2,20.5,2.9,30.8c0.3,3.6,1.7,5.5,4.9,6.5C1782.2,1116,1784.4,1117.7,1788.4,1117.5z
M1481.1,1078.1c0-2.7,0.1-4.4,0-6c-0.3-7.8-0.3-8.1-8.2-8.7c-22.9-1.7-45.8-3.3-68.7-4.7c-19.9-1.2-39.9-0.9-59.9-1
c-4.6,0-5.5,2-5.4,6c0.1,3.7,0.3,6.7,5.2,6.2c1.3-0.1,2.7,0,4,0c34.3-0.3,68.4,1.9,102.5,5.6
C1460.6,1076.6,1470.4,1078.5,1481.1,1078.1z M1480,903c-15.3,1.6-30.8,3.6-46.3,4.8c-29.5,2.4-59.1,4.7-88.8,3.9
c-5.1-0.1-6.1,2-6,6.5c0.1,4.1,1.1,6.3,5.6,5.8c1.3-0.2,2.7,0,4,0c42.3,0.3,84.5-2.6,126.7-5.9c4.2-0.3,6.2-1.6,5.8-5.9
C1480.7,909.3,1482,906.3,1480,903z M1192.2,1078.2c3.1-0.2,5.4-0.3,7.6-0.5c37-4.6,74-7.6,111.3-7.8c6,0,12-0.1,18,0
c3.7,0.1,5.3-1.1,5.4-5.1c0.1-6.5,0-6.9-7.3-7c-40.6-0.8-81.1,1.7-121.6,4.6C1193.1,1063.3,1193.2,1063.6,1192.2,1078.2z
M1192.6,903.2c0.2,14.4,0.2,14.4,12.8,16c3,0.4,6,0.6,8.9,0.8c19.3,1.1,38.5,2,57.8,3.2c18.6,1.2,37.2,0.8,55.9,0.8
c6.4,0,6.3-0.5,6.6-6.8c0.2-5.8-3.3-5.5-7.1-5.4c-28.3,0.8-56.5-1.5-84.7-3.6C1226.3,907,1209.8,905.4,1192.6,903.2z M653.6,694.6
c2,12.3,3.8,22.4,5.1,32.6c0.5,3.7,1.7,6.1,4.8,8c14.2,8.8,27.5,18.8,42.2,26.6c2.7,1.4,6.2,6.3,8.9,2.4c2.7-3.7-2.6-6.2-5-8.5
c-19-17.3-36.7-35.8-51.8-56.7C657,698.1,656,697.3,653.6,694.6z M654,1284.6c0.8,0,1.3,0.1,1.5,0c1.4-1.4,2.9-2.7,4-4.3
c14.6-20.8,32.8-38.3,51.2-55.5c1.2-1.1,2.6-2.1,3.6-3.4c1.4-1.7,1.5-3.6-0.2-5.7c-2.9,0.3-5.3,2.1-7.8,3.6
c-10.2,6.3-19.8,13.5-30.5,18.8c-12.9,6.4-19.3,15.7-19.2,30.2C656.6,1273.7,654.9,1279.1,654,1284.6z M2019.1,698.8
c-0.5-0.3-1-0.6-1.5-0.9c-1,1.2-2,2.4-3,3.6c-6.8,9-13,18.4-21.4,26.1c-11,10.1-20.5,21.7-32.4,30.8c-1.9,1.5-3.6,3.2-2.1,5.9
c2,3.5,4.1,0.5,5.7-0.4c15.4-9.7,30.8-19.6,46.1-29.5c2.3-1.5,4-3.1,4.4-6.2C2016.2,718.5,2017.7,708.6,2019.1,698.8z
M2017.1,1283.9c0.7-0.2,1.4-0.5,2.1-0.7c-1.5-10.1-3-20.2-4.4-30.4c-0.4-2.6-1.9-4.1-3.9-5.4c-15.6-10-31.3-20.1-46.9-30
c-1.5-1-3.3-2.8-5-0.5c-1.7,2.4-0.3,4.2,1.7,5.9c4,3.4,8.1,6.9,11.8,10.6C1988.4,1249.3,2005.3,1264.3,2017.1,1283.9z M1044.6,1034
c0.2,0,0.3,0,0.5,0c0-11-0.1-21.9,0.1-32.9c0.1-4.2-2-4.8-5.5-4.9c-4.5-0.1-4.6,2.6-4.6,5.9c0.1,22.3,1.2,44.5,2.4,66.8
c0.1,1,0.2,2,0.4,3c0.2,1.2,1.2,1.8,2.1,1.7c0.8-0.1,1.9-1,2.2-1.7c0.5-1.5,0.7-3.2,0.8-4.9C1043.5,1055.9,1044,1045,1044.6,1034z
M1636.4,1033.8c0.7,0,1.4,0,2.1,0c0-10.6,0-21.2,0-31.8c0-2.7,0.2-5.3-3.6-5.8c-4.7-0.6-6,0.4-6,6.1c-0.1,21.9,0.2,43.7,2.1,65.6
c0.1,1.3,0.1,2.7,0.6,3.9c0.3,0.7,1.5,1.6,2.2,1.5c0.7,0,1.7-1.1,1.9-1.8c0.5-1.9,0.7-3.9,0.7-5.9
C1636.4,1055,1636.4,1044.4,1636.4,1033.8z M1638.5,948.1c-0.2,0-0.5,0-0.7,0c-0.5-11.6-1-23.3-1.6-34.9c-0.1-1.9,0.2-4.8-2.3-4.9
c-3.3-0.1-2.8,3-3,5.2c-0.3,3-0.4,6-0.7,9c-1.5,19-1.4,37.9-1.4,56.9c0,5.3,2,6.8,6.4,5.8c3.8-0.8,3.3-3.6,3.3-6.2
C1638.5,968.8,1638.5,958.4,1638.5,948.1z M1045.1,948c-0.1,0-0.3,0-0.4,0c-0.6-11.9-1.2-23.9-1.9-35.8c-0.1-1.9-0.4-4.3-3.2-3.9
c-2.2,0.3-1.9,2.4-2,4c-1.3,22.5-2.3,45.1-2.5,67.7c0,3.7,1.2,5.4,5.1,5.4c4.1,0.1,5-1.9,5-5.5C1045,969.3,1045.1,958.7,1045.1,948
z M741.9,740.1c-4.9,5.5-6.3,11-8.3,16.2c-0.8,2-0.8,4,1.2,5.3c6.2,4.1,11.6,9.4,19.7,13C751.6,762.1,748.8,750.9,741.9,740.1z
M1931.7,1241.3c11.7-16.6,11.5-18.2-3-28.3c-2.7-1.9-4.9-4.5-9.1-4.9C1921.9,1219.6,1924.7,1230.7,1931.7,1241.3z M1931.6,740
c-6.6,10.9-9.8,22-12.1,33.9c4-0.9,6.1-3.1,8.5-4.8C1943.5,758.3,1943.6,757.2,1931.6,740z M742.1,1241.3
c6.6-11.1,9.8-21.9,12.2-34.3c-8,3.6-13.2,8.8-19.2,12.6c-2,1.3-2.3,3.1-1.6,5.1C735.6,1230.2,736.7,1236.2,742.1,1241.3z
M597.6,1464.4c16.9-6.5,17.3-8.9,14-24.4C605.8,1447.2,601.7,1455,597.6,1464.4z M2061.5,1442.5c-5.1,14.3,5.6,17.3,12.5,22.8
C2072.1,1457,2066.2,1451,2061.5,1442.5z M1781.6,836.9c-1,5.2-3.6,8.5-1.8,12.6C1785.7,846.2,1785.7,846.2,1781.6,836.9z
M895.4,849.3c-1.5-4.5-1.6-7.9-3.8-11.1C887.9,846.3,887.9,846.3,895.4,849.3z M630.1,1284.6c-0.4-5.3,4.1-10.2,1.8-17
C627.4,1273.6,629.9,1279.2,630.1,1284.6z M629.9,697.1c0.2,5.4-3,11.4,2.4,16.6C633.8,707.4,630.4,702.5,629.9,697.1z
M2044.5,1284.5c-1-5.2,1.5-10.3-2.8-15C2040.3,1275.4,2042.4,1279.6,2044.5,1284.5z M2041.9,712.1c4.1-4.8,1.6-10,2.2-14.6
C2043,702.1,2039.8,706.2,2041.9,712.1z M892.2,1143.9c0.4-4.2,3.5-7.3,1.6-11.5C887.6,1136,887.6,1136,892.2,1143.9z
M1779.6,1132.2c-1,2.9-0.4,4.8,0.4,6.5c0.7,1.5,0.4,3.6,2.3,4.6C1785.7,1135.3,1785.7,1135.3,1779.6,1132.2z M729.5,774
c1.5,1.6,2.6,4,5.8,3.2C733.9,774.7,732.2,773.5,729.5,774z M737.9,778.4c-0.2,0.2-0.5,0.4-0.7,0.6c0.2,0.2,0.4,0.6,0.6,0.6
c0.2,0,0.5-0.3,0.7-0.5C738.4,778.8,738.1,778.6,737.9,778.4z"/>
<path class="st1" d="M534.9,1353.5c4.5,11.6,7.4,22.2,12.2,32c4.7,9.6,9.8,18.8,17.1,26.8c10.6,11.6,28.8,15.5,42.7,2.9
c1.4-1.3,3.1-2.4,5.5-4.2c1.4,7.2,1.7,13.2-2.5,19.1c-8.4,11.7-14.8,24.4-20.8,37.4c-1.7,3.7-3.7,6.7-7.3,9.1
c-52.7,33.7-105.3,67.6-158,101.4c-1.1,0.7-2.1,1.7-4.1,1.1c0-195.1,0-390.4,0-585.6c2.1-1.2,4.1-0.9,6-0.8
c21.6,0.5,43.3,1.1,64.9,1.3c5.3,0,6.7,1.8,6.7,6.9c0.5,40,1.4,79.9,3.9,119.8c2.4,38.2,5.6,76.4,10.6,114.3
c4.6,34.7,10.1,69.1,19.2,103c1.1,4,0.6,7.1-2.3,10c-0.9,0.9-1.7,2.1-2.6,3.1c-22.1,25.1-40.6,52.7-57.5,81.5
c-11.6,19.7-19.9,41-28.9,61.9c14.9-23.9,25.7-50.1,41.8-73.5C497.6,1397.6,514.9,1375.3,534.9,1353.5z"/>
<path class="st1" d="M534.6,627.2c-40-41.2-69.6-89-94.2-140.1c0.7,6.2,4,11.2,6.2,16.6c12.7,31.7,30.5,60.6,49.6,88.7
c9.3,13.7,20.4,26.3,31,39.1c3.8,4.5,5,8.7,3.4,14.4c-9.6,35.4-15,71.6-19.6,107.9c-9.4,74.5-12.5,149.3-13.7,224.3c0,1-0.1,2,0,3
c0.6,4.7-1.2,6.5-6.1,6.4c-8.3-0.2-16.7,0.2-25,0.4c-13.7,0.3-27.3,0.4-41,1c-4.8,0.2-6.8-1.2-6.2-6.1c0.3-2,0-4,0-6
c0-187.9,0-375.8,0-563.7c0-3.5,0-7.1,0-10.7c4.2-0.1,6.3,2.3,8.7,3.9c50.4,32.4,100.8,64.9,151.3,97.1c5,3.2,8.4,7.1,10.8,12.4
c5.1,11.2,10.4,22.4,17.8,32.3c4.9,6.6,6.7,13.5,5.1,21.9c-3.6-0.1-5.6-3.1-7.9-5c-12-9.8-28.1-7.7-38.9,2.8
c-12.6,12.2-19,27.9-25.2,43.7C538.8,616.5,538.2,622.2,534.6,627.2z"/>
<path class="st1" d="M2060.9,1411.1c7.4,4.3,12.3,10.3,20.6,11.4c9.8,1.4,17.7-1.4,24.8-7.5c8.5-7.3,13.9-16.9,18.8-26.8
c5.1-10.4,8.6-21.5,12.7-32.5c24.7,24.7,73.3,78.9,107.2,169.7c-1.1-8.8-3.5-17.2-6.8-25.4c-13.2-33.9-29.3-66.4-49.3-96.8
c-12.8-19.5-27.4-37.7-42.4-55.5c-3.4-4-4.8-7.1-3.5-12.3c12.8-48.8,19.1-98.6,24-148.7c5.9-59.7,8.4-119.6,8.9-179.7
c0-1.3,0-2.7,0.1-4c0.2-7,0.2-7.1,7-7.4c20-0.7,40-1.3,60-1.9c2.3-0.1,4.7,0,7-0.1c3.5-0.2,4.8,1.4,4.6,4.8c-0.1,2,0,4,0,6
c0,188.3,0,376.6,0,565c0,3.2,0,6.4,0,9.8c-4.5-0.2-6.9-2.8-9.5-4.5c-47.7-30.6-95.2-61.4-143-91.7c-9-5.7-16-11.6-19.9-22.4
c-3.7-10.1-11.2-18.9-17.1-28.2C2060.8,1426.2,2058.6,1419.6,2060.9,1411.1z"/>
<path class="st1" d="M2244.8,454.3c-23,66.2-59.6,121.8-106.6,172.4c-4.5-11.6-7.7-22.6-12.8-32.9c-4-8.1-8.4-15.8-14.4-22.7
c-12.6-14.4-29.9-17.7-45.6-3.8c-1.1,1-2,2.7-4.6,2.6c-1.4-5.5-1.1-10.5,2-15.7c8.8-14.9,18.9-29,25.8-45c1-2.3,2.7-3.7,4.7-4.9
c52.7-33.9,105.3-67.7,158-101.5c0.5-0.3,1.3-0.2,2.1-0.4c2.3,2.1,1.2,4.9,1.2,7.3c0.1,189.6,0.1,379.3,0.1,568.9
c0,1.7-0.1,3.3,0,5c0.2,3.6-1.7,4.6-4.9,4.5c-5-0.2-10-0.3-15-0.5c-17-0.5-34-1-51-1.4c-7.4-0.2-7.5-0.2-7.6-7.8
c-0.5-35.3-1.1-70.6-3.1-105.9c-2.6-45.9-6.4-91.7-12.8-137.3c-4.2-30.3-9.4-60.5-17.3-90.2c-1.1-4-0.2-6.6,2.5-9.8
c27.4-31.6,51.5-65.6,70.4-102.8c11.3-22.2,21.7-44.9,28.2-69.2C2244.6,460.9,2245.4,458.8,2244.8,454.3z"/>
<path class="st1" d="M553.7,662.8c3.8,0.2,5.2,2.8,6.9,4.6c14.5,15.2,30.3,29,46.9,41.9c4.1,3.2,5.8,6.4,5.8,11.7
c-0.2,85.6-0.1,171.2-0.1,256.8c0,7.5-0.1,7.7-7.6,7.8c-25,0.4-50,0.6-74.9,1.1c-4.3,0.1-5.9-1.2-5.7-5.6c0.4-13,0.2-26,0.5-39
c1.5-60.3,4.7-120.5,11.8-180.4c3.7-31.7,8.3-63.3,15.4-94.5C553,665.6,553.4,664.1,553.7,662.8z"/>
<path class="st1" d="M553.6,1319.7c-22.4-107.4-27.4-215.3-28.7-323.3c3.3-2.5,6.4-1.5,9.3-1.5c23.3,0.3,46.7,0.7,70,1.1
c9.5,0.1,9.2-1.2,9.2,9.2c0,72.3,0,144.7,0,217c0,13.3-0.1,26.7,0.1,40c0,3.6-0.7,6.2-3.8,8.6c-18.7,14.5-36.3,30.3-52.7,47.4
C556.3,1318.5,555.6,1318.7,553.6,1319.7z"/>
<path class="st1" d="M2119.8,1319.2c-6.7-6.5-12.1-12.3-18-17.5c-11.8-10.2-23.8-20.2-36-30.1c-4.2-3.4-5.9-7.1-5.9-12.7
c0.2-84.6,0.1-169.1,0.1-253.7c0-8.4,0.1-8.5,8-8.5c24.3-0.2,48.6-0.4,72.9-0.5c7.5,0,7.7,0,7.6,7.7c-0.3,45.6-1.9,91.2-5.1,136.7
c-2.7,39.9-6.7,79.6-12.5,119.1C2128.1,1279,2124.6,1298.3,2119.8,1319.2z"/>
<path class="st1" d="M2120.4,664.4c12.3,57.5,18.7,115.7,22.8,174.2c3.1,44.9,4.7,89.8,5.2,134.8c0,2.3-0.1,4.7,0.1,7
c0.3,4-1.7,5.2-5.4,5.1c-6.7-0.2-13.3-0.3-20-0.3c-18.3-0.1-36.7-0.2-55-0.4c-8-0.1-8.2-0.1-8.2-8.2c0-52.3,0-104.6,0-157
c0-32,0.2-64-0.2-96c-0.1-6.3,2-10.3,6.8-14.3c17.1-14.3,35.1-27.7,50.2-44.2C2117.5,664.3,2118.3,663.8,2120.4,664.4z"/>
<path class="st1" d="M878.7,1138.5c2,4.9,3.2,9.1,5.3,12.8c4,7.2,10.5,7.1,14.6,0c3-5.3,4.3-11.2,5.2-17.1c0.8-4.7,3.2-7.1,7.8-8.9
c37.6-14.8,76.3-25.6,115.5-34.7c5-1.1,6.6,0.4,7.3,4.9c0.6,3.5,0.7,8.4,5.3,8.5c5.3,0.1,5.5-5.2,5.8-9c0.5-7.1,3.9-10,10.9-11.2
c40-7,79.8-14.9,120.4-18c2.4-0.2,5.9-2.1,6.9,1.5c1,3.7,1.8,7.8,0.4,11.7c-0.7,1.8-2.9,1.2-4.4,1.5c-9.5,1.6-19.1,2.8-28.6,4.6
c-39.6,7.4-78.6,17-117,29.2c-59.9,19-116.5,45.2-170.8,76.4c-41.6,24-80.2,52.3-117.2,83.1c-39.4,32.8-74.3,69.6-105.4,110.2
c-3,3.9-6.2,7.7-9.4,11.6c-0.1,0.2-0.6,0.2-2.1,0.5c0.2-7-1.1-13.6,1.5-20.3c7.6-19.2,12.9-39.2,16.7-59.5
c1.8-9.6,6.2-17.5,12-24.8c17.2-21.7,36.5-41.5,58.8-58.9c3.4,2.2,3.5,6.6,5.4,9.8c4,7,7.8,14.4,17.4,14.4
c9.6,0,13.8-7.2,17.3-14.4c6.5-13.3,9.4-27.7,12.2-42c0.8-3.9,2.2-6.5,5.7-8.6C808.9,1171.4,842.7,1153.5,878.7,1138.5z"/>
<path class="st1" d="M1032.8,891.9c-42-9-82.2-20.4-121.3-35.7c-4.8-1.9-6.9-4.4-7.7-9c-0.9-5.6-2.3-11.1-4.8-16.2
c-1.5-3.2-3.7-5.9-7.3-6.1c-4.1-0.2-6.8,2.5-8.3,6.1c-1.6,3.6-2.8,7.4-4.7,12.2c-11.4-5.2-22.6-10-33.5-15.2
c-23.1-11.1-45.4-23.7-67.3-37.1c-4.3-2.6-6.6-5.5-7.5-10.7c-2.1-11.4-4.6-22.8-8.9-33.7c-2.1-5.3-4.2-10.5-7.8-15
c-7.6-9.4-17.5-9.4-25.3,0.1c-4.2,5.2-6,11.7-9.8,18.2c-11.1-9.9-22.3-18.8-32.2-28.9c-10.2-10.4-19.5-21.8-28.7-33.3
c-5.6-7-8.9-15.3-10.6-24.4c-3.7-19.3-8.6-38.3-16-56.5c-2.9-7.1-1.4-14.1-0.5-21.8c3.9,4.8,7.9,9.4,11.6,14.3
c23.7,32,51,60.7,80.1,87.7c19,17.6,39.4,33.9,59.9,49.7c45,34.7,93.8,63.3,144.7,87.9c39.2,18.9,79.8,34.7,121.6,47.1
c41.2,12.2,82.8,22.3,125.4,28.2c1,0.1,2,0.2,3,0.4c7.3,1.2,10.5,7.7,7,14.2c-1.3,2.3-3.4,1.7-5.3,1.6c-8-0.7-15.9-1.5-23.8-2.4
c-32.8-3.7-65.1-10.2-97.5-15.7c-7.1-1.2-11.5-3.9-11.7-11.6c0-1.3-0.4-2.6-0.8-3.9c-0.8-2.6-1.9-5.3-5.3-4.9
c-2.7,0.3-3.7,2.6-4.4,4.9C1034.4,885.3,1033.7,888.2,1032.8,891.9z"/>
<path class="st1" d="M1953.8,1233.7c3.8,1,5.8,3.4,8,5.3c21,19.1,40.7,39.4,57.6,62.2c3,4.1,4.9,8.4,6,13.5
c4.5,20.5,9.3,40.9,17.1,60.4c1.3,3.1,2.4,6.3,1.8,9.7c-0.6,3.6,2.4,8.9-2.1,10.4c-2.5,0.8-5.4-3.9-7.3-6.7
c-35.7-51.5-81-93.6-129.4-132.6c-40.8-32.9-84.7-61.2-131-85.6c-39.5-20.8-80.6-37.9-123-52c-45.9-15.2-92.6-26.8-140.3-34.3
c-6.2-1-12.4-2.7-18.7-3.1c-6.5-0.4-5.5-4.5-5.2-8.4c0.4-5.4,2.1-7.1,7.5-6.6c8.3,0.7,16.5,1.9,24.8,2.7
c33.2,3.4,65.8,10.1,98.6,15.7c6.3,1.1,9.2,3.4,9.6,10c0.2,3.6,0.3,9.3,5.3,9.6c6,0.4,5.5-5.6,6.3-9.5c0.7-3.9,2.4-5,6.2-4
c39.4,9.7,78.9,19.2,116.7,34.3c4.5,1.8,6.8,4.2,7.5,9.1c0.9,5.9,2.2,11.8,5.1,17.2c1.7,3.1,3.8,5.6,7.6,5.6c3.8,0,6-2.5,7.6-5.6
c0.9-1.8,1.7-3.6,2.3-5.5c2.8-7.7,2.9-7.9,9.9-4.7c32.6,15.3,64.6,31.4,95,50.8c3.4,2.2,5.1,4.7,5.8,8.6
c2.1,12.1,4.9,24.1,9.4,35.7c1.9,5,4.1,9.8,7.5,14c7.5,9.4,17.7,9.4,25.3,0C1948.9,1245.3,1950.9,1239.5,1953.8,1233.7z"/>
<path class="st1" d="M2043.6,585.1c0.9,7.5,1.7,14.1-0.8,20.4c-8,20.2-13.1,41.2-17.6,62.4c-0.9,4.4-2.6,8.3-5.2,11.7
c-17.8,23.8-38.4,45.2-60.6,64.9c-1.5,1.3-3.2,2.3-5.1,3.8c-2.9-5.3-5.1-10.4-8.2-14.9c-8-11.8-19.3-11.8-27.5-0.1
c-7.4,10.6-11.4,23-12.8,35.4c-1.7,14.6-9.4,22.2-21.3,29.3c-25.5,15.2-52.1,28.1-78.7,41.2c-2.4,1.2-5,2-7.3,3.1
c-2.7,1.3-4.3,0.7-5.2-2.2c-1-3.2-1.9-6.4-3.4-9.3c-4.2-7.7-10.8-7.7-15.1-0.1c-2.5,4.4-3.9,9.3-4.5,14.2c-0.8,7-4.2,10.7-11,13.3
c-27.5,10.2-55.4,18.9-83.9,25.7c-10,2.4-20.1,4.8-30,7.5c-3.8,1-5.4,0-6.1-3.9c-0.7-4-0.2-9.9-6.2-9.6c-5.1,0.2-5.3,6-5.4,9.5
c-0.1,7.5-4.3,9-10.5,10.2c-39.6,7.4-79.4,13.9-119.5,18.1c-2.8,0.3-6.7,2.3-8.1-1.3c-1.4-3.6-1.7-7.9-0.3-11.7
c0.9-2.4,3.9-2,6.2-2.4c13.8-2.4,27.6-4.4,41.3-7c42.1-8.2,83.5-19.3,124.2-33.1c57.6-19.6,112.2-45.7,163.9-77.3
c38.1-23.2,73.8-50.1,107.6-79.4c37.9-32.8,72.6-68.6,102.5-109C2037.5,591.3,2039.4,587.8,2043.6,585.1z"/>
<path class="st1" d="M777.2,1163.8c-0.9-7.9,1.2-15.4,2-23c4.4-45.1,6.3-90.3,6.6-135.6c0.1-8.6,0.2-8.7,8.8-8.7
c22-0.1,44-0.1,65.9,0c8.3,0,8.4,0.1,8.5,7.9c0.2,37.7,2,75.2,6.5,112.6c0.5,3.8,0.3,6-3.9,7.5c-31.7,11.3-62.2,25.3-92.6,39.6
C778.8,1164.3,778.4,1164.1,777.2,1163.8z"/>
<path class="st1" d="M776.8,817.8c3.1-0.6,4.7,0.6,6.5,1.4c28.2,13.5,56.6,26.5,86.2,36.8c5.4,1.9,6.7,4.3,6,10.1
c-4.5,36.7-6.3,73.6-6.4,110.6c0,8.1,0.7,8.3-8.1,8.3c-22.6-0.1-45.3,0-67.9-0.1c-7,0-7.1-0.2-7.2-7.3c-0.4-28.6-0.9-57.3-2.7-85.9
c-1.3-21.3-2.9-42.5-5.5-63.7C777.2,824.9,777.1,821.6,776.8,817.8z"/>
<path class="st1" d="M1896.4,1163.9c-3.3,0.5-5.7-1.5-8.4-2.8c-27.2-13.3-54.8-25.7-83.3-36.1c-7.4-2.7-7.2-2.9-6.3-10.6
c4.4-35.8,5.3-71.8,6.4-107.7c0.1-1.7,0.1-3.3,0-5c-0.3-3.8,1.3-5.4,5.2-5.3c6.3,0.2,12.7,0.1,19,0.1c16.7,0,33.3,0,50,0.1
c8.4,0,8.4,0.1,8.6,8.8c0.8,30.3,1.2,60.6,3.3,90.9c1.4,19.9,2.9,39.9,5.3,59.7C1896.5,1158.6,1897.4,1161.2,1896.4,1163.9z"/>
<path class="st1" d="M1896.8,817.7c-1.3,13.9-2.7,27.7-3.9,41.6c-3.5,38.5-4.5,77-5.2,115.6c0,1.7-0.2,3.3-0.1,5
c0.4,4-1.5,5.5-5.2,5.1c-1-0.1-2,0-3,0c-22.6,0-45.3-0.1-67.9,0.2c-5,0-6.8-1.2-6.8-6.6c-0.4-28.3-1.2-56.5-3.6-84.7
c-0.8-8.9-1.6-17.9-2.7-26.8c-0.9-7.6-1.1-7.8,6.5-10.6c23-8.7,45.8-17.8,67.9-28.7c6-2.9,12-5.7,18-8.6
C1892.5,818.3,1894.3,817.4,1896.8,817.7z"/>
<path class="st1" d="M908.3,1109.6c-1-4.4,0.4-8.6,0.8-12.8c2.5-30.9,4-61.8,4-92.9c0-7.4,0.2-7.5,7.9-7.5
c33.3-0.1,66.7-0.2,100-0.2c7.8,0,8.1,0.1,8,7.4c-0.3,22,1,44,2.3,65.9c0.3,4.5-1,6.5-5.5,7.4c-38.7,8.5-76.4,20.9-114.2,32.5
C910.7,1109.8,909.7,1109.6,908.3,1109.6z"/>
<path class="st1" d="M1765.3,1110.4c-9.6-3-18.1-5.6-26.6-8.4c-29.1-9.6-58.7-17.4-88.5-24.5c-8.9-2.1-8.1-1.4-7.6-10.4
c1.2-21,1.6-42,2.3-62.9c0.2-7.7,0.2-7.9,7.5-7.9c33.3,0,66.7,0.1,100,0.2c7.8,0,8.2,0.3,8.2,7.7c0.2,34,1.7,68,4.8,101.8
C1765.4,1106.9,1765.3,1107.9,1765.3,1110.4z"/>
<path class="st1" d="M908.8,872.2c2.8-1.1,5,0.8,7.5,1.5c36.1,10.3,71.6,22.9,108.5,30.5c5.5,1.1,6.7,3.6,6.4,8.8
c-1.3,21.9-2.6,43.9-2.1,65.9c0.1,4.7-1.2,6.5-6.1,6.4c-34.7-0.2-69.3-0.2-104-0.1c-4.1,0-6.1-1-6-5.6c0.4-35.6-2-71.2-4.8-106.7
C908.2,872.8,908.6,872.5,908.8,872.2z"/>
<path class="st1" d="M1765.6,872.4c-0.9,10.7-1.8,20.9-2.5,31.2c-1.7,24.6-2.4,49.2-2.6,73.8c-0.1,7.7-0.2,7.8-7.7,7.8
c-33.6,0.1-67.2,0.1-100.8,0.1c-6.9,0-6.9-0.2-7.1-7.1c-0.8-21.9-1.3-43.9-2.6-65.8c-0.3-5.4,1.8-7,6.2-8
c14.5-3.5,29-7.1,43.5-10.8c22.5-5.7,44.5-13.5,66.7-20.3C1760.5,872.7,1762.3,871.8,1765.6,872.4z"/>
<path class="st1" d="M725.8,1192.5c-5.9-34.3-8.8-67.4-11-100.6c-1.8-27.6-2.2-55.2-3-82.8c-0.1-2.3,0.1-4.7-0.1-7
c-0.4-4.4,1.6-5.8,5.9-5.7c9,0.2,18,0,27,0c6.7,0,13.3-0.1,20,0c7.2,0.1,7.4,0.2,7.3,7.1c-0.5,23.6-1,47.2-2,70.8
c-1.2,27.2-3.3,54.4-6.7,81.5c-0.2,1.3-0.3,2.6-0.4,4c-0.4,9.2-3.4,16-13,19.7C741.7,1182.7,734.5,1187.7,725.8,1192.5z"/>
<path class="st1" d="M1946.2,1191.7c-10.1-5.6-20.4-11.5-30.8-17.2c-3.3-1.8-3.1-5-3.5-7.8c-1.4-10.2-2.4-20.5-3.5-30.7
c-4.7-43.8-6-87.7-6.7-131.6c-0.1-7.7,0.1-7.8,7.7-7.8c15.3-0.1,30.6,0.1,46-0.1c4.6-0.1,6.5,1.1,6.4,6.1
c-0.5,34-1.2,67.9-3.7,101.8c-2,26.9-4.5,53.7-8.9,80.4C1948.7,1187,1949.1,1189.5,1946.2,1191.7z"/>
<path class="st1" d="M1947.7,789.1c5.8,32.7,8.6,63.8,10.8,95c2.1,30.6,2.8,61.2,3.1,91.9c0.1,9.6,0.3,8.9-8.3,8.9
c-15,0-30-0.1-45,0.2c-5.3,0.1-6.9-1.8-6.7-7c0.9-25.3,1.3-50.6,2.5-75.9c1.3-27.3,3.9-54.4,6.9-81.6c0.9-8.3,3.3-14.2,11.5-17.8
C1930.8,799.2,1938.6,794.1,1947.7,789.1z"/>
<path class="st1" d="M726,789.2c11.3,6.2,21.7,11.9,32.2,17.5c2.9,1.6,2.8,4.2,3.2,6.8c2.7,15.8,4.1,31.7,5.5,47.6
c3.3,37.2,4.5,74.5,5,111.8c0,2.3,0,4.7,0.1,7c0.1,3.1-0.8,5-4.4,5c-17-0.1-34-0.1-51-0.1c-3.3,0-4.8-1.5-4.7-4.8
c0.2-5.7,0-11.3,0.2-17c1-42.6,2.7-85.2,7.3-127.7C720.9,820.5,722.6,805.7,726,789.2z"/>
<path class="st1" d="M1625.1,910.5c-0.6,9.4-1.4,18-1.7,26.5c-0.4,13-0.4,26-0.6,38.9c-0.1,9.3-0.2,9.4-9.2,9.4
c-30.6,0.1-61.3,0.1-91.9,0.1c-8.3,0-16.7-0.1-25,0.1c-4.2,0.1-5.9-1.3-5.9-5.7c0-15-0.1-30-0.7-44.9c-0.2-4.8,2.5-5.5,6-6
c10.9-1.3,21.8-3.6,32.7-3.7c23.9-0.2,46.7-6.5,70-10C1607.1,913.9,1615.2,912.2,1625.1,910.5z"/>
<path class="st1" d="M1625.2,1071.1c-23.5-4.2-45.7-8.1-67.9-12.1c-7.9-1.4-15.8-2.4-23.8-2.5c-12.7-0.1-25.1-2.6-37.7-4
c-4-0.4-5.8-1.6-5.6-6.1c0.5-14.6,0.6-29.3,0.6-43.9c0-4.5,1-6.5,6.1-6.5c39.9,0.2,79.9,0.2,119.8,0.1c5,0,6.1,2.1,6.1,6.6
C1622.9,1025,1623,1047.3,1625.2,1071.1z"/>
<path class="st1" d="M1049.3,1070.4c-1-5.9-0.1-11.8,0.3-17.7c0.9-16.3,1.4-32.6,1.4-48.9c0-7.3,0.2-7.5,6.8-7.5
c39.3-0.1,78.5-0.1,117.8-0.1c7,0,7.1,0.2,7.2,7.2c0.2,14,0,27.9,0.5,41.9c0.2,5.2-1.5,6.6-6.6,7c-16.9,1.3-33.8,2.7-50.6,5.2
c-23.3,3.4-46.7,6.8-69.7,12.3C1054.2,1070.3,1051.9,1071.4,1049.3,1070.4z"/>
<path class="st1" d="M1049.8,910.2c22.1,5.7,44.8,8.7,67.4,12.5c16.4,2.8,33.1,3.3,49.6,5.2c16.4,1.9,16.4,1.5,16.3,17.4
c-0.1,11-0.3,22-0.2,32.9c0.1,4.9-1,7.2-6.6,7.2c-39.9-0.2-79.9-0.2-119.8-0.1c-3.4,0-5.7-0.6-5.6-4.8c0.6-22.6-1.3-45.2-1.9-67.8
C1049,912,1049.4,911.4,1049.8,910.2z"/>
<path class="st1" d="M1260.5,985.4c-21.6,0-43.2,0-64.9,0c-8,0-8.2-0.1-8.1-8.4c0.2-13.6,0.6-27.3,0.8-40.9c0.1-4.2,1.4-5.6,6-5.1
c41.7,4.1,83.5,6.2,125.5,6.2c2.3,0,4.7,0,7,0c7.5,0.2,7.5,0.2,7.6,7.9c0.1,10.3,0.1,20.6,0.1,30.9c0,9.3,0,9.4-9.1,9.4
C1303.8,985.4,1282.2,985.4,1260.5,985.4z"/>
<path class="st1" d="M1485,931.5c0.3,16.2,0.5,32.7,0.8,49.3c0.1,3.4-1.7,4.8-4.9,4.6c-2-0.1-4,0-6,0c-41.6,0-83.2,0-124.9,0
c-2,0-4-0.1-6,0c-4,0.3-5.1-1.8-5-5.5c0.1-11.7,0-23.3,0.1-35c0.1-7.6,0.1-7.8,7.6-7.8c17.3,0.2,34.6-0.6,51.9-1.1
c26.3-0.7,52.5-3.1,78.7-5.1C1479.6,930.9,1481.8,930,1485,931.5z"/>
<path class="st1" d="M1485,1051.1c-35.4-3-69.9-5.3-104.5-6.4c-11.6-0.4-23.3-0.5-34.9-0.4c-4.4,0-6.7-0.8-6.5-6
c0.4-12,0.3-24,0-36c-0.1-4.8,1.6-6.3,6.3-6.2c45,0.1,89.9,0.1,134.9,0c3.5,0,5.8,0.7,5.7,4.8
C1485.5,1017.5,1485.3,1034.1,1485,1051.1z"/>
<path class="st1" d="M1188.4,1050.6c-0.3-17.2-0.5-33.7-0.8-50.3c0-2.9,1.6-4.1,4.4-4.1c1.7,0,3.3,0,5,0c42.6,0,85.1,0,127.7,0
c1.7,0,3.3,0.1,5,0c3.3-0.2,4.9,1.2,4.9,4.6c-0.1,13-0.1,25.9,0,38.9c0,3.6-1.7,4.7-4.9,4.5c-0.7,0-1.3,0-2,0
C1281.6,1043.7,1235.9,1047.2,1188.4,1050.6z"/>
<path class="st1" d="M2010.7,1230.1c-5.1-1.9-9-5.1-13.1-7.7c-9.9-6.1-19.6-12.6-29.8-18.3c-4.8-2.6-5.5-5.6-4.5-10.4
c3.9-20,6.2-40.1,8.2-60.4c4-41.2,5.8-82.4,6.1-123.8c0-3,0.1-6,0-9c-0.1-3,1.1-3.8,4.1-4.1c16.5-1.5,16.7-1.4,16.8,14.7
c0.4,66.4,3.9,132.5,10.7,198.6C2009.9,1216.3,2011.6,1222.7,2010.7,1230.1z"/>
<path class="st1" d="M2011.2,752.5c-1.6,16.9-3.2,33.8-4.8,50.6c-3,30.8-4.6,61.7-6.1,92.6c-1.3,27.3-1.5,54.5-1.8,81.8
c-0.1,7.1-0.2,7.3-7.1,7.3c-16.3,0-13.6,1.3-13.7-12.8c-0.5-54.3-3.2-108.4-10.6-162.2c-1.1-7.9-2.5-15.8-3.9-23.6
c-0.7-3.6-0.1-6,3.5-8.2c13.4-8,26.6-16.4,39.8-24.7C2007.6,752.8,2008.5,751.7,2011.2,752.5z"/>
<path class="st1" d="M662.1,1231.5c1.9-20.4,3.7-38.6,5.4-56.8c3-31.9,4.7-63.8,6-95.8c1-24.6,1.6-49.3,1.6-74
c0-8.3,0.1-8.3,7.9-8.4c1.3,0,2.7,0,4,0c8.6,0.1,8.7,0.1,8.8,8.7c0.4,40,1.8,80,5.4,119.8c2,22.9,4.4,45.8,9,68.3
c1.1,5.5,0.2,8.5-5.1,11c-12.7,6-23.9,14.5-35.5,22.2C667.7,1227.9,665.8,1229.1,662.1,1231.5z"/>
<path class="st1" d="M662.9,752.1c2.6-0.9,3.9,1,5.4,2.1c12.4,8.4,24.6,17.1,38,23.9c3.3,1.7,5,3.6,4.2,7.7
c-11.3,61.8-13.7,124.4-14.8,187c-0.2,13.5,2.2,12.1-12.3,12.1c-8.4,0-8.5-0.1-8.5-8.9c0.1-52.7-2.6-105.3-6.4-157.8
c-1.5-20.6-4-41.1-6-61.7C662.5,755.1,661.9,753.4,662.9,752.1z"/>
<path class="st1" d="M2044.3,728.6c0,3.4,0,6.3,0,9.2c0,78.9,0,157.9,0,236.8c0,1.7-0.1,3.3,0,5c0.4,3.9-1.3,5.4-5.2,5.3
c-5.9-0.2-11.9,0-18.3,0c0-26.3,0.3-51.6,1.3-76.8c1.7-42.3,4.4-84.5,8.9-126.5c1.6-14.9,3.9-29.7,5.6-44.6
C2037.2,732.4,2038.9,729.7,2044.3,728.6z"/>
<path class="st1" d="M2044.3,1253c-5.6-1.2-7-4.1-7.7-8.6c-3.1-19.4-5.2-38.9-7-58.5c-5.6-59-8.7-118.2-8.7-177.6
c0-13.6-1.4-11.7,11.5-11.8c13.3,0,11.9-1.5,11.9,11.5c0,78.6,0,157.2,0,235.8C2044.3,1246.6,2044.3,1249.3,2044.3,1253z"/>
<path class="st1" d="M629.2,728.7c5.8,1.6,7.4,4.6,7.9,9c2.3,21.2,5.3,42.2,7.3,63.4c5.3,55.3,7.7,110.8,8.5,166.4
c0,3.3,0.1,6.7,0,10c-0.1,7.5-0.1,7.5-7.7,7.7c-16,0.5-16,0.5-16-15.3c0-76.5,0-153.1,0-229.6C629.2,736.7,629.2,733,629.2,728.7z"
/>
<path class="st1" d="M629.2,1253.2c0-6,0-10.6,0-15.3c0-75.7,0-151.3,0-227c0-2,0-4,0-6c-0.1-9.2-0.3-8.7,8.7-8.6
c15.1,0,15.2,0,15,15.6c-0.7,62.7-3.8,125.2-10.3,187.6c-1.5,14.9-3.9,29.7-5.5,44.6C636.6,1248.7,634.5,1251.1,629.2,1253.2z"/>
<path class="st1" d="M2060,697.5c0-15.9,0-29.5,0-43c0-3.7,0.3-7.3-0.1-11c-2-19,6.4-34.8,15.3-50.5c1-1.7,2.3-3.2,3.6-4.7
c5.4-6.2,8.1-6.2,13.4,0.4c6.3,7.8,10.3,17.1,13.7,26.4c3.2,8.7,5.9,17.6,8.7,26.5c1.3,4,1,7.1-2.4,10.7
C2096.7,668.2,2079,681.6,2060,697.5z"/>
<path class="st1" d="M613.2,1283.1c0,24.5,1.5,47.6-0.5,70.3c-1.4,15.4-9,29.8-19.7,41.7c-3.8,4.2-5.7,3.9-10-0.4
c-4.3-4.3-7.2-9.6-9.8-15c-6.7-13.8-11.2-28.4-15.3-43.2c-0.8-3-1.3-5.6,1.5-8.1C576.3,1313,592.4,1296.7,613.2,1283.1z"/>
<path class="st1" d="M2060,1284.4c19.3,15.5,36.8,29.3,52.5,45.2c2.6,2.6,3.7,5.1,2.6,8.8c-5.1,17.2-10,34.5-19.7,50
c-7.7,12.2-12.4,12.7-19.8,0.6c-9.5-15.4-16.9-31.4-15.8-50.5C2060.7,1321.2,2060,1303.9,2060,1284.4z"/>
<path class="st1" d="M611.9,698.2c-8.5-6.7-17.1-12.8-25-19.7c-9.1-7.8-17.5-16.4-26.3-24.5c-2.3-2.2-4-4.1-3-7.7
c5.3-18.2,10.3-36.5,20.4-52.8c7.8-12.7,12.8-13,20.4-0.3c8.9,14.9,16.3,30.2,15.1,48.6c-1,15.9-0.2,31.9-0.3,47.9
C613.2,692.2,614,694.9,611.9,698.2z"/>
<path class="st1" d="M1788.2,864c4.1,27,4.9,53.5,6.1,80.1c0.5,11.3,0.2,22.6,0.7,33.9c0.2,5.2-1.6,7.4-6.7,7.3
c-4.7-0.1-9.3-0.3-14,0c-4.6,0.3-5.5-2-5.4-6c0.2-26.3,1.3-52.5,3.2-78.8c0.5-7.6,1.2-15.2,2.1-22.8
C1775.6,867.2,1775.7,867.2,1788.2,864z"/>
<path class="st1" d="M885.3,1118.4c-5.1-39.1-6.6-77.6-6.6-116.3c0-4.2,1.3-5.9,5.7-5.8c25,0.7,20.1-4.1,19.8,19.4
c-0.4,29.6-1.5,59.3-5,88.8C897.9,1115.4,898.1,1115.4,885.3,1118.4z"/>
<path class="st1" d="M885.7,863.5c12.4,2.1,12.2,2.1,13.5,13.8c3.8,33,4.6,66.2,5.3,99.4c0.2,8.3,0,8.5-7.9,8.4
c-5.6-0.1-12.6,2.4-16.3-1.2c-3.8-3.6-1.3-10.7-1.3-16.2c0.1-33.9,1.8-67.7,6.1-101.4C885.1,865.4,885.4,864.5,885.7,863.5z"/>
<path class="st1" d="M1788.4,1117.5c-4,0.3-6.2-1.5-8.7-2.2c-3.2-0.9-4.6-2.9-4.9-6.5c-0.7-10.3-2.2-20.5-2.9-30.8
c-1.5-24.6-2.8-49.2-2.8-73.9c0-7.5,0.1-7.8,7.7-7.6c5.9,0.2,13.3-2.7,17.2,1.6c3.4,3.7,1,10.9,0.9,16.5
C1794.3,1048.8,1792.8,1083.1,1788.4,1117.5z"/>
<path class="st1" d="M1481.1,1078.1c-10.7,0.3-20.5-1.5-30.3-2.6c-34.1-3.7-68.2-5.9-102.5-5.6c-1.3,0-2.7-0.1-4,0
c-4.9,0.5-5.1-2.5-5.2-6.2c-0.1-4,0.7-6,5.4-6c20,0.1,39.9-0.2,59.9,1c22.9,1.4,45.8,3,68.7,4.7c7.9,0.6,7.8,0.9,8.2,8.7
C1481.2,1073.7,1481.1,1075.4,1481.1,1078.1z"/>
<path class="st1" d="M1480,903c2,3.3,0.7,6.3,1,9.2c0.4,4.3-1.6,5.6-5.8,5.9c-42.2,3.3-84.3,6.2-126.7,5.9c-1.3,0-2.7-0.1-4,0
c-4.5,0.5-5.5-1.7-5.6-5.8c-0.1-4.5,0.9-6.6,6-6.5c29.7,0.7,59.2-1.5,88.8-3.9C1449.3,906.5,1464.7,904.6,1480,903z"/>
<path class="st1" d="M1192.2,1078.2c1-14.7,0.9-15,13.5-15.8c40.5-2.9,80.9-5.4,121.6-4.6c7.3,0.1,7.4,0.5,7.3,7
c-0.1,4-1.7,5.2-5.4,5.1c-6-0.2-12-0.1-18,0c-37.3,0.2-74.3,3.2-111.3,7.8C1197.6,1078,1195.3,1078,1192.2,1078.2z"/>
<path class="st1" d="M1192.6,903.2c17.2,2.2,33.7,3.8,50.2,5c28.2,2.1,56.4,4.4,84.7,3.6c3.8-0.1,7.3-0.4,7.1,5.4
c-0.3,6.3-0.2,6.8-6.6,6.8c-18.6,0-37.2,0.4-55.9-0.8c-19.2-1.2-38.5-2.1-57.8-3.2c-3-0.2-6-0.4-8.9-0.8
C1192.8,917.6,1192.8,917.6,1192.6,903.2z"/>
<path class="st1" d="M653.6,694.6c2.5,2.7,3.4,3.5,4.2,4.6c15.1,20.9,32.8,39.4,51.8,56.7c2.5,2.3,7.7,4.8,5,8.5
c-2.7,3.9-6.2-1-8.9-2.4c-14.7-7.8-28.1-17.9-42.2-26.6c-3.1-1.9-4.3-4.4-4.8-8C657.4,717,655.5,706.9,653.6,694.6z"/>
<path class="st1" d="M654,1284.6c0.9-5.5,2.7-11,2.6-16.4c-0.1-14.4,6.3-23.8,19.2-30.2c10.6-5.3,20.3-12.5,30.5-18.8
c2.5-1.5,4.9-3.3,7.8-3.6c1.6,2.1,1.5,4,0.2,5.7c-1,1.3-2.4,2.3-3.6,3.4c-18.4,17.3-36.5,34.8-51.2,55.5c-1.1,1.6-2.6,2.9-4,4.3
C655.3,1284.8,654.8,1284.6,654,1284.6z"/>
<path class="st1" d="M2019.1,698.8c-1.4,9.8-3,19.6-4.2,29.5c-0.4,3-2.2,4.7-4.4,6.2c-15.3,9.9-30.7,19.7-46.1,29.5
c-1.5,1-3.7,4-5.7,0.4c-1.5-2.6,0.2-4.4,2.1-5.9c11.9-9.2,21.5-20.7,32.4-30.8c8.4-7.7,14.5-17.2,21.4-26.1c0.9-1.2,2-2.4,3-3.6
C2018.1,698.2,2018.6,698.5,2019.1,698.8z"/>
<path class="st1" d="M2017.1,1283.9c-11.8-19.6-28.7-34.6-44.6-50.5c-3.7-3.7-7.8-7.2-11.8-10.6c-1.9-1.7-3.4-3.4-1.7-5.9
c1.7-2.4,3.5-0.5,5,0.5c15.7,10,31.3,20,46.9,30c2,1.3,3.5,2.9,3.9,5.4c1.4,10.1,2.9,20.2,4.4,30.4
C2018.5,1283.4,2017.8,1283.7,2017.1,1283.9z"/>
<path class="st1" d="M1044.6,1034c-0.6,11-1.1,21.9-1.7,32.9c-0.1,1.6-0.3,3.3-0.8,4.9c-0.3,0.8-1.4,1.6-2.2,1.7
c-0.9,0.1-1.9-0.5-2.1-1.7c-0.1-1-0.3-2-0.4-3c-1.2-22.2-2.3-44.5-2.4-66.8c0-3.2,0-6,4.6-5.9c3.5,0.1,5.6,0.7,5.5,4.9
c-0.2,11-0.1,21.9-0.1,32.9C1044.9,1034,1044.8,1034,1044.6,1034z"/>
<path class="st1" d="M1636.4,1033.8c0,10.6,0,21.2,0,31.8c0,2-0.3,4-0.7,5.9c-0.2,0.8-1.2,1.8-1.9,1.8c-0.8,0-1.9-0.8-2.2-1.5
c-0.5-1.2-0.5-2.6-0.6-3.9c-1.9-21.8-2.2-43.7-2.1-65.6c0-5.7,1.3-6.7,6-6.1c3.9,0.5,3.6,3.1,3.6,5.8c0,10.6,0,21.2,0,31.8
C1637.8,1033.8,1637.1,1033.8,1636.4,1033.8z"/>
<path class="st1" d="M1638.5,948.1c0,10.3,0,20.6,0,31c0,2.6,0.5,5.4-3.3,6.2c-4.4,0.9-6.4-0.5-6.4-5.8c0-19,0-38,1.4-56.9
c0.2-3,0.4-6,0.7-9c0.2-2.2-0.3-5.4,3-5.2c2.5,0.1,2.2,3,2.3,4.9c0.6,11.6,1.1,23.3,1.6,34.9C1638,948.1,1638.2,948.1,1638.5,948.1
z"/>
<path class="st1" d="M1045.1,948c0,10.6-0.1,21.2,0,31.9c0,3.6-0.9,5.5-5,5.5c-3.9-0.1-5.1-1.7-5.1-5.4c0.1-22.6,1.2-45.1,2.5-67.7
c0.1-1.6-0.1-3.7,2-4c2.8-0.4,3.1,1.9,3.2,3.9c0.7,11.9,1.3,23.9,1.9,35.8C1044.8,948,1044.9,948,1045.1,948z"/>
<path class="st1" d="M741.9,740.1c7,10.8,9.7,22,12.6,34.5c-8.1-3.6-13.5-8.9-19.7-13c-2-1.3-2-3.3-1.2-5.3
C735.6,751.1,737,745.6,741.9,740.1z"/>
<path class="st1" d="M1931.7,1241.3c-6.9-10.6-9.7-21.7-12.1-33.3c4.2,0.4,6.4,3.1,9.1,4.9
C1943.2,1223.1,1943.4,1224.7,1931.7,1241.3z"/>
<path class="st1" d="M1931.6,740c12,17.2,11.9,18.3-3.6,29.1c-2.4,1.7-4.5,3.9-8.5,4.8C1921.8,761.9,1925,750.9,1931.6,740z"/>
<path class="st1" d="M742.1,1241.3c-5.3-5.2-6.5-11.1-8.6-16.5c-0.8-2-0.5-3.8,1.6-5.1c6-3.9,11.2-9.1,19.2-12.6
C751.8,1219.5,748.7,1230.3,742.1,1241.3z"/>
<path class="st1" d="M599.6,517.5c0.6,0.1,1.4,0.1,1.9,0.4c12.6,7.1,13,7.9,10.7,24.4c-5.9-7.9-11-15.2-13.4-24.1
C599,518,599.3,517.7,599.6,517.5z"/>
<path class="st1" d="M597.6,1464.4c4-9.4,8.1-17.2,14-24.4C614.9,1455.5,614.5,1457.9,597.6,1464.4z"/>
<path class="st1" d="M2072.4,516.4c2.2,0.6,2.3,2,1.3,3.7c-3.8,6.1-7.7,12.1-11.9,18.6c-2.8-10.6-2.1-12.7,5.3-17.8
c1.9-1.3,4.2-2,5.3-4.3L2072.4,516.4z"/>
<path class="st1" d="M2061.5,1442.5c4.7,8.5,10.5,14.5,12.5,22.8C2067.2,1459.8,2056.4,1456.8,2061.5,1442.5z"/>
<path class="st1" d="M1781.6,836.9c4.2,9.3,4.2,9.3-1.8,12.6C1777.9,845.4,1780.6,842.1,1781.6,836.9z"/>
<path class="st1" d="M895.4,849.3c-7.5-2.9-7.5-2.9-3.8-11.1C893.8,841.3,893.9,844.7,895.4,849.3z"/>
<path class="st1" d="M630.1,1284.6c-0.2-5.4-2.7-11,1.8-17C634.2,1274.4,629.7,1279.3,630.1,1284.6z"/>
<path class="st1" d="M629.9,697.1c0.5,5.4,3.9,10.3,2.4,16.6C626.9,708.5,630.1,702.5,629.9,697.1z"/>
<path class="st1" d="M2044.5,1284.5c-2.2-5-4.3-9.2-2.8-15C2046.1,1274.3,2043.5,1279.4,2044.5,1284.5z"/>
<path class="st1" d="M2041.9,712.1c-2.2-6,1.1-10.1,2.2-14.6C2043.5,702.2,2046,707.4,2041.9,712.1z"/>
<path class="st1" d="M892.2,1143.9c-4.6-7.9-4.6-7.9,1.6-11.5C895.7,1136.5,892.6,1139.7,892.2,1143.9z"/>
<path class="st1" d="M1779.6,1132.2c6.1,3.1,6.1,3.1,2.7,11.1c-1.9-1-1.7-3.1-2.3-4.6C1779.2,1137,1778.6,1135.1,1779.6,1132.2z"/>
<path class="st1" d="M1934.8,1202.5c4.1-0.6,6.5,2.5,11.4,4.9c-6.3,0.5-8.2-3.5-11.3-5.1L1934.8,1202.5z"/>
<path class="st1" d="M1935.8,777c3.1,0.9,4.5-3.5,7.8-2.9c-1.8,4-4.7,3.9-7.9,2.8L1935.8,777z"/>
<path class="st1" d="M729.5,774c2.7-0.5,4.4,0.6,5.8,3.2C732.1,778,731,775.7,729.5,774z"/>
<path class="st1" d="M734,1202.5c2.3,4.6-1.2,4.8-4.2,5.9c-0.6-3.7,3.7-3.3,4.2-5.7C734,1202.7,734,1202.5,734,1202.5z"/>
<path class="st1" d="M1935.2,780.3c-0.4,0.2-0.8,0.5-1.2,0.7c0.3,0,0.6,0.1,0.8,0c0.2-0.1,0.4-0.3,0.6-0.5
C1935.5,780.5,1935.2,780.3,1935.2,780.3z"/>
<path class="st1" d="M740,1203.1c-1.9-0.8-4.2,2.9-6-0.5c0,0,0,0.2,0,0.1C736,1202.8,738,1202.9,740,1203.1L740,1203.1z"/>
<path class="st1" d="M1935.7,777c1.4,1.3,0.9,2.4-0.5,3.3c0,0,0.3,0.3,0.3,0.3c0.1-1.2,0.3-2.3,0.4-3.5
C1935.8,777,1935.7,777,1935.7,777z"/>
<path class="st1" d="M737.9,778.4c0.2,0.2,0.4,0.5,0.6,0.7c-0.2,0.2-0.5,0.5-0.7,0.5c-0.2,0-0.4-0.4-0.6-0.6
C737.4,778.8,737.7,778.6,737.9,778.4z"/>
<path class="st1" d="M740,1203.1c0-0.3,0-0.8-0.2-1c-0.8-0.8-0.7-1.1,0.4-0.8C739.5,1201.9,739.4,1202.4,740,1203.1
C740,1203.1,740,1203.1,740,1203.1z"/>
<path class="st1" d="M1934.9,1202.3c0-0.2-0.1-0.3-0.1-0.5c0,0-0.1,0.1-0.1,0.1c0.1,0.2,0.1,0.4,0.2,0.6
C1934.8,1202.5,1934.9,1202.3,1934.9,1202.3z"/>
<path class="st1" d="M2072.4,516.6c0.7,0.4,1.4,0.8,2.1,1.2c-0.2,0.2-0.4,0.7-0.6,0.7c-1.1-0.2-1.5-1-1.4-2
C2072.4,516.4,2072.4,516.6,2072.4,516.6z"/>
<path class="st1" d="M598.7,518.3c-1-0.2-1.8-0.8-1-1.6c0.8-0.9,1.4,0,1.8,0.8C599.3,517.7,599,518,598.7,518.3z"/>
</g>
</svg>

After

Width:  |  Height:  |  Size: 48 KiB

View File

@ -0,0 +1,458 @@
<?xml version="1.0" encoding="utf-8"?>
<!-- Generator: Adobe Illustrator 24.2.1, SVG Export Plug-In . SVG Version: 6.00 Build 0) -->
<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
viewBox="0 0 2670 1980" style="enable-background:new 0 0 2670 1980;" xml:space="preserve">
<style type="text/css">
.st0{fill:#FCFCFB;}
.st1{fill:none;}
</style>
<g>
<path class="st0" d="M2275.3,1000.1c-0.1,2.3,0,4.7,0,7c0,189.3,0,378.7,0,568c0,6,0.1,12.1-0.8,17.9c-2.4,15.2-12.9,17.4-23.6,10
c-45.4-31-91.2-61.4-136.9-91.9l-30.1-20.3c-9.8-6.3-19.2-13.2-29.2-19.3c-6.7-4.1-9.8-9.7-9.9-17.2c-0.1-9.3-0.6-18.7-0.3-28
c0.2-7.3-1.7-13-6.5-19c-18.6-23.3-36.2-47.3-56.9-69c-18-18.8-36.7-36.8-56.4-53.8c-31.8-27.5-65.4-52.7-101-74.9
c-32.7-20.4-66.8-38.4-102.2-54c-50.7-22.4-103.2-39.2-157.1-51.5c-36.7-8.4-73.8-14.4-111.2-18.5c-34.5-3.8-69-6.5-103.7-5.8
c-23.1,0.5-46.2-1.3-69.3,0.4c-22.9,1.8-45.9,3-68.7,5.6c-72.7,8.4-143.7,24.6-212.1,50.6c-65.5,24.8-127.4,57-184.9,97.4
c-46.8,32.8-90.1,69.8-129.5,111.1c-18.1,19-35.1,39.2-50.5,60.6c-3.9,5.5-5.5,11-5.3,17.6c0.4,9.3,0,18.7,0,28
c0,9.3-3.1,16.4-11.8,21.4c-10.4,5.9-19.9,13.4-30.1,19.8l-25.4,17.2c-11.6,7.8-23.3,15.5-34.9,23.3c-34.3,23-68.6,46-102.9,69
c-14.6,9.7-23.6,5.5-25.2-12.1c-0.5-5.3-0.5-10.7-0.5-16c0-189.7,0-379.3,0-569l-0.1-26.1c0.2-190.3,0.1-380.7,0.2-571
c0-5.7,0.4-11.3,0.5-17c0.1-3.7,1.2-7.3,3-10.5c3.4-5.9,9.3-7.7,15.4-4.8c2.7,1.3,5.1,3.1,7.6,4.8c44.3,29.6,88.5,59.3,132.8,88.9
l29.7,20.1c10.8,6.6,20.9,14.3,31.7,20.9c6.7,4.1,9.7,9.8,9.8,17.3c0.2,11,0.4,22,0.2,33c-0.1,5.2,1.2,9.6,4.2,13.9
c15.4,22.2,33,42.8,51.6,62.2c53.9,56.4,114.1,105.1,181.4,144.6c38.4,22.6,78.3,42.4,119.9,58.6c40,15.6,80.9,28.3,122.8,38
c52.7,12.1,106,19.3,160,22.6c18.3,1.1,36.6,2.1,54.9,1.5c17.7-0.6,35.5,1.1,53.2-0.1c17.6-1.1,35.2-2.3,52.8-3.6
c35.9-2.5,71.4-8.5,106.7-15.1c53.4-9.9,105.5-25,156-45.1c53.8-21.5,105.1-48.2,153.2-80.9c43.4-29.5,83.6-62.7,121.1-99.3
c27.3-26.6,50.3-56.8,73.9-86.6c2.9-3.6,3.6-7.6,3.6-12.1c-0.1-10.7,0.2-21.3,0.4-32c0.1-7.5,3.2-13.1,9.9-17.2
c10.2-6.2,19.9-13.4,30-19.8l23.2-16.3c4.2-1.3,7.4-4.3,11-6.7c42.9-28.7,85.8-57.4,128.7-86.1c1.1-0.7,2.3-1.4,3.3-2.3
c10.9-9.1,21.4-4.7,23.8,10.7c0.8,5.6,0.6,11.3,0.6,17c0,189.3,0,378.7,0,568L2275.3,1000.1z M740,1203.1c-2-0.2-4-0.3-6-0.5
c-0.5,2.5-4.8,2-4.2,5.8c2.9-1,6.4-1.2,4.2-5.8C735.8,1206,738.1,1202.3,740,1203.1c-0.6-0.7-0.5-1.3,0.2-1.7
c-1.1-0.3-1.1-0.1-0.4,0.8C740,1202.3,739.9,1202.8,740,1203.1z M1935.5,780.5c-0.2,0.2-0.4,0.4-0.6,0.4c-0.3,0.1-0.6,0-0.8,0
c0.4-0.2,0.8-0.5,1.1-0.8c1.4-0.9,2-1.9,0.5-3.3c3.2,1.1,6.1,1.2,7.9-2.8c-3.3-0.7-4.6,3.8-7.7,2.9
C1935.7,778.2,1935.6,779.4,1935.5,780.5z M1934.8,1202.5c-0.1-0.2-0.1-0.4-0.2-0.6c0,0,0.1-0.1,0.1-0.1c0,0.2,0.1,0.3,0.1,0.5
c3.1,1.6,5,5.7,11.3,5.1C1941.3,1205,1938.9,1202,1934.8,1202.5z M2072.4,516.6c-1,2.3-3.4,3-5.3,4.3c-7.4,5.1-8.2,7.2-5.3,17.8
c4.2-6.5,8-12.6,11.9-18.6c1.1-1.7,0.9-3.1-1.3-3.6c0,1,0.3,1.8,1.4,2c0.2,0,0.4-0.4,0.6-0.7C2073.8,517.4,2073.1,517,2072.4,516.6
z M599.6,517.5c-0.4-0.8-1.1-1.7-1.8-0.8c-0.8,0.8,0.1,1.5,1,1.6c2.4,8.9,7.5,16.2,13.4,24.1c2.3-16.5,1.9-17.3-10.7-24.4
C600.9,517.6,600.2,517.6,599.6,517.5z M534.9,1353.5c-20,21.8-37.3,44.1-53.4,67.4c-16.1,23.4-26.8,49.5-41.8,73.5
c9-21,17.4-42.2,28.9-61.9c16.9-28.7,35.4-56.4,57.5-81.5c0.9-1,1.6-2.1,2.6-3.1c2.9-2.9,3.3-6,2.3-10
c-9.1-33.8-14.6-68.3-19.2-103c-5-38-8.3-76.1-10.6-114.3c-2.5-39.9-3.4-79.9-3.9-119.8c-0.1-5.1-1.4-6.8-6.7-6.9
c-21.6-0.2-43.3-0.8-64.9-1.3c-2,0-4-0.4-6,0.8c0,195.2,0,390.5,0,585.6c2.1,0.6,3.1-0.4,4.1-1.1c52.7-33.8,105.3-67.7,158-101.4
c3.6-2.3,5.7-5.4,7.3-9.1c6-13,12.5-25.8,20.8-37.4c4.2-5.9,3.9-11.9,2.5-19.1c-2.4,1.8-4,2.9-5.5,4.2
c-13.9,12.6-32.1,8.7-42.7-2.9c-7.3-8-12.4-17.2-17.1-26.8C542.3,1375.7,539.4,1365.1,534.9,1353.5z M534.6,627.2
c3.6-5,4.3-10.7,6.3-15.9c6.2-15.8,12.6-31.5,25.2-43.7c10.8-10.5,26.8-12.6,38.9-2.8c2.3,1.9,4.3,5,7.9,5
c1.6-8.3-0.2-15.3-5.1-21.9c-7.4-9.9-12.7-21-17.8-32.3c-2.4-5.3-5.8-9.2-10.8-12.4c-50.5-32.2-100.9-64.7-151.3-97.1
c-2.4-1.5-4.5-3.9-8.7-3.9c0,3.7,0,7.2,0,10.7c0,187.9,0,375.8,0,563.7c0,2,0.2,4,0,6c-0.6,4.9,1.4,6.3,6.2,6.1
c13.6-0.6,27.3-0.7,41-1c8.3-0.2,16.7-0.6,25-0.4c4.9,0.1,6.7-1.8,6.1-6.4c-0.1-1,0-2,0-3c1.2-75,4.3-149.8,13.7-224.3
c4.6-36.3,10-72.5,19.6-107.9c1.6-5.7,0.4-9.9-3.4-14.4c-10.7-12.8-21.7-25.4-31-39.1c-19-28.1-36.8-57-49.6-88.7
c-2.2-5.4-5.5-10.4-6.2-16.6C465,538.3,494.6,586,534.6,627.2z M2060.9,1411.1c-2.3,8.5-0.1,15.1,4,21.4
c5.9,9.3,13.4,18.1,17.1,28.2c3.9,10.7,10.9,16.6,19.9,22.4c47.8,30.3,95.3,61.1,143,91.7c2.7,1.7,5,4.3,9.5,4.5c0-3.4,0-6.6,0-9.8
c0-188.3,0-376.6,0-565c0-2-0.1-4,0-6c0.2-3.4-1.1-5-4.6-4.8c-2.3,0.1-4.7,0-7,0.1c-20,0.6-40,1.2-60,1.9c-6.7,0.2-6.7,0.4-7,7.4
c0,1.3,0,2.7-0.1,4c-0.4,60-3,119.9-8.9,179.7c-5,50.1-11.3,99.9-24,148.7c-1.4,5.2,0.1,8.3,3.5,12.3c15,17.8,29.6,36.1,42.4,55.5
c20,30.4,36.1,62.9,49.3,96.8c3.2,8.2,5.7,16.6,6.8,25.4c-33.9-90.8-82.6-145-107.2-169.7c-4.1,11-7.5,22.1-12.7,32.5
c-4.9,9.9-10.3,19.5-18.8,26.8c-7.1,6.1-15.1,8.9-24.8,7.5C2073.2,1421.4,2068.3,1415.5,2060.9,1411.1z M2244.8,454.3
c0.6,4.5-0.2,6.7-0.8,8.9c-6.5,24.2-17,47-28.2,69.2c-18.9,37.2-43,71.2-70.4,102.8c-2.7,3.2-3.6,5.8-2.5,9.8
c7.9,29.6,13.1,59.8,17.3,90.2c6.4,45.6,10.2,91.3,12.8,137.3c2,35.3,2.6,70.6,3.1,105.9c0.1,7.6,0.2,7.6,7.6,7.8
c17,0.5,34,1,51,1.4c5,0.1,10,0.2,15,0.5c3.2,0.1,5.1-0.9,4.9-4.5c-0.1-1.7,0-3.3,0-5c0-189.6,0-379.3-0.1-568.9
c0-2.4,1.1-5.2-1.2-7.3c-0.8,0.1-1.6,0-2.1,0.4c-52.7,33.8-105.3,67.7-158,101.5c-2,1.3-3.7,2.6-4.7,4.9c-6.9,16-16.9,30.1-25.8,45
c-3.1,5.2-3.4,10.2-2,15.7c2.6,0.1,3.5-1.6,4.6-2.6c15.7-13.9,33-10.6,45.6,3.8c6,6.8,10.4,14.6,14.4,22.7
c5.1,10.3,8.3,21.3,12.8,32.9C2185.2,576.1,2221.7,520.5,2244.8,454.3z M553.7,662.8c-0.3,1.3-0.7,2.8-1,4.3
c-7.1,31.2-11.7,62.8-15.4,94.5c-7.1,59.9-10.3,120.1-11.8,180.4c-0.3,13-0.2,26-0.5,39c-0.1,4.4,1.5,5.7,5.7,5.6
c25-0.5,50-0.7,74.9-1.1c7.4-0.1,7.6-0.2,7.6-7.8c0-85.6,0-171.2,0.1-256.8c0-5.3-1.7-8.5-5.8-11.7c-16.6-12.8-32.4-26.6-46.9-41.9
C559,665.6,557.6,663,553.7,662.8z M553.6,1319.7c2.1-1,2.8-1.2,3.2-1.6c16.4-17.1,34-32.9,52.7-47.4c3-2.4,3.8-5,3.8-8.6
c-0.1-13.3-0.1-26.7-0.1-40c0-72.3,0-144.7,0-217c0-10.3,0.4-9-9.2-9.2c-23.3-0.4-46.7-0.8-70-1.1c-2.8,0-5.9-1-9.3,1.5
C526.2,1104.4,531.1,1212.3,553.6,1319.7z M2119.8,1319.2c4.8-20.9,8.3-40.2,11.1-59.6c5.8-39.5,9.8-79.2,12.5-119.1
c3.1-45.5,4.7-91.1,5.1-136.7c0.1-7.6-0.1-7.7-7.6-7.7c-24.3,0.1-48.6,0.3-72.9,0.5c-7.9,0.1-8,0.1-8,8.5c0,84.6,0,169.1-0.1,253.7
c0,5.6,1.7,9.3,5.9,12.7c12.1,9.9,24.2,19.9,36,30.1C2107.7,1306.9,2113.1,1312.7,2119.8,1319.2z M2120.4,664.4
c-2.1-0.6-2.9-0.1-3.6,0.7c-15.2,16.5-33.1,29.9-50.2,44.2c-4.8,4-6.8,8-6.8,14.3c0.3,32,0.2,64,0.2,96c0,52.3,0,104.6,0,157
c0,8.1,0.1,8.2,8.2,8.2c18.3,0.2,36.7,0.2,55,0.4c6.7,0.1,13.3,0.1,20,0.3c3.7,0.1,5.7-1.1,5.4-5.1c-0.2-2.3,0-4.7-0.1-7
c-0.5-45-2.1-89.9-5.2-134.8C2139.2,780.1,2132.7,721.9,2120.4,664.4z M878.7,1138.5c-36,15-69.8,32.9-102.3,53.1
c-3.4,2.1-4.9,4.7-5.7,8.6c-2.8,14.4-5.7,28.8-12.2,42c-3.6,7.3-7.7,14.4-17.3,14.4c-9.6,0-13.4-7.4-17.4-14.4
c-1.9-3.2-2-7.6-5.4-9.8c-22.3,17.4-41.6,37.2-58.8,58.9c-5.8,7.3-10.2,15.3-12,24.8c-3.8,20.3-9.1,40.2-16.7,59.5
c-2.6,6.7-1.3,13.3-1.5,20.3c1.5-0.4,2-0.3,2.1-0.5c3.2-3.8,6.4-7.6,9.4-11.6c31.2-40.5,66-77.4,105.4-110.2
c36.9-30.8,75.6-59.1,117.2-83.1c54.3-31.2,110.9-57.4,170.8-76.4c38.4-12.2,77.4-21.8,117-29.2c9.5-1.8,19.1-3,28.6-4.6
c1.5-0.3,3.8,0.3,4.4-1.5c1.4-3.9,0.6-8-0.4-11.7c-1-3.6-4.5-1.7-6.9-1.5c-40.6,3.2-80.4,11-120.4,18c-7,1.2-10.4,4.1-10.9,11.2
c-0.3,3.8-0.5,9.1-5.8,9c-4.6-0.1-4.7-5-5.3-8.5c-0.7-4.5-2.4-6-7.3-4.9c-39.3,9.1-78,19.9-115.5,34.7c-4.6,1.8-7.1,4.2-7.8,8.9
c-0.9,5.9-2.2,11.8-5.2,17.1c-4.1,7.1-10.6,7.2-14.6,0C881.9,1147.7,880.7,1143.4,878.7,1138.5z M1032.8,891.9
c0.9-3.8,1.6-6.7,2.4-9.5c0.7-2.3,1.7-4.6,4.4-4.9c3.4-0.3,4.4,2.3,5.3,4.9c0.4,1.3,0.8,2.6,0.8,3.9c0.1,7.7,4.5,10.4,11.7,11.6
c32.5,5.5,64.8,11.9,97.5,15.7c7.9,0.9,15.9,1.7,23.8,2.4c1.9,0.2,4,0.7,5.3-1.6c3.5-6.5,0.3-13-7-14.2c-1-0.2-2-0.2-3-0.4
c-42.6-5.8-84.2-15.9-125.4-28.2c-41.8-12.4-82.5-28.2-121.6-47.1c-51-24.6-99.8-53.3-144.7-87.9c-20.6-15.8-40.9-32.1-59.9-49.7
c-29.1-26.9-56.4-55.7-80.1-87.7c-3.7-4.9-7.7-9.6-11.6-14.3c-0.8,7.6-2.4,14.7,0.5,21.8c7.4,18.2,12.4,37.2,16,56.5
c1.7,9.1,5,17.3,10.6,24.4c9.2,11.4,18.5,22.8,28.7,33.3c9.9,10.1,21.1,19,32.2,28.9c3.8-6.5,5.6-13,9.8-18.2
c7.7-9.4,17.7-9.5,25.3-0.1c3.6,4.5,5.7,9.7,7.8,15c4.3,10.9,6.8,22.3,8.9,33.7c0.9,5.2,3.3,8.1,7.5,10.7
c21.9,13.4,44.1,26,67.3,37.1c10.9,5.2,22.1,10,33.5,15.2c1.8-4.8,3.1-8.6,4.7-12.2c1.6-3.5,4.2-6.3,8.3-6.1
c3.6,0.2,5.8,2.9,7.3,6.1c2.4,5.2,3.8,10.6,4.8,16.2c0.8,4.6,2.9,7.2,7.7,9C950.6,871.5,990.8,883,1032.8,891.9z M1953.8,1233.7
c-2.9,5.8-4.9,11.5-8.8,16.2c-7.6,9.3-17.8,9.3-25.3,0c-3.4-4.2-5.5-9-7.5-14c-4.4-11.5-7.2-23.5-9.4-35.7
c-0.7-3.9-2.4-6.4-5.8-8.6c-30.4-19.4-62.4-35.6-95-50.8c-6.9-3.3-7-3-9.9,4.7c-0.7,1.9-1.5,3.7-2.3,5.5c-1.6,3.1-3.7,5.6-7.6,5.6
c-3.8,0-5.9-2.5-7.6-5.6c-2.9-5.4-4.3-11.3-5.1-17.2c-0.7-5-3-7.3-7.5-9.1c-37.8-15.2-77.3-24.6-116.7-34.3c-3.8-0.9-5.4,0.2-6.2,4
c-0.7,3.9-0.2,9.9-6.3,9.5c-5-0.3-5.1-6.1-5.3-9.6c-0.4-6.6-3.3-8.9-9.6-10c-32.8-5.6-65.4-12.2-98.6-15.7
c-8.3-0.9-16.5-2.1-24.8-2.7c-5.4-0.4-7.1,1.2-7.5,6.6c-0.3,3.9-1.3,8,5.2,8.4c6.3,0.4,12.4,2.1,18.7,3.1
c47.7,7.5,94.4,19.1,140.3,34.3c42.4,14,83.5,31.2,123,52c46.3,24.4,90.2,52.6,131,85.6c48.5,39.1,93.7,81.2,129.4,132.6
c1.9,2.8,4.8,7.5,7.3,6.7c4.5-1.5,1.4-6.8,2.1-10.4c0.6-3.4-0.5-6.6-1.8-9.7c-7.8-19.5-12.6-40-17.1-60.4c-1.1-5.1-3-9.4-6-13.5
c-17-22.8-36.7-43.1-57.6-62.2C1959.6,1237.1,1957.6,1234.7,1953.8,1233.7z M2043.6,585.1c-4.2,2.7-6.2,6.2-8.5,9.2
c-29.9,40.3-64.6,76.1-102.5,109c-33.8,29.3-69.5,56.2-107.6,79.4c-51.8,31.6-106.3,57.7-163.9,77.3
c-40.7,13.8-82.1,24.9-124.2,33.1c-13.7,2.7-27.6,4.7-41.3,7c-2.2,0.4-5.3-0.1-6.2,2.4c-1.4,3.8-1.1,8.2,0.3,11.7
c1.4,3.5,5.3,1.6,8.1,1.3c40.1-4.2,79.8-10.7,119.5-18.1c6.2-1.2,10.4-2.7,10.5-10.2c0-3.5,0.3-9.3,5.4-9.5c6-0.3,5.5,5.7,6.2,9.6
c0.7,3.9,2.3,4.9,6.1,3.9c10-2.7,20-5.1,30-7.5c28.5-6.9,56.4-15.6,83.9-25.7c6.8-2.5,10.2-6.3,11-13.3c0.5-5,2-9.8,4.5-14.2
c4.2-7.6,10.8-7.6,15.1,0.1c1.6,2.9,2.5,6.2,3.4,9.3c0.9,2.9,2.5,3.5,5.2,2.2c2.4-1.2,5-1.9,7.3-3.1c26.6-13.1,53.2-26,78.7-41.2
c11.9-7.1,19.6-14.7,21.3-29.3c1.4-12.4,5.5-24.8,12.8-35.4c8.1-11.7,19.5-11.7,27.5,0.1c3.1,4.6,5.4,9.7,8.2,14.9
c1.9-1.4,3.6-2.5,5.1-3.8c22.2-19.8,42.7-41.1,60.6-64.9c2.6-3.5,4.3-7.3,5.2-11.7c4.5-21.2,9.6-42.2,17.6-62.4
C2045.3,599.2,2044.5,592.6,2043.6,585.1z M777.2,1163.8c1.2,0.2,1.6,0.4,1.9,0.3c30.4-14.3,60.9-28.3,92.6-39.6
c4.2-1.5,4.4-3.7,3.9-7.5c-4.5-37.4-6.3-75-6.5-112.6c0-7.8-0.2-7.9-8.5-7.9c-22,0-44,0-65.9,0c-8.6,0-8.8,0.1-8.8,8.7
c-0.4,45.3-2.2,90.5-6.6,135.6C778.4,1148.4,776.3,1155.9,777.2,1163.8z M776.8,817.8c0.3,3.8,0.5,7.1,0.9,10.4
c2.5,21.2,4.1,42.4,5.5,63.7c1.8,28.6,2.3,57.2,2.7,85.9c0.1,7.1,0.3,7.3,7.2,7.3c22.6,0.1,45.3,0.1,67.9,0.1
c8.8,0,8.1-0.2,8.1-8.3c0.1-37,1.9-73.9,6.4-110.6c0.7-5.8-0.5-8.2-6-10.1c-29.5-10.3-58-23.3-86.2-36.8
C781.5,818.4,779.9,817.2,776.8,817.8z M1896.4,1163.9c1-2.7,0.2-5.3-0.1-7.9c-2.4-19.8-3.9-39.8-5.3-59.7
c-2.1-30.3-2.5-60.6-3.3-90.9c-0.2-8.7-0.2-8.8-8.6-8.8c-16.7-0.1-33.3,0-50-0.1c-6.3,0-12.7,0.2-19-0.1c-3.9-0.1-5.5,1.4-5.2,5.3
c0.1,1.7,0.1,3.3,0,5c-1.1,36-2,72-6.4,107.7c-0.9,7.7-1.1,7.9,6.3,10.6c28.5,10.4,56.1,22.9,83.3,36.1
C1890.6,1162.4,1893.1,1164.4,1896.4,1163.9z M1896.8,817.7c-2.5-0.3-4.3,0.6-6.1,1.5c-6,2.9-12,5.6-18,8.6
c-22.1,10.9-44.9,20.1-67.9,28.7c-7.5,2.8-7.3,3-6.5,10.6c1,8.9,1.9,17.9,2.7,26.8c2.4,28.2,3.2,56.5,3.6,84.7
c0.1,5.4,1.8,6.7,6.8,6.6c22.6-0.2,45.3-0.1,67.9-0.2c1,0,2-0.1,3,0c3.8,0.3,5.6-1.2,5.2-5.1c-0.2-1.6,0-3.3,0.1-5
c0.7-38.6,1.7-77.2,5.2-115.6C1894.1,845.4,1895.5,831.6,1896.8,817.7z M908.3,1109.6c1.4,0,2.4,0.2,3.2,0
c37.8-11.7,75.5-24,114.2-32.5c4.5-1,5.8-2.9,5.5-7.4c-1.3-22-2.5-43.9-2.3-65.9c0.1-7.3-0.2-7.4-8-7.4c-33.3,0-66.7,0.1-100,0.2
c-7.7,0-7.9,0.1-7.9,7.5c0,31-1.6,61.9-4,92.9C908.8,1101,907.3,1105.2,908.3,1109.6z M1765.3,1110.4c0-2.5,0.1-3.5,0-4.4
c-3.1-33.9-4.7-67.8-4.8-101.8c0-7.4-0.4-7.6-8.2-7.7c-33.3-0.1-66.7-0.2-100-0.2c-7.3,0-7.2,0.1-7.5,7.9c-0.7,21-1.1,42-2.3,62.9
c-0.5,9-1.3,8.3,7.6,10.4c29.8,7.1,59.4,14.9,88.5,24.5C1747.2,1104.8,1755.7,1107.4,1765.3,1110.4z M908.8,872.2
c-0.2,0.3-0.6,0.6-0.6,0.8c2.8,35.5,5.2,71,4.8,106.7c-0.1,4.6,1.9,5.6,6,5.6c34.7-0.1,69.3-0.1,104,0.1c4.9,0,6.2-1.8,6.1-6.4
c-0.5-22,0.8-44,2.1-65.9c0.3-5.2-0.9-7.7-6.4-8.8c-36.9-7.6-72.4-20.1-108.5-30.5C913.8,873,911.6,871.1,908.8,872.2z
M1765.6,872.4c-3.2-0.6-5.1,0.3-7,0.9c-22.2,6.7-44.2,14.5-66.7,20.3c-14.5,3.7-29,7.3-43.5,10.8c-4.4,1.1-6.5,2.6-6.2,8
c1.3,21.9,1.8,43.9,2.6,65.8c0.2,7,0.2,7.1,7.1,7.1c33.6,0,67.2,0,100.8-0.1c7.5,0,7.7-0.2,7.7-7.8c0.2-24.6,0.9-49.2,2.6-73.8
C1763.8,893.4,1764.7,883.1,1765.6,872.4z M725.8,1192.5c8.7-4.8,15.8-9.8,23.8-12.8c9.6-3.7,12.7-10.5,13-19.7
c0.1-1.3,0.2-2.7,0.4-4c3.4-27.1,5.6-54.3,6.7-81.5c1-23.6,1.5-47.2,2-70.8c0.2-7-0.1-7-7.3-7.1c-6.7-0.1-13.3,0-20,0
c-9,0-18,0.2-27,0c-4.2-0.1-6.2,1.3-5.9,5.7c0.2,2.3,0,4.7,0.1,7c0.8,27.6,1.2,55.2,3,82.8C717,1125.1,719.9,1158.2,725.8,1192.5z
M1946.2,1191.7c2.9-2.2,2.4-4.7,2.8-6.9c4.5-26.6,6.9-53.5,8.9-80.4c2.5-33.9,3.2-67.9,3.7-101.8c0.1-5-1.8-6.2-6.4-6.1
c-15.3,0.2-30.6,0-46,0.1c-7.6,0-7.8,0.2-7.7,7.8c0.7,44,2,87.9,6.7,131.6c1.1,10.3,2.2,20.5,3.5,30.7c0.4,2.8,0.2,6,3.5,7.8
C1925.8,1180.2,1936.2,1186.1,1946.2,1191.7z M1947.7,789.1c-9.1,5-16.9,10.1-25.3,13.8c-8.2,3.6-10.5,9.5-11.5,17.8
c-3,27.1-5.6,54.3-6.9,81.6c-1.2,25.3-1.5,50.6-2.5,75.9c-0.2,5.1,1.5,7,6.7,7c15-0.3,30-0.1,45-0.2c8.6,0,8.4,0.6,8.3-8.9
c-0.3-30.6-1-61.3-3.1-91.9C1956.3,852.9,1953.6,821.8,1947.7,789.1z M726,789.2c-3.3,16.4-5.1,31.3-6.7,46.2
c-4.6,42.4-6.3,85-7.3,127.7c-0.1,5.7,0,11.3-0.2,17c-0.1,3.4,1.4,4.8,4.7,4.8c17,0,34,0,51,0.1c3.6,0,4.6-1.9,4.4-5
c-0.1-2.3-0.1-4.7-0.1-7c-0.5-37.3-1.7-74.6-5-111.8c-1.4-15.9-2.8-31.9-5.5-47.6c-0.4-2.6-0.3-5.2-3.2-6.8
C747.6,801.1,737.2,795.4,726,789.2z M1625.1,910.5c-9.8,1.8-18,3.4-26.2,4.6c-23.3,3.5-46.2,9.9-70,10
c-10.9,0.1-21.8,2.4-32.7,3.7c-3.5,0.4-6.2,1.1-6,6c0.5,15,0.7,30,0.7,44.9c0,4.4,1.7,5.8,5.9,5.7c8.3-0.2,16.6,0,25-0.1
c30.6,0,61.3,0,91.9-0.1c9,0,9-0.1,9.2-9.4c0.2-13,0.2-26,0.6-38.9C1623.7,928.4,1624.5,919.8,1625.1,910.5z M1625.2,1071.1
c-2.2-23.8-2.3-46.1-2.4-68.4c0-4.5-1-6.6-6.1-6.6c-39.9,0.1-79.9,0.1-119.8-0.1c-5.1,0-6.1,2-6.1,6.5c0,14.6-0.1,29.3-0.6,43.9
c-0.1,4.5,1.6,5.6,5.6,6.1c12.6,1.4,25,3.8,37.7,4c8,0.1,15.9,1.1,23.8,2.5C1579.5,1063,1601.7,1066.9,1625.2,1071.1z
M1049.3,1070.4c2.6,1,4.9-0.1,7-0.6c23-5.5,46.3-8.9,69.7-12.3c16.7-2.5,33.7-3.8,50.6-5.2c5.1-0.4,6.8-1.9,6.6-7
c-0.4-14-0.3-27.9-0.5-41.9c-0.1-7-0.2-7.2-7.2-7.2c-39.3,0-78.5,0-117.8,0.1c-6.7,0-6.8,0.2-6.8,7.5c0,16.3-0.5,32.6-1.4,48.9
C1049.3,1058.6,1048.4,1064.5,1049.3,1070.4z M1049.8,910.2c-0.4,1.2-0.8,1.8-0.8,2.4c0.6,22.6,2.5,45.2,1.9,67.8
c-0.1,4.2,2.1,4.8,5.6,4.8c39.9,0,79.9-0.1,119.8,0.1c5.5,0,6.7-2.3,6.6-7.2c-0.2-11,0.1-22,0.2-32.9c0.1-15.9,0.1-15.5-16.3-17.4
c-16.5-1.9-33.1-2.4-49.6-5.2C1094.7,918.9,1072,915.9,1049.8,910.2z M1260.5,985.4c21.6,0,43.2,0,64.9,0c9.1,0,9.1-0.1,9.1-9.4
c0-10.3,0-20.6-0.1-30.9c-0.1-7.7-0.1-7.7-7.6-7.9c-2.3-0.1-4.7,0-7,0c-41.9,0-83.8-2.2-125.5-6.2c-4.6-0.4-5.9,0.9-6,5.1
c-0.2,13.6-0.6,27.3-0.8,40.9c-0.1,8.3,0.1,8.4,8.1,8.4C1217.3,985.4,1238.9,985.4,1260.5,985.4z M1485,931.5
c-3.1-1.4-5.4-0.6-7.7-0.4c-26.2,1.9-52.4,4.3-78.7,5.1c-17.3,0.5-34.6,1.3-51.9,1.1c-7.4-0.1-7.5,0.1-7.6,7.8
c-0.1,11.7,0,23.3-0.1,35c0,3.7,1,5.8,5,5.5c2-0.1,4,0,6,0c41.6,0,83.2,0,124.9,0c2,0,4-0.1,6,0c3.3,0.1,5-1.2,4.9-4.6
C1485.5,964.2,1485.3,947.6,1485,931.5z M1485,1051.1c0.3-17,0.5-33.6,0.9-50.1c0.1-4.2-2.2-4.8-5.7-4.8c-45,0.1-89.9,0.1-134.9,0
c-4.7,0-6.5,1.4-6.3,6.2c0.3,12,0.4,24,0,36c-0.2,5.2,2.1,6,6.5,6c11.6-0.1,23.3,0.1,34.9,0.4
C1415,1045.8,1449.5,1048.1,1485,1051.1z M1188.4,1050.6c47.5-3.4,93.2-6.8,139.2-6.4c0.7,0,1.3,0,2,0c3.2,0.2,4.9-0.9,4.9-4.5
c-0.1-13,0-25.9,0-38.9c0-3.4-1.6-4.8-4.9-4.6c-1.7,0.1-3.3,0-5,0c-42.6,0-85.1,0-127.7,0c-1.7,0-3.3,0.1-5,0
c-2.8-0.1-4.4,1.1-4.4,4.1C1187.9,1016.8,1188.1,1033.4,1188.4,1050.6z M2010.7,1230.1c0.9-7.3-0.9-13.8-1.5-20.3
c-6.8-66-10.3-132.2-10.7-198.6c-0.1-16.1-0.3-16.3-16.8-14.7c-2.9,0.3-4.2,1.1-4.1,4.1c0.1,3,0,6,0,9
c-0.3,41.3-2.1,82.6-6.1,123.8c-2,20.2-4.3,40.4-8.2,60.4c-0.9,4.8-0.2,7.8,4.5,10.4c10.2,5.7,19.9,12.2,29.8,18.3
C2001.8,1225,2005.6,1228.2,2010.7,1230.1z M2011.2,752.5c-2.7-0.8-3.6,0.3-4.7,1c-13.3,8.3-26.4,16.7-39.8,24.7
c-3.6,2.1-4.2,4.5-3.5,8.2c1.4,7.8,2.8,15.7,3.9,23.6c7.4,53.8,10.1,107.9,10.6,162.2c0.1,14.1-2.6,12.8,13.7,12.8
c7,0,7-0.2,7.1-7.3c0.4-27.3,0.6-54.6,1.8-81.8c1.4-30.9,3.1-61.8,6.1-92.6C2008,786.3,2009.6,769.5,2011.2,752.5z M662.1,1231.5
c3.7-2.4,5.6-3.7,7.5-4.9c11.6-7.7,22.9-16.2,35.5-22.2c5.3-2.5,6.2-5.5,5.1-11c-4.6-22.5-7-45.4-9-68.3
c-3.5-39.9-5-79.8-5.4-119.8c-0.1-8.6-0.2-8.6-8.8-8.7c-1.3,0-2.7,0-4,0c-7.9,0.1-8,0.1-7.9,8.4c0,24.7-0.6,49.3-1.6,74
c-1.3,32-3,63.9-6,95.8C665.8,1192.9,664.1,1211.1,662.1,1231.5z M662.9,752.1c-1,1.4-0.4,3-0.3,4.6c2,20.5,4.6,41.1,6,61.7
c3.8,52.5,6.5,105.1,6.4,157.8c0,8.8,0.1,8.9,8.5,8.9c14.5-0.1,12.1,1.3,12.3-12.1c1.1-62.7,3.5-125.2,14.8-187
c0.8-4.1-0.9-6-4.2-7.7c-13.4-6.8-25.6-15.5-38-23.9C666.8,753.1,665.5,751.1,662.9,752.1z M2044.3,728.6c-5.4,1.1-7.1,3.9-7.7,8.3
c-1.7,14.9-4,29.7-5.6,44.6c-4.6,42.1-7.2,84.2-8.9,126.5c-1,25.3-1.3,50.5-1.3,76.8c6.5,0,12.4-0.1,18.3,0
c3.8,0.1,5.5-1.4,5.2-5.3c-0.2-1.7,0-3.3,0-5c0-78.9,0-157.9,0-236.8C2044.3,734.9,2044.3,732,2044.3,728.6z M2044.3,1253
c0-3.7,0-6.4,0-9c0-78.6,0-157.2,0-235.8c0-13,1.3-11.5-11.9-11.5c-13,0-11.5-1.8-11.5,11.8c0,59.3,3.1,118.5,8.7,177.6
c1.9,19.5,4,39.1,7,58.5C2037.3,1248.9,2038.8,1251.8,2044.3,1253z M629.2,728.7c0,4.3,0,8,0,11.6c0,76.5,0,153.1,0,229.6
c0,15.8,0,15.8,16,15.3c7.6-0.2,7.6-0.2,7.7-7.7c0.1-3.3,0-6.7,0-10c-0.8-55.6-3.1-111.1-8.5-166.4c-2-21.2-5-42.3-7.3-63.4
C636.6,733.3,635,730.3,629.2,728.7z M629.2,1253.2c5.3-2.1,7.4-4.5,7.9-9.2c1.6-14.9,4-29.7,5.5-44.6
c6.5-62.4,9.5-124.9,10.3-187.6c0.2-15.6,0.1-15.6-15-15.6c-9,0-8.8-0.6-8.7,8.6c0,2,0,4,0,6c0,75.7,0,151.3,0,227
C629.2,1242.5,629.2,1247.2,629.2,1253.2z M2060,697.5c19-15.9,36.7-29.3,52.3-45.3c3.5-3.5,3.7-6.7,2.4-10.7
c-2.8-8.9-5.4-17.8-8.7-26.5c-3.5-9.3-7.4-18.5-13.7-26.4c-5.3-6.6-8-6.6-13.4-0.4c-1.3,1.5-2.7,3-3.6,4.7
c-8.9,15.7-17.3,31.5-15.3,50.5c0.4,3.6,0.1,7.3,0.1,11C2060,668,2060,681.5,2060,697.5z M613.2,1283.1
c-20.8,13.6-36.9,29.9-53.8,45.4c-2.8,2.6-2.3,5.1-1.5,8.1c4.1,14.7,8.6,29.3,15.3,43.2c2.6,5.4,5.4,10.7,9.8,15
c4.3,4.2,6.2,4.6,10,0.4c10.7-11.9,18.4-26.3,19.7-41.7C614.7,1330.6,613.2,1307.6,613.2,1283.1z M2060,1284.4
c0,19.5,0.7,36.8-0.2,54c-1.1,19.1,6.3,35.2,15.8,50.5c7.4,12.1,12.1,11.7,19.8-0.6c9.7-15.4,14.6-32.7,19.7-50
c1.1-3.7,0-6.2-2.6-8.8C2096.8,1313.6,2079.3,1299.9,2060,1284.4z M611.9,698.2c2.1-3.3,1.3-5.9,1.3-8.4c0.1-16-0.8-32,0.3-47.9
c1.2-18.4-6.2-33.8-15.1-48.6c-7.6-12.7-12.6-12.4-20.4,0.3c-10.1,16.3-15.1,34.6-20.4,52.8c-1,3.6,0.6,5.5,3,7.7
c8.8,8.1,17.3,16.7,26.3,24.5C594.8,685.4,603.3,691.5,611.9,698.2z M1788.2,864c-12.5,3.2-12.6,3.2-13.9,13.8
c-0.9,7.6-1.6,15.2-2.1,22.8c-1.8,26.2-2.9,52.5-3.2,78.8c0,4,0.8,6.3,5.4,6c4.6-0.3,9.3-0.2,14,0c5.1,0.1,6.9-2.1,6.7-7.3
c-0.5-11.3-0.2-22.6-0.7-33.9C1793.2,917.5,1792.3,890.9,1788.2,864z M885.3,1118.4c12.8-3,12.5-3,13.8-13.9
c3.5-29.5,4.6-59.1,5-88.8c0.3-23.5,5.2-18.7-19.8-19.4c-4.4-0.1-5.7,1.5-5.7,5.8C878.8,1040.8,880.3,1079.4,885.3,1118.4z
M885.7,863.5c-0.2,0.9-0.6,1.9-0.7,2.9c-4.3,33.7-6,67.5-6.1,101.4c0,5.5-2.5,12.6,1.3,16.2c3.8,3.6,10.8,1.1,16.3,1.2
c7.9,0.1,8.1-0.1,7.9-8.4c-0.6-33.2-1.5-66.4-5.3-99.4C897.9,865.6,898,865.6,885.7,863.5z M1788.4,1117.5
c4.4-34.4,5.9-68.6,6.5-102.9c0.1-5.6,2.4-12.7-0.9-16.5c-3.9-4.3-11.3-1.4-17.2-1.6c-7.6-0.2-7.7,0.1-7.7,7.6
c0,24.7,1.3,49.3,2.8,73.9c0.6,10.3,2.2,20.5,2.9,30.8c0.3,3.6,1.7,5.5,4.9,6.5C1782.2,1116,1784.4,1117.7,1788.4,1117.5z
M1481.1,1078.1c0-2.7,0.1-4.4,0-6c-0.3-7.8-0.3-8.1-8.2-8.7c-22.9-1.7-45.8-3.3-68.7-4.7c-19.9-1.2-39.9-0.9-59.9-1
c-4.6,0-5.5,2-5.4,6c0.1,3.7,0.3,6.7,5.2,6.2c1.3-0.1,2.7,0,4,0c34.3-0.3,68.4,1.9,102.5,5.6
C1460.6,1076.6,1470.4,1078.5,1481.1,1078.1z M1480,903c-15.3,1.6-30.8,3.6-46.3,4.8c-29.5,2.4-59.1,4.7-88.8,3.9
c-5.1-0.1-6.1,2-6,6.5c0.1,4.1,1.1,6.3,5.6,5.8c1.3-0.2,2.7,0,4,0c42.3,0.3,84.5-2.6,126.7-5.9c4.2-0.3,6.2-1.6,5.8-5.9
C1480.7,909.3,1482,906.3,1480,903z M1192.2,1078.2c3.1-0.2,5.4-0.3,7.6-0.5c37-4.6,74-7.6,111.3-7.8c6,0,12-0.1,18,0
c3.7,0.1,5.3-1.1,5.4-5.1c0.1-6.5,0-6.9-7.3-7c-40.6-0.8-81.1,1.7-121.6,4.6C1193.1,1063.3,1193.2,1063.6,1192.2,1078.2z
M1192.6,903.2c0.2,14.4,0.2,14.4,12.8,16c3,0.4,6,0.6,8.9,0.8c19.3,1.1,38.5,2,57.8,3.2c18.6,1.2,37.2,0.8,55.9,0.8
c6.4,0,6.3-0.5,6.6-6.8c0.2-5.8-3.3-5.5-7.1-5.4c-28.3,0.8-56.5-1.5-84.7-3.6C1226.3,907,1209.8,905.4,1192.6,903.2z M653.6,694.6
c2,12.3,3.8,22.4,5.1,32.6c0.5,3.7,1.7,6.1,4.8,8c14.2,8.8,27.5,18.8,42.2,26.6c2.7,1.4,6.2,6.3,8.9,2.4c2.7-3.7-2.6-6.2-5-8.5
c-19-17.3-36.7-35.8-51.8-56.7C657,698.1,656,697.3,653.6,694.6z M654,1284.6c0.8,0,1.3,0.1,1.5,0c1.4-1.4,2.9-2.7,4-4.3
c14.6-20.8,32.8-38.3,51.2-55.5c1.2-1.1,2.6-2.1,3.6-3.4c1.4-1.7,1.5-3.6-0.2-5.7c-2.9,0.3-5.3,2.1-7.8,3.6
c-10.2,6.3-19.8,13.5-30.5,18.8c-12.9,6.4-19.3,15.7-19.2,30.2C656.6,1273.7,654.9,1279.1,654,1284.6z M2019.1,698.8
c-0.5-0.3-1-0.6-1.5-0.9c-1,1.2-2,2.4-3,3.6c-6.8,9-13,18.4-21.4,26.1c-11,10.1-20.5,21.7-32.4,30.8c-1.9,1.5-3.6,3.2-2.1,5.9
c2,3.5,4.1,0.5,5.7-0.4c15.4-9.7,30.8-19.6,46.1-29.5c2.3-1.5,4-3.1,4.4-6.2C2016.2,718.5,2017.7,708.6,2019.1,698.8z
M2017.1,1283.9c0.7-0.2,1.4-0.5,2.1-0.7c-1.5-10.1-3-20.2-4.4-30.4c-0.4-2.6-1.9-4.1-3.9-5.4c-15.6-10-31.3-20.1-46.9-30
c-1.5-1-3.3-2.8-5-0.5c-1.7,2.4-0.3,4.2,1.7,5.9c4,3.4,8.1,6.9,11.8,10.6C1988.4,1249.3,2005.3,1264.3,2017.1,1283.9z M1044.6,1034
c0.2,0,0.3,0,0.5,0c0-11-0.1-21.9,0.1-32.9c0.1-4.2-2-4.8-5.5-4.9c-4.5-0.1-4.6,2.6-4.6,5.9c0.1,22.3,1.2,44.5,2.4,66.8
c0.1,1,0.2,2,0.4,3c0.2,1.2,1.2,1.8,2.1,1.7c0.8-0.1,1.9-1,2.2-1.7c0.5-1.5,0.7-3.2,0.8-4.9C1043.5,1055.9,1044,1045,1044.6,1034z
M1636.4,1033.8c0.7,0,1.4,0,2.1,0c0-10.6,0-21.2,0-31.8c0-2.7,0.2-5.3-3.6-5.8c-4.7-0.6-6,0.4-6,6.1c-0.1,21.9,0.2,43.7,2.1,65.6
c0.1,1.3,0.1,2.7,0.6,3.9c0.3,0.7,1.5,1.6,2.2,1.5c0.7,0,1.7-1.1,1.9-1.8c0.5-1.9,0.7-3.9,0.7-5.9
C1636.4,1055,1636.4,1044.4,1636.4,1033.8z M1638.5,948.1c-0.2,0-0.5,0-0.7,0c-0.5-11.6-1-23.3-1.6-34.9c-0.1-1.9,0.2-4.8-2.3-4.9
c-3.3-0.1-2.8,3-3,5.2c-0.3,3-0.4,6-0.7,9c-1.5,19-1.4,37.9-1.4,56.9c0,5.3,2,6.8,6.4,5.8c3.8-0.8,3.3-3.6,3.3-6.2
C1638.5,968.8,1638.5,958.4,1638.5,948.1z M1045.1,948c-0.1,0-0.3,0-0.4,0c-0.6-11.9-1.2-23.9-1.9-35.8c-0.1-1.9-0.4-4.3-3.2-3.9
c-2.2,0.3-1.9,2.4-2,4c-1.3,22.5-2.3,45.1-2.5,67.7c0,3.7,1.2,5.4,5.1,5.4c4.1,0.1,5-1.9,5-5.5C1045,969.3,1045.1,958.7,1045.1,948
z M741.9,740.1c-4.9,5.5-6.3,11-8.3,16.2c-0.8,2-0.8,4,1.2,5.3c6.2,4.1,11.6,9.4,19.7,13C751.6,762.1,748.8,750.9,741.9,740.1z
M1931.7,1241.3c11.7-16.6,11.5-18.2-3-28.3c-2.7-1.9-4.9-4.5-9.1-4.9C1921.9,1219.6,1924.7,1230.7,1931.7,1241.3z M1931.6,740
c-6.6,10.9-9.8,22-12.1,33.9c4-0.9,6.1-3.1,8.5-4.8C1943.5,758.3,1943.6,757.2,1931.6,740z M742.1,1241.3
c6.6-11.1,9.8-21.9,12.2-34.3c-8,3.6-13.2,8.8-19.2,12.6c-2,1.3-2.3,3.1-1.6,5.1C735.6,1230.2,736.7,1236.2,742.1,1241.3z
M597.6,1464.4c16.9-6.5,17.3-8.9,14-24.4C605.8,1447.2,601.7,1455,597.6,1464.4z M2061.5,1442.5c-5.1,14.3,5.6,17.3,12.5,22.8
C2072.1,1457,2066.2,1451,2061.5,1442.5z M1781.6,836.9c-1,5.2-3.6,8.5-1.8,12.6C1785.7,846.2,1785.7,846.2,1781.6,836.9z
M895.4,849.3c-1.5-4.5-1.6-7.9-3.8-11.1C887.9,846.3,887.9,846.3,895.4,849.3z M630.1,1284.6c-0.4-5.3,4.1-10.2,1.8-17
C627.4,1273.6,629.9,1279.2,630.1,1284.6z M629.9,697.1c0.2,5.4-3,11.4,2.4,16.6C633.8,707.4,630.4,702.5,629.9,697.1z
M2044.5,1284.5c-1-5.2,1.5-10.3-2.8-15C2040.3,1275.4,2042.4,1279.6,2044.5,1284.5z M2041.9,712.1c4.1-4.8,1.6-10,2.2-14.6
C2043,702.1,2039.8,706.2,2041.9,712.1z M892.2,1143.9c0.4-4.2,3.5-7.3,1.6-11.5C887.6,1136,887.6,1136,892.2,1143.9z
M1779.6,1132.2c-1,2.9-0.4,4.8,0.4,6.5c0.7,1.5,0.4,3.6,2.3,4.6C1785.7,1135.3,1785.7,1135.3,1779.6,1132.2z M729.5,774
c1.5,1.6,2.6,4,5.8,3.2C733.9,774.7,732.2,773.5,729.5,774z M737.9,778.4c-0.2,0.2-0.5,0.4-0.7,0.6c0.2,0.2,0.4,0.6,0.6,0.6
c0.2,0,0.5-0.3,0.7-0.5C738.4,778.8,738.1,778.6,737.9,778.4z"/>
<path class="st1" d="M534.9,1353.5c4.5,11.6,7.4,22.2,12.2,32c4.7,9.6,9.8,18.8,17.1,26.8c10.6,11.6,28.8,15.5,42.7,2.9
c1.4-1.3,3.1-2.4,5.5-4.2c1.4,7.2,1.7,13.2-2.5,19.1c-8.4,11.7-14.8,24.4-20.8,37.4c-1.7,3.7-3.7,6.7-7.3,9.1
c-52.7,33.7-105.3,67.6-158,101.4c-1.1,0.7-2.1,1.7-4.1,1.1c0-195.1,0-390.4,0-585.6c2.1-1.2,4.1-0.9,6-0.8
c21.6,0.5,43.3,1.1,64.9,1.3c5.3,0,6.7,1.8,6.7,6.9c0.5,40,1.4,79.9,3.9,119.8c2.4,38.2,5.6,76.4,10.6,114.3
c4.6,34.7,10.1,69.1,19.2,103c1.1,4,0.6,7.1-2.3,10c-0.9,0.9-1.7,2.1-2.6,3.1c-22.1,25.1-40.6,52.7-57.5,81.5
c-11.6,19.7-19.9,41-28.9,61.9c14.9-23.9,25.7-50.1,41.8-73.5C497.6,1397.6,514.9,1375.3,534.9,1353.5z"/>
<path class="st1" d="M534.6,627.2c-40-41.2-69.6-89-94.2-140.1c0.7,6.2,4,11.2,6.2,16.6c12.7,31.7,30.5,60.6,49.6,88.7
c9.3,13.7,20.4,26.3,31,39.1c3.8,4.5,5,8.7,3.4,14.4c-9.6,35.4-15,71.6-19.6,107.9c-9.4,74.5-12.5,149.3-13.7,224.3c0,1-0.1,2,0,3
c0.6,4.7-1.2,6.5-6.1,6.4c-8.3-0.2-16.7,0.2-25,0.4c-13.7,0.3-27.3,0.4-41,1c-4.8,0.2-6.8-1.2-6.2-6.1c0.3-2,0-4,0-6
c0-187.9,0-375.8,0-563.7c0-3.5,0-7.1,0-10.7c4.2-0.1,6.3,2.3,8.7,3.9c50.4,32.4,100.8,64.9,151.3,97.1c5,3.2,8.4,7.1,10.8,12.4
c5.1,11.2,10.4,22.4,17.8,32.3c4.9,6.6,6.7,13.5,5.1,21.9c-3.6-0.1-5.6-3.1-7.9-5c-12-9.8-28.1-7.7-38.9,2.8
c-12.6,12.2-19,27.9-25.2,43.7C538.8,616.5,538.2,622.2,534.6,627.2z"/>
<path class="st1" d="M2060.9,1411.1c7.4,4.3,12.3,10.3,20.6,11.4c9.8,1.4,17.7-1.4,24.8-7.5c8.5-7.3,13.9-16.9,18.8-26.8
c5.1-10.4,8.6-21.5,12.7-32.5c24.7,24.7,73.3,78.9,107.2,169.7c-1.1-8.8-3.5-17.2-6.8-25.4c-13.2-33.9-29.3-66.4-49.3-96.8
c-12.8-19.5-27.4-37.7-42.4-55.5c-3.4-4-4.8-7.1-3.5-12.3c12.8-48.8,19.1-98.6,24-148.7c5.9-59.7,8.4-119.6,8.9-179.7
c0-1.3,0-2.7,0.1-4c0.2-7,0.2-7.1,7-7.4c20-0.7,40-1.3,60-1.9c2.3-0.1,4.7,0,7-0.1c3.5-0.2,4.8,1.4,4.6,4.8c-0.1,2,0,4,0,6
c0,188.3,0,376.6,0,565c0,3.2,0,6.4,0,9.8c-4.5-0.2-6.9-2.8-9.5-4.5c-47.7-30.6-95.2-61.4-143-91.7c-9-5.7-16-11.6-19.9-22.4
c-3.7-10.1-11.2-18.9-17.1-28.2C2060.8,1426.2,2058.6,1419.6,2060.9,1411.1z"/>
<path class="st1" d="M2244.8,454.3c-23,66.2-59.6,121.8-106.6,172.4c-4.5-11.6-7.7-22.6-12.8-32.9c-4-8.1-8.4-15.8-14.4-22.7
c-12.6-14.4-29.9-17.7-45.6-3.8c-1.1,1-2,2.7-4.6,2.6c-1.4-5.5-1.1-10.5,2-15.7c8.8-14.9,18.9-29,25.8-45c1-2.3,2.7-3.7,4.7-4.9
c52.7-33.9,105.3-67.7,158-101.5c0.5-0.3,1.3-0.2,2.1-0.4c2.3,2.1,1.2,4.9,1.2,7.3c0.1,189.6,0.1,379.3,0.1,568.9
c0,1.7-0.1,3.3,0,5c0.2,3.6-1.7,4.6-4.9,4.5c-5-0.2-10-0.3-15-0.5c-17-0.5-34-1-51-1.4c-7.4-0.2-7.5-0.2-7.6-7.8
c-0.5-35.3-1.1-70.6-3.1-105.9c-2.6-45.9-6.4-91.7-12.8-137.3c-4.2-30.3-9.4-60.5-17.3-90.2c-1.1-4-0.2-6.6,2.5-9.8
c27.4-31.6,51.5-65.6,70.4-102.8c11.3-22.2,21.7-44.9,28.2-69.2C2244.6,460.9,2245.4,458.8,2244.8,454.3z"/>
<path class="st1" d="M553.7,662.8c3.8,0.2,5.2,2.8,6.9,4.6c14.5,15.2,30.3,29,46.9,41.9c4.1,3.2,5.8,6.4,5.8,11.7
c-0.2,85.6-0.1,171.2-0.1,256.8c0,7.5-0.1,7.7-7.6,7.8c-25,0.4-50,0.6-74.9,1.1c-4.3,0.1-5.9-1.2-5.7-5.6c0.4-13,0.2-26,0.5-39
c1.5-60.3,4.7-120.5,11.8-180.4c3.7-31.7,8.3-63.3,15.4-94.5C553,665.6,553.4,664.1,553.7,662.8z"/>
<path class="st1" d="M553.6,1319.7c-22.4-107.4-27.4-215.3-28.7-323.3c3.3-2.5,6.4-1.5,9.3-1.5c23.3,0.3,46.7,0.7,70,1.1
c9.5,0.1,9.2-1.2,9.2,9.2c0,72.3,0,144.7,0,217c0,13.3-0.1,26.7,0.1,40c0,3.6-0.7,6.2-3.8,8.6c-18.7,14.5-36.3,30.3-52.7,47.4
C556.3,1318.5,555.6,1318.7,553.6,1319.7z"/>
<path class="st1" d="M2119.8,1319.2c-6.7-6.5-12.1-12.3-18-17.5c-11.8-10.2-23.8-20.2-36-30.1c-4.2-3.4-5.9-7.1-5.9-12.7
c0.2-84.6,0.1-169.1,0.1-253.7c0-8.4,0.1-8.5,8-8.5c24.3-0.2,48.6-0.4,72.9-0.5c7.5,0,7.7,0,7.6,7.7c-0.3,45.6-1.9,91.2-5.1,136.7
c-2.7,39.9-6.7,79.6-12.5,119.1C2128.1,1279,2124.6,1298.3,2119.8,1319.2z"/>
<path class="st1" d="M2120.4,664.4c12.3,57.5,18.7,115.7,22.8,174.2c3.1,44.9,4.7,89.8,5.2,134.8c0,2.3-0.1,4.7,0.1,7
c0.3,4-1.7,5.2-5.4,5.1c-6.7-0.2-13.3-0.3-20-0.3c-18.3-0.1-36.7-0.2-55-0.4c-8-0.1-8.2-0.1-8.2-8.2c0-52.3,0-104.6,0-157
c0-32,0.2-64-0.2-96c-0.1-6.3,2-10.3,6.8-14.3c17.1-14.3,35.1-27.7,50.2-44.2C2117.5,664.3,2118.3,663.8,2120.4,664.4z"/>
<path class="st1" d="M878.7,1138.5c2,4.9,3.2,9.1,5.3,12.8c4,7.2,10.5,7.1,14.6,0c3-5.3,4.3-11.2,5.2-17.1c0.8-4.7,3.2-7.1,7.8-8.9
c37.6-14.8,76.3-25.6,115.5-34.7c5-1.1,6.6,0.4,7.3,4.9c0.6,3.5,0.7,8.4,5.3,8.5c5.3,0.1,5.5-5.2,5.8-9c0.5-7.1,3.9-10,10.9-11.2
c40-7,79.8-14.9,120.4-18c2.4-0.2,5.9-2.1,6.9,1.5c1,3.7,1.8,7.8,0.4,11.7c-0.7,1.8-2.9,1.2-4.4,1.5c-9.5,1.6-19.1,2.8-28.6,4.6
c-39.6,7.4-78.6,17-117,29.2c-59.9,19-116.5,45.2-170.8,76.4c-41.6,24-80.2,52.3-117.2,83.1c-39.4,32.8-74.3,69.6-105.4,110.2
c-3,3.9-6.2,7.7-9.4,11.6c-0.1,0.2-0.6,0.2-2.1,0.5c0.2-7-1.1-13.6,1.5-20.3c7.6-19.2,12.9-39.2,16.7-59.5
c1.8-9.6,6.2-17.5,12-24.8c17.2-21.7,36.5-41.5,58.8-58.9c3.4,2.2,3.5,6.6,5.4,9.8c4,7,7.8,14.4,17.4,14.4
c9.6,0,13.8-7.2,17.3-14.4c6.5-13.3,9.4-27.7,12.2-42c0.8-3.9,2.2-6.5,5.7-8.6C808.9,1171.4,842.7,1153.5,878.7,1138.5z"/>
<path class="st1" d="M1032.8,891.9c-42-9-82.2-20.4-121.3-35.7c-4.8-1.9-6.9-4.4-7.7-9c-0.9-5.6-2.3-11.1-4.8-16.2
c-1.5-3.2-3.7-5.9-7.3-6.1c-4.1-0.2-6.8,2.5-8.3,6.1c-1.6,3.6-2.8,7.4-4.7,12.2c-11.4-5.2-22.6-10-33.5-15.2
c-23.1-11.1-45.4-23.7-67.3-37.1c-4.3-2.6-6.6-5.5-7.5-10.7c-2.1-11.4-4.6-22.8-8.9-33.7c-2.1-5.3-4.2-10.5-7.8-15
c-7.6-9.4-17.5-9.4-25.3,0.1c-4.2,5.2-6,11.7-9.8,18.2c-11.1-9.9-22.3-18.8-32.2-28.9c-10.2-10.4-19.5-21.8-28.7-33.3
c-5.6-7-8.9-15.3-10.6-24.4c-3.7-19.3-8.6-38.3-16-56.5c-2.9-7.1-1.4-14.1-0.5-21.8c3.9,4.8,7.9,9.4,11.6,14.3
c23.7,32,51,60.7,80.1,87.7c19,17.6,39.4,33.9,59.9,49.7c45,34.7,93.8,63.3,144.7,87.9c39.2,18.9,79.8,34.7,121.6,47.1
c41.2,12.2,82.8,22.3,125.4,28.2c1,0.1,2,0.2,3,0.4c7.3,1.2,10.5,7.7,7,14.2c-1.3,2.3-3.4,1.7-5.3,1.6c-8-0.7-15.9-1.5-23.8-2.4
c-32.8-3.7-65.1-10.2-97.5-15.7c-7.1-1.2-11.5-3.9-11.7-11.6c0-1.3-0.4-2.6-0.8-3.9c-0.8-2.6-1.9-5.3-5.3-4.9
c-2.7,0.3-3.7,2.6-4.4,4.9C1034.4,885.3,1033.7,888.2,1032.8,891.9z"/>
<path class="st1" d="M1953.8,1233.7c3.8,1,5.8,3.4,8,5.3c21,19.1,40.7,39.4,57.6,62.2c3,4.1,4.9,8.4,6,13.5
c4.5,20.5,9.3,40.9,17.1,60.4c1.3,3.1,2.4,6.3,1.8,9.7c-0.6,3.6,2.4,8.9-2.1,10.4c-2.5,0.8-5.4-3.9-7.3-6.7
c-35.7-51.5-81-93.6-129.4-132.6c-40.8-32.9-84.7-61.2-131-85.6c-39.5-20.8-80.6-37.9-123-52c-45.9-15.2-92.6-26.8-140.3-34.3
c-6.2-1-12.4-2.7-18.7-3.1c-6.5-0.4-5.5-4.5-5.2-8.4c0.4-5.4,2.1-7.1,7.5-6.6c8.3,0.7,16.5,1.9,24.8,2.7
c33.2,3.4,65.8,10.1,98.6,15.7c6.3,1.1,9.2,3.4,9.6,10c0.2,3.6,0.3,9.3,5.3,9.6c6,0.4,5.5-5.6,6.3-9.5c0.7-3.9,2.4-5,6.2-4
c39.4,9.7,78.9,19.2,116.7,34.3c4.5,1.8,6.8,4.2,7.5,9.1c0.9,5.9,2.2,11.8,5.1,17.2c1.7,3.1,3.8,5.6,7.6,5.6c3.8,0,6-2.5,7.6-5.6
c0.9-1.8,1.7-3.6,2.3-5.5c2.8-7.7,2.9-7.9,9.9-4.7c32.6,15.3,64.6,31.4,95,50.8c3.4,2.2,5.1,4.7,5.8,8.6
c2.1,12.1,4.9,24.1,9.4,35.7c1.9,5,4.1,9.8,7.5,14c7.5,9.4,17.7,9.4,25.3,0C1948.9,1245.3,1950.9,1239.5,1953.8,1233.7z"/>
<path class="st1" d="M2043.6,585.1c0.9,7.5,1.7,14.1-0.8,20.4c-8,20.2-13.1,41.2-17.6,62.4c-0.9,4.4-2.6,8.3-5.2,11.7
c-17.8,23.8-38.4,45.2-60.6,64.9c-1.5,1.3-3.2,2.3-5.1,3.8c-2.9-5.3-5.1-10.4-8.2-14.9c-8-11.8-19.3-11.8-27.5-0.1
c-7.4,10.6-11.4,23-12.8,35.4c-1.7,14.6-9.4,22.2-21.3,29.3c-25.5,15.2-52.1,28.1-78.7,41.2c-2.4,1.2-5,2-7.3,3.1
c-2.7,1.3-4.3,0.7-5.2-2.2c-1-3.2-1.9-6.4-3.4-9.3c-4.2-7.7-10.8-7.7-15.1-0.1c-2.5,4.4-3.9,9.3-4.5,14.2c-0.8,7-4.2,10.7-11,13.3
c-27.5,10.2-55.4,18.9-83.9,25.7c-10,2.4-20.1,4.8-30,7.5c-3.8,1-5.4,0-6.1-3.9c-0.7-4-0.2-9.9-6.2-9.6c-5.1,0.2-5.3,6-5.4,9.5
c-0.1,7.5-4.3,9-10.5,10.2c-39.6,7.4-79.4,13.9-119.5,18.1c-2.8,0.3-6.7,2.3-8.1-1.3c-1.4-3.6-1.7-7.9-0.3-11.7
c0.9-2.4,3.9-2,6.2-2.4c13.8-2.4,27.6-4.4,41.3-7c42.1-8.2,83.5-19.3,124.2-33.1c57.6-19.6,112.2-45.7,163.9-77.3
c38.1-23.2,73.8-50.1,107.6-79.4c37.9-32.8,72.6-68.6,102.5-109C2037.5,591.3,2039.4,587.8,2043.6,585.1z"/>
<path class="st1" d="M777.2,1163.8c-0.9-7.9,1.2-15.4,2-23c4.4-45.1,6.3-90.3,6.6-135.6c0.1-8.6,0.2-8.7,8.8-8.7
c22-0.1,44-0.1,65.9,0c8.3,0,8.4,0.1,8.5,7.9c0.2,37.7,2,75.2,6.5,112.6c0.5,3.8,0.3,6-3.9,7.5c-31.7,11.3-62.2,25.3-92.6,39.6
C778.8,1164.3,778.4,1164.1,777.2,1163.8z"/>
<path class="st1" d="M776.8,817.8c3.1-0.6,4.7,0.6,6.5,1.4c28.2,13.5,56.6,26.5,86.2,36.8c5.4,1.9,6.7,4.3,6,10.1
c-4.5,36.7-6.3,73.6-6.4,110.6c0,8.1,0.7,8.3-8.1,8.3c-22.6-0.1-45.3,0-67.9-0.1c-7,0-7.1-0.2-7.2-7.3c-0.4-28.6-0.9-57.3-2.7-85.9
c-1.3-21.3-2.9-42.5-5.5-63.7C777.2,824.9,777.1,821.6,776.8,817.8z"/>
<path class="st1" d="M1896.4,1163.9c-3.3,0.5-5.7-1.5-8.4-2.8c-27.2-13.3-54.8-25.7-83.3-36.1c-7.4-2.7-7.2-2.9-6.3-10.6
c4.4-35.8,5.3-71.8,6.4-107.7c0.1-1.7,0.1-3.3,0-5c-0.3-3.8,1.3-5.4,5.2-5.3c6.3,0.2,12.7,0.1,19,0.1c16.7,0,33.3,0,50,0.1
c8.4,0,8.4,0.1,8.6,8.8c0.8,30.3,1.2,60.6,3.3,90.9c1.4,19.9,2.9,39.9,5.3,59.7C1896.5,1158.6,1897.4,1161.2,1896.4,1163.9z"/>
<path class="st1" d="M1896.8,817.7c-1.3,13.9-2.7,27.7-3.9,41.6c-3.5,38.5-4.5,77-5.2,115.6c0,1.7-0.2,3.3-0.1,5
c0.4,4-1.5,5.5-5.2,5.1c-1-0.1-2,0-3,0c-22.6,0-45.3-0.1-67.9,0.2c-5,0-6.8-1.2-6.8-6.6c-0.4-28.3-1.2-56.5-3.6-84.7
c-0.8-8.9-1.6-17.9-2.7-26.8c-0.9-7.6-1.1-7.8,6.5-10.6c23-8.7,45.8-17.8,67.9-28.7c6-2.9,12-5.7,18-8.6
C1892.5,818.3,1894.3,817.4,1896.8,817.7z"/>
<path class="st1" d="M908.3,1109.6c-1-4.4,0.4-8.6,0.8-12.8c2.5-30.9,4-61.8,4-92.9c0-7.4,0.2-7.5,7.9-7.5
c33.3-0.1,66.7-0.2,100-0.2c7.8,0,8.1,0.1,8,7.4c-0.3,22,1,44,2.3,65.9c0.3,4.5-1,6.5-5.5,7.4c-38.7,8.5-76.4,20.9-114.2,32.5
C910.7,1109.8,909.7,1109.6,908.3,1109.6z"/>
<path class="st1" d="M1765.3,1110.4c-9.6-3-18.1-5.6-26.6-8.4c-29.1-9.6-58.7-17.4-88.5-24.5c-8.9-2.1-8.1-1.4-7.6-10.4
c1.2-21,1.6-42,2.3-62.9c0.2-7.7,0.2-7.9,7.5-7.9c33.3,0,66.7,0.1,100,0.2c7.8,0,8.2,0.3,8.2,7.7c0.2,34,1.7,68,4.8,101.8
C1765.4,1106.9,1765.3,1107.9,1765.3,1110.4z"/>
<path class="st1" d="M908.8,872.2c2.8-1.1,5,0.8,7.5,1.5c36.1,10.3,71.6,22.9,108.5,30.5c5.5,1.1,6.7,3.6,6.4,8.8
c-1.3,21.9-2.6,43.9-2.1,65.9c0.1,4.7-1.2,6.5-6.1,6.4c-34.7-0.2-69.3-0.2-104-0.1c-4.1,0-6.1-1-6-5.6c0.4-35.6-2-71.2-4.8-106.7
C908.2,872.8,908.6,872.5,908.8,872.2z"/>
<path class="st1" d="M1765.6,872.4c-0.9,10.7-1.8,20.9-2.5,31.2c-1.7,24.6-2.4,49.2-2.6,73.8c-0.1,7.7-0.2,7.8-7.7,7.8
c-33.6,0.1-67.2,0.1-100.8,0.1c-6.9,0-6.9-0.2-7.1-7.1c-0.8-21.9-1.3-43.9-2.6-65.8c-0.3-5.4,1.8-7,6.2-8
c14.5-3.5,29-7.1,43.5-10.8c22.5-5.7,44.5-13.5,66.7-20.3C1760.5,872.7,1762.3,871.8,1765.6,872.4z"/>
<path class="st1" d="M725.8,1192.5c-5.9-34.3-8.8-67.4-11-100.6c-1.8-27.6-2.2-55.2-3-82.8c-0.1-2.3,0.1-4.7-0.1-7
c-0.4-4.4,1.6-5.8,5.9-5.7c9,0.2,18,0,27,0c6.7,0,13.3-0.1,20,0c7.2,0.1,7.4,0.2,7.3,7.1c-0.5,23.6-1,47.2-2,70.8
c-1.2,27.2-3.3,54.4-6.7,81.5c-0.2,1.3-0.3,2.6-0.4,4c-0.4,9.2-3.4,16-13,19.7C741.7,1182.7,734.5,1187.7,725.8,1192.5z"/>
<path class="st1" d="M1946.2,1191.7c-10.1-5.6-20.4-11.5-30.8-17.2c-3.3-1.8-3.1-5-3.5-7.8c-1.4-10.2-2.4-20.5-3.5-30.7
c-4.7-43.8-6-87.7-6.7-131.6c-0.1-7.7,0.1-7.8,7.7-7.8c15.3-0.1,30.6,0.1,46-0.1c4.6-0.1,6.5,1.1,6.4,6.1
c-0.5,34-1.2,67.9-3.7,101.8c-2,26.9-4.5,53.7-8.9,80.4C1948.7,1187,1949.1,1189.5,1946.2,1191.7z"/>
<path class="st1" d="M1947.7,789.1c5.8,32.7,8.6,63.8,10.8,95c2.1,30.6,2.8,61.2,3.1,91.9c0.1,9.6,0.3,8.9-8.3,8.9
c-15,0-30-0.1-45,0.2c-5.3,0.1-6.9-1.8-6.7-7c0.9-25.3,1.3-50.6,2.5-75.9c1.3-27.3,3.9-54.4,6.9-81.6c0.9-8.3,3.3-14.2,11.5-17.8
C1930.8,799.2,1938.6,794.1,1947.7,789.1z"/>
<path class="st1" d="M726,789.2c11.3,6.2,21.7,11.9,32.2,17.5c2.9,1.6,2.8,4.2,3.2,6.8c2.7,15.8,4.1,31.7,5.5,47.6
c3.3,37.2,4.5,74.5,5,111.8c0,2.3,0,4.7,0.1,7c0.1,3.1-0.8,5-4.4,5c-17-0.1-34-0.1-51-0.1c-3.3,0-4.8-1.5-4.7-4.8
c0.2-5.7,0-11.3,0.2-17c1-42.6,2.7-85.2,7.3-127.7C720.9,820.5,722.6,805.7,726,789.2z"/>
<path class="st1" d="M1625.1,910.5c-0.6,9.4-1.4,18-1.7,26.5c-0.4,13-0.4,26-0.6,38.9c-0.1,9.3-0.2,9.4-9.2,9.4
c-30.6,0.1-61.3,0.1-91.9,0.1c-8.3,0-16.7-0.1-25,0.1c-4.2,0.1-5.9-1.3-5.9-5.7c0-15-0.1-30-0.7-44.9c-0.2-4.8,2.5-5.5,6-6
c10.9-1.3,21.8-3.6,32.7-3.7c23.9-0.2,46.7-6.5,70-10C1607.1,913.9,1615.2,912.2,1625.1,910.5z"/>
<path class="st1" d="M1625.2,1071.1c-23.5-4.2-45.7-8.1-67.9-12.1c-7.9-1.4-15.8-2.4-23.8-2.5c-12.7-0.1-25.1-2.6-37.7-4
c-4-0.4-5.8-1.6-5.6-6.1c0.5-14.6,0.6-29.3,0.6-43.9c0-4.5,1-6.5,6.1-6.5c39.9,0.2,79.9,0.2,119.8,0.1c5,0,6.1,2.1,6.1,6.6
C1622.9,1025,1623,1047.3,1625.2,1071.1z"/>
<path class="st1" d="M1049.3,1070.4c-1-5.9-0.1-11.8,0.3-17.7c0.9-16.3,1.4-32.6,1.4-48.9c0-7.3,0.2-7.5,6.8-7.5
c39.3-0.1,78.5-0.1,117.8-0.1c7,0,7.1,0.2,7.2,7.2c0.2,14,0,27.9,0.5,41.9c0.2,5.2-1.5,6.6-6.6,7c-16.9,1.3-33.8,2.7-50.6,5.2
c-23.3,3.4-46.7,6.8-69.7,12.3C1054.2,1070.3,1051.9,1071.4,1049.3,1070.4z"/>
<path class="st1" d="M1049.8,910.2c22.1,5.7,44.8,8.7,67.4,12.5c16.4,2.8,33.1,3.3,49.6,5.2c16.4,1.9,16.4,1.5,16.3,17.4
c-0.1,11-0.3,22-0.2,32.9c0.1,4.9-1,7.2-6.6,7.2c-39.9-0.2-79.9-0.2-119.8-0.1c-3.4,0-5.7-0.6-5.6-4.8c0.6-22.6-1.3-45.2-1.9-67.8
C1049,912,1049.4,911.4,1049.8,910.2z"/>
<path class="st1" d="M1260.5,985.4c-21.6,0-43.2,0-64.9,0c-8,0-8.2-0.1-8.1-8.4c0.2-13.6,0.6-27.3,0.8-40.9c0.1-4.2,1.4-5.6,6-5.1
c41.7,4.1,83.5,6.2,125.5,6.2c2.3,0,4.7,0,7,0c7.5,0.2,7.5,0.2,7.6,7.9c0.1,10.3,0.1,20.6,0.1,30.9c0,9.3,0,9.4-9.1,9.4
C1303.8,985.4,1282.2,985.4,1260.5,985.4z"/>
<path class="st1" d="M1485,931.5c0.3,16.2,0.5,32.7,0.8,49.3c0.1,3.4-1.7,4.8-4.9,4.6c-2-0.1-4,0-6,0c-41.6,0-83.2,0-124.9,0
c-2,0-4-0.1-6,0c-4,0.3-5.1-1.8-5-5.5c0.1-11.7,0-23.3,0.1-35c0.1-7.6,0.1-7.8,7.6-7.8c17.3,0.2,34.6-0.6,51.9-1.1
c26.3-0.7,52.5-3.1,78.7-5.1C1479.6,930.9,1481.8,930,1485,931.5z"/>
<path class="st1" d="M1485,1051.1c-35.4-3-69.9-5.3-104.5-6.4c-11.6-0.4-23.3-0.5-34.9-0.4c-4.4,0-6.7-0.8-6.5-6
c0.4-12,0.3-24,0-36c-0.1-4.8,1.6-6.3,6.3-6.2c45,0.1,89.9,0.1,134.9,0c3.5,0,5.8,0.7,5.7,4.8
C1485.5,1017.5,1485.3,1034.1,1485,1051.1z"/>
<path class="st1" d="M1188.4,1050.6c-0.3-17.2-0.5-33.7-0.8-50.3c0-2.9,1.6-4.1,4.4-4.1c1.7,0,3.3,0,5,0c42.6,0,85.1,0,127.7,0
c1.7,0,3.3,0.1,5,0c3.3-0.2,4.9,1.2,4.9,4.6c-0.1,13-0.1,25.9,0,38.9c0,3.6-1.7,4.7-4.9,4.5c-0.7,0-1.3,0-2,0
C1281.6,1043.7,1235.9,1047.2,1188.4,1050.6z"/>
<path class="st1" d="M2010.7,1230.1c-5.1-1.9-9-5.1-13.1-7.7c-9.9-6.1-19.6-12.6-29.8-18.3c-4.8-2.6-5.5-5.6-4.5-10.4
c3.9-20,6.2-40.1,8.2-60.4c4-41.2,5.8-82.4,6.1-123.8c0-3,0.1-6,0-9c-0.1-3,1.1-3.8,4.1-4.1c16.5-1.5,16.7-1.4,16.8,14.7
c0.4,66.4,3.9,132.5,10.7,198.6C2009.9,1216.3,2011.6,1222.7,2010.7,1230.1z"/>
<path class="st1" d="M2011.2,752.5c-1.6,16.9-3.2,33.8-4.8,50.6c-3,30.8-4.6,61.7-6.1,92.6c-1.3,27.3-1.5,54.5-1.8,81.8
c-0.1,7.1-0.2,7.3-7.1,7.3c-16.3,0-13.6,1.3-13.7-12.8c-0.5-54.3-3.2-108.4-10.6-162.2c-1.1-7.9-2.5-15.8-3.9-23.6
c-0.7-3.6-0.1-6,3.5-8.2c13.4-8,26.6-16.4,39.8-24.7C2007.6,752.8,2008.5,751.7,2011.2,752.5z"/>
<path class="st1" d="M662.1,1231.5c1.9-20.4,3.7-38.6,5.4-56.8c3-31.9,4.7-63.8,6-95.8c1-24.6,1.6-49.3,1.6-74
c0-8.3,0.1-8.3,7.9-8.4c1.3,0,2.7,0,4,0c8.6,0.1,8.7,0.1,8.8,8.7c0.4,40,1.8,80,5.4,119.8c2,22.9,4.4,45.8,9,68.3
c1.1,5.5,0.2,8.5-5.1,11c-12.7,6-23.9,14.5-35.5,22.2C667.7,1227.9,665.8,1229.1,662.1,1231.5z"/>
<path class="st1" d="M662.9,752.1c2.6-0.9,3.9,1,5.4,2.1c12.4,8.4,24.6,17.1,38,23.9c3.3,1.7,5,3.6,4.2,7.7
c-11.3,61.8-13.7,124.4-14.8,187c-0.2,13.5,2.2,12.1-12.3,12.1c-8.4,0-8.5-0.1-8.5-8.9c0.1-52.7-2.6-105.3-6.4-157.8
c-1.5-20.6-4-41.1-6-61.7C662.5,755.1,661.9,753.4,662.9,752.1z"/>
<path class="st1" d="M2044.3,728.6c0,3.4,0,6.3,0,9.2c0,78.9,0,157.9,0,236.8c0,1.7-0.1,3.3,0,5c0.4,3.9-1.3,5.4-5.2,5.3
c-5.9-0.2-11.9,0-18.3,0c0-26.3,0.3-51.6,1.3-76.8c1.7-42.3,4.4-84.5,8.9-126.5c1.6-14.9,3.9-29.7,5.6-44.6
C2037.2,732.4,2038.9,729.7,2044.3,728.6z"/>
<path class="st1" d="M2044.3,1253c-5.6-1.2-7-4.1-7.7-8.6c-3.1-19.4-5.2-38.9-7-58.5c-5.6-59-8.7-118.2-8.7-177.6
c0-13.6-1.4-11.7,11.5-11.8c13.3,0,11.9-1.5,11.9,11.5c0,78.6,0,157.2,0,235.8C2044.3,1246.6,2044.3,1249.3,2044.3,1253z"/>
<path class="st1" d="M629.2,728.7c5.8,1.6,7.4,4.6,7.9,9c2.3,21.2,5.3,42.2,7.3,63.4c5.3,55.3,7.7,110.8,8.5,166.4
c0,3.3,0.1,6.7,0,10c-0.1,7.5-0.1,7.5-7.7,7.7c-16,0.5-16,0.5-16-15.3c0-76.5,0-153.1,0-229.6C629.2,736.7,629.2,733,629.2,728.7z"
/>
<path class="st1" d="M629.2,1253.2c0-6,0-10.6,0-15.3c0-75.7,0-151.3,0-227c0-2,0-4,0-6c-0.1-9.2-0.3-8.7,8.7-8.6
c15.1,0,15.2,0,15,15.6c-0.7,62.7-3.8,125.2-10.3,187.6c-1.5,14.9-3.9,29.7-5.5,44.6C636.6,1248.7,634.5,1251.1,629.2,1253.2z"/>
<path class="st1" d="M2060,697.5c0-15.9,0-29.5,0-43c0-3.7,0.3-7.3-0.1-11c-2-19,6.4-34.8,15.3-50.5c1-1.7,2.3-3.2,3.6-4.7
c5.4-6.2,8.1-6.2,13.4,0.4c6.3,7.8,10.3,17.1,13.7,26.4c3.2,8.7,5.9,17.6,8.7,26.5c1.3,4,1,7.1-2.4,10.7
C2096.7,668.2,2079,681.6,2060,697.5z"/>
<path class="st1" d="M613.2,1283.1c0,24.5,1.5,47.6-0.5,70.3c-1.4,15.4-9,29.8-19.7,41.7c-3.8,4.2-5.7,3.9-10-0.4
c-4.3-4.3-7.2-9.6-9.8-15c-6.7-13.8-11.2-28.4-15.3-43.2c-0.8-3-1.3-5.6,1.5-8.1C576.3,1313,592.4,1296.7,613.2,1283.1z"/>
<path class="st1" d="M2060,1284.4c19.3,15.5,36.8,29.3,52.5,45.2c2.6,2.6,3.7,5.1,2.6,8.8c-5.1,17.2-10,34.5-19.7,50
c-7.7,12.2-12.4,12.7-19.8,0.6c-9.5-15.4-16.9-31.4-15.8-50.5C2060.7,1321.2,2060,1303.9,2060,1284.4z"/>
<path class="st1" d="M611.9,698.2c-8.5-6.7-17.1-12.8-25-19.7c-9.1-7.8-17.5-16.4-26.3-24.5c-2.3-2.2-4-4.1-3-7.7
c5.3-18.2,10.3-36.5,20.4-52.8c7.8-12.7,12.8-13,20.4-0.3c8.9,14.9,16.3,30.2,15.1,48.6c-1,15.9-0.2,31.9-0.3,47.9
C613.2,692.2,614,694.9,611.9,698.2z"/>
<path class="st1" d="M1788.2,864c4.1,27,4.9,53.5,6.1,80.1c0.5,11.3,0.2,22.6,0.7,33.9c0.2,5.2-1.6,7.4-6.7,7.3
c-4.7-0.1-9.3-0.3-14,0c-4.6,0.3-5.5-2-5.4-6c0.2-26.3,1.3-52.5,3.2-78.8c0.5-7.6,1.2-15.2,2.1-22.8
C1775.6,867.2,1775.7,867.2,1788.2,864z"/>
<path class="st1" d="M885.3,1118.4c-5.1-39.1-6.6-77.6-6.6-116.3c0-4.2,1.3-5.9,5.7-5.8c25,0.7,20.1-4.1,19.8,19.4
c-0.4,29.6-1.5,59.3-5,88.8C897.9,1115.4,898.1,1115.4,885.3,1118.4z"/>
<path class="st1" d="M885.7,863.5c12.4,2.1,12.2,2.1,13.5,13.8c3.8,33,4.6,66.2,5.3,99.4c0.2,8.3,0,8.5-7.9,8.4
c-5.6-0.1-12.6,2.4-16.3-1.2c-3.8-3.6-1.3-10.7-1.3-16.2c0.1-33.9,1.8-67.7,6.1-101.4C885.1,865.4,885.4,864.5,885.7,863.5z"/>
<path class="st1" d="M1788.4,1117.5c-4,0.3-6.2-1.5-8.7-2.2c-3.2-0.9-4.6-2.9-4.9-6.5c-0.7-10.3-2.2-20.5-2.9-30.8
c-1.5-24.6-2.8-49.2-2.8-73.9c0-7.5,0.1-7.8,7.7-7.6c5.9,0.2,13.3-2.7,17.2,1.6c3.4,3.7,1,10.9,0.9,16.5
C1794.3,1048.8,1792.8,1083.1,1788.4,1117.5z"/>
<path class="st1" d="M1481.1,1078.1c-10.7,0.3-20.5-1.5-30.3-2.6c-34.1-3.7-68.2-5.9-102.5-5.6c-1.3,0-2.7-0.1-4,0
c-4.9,0.5-5.1-2.5-5.2-6.2c-0.1-4,0.7-6,5.4-6c20,0.1,39.9-0.2,59.9,1c22.9,1.4,45.8,3,68.7,4.7c7.9,0.6,7.8,0.9,8.2,8.7
C1481.2,1073.7,1481.1,1075.4,1481.1,1078.1z"/>
<path class="st1" d="M1480,903c2,3.3,0.7,6.3,1,9.2c0.4,4.3-1.6,5.6-5.8,5.9c-42.2,3.3-84.3,6.2-126.7,5.9c-1.3,0-2.7-0.1-4,0
c-4.5,0.5-5.5-1.7-5.6-5.8c-0.1-4.5,0.9-6.6,6-6.5c29.7,0.7,59.2-1.5,88.8-3.9C1449.3,906.5,1464.7,904.6,1480,903z"/>
<path class="st1" d="M1192.2,1078.2c1-14.7,0.9-15,13.5-15.8c40.5-2.9,80.9-5.4,121.6-4.6c7.3,0.1,7.4,0.5,7.3,7
c-0.1,4-1.7,5.2-5.4,5.1c-6-0.2-12-0.1-18,0c-37.3,0.2-74.3,3.2-111.3,7.8C1197.6,1078,1195.3,1078,1192.2,1078.2z"/>
<path class="st1" d="M1192.6,903.2c17.2,2.2,33.7,3.8,50.2,5c28.2,2.1,56.4,4.4,84.7,3.6c3.8-0.1,7.3-0.4,7.1,5.4
c-0.3,6.3-0.2,6.8-6.6,6.8c-18.6,0-37.2,0.4-55.9-0.8c-19.2-1.2-38.5-2.1-57.8-3.2c-3-0.2-6-0.4-8.9-0.8
C1192.8,917.6,1192.8,917.6,1192.6,903.2z"/>
<path class="st1" d="M653.6,694.6c2.5,2.7,3.4,3.5,4.2,4.6c15.1,20.9,32.8,39.4,51.8,56.7c2.5,2.3,7.7,4.8,5,8.5
c-2.7,3.9-6.2-1-8.9-2.4c-14.7-7.8-28.1-17.9-42.2-26.6c-3.1-1.9-4.3-4.4-4.8-8C657.4,717,655.5,706.9,653.6,694.6z"/>
<path class="st1" d="M654,1284.6c0.9-5.5,2.7-11,2.6-16.4c-0.1-14.4,6.3-23.8,19.2-30.2c10.6-5.3,20.3-12.5,30.5-18.8
c2.5-1.5,4.9-3.3,7.8-3.6c1.6,2.1,1.5,4,0.2,5.7c-1,1.3-2.4,2.3-3.6,3.4c-18.4,17.3-36.5,34.8-51.2,55.5c-1.1,1.6-2.6,2.9-4,4.3
C655.3,1284.8,654.8,1284.6,654,1284.6z"/>
<path class="st1" d="M2019.1,698.8c-1.4,9.8-3,19.6-4.2,29.5c-0.4,3-2.2,4.7-4.4,6.2c-15.3,9.9-30.7,19.7-46.1,29.5
c-1.5,1-3.7,4-5.7,0.4c-1.5-2.6,0.2-4.4,2.1-5.9c11.9-9.2,21.5-20.7,32.4-30.8c8.4-7.7,14.5-17.2,21.4-26.1c0.9-1.2,2-2.4,3-3.6
C2018.1,698.2,2018.6,698.5,2019.1,698.8z"/>
<path class="st1" d="M2017.1,1283.9c-11.8-19.6-28.7-34.6-44.6-50.5c-3.7-3.7-7.8-7.2-11.8-10.6c-1.9-1.7-3.4-3.4-1.7-5.9
c1.7-2.4,3.5-0.5,5,0.5c15.7,10,31.3,20,46.9,30c2,1.3,3.5,2.9,3.9,5.4c1.4,10.1,2.9,20.2,4.4,30.4
C2018.5,1283.4,2017.8,1283.7,2017.1,1283.9z"/>
<path class="st1" d="M1044.6,1034c-0.6,11-1.1,21.9-1.7,32.9c-0.1,1.6-0.3,3.3-0.8,4.9c-0.3,0.8-1.4,1.6-2.2,1.7
c-0.9,0.1-1.9-0.5-2.1-1.7c-0.1-1-0.3-2-0.4-3c-1.2-22.2-2.3-44.5-2.4-66.8c0-3.2,0-6,4.6-5.9c3.5,0.1,5.6,0.7,5.5,4.9
c-0.2,11-0.1,21.9-0.1,32.9C1044.9,1034,1044.8,1034,1044.6,1034z"/>
<path class="st1" d="M1636.4,1033.8c0,10.6,0,21.2,0,31.8c0,2-0.3,4-0.7,5.9c-0.2,0.8-1.2,1.8-1.9,1.8c-0.8,0-1.9-0.8-2.2-1.5
c-0.5-1.2-0.5-2.6-0.6-3.9c-1.9-21.8-2.2-43.7-2.1-65.6c0-5.7,1.3-6.7,6-6.1c3.9,0.5,3.6,3.1,3.6,5.8c0,10.6,0,21.2,0,31.8
C1637.8,1033.8,1637.1,1033.8,1636.4,1033.8z"/>
<path class="st1" d="M1638.5,948.1c0,10.3,0,20.6,0,31c0,2.6,0.5,5.4-3.3,6.2c-4.4,0.9-6.4-0.5-6.4-5.8c0-19,0-38,1.4-56.9
c0.2-3,0.4-6,0.7-9c0.2-2.2-0.3-5.4,3-5.2c2.5,0.1,2.2,3,2.3,4.9c0.6,11.6,1.1,23.3,1.6,34.9C1638,948.1,1638.2,948.1,1638.5,948.1
z"/>
<path class="st1" d="M1045.1,948c0,10.6-0.1,21.2,0,31.9c0,3.6-0.9,5.5-5,5.5c-3.9-0.1-5.1-1.7-5.1-5.4c0.1-22.6,1.2-45.1,2.5-67.7
c0.1-1.6-0.1-3.7,2-4c2.8-0.4,3.1,1.9,3.2,3.9c0.7,11.9,1.3,23.9,1.9,35.8C1044.8,948,1044.9,948,1045.1,948z"/>
<path class="st1" d="M741.9,740.1c7,10.8,9.7,22,12.6,34.5c-8.1-3.6-13.5-8.9-19.7-13c-2-1.3-2-3.3-1.2-5.3
C735.6,751.1,737,745.6,741.9,740.1z"/>
<path class="st1" d="M1931.7,1241.3c-6.9-10.6-9.7-21.7-12.1-33.3c4.2,0.4,6.4,3.1,9.1,4.9
C1943.2,1223.1,1943.4,1224.7,1931.7,1241.3z"/>
<path class="st1" d="M1931.6,740c12,17.2,11.9,18.3-3.6,29.1c-2.4,1.7-4.5,3.9-8.5,4.8C1921.8,761.9,1925,750.9,1931.6,740z"/>
<path class="st1" d="M742.1,1241.3c-5.3-5.2-6.5-11.1-8.6-16.5c-0.8-2-0.5-3.8,1.6-5.1c6-3.9,11.2-9.1,19.2-12.6
C751.8,1219.5,748.7,1230.3,742.1,1241.3z"/>
<path class="st1" d="M599.6,517.5c0.6,0.1,1.4,0.1,1.9,0.4c12.6,7.1,13,7.9,10.7,24.4c-5.9-7.9-11-15.2-13.4-24.1
C599,518,599.3,517.7,599.6,517.5z"/>
<path class="st1" d="M597.6,1464.4c4-9.4,8.1-17.2,14-24.4C614.9,1455.5,614.5,1457.9,597.6,1464.4z"/>
<path class="st1" d="M2072.4,516.4c2.2,0.6,2.3,2,1.3,3.7c-3.8,6.1-7.7,12.1-11.9,18.6c-2.8-10.6-2.1-12.7,5.3-17.8
c1.9-1.3,4.2-2,5.3-4.3L2072.4,516.4z"/>
<path class="st1" d="M2061.5,1442.5c4.7,8.5,10.5,14.5,12.5,22.8C2067.2,1459.8,2056.4,1456.8,2061.5,1442.5z"/>
<path class="st1" d="M1781.6,836.9c4.2,9.3,4.2,9.3-1.8,12.6C1777.9,845.4,1780.6,842.1,1781.6,836.9z"/>
<path class="st1" d="M895.4,849.3c-7.5-2.9-7.5-2.9-3.8-11.1C893.8,841.3,893.9,844.7,895.4,849.3z"/>
<path class="st1" d="M630.1,1284.6c-0.2-5.4-2.7-11,1.8-17C634.2,1274.4,629.7,1279.3,630.1,1284.6z"/>
<path class="st1" d="M629.9,697.1c0.5,5.4,3.9,10.3,2.4,16.6C626.9,708.5,630.1,702.5,629.9,697.1z"/>
<path class="st1" d="M2044.5,1284.5c-2.2-5-4.3-9.2-2.8-15C2046.1,1274.3,2043.5,1279.4,2044.5,1284.5z"/>
<path class="st1" d="M2041.9,712.1c-2.2-6,1.1-10.1,2.2-14.6C2043.5,702.2,2046,707.4,2041.9,712.1z"/>
<path class="st1" d="M892.2,1143.9c-4.6-7.9-4.6-7.9,1.6-11.5C895.7,1136.5,892.6,1139.7,892.2,1143.9z"/>
<path class="st1" d="M1779.6,1132.2c6.1,3.1,6.1,3.1,2.7,11.1c-1.9-1-1.7-3.1-2.3-4.6C1779.2,1137,1778.6,1135.1,1779.6,1132.2z"/>
<path class="st1" d="M1934.8,1202.5c4.1-0.6,6.5,2.5,11.4,4.9c-6.3,0.5-8.2-3.5-11.3-5.1L1934.8,1202.5z"/>
<path class="st1" d="M1935.8,777c3.1,0.9,4.5-3.5,7.8-2.9c-1.8,4-4.7,3.9-7.9,2.8L1935.8,777z"/>
<path class="st1" d="M729.5,774c2.7-0.5,4.4,0.6,5.8,3.2C732.1,778,731,775.7,729.5,774z"/>
<path class="st1" d="M734,1202.5c2.3,4.6-1.2,4.8-4.2,5.9c-0.6-3.7,3.7-3.3,4.2-5.7C734,1202.7,734,1202.5,734,1202.5z"/>
<path class="st1" d="M1935.2,780.3c-0.4,0.2-0.8,0.5-1.2,0.7c0.3,0,0.6,0.1,0.8,0c0.2-0.1,0.4-0.3,0.6-0.5
C1935.5,780.5,1935.2,780.3,1935.2,780.3z"/>
<path class="st1" d="M740,1203.1c-1.9-0.8-4.2,2.9-6-0.5c0,0,0,0.2,0,0.1C736,1202.8,738,1202.9,740,1203.1L740,1203.1z"/>
<path class="st1" d="M1935.7,777c1.4,1.3,0.9,2.4-0.5,3.3c0,0,0.3,0.3,0.3,0.3c0.1-1.2,0.3-2.3,0.4-3.5
C1935.8,777,1935.7,777,1935.7,777z"/>
<path class="st1" d="M737.9,778.4c0.2,0.2,0.4,0.5,0.6,0.7c-0.2,0.2-0.5,0.5-0.7,0.5c-0.2,0-0.4-0.4-0.6-0.6
C737.4,778.8,737.7,778.6,737.9,778.4z"/>
<path class="st1" d="M740,1203.1c0-0.3,0-0.8-0.2-1c-0.8-0.8-0.7-1.1,0.4-0.8C739.5,1201.9,739.4,1202.4,740,1203.1
C740,1203.1,740,1203.1,740,1203.1z"/>
<path class="st1" d="M1934.9,1202.3c0-0.2-0.1-0.3-0.1-0.5c0,0-0.1,0.1-0.1,0.1c0.1,0.2,0.1,0.4,0.2,0.6
C1934.8,1202.5,1934.9,1202.3,1934.9,1202.3z"/>
<path class="st1" d="M2072.4,516.6c0.7,0.4,1.4,0.8,2.1,1.2c-0.2,0.2-0.4,0.7-0.6,0.7c-1.1-0.2-1.5-1-1.4-2
C2072.4,516.4,2072.4,516.6,2072.4,516.6z"/>
<path class="st1" d="M598.7,518.3c-1-0.2-1.8-0.8-1-1.6c0.8-0.9,1.4,0,1.8,0.8C599.3,517.7,599,518,598.7,518.3z"/>
</g>
</svg>

After

Width:  |  Height:  |  Size: 48 KiB

View File

@ -0,0 +1,458 @@
<?xml version="1.0" encoding="utf-8"?>
<!-- Generator: Adobe Illustrator 24.2.1, SVG Export Plug-In . SVG Version: 6.00 Build 0) -->
<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
viewBox="0 0 2670 1980" style="enable-background:new 0 0 2670 1980;" xml:space="preserve">
<style type="text/css">
.st0{fill:#0A0A0A;}
.st1{fill:none;}
</style>
<g>
<path class="st0" d="M2275.3,1000.1c-0.1,2.3,0,4.7,0,7c0,189.3,0,378.7,0,568c0,6,0.1,12.1-0.8,17.9c-2.4,15.2-12.9,17.4-23.6,10
c-45.4-31-91.2-61.4-136.9-91.9l-30.1-20.3c-9.8-6.3-19.2-13.2-29.2-19.3c-6.7-4.1-9.8-9.7-9.9-17.2c-0.1-9.3-0.6-18.7-0.3-28
c0.2-7.3-1.7-13-6.5-19c-18.6-23.3-36.2-47.3-56.9-69c-18-18.8-36.7-36.8-56.4-53.8c-31.8-27.5-65.4-52.7-101-74.9
c-32.7-20.4-66.8-38.4-102.2-54c-50.7-22.4-103.2-39.2-157.1-51.5c-36.7-8.4-73.8-14.4-111.2-18.5c-34.5-3.8-69-6.5-103.7-5.8
c-23.1,0.5-46.2-1.3-69.3,0.4c-22.9,1.8-45.9,3-68.7,5.6c-72.7,8.4-143.7,24.6-212.1,50.6c-65.5,24.8-127.4,57-184.9,97.4
c-46.8,32.8-90.1,69.8-129.5,111.1c-18.1,19-35.1,39.2-50.5,60.6c-3.9,5.5-5.5,11-5.3,17.6c0.4,9.3,0,18.7,0,28
c0,9.3-3.1,16.4-11.8,21.4c-10.4,5.9-19.9,13.4-30.1,19.8l-25.4,17.2c-11.6,7.8-23.3,15.5-34.9,23.3c-34.3,23-68.6,46-102.9,69
c-14.6,9.7-23.6,5.5-25.2-12.1c-0.5-5.3-0.5-10.7-0.5-16c0-189.7,0-379.3,0-569l-0.1-26.1c0.2-190.3,0.1-380.7,0.2-571
c0-5.7,0.4-11.3,0.5-17c0.1-3.7,1.2-7.3,3-10.5c3.4-5.9,9.3-7.7,15.4-4.8c2.7,1.3,5.1,3.1,7.6,4.8c44.3,29.6,88.5,59.3,132.8,88.9
l29.7,20.1c10.8,6.6,20.9,14.3,31.7,20.9c6.7,4.1,9.7,9.8,9.8,17.3c0.2,11,0.4,22,0.2,33c-0.1,5.2,1.2,9.6,4.2,13.9
c15.4,22.2,33,42.8,51.6,62.2c53.9,56.4,114.1,105.1,181.4,144.6c38.4,22.6,78.3,42.4,119.9,58.6c40,15.6,80.9,28.3,122.8,38
c52.7,12.1,106,19.3,160,22.6c18.3,1.1,36.6,2.1,54.9,1.5c17.7-0.6,35.5,1.1,53.2-0.1c17.6-1.1,35.2-2.3,52.8-3.6
c35.9-2.5,71.4-8.5,106.7-15.1c53.4-9.9,105.5-25,156-45.1c53.8-21.5,105.1-48.2,153.2-80.9c43.4-29.5,83.6-62.7,121.1-99.3
c27.3-26.6,50.3-56.8,73.9-86.6c2.9-3.6,3.6-7.6,3.6-12.1c-0.1-10.7,0.2-21.3,0.4-32c0.1-7.5,3.2-13.1,9.9-17.2
c10.2-6.2,19.9-13.4,30-19.8l23.2-16.3c4.2-1.3,7.4-4.3,11-6.7c42.9-28.7,85.8-57.4,128.7-86.1c1.1-0.7,2.3-1.4,3.3-2.3
c10.9-9.1,21.4-4.7,23.8,10.7c0.8,5.6,0.6,11.3,0.6,17c0,189.3,0,378.7,0,568L2275.3,1000.1z M740,1203.1c-2-0.2-4-0.3-6-0.5
c-0.5,2.5-4.8,2-4.2,5.8c2.9-1,6.4-1.2,4.2-5.8C735.8,1206,738.1,1202.3,740,1203.1c-0.6-0.7-0.5-1.3,0.2-1.7
c-1.1-0.3-1.1-0.1-0.4,0.8C740,1202.3,739.9,1202.8,740,1203.1z M1935.5,780.5c-0.2,0.2-0.4,0.4-0.6,0.4c-0.3,0.1-0.6,0-0.8,0
c0.4-0.2,0.8-0.5,1.1-0.8c1.4-0.9,2-1.9,0.5-3.3c3.2,1.1,6.1,1.2,7.9-2.8c-3.3-0.7-4.6,3.8-7.7,2.9
C1935.7,778.2,1935.6,779.4,1935.5,780.5z M1934.8,1202.5c-0.1-0.2-0.1-0.4-0.2-0.6c0,0,0.1-0.1,0.1-0.1c0,0.2,0.1,0.3,0.1,0.5
c3.1,1.6,5,5.7,11.3,5.1C1941.3,1205,1938.9,1202,1934.8,1202.5z M2072.4,516.6c-1,2.3-3.4,3-5.3,4.3c-7.4,5.1-8.2,7.2-5.3,17.8
c4.2-6.5,8-12.6,11.9-18.6c1.1-1.7,0.9-3.1-1.3-3.6c0,1,0.3,1.8,1.4,2c0.2,0,0.4-0.4,0.6-0.7C2073.8,517.4,2073.1,517,2072.4,516.6
z M599.6,517.5c-0.4-0.8-1.1-1.7-1.8-0.8c-0.8,0.8,0.1,1.5,1,1.6c2.4,8.9,7.5,16.2,13.4,24.1c2.3-16.5,1.9-17.3-10.7-24.4
C600.9,517.6,600.2,517.6,599.6,517.5z M534.9,1353.5c-20,21.8-37.3,44.1-53.4,67.4c-16.1,23.4-26.8,49.5-41.8,73.5
c9-21,17.4-42.2,28.9-61.9c16.9-28.7,35.4-56.4,57.5-81.5c0.9-1,1.6-2.1,2.6-3.1c2.9-2.9,3.3-6,2.3-10
c-9.1-33.8-14.6-68.3-19.2-103c-5-38-8.3-76.1-10.6-114.3c-2.5-39.9-3.4-79.9-3.9-119.8c-0.1-5.1-1.4-6.8-6.7-6.9
c-21.6-0.2-43.3-0.8-64.9-1.3c-2,0-4-0.4-6,0.8c0,195.2,0,390.5,0,585.6c2.1,0.6,3.1-0.4,4.1-1.1c52.7-33.8,105.3-67.7,158-101.4
c3.6-2.3,5.7-5.4,7.3-9.1c6-13,12.5-25.8,20.8-37.4c4.2-5.9,3.9-11.9,2.5-19.1c-2.4,1.8-4,2.9-5.5,4.2
c-13.9,12.6-32.1,8.7-42.7-2.9c-7.3-8-12.4-17.2-17.1-26.8C542.3,1375.7,539.4,1365.1,534.9,1353.5z M534.6,627.2
c3.6-5,4.3-10.7,6.3-15.9c6.2-15.8,12.6-31.5,25.2-43.7c10.8-10.5,26.8-12.6,38.9-2.8c2.3,1.9,4.3,5,7.9,5
c1.6-8.3-0.2-15.3-5.1-21.9c-7.4-9.9-12.7-21-17.8-32.3c-2.4-5.3-5.8-9.2-10.8-12.4c-50.5-32.2-100.9-64.7-151.3-97.1
c-2.4-1.5-4.5-3.9-8.7-3.9c0,3.7,0,7.2,0,10.7c0,187.9,0,375.8,0,563.7c0,2,0.2,4,0,6c-0.6,4.9,1.4,6.3,6.2,6.1
c13.6-0.6,27.3-0.7,41-1c8.3-0.2,16.7-0.6,25-0.4c4.9,0.1,6.7-1.8,6.1-6.4c-0.1-1,0-2,0-3c1.2-75,4.3-149.8,13.7-224.3
c4.6-36.3,10-72.5,19.6-107.9c1.6-5.7,0.4-9.9-3.4-14.4c-10.7-12.8-21.7-25.4-31-39.1c-19-28.1-36.8-57-49.6-88.7
c-2.2-5.4-5.5-10.4-6.2-16.6C465,538.3,494.6,586,534.6,627.2z M2060.9,1411.1c-2.3,8.5-0.1,15.1,4,21.4
c5.9,9.3,13.4,18.1,17.1,28.2c3.9,10.7,10.9,16.6,19.9,22.4c47.8,30.3,95.3,61.1,143,91.7c2.7,1.7,5,4.3,9.5,4.5c0-3.4,0-6.6,0-9.8
c0-188.3,0-376.6,0-565c0-2-0.1-4,0-6c0.2-3.4-1.1-5-4.6-4.8c-2.3,0.1-4.7,0-7,0.1c-20,0.6-40,1.2-60,1.9c-6.7,0.2-6.7,0.4-7,7.4
c0,1.3,0,2.7-0.1,4c-0.4,60-3,119.9-8.9,179.7c-5,50.1-11.3,99.9-24,148.7c-1.4,5.2,0.1,8.3,3.5,12.3c15,17.8,29.6,36.1,42.4,55.5
c20,30.4,36.1,62.9,49.3,96.8c3.2,8.2,5.7,16.6,6.8,25.4c-33.9-90.8-82.6-145-107.2-169.7c-4.1,11-7.5,22.1-12.7,32.5
c-4.9,9.9-10.3,19.5-18.8,26.8c-7.1,6.1-15.1,8.9-24.8,7.5C2073.2,1421.4,2068.3,1415.5,2060.9,1411.1z M2244.8,454.3
c0.6,4.5-0.2,6.7-0.8,8.9c-6.5,24.2-17,47-28.2,69.2c-18.9,37.2-43,71.2-70.4,102.8c-2.7,3.2-3.6,5.8-2.5,9.8
c7.9,29.6,13.1,59.8,17.3,90.2c6.4,45.6,10.2,91.3,12.8,137.3c2,35.3,2.6,70.6,3.1,105.9c0.1,7.6,0.2,7.6,7.6,7.8
c17,0.5,34,1,51,1.4c5,0.1,10,0.2,15,0.5c3.2,0.1,5.1-0.9,4.9-4.5c-0.1-1.7,0-3.3,0-5c0-189.6,0-379.3-0.1-568.9
c0-2.4,1.1-5.2-1.2-7.3c-0.8,0.1-1.6,0-2.1,0.4c-52.7,33.8-105.3,67.7-158,101.5c-2,1.3-3.7,2.6-4.7,4.9c-6.9,16-16.9,30.1-25.8,45
c-3.1,5.2-3.4,10.2-2,15.7c2.6,0.1,3.5-1.6,4.6-2.6c15.7-13.9,33-10.6,45.6,3.8c6,6.8,10.4,14.6,14.4,22.7
c5.1,10.3,8.3,21.3,12.8,32.9C2185.2,576.1,2221.7,520.5,2244.8,454.3z M553.7,662.8c-0.3,1.3-0.7,2.8-1,4.3
c-7.1,31.2-11.7,62.8-15.4,94.5c-7.1,59.9-10.3,120.1-11.8,180.4c-0.3,13-0.2,26-0.5,39c-0.1,4.4,1.5,5.7,5.7,5.6
c25-0.5,50-0.7,74.9-1.1c7.4-0.1,7.6-0.2,7.6-7.8c0-85.6,0-171.2,0.1-256.8c0-5.3-1.7-8.5-5.8-11.7c-16.6-12.8-32.4-26.6-46.9-41.9
C559,665.6,557.6,663,553.7,662.8z M553.6,1319.7c2.1-1,2.8-1.2,3.2-1.6c16.4-17.1,34-32.9,52.7-47.4c3-2.4,3.8-5,3.8-8.6
c-0.1-13.3-0.1-26.7-0.1-40c0-72.3,0-144.7,0-217c0-10.3,0.4-9-9.2-9.2c-23.3-0.4-46.7-0.8-70-1.1c-2.8,0-5.9-1-9.3,1.5
C526.2,1104.4,531.1,1212.3,553.6,1319.7z M2119.8,1319.2c4.8-20.9,8.3-40.2,11.1-59.6c5.8-39.5,9.8-79.2,12.5-119.1
c3.1-45.5,4.7-91.1,5.1-136.7c0.1-7.6-0.1-7.7-7.6-7.7c-24.3,0.1-48.6,0.3-72.9,0.5c-7.9,0.1-8,0.1-8,8.5c0,84.6,0,169.1-0.1,253.7
c0,5.6,1.7,9.3,5.9,12.7c12.1,9.9,24.2,19.9,36,30.1C2107.7,1306.9,2113.1,1312.7,2119.8,1319.2z M2120.4,664.4
c-2.1-0.6-2.9-0.1-3.6,0.7c-15.2,16.5-33.1,29.9-50.2,44.2c-4.8,4-6.8,8-6.8,14.3c0.3,32,0.2,64,0.2,96c0,52.3,0,104.6,0,157
c0,8.1,0.1,8.2,8.2,8.2c18.3,0.2,36.7,0.2,55,0.4c6.7,0.1,13.3,0.1,20,0.3c3.7,0.1,5.7-1.1,5.4-5.1c-0.2-2.3,0-4.7-0.1-7
c-0.5-45-2.1-89.9-5.2-134.8C2139.2,780.1,2132.7,721.9,2120.4,664.4z M878.7,1138.5c-36,15-69.8,32.9-102.3,53.1
c-3.4,2.1-4.9,4.7-5.7,8.6c-2.8,14.4-5.7,28.8-12.2,42c-3.6,7.3-7.7,14.4-17.3,14.4c-9.6,0-13.4-7.4-17.4-14.4
c-1.9-3.2-2-7.6-5.4-9.8c-22.3,17.4-41.6,37.2-58.8,58.9c-5.8,7.3-10.2,15.3-12,24.8c-3.8,20.3-9.1,40.2-16.7,59.5
c-2.6,6.7-1.3,13.3-1.5,20.3c1.5-0.4,2-0.3,2.1-0.5c3.2-3.8,6.4-7.6,9.4-11.6c31.2-40.5,66-77.4,105.4-110.2
c36.9-30.8,75.6-59.1,117.2-83.1c54.3-31.2,110.9-57.4,170.8-76.4c38.4-12.2,77.4-21.8,117-29.2c9.5-1.8,19.1-3,28.6-4.6
c1.5-0.3,3.8,0.3,4.4-1.5c1.4-3.9,0.6-8-0.4-11.7c-1-3.6-4.5-1.7-6.9-1.5c-40.6,3.2-80.4,11-120.4,18c-7,1.2-10.4,4.1-10.9,11.2
c-0.3,3.8-0.5,9.1-5.8,9c-4.6-0.1-4.7-5-5.3-8.5c-0.7-4.5-2.4-6-7.3-4.9c-39.3,9.1-78,19.9-115.5,34.7c-4.6,1.8-7.1,4.2-7.8,8.9
c-0.9,5.9-2.2,11.8-5.2,17.1c-4.1,7.1-10.6,7.2-14.6,0C881.9,1147.7,880.7,1143.4,878.7,1138.5z M1032.8,891.9
c0.9-3.8,1.6-6.7,2.4-9.5c0.7-2.3,1.7-4.6,4.4-4.9c3.4-0.3,4.4,2.3,5.3,4.9c0.4,1.3,0.8,2.6,0.8,3.9c0.1,7.7,4.5,10.4,11.7,11.6
c32.5,5.5,64.8,11.9,97.5,15.7c7.9,0.9,15.9,1.7,23.8,2.4c1.9,0.2,4,0.7,5.3-1.6c3.5-6.5,0.3-13-7-14.2c-1-0.2-2-0.2-3-0.4
c-42.6-5.8-84.2-15.9-125.4-28.2c-41.8-12.4-82.5-28.2-121.6-47.1c-51-24.6-99.8-53.3-144.7-87.9c-20.6-15.8-40.9-32.1-59.9-49.7
c-29.1-26.9-56.4-55.7-80.1-87.7c-3.7-4.9-7.7-9.6-11.6-14.3c-0.8,7.6-2.4,14.7,0.5,21.8c7.4,18.2,12.4,37.2,16,56.5
c1.7,9.1,5,17.3,10.6,24.4c9.2,11.4,18.5,22.8,28.7,33.3c9.9,10.1,21.1,19,32.2,28.9c3.8-6.5,5.6-13,9.8-18.2
c7.7-9.4,17.7-9.5,25.3-0.1c3.6,4.5,5.7,9.7,7.8,15c4.3,10.9,6.8,22.3,8.9,33.7c0.9,5.2,3.3,8.1,7.5,10.7
c21.9,13.4,44.1,26,67.3,37.1c10.9,5.2,22.1,10,33.5,15.2c1.8-4.8,3.1-8.6,4.7-12.2c1.6-3.5,4.2-6.3,8.3-6.1
c3.6,0.2,5.8,2.9,7.3,6.1c2.4,5.2,3.8,10.6,4.8,16.2c0.8,4.6,2.9,7.2,7.7,9C950.6,871.5,990.8,883,1032.8,891.9z M1953.8,1233.7
c-2.9,5.8-4.9,11.5-8.8,16.2c-7.6,9.3-17.8,9.3-25.3,0c-3.4-4.2-5.5-9-7.5-14c-4.4-11.5-7.2-23.5-9.4-35.7
c-0.7-3.9-2.4-6.4-5.8-8.6c-30.4-19.4-62.4-35.6-95-50.8c-6.9-3.3-7-3-9.9,4.7c-0.7,1.9-1.5,3.7-2.3,5.5c-1.6,3.1-3.7,5.6-7.6,5.6
c-3.8,0-5.9-2.5-7.6-5.6c-2.9-5.4-4.3-11.3-5.1-17.2c-0.7-5-3-7.3-7.5-9.1c-37.8-15.2-77.3-24.6-116.7-34.3c-3.8-0.9-5.4,0.2-6.2,4
c-0.7,3.9-0.2,9.9-6.3,9.5c-5-0.3-5.1-6.1-5.3-9.6c-0.4-6.6-3.3-8.9-9.6-10c-32.8-5.6-65.4-12.2-98.6-15.7
c-8.3-0.9-16.5-2.1-24.8-2.7c-5.4-0.4-7.1,1.2-7.5,6.6c-0.3,3.9-1.3,8,5.2,8.4c6.3,0.4,12.4,2.1,18.7,3.1
c47.7,7.5,94.4,19.1,140.3,34.3c42.4,14,83.5,31.2,123,52c46.3,24.4,90.2,52.6,131,85.6c48.5,39.1,93.7,81.2,129.4,132.6
c1.9,2.8,4.8,7.5,7.3,6.7c4.5-1.5,1.4-6.8,2.1-10.4c0.6-3.4-0.5-6.6-1.8-9.7c-7.8-19.5-12.6-40-17.1-60.4c-1.1-5.1-3-9.4-6-13.5
c-17-22.8-36.7-43.1-57.6-62.2C1959.6,1237.1,1957.6,1234.7,1953.8,1233.7z M2043.6,585.1c-4.2,2.7-6.2,6.2-8.5,9.2
c-29.9,40.3-64.6,76.1-102.5,109c-33.8,29.3-69.5,56.2-107.6,79.4c-51.8,31.6-106.3,57.7-163.9,77.3
c-40.7,13.8-82.1,24.9-124.2,33.1c-13.7,2.7-27.6,4.7-41.3,7c-2.2,0.4-5.3-0.1-6.2,2.4c-1.4,3.8-1.1,8.2,0.3,11.7
c1.4,3.5,5.3,1.6,8.1,1.3c40.1-4.2,79.8-10.7,119.5-18.1c6.2-1.2,10.4-2.7,10.5-10.2c0-3.5,0.3-9.3,5.4-9.5c6-0.3,5.5,5.7,6.2,9.6
c0.7,3.9,2.3,4.9,6.1,3.9c10-2.7,20-5.1,30-7.5c28.5-6.9,56.4-15.6,83.9-25.7c6.8-2.5,10.2-6.3,11-13.3c0.5-5,2-9.8,4.5-14.2
c4.2-7.6,10.8-7.6,15.1,0.1c1.6,2.9,2.5,6.2,3.4,9.3c0.9,2.9,2.5,3.5,5.2,2.2c2.4-1.2,5-1.9,7.3-3.1c26.6-13.1,53.2-26,78.7-41.2
c11.9-7.1,19.6-14.7,21.3-29.3c1.4-12.4,5.5-24.8,12.8-35.4c8.1-11.7,19.5-11.7,27.5,0.1c3.1,4.6,5.4,9.7,8.2,14.9
c1.9-1.4,3.6-2.5,5.1-3.8c22.2-19.8,42.7-41.1,60.6-64.9c2.6-3.5,4.3-7.3,5.2-11.7c4.5-21.2,9.6-42.2,17.6-62.4
C2045.3,599.2,2044.5,592.6,2043.6,585.1z M777.2,1163.8c1.2,0.2,1.6,0.4,1.9,0.3c30.4-14.3,60.9-28.3,92.6-39.6
c4.2-1.5,4.4-3.7,3.9-7.5c-4.5-37.4-6.3-75-6.5-112.6c0-7.8-0.2-7.9-8.5-7.9c-22,0-44,0-65.9,0c-8.6,0-8.8,0.1-8.8,8.7
c-0.4,45.3-2.2,90.5-6.6,135.6C778.4,1148.4,776.3,1155.9,777.2,1163.8z M776.8,817.8c0.3,3.8,0.5,7.1,0.9,10.4
c2.5,21.2,4.1,42.4,5.5,63.7c1.8,28.6,2.3,57.2,2.7,85.9c0.1,7.1,0.3,7.3,7.2,7.3c22.6,0.1,45.3,0.1,67.9,0.1
c8.8,0,8.1-0.2,8.1-8.3c0.1-37,1.9-73.9,6.4-110.6c0.7-5.8-0.5-8.2-6-10.1c-29.5-10.3-58-23.3-86.2-36.8
C781.5,818.4,779.9,817.2,776.8,817.8z M1896.4,1163.9c1-2.7,0.2-5.3-0.1-7.9c-2.4-19.8-3.9-39.8-5.3-59.7
c-2.1-30.3-2.5-60.6-3.3-90.9c-0.2-8.7-0.2-8.8-8.6-8.8c-16.7-0.1-33.3,0-50-0.1c-6.3,0-12.7,0.2-19-0.1c-3.9-0.1-5.5,1.4-5.2,5.3
c0.1,1.7,0.1,3.3,0,5c-1.1,36-2,72-6.4,107.7c-0.9,7.7-1.1,7.9,6.3,10.6c28.5,10.4,56.1,22.9,83.3,36.1
C1890.6,1162.4,1893.1,1164.4,1896.4,1163.9z M1896.8,817.7c-2.5-0.3-4.3,0.6-6.1,1.5c-6,2.9-12,5.6-18,8.6
c-22.1,10.9-44.9,20.1-67.9,28.7c-7.5,2.8-7.3,3-6.5,10.6c1,8.9,1.9,17.9,2.7,26.8c2.4,28.2,3.2,56.5,3.6,84.7
c0.1,5.4,1.8,6.7,6.8,6.6c22.6-0.2,45.3-0.1,67.9-0.2c1,0,2-0.1,3,0c3.8,0.3,5.6-1.2,5.2-5.1c-0.2-1.6,0-3.3,0.1-5
c0.7-38.6,1.7-77.2,5.2-115.6C1894.1,845.4,1895.5,831.6,1896.8,817.7z M908.3,1109.6c1.4,0,2.4,0.2,3.2,0
c37.8-11.7,75.5-24,114.2-32.5c4.5-1,5.8-2.9,5.5-7.4c-1.3-22-2.5-43.9-2.3-65.9c0.1-7.3-0.2-7.4-8-7.4c-33.3,0-66.7,0.1-100,0.2
c-7.7,0-7.9,0.1-7.9,7.5c0,31-1.6,61.9-4,92.9C908.8,1101,907.3,1105.2,908.3,1109.6z M1765.3,1110.4c0-2.5,0.1-3.5,0-4.4
c-3.1-33.9-4.7-67.8-4.8-101.8c0-7.4-0.4-7.6-8.2-7.7c-33.3-0.1-66.7-0.2-100-0.2c-7.3,0-7.2,0.1-7.5,7.9c-0.7,21-1.1,42-2.3,62.9
c-0.5,9-1.3,8.3,7.6,10.4c29.8,7.1,59.4,14.9,88.5,24.5C1747.2,1104.8,1755.7,1107.4,1765.3,1110.4z M908.8,872.2
c-0.2,0.3-0.6,0.6-0.6,0.8c2.8,35.5,5.2,71,4.8,106.7c-0.1,4.6,1.9,5.6,6,5.6c34.7-0.1,69.3-0.1,104,0.1c4.9,0,6.2-1.8,6.1-6.4
c-0.5-22,0.8-44,2.1-65.9c0.3-5.2-0.9-7.7-6.4-8.8c-36.9-7.6-72.4-20.1-108.5-30.5C913.8,873,911.6,871.1,908.8,872.2z
M1765.6,872.4c-3.2-0.6-5.1,0.3-7,0.9c-22.2,6.7-44.2,14.5-66.7,20.3c-14.5,3.7-29,7.3-43.5,10.8c-4.4,1.1-6.5,2.6-6.2,8
c1.3,21.9,1.8,43.9,2.6,65.8c0.2,7,0.2,7.1,7.1,7.1c33.6,0,67.2,0,100.8-0.1c7.5,0,7.7-0.2,7.7-7.8c0.2-24.6,0.9-49.2,2.6-73.8
C1763.8,893.4,1764.7,883.1,1765.6,872.4z M725.8,1192.5c8.7-4.8,15.8-9.8,23.8-12.8c9.6-3.7,12.7-10.5,13-19.7
c0.1-1.3,0.2-2.7,0.4-4c3.4-27.1,5.6-54.3,6.7-81.5c1-23.6,1.5-47.2,2-70.8c0.2-7-0.1-7-7.3-7.1c-6.7-0.1-13.3,0-20,0
c-9,0-18,0.2-27,0c-4.2-0.1-6.2,1.3-5.9,5.7c0.2,2.3,0,4.7,0.1,7c0.8,27.6,1.2,55.2,3,82.8C717,1125.1,719.9,1158.2,725.8,1192.5z
M1946.2,1191.7c2.9-2.2,2.4-4.7,2.8-6.9c4.5-26.6,6.9-53.5,8.9-80.4c2.5-33.9,3.2-67.9,3.7-101.8c0.1-5-1.8-6.2-6.4-6.1
c-15.3,0.2-30.6,0-46,0.1c-7.6,0-7.8,0.2-7.7,7.8c0.7,44,2,87.9,6.7,131.6c1.1,10.3,2.2,20.5,3.5,30.7c0.4,2.8,0.2,6,3.5,7.8
C1925.8,1180.2,1936.2,1186.1,1946.2,1191.7z M1947.7,789.1c-9.1,5-16.9,10.1-25.3,13.8c-8.2,3.6-10.5,9.5-11.5,17.8
c-3,27.1-5.6,54.3-6.9,81.6c-1.2,25.3-1.5,50.6-2.5,75.9c-0.2,5.1,1.5,7,6.7,7c15-0.3,30-0.1,45-0.2c8.6,0,8.4,0.6,8.3-8.9
c-0.3-30.6-1-61.3-3.1-91.9C1956.3,852.9,1953.6,821.8,1947.7,789.1z M726,789.2c-3.3,16.4-5.1,31.3-6.7,46.2
c-4.6,42.4-6.3,85-7.3,127.7c-0.1,5.7,0,11.3-0.2,17c-0.1,3.4,1.4,4.8,4.7,4.8c17,0,34,0,51,0.1c3.6,0,4.6-1.9,4.4-5
c-0.1-2.3-0.1-4.7-0.1-7c-0.5-37.3-1.7-74.6-5-111.8c-1.4-15.9-2.8-31.9-5.5-47.6c-0.4-2.6-0.3-5.2-3.2-6.8
C747.6,801.1,737.2,795.4,726,789.2z M1625.1,910.5c-9.8,1.8-18,3.4-26.2,4.6c-23.3,3.5-46.2,9.9-70,10
c-10.9,0.1-21.8,2.4-32.7,3.7c-3.5,0.4-6.2,1.1-6,6c0.5,15,0.7,30,0.7,44.9c0,4.4,1.7,5.8,5.9,5.7c8.3-0.2,16.6,0,25-0.1
c30.6,0,61.3,0,91.9-0.1c9,0,9-0.1,9.2-9.4c0.2-13,0.2-26,0.6-38.9C1623.7,928.4,1624.5,919.8,1625.1,910.5z M1625.2,1071.1
c-2.2-23.8-2.3-46.1-2.4-68.4c0-4.5-1-6.6-6.1-6.6c-39.9,0.1-79.9,0.1-119.8-0.1c-5.1,0-6.1,2-6.1,6.5c0,14.6-0.1,29.3-0.6,43.9
c-0.1,4.5,1.6,5.6,5.6,6.1c12.6,1.4,25,3.8,37.7,4c8,0.1,15.9,1.1,23.8,2.5C1579.5,1063,1601.7,1066.9,1625.2,1071.1z
M1049.3,1070.4c2.6,1,4.9-0.1,7-0.6c23-5.5,46.3-8.9,69.7-12.3c16.7-2.5,33.7-3.8,50.6-5.2c5.1-0.4,6.8-1.9,6.6-7
c-0.4-14-0.3-27.9-0.5-41.9c-0.1-7-0.2-7.2-7.2-7.2c-39.3,0-78.5,0-117.8,0.1c-6.7,0-6.8,0.2-6.8,7.5c0,16.3-0.5,32.6-1.4,48.9
C1049.3,1058.6,1048.4,1064.5,1049.3,1070.4z M1049.8,910.2c-0.4,1.2-0.8,1.8-0.8,2.4c0.6,22.6,2.5,45.2,1.9,67.8
c-0.1,4.2,2.1,4.8,5.6,4.8c39.9,0,79.9-0.1,119.8,0.1c5.5,0,6.7-2.3,6.6-7.2c-0.2-11,0.1-22,0.2-32.9c0.1-15.9,0.1-15.5-16.3-17.4
c-16.5-1.9-33.1-2.4-49.6-5.2C1094.7,918.9,1072,915.9,1049.8,910.2z M1260.5,985.4c21.6,0,43.2,0,64.9,0c9.1,0,9.1-0.1,9.1-9.4
c0-10.3,0-20.6-0.1-30.9c-0.1-7.7-0.1-7.7-7.6-7.9c-2.3-0.1-4.7,0-7,0c-41.9,0-83.8-2.2-125.5-6.2c-4.6-0.4-5.9,0.9-6,5.1
c-0.2,13.6-0.6,27.3-0.8,40.9c-0.1,8.3,0.1,8.4,8.1,8.4C1217.3,985.4,1238.9,985.4,1260.5,985.4z M1485,931.5
c-3.1-1.4-5.4-0.6-7.7-0.4c-26.2,1.9-52.4,4.3-78.7,5.1c-17.3,0.5-34.6,1.3-51.9,1.1c-7.4-0.1-7.5,0.1-7.6,7.8
c-0.1,11.7,0,23.3-0.1,35c0,3.7,1,5.8,5,5.5c2-0.1,4,0,6,0c41.6,0,83.2,0,124.9,0c2,0,4-0.1,6,0c3.3,0.1,5-1.2,4.9-4.6
C1485.5,964.2,1485.3,947.6,1485,931.5z M1485,1051.1c0.3-17,0.5-33.6,0.9-50.1c0.1-4.2-2.2-4.8-5.7-4.8c-45,0.1-89.9,0.1-134.9,0
c-4.7,0-6.5,1.4-6.3,6.2c0.3,12,0.4,24,0,36c-0.2,5.2,2.1,6,6.5,6c11.6-0.1,23.3,0.1,34.9,0.4
C1415,1045.8,1449.5,1048.1,1485,1051.1z M1188.4,1050.6c47.5-3.4,93.2-6.8,139.2-6.4c0.7,0,1.3,0,2,0c3.2,0.2,4.9-0.9,4.9-4.5
c-0.1-13,0-25.9,0-38.9c0-3.4-1.6-4.8-4.9-4.6c-1.7,0.1-3.3,0-5,0c-42.6,0-85.1,0-127.7,0c-1.7,0-3.3,0.1-5,0
c-2.8-0.1-4.4,1.1-4.4,4.1C1187.9,1016.8,1188.1,1033.4,1188.4,1050.6z M2010.7,1230.1c0.9-7.3-0.9-13.8-1.5-20.3
c-6.8-66-10.3-132.2-10.7-198.6c-0.1-16.1-0.3-16.3-16.8-14.7c-2.9,0.3-4.2,1.1-4.1,4.1c0.1,3,0,6,0,9
c-0.3,41.3-2.1,82.6-6.1,123.8c-2,20.2-4.3,40.4-8.2,60.4c-0.9,4.8-0.2,7.8,4.5,10.4c10.2,5.7,19.9,12.2,29.8,18.3
C2001.8,1225,2005.6,1228.2,2010.7,1230.1z M2011.2,752.5c-2.7-0.8-3.6,0.3-4.7,1c-13.3,8.3-26.4,16.7-39.8,24.7
c-3.6,2.1-4.2,4.5-3.5,8.2c1.4,7.8,2.8,15.7,3.9,23.6c7.4,53.8,10.1,107.9,10.6,162.2c0.1,14.1-2.6,12.8,13.7,12.8
c7,0,7-0.2,7.1-7.3c0.4-27.3,0.6-54.6,1.8-81.8c1.4-30.9,3.1-61.8,6.1-92.6C2008,786.3,2009.6,769.5,2011.2,752.5z M662.1,1231.5
c3.7-2.4,5.6-3.7,7.5-4.9c11.6-7.7,22.9-16.2,35.5-22.2c5.3-2.5,6.2-5.5,5.1-11c-4.6-22.5-7-45.4-9-68.3
c-3.5-39.9-5-79.8-5.4-119.8c-0.1-8.6-0.2-8.6-8.8-8.7c-1.3,0-2.7,0-4,0c-7.9,0.1-8,0.1-7.9,8.4c0,24.7-0.6,49.3-1.6,74
c-1.3,32-3,63.9-6,95.8C665.8,1192.9,664.1,1211.1,662.1,1231.5z M662.9,752.1c-1,1.4-0.4,3-0.3,4.6c2,20.5,4.6,41.1,6,61.7
c3.8,52.5,6.5,105.1,6.4,157.8c0,8.8,0.1,8.9,8.5,8.9c14.5-0.1,12.1,1.3,12.3-12.1c1.1-62.7,3.5-125.2,14.8-187
c0.8-4.1-0.9-6-4.2-7.7c-13.4-6.8-25.6-15.5-38-23.9C666.8,753.1,665.5,751.1,662.9,752.1z M2044.3,728.6c-5.4,1.1-7.1,3.9-7.7,8.3
c-1.7,14.9-4,29.7-5.6,44.6c-4.6,42.1-7.2,84.2-8.9,126.5c-1,25.3-1.3,50.5-1.3,76.8c6.5,0,12.4-0.1,18.3,0
c3.8,0.1,5.5-1.4,5.2-5.3c-0.2-1.7,0-3.3,0-5c0-78.9,0-157.9,0-236.8C2044.3,734.9,2044.3,732,2044.3,728.6z M2044.3,1253
c0-3.7,0-6.4,0-9c0-78.6,0-157.2,0-235.8c0-13,1.3-11.5-11.9-11.5c-13,0-11.5-1.8-11.5,11.8c0,59.3,3.1,118.5,8.7,177.6
c1.9,19.5,4,39.1,7,58.5C2037.3,1248.9,2038.8,1251.8,2044.3,1253z M629.2,728.7c0,4.3,0,8,0,11.6c0,76.5,0,153.1,0,229.6
c0,15.8,0,15.8,16,15.3c7.6-0.2,7.6-0.2,7.7-7.7c0.1-3.3,0-6.7,0-10c-0.8-55.6-3.1-111.1-8.5-166.4c-2-21.2-5-42.3-7.3-63.4
C636.6,733.3,635,730.3,629.2,728.7z M629.2,1253.2c5.3-2.1,7.4-4.5,7.9-9.2c1.6-14.9,4-29.7,5.5-44.6
c6.5-62.4,9.5-124.9,10.3-187.6c0.2-15.6,0.1-15.6-15-15.6c-9,0-8.8-0.6-8.7,8.6c0,2,0,4,0,6c0,75.7,0,151.3,0,227
C629.2,1242.5,629.2,1247.2,629.2,1253.2z M2060,697.5c19-15.9,36.7-29.3,52.3-45.3c3.5-3.5,3.7-6.7,2.4-10.7
c-2.8-8.9-5.4-17.8-8.7-26.5c-3.5-9.3-7.4-18.5-13.7-26.4c-5.3-6.6-8-6.6-13.4-0.4c-1.3,1.5-2.7,3-3.6,4.7
c-8.9,15.7-17.3,31.5-15.3,50.5c0.4,3.6,0.1,7.3,0.1,11C2060,668,2060,681.5,2060,697.5z M613.2,1283.1
c-20.8,13.6-36.9,29.9-53.8,45.4c-2.8,2.6-2.3,5.1-1.5,8.1c4.1,14.7,8.6,29.3,15.3,43.2c2.6,5.4,5.4,10.7,9.8,15
c4.3,4.2,6.2,4.6,10,0.4c10.7-11.9,18.4-26.3,19.7-41.7C614.7,1330.6,613.2,1307.6,613.2,1283.1z M2060,1284.4
c0,19.5,0.7,36.8-0.2,54c-1.1,19.1,6.3,35.2,15.8,50.5c7.4,12.1,12.1,11.7,19.8-0.6c9.7-15.4,14.6-32.7,19.7-50
c1.1-3.7,0-6.2-2.6-8.8C2096.8,1313.6,2079.3,1299.9,2060,1284.4z M611.9,698.2c2.1-3.3,1.3-5.9,1.3-8.4c0.1-16-0.8-32,0.3-47.9
c1.2-18.4-6.2-33.8-15.1-48.6c-7.6-12.7-12.6-12.4-20.4,0.3c-10.1,16.3-15.1,34.6-20.4,52.8c-1,3.6,0.6,5.5,3,7.7
c8.8,8.1,17.3,16.7,26.3,24.5C594.8,685.4,603.3,691.5,611.9,698.2z M1788.2,864c-12.5,3.2-12.6,3.2-13.9,13.8
c-0.9,7.6-1.6,15.2-2.1,22.8c-1.8,26.2-2.9,52.5-3.2,78.8c0,4,0.8,6.3,5.4,6c4.6-0.3,9.3-0.2,14,0c5.1,0.1,6.9-2.1,6.7-7.3
c-0.5-11.3-0.2-22.6-0.7-33.9C1793.2,917.5,1792.3,890.9,1788.2,864z M885.3,1118.4c12.8-3,12.5-3,13.8-13.9
c3.5-29.5,4.6-59.1,5-88.8c0.3-23.5,5.2-18.7-19.8-19.4c-4.4-0.1-5.7,1.5-5.7,5.8C878.8,1040.8,880.3,1079.4,885.3,1118.4z
M885.7,863.5c-0.2,0.9-0.6,1.9-0.7,2.9c-4.3,33.7-6,67.5-6.1,101.4c0,5.5-2.5,12.6,1.3,16.2c3.8,3.6,10.8,1.1,16.3,1.2
c7.9,0.1,8.1-0.1,7.9-8.4c-0.6-33.2-1.5-66.4-5.3-99.4C897.9,865.6,898,865.6,885.7,863.5z M1788.4,1117.5
c4.4-34.4,5.9-68.6,6.5-102.9c0.1-5.6,2.4-12.7-0.9-16.5c-3.9-4.3-11.3-1.4-17.2-1.6c-7.6-0.2-7.7,0.1-7.7,7.6
c0,24.7,1.3,49.3,2.8,73.9c0.6,10.3,2.2,20.5,2.9,30.8c0.3,3.6,1.7,5.5,4.9,6.5C1782.2,1116,1784.4,1117.7,1788.4,1117.5z
M1481.1,1078.1c0-2.7,0.1-4.4,0-6c-0.3-7.8-0.3-8.1-8.2-8.7c-22.9-1.7-45.8-3.3-68.7-4.7c-19.9-1.2-39.9-0.9-59.9-1
c-4.6,0-5.5,2-5.4,6c0.1,3.7,0.3,6.7,5.2,6.2c1.3-0.1,2.7,0,4,0c34.3-0.3,68.4,1.9,102.5,5.6
C1460.6,1076.6,1470.4,1078.5,1481.1,1078.1z M1480,903c-15.3,1.6-30.8,3.6-46.3,4.8c-29.5,2.4-59.1,4.7-88.8,3.9
c-5.1-0.1-6.1,2-6,6.5c0.1,4.1,1.1,6.3,5.6,5.8c1.3-0.2,2.7,0,4,0c42.3,0.3,84.5-2.6,126.7-5.9c4.2-0.3,6.2-1.6,5.8-5.9
C1480.7,909.3,1482,906.3,1480,903z M1192.2,1078.2c3.1-0.2,5.4-0.3,7.6-0.5c37-4.6,74-7.6,111.3-7.8c6,0,12-0.1,18,0
c3.7,0.1,5.3-1.1,5.4-5.1c0.1-6.5,0-6.9-7.3-7c-40.6-0.8-81.1,1.7-121.6,4.6C1193.1,1063.3,1193.2,1063.6,1192.2,1078.2z
M1192.6,903.2c0.2,14.4,0.2,14.4,12.8,16c3,0.4,6,0.6,8.9,0.8c19.3,1.1,38.5,2,57.8,3.2c18.6,1.2,37.2,0.8,55.9,0.8
c6.4,0,6.3-0.5,6.6-6.8c0.2-5.8-3.3-5.5-7.1-5.4c-28.3,0.8-56.5-1.5-84.7-3.6C1226.3,907,1209.8,905.4,1192.6,903.2z M653.6,694.6
c2,12.3,3.8,22.4,5.1,32.6c0.5,3.7,1.7,6.1,4.8,8c14.2,8.8,27.5,18.8,42.2,26.6c2.7,1.4,6.2,6.3,8.9,2.4c2.7-3.7-2.6-6.2-5-8.5
c-19-17.3-36.7-35.8-51.8-56.7C657,698.1,656,697.3,653.6,694.6z M654,1284.6c0.8,0,1.3,0.1,1.5,0c1.4-1.4,2.9-2.7,4-4.3
c14.6-20.8,32.8-38.3,51.2-55.5c1.2-1.1,2.6-2.1,3.6-3.4c1.4-1.7,1.5-3.6-0.2-5.7c-2.9,0.3-5.3,2.1-7.8,3.6
c-10.2,6.3-19.8,13.5-30.5,18.8c-12.9,6.4-19.3,15.7-19.2,30.2C656.6,1273.7,654.9,1279.1,654,1284.6z M2019.1,698.8
c-0.5-0.3-1-0.6-1.5-0.9c-1,1.2-2,2.4-3,3.6c-6.8,9-13,18.4-21.4,26.1c-11,10.1-20.5,21.7-32.4,30.8c-1.9,1.5-3.6,3.2-2.1,5.9
c2,3.5,4.1,0.5,5.7-0.4c15.4-9.7,30.8-19.6,46.1-29.5c2.3-1.5,4-3.1,4.4-6.2C2016.2,718.5,2017.7,708.6,2019.1,698.8z
M2017.1,1283.9c0.7-0.2,1.4-0.5,2.1-0.7c-1.5-10.1-3-20.2-4.4-30.4c-0.4-2.6-1.9-4.1-3.9-5.4c-15.6-10-31.3-20.1-46.9-30
c-1.5-1-3.3-2.8-5-0.5c-1.7,2.4-0.3,4.2,1.7,5.9c4,3.4,8.1,6.9,11.8,10.6C1988.4,1249.3,2005.3,1264.3,2017.1,1283.9z M1044.6,1034
c0.2,0,0.3,0,0.5,0c0-11-0.1-21.9,0.1-32.9c0.1-4.2-2-4.8-5.5-4.9c-4.5-0.1-4.6,2.6-4.6,5.9c0.1,22.3,1.2,44.5,2.4,66.8
c0.1,1,0.2,2,0.4,3c0.2,1.2,1.2,1.8,2.1,1.7c0.8-0.1,1.9-1,2.2-1.7c0.5-1.5,0.7-3.2,0.8-4.9C1043.5,1055.9,1044,1045,1044.6,1034z
M1636.4,1033.8c0.7,0,1.4,0,2.1,0c0-10.6,0-21.2,0-31.8c0-2.7,0.2-5.3-3.6-5.8c-4.7-0.6-6,0.4-6,6.1c-0.1,21.9,0.2,43.7,2.1,65.6
c0.1,1.3,0.1,2.7,0.6,3.9c0.3,0.7,1.5,1.6,2.2,1.5c0.7,0,1.7-1.1,1.9-1.8c0.5-1.9,0.7-3.9,0.7-5.9
C1636.4,1055,1636.4,1044.4,1636.4,1033.8z M1638.5,948.1c-0.2,0-0.5,0-0.7,0c-0.5-11.6-1-23.3-1.6-34.9c-0.1-1.9,0.2-4.8-2.3-4.9
c-3.3-0.1-2.8,3-3,5.2c-0.3,3-0.4,6-0.7,9c-1.5,19-1.4,37.9-1.4,56.9c0,5.3,2,6.8,6.4,5.8c3.8-0.8,3.3-3.6,3.3-6.2
C1638.5,968.8,1638.5,958.4,1638.5,948.1z M1045.1,948c-0.1,0-0.3,0-0.4,0c-0.6-11.9-1.2-23.9-1.9-35.8c-0.1-1.9-0.4-4.3-3.2-3.9
c-2.2,0.3-1.9,2.4-2,4c-1.3,22.5-2.3,45.1-2.5,67.7c0,3.7,1.2,5.4,5.1,5.4c4.1,0.1,5-1.9,5-5.5C1045,969.3,1045.1,958.7,1045.1,948
z M741.9,740.1c-4.9,5.5-6.3,11-8.3,16.2c-0.8,2-0.8,4,1.2,5.3c6.2,4.1,11.6,9.4,19.7,13C751.6,762.1,748.8,750.9,741.9,740.1z
M1931.7,1241.3c11.7-16.6,11.5-18.2-3-28.3c-2.7-1.9-4.9-4.5-9.1-4.9C1921.9,1219.6,1924.7,1230.7,1931.7,1241.3z M1931.6,740
c-6.6,10.9-9.8,22-12.1,33.9c4-0.9,6.1-3.1,8.5-4.8C1943.5,758.3,1943.6,757.2,1931.6,740z M742.1,1241.3
c6.6-11.1,9.8-21.9,12.2-34.3c-8,3.6-13.2,8.8-19.2,12.6c-2,1.3-2.3,3.1-1.6,5.1C735.6,1230.2,736.7,1236.2,742.1,1241.3z
M597.6,1464.4c16.9-6.5,17.3-8.9,14-24.4C605.8,1447.2,601.7,1455,597.6,1464.4z M2061.5,1442.5c-5.1,14.3,5.6,17.3,12.5,22.8
C2072.1,1457,2066.2,1451,2061.5,1442.5z M1781.6,836.9c-1,5.2-3.6,8.5-1.8,12.6C1785.7,846.2,1785.7,846.2,1781.6,836.9z
M895.4,849.3c-1.5-4.5-1.6-7.9-3.8-11.1C887.9,846.3,887.9,846.3,895.4,849.3z M630.1,1284.6c-0.4-5.3,4.1-10.2,1.8-17
C627.4,1273.6,629.9,1279.2,630.1,1284.6z M629.9,697.1c0.2,5.4-3,11.4,2.4,16.6C633.8,707.4,630.4,702.5,629.9,697.1z
M2044.5,1284.5c-1-5.2,1.5-10.3-2.8-15C2040.3,1275.4,2042.4,1279.6,2044.5,1284.5z M2041.9,712.1c4.1-4.8,1.6-10,2.2-14.6
C2043,702.1,2039.8,706.2,2041.9,712.1z M892.2,1143.9c0.4-4.2,3.5-7.3,1.6-11.5C887.6,1136,887.6,1136,892.2,1143.9z
M1779.6,1132.2c-1,2.9-0.4,4.8,0.4,6.5c0.7,1.5,0.4,3.6,2.3,4.6C1785.7,1135.3,1785.7,1135.3,1779.6,1132.2z M729.5,774
c1.5,1.6,2.6,4,5.8,3.2C733.9,774.7,732.2,773.5,729.5,774z M737.9,778.4c-0.2,0.2-0.5,0.4-0.7,0.6c0.2,0.2,0.4,0.6,0.6,0.6
c0.2,0,0.5-0.3,0.7-0.5C738.4,778.8,738.1,778.6,737.9,778.4z"/>
<path class="st1" d="M534.9,1353.5c4.5,11.6,7.4,22.2,12.2,32c4.7,9.6,9.8,18.8,17.1,26.8c10.6,11.6,28.8,15.5,42.7,2.9
c1.4-1.3,3.1-2.4,5.5-4.2c1.4,7.2,1.7,13.2-2.5,19.1c-8.4,11.7-14.8,24.4-20.8,37.4c-1.7,3.7-3.7,6.7-7.3,9.1
c-52.7,33.7-105.3,67.6-158,101.4c-1.1,0.7-2.1,1.7-4.1,1.1c0-195.1,0-390.4,0-585.6c2.1-1.2,4.1-0.9,6-0.8
c21.6,0.5,43.3,1.1,64.9,1.3c5.3,0,6.7,1.8,6.7,6.9c0.5,40,1.4,79.9,3.9,119.8c2.4,38.2,5.6,76.4,10.6,114.3
c4.6,34.7,10.1,69.1,19.2,103c1.1,4,0.6,7.1-2.3,10c-0.9,0.9-1.7,2.1-2.6,3.1c-22.1,25.1-40.6,52.7-57.5,81.5
c-11.6,19.7-19.9,41-28.9,61.9c14.9-23.9,25.7-50.1,41.8-73.5C497.6,1397.6,514.9,1375.3,534.9,1353.5z"/>
<path class="st1" d="M534.6,627.2c-40-41.2-69.6-89-94.2-140.1c0.7,6.2,4,11.2,6.2,16.6c12.7,31.7,30.5,60.6,49.6,88.7
c9.3,13.7,20.4,26.3,31,39.1c3.8,4.5,5,8.7,3.4,14.4c-9.6,35.4-15,71.6-19.6,107.9c-9.4,74.5-12.5,149.3-13.7,224.3c0,1-0.1,2,0,3
c0.6,4.7-1.2,6.5-6.1,6.4c-8.3-0.2-16.7,0.2-25,0.4c-13.7,0.3-27.3,0.4-41,1c-4.8,0.2-6.8-1.2-6.2-6.1c0.3-2,0-4,0-6
c0-187.9,0-375.8,0-563.7c0-3.5,0-7.1,0-10.7c4.2-0.1,6.3,2.3,8.7,3.9c50.4,32.4,100.8,64.9,151.3,97.1c5,3.2,8.4,7.1,10.8,12.4
c5.1,11.2,10.4,22.4,17.8,32.3c4.9,6.6,6.7,13.5,5.1,21.9c-3.6-0.1-5.6-3.1-7.9-5c-12-9.8-28.1-7.7-38.9,2.8
c-12.6,12.2-19,27.9-25.2,43.7C538.8,616.5,538.2,622.2,534.6,627.2z"/>
<path class="st1" d="M2060.9,1411.1c7.4,4.3,12.3,10.3,20.6,11.4c9.8,1.4,17.7-1.4,24.8-7.5c8.5-7.3,13.9-16.9,18.8-26.8
c5.1-10.4,8.6-21.5,12.7-32.5c24.7,24.7,73.3,78.9,107.2,169.7c-1.1-8.8-3.5-17.2-6.8-25.4c-13.2-33.9-29.3-66.4-49.3-96.8
c-12.8-19.5-27.4-37.7-42.4-55.5c-3.4-4-4.8-7.1-3.5-12.3c12.8-48.8,19.1-98.6,24-148.7c5.9-59.7,8.4-119.6,8.9-179.7
c0-1.3,0-2.7,0.1-4c0.2-7,0.2-7.1,7-7.4c20-0.7,40-1.3,60-1.9c2.3-0.1,4.7,0,7-0.1c3.5-0.2,4.8,1.4,4.6,4.8c-0.1,2,0,4,0,6
c0,188.3,0,376.6,0,565c0,3.2,0,6.4,0,9.8c-4.5-0.2-6.9-2.8-9.5-4.5c-47.7-30.6-95.2-61.4-143-91.7c-9-5.7-16-11.6-19.9-22.4
c-3.7-10.1-11.2-18.9-17.1-28.2C2060.8,1426.2,2058.6,1419.6,2060.9,1411.1z"/>
<path class="st1" d="M2244.8,454.3c-23,66.2-59.6,121.8-106.6,172.4c-4.5-11.6-7.7-22.6-12.8-32.9c-4-8.1-8.4-15.8-14.4-22.7
c-12.6-14.4-29.9-17.7-45.6-3.8c-1.1,1-2,2.7-4.6,2.6c-1.4-5.5-1.1-10.5,2-15.7c8.8-14.9,18.9-29,25.8-45c1-2.3,2.7-3.7,4.7-4.9
c52.7-33.9,105.3-67.7,158-101.5c0.5-0.3,1.3-0.2,2.1-0.4c2.3,2.1,1.2,4.9,1.2,7.3c0.1,189.6,0.1,379.3,0.1,568.9
c0,1.7-0.1,3.3,0,5c0.2,3.6-1.7,4.6-4.9,4.5c-5-0.2-10-0.3-15-0.5c-17-0.5-34-1-51-1.4c-7.4-0.2-7.5-0.2-7.6-7.8
c-0.5-35.3-1.1-70.6-3.1-105.9c-2.6-45.9-6.4-91.7-12.8-137.3c-4.2-30.3-9.4-60.5-17.3-90.2c-1.1-4-0.2-6.6,2.5-9.8
c27.4-31.6,51.5-65.6,70.4-102.8c11.3-22.2,21.7-44.9,28.2-69.2C2244.6,460.9,2245.4,458.8,2244.8,454.3z"/>
<path class="st1" d="M553.7,662.8c3.8,0.2,5.2,2.8,6.9,4.6c14.5,15.2,30.3,29,46.9,41.9c4.1,3.2,5.8,6.4,5.8,11.7
c-0.2,85.6-0.1,171.2-0.1,256.8c0,7.5-0.1,7.7-7.6,7.8c-25,0.4-50,0.6-74.9,1.1c-4.3,0.1-5.9-1.2-5.7-5.6c0.4-13,0.2-26,0.5-39
c1.5-60.3,4.7-120.5,11.8-180.4c3.7-31.7,8.3-63.3,15.4-94.5C553,665.6,553.4,664.1,553.7,662.8z"/>
<path class="st1" d="M553.6,1319.7c-22.4-107.4-27.4-215.3-28.7-323.3c3.3-2.5,6.4-1.5,9.3-1.5c23.3,0.3,46.7,0.7,70,1.1
c9.5,0.1,9.2-1.2,9.2,9.2c0,72.3,0,144.7,0,217c0,13.3-0.1,26.7,0.1,40c0,3.6-0.7,6.2-3.8,8.6c-18.7,14.5-36.3,30.3-52.7,47.4
C556.3,1318.5,555.6,1318.7,553.6,1319.7z"/>
<path class="st1" d="M2119.8,1319.2c-6.7-6.5-12.1-12.3-18-17.5c-11.8-10.2-23.8-20.2-36-30.1c-4.2-3.4-5.9-7.1-5.9-12.7
c0.2-84.6,0.1-169.1,0.1-253.7c0-8.4,0.1-8.5,8-8.5c24.3-0.2,48.6-0.4,72.9-0.5c7.5,0,7.7,0,7.6,7.7c-0.3,45.6-1.9,91.2-5.1,136.7
c-2.7,39.9-6.7,79.6-12.5,119.1C2128.1,1279,2124.6,1298.3,2119.8,1319.2z"/>
<path class="st1" d="M2120.4,664.4c12.3,57.5,18.7,115.7,22.8,174.2c3.1,44.9,4.7,89.8,5.2,134.8c0,2.3-0.1,4.7,0.1,7
c0.3,4-1.7,5.2-5.4,5.1c-6.7-0.2-13.3-0.3-20-0.3c-18.3-0.1-36.7-0.2-55-0.4c-8-0.1-8.2-0.1-8.2-8.2c0-52.3,0-104.6,0-157
c0-32,0.2-64-0.2-96c-0.1-6.3,2-10.3,6.8-14.3c17.1-14.3,35.1-27.7,50.2-44.2C2117.5,664.3,2118.3,663.8,2120.4,664.4z"/>
<path class="st1" d="M878.7,1138.5c2,4.9,3.2,9.1,5.3,12.8c4,7.2,10.5,7.1,14.6,0c3-5.3,4.3-11.2,5.2-17.1c0.8-4.7,3.2-7.1,7.8-8.9
c37.6-14.8,76.3-25.6,115.5-34.7c5-1.1,6.6,0.4,7.3,4.9c0.6,3.5,0.7,8.4,5.3,8.5c5.3,0.1,5.5-5.2,5.8-9c0.5-7.1,3.9-10,10.9-11.2
c40-7,79.8-14.9,120.4-18c2.4-0.2,5.9-2.1,6.9,1.5c1,3.7,1.8,7.8,0.4,11.7c-0.7,1.8-2.9,1.2-4.4,1.5c-9.5,1.6-19.1,2.8-28.6,4.6
c-39.6,7.4-78.6,17-117,29.2c-59.9,19-116.5,45.2-170.8,76.4c-41.6,24-80.2,52.3-117.2,83.1c-39.4,32.8-74.3,69.6-105.4,110.2
c-3,3.9-6.2,7.7-9.4,11.6c-0.1,0.2-0.6,0.2-2.1,0.5c0.2-7-1.1-13.6,1.5-20.3c7.6-19.2,12.9-39.2,16.7-59.5
c1.8-9.6,6.2-17.5,12-24.8c17.2-21.7,36.5-41.5,58.8-58.9c3.4,2.2,3.5,6.6,5.4,9.8c4,7,7.8,14.4,17.4,14.4
c9.6,0,13.8-7.2,17.3-14.4c6.5-13.3,9.4-27.7,12.2-42c0.8-3.9,2.2-6.5,5.7-8.6C808.9,1171.4,842.7,1153.5,878.7,1138.5z"/>
<path class="st1" d="M1032.8,891.9c-42-9-82.2-20.4-121.3-35.7c-4.8-1.9-6.9-4.4-7.7-9c-0.9-5.6-2.3-11.1-4.8-16.2
c-1.5-3.2-3.7-5.9-7.3-6.1c-4.1-0.2-6.8,2.5-8.3,6.1c-1.6,3.6-2.8,7.4-4.7,12.2c-11.4-5.2-22.6-10-33.5-15.2
c-23.1-11.1-45.4-23.7-67.3-37.1c-4.3-2.6-6.6-5.5-7.5-10.7c-2.1-11.4-4.6-22.8-8.9-33.7c-2.1-5.3-4.2-10.5-7.8-15
c-7.6-9.4-17.5-9.4-25.3,0.1c-4.2,5.2-6,11.7-9.8,18.2c-11.1-9.9-22.3-18.8-32.2-28.9c-10.2-10.4-19.5-21.8-28.7-33.3
c-5.6-7-8.9-15.3-10.6-24.4c-3.7-19.3-8.6-38.3-16-56.5c-2.9-7.1-1.4-14.1-0.5-21.8c3.9,4.8,7.9,9.4,11.6,14.3
c23.7,32,51,60.7,80.1,87.7c19,17.6,39.4,33.9,59.9,49.7c45,34.7,93.8,63.3,144.7,87.9c39.2,18.9,79.8,34.7,121.6,47.1
c41.2,12.2,82.8,22.3,125.4,28.2c1,0.1,2,0.2,3,0.4c7.3,1.2,10.5,7.7,7,14.2c-1.3,2.3-3.4,1.7-5.3,1.6c-8-0.7-15.9-1.5-23.8-2.4
c-32.8-3.7-65.1-10.2-97.5-15.7c-7.1-1.2-11.5-3.9-11.7-11.6c0-1.3-0.4-2.6-0.8-3.9c-0.8-2.6-1.9-5.3-5.3-4.9
c-2.7,0.3-3.7,2.6-4.4,4.9C1034.4,885.3,1033.7,888.2,1032.8,891.9z"/>
<path class="st1" d="M1953.8,1233.7c3.8,1,5.8,3.4,8,5.3c21,19.1,40.7,39.4,57.6,62.2c3,4.1,4.9,8.4,6,13.5
c4.5,20.5,9.3,40.9,17.1,60.4c1.3,3.1,2.4,6.3,1.8,9.7c-0.6,3.6,2.4,8.9-2.1,10.4c-2.5,0.8-5.4-3.9-7.3-6.7
c-35.7-51.5-81-93.6-129.4-132.6c-40.8-32.9-84.7-61.2-131-85.6c-39.5-20.8-80.6-37.9-123-52c-45.9-15.2-92.6-26.8-140.3-34.3
c-6.2-1-12.4-2.7-18.7-3.1c-6.5-0.4-5.5-4.5-5.2-8.4c0.4-5.4,2.1-7.1,7.5-6.6c8.3,0.7,16.5,1.9,24.8,2.7
c33.2,3.4,65.8,10.1,98.6,15.7c6.3,1.1,9.2,3.4,9.6,10c0.2,3.6,0.3,9.3,5.3,9.6c6,0.4,5.5-5.6,6.3-9.5c0.7-3.9,2.4-5,6.2-4
c39.4,9.7,78.9,19.2,116.7,34.3c4.5,1.8,6.8,4.2,7.5,9.1c0.9,5.9,2.2,11.8,5.1,17.2c1.7,3.1,3.8,5.6,7.6,5.6c3.8,0,6-2.5,7.6-5.6
c0.9-1.8,1.7-3.6,2.3-5.5c2.8-7.7,2.9-7.9,9.9-4.7c32.6,15.3,64.6,31.4,95,50.8c3.4,2.2,5.1,4.7,5.8,8.6
c2.1,12.1,4.9,24.1,9.4,35.7c1.9,5,4.1,9.8,7.5,14c7.5,9.4,17.7,9.4,25.3,0C1948.9,1245.3,1950.9,1239.5,1953.8,1233.7z"/>
<path class="st1" d="M2043.6,585.1c0.9,7.5,1.7,14.1-0.8,20.4c-8,20.2-13.1,41.2-17.6,62.4c-0.9,4.4-2.6,8.3-5.2,11.7
c-17.8,23.8-38.4,45.2-60.6,64.9c-1.5,1.3-3.2,2.3-5.1,3.8c-2.9-5.3-5.1-10.4-8.2-14.9c-8-11.8-19.3-11.8-27.5-0.1
c-7.4,10.6-11.4,23-12.8,35.4c-1.7,14.6-9.4,22.2-21.3,29.3c-25.5,15.2-52.1,28.1-78.7,41.2c-2.4,1.2-5,2-7.3,3.1
c-2.7,1.3-4.3,0.7-5.2-2.2c-1-3.2-1.9-6.4-3.4-9.3c-4.2-7.7-10.8-7.7-15.1-0.1c-2.5,4.4-3.9,9.3-4.5,14.2c-0.8,7-4.2,10.7-11,13.3
c-27.5,10.2-55.4,18.9-83.9,25.7c-10,2.4-20.1,4.8-30,7.5c-3.8,1-5.4,0-6.1-3.9c-0.7-4-0.2-9.9-6.2-9.6c-5.1,0.2-5.3,6-5.4,9.5
c-0.1,7.5-4.3,9-10.5,10.2c-39.6,7.4-79.4,13.9-119.5,18.1c-2.8,0.3-6.7,2.3-8.1-1.3c-1.4-3.6-1.7-7.9-0.3-11.7
c0.9-2.4,3.9-2,6.2-2.4c13.8-2.4,27.6-4.4,41.3-7c42.1-8.2,83.5-19.3,124.2-33.1c57.6-19.6,112.2-45.7,163.9-77.3
c38.1-23.2,73.8-50.1,107.6-79.4c37.9-32.8,72.6-68.6,102.5-109C2037.5,591.3,2039.4,587.8,2043.6,585.1z"/>
<path class="st1" d="M777.2,1163.8c-0.9-7.9,1.2-15.4,2-23c4.4-45.1,6.3-90.3,6.6-135.6c0.1-8.6,0.2-8.7,8.8-8.7
c22-0.1,44-0.1,65.9,0c8.3,0,8.4,0.1,8.5,7.9c0.2,37.7,2,75.2,6.5,112.6c0.5,3.8,0.3,6-3.9,7.5c-31.7,11.3-62.2,25.3-92.6,39.6
C778.8,1164.3,778.4,1164.1,777.2,1163.8z"/>
<path class="st1" d="M776.8,817.8c3.1-0.6,4.7,0.6,6.5,1.4c28.2,13.5,56.6,26.5,86.2,36.8c5.4,1.9,6.7,4.3,6,10.1
c-4.5,36.7-6.3,73.6-6.4,110.6c0,8.1,0.7,8.3-8.1,8.3c-22.6-0.1-45.3,0-67.9-0.1c-7,0-7.1-0.2-7.2-7.3c-0.4-28.6-0.9-57.3-2.7-85.9
c-1.3-21.3-2.9-42.5-5.5-63.7C777.2,824.9,777.1,821.6,776.8,817.8z"/>
<path class="st1" d="M1896.4,1163.9c-3.3,0.5-5.7-1.5-8.4-2.8c-27.2-13.3-54.8-25.7-83.3-36.1c-7.4-2.7-7.2-2.9-6.3-10.6
c4.4-35.8,5.3-71.8,6.4-107.7c0.1-1.7,0.1-3.3,0-5c-0.3-3.8,1.3-5.4,5.2-5.3c6.3,0.2,12.7,0.1,19,0.1c16.7,0,33.3,0,50,0.1
c8.4,0,8.4,0.1,8.6,8.8c0.8,30.3,1.2,60.6,3.3,90.9c1.4,19.9,2.9,39.9,5.3,59.7C1896.5,1158.6,1897.4,1161.2,1896.4,1163.9z"/>
<path class="st1" d="M1896.8,817.7c-1.3,13.9-2.7,27.7-3.9,41.6c-3.5,38.5-4.5,77-5.2,115.6c0,1.7-0.2,3.3-0.1,5
c0.4,4-1.5,5.5-5.2,5.1c-1-0.1-2,0-3,0c-22.6,0-45.3-0.1-67.9,0.2c-5,0-6.8-1.2-6.8-6.6c-0.4-28.3-1.2-56.5-3.6-84.7
c-0.8-8.9-1.6-17.9-2.7-26.8c-0.9-7.6-1.1-7.8,6.5-10.6c23-8.7,45.8-17.8,67.9-28.7c6-2.9,12-5.7,18-8.6
C1892.5,818.3,1894.3,817.4,1896.8,817.7z"/>
<path class="st1" d="M908.3,1109.6c-1-4.4,0.4-8.6,0.8-12.8c2.5-30.9,4-61.8,4-92.9c0-7.4,0.2-7.5,7.9-7.5
c33.3-0.1,66.7-0.2,100-0.2c7.8,0,8.1,0.1,8,7.4c-0.3,22,1,44,2.3,65.9c0.3,4.5-1,6.5-5.5,7.4c-38.7,8.5-76.4,20.9-114.2,32.5
C910.7,1109.8,909.7,1109.6,908.3,1109.6z"/>
<path class="st1" d="M1765.3,1110.4c-9.6-3-18.1-5.6-26.6-8.4c-29.1-9.6-58.7-17.4-88.5-24.5c-8.9-2.1-8.1-1.4-7.6-10.4
c1.2-21,1.6-42,2.3-62.9c0.2-7.7,0.2-7.9,7.5-7.9c33.3,0,66.7,0.1,100,0.2c7.8,0,8.2,0.3,8.2,7.7c0.2,34,1.7,68,4.8,101.8
C1765.4,1106.9,1765.3,1107.9,1765.3,1110.4z"/>
<path class="st1" d="M908.8,872.2c2.8-1.1,5,0.8,7.5,1.5c36.1,10.3,71.6,22.9,108.5,30.5c5.5,1.1,6.7,3.6,6.4,8.8
c-1.3,21.9-2.6,43.9-2.1,65.9c0.1,4.7-1.2,6.5-6.1,6.4c-34.7-0.2-69.3-0.2-104-0.1c-4.1,0-6.1-1-6-5.6c0.4-35.6-2-71.2-4.8-106.7
C908.2,872.8,908.6,872.5,908.8,872.2z"/>
<path class="st1" d="M1765.6,872.4c-0.9,10.7-1.8,20.9-2.5,31.2c-1.7,24.6-2.4,49.2-2.6,73.8c-0.1,7.7-0.2,7.8-7.7,7.8
c-33.6,0.1-67.2,0.1-100.8,0.1c-6.9,0-6.9-0.2-7.1-7.1c-0.8-21.9-1.3-43.9-2.6-65.8c-0.3-5.4,1.8-7,6.2-8
c14.5-3.5,29-7.1,43.5-10.8c22.5-5.7,44.5-13.5,66.7-20.3C1760.5,872.7,1762.3,871.8,1765.6,872.4z"/>
<path class="st1" d="M725.8,1192.5c-5.9-34.3-8.8-67.4-11-100.6c-1.8-27.6-2.2-55.2-3-82.8c-0.1-2.3,0.1-4.7-0.1-7
c-0.4-4.4,1.6-5.8,5.9-5.7c9,0.2,18,0,27,0c6.7,0,13.3-0.1,20,0c7.2,0.1,7.4,0.2,7.3,7.1c-0.5,23.6-1,47.2-2,70.8
c-1.2,27.2-3.3,54.4-6.7,81.5c-0.2,1.3-0.3,2.6-0.4,4c-0.4,9.2-3.4,16-13,19.7C741.7,1182.7,734.5,1187.7,725.8,1192.5z"/>
<path class="st1" d="M1946.2,1191.7c-10.1-5.6-20.4-11.5-30.8-17.2c-3.3-1.8-3.1-5-3.5-7.8c-1.4-10.2-2.4-20.5-3.5-30.7
c-4.7-43.8-6-87.7-6.7-131.6c-0.1-7.7,0.1-7.8,7.7-7.8c15.3-0.1,30.6,0.1,46-0.1c4.6-0.1,6.5,1.1,6.4,6.1
c-0.5,34-1.2,67.9-3.7,101.8c-2,26.9-4.5,53.7-8.9,80.4C1948.7,1187,1949.1,1189.5,1946.2,1191.7z"/>
<path class="st1" d="M1947.7,789.1c5.8,32.7,8.6,63.8,10.8,95c2.1,30.6,2.8,61.2,3.1,91.9c0.1,9.6,0.3,8.9-8.3,8.9
c-15,0-30-0.1-45,0.2c-5.3,0.1-6.9-1.8-6.7-7c0.9-25.3,1.3-50.6,2.5-75.9c1.3-27.3,3.9-54.4,6.9-81.6c0.9-8.3,3.3-14.2,11.5-17.8
C1930.8,799.2,1938.6,794.1,1947.7,789.1z"/>
<path class="st1" d="M726,789.2c11.3,6.2,21.7,11.9,32.2,17.5c2.9,1.6,2.8,4.2,3.2,6.8c2.7,15.8,4.1,31.7,5.5,47.6
c3.3,37.2,4.5,74.5,5,111.8c0,2.3,0,4.7,0.1,7c0.1,3.1-0.8,5-4.4,5c-17-0.1-34-0.1-51-0.1c-3.3,0-4.8-1.5-4.7-4.8
c0.2-5.7,0-11.3,0.2-17c1-42.6,2.7-85.2,7.3-127.7C720.9,820.5,722.6,805.7,726,789.2z"/>
<path class="st1" d="M1625.1,910.5c-0.6,9.4-1.4,18-1.7,26.5c-0.4,13-0.4,26-0.6,38.9c-0.1,9.3-0.2,9.4-9.2,9.4
c-30.6,0.1-61.3,0.1-91.9,0.1c-8.3,0-16.7-0.1-25,0.1c-4.2,0.1-5.9-1.3-5.9-5.7c0-15-0.1-30-0.7-44.9c-0.2-4.8,2.5-5.5,6-6
c10.9-1.3,21.8-3.6,32.7-3.7c23.9-0.2,46.7-6.5,70-10C1607.1,913.9,1615.2,912.2,1625.1,910.5z"/>
<path class="st1" d="M1625.2,1071.1c-23.5-4.2-45.7-8.1-67.9-12.1c-7.9-1.4-15.8-2.4-23.8-2.5c-12.7-0.1-25.1-2.6-37.7-4
c-4-0.4-5.8-1.6-5.6-6.1c0.5-14.6,0.6-29.3,0.6-43.9c0-4.5,1-6.5,6.1-6.5c39.9,0.2,79.9,0.2,119.8,0.1c5,0,6.1,2.1,6.1,6.6
C1622.9,1025,1623,1047.3,1625.2,1071.1z"/>
<path class="st1" d="M1049.3,1070.4c-1-5.9-0.1-11.8,0.3-17.7c0.9-16.3,1.4-32.6,1.4-48.9c0-7.3,0.2-7.5,6.8-7.5
c39.3-0.1,78.5-0.1,117.8-0.1c7,0,7.1,0.2,7.2,7.2c0.2,14,0,27.9,0.5,41.9c0.2,5.2-1.5,6.6-6.6,7c-16.9,1.3-33.8,2.7-50.6,5.2
c-23.3,3.4-46.7,6.8-69.7,12.3C1054.2,1070.3,1051.9,1071.4,1049.3,1070.4z"/>
<path class="st1" d="M1049.8,910.2c22.1,5.7,44.8,8.7,67.4,12.5c16.4,2.8,33.1,3.3,49.6,5.2c16.4,1.9,16.4,1.5,16.3,17.4
c-0.1,11-0.3,22-0.2,32.9c0.1,4.9-1,7.2-6.6,7.2c-39.9-0.2-79.9-0.2-119.8-0.1c-3.4,0-5.7-0.6-5.6-4.8c0.6-22.6-1.3-45.2-1.9-67.8
C1049,912,1049.4,911.4,1049.8,910.2z"/>
<path class="st1" d="M1260.5,985.4c-21.6,0-43.2,0-64.9,0c-8,0-8.2-0.1-8.1-8.4c0.2-13.6,0.6-27.3,0.8-40.9c0.1-4.2,1.4-5.6,6-5.1
c41.7,4.1,83.5,6.2,125.5,6.2c2.3,0,4.7,0,7,0c7.5,0.2,7.5,0.2,7.6,7.9c0.1,10.3,0.1,20.6,0.1,30.9c0,9.3,0,9.4-9.1,9.4
C1303.8,985.4,1282.2,985.4,1260.5,985.4z"/>
<path class="st1" d="M1485,931.5c0.3,16.2,0.5,32.7,0.8,49.3c0.1,3.4-1.7,4.8-4.9,4.6c-2-0.1-4,0-6,0c-41.6,0-83.2,0-124.9,0
c-2,0-4-0.1-6,0c-4,0.3-5.1-1.8-5-5.5c0.1-11.7,0-23.3,0.1-35c0.1-7.6,0.1-7.8,7.6-7.8c17.3,0.2,34.6-0.6,51.9-1.1
c26.3-0.7,52.5-3.1,78.7-5.1C1479.6,930.9,1481.8,930,1485,931.5z"/>
<path class="st1" d="M1485,1051.1c-35.4-3-69.9-5.3-104.5-6.4c-11.6-0.4-23.3-0.5-34.9-0.4c-4.4,0-6.7-0.8-6.5-6
c0.4-12,0.3-24,0-36c-0.1-4.8,1.6-6.3,6.3-6.2c45,0.1,89.9,0.1,134.9,0c3.5,0,5.8,0.7,5.7,4.8
C1485.5,1017.5,1485.3,1034.1,1485,1051.1z"/>
<path class="st1" d="M1188.4,1050.6c-0.3-17.2-0.5-33.7-0.8-50.3c0-2.9,1.6-4.1,4.4-4.1c1.7,0,3.3,0,5,0c42.6,0,85.1,0,127.7,0
c1.7,0,3.3,0.1,5,0c3.3-0.2,4.9,1.2,4.9,4.6c-0.1,13-0.1,25.9,0,38.9c0,3.6-1.7,4.7-4.9,4.5c-0.7,0-1.3,0-2,0
C1281.6,1043.7,1235.9,1047.2,1188.4,1050.6z"/>
<path class="st1" d="M2010.7,1230.1c-5.1-1.9-9-5.1-13.1-7.7c-9.9-6.1-19.6-12.6-29.8-18.3c-4.8-2.6-5.5-5.6-4.5-10.4
c3.9-20,6.2-40.1,8.2-60.4c4-41.2,5.8-82.4,6.1-123.8c0-3,0.1-6,0-9c-0.1-3,1.1-3.8,4.1-4.1c16.5-1.5,16.7-1.4,16.8,14.7
c0.4,66.4,3.9,132.5,10.7,198.6C2009.9,1216.3,2011.6,1222.7,2010.7,1230.1z"/>
<path class="st1" d="M2011.2,752.5c-1.6,16.9-3.2,33.8-4.8,50.6c-3,30.8-4.6,61.7-6.1,92.6c-1.3,27.3-1.5,54.5-1.8,81.8
c-0.1,7.1-0.2,7.3-7.1,7.3c-16.3,0-13.6,1.3-13.7-12.8c-0.5-54.3-3.2-108.4-10.6-162.2c-1.1-7.9-2.5-15.8-3.9-23.6
c-0.7-3.6-0.1-6,3.5-8.2c13.4-8,26.6-16.4,39.8-24.7C2007.6,752.8,2008.5,751.7,2011.2,752.5z"/>
<path class="st1" d="M662.1,1231.5c1.9-20.4,3.7-38.6,5.4-56.8c3-31.9,4.7-63.8,6-95.8c1-24.6,1.6-49.3,1.6-74
c0-8.3,0.1-8.3,7.9-8.4c1.3,0,2.7,0,4,0c8.6,0.1,8.7,0.1,8.8,8.7c0.4,40,1.8,80,5.4,119.8c2,22.9,4.4,45.8,9,68.3
c1.1,5.5,0.2,8.5-5.1,11c-12.7,6-23.9,14.5-35.5,22.2C667.7,1227.9,665.8,1229.1,662.1,1231.5z"/>
<path class="st1" d="M662.9,752.1c2.6-0.9,3.9,1,5.4,2.1c12.4,8.4,24.6,17.1,38,23.9c3.3,1.7,5,3.6,4.2,7.7
c-11.3,61.8-13.7,124.4-14.8,187c-0.2,13.5,2.2,12.1-12.3,12.1c-8.4,0-8.5-0.1-8.5-8.9c0.1-52.7-2.6-105.3-6.4-157.8
c-1.5-20.6-4-41.1-6-61.7C662.5,755.1,661.9,753.4,662.9,752.1z"/>
<path class="st1" d="M2044.3,728.6c0,3.4,0,6.3,0,9.2c0,78.9,0,157.9,0,236.8c0,1.7-0.1,3.3,0,5c0.4,3.9-1.3,5.4-5.2,5.3
c-5.9-0.2-11.9,0-18.3,0c0-26.3,0.3-51.6,1.3-76.8c1.7-42.3,4.4-84.5,8.9-126.5c1.6-14.9,3.9-29.7,5.6-44.6
C2037.2,732.4,2038.9,729.7,2044.3,728.6z"/>
<path class="st1" d="M2044.3,1253c-5.6-1.2-7-4.1-7.7-8.6c-3.1-19.4-5.2-38.9-7-58.5c-5.6-59-8.7-118.2-8.7-177.6
c0-13.6-1.4-11.7,11.5-11.8c13.3,0,11.9-1.5,11.9,11.5c0,78.6,0,157.2,0,235.8C2044.3,1246.6,2044.3,1249.3,2044.3,1253z"/>
<path class="st1" d="M629.2,728.7c5.8,1.6,7.4,4.6,7.9,9c2.3,21.2,5.3,42.2,7.3,63.4c5.3,55.3,7.7,110.8,8.5,166.4
c0,3.3,0.1,6.7,0,10c-0.1,7.5-0.1,7.5-7.7,7.7c-16,0.5-16,0.5-16-15.3c0-76.5,0-153.1,0-229.6C629.2,736.7,629.2,733,629.2,728.7z"
/>
<path class="st1" d="M629.2,1253.2c0-6,0-10.6,0-15.3c0-75.7,0-151.3,0-227c0-2,0-4,0-6c-0.1-9.2-0.3-8.7,8.7-8.6
c15.1,0,15.2,0,15,15.6c-0.7,62.7-3.8,125.2-10.3,187.6c-1.5,14.9-3.9,29.7-5.5,44.6C636.6,1248.7,634.5,1251.1,629.2,1253.2z"/>
<path class="st1" d="M2060,697.5c0-15.9,0-29.5,0-43c0-3.7,0.3-7.3-0.1-11c-2-19,6.4-34.8,15.3-50.5c1-1.7,2.3-3.2,3.6-4.7
c5.4-6.2,8.1-6.2,13.4,0.4c6.3,7.8,10.3,17.1,13.7,26.4c3.2,8.7,5.9,17.6,8.7,26.5c1.3,4,1,7.1-2.4,10.7
C2096.7,668.2,2079,681.6,2060,697.5z"/>
<path class="st1" d="M613.2,1283.1c0,24.5,1.5,47.6-0.5,70.3c-1.4,15.4-9,29.8-19.7,41.7c-3.8,4.2-5.7,3.9-10-0.4
c-4.3-4.3-7.2-9.6-9.8-15c-6.7-13.8-11.2-28.4-15.3-43.2c-0.8-3-1.3-5.6,1.5-8.1C576.3,1313,592.4,1296.7,613.2,1283.1z"/>
<path class="st1" d="M2060,1284.4c19.3,15.5,36.8,29.3,52.5,45.2c2.6,2.6,3.7,5.1,2.6,8.8c-5.1,17.2-10,34.5-19.7,50
c-7.7,12.2-12.4,12.7-19.8,0.6c-9.5-15.4-16.9-31.4-15.8-50.5C2060.7,1321.2,2060,1303.9,2060,1284.4z"/>
<path class="st1" d="M611.9,698.2c-8.5-6.7-17.1-12.8-25-19.7c-9.1-7.8-17.5-16.4-26.3-24.5c-2.3-2.2-4-4.1-3-7.7
c5.3-18.2,10.3-36.5,20.4-52.8c7.8-12.7,12.8-13,20.4-0.3c8.9,14.9,16.3,30.2,15.1,48.6c-1,15.9-0.2,31.9-0.3,47.9
C613.2,692.2,614,694.9,611.9,698.2z"/>
<path class="st1" d="M1788.2,864c4.1,27,4.9,53.5,6.1,80.1c0.5,11.3,0.2,22.6,0.7,33.9c0.2,5.2-1.6,7.4-6.7,7.3
c-4.7-0.1-9.3-0.3-14,0c-4.6,0.3-5.5-2-5.4-6c0.2-26.3,1.3-52.5,3.2-78.8c0.5-7.6,1.2-15.2,2.1-22.8
C1775.6,867.2,1775.7,867.2,1788.2,864z"/>
<path class="st1" d="M885.3,1118.4c-5.1-39.1-6.6-77.6-6.6-116.3c0-4.2,1.3-5.9,5.7-5.8c25,0.7,20.1-4.1,19.8,19.4
c-0.4,29.6-1.5,59.3-5,88.8C897.9,1115.4,898.1,1115.4,885.3,1118.4z"/>
<path class="st1" d="M885.7,863.5c12.4,2.1,12.2,2.1,13.5,13.8c3.8,33,4.6,66.2,5.3,99.4c0.2,8.3,0,8.5-7.9,8.4
c-5.6-0.1-12.6,2.4-16.3-1.2c-3.8-3.6-1.3-10.7-1.3-16.2c0.1-33.9,1.8-67.7,6.1-101.4C885.1,865.4,885.4,864.5,885.7,863.5z"/>
<path class="st1" d="M1788.4,1117.5c-4,0.3-6.2-1.5-8.7-2.2c-3.2-0.9-4.6-2.9-4.9-6.5c-0.7-10.3-2.2-20.5-2.9-30.8
c-1.5-24.6-2.8-49.2-2.8-73.9c0-7.5,0.1-7.8,7.7-7.6c5.9,0.2,13.3-2.7,17.2,1.6c3.4,3.7,1,10.9,0.9,16.5
C1794.3,1048.8,1792.8,1083.1,1788.4,1117.5z"/>
<path class="st1" d="M1481.1,1078.1c-10.7,0.3-20.5-1.5-30.3-2.6c-34.1-3.7-68.2-5.9-102.5-5.6c-1.3,0-2.7-0.1-4,0
c-4.9,0.5-5.1-2.5-5.2-6.2c-0.1-4,0.7-6,5.4-6c20,0.1,39.9-0.2,59.9,1c22.9,1.4,45.8,3,68.7,4.7c7.9,0.6,7.8,0.9,8.2,8.7
C1481.2,1073.7,1481.1,1075.4,1481.1,1078.1z"/>
<path class="st1" d="M1480,903c2,3.3,0.7,6.3,1,9.2c0.4,4.3-1.6,5.6-5.8,5.9c-42.2,3.3-84.3,6.2-126.7,5.9c-1.3,0-2.7-0.1-4,0
c-4.5,0.5-5.5-1.7-5.6-5.8c-0.1-4.5,0.9-6.6,6-6.5c29.7,0.7,59.2-1.5,88.8-3.9C1449.3,906.5,1464.7,904.6,1480,903z"/>
<path class="st1" d="M1192.2,1078.2c1-14.7,0.9-15,13.5-15.8c40.5-2.9,80.9-5.4,121.6-4.6c7.3,0.1,7.4,0.5,7.3,7
c-0.1,4-1.7,5.2-5.4,5.1c-6-0.2-12-0.1-18,0c-37.3,0.2-74.3,3.2-111.3,7.8C1197.6,1078,1195.3,1078,1192.2,1078.2z"/>
<path class="st1" d="M1192.6,903.2c17.2,2.2,33.7,3.8,50.2,5c28.2,2.1,56.4,4.4,84.7,3.6c3.8-0.1,7.3-0.4,7.1,5.4
c-0.3,6.3-0.2,6.8-6.6,6.8c-18.6,0-37.2,0.4-55.9-0.8c-19.2-1.2-38.5-2.1-57.8-3.2c-3-0.2-6-0.4-8.9-0.8
C1192.8,917.6,1192.8,917.6,1192.6,903.2z"/>
<path class="st1" d="M653.6,694.6c2.5,2.7,3.4,3.5,4.2,4.6c15.1,20.9,32.8,39.4,51.8,56.7c2.5,2.3,7.7,4.8,5,8.5
c-2.7,3.9-6.2-1-8.9-2.4c-14.7-7.8-28.1-17.9-42.2-26.6c-3.1-1.9-4.3-4.4-4.8-8C657.4,717,655.5,706.9,653.6,694.6z"/>
<path class="st1" d="M654,1284.6c0.9-5.5,2.7-11,2.6-16.4c-0.1-14.4,6.3-23.8,19.2-30.2c10.6-5.3,20.3-12.5,30.5-18.8
c2.5-1.5,4.9-3.3,7.8-3.6c1.6,2.1,1.5,4,0.2,5.7c-1,1.3-2.4,2.3-3.6,3.4c-18.4,17.3-36.5,34.8-51.2,55.5c-1.1,1.6-2.6,2.9-4,4.3
C655.3,1284.8,654.8,1284.6,654,1284.6z"/>
<path class="st1" d="M2019.1,698.8c-1.4,9.8-3,19.6-4.2,29.5c-0.4,3-2.2,4.7-4.4,6.2c-15.3,9.9-30.7,19.7-46.1,29.5
c-1.5,1-3.7,4-5.7,0.4c-1.5-2.6,0.2-4.4,2.1-5.9c11.9-9.2,21.5-20.7,32.4-30.8c8.4-7.7,14.5-17.2,21.4-26.1c0.9-1.2,2-2.4,3-3.6
C2018.1,698.2,2018.6,698.5,2019.1,698.8z"/>
<path class="st1" d="M2017.1,1283.9c-11.8-19.6-28.7-34.6-44.6-50.5c-3.7-3.7-7.8-7.2-11.8-10.6c-1.9-1.7-3.4-3.4-1.7-5.9
c1.7-2.4,3.5-0.5,5,0.5c15.7,10,31.3,20,46.9,30c2,1.3,3.5,2.9,3.9,5.4c1.4,10.1,2.9,20.2,4.4,30.4
C2018.5,1283.4,2017.8,1283.7,2017.1,1283.9z"/>
<path class="st1" d="M1044.6,1034c-0.6,11-1.1,21.9-1.7,32.9c-0.1,1.6-0.3,3.3-0.8,4.9c-0.3,0.8-1.4,1.6-2.2,1.7
c-0.9,0.1-1.9-0.5-2.1-1.7c-0.1-1-0.3-2-0.4-3c-1.2-22.2-2.3-44.5-2.4-66.8c0-3.2,0-6,4.6-5.9c3.5,0.1,5.6,0.7,5.5,4.9
c-0.2,11-0.1,21.9-0.1,32.9C1044.9,1034,1044.8,1034,1044.6,1034z"/>
<path class="st1" d="M1636.4,1033.8c0,10.6,0,21.2,0,31.8c0,2-0.3,4-0.7,5.9c-0.2,0.8-1.2,1.8-1.9,1.8c-0.8,0-1.9-0.8-2.2-1.5
c-0.5-1.2-0.5-2.6-0.6-3.9c-1.9-21.8-2.2-43.7-2.1-65.6c0-5.7,1.3-6.7,6-6.1c3.9,0.5,3.6,3.1,3.6,5.8c0,10.6,0,21.2,0,31.8
C1637.8,1033.8,1637.1,1033.8,1636.4,1033.8z"/>
<path class="st1" d="M1638.5,948.1c0,10.3,0,20.6,0,31c0,2.6,0.5,5.4-3.3,6.2c-4.4,0.9-6.4-0.5-6.4-5.8c0-19,0-38,1.4-56.9
c0.2-3,0.4-6,0.7-9c0.2-2.2-0.3-5.4,3-5.2c2.5,0.1,2.2,3,2.3,4.9c0.6,11.6,1.1,23.3,1.6,34.9C1638,948.1,1638.2,948.1,1638.5,948.1
z"/>
<path class="st1" d="M1045.1,948c0,10.6-0.1,21.2,0,31.9c0,3.6-0.9,5.5-5,5.5c-3.9-0.1-5.1-1.7-5.1-5.4c0.1-22.6,1.2-45.1,2.5-67.7
c0.1-1.6-0.1-3.7,2-4c2.8-0.4,3.1,1.9,3.2,3.9c0.7,11.9,1.3,23.9,1.9,35.8C1044.8,948,1044.9,948,1045.1,948z"/>
<path class="st1" d="M741.9,740.1c7,10.8,9.7,22,12.6,34.5c-8.1-3.6-13.5-8.9-19.7-13c-2-1.3-2-3.3-1.2-5.3
C735.6,751.1,737,745.6,741.9,740.1z"/>
<path class="st1" d="M1931.7,1241.3c-6.9-10.6-9.7-21.7-12.1-33.3c4.2,0.4,6.4,3.1,9.1,4.9
C1943.2,1223.1,1943.4,1224.7,1931.7,1241.3z"/>
<path class="st1" d="M1931.6,740c12,17.2,11.9,18.3-3.6,29.1c-2.4,1.7-4.5,3.9-8.5,4.8C1921.8,761.9,1925,750.9,1931.6,740z"/>
<path class="st1" d="M742.1,1241.3c-5.3-5.2-6.5-11.1-8.6-16.5c-0.8-2-0.5-3.8,1.6-5.1c6-3.9,11.2-9.1,19.2-12.6
C751.8,1219.5,748.7,1230.3,742.1,1241.3z"/>
<path class="st1" d="M599.6,517.5c0.6,0.1,1.4,0.1,1.9,0.4c12.6,7.1,13,7.9,10.7,24.4c-5.9-7.9-11-15.2-13.4-24.1
C599,518,599.3,517.7,599.6,517.5z"/>
<path class="st1" d="M597.6,1464.4c4-9.4,8.1-17.2,14-24.4C614.9,1455.5,614.5,1457.9,597.6,1464.4z"/>
<path class="st1" d="M2072.4,516.4c2.2,0.6,2.3,2,1.3,3.7c-3.8,6.1-7.7,12.1-11.9,18.6c-2.8-10.6-2.1-12.7,5.3-17.8
c1.9-1.3,4.2-2,5.3-4.3L2072.4,516.4z"/>
<path class="st1" d="M2061.5,1442.5c4.7,8.5,10.5,14.5,12.5,22.8C2067.2,1459.8,2056.4,1456.8,2061.5,1442.5z"/>
<path class="st1" d="M1781.6,836.9c4.2,9.3,4.2,9.3-1.8,12.6C1777.9,845.4,1780.6,842.1,1781.6,836.9z"/>
<path class="st1" d="M895.4,849.3c-7.5-2.9-7.5-2.9-3.8-11.1C893.8,841.3,893.9,844.7,895.4,849.3z"/>
<path class="st1" d="M630.1,1284.6c-0.2-5.4-2.7-11,1.8-17C634.2,1274.4,629.7,1279.3,630.1,1284.6z"/>
<path class="st1" d="M629.9,697.1c0.5,5.4,3.9,10.3,2.4,16.6C626.9,708.5,630.1,702.5,629.9,697.1z"/>
<path class="st1" d="M2044.5,1284.5c-2.2-5-4.3-9.2-2.8-15C2046.1,1274.3,2043.5,1279.4,2044.5,1284.5z"/>
<path class="st1" d="M2041.9,712.1c-2.2-6,1.1-10.1,2.2-14.6C2043.5,702.2,2046,707.4,2041.9,712.1z"/>
<path class="st1" d="M892.2,1143.9c-4.6-7.9-4.6-7.9,1.6-11.5C895.7,1136.5,892.6,1139.7,892.2,1143.9z"/>
<path class="st1" d="M1779.6,1132.2c6.1,3.1,6.1,3.1,2.7,11.1c-1.9-1-1.7-3.1-2.3-4.6C1779.2,1137,1778.6,1135.1,1779.6,1132.2z"/>
<path class="st1" d="M1934.8,1202.5c4.1-0.6,6.5,2.5,11.4,4.9c-6.3,0.5-8.2-3.5-11.3-5.1L1934.8,1202.5z"/>
<path class="st1" d="M1935.8,777c3.1,0.9,4.5-3.5,7.8-2.9c-1.8,4-4.7,3.9-7.9,2.8L1935.8,777z"/>
<path class="st1" d="M729.5,774c2.7-0.5,4.4,0.6,5.8,3.2C732.1,778,731,775.7,729.5,774z"/>
<path class="st1" d="M734,1202.5c2.3,4.6-1.2,4.8-4.2,5.9c-0.6-3.7,3.7-3.3,4.2-5.7C734,1202.7,734,1202.5,734,1202.5z"/>
<path class="st1" d="M1935.2,780.3c-0.4,0.2-0.8,0.5-1.2,0.7c0.3,0,0.6,0.1,0.8,0c0.2-0.1,0.4-0.3,0.6-0.5
C1935.5,780.5,1935.2,780.3,1935.2,780.3z"/>
<path class="st1" d="M740,1203.1c-1.9-0.8-4.2,2.9-6-0.5c0,0,0,0.2,0,0.1C736,1202.8,738,1202.9,740,1203.1L740,1203.1z"/>
<path class="st1" d="M1935.7,777c1.4,1.3,0.9,2.4-0.5,3.3c0,0,0.3,0.3,0.3,0.3c0.1-1.2,0.3-2.3,0.4-3.5
C1935.8,777,1935.7,777,1935.7,777z"/>
<path class="st1" d="M737.9,778.4c0.2,0.2,0.4,0.5,0.6,0.7c-0.2,0.2-0.5,0.5-0.7,0.5c-0.2,0-0.4-0.4-0.6-0.6
C737.4,778.8,737.7,778.6,737.9,778.4z"/>
<path class="st1" d="M740,1203.1c0-0.3,0-0.8-0.2-1c-0.8-0.8-0.7-1.1,0.4-0.8C739.5,1201.9,739.4,1202.4,740,1203.1
C740,1203.1,740,1203.1,740,1203.1z"/>
<path class="st1" d="M1934.9,1202.3c0-0.2-0.1-0.3-0.1-0.5c0,0-0.1,0.1-0.1,0.1c0.1,0.2,0.1,0.4,0.2,0.6
C1934.8,1202.5,1934.9,1202.3,1934.9,1202.3z"/>
<path class="st1" d="M2072.4,516.6c0.7,0.4,1.4,0.8,2.1,1.2c-0.2,0.2-0.4,0.7-0.6,0.7c-1.1-0.2-1.5-1-1.4-2
C2072.4,516.4,2072.4,516.6,2072.4,516.6z"/>
<path class="st1" d="M598.7,518.3c-1-0.2-1.8-0.8-1-1.6c0.8-0.9,1.4,0,1.8,0.8C599.3,517.7,599,518,598.7,518.3z"/>
</g>
</svg>

After

Width:  |  Height:  |  Size: 48 KiB

View File

@ -0,0 +1,455 @@
<?xml version="1.0" encoding="utf-8"?>
<!-- Generator: Adobe Illustrator 24.2.1, SVG Export Plug-In . SVG Version: 6.00 Build 0) -->
<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
viewBox="0 0 2670 1980" style="enable-background:new 0 0 2670 1980;" xml:space="preserve">
<g>
<path fill="#808080" d="M2275.3,1000.1c-0.1,2.3,0,4.7,0,7c0,189.3,0,378.7,0,568c0,6,0.1,12.1-0.8,17.9c-2.4,15.2-12.9,17.4-23.6,10
c-45.4-31-91.2-61.4-136.9-91.9l-30.1-20.3c-9.8-6.3-19.2-13.2-29.2-19.3c-6.7-4.1-9.8-9.7-9.9-17.2c-0.1-9.3-0.6-18.7-0.3-28
c0.2-7.3-1.7-13-6.5-19c-18.6-23.3-36.2-47.3-56.9-69c-18-18.8-36.7-36.8-56.4-53.8c-31.8-27.5-65.4-52.7-101-74.9
c-32.7-20.4-66.8-38.4-102.2-54c-50.7-22.4-103.2-39.2-157.1-51.5c-36.7-8.4-73.8-14.4-111.2-18.5c-34.5-3.8-69-6.5-103.7-5.8
c-23.1,0.5-46.2-1.3-69.3,0.4c-22.9,1.8-45.9,3-68.7,5.6c-72.7,8.4-143.7,24.6-212.1,50.6c-65.5,24.8-127.4,57-184.9,97.4
c-46.8,32.8-90.1,69.8-129.5,111.1c-18.1,19-35.1,39.2-50.5,60.6c-3.9,5.5-5.5,11-5.3,17.6c0.4,9.3,0,18.7,0,28
c0,9.3-3.1,16.4-11.8,21.4c-10.4,5.9-19.9,13.4-30.1,19.8l-25.4,17.2c-11.6,7.8-23.3,15.5-34.9,23.3c-34.3,23-68.6,46-102.9,69
c-14.6,9.7-23.6,5.5-25.2-12.1c-0.5-5.3-0.5-10.7-0.5-16c0-189.7,0-379.3,0-569l-0.1-26.1c0.2-190.3,0.1-380.7,0.2-571
c0-5.7,0.4-11.3,0.5-17c0.1-3.7,1.2-7.3,3-10.5c3.4-5.9,9.3-7.7,15.4-4.8c2.7,1.3,5.1,3.1,7.6,4.8c44.3,29.6,88.5,59.3,132.8,88.9
l29.7,20.1c10.8,6.6,20.9,14.3,31.7,20.9c6.7,4.1,9.7,9.8,9.8,17.3c0.2,11,0.4,22,0.2,33c-0.1,5.2,1.2,9.6,4.2,13.9
c15.4,22.2,33,42.8,51.6,62.2c53.9,56.4,114.1,105.1,181.4,144.6c38.4,22.6,78.3,42.4,119.9,58.6c40,15.6,80.9,28.3,122.8,38
c52.7,12.1,106,19.3,160,22.6c18.3,1.1,36.6,2.1,54.9,1.5c17.7-0.6,35.5,1.1,53.2-0.1c17.6-1.1,35.2-2.3,52.8-3.6
c35.9-2.5,71.4-8.5,106.7-15.1c53.4-9.9,105.5-25,156-45.1c53.8-21.5,105.1-48.2,153.2-80.9c43.4-29.5,83.6-62.7,121.1-99.3
c27.3-26.6,50.3-56.8,73.9-86.6c2.9-3.6,3.6-7.6,3.6-12.1c-0.1-10.7,0.2-21.3,0.4-32c0.1-7.5,3.2-13.1,9.9-17.2
c10.2-6.2,19.9-13.4,30-19.8l23.2-16.3c4.2-1.3,7.4-4.3,11-6.7c42.9-28.7,85.8-57.4,128.7-86.1c1.1-0.7,2.3-1.4,3.3-2.3
c10.9-9.1,21.4-4.7,23.8,10.7c0.8,5.6,0.6,11.3,0.6,17c0,189.3,0,378.7,0,568L2275.3,1000.1z M740,1203.1c-2-0.2-4-0.3-6-0.5
c-0.5,2.5-4.8,2-4.2,5.8c2.9-1,6.4-1.2,4.2-5.8C735.8,1206,738.1,1202.3,740,1203.1c-0.6-0.7-0.5-1.3,0.2-1.7
c-1.1-0.3-1.1-0.1-0.4,0.8C740,1202.3,739.9,1202.8,740,1203.1z M1935.5,780.5c-0.2,0.2-0.4,0.4-0.6,0.4c-0.3,0.1-0.6,0-0.8,0
c0.4-0.2,0.8-0.5,1.1-0.8c1.4-0.9,2-1.9,0.5-3.3c3.2,1.1,6.1,1.2,7.9-2.8c-3.3-0.7-4.6,3.8-7.7,2.9
C1935.7,778.2,1935.6,779.4,1935.5,780.5z M1934.8,1202.5c-0.1-0.2-0.1-0.4-0.2-0.6c0,0,0.1-0.1,0.1-0.1c0,0.2,0.1,0.3,0.1,0.5
c3.1,1.6,5,5.7,11.3,5.1C1941.3,1205,1938.9,1202,1934.8,1202.5z M2072.4,516.6c-1,2.3-3.4,3-5.3,4.3c-7.4,5.1-8.2,7.2-5.3,17.8
c4.2-6.5,8-12.6,11.9-18.6c1.1-1.7,0.9-3.1-1.3-3.6c0,1,0.3,1.8,1.4,2c0.2,0,0.4-0.4,0.6-0.7C2073.8,517.4,2073.1,517,2072.4,516.6
z M599.6,517.5c-0.4-0.8-1.1-1.7-1.8-0.8c-0.8,0.8,0.1,1.5,1,1.6c2.4,8.9,7.5,16.2,13.4,24.1c2.3-16.5,1.9-17.3-10.7-24.4
C600.9,517.6,600.2,517.6,599.6,517.5z M534.9,1353.5c-20,21.8-37.3,44.1-53.4,67.4c-16.1,23.4-26.8,49.5-41.8,73.5
c9-21,17.4-42.2,28.9-61.9c16.9-28.7,35.4-56.4,57.5-81.5c0.9-1,1.6-2.1,2.6-3.1c2.9-2.9,3.3-6,2.3-10
c-9.1-33.8-14.6-68.3-19.2-103c-5-38-8.3-76.1-10.6-114.3c-2.5-39.9-3.4-79.9-3.9-119.8c-0.1-5.1-1.4-6.8-6.7-6.9
c-21.6-0.2-43.3-0.8-64.9-1.3c-2,0-4-0.4-6,0.8c0,195.2,0,390.5,0,585.6c2.1,0.6,3.1-0.4,4.1-1.1c52.7-33.8,105.3-67.7,158-101.4
c3.6-2.3,5.7-5.4,7.3-9.1c6-13,12.5-25.8,20.8-37.4c4.2-5.9,3.9-11.9,2.5-19.1c-2.4,1.8-4,2.9-5.5,4.2
c-13.9,12.6-32.1,8.7-42.7-2.9c-7.3-8-12.4-17.2-17.1-26.8C542.3,1375.7,539.4,1365.1,534.9,1353.5z M534.6,627.2
c3.6-5,4.3-10.7,6.3-15.9c6.2-15.8,12.6-31.5,25.2-43.7c10.8-10.5,26.8-12.6,38.9-2.8c2.3,1.9,4.3,5,7.9,5
c1.6-8.3-0.2-15.3-5.1-21.9c-7.4-9.9-12.7-21-17.8-32.3c-2.4-5.3-5.8-9.2-10.8-12.4c-50.5-32.2-100.9-64.7-151.3-97.1
c-2.4-1.5-4.5-3.9-8.7-3.9c0,3.7,0,7.2,0,10.7c0,187.9,0,375.8,0,563.7c0,2,0.2,4,0,6c-0.6,4.9,1.4,6.3,6.2,6.1
c13.6-0.6,27.3-0.7,41-1c8.3-0.2,16.7-0.6,25-0.4c4.9,0.1,6.7-1.8,6.1-6.4c-0.1-1,0-2,0-3c1.2-75,4.3-149.8,13.7-224.3
c4.6-36.3,10-72.5,19.6-107.9c1.6-5.7,0.4-9.9-3.4-14.4c-10.7-12.8-21.7-25.4-31-39.1c-19-28.1-36.8-57-49.6-88.7
c-2.2-5.4-5.5-10.4-6.2-16.6C465,538.3,494.6,586,534.6,627.2z M2060.9,1411.1c-2.3,8.5-0.1,15.1,4,21.4
c5.9,9.3,13.4,18.1,17.1,28.2c3.9,10.7,10.9,16.6,19.9,22.4c47.8,30.3,95.3,61.1,143,91.7c2.7,1.7,5,4.3,9.5,4.5c0-3.4,0-6.6,0-9.8
c0-188.3,0-376.6,0-565c0-2-0.1-4,0-6c0.2-3.4-1.1-5-4.6-4.8c-2.3,0.1-4.7,0-7,0.1c-20,0.6-40,1.2-60,1.9c-6.7,0.2-6.7,0.4-7,7.4
c0,1.3,0,2.7-0.1,4c-0.4,60-3,119.9-8.9,179.7c-5,50.1-11.3,99.9-24,148.7c-1.4,5.2,0.1,8.3,3.5,12.3c15,17.8,29.6,36.1,42.4,55.5
c20,30.4,36.1,62.9,49.3,96.8c3.2,8.2,5.7,16.6,6.8,25.4c-33.9-90.8-82.6-145-107.2-169.7c-4.1,11-7.5,22.1-12.7,32.5
c-4.9,9.9-10.3,19.5-18.8,26.8c-7.1,6.1-15.1,8.9-24.8,7.5C2073.2,1421.4,2068.3,1415.5,2060.9,1411.1z M2244.8,454.3
c0.6,4.5-0.2,6.7-0.8,8.9c-6.5,24.2-17,47-28.2,69.2c-18.9,37.2-43,71.2-70.4,102.8c-2.7,3.2-3.6,5.8-2.5,9.8
c7.9,29.6,13.1,59.8,17.3,90.2c6.4,45.6,10.2,91.3,12.8,137.3c2,35.3,2.6,70.6,3.1,105.9c0.1,7.6,0.2,7.6,7.6,7.8
c17,0.5,34,1,51,1.4c5,0.1,10,0.2,15,0.5c3.2,0.1,5.1-0.9,4.9-4.5c-0.1-1.7,0-3.3,0-5c0-189.6,0-379.3-0.1-568.9
c0-2.4,1.1-5.2-1.2-7.3c-0.8,0.1-1.6,0-2.1,0.4c-52.7,33.8-105.3,67.7-158,101.5c-2,1.3-3.7,2.6-4.7,4.9c-6.9,16-16.9,30.1-25.8,45
c-3.1,5.2-3.4,10.2-2,15.7c2.6,0.1,3.5-1.6,4.6-2.6c15.7-13.9,33-10.6,45.6,3.8c6,6.8,10.4,14.6,14.4,22.7
c5.1,10.3,8.3,21.3,12.8,32.9C2185.2,576.1,2221.7,520.5,2244.8,454.3z M553.7,662.8c-0.3,1.3-0.7,2.8-1,4.3
c-7.1,31.2-11.7,62.8-15.4,94.5c-7.1,59.9-10.3,120.1-11.8,180.4c-0.3,13-0.2,26-0.5,39c-0.1,4.4,1.5,5.7,5.7,5.6
c25-0.5,50-0.7,74.9-1.1c7.4-0.1,7.6-0.2,7.6-7.8c0-85.6,0-171.2,0.1-256.8c0-5.3-1.7-8.5-5.8-11.7c-16.6-12.8-32.4-26.6-46.9-41.9
C559,665.6,557.6,663,553.7,662.8z M553.6,1319.7c2.1-1,2.8-1.2,3.2-1.6c16.4-17.1,34-32.9,52.7-47.4c3-2.4,3.8-5,3.8-8.6
c-0.1-13.3-0.1-26.7-0.1-40c0-72.3,0-144.7,0-217c0-10.3,0.4-9-9.2-9.2c-23.3-0.4-46.7-0.8-70-1.1c-2.8,0-5.9-1-9.3,1.5
C526.2,1104.4,531.1,1212.3,553.6,1319.7z M2119.8,1319.2c4.8-20.9,8.3-40.2,11.1-59.6c5.8-39.5,9.8-79.2,12.5-119.1
c3.1-45.5,4.7-91.1,5.1-136.7c0.1-7.6-0.1-7.7-7.6-7.7c-24.3,0.1-48.6,0.3-72.9,0.5c-7.9,0.1-8,0.1-8,8.5c0,84.6,0,169.1-0.1,253.7
c0,5.6,1.7,9.3,5.9,12.7c12.1,9.9,24.2,19.9,36,30.1C2107.7,1306.9,2113.1,1312.7,2119.8,1319.2z M2120.4,664.4
c-2.1-0.6-2.9-0.1-3.6,0.7c-15.2,16.5-33.1,29.9-50.2,44.2c-4.8,4-6.8,8-6.8,14.3c0.3,32,0.2,64,0.2,96c0,52.3,0,104.6,0,157
c0,8.1,0.1,8.2,8.2,8.2c18.3,0.2,36.7,0.2,55,0.4c6.7,0.1,13.3,0.1,20,0.3c3.7,0.1,5.7-1.1,5.4-5.1c-0.2-2.3,0-4.7-0.1-7
c-0.5-45-2.1-89.9-5.2-134.8C2139.2,780.1,2132.7,721.9,2120.4,664.4z M878.7,1138.5c-36,15-69.8,32.9-102.3,53.1
c-3.4,2.1-4.9,4.7-5.7,8.6c-2.8,14.4-5.7,28.8-12.2,42c-3.6,7.3-7.7,14.4-17.3,14.4c-9.6,0-13.4-7.4-17.4-14.4
c-1.9-3.2-2-7.6-5.4-9.8c-22.3,17.4-41.6,37.2-58.8,58.9c-5.8,7.3-10.2,15.3-12,24.8c-3.8,20.3-9.1,40.2-16.7,59.5
c-2.6,6.7-1.3,13.3-1.5,20.3c1.5-0.4,2-0.3,2.1-0.5c3.2-3.8,6.4-7.6,9.4-11.6c31.2-40.5,66-77.4,105.4-110.2
c36.9-30.8,75.6-59.1,117.2-83.1c54.3-31.2,110.9-57.4,170.8-76.4c38.4-12.2,77.4-21.8,117-29.2c9.5-1.8,19.1-3,28.6-4.6
c1.5-0.3,3.8,0.3,4.4-1.5c1.4-3.9,0.6-8-0.4-11.7c-1-3.6-4.5-1.7-6.9-1.5c-40.6,3.2-80.4,11-120.4,18c-7,1.2-10.4,4.1-10.9,11.2
c-0.3,3.8-0.5,9.1-5.8,9c-4.6-0.1-4.7-5-5.3-8.5c-0.7-4.5-2.4-6-7.3-4.9c-39.3,9.1-78,19.9-115.5,34.7c-4.6,1.8-7.1,4.2-7.8,8.9
c-0.9,5.9-2.2,11.8-5.2,17.1c-4.1,7.1-10.6,7.2-14.6,0C881.9,1147.7,880.7,1143.4,878.7,1138.5z M1032.8,891.9
c0.9-3.8,1.6-6.7,2.4-9.5c0.7-2.3,1.7-4.6,4.4-4.9c3.4-0.3,4.4,2.3,5.3,4.9c0.4,1.3,0.8,2.6,0.8,3.9c0.1,7.7,4.5,10.4,11.7,11.6
c32.5,5.5,64.8,11.9,97.5,15.7c7.9,0.9,15.9,1.7,23.8,2.4c1.9,0.2,4,0.7,5.3-1.6c3.5-6.5,0.3-13-7-14.2c-1-0.2-2-0.2-3-0.4
c-42.6-5.8-84.2-15.9-125.4-28.2c-41.8-12.4-82.5-28.2-121.6-47.1c-51-24.6-99.8-53.3-144.7-87.9c-20.6-15.8-40.9-32.1-59.9-49.7
c-29.1-26.9-56.4-55.7-80.1-87.7c-3.7-4.9-7.7-9.6-11.6-14.3c-0.8,7.6-2.4,14.7,0.5,21.8c7.4,18.2,12.4,37.2,16,56.5
c1.7,9.1,5,17.3,10.6,24.4c9.2,11.4,18.5,22.8,28.7,33.3c9.9,10.1,21.1,19,32.2,28.9c3.8-6.5,5.6-13,9.8-18.2
c7.7-9.4,17.7-9.5,25.3-0.1c3.6,4.5,5.7,9.7,7.8,15c4.3,10.9,6.8,22.3,8.9,33.7c0.9,5.2,3.3,8.1,7.5,10.7
c21.9,13.4,44.1,26,67.3,37.1c10.9,5.2,22.1,10,33.5,15.2c1.8-4.8,3.1-8.6,4.7-12.2c1.6-3.5,4.2-6.3,8.3-6.1
c3.6,0.2,5.8,2.9,7.3,6.1c2.4,5.2,3.8,10.6,4.8,16.2c0.8,4.6,2.9,7.2,7.7,9C950.6,871.5,990.8,883,1032.8,891.9z M1953.8,1233.7
c-2.9,5.8-4.9,11.5-8.8,16.2c-7.6,9.3-17.8,9.3-25.3,0c-3.4-4.2-5.5-9-7.5-14c-4.4-11.5-7.2-23.5-9.4-35.7
c-0.7-3.9-2.4-6.4-5.8-8.6c-30.4-19.4-62.4-35.6-95-50.8c-6.9-3.3-7-3-9.9,4.7c-0.7,1.9-1.5,3.7-2.3,5.5c-1.6,3.1-3.7,5.6-7.6,5.6
c-3.8,0-5.9-2.5-7.6-5.6c-2.9-5.4-4.3-11.3-5.1-17.2c-0.7-5-3-7.3-7.5-9.1c-37.8-15.2-77.3-24.6-116.7-34.3c-3.8-0.9-5.4,0.2-6.2,4
c-0.7,3.9-0.2,9.9-6.3,9.5c-5-0.3-5.1-6.1-5.3-9.6c-0.4-6.6-3.3-8.9-9.6-10c-32.8-5.6-65.4-12.2-98.6-15.7
c-8.3-0.9-16.5-2.1-24.8-2.7c-5.4-0.4-7.1,1.2-7.5,6.6c-0.3,3.9-1.3,8,5.2,8.4c6.3,0.4,12.4,2.1,18.7,3.1
c47.7,7.5,94.4,19.1,140.3,34.3c42.4,14,83.5,31.2,123,52c46.3,24.4,90.2,52.6,131,85.6c48.5,39.1,93.7,81.2,129.4,132.6
c1.9,2.8,4.8,7.5,7.3,6.7c4.5-1.5,1.4-6.8,2.1-10.4c0.6-3.4-0.5-6.6-1.8-9.7c-7.8-19.5-12.6-40-17.1-60.4c-1.1-5.1-3-9.4-6-13.5
c-17-22.8-36.7-43.1-57.6-62.2C1959.6,1237.1,1957.6,1234.7,1953.8,1233.7z M2043.6,585.1c-4.2,2.7-6.2,6.2-8.5,9.2
c-29.9,40.3-64.6,76.1-102.5,109c-33.8,29.3-69.5,56.2-107.6,79.4c-51.8,31.6-106.3,57.7-163.9,77.3
c-40.7,13.8-82.1,24.9-124.2,33.1c-13.7,2.7-27.6,4.7-41.3,7c-2.2,0.4-5.3-0.1-6.2,2.4c-1.4,3.8-1.1,8.2,0.3,11.7
c1.4,3.5,5.3,1.6,8.1,1.3c40.1-4.2,79.8-10.7,119.5-18.1c6.2-1.2,10.4-2.7,10.5-10.2c0-3.5,0.3-9.3,5.4-9.5c6-0.3,5.5,5.7,6.2,9.6
c0.7,3.9,2.3,4.9,6.1,3.9c10-2.7,20-5.1,30-7.5c28.5-6.9,56.4-15.6,83.9-25.7c6.8-2.5,10.2-6.3,11-13.3c0.5-5,2-9.8,4.5-14.2
c4.2-7.6,10.8-7.6,15.1,0.1c1.6,2.9,2.5,6.2,3.4,9.3c0.9,2.9,2.5,3.5,5.2,2.2c2.4-1.2,5-1.9,7.3-3.1c26.6-13.1,53.2-26,78.7-41.2
c11.9-7.1,19.6-14.7,21.3-29.3c1.4-12.4,5.5-24.8,12.8-35.4c8.1-11.7,19.5-11.7,27.5,0.1c3.1,4.6,5.4,9.7,8.2,14.9
c1.9-1.4,3.6-2.5,5.1-3.8c22.2-19.8,42.7-41.1,60.6-64.9c2.6-3.5,4.3-7.3,5.2-11.7c4.5-21.2,9.6-42.2,17.6-62.4
C2045.3,599.2,2044.5,592.6,2043.6,585.1z M777.2,1163.8c1.2,0.2,1.6,0.4,1.9,0.3c30.4-14.3,60.9-28.3,92.6-39.6
c4.2-1.5,4.4-3.7,3.9-7.5c-4.5-37.4-6.3-75-6.5-112.6c0-7.8-0.2-7.9-8.5-7.9c-22,0-44,0-65.9,0c-8.6,0-8.8,0.1-8.8,8.7
c-0.4,45.3-2.2,90.5-6.6,135.6C778.4,1148.4,776.3,1155.9,777.2,1163.8z M776.8,817.8c0.3,3.8,0.5,7.1,0.9,10.4
c2.5,21.2,4.1,42.4,5.5,63.7c1.8,28.6,2.3,57.2,2.7,85.9c0.1,7.1,0.3,7.3,7.2,7.3c22.6,0.1,45.3,0.1,67.9,0.1
c8.8,0,8.1-0.2,8.1-8.3c0.1-37,1.9-73.9,6.4-110.6c0.7-5.8-0.5-8.2-6-10.1c-29.5-10.3-58-23.3-86.2-36.8
C781.5,818.4,779.9,817.2,776.8,817.8z M1896.4,1163.9c1-2.7,0.2-5.3-0.1-7.9c-2.4-19.8-3.9-39.8-5.3-59.7
c-2.1-30.3-2.5-60.6-3.3-90.9c-0.2-8.7-0.2-8.8-8.6-8.8c-16.7-0.1-33.3,0-50-0.1c-6.3,0-12.7,0.2-19-0.1c-3.9-0.1-5.5,1.4-5.2,5.3
c0.1,1.7,0.1,3.3,0,5c-1.1,36-2,72-6.4,107.7c-0.9,7.7-1.1,7.9,6.3,10.6c28.5,10.4,56.1,22.9,83.3,36.1
C1890.6,1162.4,1893.1,1164.4,1896.4,1163.9z M1896.8,817.7c-2.5-0.3-4.3,0.6-6.1,1.5c-6,2.9-12,5.6-18,8.6
c-22.1,10.9-44.9,20.1-67.9,28.7c-7.5,2.8-7.3,3-6.5,10.6c1,8.9,1.9,17.9,2.7,26.8c2.4,28.2,3.2,56.5,3.6,84.7
c0.1,5.4,1.8,6.7,6.8,6.6c22.6-0.2,45.3-0.1,67.9-0.2c1,0,2-0.1,3,0c3.8,0.3,5.6-1.2,5.2-5.1c-0.2-1.6,0-3.3,0.1-5
c0.7-38.6,1.7-77.2,5.2-115.6C1894.1,845.4,1895.5,831.6,1896.8,817.7z M908.3,1109.6c1.4,0,2.4,0.2,3.2,0
c37.8-11.7,75.5-24,114.2-32.5c4.5-1,5.8-2.9,5.5-7.4c-1.3-22-2.5-43.9-2.3-65.9c0.1-7.3-0.2-7.4-8-7.4c-33.3,0-66.7,0.1-100,0.2
c-7.7,0-7.9,0.1-7.9,7.5c0,31-1.6,61.9-4,92.9C908.8,1101,907.3,1105.2,908.3,1109.6z M1765.3,1110.4c0-2.5,0.1-3.5,0-4.4
c-3.1-33.9-4.7-67.8-4.8-101.8c0-7.4-0.4-7.6-8.2-7.7c-33.3-0.1-66.7-0.2-100-0.2c-7.3,0-7.2,0.1-7.5,7.9c-0.7,21-1.1,42-2.3,62.9
c-0.5,9-1.3,8.3,7.6,10.4c29.8,7.1,59.4,14.9,88.5,24.5C1747.2,1104.8,1755.7,1107.4,1765.3,1110.4z M908.8,872.2
c-0.2,0.3-0.6,0.6-0.6,0.8c2.8,35.5,5.2,71,4.8,106.7c-0.1,4.6,1.9,5.6,6,5.6c34.7-0.1,69.3-0.1,104,0.1c4.9,0,6.2-1.8,6.1-6.4
c-0.5-22,0.8-44,2.1-65.9c0.3-5.2-0.9-7.7-6.4-8.8c-36.9-7.6-72.4-20.1-108.5-30.5C913.8,873,911.6,871.1,908.8,872.2z
M1765.6,872.4c-3.2-0.6-5.1,0.3-7,0.9c-22.2,6.7-44.2,14.5-66.7,20.3c-14.5,3.7-29,7.3-43.5,10.8c-4.4,1.1-6.5,2.6-6.2,8
c1.3,21.9,1.8,43.9,2.6,65.8c0.2,7,0.2,7.1,7.1,7.1c33.6,0,67.2,0,100.8-0.1c7.5,0,7.7-0.2,7.7-7.8c0.2-24.6,0.9-49.2,2.6-73.8
C1763.8,893.4,1764.7,883.1,1765.6,872.4z M725.8,1192.5c8.7-4.8,15.8-9.8,23.8-12.8c9.6-3.7,12.7-10.5,13-19.7
c0.1-1.3,0.2-2.7,0.4-4c3.4-27.1,5.6-54.3,6.7-81.5c1-23.6,1.5-47.2,2-70.8c0.2-7-0.1-7-7.3-7.1c-6.7-0.1-13.3,0-20,0
c-9,0-18,0.2-27,0c-4.2-0.1-6.2,1.3-5.9,5.7c0.2,2.3,0,4.7,0.1,7c0.8,27.6,1.2,55.2,3,82.8C717,1125.1,719.9,1158.2,725.8,1192.5z
M1946.2,1191.7c2.9-2.2,2.4-4.7,2.8-6.9c4.5-26.6,6.9-53.5,8.9-80.4c2.5-33.9,3.2-67.9,3.7-101.8c0.1-5-1.8-6.2-6.4-6.1
c-15.3,0.2-30.6,0-46,0.1c-7.6,0-7.8,0.2-7.7,7.8c0.7,44,2,87.9,6.7,131.6c1.1,10.3,2.2,20.5,3.5,30.7c0.4,2.8,0.2,6,3.5,7.8
C1925.8,1180.2,1936.2,1186.1,1946.2,1191.7z M1947.7,789.1c-9.1,5-16.9,10.1-25.3,13.8c-8.2,3.6-10.5,9.5-11.5,17.8
c-3,27.1-5.6,54.3-6.9,81.6c-1.2,25.3-1.5,50.6-2.5,75.9c-0.2,5.1,1.5,7,6.7,7c15-0.3,30-0.1,45-0.2c8.6,0,8.4,0.6,8.3-8.9
c-0.3-30.6-1-61.3-3.1-91.9C1956.3,852.9,1953.6,821.8,1947.7,789.1z M726,789.2c-3.3,16.4-5.1,31.3-6.7,46.2
c-4.6,42.4-6.3,85-7.3,127.7c-0.1,5.7,0,11.3-0.2,17c-0.1,3.4,1.4,4.8,4.7,4.8c17,0,34,0,51,0.1c3.6,0,4.6-1.9,4.4-5
c-0.1-2.3-0.1-4.7-0.1-7c-0.5-37.3-1.7-74.6-5-111.8c-1.4-15.9-2.8-31.9-5.5-47.6c-0.4-2.6-0.3-5.2-3.2-6.8
C747.6,801.1,737.2,795.4,726,789.2z M1625.1,910.5c-9.8,1.8-18,3.4-26.2,4.6c-23.3,3.5-46.2,9.9-70,10
c-10.9,0.1-21.8,2.4-32.7,3.7c-3.5,0.4-6.2,1.1-6,6c0.5,15,0.7,30,0.7,44.9c0,4.4,1.7,5.8,5.9,5.7c8.3-0.2,16.6,0,25-0.1
c30.6,0,61.3,0,91.9-0.1c9,0,9-0.1,9.2-9.4c0.2-13,0.2-26,0.6-38.9C1623.7,928.4,1624.5,919.8,1625.1,910.5z M1625.2,1071.1
c-2.2-23.8-2.3-46.1-2.4-68.4c0-4.5-1-6.6-6.1-6.6c-39.9,0.1-79.9,0.1-119.8-0.1c-5.1,0-6.1,2-6.1,6.5c0,14.6-0.1,29.3-0.6,43.9
c-0.1,4.5,1.6,5.6,5.6,6.1c12.6,1.4,25,3.8,37.7,4c8,0.1,15.9,1.1,23.8,2.5C1579.5,1063,1601.7,1066.9,1625.2,1071.1z
M1049.3,1070.4c2.6,1,4.9-0.1,7-0.6c23-5.5,46.3-8.9,69.7-12.3c16.7-2.5,33.7-3.8,50.6-5.2c5.1-0.4,6.8-1.9,6.6-7
c-0.4-14-0.3-27.9-0.5-41.9c-0.1-7-0.2-7.2-7.2-7.2c-39.3,0-78.5,0-117.8,0.1c-6.7,0-6.8,0.2-6.8,7.5c0,16.3-0.5,32.6-1.4,48.9
C1049.3,1058.6,1048.4,1064.5,1049.3,1070.4z M1049.8,910.2c-0.4,1.2-0.8,1.8-0.8,2.4c0.6,22.6,2.5,45.2,1.9,67.8
c-0.1,4.2,2.1,4.8,5.6,4.8c39.9,0,79.9-0.1,119.8,0.1c5.5,0,6.7-2.3,6.6-7.2c-0.2-11,0.1-22,0.2-32.9c0.1-15.9,0.1-15.5-16.3-17.4
c-16.5-1.9-33.1-2.4-49.6-5.2C1094.7,918.9,1072,915.9,1049.8,910.2z M1260.5,985.4c21.6,0,43.2,0,64.9,0c9.1,0,9.1-0.1,9.1-9.4
c0-10.3,0-20.6-0.1-30.9c-0.1-7.7-0.1-7.7-7.6-7.9c-2.3-0.1-4.7,0-7,0c-41.9,0-83.8-2.2-125.5-6.2c-4.6-0.4-5.9,0.9-6,5.1
c-0.2,13.6-0.6,27.3-0.8,40.9c-0.1,8.3,0.1,8.4,8.1,8.4C1217.3,985.4,1238.9,985.4,1260.5,985.4z M1485,931.5
c-3.1-1.4-5.4-0.6-7.7-0.4c-26.2,1.9-52.4,4.3-78.7,5.1c-17.3,0.5-34.6,1.3-51.9,1.1c-7.4-0.1-7.5,0.1-7.6,7.8
c-0.1,11.7,0,23.3-0.1,35c0,3.7,1,5.8,5,5.5c2-0.1,4,0,6,0c41.6,0,83.2,0,124.9,0c2,0,4-0.1,6,0c3.3,0.1,5-1.2,4.9-4.6
C1485.5,964.2,1485.3,947.6,1485,931.5z M1485,1051.1c0.3-17,0.5-33.6,0.9-50.1c0.1-4.2-2.2-4.8-5.7-4.8c-45,0.1-89.9,0.1-134.9,0
c-4.7,0-6.5,1.4-6.3,6.2c0.3,12,0.4,24,0,36c-0.2,5.2,2.1,6,6.5,6c11.6-0.1,23.3,0.1,34.9,0.4
C1415,1045.8,1449.5,1048.1,1485,1051.1z M1188.4,1050.6c47.5-3.4,93.2-6.8,139.2-6.4c0.7,0,1.3,0,2,0c3.2,0.2,4.9-0.9,4.9-4.5
c-0.1-13,0-25.9,0-38.9c0-3.4-1.6-4.8-4.9-4.6c-1.7,0.1-3.3,0-5,0c-42.6,0-85.1,0-127.7,0c-1.7,0-3.3,0.1-5,0
c-2.8-0.1-4.4,1.1-4.4,4.1C1187.9,1016.8,1188.1,1033.4,1188.4,1050.6z M2010.7,1230.1c0.9-7.3-0.9-13.8-1.5-20.3
c-6.8-66-10.3-132.2-10.7-198.6c-0.1-16.1-0.3-16.3-16.8-14.7c-2.9,0.3-4.2,1.1-4.1,4.1c0.1,3,0,6,0,9
c-0.3,41.3-2.1,82.6-6.1,123.8c-2,20.2-4.3,40.4-8.2,60.4c-0.9,4.8-0.2,7.8,4.5,10.4c10.2,5.7,19.9,12.2,29.8,18.3
C2001.8,1225,2005.6,1228.2,2010.7,1230.1z M2011.2,752.5c-2.7-0.8-3.6,0.3-4.7,1c-13.3,8.3-26.4,16.7-39.8,24.7
c-3.6,2.1-4.2,4.5-3.5,8.2c1.4,7.8,2.8,15.7,3.9,23.6c7.4,53.8,10.1,107.9,10.6,162.2c0.1,14.1-2.6,12.8,13.7,12.8
c7,0,7-0.2,7.1-7.3c0.4-27.3,0.6-54.6,1.8-81.8c1.4-30.9,3.1-61.8,6.1-92.6C2008,786.3,2009.6,769.5,2011.2,752.5z M662.1,1231.5
c3.7-2.4,5.6-3.7,7.5-4.9c11.6-7.7,22.9-16.2,35.5-22.2c5.3-2.5,6.2-5.5,5.1-11c-4.6-22.5-7-45.4-9-68.3
c-3.5-39.9-5-79.8-5.4-119.8c-0.1-8.6-0.2-8.6-8.8-8.7c-1.3,0-2.7,0-4,0c-7.9,0.1-8,0.1-7.9,8.4c0,24.7-0.6,49.3-1.6,74
c-1.3,32-3,63.9-6,95.8C665.8,1192.9,664.1,1211.1,662.1,1231.5z M662.9,752.1c-1,1.4-0.4,3-0.3,4.6c2,20.5,4.6,41.1,6,61.7
c3.8,52.5,6.5,105.1,6.4,157.8c0,8.8,0.1,8.9,8.5,8.9c14.5-0.1,12.1,1.3,12.3-12.1c1.1-62.7,3.5-125.2,14.8-187
c0.8-4.1-0.9-6-4.2-7.7c-13.4-6.8-25.6-15.5-38-23.9C666.8,753.1,665.5,751.1,662.9,752.1z M2044.3,728.6c-5.4,1.1-7.1,3.9-7.7,8.3
c-1.7,14.9-4,29.7-5.6,44.6c-4.6,42.1-7.2,84.2-8.9,126.5c-1,25.3-1.3,50.5-1.3,76.8c6.5,0,12.4-0.1,18.3,0
c3.8,0.1,5.5-1.4,5.2-5.3c-0.2-1.7,0-3.3,0-5c0-78.9,0-157.9,0-236.8C2044.3,734.9,2044.3,732,2044.3,728.6z M2044.3,1253
c0-3.7,0-6.4,0-9c0-78.6,0-157.2,0-235.8c0-13,1.3-11.5-11.9-11.5c-13,0-11.5-1.8-11.5,11.8c0,59.3,3.1,118.5,8.7,177.6
c1.9,19.5,4,39.1,7,58.5C2037.3,1248.9,2038.8,1251.8,2044.3,1253z M629.2,728.7c0,4.3,0,8,0,11.6c0,76.5,0,153.1,0,229.6
c0,15.8,0,15.8,16,15.3c7.6-0.2,7.6-0.2,7.7-7.7c0.1-3.3,0-6.7,0-10c-0.8-55.6-3.1-111.1-8.5-166.4c-2-21.2-5-42.3-7.3-63.4
C636.6,733.3,635,730.3,629.2,728.7z M629.2,1253.2c5.3-2.1,7.4-4.5,7.9-9.2c1.6-14.9,4-29.7,5.5-44.6
c6.5-62.4,9.5-124.9,10.3-187.6c0.2-15.6,0.1-15.6-15-15.6c-9,0-8.8-0.6-8.7,8.6c0,2,0,4,0,6c0,75.7,0,151.3,0,227
C629.2,1242.5,629.2,1247.2,629.2,1253.2z M2060,697.5c19-15.9,36.7-29.3,52.3-45.3c3.5-3.5,3.7-6.7,2.4-10.7
c-2.8-8.9-5.4-17.8-8.7-26.5c-3.5-9.3-7.4-18.5-13.7-26.4c-5.3-6.6-8-6.6-13.4-0.4c-1.3,1.5-2.7,3-3.6,4.7
c-8.9,15.7-17.3,31.5-15.3,50.5c0.4,3.6,0.1,7.3,0.1,11C2060,668,2060,681.5,2060,697.5z M613.2,1283.1
c-20.8,13.6-36.9,29.9-53.8,45.4c-2.8,2.6-2.3,5.1-1.5,8.1c4.1,14.7,8.6,29.3,15.3,43.2c2.6,5.4,5.4,10.7,9.8,15
c4.3,4.2,6.2,4.6,10,0.4c10.7-11.9,18.4-26.3,19.7-41.7C614.7,1330.6,613.2,1307.6,613.2,1283.1z M2060,1284.4
c0,19.5,0.7,36.8-0.2,54c-1.1,19.1,6.3,35.2,15.8,50.5c7.4,12.1,12.1,11.7,19.8-0.6c9.7-15.4,14.6-32.7,19.7-50
c1.1-3.7,0-6.2-2.6-8.8C2096.8,1313.6,2079.3,1299.9,2060,1284.4z M611.9,698.2c2.1-3.3,1.3-5.9,1.3-8.4c0.1-16-0.8-32,0.3-47.9
c1.2-18.4-6.2-33.8-15.1-48.6c-7.6-12.7-12.6-12.4-20.4,0.3c-10.1,16.3-15.1,34.6-20.4,52.8c-1,3.6,0.6,5.5,3,7.7
c8.8,8.1,17.3,16.7,26.3,24.5C594.8,685.4,603.3,691.5,611.9,698.2z M1788.2,864c-12.5,3.2-12.6,3.2-13.9,13.8
c-0.9,7.6-1.6,15.2-2.1,22.8c-1.8,26.2-2.9,52.5-3.2,78.8c0,4,0.8,6.3,5.4,6c4.6-0.3,9.3-0.2,14,0c5.1,0.1,6.9-2.1,6.7-7.3
c-0.5-11.3-0.2-22.6-0.7-33.9C1793.2,917.5,1792.3,890.9,1788.2,864z M885.3,1118.4c12.8-3,12.5-3,13.8-13.9
c3.5-29.5,4.6-59.1,5-88.8c0.3-23.5,5.2-18.7-19.8-19.4c-4.4-0.1-5.7,1.5-5.7,5.8C878.8,1040.8,880.3,1079.4,885.3,1118.4z
M885.7,863.5c-0.2,0.9-0.6,1.9-0.7,2.9c-4.3,33.7-6,67.5-6.1,101.4c0,5.5-2.5,12.6,1.3,16.2c3.8,3.6,10.8,1.1,16.3,1.2
c7.9,0.1,8.1-0.1,7.9-8.4c-0.6-33.2-1.5-66.4-5.3-99.4C897.9,865.6,898,865.6,885.7,863.5z M1788.4,1117.5
c4.4-34.4,5.9-68.6,6.5-102.9c0.1-5.6,2.4-12.7-0.9-16.5c-3.9-4.3-11.3-1.4-17.2-1.6c-7.6-0.2-7.7,0.1-7.7,7.6
c0,24.7,1.3,49.3,2.8,73.9c0.6,10.3,2.2,20.5,2.9,30.8c0.3,3.6,1.7,5.5,4.9,6.5C1782.2,1116,1784.4,1117.7,1788.4,1117.5z
M1481.1,1078.1c0-2.7,0.1-4.4,0-6c-0.3-7.8-0.3-8.1-8.2-8.7c-22.9-1.7-45.8-3.3-68.7-4.7c-19.9-1.2-39.9-0.9-59.9-1
c-4.6,0-5.5,2-5.4,6c0.1,3.7,0.3,6.7,5.2,6.2c1.3-0.1,2.7,0,4,0c34.3-0.3,68.4,1.9,102.5,5.6
C1460.6,1076.6,1470.4,1078.5,1481.1,1078.1z M1480,903c-15.3,1.6-30.8,3.6-46.3,4.8c-29.5,2.4-59.1,4.7-88.8,3.9
c-5.1-0.1-6.1,2-6,6.5c0.1,4.1,1.1,6.3,5.6,5.8c1.3-0.2,2.7,0,4,0c42.3,0.3,84.5-2.6,126.7-5.9c4.2-0.3,6.2-1.6,5.8-5.9
C1480.7,909.3,1482,906.3,1480,903z M1192.2,1078.2c3.1-0.2,5.4-0.3,7.6-0.5c37-4.6,74-7.6,111.3-7.8c6,0,12-0.1,18,0
c3.7,0.1,5.3-1.1,5.4-5.1c0.1-6.5,0-6.9-7.3-7c-40.6-0.8-81.1,1.7-121.6,4.6C1193.1,1063.3,1193.2,1063.6,1192.2,1078.2z
M1192.6,903.2c0.2,14.4,0.2,14.4,12.8,16c3,0.4,6,0.6,8.9,0.8c19.3,1.1,38.5,2,57.8,3.2c18.6,1.2,37.2,0.8,55.9,0.8
c6.4,0,6.3-0.5,6.6-6.8c0.2-5.8-3.3-5.5-7.1-5.4c-28.3,0.8-56.5-1.5-84.7-3.6C1226.3,907,1209.8,905.4,1192.6,903.2z M653.6,694.6
c2,12.3,3.8,22.4,5.1,32.6c0.5,3.7,1.7,6.1,4.8,8c14.2,8.8,27.5,18.8,42.2,26.6c2.7,1.4,6.2,6.3,8.9,2.4c2.7-3.7-2.6-6.2-5-8.5
c-19-17.3-36.7-35.8-51.8-56.7C657,698.1,656,697.3,653.6,694.6z M654,1284.6c0.8,0,1.3,0.1,1.5,0c1.4-1.4,2.9-2.7,4-4.3
c14.6-20.8,32.8-38.3,51.2-55.5c1.2-1.1,2.6-2.1,3.6-3.4c1.4-1.7,1.5-3.6-0.2-5.7c-2.9,0.3-5.3,2.1-7.8,3.6
c-10.2,6.3-19.8,13.5-30.5,18.8c-12.9,6.4-19.3,15.7-19.2,30.2C656.6,1273.7,654.9,1279.1,654,1284.6z M2019.1,698.8
c-0.5-0.3-1-0.6-1.5-0.9c-1,1.2-2,2.4-3,3.6c-6.8,9-13,18.4-21.4,26.1c-11,10.1-20.5,21.7-32.4,30.8c-1.9,1.5-3.6,3.2-2.1,5.9
c2,3.5,4.1,0.5,5.7-0.4c15.4-9.7,30.8-19.6,46.1-29.5c2.3-1.5,4-3.1,4.4-6.2C2016.2,718.5,2017.7,708.6,2019.1,698.8z
M2017.1,1283.9c0.7-0.2,1.4-0.5,2.1-0.7c-1.5-10.1-3-20.2-4.4-30.4c-0.4-2.6-1.9-4.1-3.9-5.4c-15.6-10-31.3-20.1-46.9-30
c-1.5-1-3.3-2.8-5-0.5c-1.7,2.4-0.3,4.2,1.7,5.9c4,3.4,8.1,6.9,11.8,10.6C1988.4,1249.3,2005.3,1264.3,2017.1,1283.9z M1044.6,1034
c0.2,0,0.3,0,0.5,0c0-11-0.1-21.9,0.1-32.9c0.1-4.2-2-4.8-5.5-4.9c-4.5-0.1-4.6,2.6-4.6,5.9c0.1,22.3,1.2,44.5,2.4,66.8
c0.1,1,0.2,2,0.4,3c0.2,1.2,1.2,1.8,2.1,1.7c0.8-0.1,1.9-1,2.2-1.7c0.5-1.5,0.7-3.2,0.8-4.9C1043.5,1055.9,1044,1045,1044.6,1034z
M1636.4,1033.8c0.7,0,1.4,0,2.1,0c0-10.6,0-21.2,0-31.8c0-2.7,0.2-5.3-3.6-5.8c-4.7-0.6-6,0.4-6,6.1c-0.1,21.9,0.2,43.7,2.1,65.6
c0.1,1.3,0.1,2.7,0.6,3.9c0.3,0.7,1.5,1.6,2.2,1.5c0.7,0,1.7-1.1,1.9-1.8c0.5-1.9,0.7-3.9,0.7-5.9
C1636.4,1055,1636.4,1044.4,1636.4,1033.8z M1638.5,948.1c-0.2,0-0.5,0-0.7,0c-0.5-11.6-1-23.3-1.6-34.9c-0.1-1.9,0.2-4.8-2.3-4.9
c-3.3-0.1-2.8,3-3,5.2c-0.3,3-0.4,6-0.7,9c-1.5,19-1.4,37.9-1.4,56.9c0,5.3,2,6.8,6.4,5.8c3.8-0.8,3.3-3.6,3.3-6.2
C1638.5,968.8,1638.5,958.4,1638.5,948.1z M1045.1,948c-0.1,0-0.3,0-0.4,0c-0.6-11.9-1.2-23.9-1.9-35.8c-0.1-1.9-0.4-4.3-3.2-3.9
c-2.2,0.3-1.9,2.4-2,4c-1.3,22.5-2.3,45.1-2.5,67.7c0,3.7,1.2,5.4,5.1,5.4c4.1,0.1,5-1.9,5-5.5C1045,969.3,1045.1,958.7,1045.1,948
z M741.9,740.1c-4.9,5.5-6.3,11-8.3,16.2c-0.8,2-0.8,4,1.2,5.3c6.2,4.1,11.6,9.4,19.7,13C751.6,762.1,748.8,750.9,741.9,740.1z
M1931.7,1241.3c11.7-16.6,11.5-18.2-3-28.3c-2.7-1.9-4.9-4.5-9.1-4.9C1921.9,1219.6,1924.7,1230.7,1931.7,1241.3z M1931.6,740
c-6.6,10.9-9.8,22-12.1,33.9c4-0.9,6.1-3.1,8.5-4.8C1943.5,758.3,1943.6,757.2,1931.6,740z M742.1,1241.3
c6.6-11.1,9.8-21.9,12.2-34.3c-8,3.6-13.2,8.8-19.2,12.6c-2,1.3-2.3,3.1-1.6,5.1C735.6,1230.2,736.7,1236.2,742.1,1241.3z
M597.6,1464.4c16.9-6.5,17.3-8.9,14-24.4C605.8,1447.2,601.7,1455,597.6,1464.4z M2061.5,1442.5c-5.1,14.3,5.6,17.3,12.5,22.8
C2072.1,1457,2066.2,1451,2061.5,1442.5z M1781.6,836.9c-1,5.2-3.6,8.5-1.8,12.6C1785.7,846.2,1785.7,846.2,1781.6,836.9z
M895.4,849.3c-1.5-4.5-1.6-7.9-3.8-11.1C887.9,846.3,887.9,846.3,895.4,849.3z M630.1,1284.6c-0.4-5.3,4.1-10.2,1.8-17
C627.4,1273.6,629.9,1279.2,630.1,1284.6z M629.9,697.1c0.2,5.4-3,11.4,2.4,16.6C633.8,707.4,630.4,702.5,629.9,697.1z
M2044.5,1284.5c-1-5.2,1.5-10.3-2.8-15C2040.3,1275.4,2042.4,1279.6,2044.5,1284.5z M2041.9,712.1c4.1-4.8,1.6-10,2.2-14.6
C2043,702.1,2039.8,706.2,2041.9,712.1z M892.2,1143.9c0.4-4.2,3.5-7.3,1.6-11.5C887.6,1136,887.6,1136,892.2,1143.9z
M1779.6,1132.2c-1,2.9-0.4,4.8,0.4,6.5c0.7,1.5,0.4,3.6,2.3,4.6C1785.7,1135.3,1785.7,1135.3,1779.6,1132.2z M729.5,774
c1.5,1.6,2.6,4,5.8,3.2C733.9,774.7,732.2,773.5,729.5,774z M737.9,778.4c-0.2,0.2-0.5,0.4-0.7,0.6c0.2,0.2,0.4,0.6,0.6,0.6
c0.2,0,0.5-0.3,0.7-0.5C738.4,778.8,738.1,778.6,737.9,778.4z"/>
<path fill="none" d="M534.9,1353.5c4.5,11.6,7.4,22.2,12.2,32c4.7,9.6,9.8,18.8,17.1,26.8c10.6,11.6,28.8,15.5,42.7,2.9
c1.4-1.3,3.1-2.4,5.5-4.2c1.4,7.2,1.7,13.2-2.5,19.1c-8.4,11.7-14.8,24.4-20.8,37.4c-1.7,3.7-3.7,6.7-7.3,9.1
c-52.7,33.7-105.3,67.6-158,101.4c-1.1,0.7-2.1,1.7-4.1,1.1c0-195.1,0-390.4,0-585.6c2.1-1.2,4.1-0.9,6-0.8
c21.6,0.5,43.3,1.1,64.9,1.3c5.3,0,6.7,1.8,6.7,6.9c0.5,40,1.4,79.9,3.9,119.8c2.4,38.2,5.6,76.4,10.6,114.3
c4.6,34.7,10.1,69.1,19.2,103c1.1,4,0.6,7.1-2.3,10c-0.9,0.9-1.7,2.1-2.6,3.1c-22.1,25.1-40.6,52.7-57.5,81.5
c-11.6,19.7-19.9,41-28.9,61.9c14.9-23.9,25.7-50.1,41.8-73.5C497.6,1397.6,514.9,1375.3,534.9,1353.5z"/>
<path fill="none" d="M534.6,627.2c-40-41.2-69.6-89-94.2-140.1c0.7,6.2,4,11.2,6.2,16.6c12.7,31.7,30.5,60.6,49.6,88.7
c9.3,13.7,20.4,26.3,31,39.1c3.8,4.5,5,8.7,3.4,14.4c-9.6,35.4-15,71.6-19.6,107.9c-9.4,74.5-12.5,149.3-13.7,224.3c0,1-0.1,2,0,3
c0.6,4.7-1.2,6.5-6.1,6.4c-8.3-0.2-16.7,0.2-25,0.4c-13.7,0.3-27.3,0.4-41,1c-4.8,0.2-6.8-1.2-6.2-6.1c0.3-2,0-4,0-6
c0-187.9,0-375.8,0-563.7c0-3.5,0-7.1,0-10.7c4.2-0.1,6.3,2.3,8.7,3.9c50.4,32.4,100.8,64.9,151.3,97.1c5,3.2,8.4,7.1,10.8,12.4
c5.1,11.2,10.4,22.4,17.8,32.3c4.9,6.6,6.7,13.5,5.1,21.9c-3.6-0.1-5.6-3.1-7.9-5c-12-9.8-28.1-7.7-38.9,2.8
c-12.6,12.2-19,27.9-25.2,43.7C538.8,616.5,538.2,622.2,534.6,627.2z"/>
<path fill="none" d="M2060.9,1411.1c7.4,4.3,12.3,10.3,20.6,11.4c9.8,1.4,17.7-1.4,24.8-7.5c8.5-7.3,13.9-16.9,18.8-26.8
c5.1-10.4,8.6-21.5,12.7-32.5c24.7,24.7,73.3,78.9,107.2,169.7c-1.1-8.8-3.5-17.2-6.8-25.4c-13.2-33.9-29.3-66.4-49.3-96.8
c-12.8-19.5-27.4-37.7-42.4-55.5c-3.4-4-4.8-7.1-3.5-12.3c12.8-48.8,19.1-98.6,24-148.7c5.9-59.7,8.4-119.6,8.9-179.7
c0-1.3,0-2.7,0.1-4c0.2-7,0.2-7.1,7-7.4c20-0.7,40-1.3,60-1.9c2.3-0.1,4.7,0,7-0.1c3.5-0.2,4.8,1.4,4.6,4.8c-0.1,2,0,4,0,6
c0,188.3,0,376.6,0,565c0,3.2,0,6.4,0,9.8c-4.5-0.2-6.9-2.8-9.5-4.5c-47.7-30.6-95.2-61.4-143-91.7c-9-5.7-16-11.6-19.9-22.4
c-3.7-10.1-11.2-18.9-17.1-28.2C2060.8,1426.2,2058.6,1419.6,2060.9,1411.1z"/>
<path fill="none" d="M2244.8,454.3c-23,66.2-59.6,121.8-106.6,172.4c-4.5-11.6-7.7-22.6-12.8-32.9c-4-8.1-8.4-15.8-14.4-22.7
c-12.6-14.4-29.9-17.7-45.6-3.8c-1.1,1-2,2.7-4.6,2.6c-1.4-5.5-1.1-10.5,2-15.7c8.8-14.9,18.9-29,25.8-45c1-2.3,2.7-3.7,4.7-4.9
c52.7-33.9,105.3-67.7,158-101.5c0.5-0.3,1.3-0.2,2.1-0.4c2.3,2.1,1.2,4.9,1.2,7.3c0.1,189.6,0.1,379.3,0.1,568.9
c0,1.7-0.1,3.3,0,5c0.2,3.6-1.7,4.6-4.9,4.5c-5-0.2-10-0.3-15-0.5c-17-0.5-34-1-51-1.4c-7.4-0.2-7.5-0.2-7.6-7.8
c-0.5-35.3-1.1-70.6-3.1-105.9c-2.6-45.9-6.4-91.7-12.8-137.3c-4.2-30.3-9.4-60.5-17.3-90.2c-1.1-4-0.2-6.6,2.5-9.8
c27.4-31.6,51.5-65.6,70.4-102.8c11.3-22.2,21.7-44.9,28.2-69.2C2244.6,460.9,2245.4,458.8,2244.8,454.3z"/>
<path fill="none" d="M553.7,662.8c3.8,0.2,5.2,2.8,6.9,4.6c14.5,15.2,30.3,29,46.9,41.9c4.1,3.2,5.8,6.4,5.8,11.7
c-0.2,85.6-0.1,171.2-0.1,256.8c0,7.5-0.1,7.7-7.6,7.8c-25,0.4-50,0.6-74.9,1.1c-4.3,0.1-5.9-1.2-5.7-5.6c0.4-13,0.2-26,0.5-39
c1.5-60.3,4.7-120.5,11.8-180.4c3.7-31.7,8.3-63.3,15.4-94.5C553,665.6,553.4,664.1,553.7,662.8z"/>
<path fill="none" d="M553.6,1319.7c-22.4-107.4-27.4-215.3-28.7-323.3c3.3-2.5,6.4-1.5,9.3-1.5c23.3,0.3,46.7,0.7,70,1.1
c9.5,0.1,9.2-1.2,9.2,9.2c0,72.3,0,144.7,0,217c0,13.3-0.1,26.7,0.1,40c0,3.6-0.7,6.2-3.8,8.6c-18.7,14.5-36.3,30.3-52.7,47.4
C556.3,1318.5,555.6,1318.7,553.6,1319.7z"/>
<path fill="none" d="M2119.8,1319.2c-6.7-6.5-12.1-12.3-18-17.5c-11.8-10.2-23.8-20.2-36-30.1c-4.2-3.4-5.9-7.1-5.9-12.7
c0.2-84.6,0.1-169.1,0.1-253.7c0-8.4,0.1-8.5,8-8.5c24.3-0.2,48.6-0.4,72.9-0.5c7.5,0,7.7,0,7.6,7.7c-0.3,45.6-1.9,91.2-5.1,136.7
c-2.7,39.9-6.7,79.6-12.5,119.1C2128.1,1279,2124.6,1298.3,2119.8,1319.2z"/>
<path fill="none" d="M2120.4,664.4c12.3,57.5,18.7,115.7,22.8,174.2c3.1,44.9,4.7,89.8,5.2,134.8c0,2.3-0.1,4.7,0.1,7
c0.3,4-1.7,5.2-5.4,5.1c-6.7-0.2-13.3-0.3-20-0.3c-18.3-0.1-36.7-0.2-55-0.4c-8-0.1-8.2-0.1-8.2-8.2c0-52.3,0-104.6,0-157
c0-32,0.2-64-0.2-96c-0.1-6.3,2-10.3,6.8-14.3c17.1-14.3,35.1-27.7,50.2-44.2C2117.5,664.3,2118.3,663.8,2120.4,664.4z"/>
<path fill="none" d="M878.7,1138.5c2,4.9,3.2,9.1,5.3,12.8c4,7.2,10.5,7.1,14.6,0c3-5.3,4.3-11.2,5.2-17.1c0.8-4.7,3.2-7.1,7.8-8.9
c37.6-14.8,76.3-25.6,115.5-34.7c5-1.1,6.6,0.4,7.3,4.9c0.6,3.5,0.7,8.4,5.3,8.5c5.3,0.1,5.5-5.2,5.8-9c0.5-7.1,3.9-10,10.9-11.2
c40-7,79.8-14.9,120.4-18c2.4-0.2,5.9-2.1,6.9,1.5c1,3.7,1.8,7.8,0.4,11.7c-0.7,1.8-2.9,1.2-4.4,1.5c-9.5,1.6-19.1,2.8-28.6,4.6
c-39.6,7.4-78.6,17-117,29.2c-59.9,19-116.5,45.2-170.8,76.4c-41.6,24-80.2,52.3-117.2,83.1c-39.4,32.8-74.3,69.6-105.4,110.2
c-3,3.9-6.2,7.7-9.4,11.6c-0.1,0.2-0.6,0.2-2.1,0.5c0.2-7-1.1-13.6,1.5-20.3c7.6-19.2,12.9-39.2,16.7-59.5
c1.8-9.6,6.2-17.5,12-24.8c17.2-21.7,36.5-41.5,58.8-58.9c3.4,2.2,3.5,6.6,5.4,9.8c4,7,7.8,14.4,17.4,14.4
c9.6,0,13.8-7.2,17.3-14.4c6.5-13.3,9.4-27.7,12.2-42c0.8-3.9,2.2-6.5,5.7-8.6C808.9,1171.4,842.7,1153.5,878.7,1138.5z"/>
<path fill="none" d="M1032.8,891.9c-42-9-82.2-20.4-121.3-35.7c-4.8-1.9-6.9-4.4-7.7-9c-0.9-5.6-2.3-11.1-4.8-16.2
c-1.5-3.2-3.7-5.9-7.3-6.1c-4.1-0.2-6.8,2.5-8.3,6.1c-1.6,3.6-2.8,7.4-4.7,12.2c-11.4-5.2-22.6-10-33.5-15.2
c-23.1-11.1-45.4-23.7-67.3-37.1c-4.3-2.6-6.6-5.5-7.5-10.7c-2.1-11.4-4.6-22.8-8.9-33.7c-2.1-5.3-4.2-10.5-7.8-15
c-7.6-9.4-17.5-9.4-25.3,0.1c-4.2,5.2-6,11.7-9.8,18.2c-11.1-9.9-22.3-18.8-32.2-28.9c-10.2-10.4-19.5-21.8-28.7-33.3
c-5.6-7-8.9-15.3-10.6-24.4c-3.7-19.3-8.6-38.3-16-56.5c-2.9-7.1-1.4-14.1-0.5-21.8c3.9,4.8,7.9,9.4,11.6,14.3
c23.7,32,51,60.7,80.1,87.7c19,17.6,39.4,33.9,59.9,49.7c45,34.7,93.8,63.3,144.7,87.9c39.2,18.9,79.8,34.7,121.6,47.1
c41.2,12.2,82.8,22.3,125.4,28.2c1,0.1,2,0.2,3,0.4c7.3,1.2,10.5,7.7,7,14.2c-1.3,2.3-3.4,1.7-5.3,1.6c-8-0.7-15.9-1.5-23.8-2.4
c-32.8-3.7-65.1-10.2-97.5-15.7c-7.1-1.2-11.5-3.9-11.7-11.6c0-1.3-0.4-2.6-0.8-3.9c-0.8-2.6-1.9-5.3-5.3-4.9
c-2.7,0.3-3.7,2.6-4.4,4.9C1034.4,885.3,1033.7,888.2,1032.8,891.9z"/>
<path fill="none" d="M1953.8,1233.7c3.8,1,5.8,3.4,8,5.3c21,19.1,40.7,39.4,57.6,62.2c3,4.1,4.9,8.4,6,13.5
c4.5,20.5,9.3,40.9,17.1,60.4c1.3,3.1,2.4,6.3,1.8,9.7c-0.6,3.6,2.4,8.9-2.1,10.4c-2.5,0.8-5.4-3.9-7.3-6.7
c-35.7-51.5-81-93.6-129.4-132.6c-40.8-32.9-84.7-61.2-131-85.6c-39.5-20.8-80.6-37.9-123-52c-45.9-15.2-92.6-26.8-140.3-34.3
c-6.2-1-12.4-2.7-18.7-3.1c-6.5-0.4-5.5-4.5-5.2-8.4c0.4-5.4,2.1-7.1,7.5-6.6c8.3,0.7,16.5,1.9,24.8,2.7
c33.2,3.4,65.8,10.1,98.6,15.7c6.3,1.1,9.2,3.4,9.6,10c0.2,3.6,0.3,9.3,5.3,9.6c6,0.4,5.5-5.6,6.3-9.5c0.7-3.9,2.4-5,6.2-4
c39.4,9.7,78.9,19.2,116.7,34.3c4.5,1.8,6.8,4.2,7.5,9.1c0.9,5.9,2.2,11.8,5.1,17.2c1.7,3.1,3.8,5.6,7.6,5.6c3.8,0,6-2.5,7.6-5.6
c0.9-1.8,1.7-3.6,2.3-5.5c2.8-7.7,2.9-7.9,9.9-4.7c32.6,15.3,64.6,31.4,95,50.8c3.4,2.2,5.1,4.7,5.8,8.6
c2.1,12.1,4.9,24.1,9.4,35.7c1.9,5,4.1,9.8,7.5,14c7.5,9.4,17.7,9.4,25.3,0C1948.9,1245.3,1950.9,1239.5,1953.8,1233.7z"/>
<path fill="none" d="M2043.6,585.1c0.9,7.5,1.7,14.1-0.8,20.4c-8,20.2-13.1,41.2-17.6,62.4c-0.9,4.4-2.6,8.3-5.2,11.7
c-17.8,23.8-38.4,45.2-60.6,64.9c-1.5,1.3-3.2,2.3-5.1,3.8c-2.9-5.3-5.1-10.4-8.2-14.9c-8-11.8-19.3-11.8-27.5-0.1
c-7.4,10.6-11.4,23-12.8,35.4c-1.7,14.6-9.4,22.2-21.3,29.3c-25.5,15.2-52.1,28.1-78.7,41.2c-2.4,1.2-5,2-7.3,3.1
c-2.7,1.3-4.3,0.7-5.2-2.2c-1-3.2-1.9-6.4-3.4-9.3c-4.2-7.7-10.8-7.7-15.1-0.1c-2.5,4.4-3.9,9.3-4.5,14.2c-0.8,7-4.2,10.7-11,13.3
c-27.5,10.2-55.4,18.9-83.9,25.7c-10,2.4-20.1,4.8-30,7.5c-3.8,1-5.4,0-6.1-3.9c-0.7-4-0.2-9.9-6.2-9.6c-5.1,0.2-5.3,6-5.4,9.5
c-0.1,7.5-4.3,9-10.5,10.2c-39.6,7.4-79.4,13.9-119.5,18.1c-2.8,0.3-6.7,2.3-8.1-1.3c-1.4-3.6-1.7-7.9-0.3-11.7
c0.9-2.4,3.9-2,6.2-2.4c13.8-2.4,27.6-4.4,41.3-7c42.1-8.2,83.5-19.3,124.2-33.1c57.6-19.6,112.2-45.7,163.9-77.3
c38.1-23.2,73.8-50.1,107.6-79.4c37.9-32.8,72.6-68.6,102.5-109C2037.5,591.3,2039.4,587.8,2043.6,585.1z"/>
<path fill="none" d="M777.2,1163.8c-0.9-7.9,1.2-15.4,2-23c4.4-45.1,6.3-90.3,6.6-135.6c0.1-8.6,0.2-8.7,8.8-8.7
c22-0.1,44-0.1,65.9,0c8.3,0,8.4,0.1,8.5,7.9c0.2,37.7,2,75.2,6.5,112.6c0.5,3.8,0.3,6-3.9,7.5c-31.7,11.3-62.2,25.3-92.6,39.6
C778.8,1164.3,778.4,1164.1,777.2,1163.8z"/>
<path fill="none" d="M776.8,817.8c3.1-0.6,4.7,0.6,6.5,1.4c28.2,13.5,56.6,26.5,86.2,36.8c5.4,1.9,6.7,4.3,6,10.1
c-4.5,36.7-6.3,73.6-6.4,110.6c0,8.1,0.7,8.3-8.1,8.3c-22.6-0.1-45.3,0-67.9-0.1c-7,0-7.1-0.2-7.2-7.3c-0.4-28.6-0.9-57.3-2.7-85.9
c-1.3-21.3-2.9-42.5-5.5-63.7C777.2,824.9,777.1,821.6,776.8,817.8z"/>
<path fill="none" d="M1896.4,1163.9c-3.3,0.5-5.7-1.5-8.4-2.8c-27.2-13.3-54.8-25.7-83.3-36.1c-7.4-2.7-7.2-2.9-6.3-10.6
c4.4-35.8,5.3-71.8,6.4-107.7c0.1-1.7,0.1-3.3,0-5c-0.3-3.8,1.3-5.4,5.2-5.3c6.3,0.2,12.7,0.1,19,0.1c16.7,0,33.3,0,50,0.1
c8.4,0,8.4,0.1,8.6,8.8c0.8,30.3,1.2,60.6,3.3,90.9c1.4,19.9,2.9,39.9,5.3,59.7C1896.5,1158.6,1897.4,1161.2,1896.4,1163.9z"/>
<path fill="none" d="M1896.8,817.7c-1.3,13.9-2.7,27.7-3.9,41.6c-3.5,38.5-4.5,77-5.2,115.6c0,1.7-0.2,3.3-0.1,5
c0.4,4-1.5,5.5-5.2,5.1c-1-0.1-2,0-3,0c-22.6,0-45.3-0.1-67.9,0.2c-5,0-6.8-1.2-6.8-6.6c-0.4-28.3-1.2-56.5-3.6-84.7
c-0.8-8.9-1.6-17.9-2.7-26.8c-0.9-7.6-1.1-7.8,6.5-10.6c23-8.7,45.8-17.8,67.9-28.7c6-2.9,12-5.7,18-8.6
C1892.5,818.3,1894.3,817.4,1896.8,817.7z"/>
<path fill="none" d="M908.3,1109.6c-1-4.4,0.4-8.6,0.8-12.8c2.5-30.9,4-61.8,4-92.9c0-7.4,0.2-7.5,7.9-7.5
c33.3-0.1,66.7-0.2,100-0.2c7.8,0,8.1,0.1,8,7.4c-0.3,22,1,44,2.3,65.9c0.3,4.5-1,6.5-5.5,7.4c-38.7,8.5-76.4,20.9-114.2,32.5
C910.7,1109.8,909.7,1109.6,908.3,1109.6z"/>
<path fill="none" d="M1765.3,1110.4c-9.6-3-18.1-5.6-26.6-8.4c-29.1-9.6-58.7-17.4-88.5-24.5c-8.9-2.1-8.1-1.4-7.6-10.4
c1.2-21,1.6-42,2.3-62.9c0.2-7.7,0.2-7.9,7.5-7.9c33.3,0,66.7,0.1,100,0.2c7.8,0,8.2,0.3,8.2,7.7c0.2,34,1.7,68,4.8,101.8
C1765.4,1106.9,1765.3,1107.9,1765.3,1110.4z"/>
<path fill="none" d="M908.8,872.2c2.8-1.1,5,0.8,7.5,1.5c36.1,10.3,71.6,22.9,108.5,30.5c5.5,1.1,6.7,3.6,6.4,8.8
c-1.3,21.9-2.6,43.9-2.1,65.9c0.1,4.7-1.2,6.5-6.1,6.4c-34.7-0.2-69.3-0.2-104-0.1c-4.1,0-6.1-1-6-5.6c0.4-35.6-2-71.2-4.8-106.7
C908.2,872.8,908.6,872.5,908.8,872.2z"/>
<path fill="none" d="M1765.6,872.4c-0.9,10.7-1.8,20.9-2.5,31.2c-1.7,24.6-2.4,49.2-2.6,73.8c-0.1,7.7-0.2,7.8-7.7,7.8
c-33.6,0.1-67.2,0.1-100.8,0.1c-6.9,0-6.9-0.2-7.1-7.1c-0.8-21.9-1.3-43.9-2.6-65.8c-0.3-5.4,1.8-7,6.2-8
c14.5-3.5,29-7.1,43.5-10.8c22.5-5.7,44.5-13.5,66.7-20.3C1760.5,872.7,1762.3,871.8,1765.6,872.4z"/>
<path fill="none" d="M725.8,1192.5c-5.9-34.3-8.8-67.4-11-100.6c-1.8-27.6-2.2-55.2-3-82.8c-0.1-2.3,0.1-4.7-0.1-7
c-0.4-4.4,1.6-5.8,5.9-5.7c9,0.2,18,0,27,0c6.7,0,13.3-0.1,20,0c7.2,0.1,7.4,0.2,7.3,7.1c-0.5,23.6-1,47.2-2,70.8
c-1.2,27.2-3.3,54.4-6.7,81.5c-0.2,1.3-0.3,2.6-0.4,4c-0.4,9.2-3.4,16-13,19.7C741.7,1182.7,734.5,1187.7,725.8,1192.5z"/>
<path fill="none" d="M1946.2,1191.7c-10.1-5.6-20.4-11.5-30.8-17.2c-3.3-1.8-3.1-5-3.5-7.8c-1.4-10.2-2.4-20.5-3.5-30.7
c-4.7-43.8-6-87.7-6.7-131.6c-0.1-7.7,0.1-7.8,7.7-7.8c15.3-0.1,30.6,0.1,46-0.1c4.6-0.1,6.5,1.1,6.4,6.1
c-0.5,34-1.2,67.9-3.7,101.8c-2,26.9-4.5,53.7-8.9,80.4C1948.7,1187,1949.1,1189.5,1946.2,1191.7z"/>
<path fill="none" d="M1947.7,789.1c5.8,32.7,8.6,63.8,10.8,95c2.1,30.6,2.8,61.2,3.1,91.9c0.1,9.6,0.3,8.9-8.3,8.9
c-15,0-30-0.1-45,0.2c-5.3,0.1-6.9-1.8-6.7-7c0.9-25.3,1.3-50.6,2.5-75.9c1.3-27.3,3.9-54.4,6.9-81.6c0.9-8.3,3.3-14.2,11.5-17.8
C1930.8,799.2,1938.6,794.1,1947.7,789.1z"/>
<path fill="none" d="M726,789.2c11.3,6.2,21.7,11.9,32.2,17.5c2.9,1.6,2.8,4.2,3.2,6.8c2.7,15.8,4.1,31.7,5.5,47.6
c3.3,37.2,4.5,74.5,5,111.8c0,2.3,0,4.7,0.1,7c0.1,3.1-0.8,5-4.4,5c-17-0.1-34-0.1-51-0.1c-3.3,0-4.8-1.5-4.7-4.8
c0.2-5.7,0-11.3,0.2-17c1-42.6,2.7-85.2,7.3-127.7C720.9,820.5,722.6,805.7,726,789.2z"/>
<path fill="none" d="M1625.1,910.5c-0.6,9.4-1.4,18-1.7,26.5c-0.4,13-0.4,26-0.6,38.9c-0.1,9.3-0.2,9.4-9.2,9.4
c-30.6,0.1-61.3,0.1-91.9,0.1c-8.3,0-16.7-0.1-25,0.1c-4.2,0.1-5.9-1.3-5.9-5.7c0-15-0.1-30-0.7-44.9c-0.2-4.8,2.5-5.5,6-6
c10.9-1.3,21.8-3.6,32.7-3.7c23.9-0.2,46.7-6.5,70-10C1607.1,913.9,1615.2,912.2,1625.1,910.5z"/>
<path fill="none" d="M1625.2,1071.1c-23.5-4.2-45.7-8.1-67.9-12.1c-7.9-1.4-15.8-2.4-23.8-2.5c-12.7-0.1-25.1-2.6-37.7-4
c-4-0.4-5.8-1.6-5.6-6.1c0.5-14.6,0.6-29.3,0.6-43.9c0-4.5,1-6.5,6.1-6.5c39.9,0.2,79.9,0.2,119.8,0.1c5,0,6.1,2.1,6.1,6.6
C1622.9,1025,1623,1047.3,1625.2,1071.1z"/>
<path fill="none" d="M1049.3,1070.4c-1-5.9-0.1-11.8,0.3-17.7c0.9-16.3,1.4-32.6,1.4-48.9c0-7.3,0.2-7.5,6.8-7.5
c39.3-0.1,78.5-0.1,117.8-0.1c7,0,7.1,0.2,7.2,7.2c0.2,14,0,27.9,0.5,41.9c0.2,5.2-1.5,6.6-6.6,7c-16.9,1.3-33.8,2.7-50.6,5.2
c-23.3,3.4-46.7,6.8-69.7,12.3C1054.2,1070.3,1051.9,1071.4,1049.3,1070.4z"/>
<path fill="none" d="M1049.8,910.2c22.1,5.7,44.8,8.7,67.4,12.5c16.4,2.8,33.1,3.3,49.6,5.2c16.4,1.9,16.4,1.5,16.3,17.4
c-0.1,11-0.3,22-0.2,32.9c0.1,4.9-1,7.2-6.6,7.2c-39.9-0.2-79.9-0.2-119.8-0.1c-3.4,0-5.7-0.6-5.6-4.8c0.6-22.6-1.3-45.2-1.9-67.8
C1049,912,1049.4,911.4,1049.8,910.2z"/>
<path fill="none" d="M1260.5,985.4c-21.6,0-43.2,0-64.9,0c-8,0-8.2-0.1-8.1-8.4c0.2-13.6,0.6-27.3,0.8-40.9c0.1-4.2,1.4-5.6,6-5.1
c41.7,4.1,83.5,6.2,125.5,6.2c2.3,0,4.7,0,7,0c7.5,0.2,7.5,0.2,7.6,7.9c0.1,10.3,0.1,20.6,0.1,30.9c0,9.3,0,9.4-9.1,9.4
C1303.8,985.4,1282.2,985.4,1260.5,985.4z"/>
<path fill="none" d="M1485,931.5c0.3,16.2,0.5,32.7,0.8,49.3c0.1,3.4-1.7,4.8-4.9,4.6c-2-0.1-4,0-6,0c-41.6,0-83.2,0-124.9,0
c-2,0-4-0.1-6,0c-4,0.3-5.1-1.8-5-5.5c0.1-11.7,0-23.3,0.1-35c0.1-7.6,0.1-7.8,7.6-7.8c17.3,0.2,34.6-0.6,51.9-1.1
c26.3-0.7,52.5-3.1,78.7-5.1C1479.6,930.9,1481.8,930,1485,931.5z"/>
<path fill="none" d="M1485,1051.1c-35.4-3-69.9-5.3-104.5-6.4c-11.6-0.4-23.3-0.5-34.9-0.4c-4.4,0-6.7-0.8-6.5-6
c0.4-12,0.3-24,0-36c-0.1-4.8,1.6-6.3,6.3-6.2c45,0.1,89.9,0.1,134.9,0c3.5,0,5.8,0.7,5.7,4.8
C1485.5,1017.5,1485.3,1034.1,1485,1051.1z"/>
<path fill="none" d="M1188.4,1050.6c-0.3-17.2-0.5-33.7-0.8-50.3c0-2.9,1.6-4.1,4.4-4.1c1.7,0,3.3,0,5,0c42.6,0,85.1,0,127.7,0
c1.7,0,3.3,0.1,5,0c3.3-0.2,4.9,1.2,4.9,4.6c-0.1,13-0.1,25.9,0,38.9c0,3.6-1.7,4.7-4.9,4.5c-0.7,0-1.3,0-2,0
C1281.6,1043.7,1235.9,1047.2,1188.4,1050.6z"/>
<path fill="none" d="M2010.7,1230.1c-5.1-1.9-9-5.1-13.1-7.7c-9.9-6.1-19.6-12.6-29.8-18.3c-4.8-2.6-5.5-5.6-4.5-10.4
c3.9-20,6.2-40.1,8.2-60.4c4-41.2,5.8-82.4,6.1-123.8c0-3,0.1-6,0-9c-0.1-3,1.1-3.8,4.1-4.1c16.5-1.5,16.7-1.4,16.8,14.7
c0.4,66.4,3.9,132.5,10.7,198.6C2009.9,1216.3,2011.6,1222.7,2010.7,1230.1z"/>
<path fill="none" d="M2011.2,752.5c-1.6,16.9-3.2,33.8-4.8,50.6c-3,30.8-4.6,61.7-6.1,92.6c-1.3,27.3-1.5,54.5-1.8,81.8
c-0.1,7.1-0.2,7.3-7.1,7.3c-16.3,0-13.6,1.3-13.7-12.8c-0.5-54.3-3.2-108.4-10.6-162.2c-1.1-7.9-2.5-15.8-3.9-23.6
c-0.7-3.6-0.1-6,3.5-8.2c13.4-8,26.6-16.4,39.8-24.7C2007.6,752.8,2008.5,751.7,2011.2,752.5z"/>
<path fill="none" d="M662.1,1231.5c1.9-20.4,3.7-38.6,5.4-56.8c3-31.9,4.7-63.8,6-95.8c1-24.6,1.6-49.3,1.6-74
c0-8.3,0.1-8.3,7.9-8.4c1.3,0,2.7,0,4,0c8.6,0.1,8.7,0.1,8.8,8.7c0.4,40,1.8,80,5.4,119.8c2,22.9,4.4,45.8,9,68.3
c1.1,5.5,0.2,8.5-5.1,11c-12.7,6-23.9,14.5-35.5,22.2C667.7,1227.9,665.8,1229.1,662.1,1231.5z"/>
<path fill="none" d="M662.9,752.1c2.6-0.9,3.9,1,5.4,2.1c12.4,8.4,24.6,17.1,38,23.9c3.3,1.7,5,3.6,4.2,7.7
c-11.3,61.8-13.7,124.4-14.8,187c-0.2,13.5,2.2,12.1-12.3,12.1c-8.4,0-8.5-0.1-8.5-8.9c0.1-52.7-2.6-105.3-6.4-157.8
c-1.5-20.6-4-41.1-6-61.7C662.5,755.1,661.9,753.4,662.9,752.1z"/>
<path fill="none" d="M2044.3,728.6c0,3.4,0,6.3,0,9.2c0,78.9,0,157.9,0,236.8c0,1.7-0.1,3.3,0,5c0.4,3.9-1.3,5.4-5.2,5.3
c-5.9-0.2-11.9,0-18.3,0c0-26.3,0.3-51.6,1.3-76.8c1.7-42.3,4.4-84.5,8.9-126.5c1.6-14.9,3.9-29.7,5.6-44.6
C2037.2,732.4,2038.9,729.7,2044.3,728.6z"/>
<path fill="none" d="M2044.3,1253c-5.6-1.2-7-4.1-7.7-8.6c-3.1-19.4-5.2-38.9-7-58.5c-5.6-59-8.7-118.2-8.7-177.6
c0-13.6-1.4-11.7,11.5-11.8c13.3,0,11.9-1.5,11.9,11.5c0,78.6,0,157.2,0,235.8C2044.3,1246.6,2044.3,1249.3,2044.3,1253z"/>
<path fill="none" d="M629.2,728.7c5.8,1.6,7.4,4.6,7.9,9c2.3,21.2,5.3,42.2,7.3,63.4c5.3,55.3,7.7,110.8,8.5,166.4
c0,3.3,0.1,6.7,0,10c-0.1,7.5-0.1,7.5-7.7,7.7c-16,0.5-16,0.5-16-15.3c0-76.5,0-153.1,0-229.6C629.2,736.7,629.2,733,629.2,728.7z"
/>
<path fill="none" d="M629.2,1253.2c0-6,0-10.6,0-15.3c0-75.7,0-151.3,0-227c0-2,0-4,0-6c-0.1-9.2-0.3-8.7,8.7-8.6
c15.1,0,15.2,0,15,15.6c-0.7,62.7-3.8,125.2-10.3,187.6c-1.5,14.9-3.9,29.7-5.5,44.6C636.6,1248.7,634.5,1251.1,629.2,1253.2z"/>
<path fill="none" d="M2060,697.5c0-15.9,0-29.5,0-43c0-3.7,0.3-7.3-0.1-11c-2-19,6.4-34.8,15.3-50.5c1-1.7,2.3-3.2,3.6-4.7
c5.4-6.2,8.1-6.2,13.4,0.4c6.3,7.8,10.3,17.1,13.7,26.4c3.2,8.7,5.9,17.6,8.7,26.5c1.3,4,1,7.1-2.4,10.7
C2096.7,668.2,2079,681.6,2060,697.5z"/>
<path fill="none" d="M613.2,1283.1c0,24.5,1.5,47.6-0.5,70.3c-1.4,15.4-9,29.8-19.7,41.7c-3.8,4.2-5.7,3.9-10-0.4
c-4.3-4.3-7.2-9.6-9.8-15c-6.7-13.8-11.2-28.4-15.3-43.2c-0.8-3-1.3-5.6,1.5-8.1C576.3,1313,592.4,1296.7,613.2,1283.1z"/>
<path fill="none" d="M2060,1284.4c19.3,15.5,36.8,29.3,52.5,45.2c2.6,2.6,3.7,5.1,2.6,8.8c-5.1,17.2-10,34.5-19.7,50
c-7.7,12.2-12.4,12.7-19.8,0.6c-9.5-15.4-16.9-31.4-15.8-50.5C2060.7,1321.2,2060,1303.9,2060,1284.4z"/>
<path fill="none" d="M611.9,698.2c-8.5-6.7-17.1-12.8-25-19.7c-9.1-7.8-17.5-16.4-26.3-24.5c-2.3-2.2-4-4.1-3-7.7
c5.3-18.2,10.3-36.5,20.4-52.8c7.8-12.7,12.8-13,20.4-0.3c8.9,14.9,16.3,30.2,15.1,48.6c-1,15.9-0.2,31.9-0.3,47.9
C613.2,692.2,614,694.9,611.9,698.2z"/>
<path fill="none" d="M1788.2,864c4.1,27,4.9,53.5,6.1,80.1c0.5,11.3,0.2,22.6,0.7,33.9c0.2,5.2-1.6,7.4-6.7,7.3
c-4.7-0.1-9.3-0.3-14,0c-4.6,0.3-5.5-2-5.4-6c0.2-26.3,1.3-52.5,3.2-78.8c0.5-7.6,1.2-15.2,2.1-22.8
C1775.6,867.2,1775.7,867.2,1788.2,864z"/>
<path fill="none" d="M885.3,1118.4c-5.1-39.1-6.6-77.6-6.6-116.3c0-4.2,1.3-5.9,5.7-5.8c25,0.7,20.1-4.1,19.8,19.4
c-0.4,29.6-1.5,59.3-5,88.8C897.9,1115.4,898.1,1115.4,885.3,1118.4z"/>
<path fill="none" d="M885.7,863.5c12.4,2.1,12.2,2.1,13.5,13.8c3.8,33,4.6,66.2,5.3,99.4c0.2,8.3,0,8.5-7.9,8.4
c-5.6-0.1-12.6,2.4-16.3-1.2c-3.8-3.6-1.3-10.7-1.3-16.2c0.1-33.9,1.8-67.7,6.1-101.4C885.1,865.4,885.4,864.5,885.7,863.5z"/>
<path fill="none" d="M1788.4,1117.5c-4,0.3-6.2-1.5-8.7-2.2c-3.2-0.9-4.6-2.9-4.9-6.5c-0.7-10.3-2.2-20.5-2.9-30.8
c-1.5-24.6-2.8-49.2-2.8-73.9c0-7.5,0.1-7.8,7.7-7.6c5.9,0.2,13.3-2.7,17.2,1.6c3.4,3.7,1,10.9,0.9,16.5
C1794.3,1048.8,1792.8,1083.1,1788.4,1117.5z"/>
<path fill="none" d="M1481.1,1078.1c-10.7,0.3-20.5-1.5-30.3-2.6c-34.1-3.7-68.2-5.9-102.5-5.6c-1.3,0-2.7-0.1-4,0
c-4.9,0.5-5.1-2.5-5.2-6.2c-0.1-4,0.7-6,5.4-6c20,0.1,39.9-0.2,59.9,1c22.9,1.4,45.8,3,68.7,4.7c7.9,0.6,7.8,0.9,8.2,8.7
C1481.2,1073.7,1481.1,1075.4,1481.1,1078.1z"/>
<path fill="none" d="M1480,903c2,3.3,0.7,6.3,1,9.2c0.4,4.3-1.6,5.6-5.8,5.9c-42.2,3.3-84.3,6.2-126.7,5.9c-1.3,0-2.7-0.1-4,0
c-4.5,0.5-5.5-1.7-5.6-5.8c-0.1-4.5,0.9-6.6,6-6.5c29.7,0.7,59.2-1.5,88.8-3.9C1449.3,906.5,1464.7,904.6,1480,903z"/>
<path fill="none" d="M1192.2,1078.2c1-14.7,0.9-15,13.5-15.8c40.5-2.9,80.9-5.4,121.6-4.6c7.3,0.1,7.4,0.5,7.3,7
c-0.1,4-1.7,5.2-5.4,5.1c-6-0.2-12-0.1-18,0c-37.3,0.2-74.3,3.2-111.3,7.8C1197.6,1078,1195.3,1078,1192.2,1078.2z"/>
<path fill="none" d="M1192.6,903.2c17.2,2.2,33.7,3.8,50.2,5c28.2,2.1,56.4,4.4,84.7,3.6c3.8-0.1,7.3-0.4,7.1,5.4
c-0.3,6.3-0.2,6.8-6.6,6.8c-18.6,0-37.2,0.4-55.9-0.8c-19.2-1.2-38.5-2.1-57.8-3.2c-3-0.2-6-0.4-8.9-0.8
C1192.8,917.6,1192.8,917.6,1192.6,903.2z"/>
<path fill="none" d="M653.6,694.6c2.5,2.7,3.4,3.5,4.2,4.6c15.1,20.9,32.8,39.4,51.8,56.7c2.5,2.3,7.7,4.8,5,8.5
c-2.7,3.9-6.2-1-8.9-2.4c-14.7-7.8-28.1-17.9-42.2-26.6c-3.1-1.9-4.3-4.4-4.8-8C657.4,717,655.5,706.9,653.6,694.6z"/>
<path fill="none" d="M654,1284.6c0.9-5.5,2.7-11,2.6-16.4c-0.1-14.4,6.3-23.8,19.2-30.2c10.6-5.3,20.3-12.5,30.5-18.8
c2.5-1.5,4.9-3.3,7.8-3.6c1.6,2.1,1.5,4,0.2,5.7c-1,1.3-2.4,2.3-3.6,3.4c-18.4,17.3-36.5,34.8-51.2,55.5c-1.1,1.6-2.6,2.9-4,4.3
C655.3,1284.8,654.8,1284.6,654,1284.6z"/>
<path fill="none" d="M2019.1,698.8c-1.4,9.8-3,19.6-4.2,29.5c-0.4,3-2.2,4.7-4.4,6.2c-15.3,9.9-30.7,19.7-46.1,29.5
c-1.5,1-3.7,4-5.7,0.4c-1.5-2.6,0.2-4.4,2.1-5.9c11.9-9.2,21.5-20.7,32.4-30.8c8.4-7.7,14.5-17.2,21.4-26.1c0.9-1.2,2-2.4,3-3.6
C2018.1,698.2,2018.6,698.5,2019.1,698.8z"/>
<path fill="none" d="M2017.1,1283.9c-11.8-19.6-28.7-34.6-44.6-50.5c-3.7-3.7-7.8-7.2-11.8-10.6c-1.9-1.7-3.4-3.4-1.7-5.9
c1.7-2.4,3.5-0.5,5,0.5c15.7,10,31.3,20,46.9,30c2,1.3,3.5,2.9,3.9,5.4c1.4,10.1,2.9,20.2,4.4,30.4
C2018.5,1283.4,2017.8,1283.7,2017.1,1283.9z"/>
<path fill="none" d="M1044.6,1034c-0.6,11-1.1,21.9-1.7,32.9c-0.1,1.6-0.3,3.3-0.8,4.9c-0.3,0.8-1.4,1.6-2.2,1.7
c-0.9,0.1-1.9-0.5-2.1-1.7c-0.1-1-0.3-2-0.4-3c-1.2-22.2-2.3-44.5-2.4-66.8c0-3.2,0-6,4.6-5.9c3.5,0.1,5.6,0.7,5.5,4.9
c-0.2,11-0.1,21.9-0.1,32.9C1044.9,1034,1044.8,1034,1044.6,1034z"/>
<path fill="none" d="M1636.4,1033.8c0,10.6,0,21.2,0,31.8c0,2-0.3,4-0.7,5.9c-0.2,0.8-1.2,1.8-1.9,1.8c-0.8,0-1.9-0.8-2.2-1.5
c-0.5-1.2-0.5-2.6-0.6-3.9c-1.9-21.8-2.2-43.7-2.1-65.6c0-5.7,1.3-6.7,6-6.1c3.9,0.5,3.6,3.1,3.6,5.8c0,10.6,0,21.2,0,31.8
C1637.8,1033.8,1637.1,1033.8,1636.4,1033.8z"/>
<path fill="none" d="M1638.5,948.1c0,10.3,0,20.6,0,31c0,2.6,0.5,5.4-3.3,6.2c-4.4,0.9-6.4-0.5-6.4-5.8c0-19,0-38,1.4-56.9
c0.2-3,0.4-6,0.7-9c0.2-2.2-0.3-5.4,3-5.2c2.5,0.1,2.2,3,2.3,4.9c0.6,11.6,1.1,23.3,1.6,34.9C1638,948.1,1638.2,948.1,1638.5,948.1
z"/>
<path fill="none" d="M1045.1,948c0,10.6-0.1,21.2,0,31.9c0,3.6-0.9,5.5-5,5.5c-3.9-0.1-5.1-1.7-5.1-5.4c0.1-22.6,1.2-45.1,2.5-67.7
c0.1-1.6-0.1-3.7,2-4c2.8-0.4,3.1,1.9,3.2,3.9c0.7,11.9,1.3,23.9,1.9,35.8C1044.8,948,1044.9,948,1045.1,948z"/>
<path fill="none" d="M741.9,740.1c7,10.8,9.7,22,12.6,34.5c-8.1-3.6-13.5-8.9-19.7-13c-2-1.3-2-3.3-1.2-5.3
C735.6,751.1,737,745.6,741.9,740.1z"/>
<path fill="none" d="M1931.7,1241.3c-6.9-10.6-9.7-21.7-12.1-33.3c4.2,0.4,6.4,3.1,9.1,4.9
C1943.2,1223.1,1943.4,1224.7,1931.7,1241.3z"/>
<path fill="none" d="M1931.6,740c12,17.2,11.9,18.3-3.6,29.1c-2.4,1.7-4.5,3.9-8.5,4.8C1921.8,761.9,1925,750.9,1931.6,740z"/>
<path fill="none" d="M742.1,1241.3c-5.3-5.2-6.5-11.1-8.6-16.5c-0.8-2-0.5-3.8,1.6-5.1c6-3.9,11.2-9.1,19.2-12.6
C751.8,1219.5,748.7,1230.3,742.1,1241.3z"/>
<path fill="none" d="M599.6,517.5c0.6,0.1,1.4,0.1,1.9,0.4c12.6,7.1,13,7.9,10.7,24.4c-5.9-7.9-11-15.2-13.4-24.1
C599,518,599.3,517.7,599.6,517.5z"/>
<path fill="none" d="M597.6,1464.4c4-9.4,8.1-17.2,14-24.4C614.9,1455.5,614.5,1457.9,597.6,1464.4z"/>
<path fill="none" d="M2072.4,516.4c2.2,0.6,2.3,2,1.3,3.7c-3.8,6.1-7.7,12.1-11.9,18.6c-2.8-10.6-2.1-12.7,5.3-17.8
c1.9-1.3,4.2-2,5.3-4.3L2072.4,516.4z"/>
<path fill="none" d="M2061.5,1442.5c4.7,8.5,10.5,14.5,12.5,22.8C2067.2,1459.8,2056.4,1456.8,2061.5,1442.5z"/>
<path fill="none" d="M1781.6,836.9c4.2,9.3,4.2,9.3-1.8,12.6C1777.9,845.4,1780.6,842.1,1781.6,836.9z"/>
<path fill="none" d="M895.4,849.3c-7.5-2.9-7.5-2.9-3.8-11.1C893.8,841.3,893.9,844.7,895.4,849.3z"/>
<path fill="none" d="M630.1,1284.6c-0.2-5.4-2.7-11,1.8-17C634.2,1274.4,629.7,1279.3,630.1,1284.6z"/>
<path fill="none" d="M629.9,697.1c0.5,5.4,3.9,10.3,2.4,16.6C626.9,708.5,630.1,702.5,629.9,697.1z"/>
<path fill="none" d="M2044.5,1284.5c-2.2-5-4.3-9.2-2.8-15C2046.1,1274.3,2043.5,1279.4,2044.5,1284.5z"/>
<path fill="none" d="M2041.9,712.1c-2.2-6,1.1-10.1,2.2-14.6C2043.5,702.2,2046,707.4,2041.9,712.1z"/>
<path fill="none" d="M892.2,1143.9c-4.6-7.9-4.6-7.9,1.6-11.5C895.7,1136.5,892.6,1139.7,892.2,1143.9z"/>
<path fill="none" d="M1779.6,1132.2c6.1,3.1,6.1,3.1,2.7,11.1c-1.9-1-1.7-3.1-2.3-4.6C1779.2,1137,1778.6,1135.1,1779.6,1132.2z"/>
<path fill="none" d="M1934.8,1202.5c4.1-0.6,6.5,2.5,11.4,4.9c-6.3,0.5-8.2-3.5-11.3-5.1L1934.8,1202.5z"/>
<path fill="none" d="M1935.8,777c3.1,0.9,4.5-3.5,7.8-2.9c-1.8,4-4.7,3.9-7.9,2.8L1935.8,777z"/>
<path fill="none" d="M729.5,774c2.7-0.5,4.4,0.6,5.8,3.2C732.1,778,731,775.7,729.5,774z"/>
<path fill="none" d="M734,1202.5c2.3,4.6-1.2,4.8-4.2,5.9c-0.6-3.7,3.7-3.3,4.2-5.7C734,1202.7,734,1202.5,734,1202.5z"/>
<path fill="none" d="M1935.2,780.3c-0.4,0.2-0.8,0.5-1.2,0.7c0.3,0,0.6,0.1,0.8,0c0.2-0.1,0.4-0.3,0.6-0.5
C1935.5,780.5,1935.2,780.3,1935.2,780.3z"/>
<path fill="none" d="M740,1203.1c-1.9-0.8-4.2,2.9-6-0.5c0,0,0,0.2,0,0.1C736,1202.8,738,1202.9,740,1203.1L740,1203.1z"/>
<path fill="none" d="M1935.7,777c1.4,1.3,0.9,2.4-0.5,3.3c0,0,0.3,0.3,0.3,0.3c0.1-1.2,0.3-2.3,0.4-3.5
C1935.8,777,1935.7,777,1935.7,777z"/>
<path fill="none" d="M737.9,778.4c0.2,0.2,0.4,0.5,0.6,0.7c-0.2,0.2-0.5,0.5-0.7,0.5c-0.2,0-0.4-0.4-0.6-0.6
C737.4,778.8,737.7,778.6,737.9,778.4z"/>
<path fill="none" d="M740,1203.1c0-0.3,0-0.8-0.2-1c-0.8-0.8-0.7-1.1,0.4-0.8C739.5,1201.9,739.4,1202.4,740,1203.1
C740,1203.1,740,1203.1,740,1203.1z"/>
<path fill="none" d="M1934.9,1202.3c0-0.2-0.1-0.3-0.1-0.5c0,0-0.1,0.1-0.1,0.1c0.1,0.2,0.1,0.4,0.2,0.6
C1934.8,1202.5,1934.9,1202.3,1934.9,1202.3z"/>
<path fill="none" d="M2072.4,516.6c0.7,0.4,1.4,0.8,2.1,1.2c-0.2,0.2-0.4,0.7-0.6,0.7c-1.1-0.2-1.5-1-1.4-2
C2072.4,516.4,2072.4,516.6,2072.4,516.6z"/>
<path fill="none" d="M598.7,518.3c-1-0.2-1.8-0.8-1-1.6c0.8-0.9,1.4,0,1.8,0.8C599.3,517.7,599,518,598.7,518.3z"/>
</g>
</svg>

After

Width:  |  Height:  |  Size: 48 KiB

View File

@ -0,0 +1,134 @@
Contexts and streaming
======================
The modern core of ``tractor``: a :class:`Context` links a task in
one actor to a task in another as a *single* structured concurrency
(SC) scope stretched across the IPC boundary — errors, results and
cancellation flow between the pair `exactly like trio`_ tasks under
a common `nursery`_. Open one with ``Portal.open_context()`` (see
:func:`tractor._context.open_context_from_portal` in
:doc:`/api/core`), then optionally bridge a bidirectional
:class:`MsgStream` between the two tasks.
.. d2:: diagrams/context_handshake.d2
:caption: The ``open_context()`` <-> ``ctx.started()`` handshake.
:margin:
:alt: parent and child actor context handshake sequence
For the guided, example-driven tour see :doc:`/guide/context`; this
page is the precise API surface.
.. currentmodule:: tractor
The ``@context`` decorator
--------------------------
.. autofunction:: context
.. note::
The decorated function **must** declare a parameter annotated
``tractor.Context`` (any param name works); the runtime injects
the context instance there on each remote invocation. Pass
``pld_spec`` to type-restrict (and validate) the payloads this
endpoint may shuttle — violations raise
:class:`MsgTypeError`. See ``examples/typed_payloads.py``.
``Context``
-----------
.. autoclass:: Context
:members: started,
wait_for_result,
cancel,
cid,
chan,
side,
cancel_called,
cancelled_caught,
cancel_acked,
canceller,
maybe_error,
outcome
.. deprecated:: 0.1.0a6
``Context.result()`` warns; use :meth:`Context.wait_for_result`.
.. note::
A :class:`Context` is **not** a :class:`trio.CancelScope`:
:meth:`Context.cancel` requests cancellation of the *remote*
peer task and does not cancel the local scope. If *you*
requested the cancel, the resulting :class:`ContextCancelled`
is absorbed at ``open_context()`` exit; a cancel originating
anywhere else (the peer, or a third-party actor recorded in
:attr:`ContextCancelled.canceller`) *is* raised locally. This
self-vs-cross-cancel rule is the key to writing correct
inter-actor teardown logic — see :doc:`/guide/context`.
Bidirectional streaming
-----------------------
.. autofunction:: tractor._streaming.open_stream_from_ctx
.. note::
:func:`~tractor._streaming.open_stream_from_ctx` is bound as
the **method-alias** ``Context.open_stream()`` — call it as
``async with ctx.open_stream() as stream:``. Both sides of the
context must enter it for the dialog to be open.
.. autoclass:: MsgStream
:members: send,
receive,
receive_nowait,
aclose,
subscribe,
ctx,
closed
.. note::
A :class:`MsgStream` is one-shot use: once closed it can never
be "re-opened" — open a fresh :class:`Context` instead. Remote
end-of-stream surfaces as :class:`StopAsyncIteration` from
``async for``; un-consumed sends overrun the receiver and raise
:class:`tractor._exceptions.StreamOverrun` unless the context
was opened with ``allow_overruns=True``.
:meth:`MsgStream.subscribe` fans a single IPC stream out to
multiple *local* tasks via a
:class:`tractor.trionics.BroadcastReceiver` (see
:doc:`/api/trionics`); the underlying allocation is idempotent and
non-reversible for the stream's lifetime. See
``examples/streaming_broadcast_fanout.py`` for the pattern in
action.
Legacy one-way streaming
------------------------
.. autofunction:: stream
.. warning::
``@tractor.stream`` and ``Portal.open_stream_from()`` are the
*legacy* one-way streaming API kept for backward compat: a
plain async-generator function streamed parent-ward with no
child-side receive leg. New code should use
``@tractor.context`` + ``ctx.open_stream()`` (bidirectional,
SC-linked, typed). Note ``ctx`` is now a reserved param name
for ``@context`` endpoints — ``@stream`` functions must use
``stream`` instead, and ``ctx.send_yield()`` is deprecated in
favor of :meth:`MsgStream.send`.
.. seealso::
:doc:`/api/errors` for :class:`ContextCancelled` /
:class:`MsgTypeError` semantics, :doc:`/api/msg` for payload
typing via ``pld_spec`` and codecs, :doc:`/api/trionics` for
the broadcast fan-out machinery, and the guided tours in
:doc:`/guide/streaming` + :doc:`/guide/cancellation`.
.. _exactly like trio: https://trio.readthedocs.io/en/latest/reference-core.html#cancellation-semantics
.. _nursery: https://trio.readthedocs.io/en/latest/reference-core.html#nurseries-and-spawning

157
docs/api/core.rst 100644
View File

@ -0,0 +1,157 @@
Runtime and spawning
====================
The core lifecycle API: boot the runtime in your root process,
spawn trio-"actors" (processes running ``trio.run()`` task trees)
under a one-cancels-all supervisor, and talk to them through
portals. This is structured concurrency (SC) applied *transitively*:
every spawned process is owned by a nursery block and errors
`always propagate`_. If you can create zombies it **is a bug**.
.. currentmodule:: tractor
Booting the runtime
-------------------
.. autofunction:: open_root_actor
.. note::
The env vars ``TRACTOR_LOGLEVEL`` and ``TRACTOR_SPAWN_METHOD``
override the ``loglevel`` / ``start_method`` params so you can
crank verbosity or swap spawn backends without touching app
code. Exactly **one** IPC transport may be enabled per actor
(see ``enable_transports`` and :doc:`/api/ipc`).
.. autofunction:: run_daemon
Spawning actors
---------------
.. d2:: diagrams/actor_tree.d2
:caption: A supervised actor (process) tree.
:margin:
:alt: root actor supervising a tree of subactors
.. autofunction:: open_nursery
.. autoclass:: ActorNursery
:members: start_actor,
run_in_actor,
cancel,
cancel_called,
cancelled_caught
.. note::
:meth:`ActorNursery.start_actor` (daemon actor + portal) is the
blessed spawning primitive; pair it with
``Portal.open_context()`` for SC-linked remote tasks.
:meth:`ActorNursery.run_in_actor` is a *convenience* one-shot —
spawn, run a single task, auto-cancel after the result — slated
to be rebuilt as a high-level wrapper, so don't design around
it as the core model.
.. deprecated:: 0.1.0a6
``ActorNursery.cancelled`` warns; use
:attr:`ActorNursery.cancel_called` and
:attr:`ActorNursery.cancelled_caught`. The ``rpc_module_paths``
kwarg is likewise deprecated in favor of ``enable_modules``.
Portals
-------
A :class:`Portal` "opens a portal" into a peer actor's memory
domain: you call functions and start SC-linked tasks *over IPC* as
though they were local, with results, errors and cancellation
flowing back `exactly like trio`_.
.. autoclass:: Portal
:members: run,
run_from_ns,
open_stream_from,
wait_for_result,
cancel_actor,
chan
.. deprecated:: 0.1.0a6
``Portal.result()`` warns; use :meth:`Portal.wait_for_result`.
The str-form ``Portal.run('mod.path', 'fn_name')`` also warns;
pass a function *object* whose module is listed in the target's
``enable_modules``. ``Portal.channel`` is the legacy spelling
of :attr:`Portal.chan`.
.. autofunction:: tractor._context.open_context_from_portal
.. note::
:func:`~tractor._context.open_context_from_portal` is bound as
the **method-alias** ``Portal.open_context()`` — that's the
spelling you should actually call:
``portal.open_context(fn, **kwargs)``. See :doc:`/api/context`
for the full ``Context`` + ``MsgStream`` API it unlocks.
.. note::
:meth:`Portal.cancel_actor` cancels the *whole* remote runtime
and process (machine-level), not a single task — use
:meth:`Context.cancel` for task-level cancellation. Pass
``raise_on_timeout=True`` to get an ``ActorTooSlowError`` you
can escalate per SC discipline (see :doc:`/api/errors`).
Clusters
--------
.. autofunction:: open_actor_cluster
Spawn a *flat* cluster of ``count`` worker actors (default: one
per core) all serving the RPC ``modules`` list, yielding a
``dict[str, Portal]`` keyed by actor name. Handy for
embarrassingly parallel fan-out; see ``examples/quick_cluster.py``.
Runtime introspection
---------------------
.. autofunction:: current_actor
.. autoclass:: Actor
:members: aid,
name,
uid,
is_registrar,
is_infected_aio,
cancel_soon
.. note::
:class:`Actor` is the per-process runtime singleton (msg loop,
RPC scheduling, IPC server) — you never instantiate it yourself
and should normally only touch the identity/introspection
surface listed above. The canonical identity type is
:attr:`Actor.aid` (a ``tractor.msg.Aid`` struct);
:attr:`Actor.uid` is the legacy ``(name, uuid)`` 2-tuple which
is still pervasive in logs and error metadata.
.. deprecated:: 0.1.0a6
``Actor.is_arbiter`` warns; use :attr:`Actor.is_registrar`.
The ``arbiter_addr`` constructor kwarg is deprecated for
``registry_addrs``.
.. autofunction:: current_ipc_ctx
.. autofunction:: is_root_process
.. autofunction:: get_runtime_vars
.. seealso::
:doc:`/api/context` for the SC-linked remote task API,
:doc:`/api/discovery` for finding actors by name, and the
guided tours in :doc:`/guide/spawning`, :doc:`/guide/rpc` and
:doc:`/guide/context`.
.. _always propagate: https://trio.readthedocs.io/en/latest/design.html#exceptions-always-propagate
.. _exactly like trio: https://trio.readthedocs.io/en/latest/reference-core.html#cancellation-semantics

92
docs/api/devx.rst 100644
View File

@ -0,0 +1,92 @@
Debugging and devx: ``tractor.devx``
====================================
Multi-process debugging that actually works: boot the tree with
``open_root_actor(debug_mode=True)`` (or pass it to
``open_nursery()``) and any crash or explicit pause in *any* actor
acquires a tree-global TTY lock and drops you into a
`pdbp`_-powered REPL — one actor at a time, ``SIGINT`` shielded,
no garbled terminals. The top-level helpers below are the daily
drivers; the rest of the toolbox lives under ``tractor.devx``.
.. currentmodule:: tractor
Pausing and post-mortems
------------------------
.. autofunction:: pause
.. autofunction:: pause_from_sync
.. note::
:func:`pause_from_sync` needs the `greenback`_ portal: boot
with ``open_root_actor(maybe_enable_greenback=True)`` (mind
the `performance implications`_). With ``debug_mode`` on, the
built-in ``breakpoint()`` is also remapped to a
``tractor``-safe equivalent.
.. autofunction:: post_mortem
.. deprecated:: 0.1.0a6
``tractor.breakpoint()`` warns and simply calls :func:`pause`
— use :func:`tractor.pause` (async) or
:func:`tractor.pause_from_sync` in new code.
Crash handling for CLIs and sync entrypoints
--------------------------------------------
.. currentmodule:: tractor.devx
.. autofunction:: open_crash_handler
.. autofunction:: maybe_open_crash_handler
.. note::
Both are *sync* context managers usable before (or without)
``trio.run()`` — wrap your CLI ``main()`` to get a post-mortem
REPL on any uncaught exception instead of a bare traceback.
Runtime hang-hunting
--------------------
.. autofunction:: enable_stack_on_sig
With stackscope_ integration enabled (also via
``open_root_actor(enable_stack_on_sig=True)`` or the
``TRACTOR_ENABLE_STACKSCOPE`` env var) a ``SIGUSR1`` triggers a
full trio task-tree dump from every actor — works on live,
*non*-debug-mode trees too:
.. code:: sh
pkill --signal SIGUSR1 -f <part-of-your-cmd>
Dumps also tee to ``/tmp/tractor-stackscope-<pid>.log`` so you
still get output under captured/CI stdio.
Lower-level debug plumbing
--------------------------
.. autofunction:: mk_pdb
.. autofunction:: maybe_wait_for_debugger
:func:`maybe_wait_for_debugger` is mainly useful in runtime/test
code that must avoid tearing down a tree while a child still
holds the global debug lock.
.. seealso::
:doc:`/api/errors` for the boxed error types you'll inspect
from the REPL, :doc:`/api/core` for the ``debug_mode`` /
``maybe_enable_greenback`` / ``enable_stack_on_sig`` boot
flags on :func:`tractor.open_root_actor`, and
:doc:`/guide/debugging` for the guided multi-actor REPL tour.
.. _pdbp: https://github.com/mdmintz/pdbp
.. _greenback: https://greenback.readthedocs.io/en/latest/
.. _performance implications: https://greenback.readthedocs.io/en/latest/principle.html#performance
.. _stackscope: https://github.com/oremanj/stackscope

View File

@ -0,0 +1,67 @@
Discovery and the registrar
===========================
Every actor registers its ``(name, uuid)`` and transport addresses
with a *registrar* actor — by default the root of the tree, or
whichever actor serves at the ``registry_addrs`` you boot with.
The discovery API lets any actor look up any other **by name** and
get back a connected :class:`~tractor.Portal`, giving you
service-discovery patterns (daemons, service trees, multi-host
meshes) without hard-coding addresses.
Lookups first scan already-connected peers before RPC-ing the
registrar, and multihomed results are ranked UDS > local TCP >
remote TCP. See ``examples/service_daemon_discovery.py`` for the
canonical daemon + lookup pattern.
.. currentmodule:: tractor
Lookup APIs
-----------
.. autofunction:: find_actor
.. autofunction:: wait_for_actor
.. autofunction:: query_actor
.. autofunction:: get_registry
.. note::
:func:`find_actor` yields ``None`` when nothing is registered
under the name (or raises with ``raise_on_none=True``);
:func:`wait_for_actor` blocks until the name appears;
:func:`query_actor` only *looks up* the address without
connecting to the target.
The ``Registrar``
-----------------
.. autoclass:: Registrar
:show-inheritance:
A :class:`Registrar` is just an :class:`~tractor.Actor` subtype
maintaining the name -> addresses table; you rarely touch it
directly beyond passing ``registry_addrs`` /
``ensure_registry=True`` to :func:`~tractor.open_root_actor`.
Check :attr:`Actor.is_registrar <tractor.Actor.is_registrar>` to
ask "am I it?".
Legacy ``Arbiter`` alias
------------------------
.. deprecated:: 0.1.0a6
``tractor.Arbiter`` survives only as a class alias of
:class:`Registrar` and all "arbiter" terminology is replaced
by "registrar"/"registry" across the API: ``get_arbiter()`` is
removed (use :func:`get_registry`) and the ``arbiter_addr``
kwarg is replaced by ``registry_addrs``.
.. seealso::
:doc:`/api/core` for booting a registrar via
``open_root_actor(registry_addrs=...)``, :doc:`/api/ipc` for
the transport/address model the registry stores, and
:doc:`/guide/discovery` for the worked walkthrough.

102
docs/api/errors.rst 100644
View File

@ -0,0 +1,102 @@
Errors and cancellation types
=============================
``tractor`` extends trio's "exceptions `always propagate`_" rule
across the process boundary: a crash in any actor is serialized as
an ``Error`` msg, shuttled over IPC, and re-raised in the linked
parent scope as a *boxed* :class:`RemoteActorError` — preserving
the original type, traceback text and source-actor identity, even
across multi-hop relays (a.k.a. "inceptions").
The most-used types below are importable from ``tractor``
directly; the remainder live in ``tractor._exceptions`` (not yet
re-exported at top level).
.. currentmodule:: tractor
Boxed remote errors
-------------------
.. autoexception:: RemoteActorError
:members: boxed_type,
src_uid,
relay_uid,
pformat
.. code:: python
try:
async with portal.open_context(ep_fn) as (ctx, first):
...
except tractor.RemoteActorError as rae:
if rae.boxed_type is ValueError:
... # remote task raised a `ValueError`
.. autoexception:: ContextCancelled
:show-inheritance:
:members: canceller
.. note::
Inspect :attr:`ContextCancelled.canceller` (the requesting
actor's uid) to distinguish a *self*-requested cancel (absorbed
at ``open_context()`` exit) from a *cross*-actor cancel (raised
locally) — the full rules live in :doc:`/api/context`.
Typed-messaging errors
----------------------
.. autoexception:: MsgTypeError
:show-inheritance:
:members: bad_msg,
expected_msg_type
An "IPC ``TypeError``": a message failed validation against the
active msg-spec / ``pld_spec`` (see :doc:`/api/msg`). Raised
sender-side for control msgs (``Started``/``Return``) and
receiver-side for stream ``Yield`` payloads.
.. autoexception:: tractor._exceptions.StreamOverrun
:show-inheritance:
The sender out-paced the receiver's buffer on a
:class:`~tractor.MsgStream` opened without
``allow_overruns=True``; subtypes :class:`trio.TooSlowError`.
Transport and runtime errors
----------------------------
.. autoexception:: TransportClosed
.. autoexception:: ModuleNotExposed
:show-inheritance:
Raised when an RPC requests a function from a module not listed
in the target actor's ``enable_modules`` allowlist —
capability-style access control, not an import bug on your end ;)
.. autoexception:: tractor._exceptions.NoRuntime
:show-inheritance:
Raised by :func:`tractor.current_actor` (and friends) when no
actor runtime is up in the current process.
.. autoexception:: tractor._exceptions.ActorTooSlowError
:show-inheritance:
A peer actor failed to ack a cancel request within the bounded
wait — the SC-sanctioned escalation signal from APIs like
``Portal.cancel_actor(raise_on_timeout=True)``. Catch it to
escalate (e.g. hard-kill via the supervising
:class:`~tractor.ActorNursery`); never just ignore it, that's how
zombies happen.
.. seealso::
:doc:`/api/context` for how cancellation and errors flow
through a :class:`~tractor.Context`, :doc:`/api/devx` for
crash-handling REPL tooling (``debug_mode``, post-mortems),
and :doc:`/guide/cancellation` for the full SC-cancellation
story.
.. _always propagate: https://trio.readthedocs.io/en/latest/design.html#exceptions-always-propagate

67
docs/api/index.rst 100644
View File

@ -0,0 +1,67 @@
API reference
=============
This is the curated reference for ``tractor``'s public surface: the
names you can import and lean on without reading runtime internals.
Everything below is re-exported at the top level (``import
tractor``) unless a page says otherwise; subsystems like
``tractor.msg``, ``tractor.trionics``, ``tractor.to_asyncio``,
``tractor.devx`` and ``tractor.log`` are importable as submodules.
``tractor`` is "just trio_" extended across processes: every API
here is designed to keep the structured concurrency (SC) rules you
already know from the `trio docs`_ intact across the process
boundary. If a name isn't documented here it's an internal — expect
it to change without notice B).
.. currentmodule:: tractor
Most-used names at a glance:
.. autosummary::
:nosignatures:
open_root_actor
open_nursery
run_daemon
ActorNursery
Portal
context
Context
MsgStream
open_actor_cluster
find_actor
wait_for_actor
get_registry
current_actor
current_ipc_ctx
is_root_process
get_runtime_vars
RemoteActorError
ContextCancelled
MsgTypeError
pause
post_mortem
Channel
.. toctree::
:maxdepth: 1
:caption: Reference pages
core
context
discovery
errors
msg
trionics
to_asyncio
devx
ipc
Where to next? If you're new, start with the runtime and spawning
APIs in :doc:`/api/core`, then graduate to the inter-actor task
linking model in :doc:`/api/context` — it's the heart of the whole
system.
.. _trio: https://github.com/python-trio/trio
.. _trio docs: https://trio.readthedocs.io/en/latest/

89
docs/api/ipc.rst 100644
View File

@ -0,0 +1,89 @@
IPC and logging
===============
Under every portal, context and stream sits a per-peer
:class:`~tractor.Channel`: a msgpack-typed messaging link wrapping
one OS transport connection. Transports are pluggable per actor
via ``enable_transports=['tcp' | 'uds']`` — TCP is the default,
UDS (unix domain sockets) gives you port-less, same-host IPC with
kernel-provided peer credentials for free — and exactly **one**
transport may currently be enabled per actor.
.. d2:: diagrams/runtime_stack.d2
:caption: Where ``Channel`` sits in the runtime stack.
:margin:
:alt: layered runtime stack from app code down to transports
Addresses are "unwrapped" tuples at the API edges:
``('host', port)`` for TCP, filesystem-path pairs for UDS. For
the full layering story — transport protocols, the IPC server,
address types and the msg loop — see
:doc:`/explain/architecture`.
.. currentmodule:: tractor
``Channel``
-----------
.. autoclass:: Channel
:members: from_addr,
send,
recv,
aclose,
connected,
apply_codec,
aid,
laddr,
raddr,
closed
.. deprecated:: 0.1.0a6
``Channel.uid`` warns; use :attr:`Channel.aid` which carries
richer (optional) identity fields beyond the legacy
``(name, uuid)`` pair.
.. note::
You rarely construct a :class:`Channel` yourself — the runtime
hands them out via :attr:`Portal.chan <tractor.Portal.chan>`
and :attr:`Context.chan <tractor.Context.chan>`. Treat the
send/recv surface as advanced API: normal apps should speak
:class:`~tractor.MsgStream` instead.
Choosing a transport
--------------------
.. literalinclude:: ../../examples/uds_transport_actor_tree.py
:caption: examples/uds_transport_actor_tree.py
:language: python
Logging
-------
``tractor.log`` provides the structured, colorized console
logging used across the runtime — with actor-name + task-aware
record headers and extra log levels below :data:`logging.DEBUG`
(``'transport'``, ``'runtime'``, ``'cancel'``, ``'devx'``) for
spelunking the runtime itself. Use it for your app too: it's
already distributed-system aware.
.. currentmodule:: tractor.log
.. autofunction:: get_logger
.. autofunction:: get_console_log
.. note::
The ``TRACTOR_LOGLEVEL`` env var overrides any caller-passed
``loglevel`` (e.g. to ``open_root_actor()``) so you can crank
console verbosity without touching code; subactors inherit
the root's level by default.
.. seealso::
:doc:`/explain/architecture` for the transport/server
internals, :doc:`/api/discovery` for how channel addresses
get registered and found, and :doc:`/api/msg` for the codec
layer every channel speaks.

107
docs/api/msg.rst 100644
View File

@ -0,0 +1,107 @@
Typed messaging: ``tractor.msg``
================================
All inter-actor communication rides a small, strictly-typed
msgpack wire protocol built from :class:`msgspec.Struct` types —
the "SC-shuttle" protocol that powers contexts, streams, RPC and
cancellation. You normally never touch these msg types directly
(the :class:`~tractor.Context` API speaks them for you) but you
*do* use this subpackage to define **payload type contracts**: per
endpoint via ``@tractor.context(pld_spec=...)`` or per channel via
custom codecs.
Violations of an active msg-spec surface as
:class:`~tractor.MsgTypeError` (see :doc:`/api/errors`); the full
typed-payload workflow is shown in ``examples/typed_payloads.py``.
.. currentmodule:: tractor.msg
The protocol message set
------------------------
.. autosummary::
:nosignatures:
PayloadMsg
Aid
SpawnSpec
Start
StartAck
Started
Yield
Stop
Return
CancelAck
Error
``Aid`` (identity handshake) and ``SpawnSpec`` (parent -> child
init) run at connection setup; ``Start``/``StartAck`` initiate an
RPC task; ``Started``/``Yield``/``Stop``/``Return`` are the
:class:`~tractor.Context` dialog phases; ``CancelAck`` and
``Error`` close the loop on cancellation and (boxed) failure.
``Msg`` is a legacy alias of ``PayloadMsg``. The union of all of
the above is exported as ``MsgType`` (also ``__msg_spec__``).
.. automodule:: tractor.msg.types
:no-members:
.. currentmodule:: tractor.msg
Codec construction and override
-------------------------------
.. autofunction:: mk_codec
.. autoclass:: MsgCodec
:members: encode,
decode,
msg_spec
.. autofunction:: mk_dec
.. autoclass:: MsgDec
:members: decode,
spec
.. autofunction:: apply_codec
.. autofunction:: current_codec
.. note::
:func:`apply_codec` swaps the codec via a
:class:`contextvars.ContextVar` — the override only applies to
the *current task* (and tasks it starts), not sibling tasks
already running in the actor. Payload-decoding is layered: the
outer codec leaves ``.pld`` fields as ``msgspec.Raw`` and each
context's payload-receiver decodes them against *its* spec
(the "cheap-or-nasty" validation pattern).
Namespace pointers
------------------
.. autoclass:: NamespacePath
:members: from_ref,
load_ref,
to_tuple
The ``'module.path:obj_name'`` :class:`str`-subtype used to
address every RPC target function over the wire (same format as
:func:`pkgutil.resolve_name`).
Pretty structs
--------------
.. autoclass:: Struct
:show-inheritance:
A :class:`msgspec.Struct` subtype with a multi-line pretty
``__repr__`` — handy as a base for your own IPC payload types so
crash logs stay readable.
.. seealso::
:doc:`/api/context` for where ``pld_spec`` typing plugs into
the ``@context`` decorator, :doc:`/api/errors` for
:class:`~tractor.MsgTypeError` semantics, and
:doc:`/guide/msging` for the guided typed-messaging tour.

View File

@ -0,0 +1,97 @@
asyncio interop: ``tractor.to_asyncio``
=======================================
"Infected asyncio" mode: spawn an actor with
``start_actor(..., infect_asyncio=True)`` and its process runs
:mod:`trio` as a `guest`_ on top of the :mod:`asyncio` loop —
letting your trio task tree drive asyncio tasks *in the same
process* while the rest of the actor tree stays pure trio. Each
trio <-> asyncio task pair is linked with structured concurrency
(SC) semantics: error or cancellation on either side tears down
both, with the cause translated cross-loop.
.. d2:: diagrams/infected_aio.d2
:caption: A trio guest driving asyncio tasks in one actor.
:margin:
:alt: trio guest mode inside an asyncio-infected actor
See ``examples/infected_asyncio_echo_server.py`` for a complete
worked example.
.. currentmodule:: tractor.to_asyncio
Starting asyncio tasks from trio
--------------------------------
.. autofunction:: open_channel_from
.. autofunction:: run_task
.. note::
:func:`open_channel_from` mirrors the
``Portal.open_context()`` handshake: the asyncio side calls
``chan.started_nowait(value)`` and that value pops out as
``first`` on the trio side. :func:`run_task` is the one-shot
form — run a single asyncio-compatible coroutine fn and return
its result to trio.
The inter-loop channel
----------------------
.. autoclass:: LinkedTaskChannel
:members: send,
receive,
wait_for_result,
subscribe,
started_nowait,
send_nowait,
get,
cancel_asyncio_task,
closed
.. note::
The trio side uses the async API
(:meth:`LinkedTaskChannel.send` /
:meth:`LinkedTaskChannel.receive`); the asyncio side uses the
loop-safe sync/await mix
(:meth:`LinkedTaskChannel.send_nowait` /
:meth:`LinkedTaskChannel.get` /
:meth:`LinkedTaskChannel.started_nowait`).
Translated exception types
--------------------------
Cross-loop failures are re-raised on the *other* side as one of
these explicit translation types, so you always know which loop
actually died first:
.. autoexception:: tractor._exceptions.AsyncioCancelled
.. autoexception:: tractor._exceptions.AsyncioTaskExited
.. autoexception:: tractor._exceptions.TrioCancelled
.. autoexception:: tractor._exceptions.TrioTaskExited
.. autoexception:: AsyncioRuntimeTranslationError
:show-inheritance:
Guest-mode entrypoint
---------------------
``run_as_asyncio_guest()`` is the runtime-internal entrypoint that
boots trio in `guest`_ mode inside an infected actor — you get it
implicitly via ``infect_asyncio=True`` and shouldn't need to call
it yourself.
.. seealso::
:doc:`/api/core` for the ``infect_asyncio`` spawn flag,
:meth:`tractor.Actor.is_infected_aio` for runtime
introspection, :doc:`/api/devx` for using the debugger from
inside asyncio tasks, and :doc:`/guide/asyncio` for the
guided tour.
.. _guest: https://trio.readthedocs.io/en/stable/reference-lowlevel.html?highlight=guest%20mode#using-guest-mode-to-run-trio-on-top-of-other-event-loops

View File

@ -0,0 +1,76 @@
Trio patterns: ``tractor.trionics``
===================================
Sugary structured concurrency (SC) patterns for plain :mod:`trio`
code — **no actor runtime required**. These helpers grew out of
real distributed-system needs in ``tractor`` apps but every one of
them works in a single-process program too; import via
``from tractor import trionics``.
.. currentmodule:: tractor.trionics
Context-manager helpers
-----------------------
.. autofunction:: gather_contexts
.. autofunction:: maybe_open_context
.. autofunction:: maybe_open_nursery
.. note::
:func:`gather_contexts` is "a nursery for async context
managers": it enters N acms concurrently and yields their
values in input order. :func:`maybe_open_context` is the
actor-wide cache/multiplex layer on top — the first task pays
the acm setup cost, later callers get ``(cache_hit=True, ...)``
and share the same value until all users exit.
Broadcast fan-out
-----------------
.. autofunction:: broadcast_receiver
.. autoclass:: BroadcastReceiver
:members: receive,
subscribe,
aclose
.. autoexception:: Lagged
:show-inheritance:
A single-producer, many-consumer broadcast layer over any
``trio``-style receive channel: non-lossy for the *fastest*
consumer while slower consumers raise :class:`Lagged` (a
:class:`trio.TooSlowError` subtype) once they fall behind the
internal ring. This is exactly the machinery behind
:meth:`tractor.MsgStream.subscribe` — see
``examples/streaming_broadcast_fanout.py``.
ExceptionGroup helpers
----------------------
.. autofunction:: collapse_eg
.. autofunction:: maybe_raise_from_masking_exc
.. note::
:func:`collapse_eg` "un-nests" single-exception
:class:`ExceptionGroup` wrappers from strict-eg ``trio``
nurseries so your ``except`` clauses match the original error;
:func:`maybe_raise_from_masking_exc` surfaces real errors that
would otherwise be masked by :class:`trio.Cancelled` during
teardown.
.. seealso::
:doc:`/api/context` for the IPC-stream consumer of
:class:`BroadcastReceiver`, :doc:`/guide/streaming` for
fan-out in a worked pipeline, and the `trio docs`_ for the
underlying channel and `nursery`_ semantics these helpers
compose.
.. _trio docs: https://trio.readthedocs.io/en/latest/
.. _nursery: https://trio.readthedocs.io/en/latest/reference-core.html#nurseries-and-spawning

View File

@ -1,105 +1,160 @@
# Configuration file for the Sphinx documentation builder.
#
# This file only contains a selection of the most common options. For a full
# list see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html
# tractor: distributed structured concurrency.
'''
Sphinx config for `tractor`'s documentation.
# -- Path setup --------------------------------------------------------------
Theme-wise we ride the `pydata_sphinx_theme` (per the
research + history in #157) skinned to a minimal
black + white look via `_static/css/custom.css`; see
the local extensions under `_ext/` for our `.. d2::`
diagram and `.. margin::` aside directives.
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
# import os
# import sys
# sys.path.insert(0, os.path.abspath('.'))
Build locally via,
# Warn about all references to unknown targets
nitpicky = True
uv run --group docs make -C docs html
# The master toctree document.
master_doc = 'index'
'''
from importlib.metadata import version as get_version
from pathlib import Path
import sys
# -- Project information -----------------------------------------------------
# local sphinx extensions live in `_ext/`:
# - `d2diagrams`: `.. d2::` diagram rendering
# - `marginalia`: `.. margin::` RHS prose-asides
sys.path.insert(
0,
str((Path(__file__).parent / '_ext').resolve()),
)
# -- project info ---------------------------------
project = 'tractor'
copyright = '2018, Tyler Goodlet'
copyright = '2018-2026, Tyler Goodlet'
author = 'Tyler Goodlet'
release: str = get_version('tractor')
version: str = release
# The full version, including alpha/beta/rc tags
release = '0.0.0a0.dev0'
# -- general config -------------------------------
# -- General configuration ---------------------------------------------------
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.autosummary',
'sphinx.ext.intersphinx',
'sphinx.ext.viewcode',
'sphinx.ext.todo',
# emit a `.nojekyll` so GitHub Pages serves the `_static/`
# + `_images/` dirs (Jekyll would otherwise drop `_`-prefixed
# paths and break all styling/assets).
'sphinx.ext.githubpages',
'sphinx_design',
'sphinx_copybutton',
'sphinxext.opengraph',
'sphinx_togglebutton',
'd2diagrams',
'marginalia',
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
exclude_patterns = [
'_build',
# the pypi/gh readme + its standalone generator
# sub-project; NOT doc-tree pages.
'README.rst',
'github_readme',
'Thumbs.db',
'.DS_Store',
]
root_doc = 'index'
# TODO: flip this on + burn down the (many) warnings
# from our informal docstring style; see the autodoc
# readiness notes from the revamp's recon pass.
nitpicky = False
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
# -- autodoc/autosummary --------------------------
autodoc_member_order = 'bysource'
autodoc_typehints = 'description'
autosummary_generate = True
# -- Options for HTML output -------------------------------------------------
# -- intersphinx ----------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = 'sphinx_book_theme'
pygments_style = 'algol_nu'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
html_theme_options = {
# 'logo': 'tractor_logo_side.svg',
# 'description': 'Structured concurrent "actors"',
"repository_url": "https://github.com/goodboy/tractor",
"use_repository_button": True,
"home_page_in_toc": False,
"show_toc_level": 1,
"path_to_docs": "docs",
}
html_sidebars = {
"**": [
"sbt-sidebar-nav.html",
# "sidebar-search-bs.html",
# 'localtoc.html',
],
# 'logo.html',
# 'github.html',
# 'relations.html',
# 'searchbox.html'
# ]
intersphinx_mapping = {
'python': (
'https://docs.python.org/3',
None,
),
'trio': (
'https://trio.readthedocs.io/en/stable',
None,
),
# NOTE, msgspec's site doesn't publish an
# `objects.inv` (404s) so no intersphinx for it.
'pytest': (
'https://docs.pytest.org/en/stable',
None,
),
}
# doesn't seem to work?
# extra_navbar = "<p>nextttt-gennnnn</p>"
# -- html output ----------------------------------
html_title = ''
html_theme = 'pydata_sphinx_theme'
html_title = 'tractor'
# canonical site root (GitHub Pages); drives <link rel=canonical>,
# og:url + any future sitemap. Update if a custom domain is added.
html_baseurl = 'https://goodboy.github.io/tractor/'
html_logo = '_static/tractor_logo_side.svg'
html_favicon = '_static/tractor_logo_side.svg'
# show_navbar_depth = 1
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
# Example configuration for intersphinx: refer to the Python standard library.
intersphinx_mapping = {
"python": ("https://docs.python.org/3", None),
"pytest": ("https://docs.pytest.org/en/latest", None),
"setuptools": ("https://setuptools.readthedocs.io/en/latest", None),
html_css_files = ['css/custom.css']
html_show_sourcelink = False
html_theme_options = {
# theme-adaptive navbar logo: faces transparent, linework
# near-black on light / near-white on dark (pydata swaps by
# the active theme). Matches the landing hero's wireframe.
'logo': {
'image_light': '_static/tractor_logo_nav_light.svg',
'image_dark': '_static/tractor_logo_nav_dark.svg',
'alt_text': 'tractor',
# text shown to the right of the navbar logo (à la
# polars).
'text': 'tractor',
},
'github_url': 'https://github.com/goodboy/tractor',
'navbar_align': 'content',
'show_toc_level': 2,
'secondary_sidebar_items': {
'**': ['page-toc'],
'index': [],
},
'use_edit_page_button': True,
'footer_start': ['copyright'],
'footer_end': ['theme-version'],
'pygments_light_style': 'algol_nu',
'pygments_dark_style': 'github-dark',
}
html_context = {
'github_user': 'goodboy',
'github_repo': 'tractor',
'github_version': 'main',
'doc_path': 'docs',
}
# -- ext: opengraph -------------------------------
ogp_site_url = 'https://goodboy.github.io/tractor/'
ogp_use_first_image = True
# -- ext: copybutton ------------------------------
copybutton_prompt_text = r'>>> |\.\.\. |\$ '
copybutton_prompt_is_regexp = True
# -- ext: todo ------------------------------------
todo_include_todos = True
# -- ext: d2diagrams (local) ----------------------
# normally resolved from PATH or the `D2_BIN` env
# var; when neither hits, the committed SVGs under
# `_diagrams/` are used as-is.
d2_bin = None
d2_args = []

View File

@ -1,51 +0,0 @@
Hot tips for ``tractor`` hackers
================================
This is a WIP guide for newcomers to the project mostly to do with
dev, testing, CI and release gotchas, reminders and best practises.
``tractor`` is a fairly novel project compared to most since it is
effectively a new way of doing distributed computing in Python and is
much closer to working with an "application level runtime" (like erlang
OTP or scala's akka project) then it is a traditional Python library.
As such, having an arsenal of tools and recipes for figuring out the
right way to debug problems when they do arise is somewhat of
a necessity.
Making a Release
----------------
We currently do nothing special here except the traditional
PyPa release recipe as in `documented by twine`_. I personally
create sub-dirs within the generated `dist/` with an explicit
release name such as `alpha3/` when there's been a sequence of
releases I've made, but it really is up to you how you like to
organize generated sdists locally.
The resulting build cmds are approximately:
.. code:: bash
python setup.py sdist -d ./dist/XXX.X/
twine upload -r testpypi dist/XXX.X/*
twine upload dist/XXX.X/*
.. _documented by twine: https://twine.readthedocs.io/en/latest/#using-twine
Debugging and monitoring actor trees
------------------------------------
TODO: but there are tips in the readme for some terminal commands
which can be used to see the process trees easily on Linux.
Using the log system to trace `trio` task flow
----------------------------------------------
TODO: the logging system is meant to be oriented around
stack "layers" of the runtime such that you can track
"logical abstraction layers" in the code such as errors, cancellation,
IPC and streaming, and the low level transport and wire protocols.

View File

@ -0,0 +1,31 @@
# tractor docs diagram: the hero supervision tree.
# A process tree of trio-task-trees; every arrow is
# a parent which *must wait* on its children.
vars: {
d2-config: {
sketch: true
theme-id: 1
pad: 16
layout-engine: elk
}
}
direction: down
root: "root actor" {
main: "main()"
an: "ActorNursery"
main -> an: opens
}
w0: "subactor\n'worker_0'" {
tasks: "trio task tree"
}
w1: "subactor\n'worker_1'" {
main: "serve()"
an: "ActorNursery"
main -> an: opens
}
gc: "subactor\n'deeper'" {
tasks: "trio task tree"
}
root.an -> w0: "spawns +\nsupervises"
root.an -> w1: "spawns +\nsupervises"
w1.an -> gc: "spawns +\nsupervises"

View File

@ -0,0 +1,33 @@
# tractor docs diagram: the inter-actor `Context`
# dialog; tractor's SC-transitive msg protocol as
# seen from both sides of `Portal.open_context()`.
vars: {
d2-config: {
sketch: true
theme-id: 1
pad: 16
layout-engine: elk
}
}
shape: sequence_diagram
parent: "parent task\n(Portal)"
child: "child actor task\n(@tractor.context fn)"
parent -> child: "Start: open_context(fn, **kwargs)"
child -> parent: "StartAck"
child -> parent: "Started: await ctx.started(value)"
parent -> child: "Yield: stream.send()"
child -> parent: "Yield: stream.send()"
parent -> child: "Stop: stream closed"
child -> parent: "Return: fn return value"
# the non-graceful endings: errors/cancels always
# propagate, both ways (`ContextCancelled` rides in
# an `Error` msg, hence the `Error: ...` labels).
"... or, instead of a graceful Return": {
parent-initiated: {
parent -> child: "ctx.cancel() | parent-side error"
child -> parent: "Error: ContextCancelled"
}
child-initiated: {
child -> parent: "Error: child raised | ContextCancelled"
}
}

View File

@ -0,0 +1,19 @@
# tractor docs diagram: multi-actor debugger REPL
# serialization via the root actor's tty lock.
vars: {
d2-config: {
sketch: true
theme-id: 1
pad: 16
layout-engine: elk
}
}
shape: sequence_diagram
mary: "subactor 'mary'"
root: "root actor\n(owns the tty)"
bob: "subactor 'bob'"
mary -> root: "tractor.pause(): acquire tty lock"
root -> mary: "granted: pdb REPL is yours"
bob -> root: "tractor.pause(): acquire tty lock"
mary -> root: "continue: release lock"
root -> bob: "granted: pdb REPL is yours"

View File

@ -0,0 +1,27 @@
# tractor docs diagram: one-cancels-all error
# propagation up a (sub-)actor tree; no zombies,
# no lost errors.
vars: {
d2-config: {
sketch: true
theme-id: 1
pad: 16
layout-engine: elk
}
}
direction: down
root: "root actor\n(ActorNursery)"
gertie: "subactor 'gertie'\n(healthy, gets cancelled)"
bobbie: "subactor 'bobbie'\nraises NameError"
root -> gertie: supervises
root -> bobbie: supervises
bobbie -> root: "RemoteActorError\n(boxed NameError)" {
style: {
stroke-dash: 3
}
}
root -> gertie: "cancel()" {
style: {
stroke-dash: 3
}
}

View File

@ -0,0 +1,23 @@
# tractor docs diagram: "infected asyncio" mode;
# trio runs as a guest on the asyncio loop with
# SC supervision linking tasks across both.
vars: {
d2-config: {
sketch: true
theme-id: 1
pad: 16
layout-engine: elk
}
}
direction: down
sub: "subactor process (infect_asyncio=True)" {
aio: "asyncio event loop" {
aiotask: "asyncio.Task\naio_echo_server()"
trio: "trio (guest mode)" {
triotask: "trio task\n@tractor.context fn"
}
trio.triotask <-> aiotask: "LinkedTaskChannel\n.send()/.receive()"
}
}
parent: "parent actor (pure trio)"
parent <-> sub.aio.trio.triotask: "IPC: Context + MsgStream"

View File

@ -0,0 +1,16 @@
# tractor docs diagram: the layered runtime view
# inside any single actor process.
vars: {
d2-config: {
sketch: true
theme-id: 1
pad: 16
layout-engine: elk
}
}
grid-rows: 4
grid-gap: 0
app: "your app: plain trio tasks + nurseries"
tractor: "tractor runtime: actors, portals,\ncontexts + streams, RPC"
ipc: "IPC Channel: msgspec-typed msgs\nover TCP | UDS transports"
os: "OS: one process per actor"

View File

@ -0,0 +1,27 @@
# tractor docs diagram: multi-actor streaming
# pipeline topology from
# examples/full_fledged_streaming_service.py
vars: {
d2-config: {
sketch: true
theme-id: 1
pad: 16
layout-engine: elk
}
}
direction: right
s0: "actor 'streamer_0'" {
fn: "stream_data(0)"
}
s1: "actor 'streamer_1'" {
fn: "stream_data(1)"
}
agg: "actor 'aggregator'" {
fn: "aggregate()"
}
main: "root actor" {
fn: "main()"
}
s0 -> agg: "async for:\nyielded ints"
s1 -> agg: "async for:\nyielded ints"
agg -> main: "deduped\nstream"

View File

@ -0,0 +1,369 @@
Anatomy of the runtime
======================
You can get a long way with ``tractor`` by treating it as "trio_
with nurseries that spawn processes". But once you start asking
*where does my msg actually go?*, *which process is that?* or
*who keeps the phonebook?*, it pays to know how the runtime hangs
together. This page walks the stack top to bottom.
.. d2:: diagrams/runtime_stack.d2
:caption: The four runtime layers inside *every* actor process.
:alt: layer cake of app tasks, tractor runtime, IPC, OS process
:width: 70%
The layer cake
--------------
Every actor process is the same four-layer sandwich:
- **your app**: plain ``trio`` tasks, nurseries and cancel
scopes; nothing special. ``tractor`` is a `structured
concurrency`_ (SC) multi-processing runtime built on trio_ and
the whole pitch is that this layer stays *just trio*: no
callbacks, no futures, no proxy objects.
- **the** ``tractor`` **runtime**: a per-process
:class:`tractor.Actor` running the msg loop and RPC task
scheduler, plus the user-facing primitives layered on it:
:class:`tractor.ActorNursery` (spawning + supervision),
:class:`tractor.Portal` (calling into a peer) and
:class:`tractor.Context` + :class:`tractor.MsgStream`
(SC-linked cross-actor task pairs and streaming).
- **IPC channels**: one :class:`tractor.Channel` per connected
peer, each wrapping a ``MsgTransport`` that ships
msgspec_-typed msgs over TCP or UDS.
- **the OS**: one process per actor, started by a swappable
spawn backend.
The property that holds it all together: SC composes *through*
the layers. A crash in a leaf actor's app task unwinds that
actor's trio tree, ships across its IPC channel as a typed
``Error`` msg, and unwinds the parent's trio tree in turn — the
"SC-transitive supervision protocol" from the README's pitch.
The whole tree cancels and errors like one big trio program; it
just happens to be spread across processes.
One actor, one process, one ``trio.run()``
------------------------------------------
A ``tractor`` "actor" is not a green thread, nor an object with
a mailbox, nor a coroutine: it's one OS process running one
:func:`trio.run` whose root task boots the runtime machinery —
msg loop, RPC task scheduler, IPC server — all embodied by a
single :class:`tractor.Actor` instance.
.. margin:: Shared nothing
Processes buy you a real `shared nothing architecture`_: no
accidentally-shared mutable state, no GIL contention, and
every actor can be inspected (or killed) like any other OS
process.
You rarely construct an :class:`~tractor.Actor` yourself; the
runtime makes exactly one per process and you grab it with
:func:`tractor.current_actor`:
.. code:: python
import tractor
actor = tractor.current_actor() # NoRuntime if none running
print(actor.aid.name) # str name, need not be unique
print(actor.aid.uuid) # uuid4 str, IS unique
print(actor.aid.pid) # the OS pid
print(actor.uid) # legacy (name, uuid) pair
Identity is carried by the ``Aid`` msg-struct (see
``tractor.msg.types``): a ``name``/``uuid``/``pid`` triple
exchanged in the very first "mailbox handshake" whenever two
actors connect. It's what the registrar stores and what shows up
in logs and proc-titles. The older ``.uid`` 2-tuple of
``(name, uuid)`` predates ``Aid`` and is still pervasive across
the codebase; treat it as the legacy spelling of the same
identity.
If this smells like the `actor model`_, sure — but as the
README warns, it probably doesn't look like what you *think* an
actor model looks like, and that's intentional. Here an "actor"
is purely a runtime-unit-of-abstraction: process +
``trio.run()`` + IPC machinery.
IPC: channels, transports, addresses
------------------------------------
Two connected actors talk through a :class:`tractor.Channel`: a
duplex, per-peer msg pipe. Each ``Channel`` wraps a
``MsgTransport`` instance which does the wire work: framing,
encode/decode and the socket itself. The encoding is msgpack
(via msgspec_) and *every* msg is an instance of one of the
runtime's tagged-union :class:`msgspec.Struct` types: the
``Aid`` handshake, ``Start``/``StartAck`` (RPC init),
``Started``/``Yield``/``Stop``/``Return`` (the ctx dialog
phases), ``Error``, etc. There is no raw-bytes mode; the
msg-spec *is* the protocol, which is exactly what lets payloads
be type-limited per-context (see ``pld_spec`` in
:doc:`/guide/context`).
Addresses come in two spellings:
- *unwrapped*: the plain-tuple form you pass to user APIs —
``('127.0.0.1', 1616)`` for tcp, or a
``(<filedir>, <filename>)`` path-pair for uds;
- *wrapped*: the internal ``TCPAddress``/``UDSAddress`` struct
types (plus libp2p-style multiaddr helpers over in
``tractor.discovery``).
You only ever need the tuple form; the runtime wraps and
unwraps at the boundaries.
TCP: the boring default
***********************
The default transport (``'tcp'``) binds each actor's IPC server
to loopback ``('127.0.0.1', <random port>)`` unless told
otherwise, and is the only choice when your tree spans hosts.
Nothing exotic: ``trio`` TCP streams + length-prefixed msgpack
framing.
UDS: same-host, creds included
******************************
Pass ``enable_transports=['uds']`` and actors instead talk over
unix-domain sockets, with socket files placed in the per-user
runtime dir (``$XDG_RUNTIME_DIR/tractor/`` on linux, the
``platformdirs`` equivalent elsewhere). Two perks over tcp on a
single host:
- no ports to fight over; addrs are just file paths,
- the kernel snitches on your peer for free: the listening side
reads the connector's ``pid`` (plus ``uid``/``gid`` on linux)
straight off the socket via ``SO_PEERCRED`` /
``LOCAL_PEERPID`` — no extra handshake msgs required B)
.. warning::
Socket-file lifetime == listening actor lifetime. On
listener teardown the runtime ``os.unlink()``\s the socket
file immediately, so any *late* connection attempt (say, a
sub-actor racing to deregister with a registrar that's
already shutting down) fails with ``FileNotFoundError``.
And ofc, UDS is same-host only.
Here's a full actor tree run entirely over uds:
.. literalinclude:: ../../examples/uds_transport_actor_tree.py
:caption: examples/uds_transport_actor_tree.py
:language: python
Picking a transport
*******************
Transport choice is per-actor via the ``enable_transports``
kwarg accepted by :func:`tractor.open_root_actor` (and proxied
through ``open_nursery()`` when it implicitly boots the
runtime), plus per-child via
``ActorNursery.start_actor(enable_transports=...)``. Two rules
the runtime enforces today:
- exactly ONE transport per actor: multi-transport actors are
on the roadmap but currently raise ``RuntimeError``;
- your ``registry_addrs`` protos must all be in
``enable_transports``: mismatches fail fast with
``ValueError`` instead of (as in darker times) hanging the
registrar handshake forever.
Spawn backends
--------------
How does an actor actually *become* a process? Via a swappable
spawn backend, selected with the ``start_method`` kwarg to
:func:`tractor.open_root_actor`:
``'trio'`` (default)
The home-grown spawner: re-exec the child as
``python -m tractor._child`` using ``trio``'s subprocess
machinery, then bootstrap it over the first IPC exchange
(the parent ships a ``SpawnSpec`` msg carrying all init
state). Supported on all platforms and the most battle
tested choice by far.
``'mp_spawn'`` / ``'mp_forkserver'``
The stdlib :mod:`multiprocessing` start-methods of the same
names (forkserver is posix-only). Mostly interesting for
ecosystem compat and start-up-latency tuning.
``'subint'`` (in development, py3.14+)
On the roadmap (not yet selectable via ``start_method`` on
this release): run each actor as a `PEP 734`_
sub-interpreter (``concurrent.interpreters``) driven on its
own OS thread *inside the parent process* — interpreter-level
shared-nothing isolation with much faster start-up. Yes, this
bends the one-actor-one-process rule; the rest of the model
is unchanged.
The ``TRACTOR_SPAWN_METHOD`` env-var beats any caller-passed
``start_method``, so you can swap backends under an unmodified
app:
.. code:: bash
TRACTOR_SPAWN_METHOD=mp_forkserver python my_app.py
One current limitation worth knowing: ``debug_mode=True`` (the
crash-to-REPL machinery) is only supported on backends whose
child-side runtime is trio-native, e.g. the default ``'trio'``;
see :doc:`/guide/debugging` for the deats.
The registrar
-------------
Discovery needs a phonebook. Every actor, as part of boot,
registers its ``Aid`` and bind-addrs with the *registrar*: an
otherwise ordinary actor (a :class:`tractor.Registrar`, subtype
of :class:`~tractor.Actor`) that keeps the name -> addrs table
for the tree; on graceful exit each actor de-registers itself.
.. margin:: Default registry addrs
With no ``registry_addrs`` passed:
- tcp: ``('127.0.0.1', 1616)``
- uds: ``registry@1616.sock``
in the runtime dir
Who *is* the registrar? Decided at root boot, rendezvous style.
:func:`tractor.open_root_actor` probes each addr in
``registry_addrs`` with a quick connect-ping, then:
- **somebody answered**: this root is a plain actor; it
registers with the existing registrar and binds random
same-proto addrs for its own IPC server;
- **nobody answered**: this root *becomes* the registrar and
binds the registry addrs itself.
So single-program trees need zero config — the root quietly
self-appoints — while multi-program setups share a registrar by
pointing every program at the same ``registry_addrs``. Pass
``ensure_registry=True`` to demand that *this* call create the
registry; it raises if the addrs are already served.
The lookup APIs — :func:`tractor.find_actor`,
:func:`tractor.wait_for_actor`, :func:`tractor.query_actor` and
:func:`tractor.get_registry` — all consult it (after first
checking already-connected peers):
.. literalinclude:: ../../examples/service_daemon_discovery.py
:caption: examples/service_daemon_discovery.py
:language: python
If you bump into "arbiter" in old issues or posts: that's the
legacy name for the same thing, surviving in-code only as the
``Arbiter = Registrar`` class alias; all current terminology is
"registrar"/"registry". Fair warning per the README: this is
still a **very naive** discovery sys (no re-election, no gossip
protocol... yet) and a registrar is expected to outlive its
registrants.
Runtime env vars
----------------
A few env-vars let you re-tune a whole tree *without touching
app code*; each wins over its corresponding kwarg:
.. list-table::
:header-rows: 1
:widths: 30 44 26
* - env-var
- effect
- vs. kwarg
* - ``TRACTOR_LOGLEVEL``
- crank (or silence) console-log verbosity for every actor
in the tree
- beats ``loglevel``
* - ``TRACTOR_SPAWN_METHOD``
- swap the process spawn backend
- beats ``start_method``
* - ``TRACTOR_ENABLE_STACKSCOPE``
- install the ``SIGUSR1`` task-tree-dump handler in every
actor, even outside ``debug_mode`` (see
:doc:`/guide/debugging`)
- OR'd with ``enable_stack_on_sig``
Spotting actors from your shell
*******************************
Every sub-actor sets an OS-level proc-title of the form
``_subactor[<name>@<pid>]`` (via ``setproctitle``, silently
skipped when not installed) so ``ps``/``htop``/``pstree`` show
*which actor is which* at a glance. The README's signature
incantation — watch a tree build and self-destruct live:
.. code:: bash
$TERM -e watch -n 0.1 "pstree -a $$" \
& python examples/nested_actor_tree.py \
&& kill $!
For scripting there are two stable cmdline markers:
.. code:: bash
pgrep -fa '_subactor\[' # live, titled sub-actors
pgrep -fa 'tractor._child' # 'trio'-backend children not
# yet (re)titled
The title also lands in the kernel ``comm`` (truncated to ~15
bytes) which survives into zombie state — that's what
``tractor``'s own test-harness reapers key off. To be crystal
clear about the contract though: you should never *need* a
reaper; if you can create zombie child processes (without using
a system signal) it **is a bug** — please report it!
Logging
-------
The runtime logs through a thin adapter over stdlib
:mod:`logging` that stamps every record with actor + task info.
Two calls get you going:
.. code:: python
from tractor.log import get_console_log, get_logger
log = get_logger(__name__) # actor/task-aware sub-logger
get_console_log('info') # attach console handler @ level
(or just pass ``loglevel='info'`` to
:func:`tractor.open_root_actor` and the console handler comes up
with the runtime).
``tractor`` adds custom levels — and matching logger methods —
that slot between the stdlib ones so you can dial in *which
runtime subsystem* you want to hear from: ``.transport()`` (5),
``.runtime()`` (15), ``.devx()`` (17), ``.cancel()`` (22), plus
a ``PDB`` (500) level for debugger chatter. E.g.
``loglevel='cancel'`` plays the whole cancellation chorus while
staying quiet about transport-layer noise. Beyond that
``tractor`` isn't opinionated about how you consume logs: it's
all stdlib ``logging`` underneath.
Where to next?
--------------
.. seealso::
- :doc:`/guide/context` — the SC-linked cross-actor task API
that rides on every ``Channel``.
- :doc:`/guide/debugging```debug_mode``, the multi-process
REPL and ``stackscope`` task-tree dumps.
- :doc:`/explain/sc-distributed`*why*
one-actor-one-process, and what kind of "actor model" this
is (and isn't).
.. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency
.. _trio: https://github.com/python-trio/trio
.. _actor model: https://en.wikipedia.org/wiki/Actor_model
.. _shared nothing architecture: https://en.wikipedia.org/wiki/Shared-nothing_architecture
.. _msgspec: https://jcristharif.com/msgspec/
.. _PEP 734: https://peps.python.org/pep-0734/

View File

@ -0,0 +1,16 @@
Big ideas
=========
The conceptual core of ``tractor``: what *structured
concurrency* (SC) means once your "tasks" are whole
processes, and how the runtime is layered to deliver
that without (much) magic.
If you only read one page in these docs make it
:doc:`sc-distributed`; if you read two, follow it with
:doc:`architecture`.
.. toctree::
:maxdepth: 2
sc-distributed
architecture

View File

@ -0,0 +1,221 @@
Structured concurrency, across processes
=========================================
``tractor`` makes one bet: the discipline that made
``trio``'s concurrency *readable* — `structured
concurrency`_ (SC) — works just as well when the
"tasks" are whole OS processes talking over a wire.
This page distills what that means, from first
principles, with as little ceremony as possible.
.. margin:: The canon
If SC is new to you, the seminal `blog post`_ is
still the best hour you'll spend on concurrent
programming; the `trio docs`_, wikipedia's SC_
page and the diagrams over at libdill-docs_ round
it out nicely.
SC in one breath
----------------
Structured concurrency is the rule that **concurrency
gets a scope**: every task is spawned *inside* a block
(a ``trio`` *nursery*) and that block **cannot exit
until every task it spawned has finished** — returned,
errored, or been cancelled.
That one rule buys you the properties you already
rely on in sequential code,
- a function call is a *black box*: when it returns,
everything it started is **done** — no secret
background tasks leaking out the sides,
- an exception **always has somewhere to go**: up the
(task) tree to a parent which is, by construction,
still there waiting,
- cancellation has a well defined *shape*: cancel a
scope and it flows down to every task inside it,
and only those.
In short: your **runtime task tree matches your source
code's indentation**. Concurrency you can read.
The leap: process-shaped tasks
------------------------------
Now swap "task" for "process".
A ``tractor`` *actor* is just a Python process running
its own ``trio.run()`` — its own private task tree,
sharing **nothing** with its siblings. You spawn
actors from an :class:`tractor.ActorNursery`, which
behaves exactly the way the name implies,
.. code:: python
async with tractor.open_nursery() as an:
portal = await an.start_actor(
'worker',
enable_modules=[__name__],
)
...
# ^ block exit == every spawned process has
# completed, errored or been cancelled, and
# been **reaped**. No exceptions, no zombies.
so the whole program becomes a *tree of process-trees*
— a `supervision tree`_ in erlang-speak — where every
arrow means "spawned by, **waited on by**, and
supervised by".
.. d2:: diagrams/actor_tree.d2
:caption: A ``tractor`` program: a process tree of
``trio`` task trees; every parent **must wait**
on its children.
:width: 85%
Causality: no process outlives its parent
-----------------------------------------
The stdlib's ``multiprocessing`` (and most "job
queue" systems) treat child processes as
fire-and-forget by default: orphans, zombies, lost
tracebacks and ``kill -9`` cleanup scripts are *your*
problem. ``tractor`` instead inherits ``trio``'s
`causality`_ discipline,
- **no spawning willy-nilly**: every actor is born
from a nursery block with a known parent,
- **lifetimes nest**: a sub-actor's entire process
tree lives strictly inside its parent's nursery
scope,
- **teardown is guaranteed**: when a scope exits (or
errors, or is cancelled) the runtime SIGINTs,
waits, and (only if it must) hard-kills + reaps
everything underneath.
We take the zombie thing personally: *if you can
create orphaned child processes without using a
system signal, it* **is a bug** — and there's a test
suite to back that sentence up.
Errors always propagate (yes, across the wire)
----------------------------------------------
In ``trio``, an exception in any task tears through
its nursery to a parent that must handle it —
`exceptions always propagate`_. ``tractor`` extends
the same guarantee across process boundaries: an
uncaught error in a remote task is
1. captured + serialized in the child,
2. shipped home over IPC as a typed ``Error`` msg,
3. re-raised in the parent **boxed** as a
:class:`tractor.RemoteActorError` carrying the
original type (``.boxed_type``), a rendered remote
traceback, and the erroring actor's id,
while the supervising nursery applies its (currently
*one-cancels-all*, just like ``trio``) strategy to any
sibling actors. A crash three processes deep arrives
at your shell as one coherent, causal traceback chain
— not a silent dead worker and a stuck queue.
Cancellation is a request, supervision is the rule
--------------------------------------------------
Cancellation likewise keeps ``trio``'s semantics
*verbatim*, just transported: cancelling an actor
nursery (or a single :class:`tractor.Context` between
two tasks in different processes) sends an explicit
cancel **request** over IPC which the remote runtime
translates into a real ``trio`` cancel-scope cancel —
then *acks back* so the requester can await
confirmation within a bounded time. Nothing is ever
"just killed" first; graceful always precedes brutal.
Because every cross-process dialog is a pair of
**linked tasks** — one on each side, each inside its
own cancel scope — SC stays *transitive*: supervision
doesn't stop at the process boundary, it tunnels
through every hop of the tree. The wire protocol that
enforces this (a small set of typed msgs:
``Start``/``Started``/``Yield``/``Stop``/``Return``/
``Error``) is detailed in :doc:`/guide/msging` and
:doc:`/guide/context`.
Hold up, is this an "actor model"?
----------------------------------
Let's stop and ask how many canon actor model papers
you've actually read ;)
From our experience many "actor systems" aren't really
"actor models" since they **don't adhere** to the `3
axioms`_ and pay even less attention to the problem of
`unbounded non-determinism`_ — which was the whole
point for creating the model in the first place.
From `the author's mouth`_, the **only** requirement
is `adherence to`_ the `3 axioms`_::
In response to a message, an actor may:
- send a finite number of new messages
- create a finite number of new actors
- designate a new behavior to process subsequent
messages
Want to debate whether that makes us a "real" actor
model? Chime in on our chat, or on one of these issues
*after you've read everything in them* ;)
- `#210 <https://github.com/goodboy/tractor/issues/210>`_
- `#18 <https://github.com/goodboy/tractor/issues/18>`_
``tractor`` adheres — actors exchange msgs, spawn
actors, and swap behaviors — **with no extra API** to
learn. What we *don't* copy is the cultural baggage:
no visible mailboxes, no untyped fire-and-forget
``send()``, no "let it crash" without a supervisor
that actually hears about it, and definitely no
shared-reference *proxy objects* pretending the
network isn't there. If our "actors" don't look like
what you expected, that's **intentional**: being an
actor model is just one property of the system; being
*structured* is the point.
Why processes at all?
---------------------
Python has a GIL; an actor model by definition shares
no state; so the *process* is the natural runtime
unit — you get real multi-core parallelism and hard
memory isolation for free. But the deeper win is
uniformity: because actors only ever talk via msgs
over a :class:`tractor.Channel` (TCP, UDS, more to
come), the **same code** runs your laptop's worker
pool and a multi-host cluster; "distributed" is a
deployment detail, not an API.
It's just ``trio``
------------------
If you remember one framing, make it this: ``tractor``
**is just** ``trio`` — with nurseries that can spawn
processes and streams that can cross them. Same
nursery discipline, same cancellation semantics, same
"how was this not always the API?" feeling, one level
up the process tree.
.. seealso::
:doc:`/explain/architecture` for how the runtime
layers deliver all of the above, and
:doc:`/start/quickstart` to feel it in ~20 lines of
code.
.. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency
.. _SC: https://en.wikipedia.org/wiki/Structured_concurrency
.. _blog post: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/
.. _trio docs: https://trio.readthedocs.io/en/latest/
.. _libdill-docs: https://sustrik.github.io/libdill/structured-concurrency.html
.. _supervision tree: https://www.erlang.org/doc/design_principles/des_princ.html
.. _causality: https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#c-c-c-c-causality-breaker
.. _exceptions always propagate: https://trio.readthedocs.io/en/latest/design.html#exceptions-always-propagate
.. _the author's mouth: https://www.youtube.com/watch?v=7erJ1DV_Tlo&t=162s
.. _adherence to: https://www.youtube.com/watch?v=7erJ1DV_Tlo&t=1821s
.. _3 axioms: https://en.wikipedia.org/wiki/Actor_model#Fundamental_concepts
.. _unbounded non-determinism: https://en.wikipedia.org/wiki/Unbounded_nondeterminism

View File

@ -0,0 +1,309 @@
Infected ``asyncio``
====================
``tractor`` is "just trio_", but the Python world is packed with
libraries that only speak ``asyncio``: websocket stacks, vendor
SDKs, that one exchange client you can't route around. Rather than
make you rewrite them, ``tractor`` lets you *quarantine* them inside
a dedicated subactor which runs both event loops at once, with full
`structured concurrency`_ (SC) guarantees maintained across the
loop boundary *and* the process tree.
In the project's own words:
Yes, we spawn a python process, run ``asyncio``, start ``trio``
on the ``asyncio`` loop, then send commands to the ``trio``
scheduled tasks to tell ``asyncio`` tasks what to do XD
We call this "infected ``asyncio``" mode: the subactor's stdlib
loop runs as the *host* with ``trio`` embedded on top in `guest`_
mode, and your ``trio`` tasks drive ``asyncio`` tasks through
a linked, SC-supervised, in-memory channel.
.. d2:: diagrams/infected_aio.d2
:caption: One process, two schedulers: ``trio`` rides the
``asyncio`` loop as a guest while the parent speaks plain
``tractor`` IPC, none the wiser.
:alt: parent actor connected over IPC to a subactor whose
asyncio loop hosts trio in guest mode, with a
LinkedTaskChannel pairing a trio task to an asyncio task
.. note::
Infected ``asyncio`` mode is **experimental**: it works (we
beat on it plenty) but parts of the API surface and some
edge-case semantics are still settling. Got opinions on the
interop design? Feel free to sling them in `#273`_!
How the infection takes hold
----------------------------
A normal subactor boots by running the ``tractor`` runtime's task
tree directly under ``trio.run()``. Pass ``infect_asyncio=True``
at spawn time and the child's entrypoint changes shape entirely:
1. the process starts the stdlib loop via ``asyncio.run()``,
2. the first ``asyncio`` task calls
``trio.lowlevel.start_guest_run()``, embedding the ``trio``
scheduler *inside* the already running ``asyncio`` loop (the
upstream `guest`_-mode feature),
3. the regular ``tractor`` runtime then boots on the guest
``trio`` side and connects back to its parent like any other
subactor.
.. margin:: Symptoms
Looks like your stdlib event loop has caught a case of "the
trios"! Don't worry, you'll barely notice; and if anything
gets too bad, your parents will know about it B)
Both schedulers interleave in a single thread, no GIL gymnastics
required. From the rest of the actor tree the infected child is
indistinguishable from any other actor: same IPC protocol, same
supervision and cancellation semantics, same zombie-safety
guarantees. The difference is purely internal: ``trio`` tasks in
that process can start and drive ``asyncio`` tasks through the
``tractor.to_asyncio`` API.
Spawning an infected subactor
-----------------------------
Just flip the flag on :meth:`tractor.ActorNursery.start_actor`:
.. code:: python
async with tractor.open_nursery() as an:
portal = await an.start_actor(
'aio_side',
enable_modules=[__name__],
infect_asyncio=True,
)
The one-shot convenience ``ActorNursery.run_in_actor()`` accepts
the same flag. The ``to_asyncio`` APIs may **only** be called from
tasks inside an infected actor; calling them anywhere else raises
a loud ``RuntimeError``. You can introspect at runtime with
``tractor.current_actor().is_infected_aio()``.
Linking tasks with ``open_channel_from()``
------------------------------------------
The core primitive is :func:`tractor.to_asyncio.open_channel_from`,
an async context manager which starts your ``asyncio`` function as
a real ``asyncio.Task`` and yields a two-way channel linking it to
the calling ``trio`` task:
.. code:: python
from tractor import to_asyncio
async with to_asyncio.open_channel_from(
aio_main, # async def aio_main(chan, **kwargs)
period=0.5, # extra kwargs are passed through
) as (chan, first):
await chan.send('tick')
The semantics deliberately mirror the inter-actor ``Context``
handshake from :doc:`/guide/context`:
- the target fn must declare a parameter literally named ``chan``;
the runtime injects the shared
:class:`~tractor.to_asyncio.LinkedTaskChannel` by keyword.
- the ``trio`` side blocks at entry until the ``asyncio`` task
calls ``chan.started_nowait(value)``; that value is delivered as
``first``, exactly like the ``(ctx, first)`` pair you get from
``Portal.open_context()`` after the child calls
``ctx.started()``.
- a first value **must** be sent from the ``asyncio`` side or the
``trio`` side will never unblock.
- on block exit the pair is torn down *together*; neither task can
outlive the other (more on this below).
A full example: the echo server
-------------------------------
Here's the canonical demo, a round-trip echo service where the
``asyncio`` task is told what to do by a ``trio`` task which is in
turn driven over IPC by the root actor:
.. literalinclude:: ../../examples/infected_asyncio_echo_server.py
:caption: examples/infected_asyncio_echo_server.py
:language: python
What's going on?
- there are three task layers: the root actor's pure ``trio``
task, the infected child's ``trio``-side ``@tractor.context``
endpoint (``trio_to_aio_echo_server()``), and the child's
``asyncio`` task (``aio_echo_server()``).
- two ``started``-style handshakes compose: the aio task's
``chan.started_nowait('start')`` unblocks the child's
``open_channel_from()`` entry, then the child relays that same
value up via ``await ctx.started(first)`` which unblocks the
root's ``open_context()`` entry. Synchronization all the way
down, er, up.
- each round trip flows: root ``stream.send()`` -> IPC -> child
``async for msg in stream`` -> ``chan.send(msg)`` -> aio
``await chan.get()`` -> ``chan.send_nowait()`` -> child
``chan.receive()`` -> ``stream.send(out)`` -> IPC -> root.
- when the root breaks out of its stream loop and exits the
context block, the child's stream ends, its channel block exits,
and the ``asyncio`` task is reaped along with it; the final
``portal.cancel_actor()`` then tears down the whole process. No
orphaned ``asyncio`` tasks, no zombie procs; if you manage to
create either it **is a bug**.
``LinkedTaskChannel``: one channel, two sides
---------------------------------------------
The same channel object is shared by both tasks; which methods you
call depends on which loop schedules your task. The ``trio`` side
gets a standard ``trio.abc.Channel`` interface while the
``asyncio`` side gets queue-flavored, mostly-sync methods:
.. list-table::
:header-rows: 1
:widths: 14 36 50
* - side
- call
- what it does
* - ``trio``
- ``await chan.send(item)``
- ship ``item`` to the ``asyncio`` task (enqueues onto an
internal ``asyncio.Queue``).
* - ``trio``
- ``await chan.receive()``
- wait for the next value from the ``asyncio`` side; the
channel also supports ``async for``.
* - ``trio``
- ``await chan.wait_for_result()``
- block until the ``asyncio`` task completes; return its
final result or raise its (translated) error.
* - ``trio``
- ``chan.subscribe()``
- acm yielding a ``BroadcastReceiver`` so N local tasks can
each consume a copy of the inbound stream (see below).
* - ``trio``
- ``chan.cancel_asyncio_task()``
- explicitly request cancellation of the linked ``asyncio``
task.
* - ``asyncio``
- ``chan.started_nowait(value)``
- deliver the "first" value; unblocks the ``trio`` side's
``open_channel_from()`` entry (mirrors ``ctx.started()``).
* - ``asyncio``
- ``await chan.get()``
- wait for the next value sent from the ``trio`` side.
* - ``asyncio``
- ``chan.send_nowait(item)``
- push a value to the ``trio`` side without blocking.
Fan-out with ``.subscribe()``
*****************************
Just like :meth:`tractor.MsgStream.subscribe` does for IPC
streams, ``chan.subscribe()`` lets multiple local ``trio`` tasks
each receive *every* value sent from the single ``asyncio`` task:
.. code:: python
async with chan.subscribe() as bcast:
async for msg in bcast:
...
The underlying broadcast machinery is lazily allocated on first
use and is *not* reversible for the channel's remaining lifetime,
so only reach for it when you actually want the fan-out.
One-shot calls with ``run_task()``
----------------------------------
When you just want a single ``asyncio`` result and no streaming
dialog, skip the channel ceremony and use
:func:`tractor.to_asyncio.run_task`:
.. code:: python
import asyncio
from tractor import to_asyncio
async def aio_fetch(url: str) -> str:
await asyncio.sleep(0.3) # pretend-IO, aio style
return f'<html>sup {url}</html>'
# from any trio task inside the infected actor:
page = await to_asyncio.run_task(aio_fetch, url='https://x.io')
It schedules the fn as an ``asyncio.Task``, waits for completion
and hands the return value back to ``trio``; think of it as the
cross-loop sibling of ``ActorNursery.run_in_actor()``. Errors and
cancellation are translated exactly as for channels.
Cross-loop errors and cancellation
----------------------------------
The paired tasks are *SC linked*: exception and cancel handling
tears down **both** sides on any unexpected error or cancellation,
in either loop. There is no fire-and-forget mode; a
``LinkedTaskChannel`` is a supervision scope just like a
``Context`` is across processes.
Because each loop has its own (incompatible) cancellation and exit
machinery, boundary crossings are translated into dedicated
exception types, all importable from ``tractor.to_asyncio``:
.. list-table::
:header-rows: 1
:widths: 26 22 52
* - exception
- raised in
- meaning
* - ``AsyncioCancelled``
- the ``trio`` task
- the linked ``asyncio`` task was cancelled by itself or
a 3rd party (i.e. *not* by the ``trio`` side).
* - ``AsyncioTaskExited``
- the ``trio`` task
- the ``asyncio`` task returned/exited early while the
``trio`` side still held the link open.
* - ``TrioCancelled``
- the ``asyncio`` task
- the ``trio`` side was cancelled (or crashed) so the
``asyncio`` task is being torn down per SC rules.
* - ``TrioTaskExited``
- the ``asyncio`` task
- the ``trio`` side exited gracefully while the ``asyncio``
task was still running; a "clean shutdown" signal much
like closing a ``trio`` mem-chan.
By default ``open_channel_from(suppress_graceful_exits=True)``
absorbs the two ``*TaskExited`` signals so happy-path teardown
stays silent; pass ``False`` when your app wants to handle early
peer-exit explicitly.
Past the task pair, everything composes with the normal actor
story: an unhandled ``asyncio`` error is translated into the
``trio`` side, propagates out of your ``@tractor.context``
endpoint, and arrives at the parent boxed as
a :class:`tractor.RemoteActorError`. One SC discipline,
end-to-end, across loops *and* processes.
Breakpoints in ``asyncio`` tasks
--------------------------------
Yes, the multi-actor REPL works here too. With
``debug_mode=True`` enabled on your tree the ``trio`` side of an
infected actor can ``await tractor.pause()`` as usual, and with
greenback enabled (``maybe_enable_greenback=True``) even the
builtin ``breakpoint()`` works from *inside* ``asyncio`` tasks;
see ``examples/debugging/asyncio_bp.py`` for the full tour. The
root-TTY locking dance behind all this is covered in
:doc:`/guide/debugging`.
Where to next?
--------------
.. seealso::
- :doc:`/guide/context` for the inter-actor handshake and
streaming APIs which this whole interop layer mirrors.
- :doc:`/guide/msging` for typing the payloads you shuttle
between actors (and loops).
- :doc:`/guide/debugging` for the multi-process REPL that
keeps working even when your loop has "the trios".
.. _trio: https://github.com/python-trio/trio
.. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency
.. _guest: https://trio.readthedocs.io/en/stable/reference-lowlevel.html?highlight=guest%20mode#using-guest-mode-to-run-trio-on-top-of-other-event-loops
.. _#273: https://github.com/goodboy/tractor/issues/273

View File

@ -0,0 +1,285 @@
Cancellation and error propagation
==================================
``tractor`` supports ``trio``'s cancellation_ system *verbatim*,
then extends it across process boundaries. If you know how to
cancel a task in ``trio`` you already know how to cancel an actor —
and its whole subtree — in ``tractor``; the runtime's job is making
that statement hold over IPC with every structured concurrency (SC)
guarantee intact.
The ground rules,
- a remote actor is **never** cancelled unless explicitly requested
(by a parent or peer), unless supervision demands it (an error
triggered one-cancels-all teardown), or unless there's a bug in
``tractor`` itself (please report it!),
- (remote) errors `always propagate`_ back to the parent
supervisor; nothing is silently dropped on the floor,
- every spawned process gets reaped no matter how it dies; if you
can create a zombie child process (without using a system signal)
it **is a bug**.
``trio`` cancellation, across the wire
--------------------------------------
Locally everything is bog-standard ``trio``: nurseries, cancel
scopes, timeouts. ``tractor`` adds exactly one twist: a cancel
scope can't physically reach into another process, so the runtime
*relays cancellation as messages*. Concretely,
- cancelling an *actor* means sending it a runtime-cancel request
msg; the target then runs its own graceful teardown — cancelling
RPC tasks, closing channels, exiting its :func:`trio.run` — and
acks the request back to the canceller,
- cancelling a single *cross-actor task* works through the
:class:`tractor.Context` layer: each ``ctx`` task-pair is
cancel-scope-linked over IPC such that either side erroring or
cancelling relays an equivalent error to the other side (see
:doc:`/guide/context` for the gory details),
- a cancel is therefore always a *request with an ack*: the
canceller does a **bounded wait** for confirmation and escalates
if the peer is unresponsive (see the teardown ladder below).
One-cancels-all supervision
---------------------------
An :class:`tractor.ActorNursery` supervises subactors `exactly like
trio`_ nurseries supervise tasks: when one child errors, the error
propagates to the supervising block and **all** sibling subactors
get cancelled before the error continues bubbling up the (process)
tree.
.. d2:: diagrams/error_propagation.d2
:caption: One-cancels-all: no zombies, no lost errors.
:alt: error propagation up a subactor tree
:width: 80%
.. literalinclude:: ../../examples/remote_error_propagation.py
:caption: examples/remote_error_propagation.py
:language: python
What's going on here?
- three healthy actors are spawned as daemons via
:meth:`tractor.ActorNursery.start_actor`; left alone they'd
happily idle forever,
- a fourth actor runs ``assert_err()`` via ``.run_in_actor()`` and
promptly trips its ``assert 0``,
- the resulting ``AssertionError`` ships back over IPC as a
serialized error msg and re-raises *boxed* inside the nursery
block as a :class:`tractor.RemoteActorError`,
- the nursery reacts like any ``trio`` nursery would: it cancels
the three healthy siblings (graceful runtime-cancel requests,
acks awaited), reaps all four processes, then re-raises,
- ``trio.run(main)`` sees that same ``RemoteActorError`` in the
parent-most process — propagation is end-to-end or bust.
This one-cancels-all style is currently the *only* supervision
strategy offered (it's the one ``trio`` gives you); more
`erlang strategies`_ are roadmap, see the bottom of this page.
The boxed-error bestiary
------------------------
All remote failures arrive locally as one of a small set of
exception types, each carrying enough metadata to work out *who*
failed, *where*, and *why*.
``RemoteActorError``
********************
The workhorse: a "boxed" exception relayed over IPC from another
actor. The original error's type, traceback string and msgdata are
preserved so you can pattern-match on what actually went wrong
remotely,
- ``.boxed_type``: the reconstructed **type** of the original
remote exception (``ValueError``, ``NameError``, what have you),
- ``.src_uid``: the ``(name, uuid)`` pair of the actor where the
error *originated*,
- ``.relay_uid`` / ``.relay_path``: when an error crosses more than
one actor boundary (grandchild -> child -> root) every relaying
actor is recorded; multi-hop boxings are lovingly referred to as
"inceptions" in the runtime internals,
- ``.pformat()``: a rich "tb box" rendering of the remote traceback
for your logs or REPL.
.. code:: python
try:
async with portal.open_context(ep) as (ctx, first):
...
except tractor.RemoteActorError as rae:
if rae.boxed_type is ValueError:
... # the remote task raised `ValueError`
``ContextCancelled``
********************
The cancel-ack for a cross-actor task pair: raised when a
:class:`tractor.Context` task is cancelled *by request*. Its
``.canceller`` attr is the uid of the actor which **requested** the
cancel, which powers the key rule,
- if **you** requested it (you called
:meth:`tractor.Context.cancel`) the resulting ctxc is *absorbed*
at ``open_context()`` exit: an expected outcome, not an error,
- if **anyone else** did — the peer task, or some third-party actor
— it *raises* locally so your code always hears about it.
The full self- vs. cross-cancel semantics are a core teaching point
of :doc:`/guide/context`; go read them there.
``MsgTypeError``
****************
An IPC-payload "type error": a msg violated the dialog's declared
payload spec. See :doc:`/guide/msging` for the typed-messaging
system which enforces it.
``TransportClosed``
*******************
The underlying IPC transport (TCP stream, UDS socket, ...) died or
closed out from under a channel. You'll normally only see this
surface when a peer hard-exits without any graceful runtime
teardown; the supervision machinery treats unexpected transport
loss on a busy channel as a failure and tears down accordingly.
Pick your blast radius
----------------------
Three cancel surfaces, three scopes of effect; choose the smallest
hammer that does the job.
.. list-table::
:header-rows: 1
:widths: 36 34 30
* - surface
- cancels
- typical use
* - :meth:`tractor.ActorNursery.cancel`
- every subactor in the nursery
- whole-tree teardown
* - :meth:`tractor.Portal.cancel_actor`
- one actor: full runtime + proc
- daemon teardown
* - :meth:`tractor.Context.cancel`
- exactly one remote task
- surgical task cancel
``ActorNursery.cancel()``
*************************
The big red button: gracefully cancel every subactor supervised by
the nursery, in parallel, with the escalation discipline below
applied per-child. It's invoked for you whenever an error hits the
nursery block (one-cancels-all); call it yourself for an orderly
early shutdown. Passing ``hard_kill=True`` skips the graceful phase
and goes straight to OS-level process termination — rarely what you
want outside tests.
``Portal.cancel_actor()``
*************************
Cancel one **whole actor**: its entire runtime, every task it's
scheduled, and (for subactors) the OS process, via a graceful
runtime-cancel request,
.. code:: python
await portal.cancel_actor() # bounded wait, bool result
await portal.cancel_actor(
raise_on_timeout=True, # no ack in time?
) # -> `ActorTooSlowError`
The wait for the peer's ack is *bounded* (default
``Portal.cancel_timeout = 0.5`` seconds, tunable per call via
``timeout=``). By default a missed ack just returns ``False``; with
``raise_on_timeout=True`` you instead get an ``ActorTooSlowError``
(from ``tractor._exceptions``) so *your* code can escalate per SC
discipline — exactly what the nursery's own teardown does
internally before resorting to OS-level signalling.
Note the granularity: this cancels an **actor**, not a task. For
one remote task use the ``Context`` layer instead.
``Context.cancel()``
********************
Request cancellation of exactly one remote task: the peer task of
an open :class:`tractor.Context`. Two things to keep straight,
- it cancels the **remote** side only; a ``Context`` is *not* a
:class:`trio.CancelScope` and your local task keeps running until
you exit the ``open_context()`` block,
- the resulting :class:`tractor.ContextCancelled` is absorbed
locally (you asked for it, after all) per the self- vs.
cross-cancel rule above.
Again, :doc:`/guide/context` covers this dance in depth.
Graceful first, hard as a last resort
-------------------------------------
.. margin:: REPL-safe by design
The hard-kill path is *skipped* whenever an actor in the tree
holds the debug-REPL lock (``debug_mode=True`` flavors):
SIGTERM raining down on a tree mid-``pdb`` session would
clobber your prompt. See :doc:`/guide/debugging`.
Every process teardown in ``tractor`` walks the same escalation
ladder, top rung first,
1. **graceful cancel request**: a runtime-cancel msg over IPC; the
target actor cancels its tasks, closes its channels and exits
its :func:`trio.run` cleanly,
2. **soft wait**: the parent waits (bounded) for the child process
to exit on its own,
3. **SIGTERM**: no ack within the bounded wait (internally an
``ActorTooSlowError``) escalates to ``proc.terminate()``,
4. **SIGKILL ultimatum**: still alive after the hard-kill timeout
(~1.6s)? The runtime logs that the "T-800" has been deployed to
collect the zombie and issues ``proc.kill()``. No survivors.
The result is the **no-zombies guarantee**: ``tractor`` tries to
protect you from zombies, no matter what. Quoting the project
manifesto,
If you can create zombie child processes (without using
a system signal) it **is a bug**.
Run the quickstart's self-destructing process-tree demo
(``examples/parallelism/we_are_processes.py``, walked through in
:doc:`/start/quickstart`) under a ``pstree`` watcher and try to
catch a
straggler; we'll wait B)
Roadmap: ``erlang``-style strategies
------------------------------------
One-cancels-all is ``trio``'s strategy and, for now, the only one
``tractor`` ships. Pluggable `erlang strategies`_ — one-for-one
restarts, rest-for-one, transient/permanent child specs and friends
(see the `supervision strategies`_ canon) — are a long-standing
roadmap item tracked in `#22`_. If supervisors are your jam that
issue is the place to sling opinions.
.. seealso::
- :doc:`/guide/context` — the cross-actor task layer where
per-task cancellation actually lives,
- :doc:`/guide/msging` — the typed msg layer that raises
:class:`tractor.MsgTypeError`,
- :doc:`/guide/debugging` — what cancellation does (and very
carefully does *not* do) while a REPL is up.
.. _cancellation: https://trio.readthedocs.io/en/latest/reference-core.html#cancellation-and-timeouts
.. _exactly like trio: https://trio.readthedocs.io/en/latest/reference-core.html#cancellation-semantics
.. _always propagate: https://trio.readthedocs.io/en/latest/design.html#exceptions-always-propagate
.. _erlang strategies: https://learnyousomeerlang.com/supervisors
.. _supervision strategies: https://www.erlang.org/doc/system/sup_princ.html
.. _#22: https://github.com/goodboy/tractor/issues/22

View File

@ -0,0 +1,127 @@
Higher-level cluster APIs
=========================
Sometimes you don't want a hand-crafted supervision tree; you want
"a pile of workers, one per core, now please". For that there's
:func:`tractor.open_actor_cluster`: a convenience wrapper which
spawns a *flat* cluster of subactors and hands you back a portal to
each,
.. code:: python
@acm
async def open_actor_cluster(
modules: list[str], # RPC allowlist for workers
count: int = cpu_count(), # one per core by default
names: list[str]|None = None, # default: 'worker_{i}'
hard_kill: bool = False, # fwd to `an.cancel()`
**runtime_kwargs, # fwd to `open_root_actor()`
) -> AsyncGenerator[dict[str, tractor.Portal], None]:
A cluster in one block
----------------------
.. literalinclude:: ../../examples/quick_cluster.py
:caption: examples/quick_cluster.py
:language: python
Walkthrough,
- ``open_actor_cluster(modules=[__name__])`` concurrently spawns
one subactor per detected core (per
:func:`multiprocessing.cpu_count`); the ``modules`` list is the
usual ``enable_modules``-style capability allowlist so workers
may run functions defined in this module,
- it yields a ``dict[str, tractor.Portal]`` mapping worker name to
portal; note the keys get prefixed with the *spawning* actor's
name, so from the root you'll see ``'root.worker_0'``,
``'root.worker_1'``, etc.,
- a plain :class:`trio.Nursery` then fans out one
``portal.run(sleepy_jane)`` per worker; each prints its actor
``.uid`` from inside its own process then naps forever — what
runs *inside* each worker (and how many tasks you point at it)
is entirely yours to compose,
- ``tractor.trionics.collapse_eg()`` un-nests the strict
``ExceptionGroup`` wrapping so the demo's ``KeyboardInterrupt``
surfaces as itself instead of arriving eg-boxed,
- on block exit the whole fleet is torn down for you via
:meth:`tractor.ActorNursery.cancel`; pass ``hard_kill=True`` at
open time to skip straight to OS-level termination instead of
the graceful ladder described in :doc:`/guide/cancellation`.
Sizing, naming, fleet-wide options
----------------------------------
``count`` doesn't have to be core-count and the auto-generated
``'worker_{i}'`` names are just the default; pass your own (the
length must match ``count`` or you get a ``ValueError``). Any
extra ``**runtime_kwargs`` pass through verbatim to
:func:`tractor.open_root_actor`, so fleet-wide runtime options are
one kwarg away,
.. code:: python
async with tractor.open_actor_cluster(
modules=['mylib.workers'],
count=4,
names=['scout', 'miner', 'smelter', 'smith'],
debug_mode=True, # whole-fleet crash-to-REPL
) as portal_map:
...
From here the composition patterns are the usual ``tractor`` fare:
``portal.run()`` for one-shot calls (as in the demo), or — for a
persistent bidirectional dialog per worker — concurrently enter N
``portal.open_context()`` blocks with
``tractor.trionics.gather_contexts()``; see :doc:`/guide/context`
for that whole layer.
Clusters vs. nurseries
----------------------
.. d2:: diagrams/actor_tree.d2
:margin:
:caption: The general shape: arbitrary nesting. A cluster is
this, minus the nesting.
:alt: a nested supervision tree of subactors
``open_actor_cluster()`` is sugar, not a new primitive: under the
hood it's just :func:`tractor.open_nursery` plus N concurrent
``start_actor()`` calls plus a ``.cancel()`` on the way out. Reach
for it when,
- you want a *flat*, homogeneous fleet (classic worker-pool or
map-style fan-out shapes),
- "one per core" — or a fixed ``count`` — is the right sizing
story,
- every child can share the same spawn options.
Drop down to a raw :class:`tractor.ActorNursery` when the topology
gets any fancier: nested trees, heterogeneous children, per-child
``debug_mode``/transport/module options, daemons mixed with
one-shot workers, and so on (see :doc:`/guide/parallelism` for a
hand-rolled pool). Either way the supervision semantics are
identical: one-cancels-all error propagation and the no-zombies
guarantee from :doc:`/guide/cancellation` apply to clusters too.
Provisional, by design
----------------------
.. note::
APIs in this section are considered **provisional**: the
signature and semantics of :func:`tractor.open_actor_cluster`
may shift as higher-level supervision machinery lands. We
encourage you to try it and provide feedback — the
`matrix channel`_ is the place to say hi, and `#22`_ tracks the
broader supervisor-strategy roadmap.
.. seealso::
- :doc:`/guide/parallelism` — worker pools built "by hand" with
plain actor nurseries (and why that's easy peasy),
- :doc:`/guide/cancellation` — the teardown machinery a cluster
inherits for free.
.. _matrix channel: https://matrix.to/#/!tractor:matrix.org
.. _#22: https://github.com/goodboy/tractor/issues/22

View File

@ -0,0 +1,344 @@
The ``Context``: a cross-actor task pair
=========================================
If you've written any trio_ you already know the contract: every
task lives in a nursery, errors always propagate, cancellation is
scoped, and nothing leaks. ``tractor`` extends that exact contract
*across processes* — the same guarantees from the seminal
`blog post`_, just with the nursery split across two memory
domains. The primitive that does it is :class:`tractor.Context`: a
**linked pair of tasks**, one in each of two actors, supervised as
a single `structured concurrency`_ (SC) scope over IPC.
.. d2:: diagrams/context_handshake.d2
:caption: The SC-transitive supervision protocol, msg by msg.
:alt: sequence diagram of the context handshake msg flow
Pretty much everything else is (or is slated to be) built on this
one primitive: ``ActorNursery.run_in_actor()`` is a convenience
for "spawn, open a context, await the result, tear down"; plain
``Portal.run()`` RPC is planned to be re-implemented on top of it;
the multi-process debugger's tree-wide REPL lock rides one. Grok
this page and the rest of the library reads as convenience
wrappers B)
The endpoint contract
---------------------
A context endpoint is an async function decorated with
:func:`tractor.context` which declares **a param annotated**
``tractor.Context`` — any param name you like, the annotation is
what's required:
.. code:: python
@tractor.context
async def trainer(
ctx: tractor.Context,
model: str,
) -> str:
await ctx.started('ready')
return f'trained {model}'
.. margin:: Who am I talking to?
Inside any context task
:func:`tractor.current_ipc_ctx` returns the
``Context`` bound to the current task; handy
in helpers that don't take ``ctx`` explicitly.
The parent (aka "opener") side invokes it through a
:class:`tractor.Portal` using ``Portal.open_context()``, passing
any extra kwargs which are shipped over the wire as the remote
task's arguments. Since the target fn is referenced by module
path, that module must be listed in the peer actor's
``enable_modules`` allowlist — RPC capability is always opt-in.
The decorator also accepts a ``pld_spec``: a type (union) which
every payload in the dialog is validated against, upgrading your
msgs to a typed contract enforced via :exc:`tractor.MsgTypeError`.
Validation strictness follows the "`cheap or nasty`_"
`(un)protocol`_ pattern: the one-shot ``Started`` payload gets the
nasty treatment (stringently round-trip checked before it's even
sent) while high-rate stream payloads stay cheap (checked only
receiver side).
The handshake, on the wire
--------------------------
Every context runs one instance of ``tractor``'s "SC-transitive
supervision protocol": a tiny fixed grammar of msgspec_-typed msgs
encapsulating *all* RPC dialogs between actors. *Transitive*
because each IPC link obeys the same rules a local nursery does —
starts are acked, completion is awaited, errors and cancels always
relay — so chaining links across a process tree composes into one
tree-wide SC scope.
The figure up top shows a full dialog; in order:
``Start``
sent by ``Portal.open_context()``: "schedule a task running
this function with these kwargs".
``StartAck``
the peer runtime confirms the task is scheduled and that the
endpoint really is a context-style fn.
``Started``
emitted when the child task calls
:meth:`tractor.Context.started`; carries the first payload
and unblocks the parent's entry of ``open_context()``.
``Yield``
one per :meth:`tractor.MsgStream.send`, flowing in *either*
direction while a stream is open.
``Stop``
graceful end-of-stream: the far side's ``async for``
terminates cleanly.
``Return``
the child fn returned; its value becomes the context's final
result. If the child raised instead, an ``Error`` msg takes
this slot carrying the boxed traceback.
``ctx.started()``: just like ``task_status.started()``
*******************************************************
The startup phase is a deliberate clone of
:meth:`trio.Nursery.start` semantics: the child decides when it's
"up", optionally handing back a first value, and the parent stays
blocked until that moment:
.. code:: python
# trio, in-process
first = await nursery.start(child_fn)
# tractor, cross-process
async with portal.open_context(child_fn) as (ctx, first):
...
The ``as (ctx, first)`` tuple is exactly that pair: the
:class:`tractor.Context` handle plus whatever value the child
passed to ``await ctx.started(value)``. And readiness is not
optional — for instance opening a stream before ``.started()``
has been called raises a ``RuntimeError``; handshake first, then
dialog.
Bidirectional streaming over a context
--------------------------------------
The canonical ping-pong (design history: `#53`_, `#223`_) — a
full-duplex msg stream between a parent and its spawned peer:
.. literalinclude:: ../../examples/rpc_bidir_streaming.py
:caption: examples/rpc_bidir_streaming.py
:language: python
What's going on?
- ``start_actor()`` spawns the daemon-style subactor
``'rpc_server'`` with this very module in its allowlist.
- ``portal.open_context(simple_rpc, data=10)`` fires the
``Start`` msg then blocks until the child task calls
``await ctx.started(data + 1)`` — hence ``sent == 11``.
- both tasks enter ``ctx.open_stream()``: a stream dialog is only
fully open once *each* side has entered its block.
- the parent seeds the first ``'ping'``; each side then echoes
the other, one ``Yield`` msg per ``stream.send()``.
- after the 9th pong the parent ``break``\ s (10 pings sent in
total) and exits its stream block, which sends ``Stop``; the
child's ``async for`` completes gracefully and its ``else``
clause asserts all 10 pings arrived.
- the 10th in-transit pong? Discarded by the implicit drain at
``open_context()`` exit, which runs the dialog down to the
child's ``Return`` (here ``None``).
- daemon actors live until told otherwise:
``portal.cancel_actor()`` reaps the subactor explicitly.
Results: the ``Return`` leg
---------------------------
Every context resolves to a final outcome. Wait on it explicitly
from the parent side:
.. code:: python
async with portal.open_context(ep) as (ctx, first):
...
result = await ctx.wait_for_result()
or just exit the block — ``__aexit__`` implicitly drains the msg
flow until the ``Return`` (or ``Error``) arrives, discarding any
in-transit ``Yield``\ s on the way. Either way the rule of
`causality`_ holds exactly as in a local nursery: **the opener
never unblocks before the remote task is done**.
For post-hoc inspection (think supervision/restart logic) the ctx
also exposes ``Context.outcome``, ``.maybe_error`` and
``.has_outcome`` — where a "result" might well be the error the
dialog ended with.
Cancellation semantics
----------------------
The part you actually came for; read it twice B)
A context's two tasks are **cancel-scope-linked across the IPC
boundary**: whatever ends one side — error, cancellation, plain
old return — is relayed such that the other side ends
equivalently. No silent half-open dialogs, no orphaned remote
tasks, ever.
``ctx.cancel()`` cancels the *remote* task
*******************************************
:meth:`tractor.Context.cancel` requests cancellation of the
**remote** task only:
.. code:: python
async with portal.open_context(ep) as (ctx, first):
await accomplish_things(ctx)
await ctx.cancel() # remote task, NOT me
A :class:`tractor.Context` is **not** a :class:`trio.CancelScope`:
the call doesn't (and can't) cancel your local task. It sends the
cancel request and waits a bounded ``timeout`` for the peer
runtime's ``CancelAck``, then your code proceeds to the block exit
as normal.
Compare scopes here: ``Portal.cancel_actor()`` is the big hammer
which cancels the peer's **entire runtime** (and thus process);
``ctx.cancel()`` is the per-dialog scalpel.
``ContextCancelled`` and the absorption rule
*********************************************
When a context task gets cancelled *by request* the requestee's
runtime reports back with a :exc:`tractor.ContextCancelled`
("ctxc") whose ``.canceller`` field holds the uid of the actor
which asked. That one field decides what you observe:
**you requested it**
i.e. ``ctxc.canceller == tractor.current_actor().uid``: the
ctxc is **absorbed** at ``open_context()`` exit — nothing
raises in your block. You asked for a graceful stop and got
it; if you care, ``await ctx.wait_for_result()`` hands the
ctxc back as a plain *value* for inspection.
**anyone else requested it**
the peer cancelling itself, or some third actor cancelling it
from the side: the ctxc **is raised** in your block. From
your scope's perspective a task you depend on was killed out
from under you and SC demands you hear about it — exactly
like a sibling crash in a `nursery`_.
In code:
.. code:: python
try:
async with portal.open_context(ep) as (ctx, first):
...
except tractor.ContextCancelled as ctxc:
# can only be a peer- or third-party cancel;
# self-requested cancels are absorbed at exit.
assert ctxc.canceller != tractor.current_actor().uid
This self- vs cross-cancel split is what makes explicit teardown
*composable*: a supervisor cancels its dialogs without try/except
noise, while unexpected cancellation anywhere in the tree still
propagates loudly like any other failure.
.. warning::
Once ``ctx.cancel()`` has been called the dialog is done: a
subsequent ``ctx.open_stream()`` raises ``RuntimeError``.
For introspection the ctx exposes trio-flavored status props:
``.cancel_called`` (this side requested), ``.cancel_acked`` (peer
confirmed), ``.cancelled_caught`` and ``.canceller``
deliberately mirroring :class:`trio.CancelScope` naming.
Errors propagate, both ways
---------------------------
A crash on either end tears down the pair, SC style:
- **child raises**: the exception ships back as an ``Error`` msg
and re-raises in the parent block boxed as a
:exc:`tractor.RemoteActorError`; the original class rides along
as ``.boxed_type`` with ``.src_uid`` naming the crashed actor.
- **parent raises** (or is cancelled) inside the block: an
equivalent error/cancel is relayed to the child task so it can
never outlive the dialog.
.. code:: python
try:
async with portal.open_context(ep) as (ctx, first):
...
except tractor.RemoteActorError as rae:
if rae.boxed_type is ValueError:
... # remote ValueError, type preserved
Errors that hop through intermediary actors on their way up the
tree ("inceptions" XD) keep the full relay trail in
``.relay_uid`` / ``.relay_path``. Payloads violating your declared
``pld_spec`` surface as the IPC analog of a ``TypeError``:
:exc:`tractor.MsgTypeError`.
Overruns and backpressure
-------------------------
Stream msgs land in a bounded per-context buffer on the receiver
side. A sender that outpaces a non-consuming receiver *overruns*
it and the runtime raises ``StreamOverrun`` (from
``tractor._exceptions``; also a :exc:`trio.TooSlowError`) instead
of buffering without bound — SC discipline applies to memory too.
Your knobs:
- ``msg_buffer_size`` on ``ctx.open_stream()`` sizes the buffer.
- ``allow_overruns=True`` (on ``Portal.open_context()`` and/or
``ctx.open_stream()``) opts in to absorbing overflow instead of
erroring — reasonable for bursty telemetry-ish feeds, just know
you're trading the error for extra buffering.
One context, one stream
-----------------------
A ``MsgStream`` is strictly **one-shot use**: once it closes —
gracefully or not, from either side — it can never be re-opened
on the same ctx. Want another round with the same peer? Open a
fresh context; they're cheap. The full close-vs-cancel teardown
story lives in :doc:`/guide/streaming`.
.. rubric:: Where to next?
:doc:`/guide/streaming` covers the rest of the msg-moving story:
the legacy one-way API, multi-actor pipelines and in-actor
broadcast fan-out. For exhaustive API detail see
:class:`tractor.Context`, :class:`tractor.MsgStream` and
:exc:`tractor.ContextCancelled`.
.. _trio: https://github.com/python-trio/trio
.. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency
.. _blog post: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/
.. _nursery: https://trio.readthedocs.io/en/latest/reference-core.html#nurseries-and-spawning
.. _causality: https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#c-c-c-c-causality-breaker
.. _cheap or nasty: https://zguide.zeromq.org/docs/chapter7/#The-Cheap-or-Nasty-Pattern
.. _(un)protocol: https://zguide.zeromq.org/docs/chapter7/#Unprotocols
.. _msgspec: https://jcristharif.com/msgspec/
.. _#53: https://github.com/goodboy/tractor/issues/53
.. _#223: https://github.com/goodboy/tractor/issues/223

View File

@ -0,0 +1,545 @@
.. _debugging:
================================
"Native" multi-process debugging
================================
``tractor`` ships the thing every ``multiprocessing`` user has
wished for and quietly assumed was impossible: a multi-process
debugger that *just works*.
Drop ``await tractor.pause()`` — or, with `greenback`_ installed,
a plain builtin ``breakpoint()`` — anywhere in any actor: the
root, a child, a grandchild, a sync helper function, even an
``asyncio`` task inside an "infected" actor. A full-featured
`pdbp`_ REPL opens *in that process*, with syntax-highlighted
source listings, tab completion and sticky mode, attached to your
one terminal.
Under the hood every REPL entry acquires a tree-global tty mutex
via an IPC request to the root actor, so prompts from concurrent
pauses and crashes never interleave. ``ctrl-c`` is shielded while
any REPL is live, so a stray ``SIGINT`` can't vaporize the tree
out from under you. And in debug mode any uncaught error drops
you into a crash REPL *first in the failing child*, then again at
each parent as the boxed :class:`~tractor.RemoteActorError` climbs
the supervision tree.
No remote-pdb sockets, no ``set_trace()`` port juggling, no
``ptrace`` attach dance: the debugger semantics you already know,
transparently extended across an entire process tree. Because
``tractor`` is a `structured concurrency`_ (SC) runtime, the
debugger composes with supervision instead of fighting it — quit
a REPL and errors keep propagating exactly like `trio`_ taught
you, ending in clean, zombie-free teardown.
We're pretty sure it's the (first ever?) "native" debugging UX
for multi-process Python B)
Enabling debug mode
-------------------
Pass ``debug_mode=True`` to your runtime entrypoint, either
:func:`tractor.open_nursery` (which forwards it to the implicitly
opened root actor) or :func:`tractor.open_root_actor` directly:
.. code:: python
async with tractor.open_nursery(
debug_mode=True, # arm the whole actor tree
) as an:
...
This arms the debug machinery *tree-wide*:
- crash handling is enabled in every actor: uncaught errors enter
a REPL before they propagate,
- the internal tty-lock module is auto-exposed over RPC to every
subactor (this is what makes the one-terminal handoff work),
- console logging is bumped to include ``PDB``-level status msgs
so you can see REPL acquire/release events as they happen.
You can instead flip it on for just one child, letting its
siblings crash-and-burn the normal way:
.. code:: python
portal = await an.start_actor(
'sketchy_worker',
debug_mode=True, # OR-ed with the tree-wide flag
)
See ``examples/debugging/per_actor_debug.py`` for a runnable
proof of the selective style.
.. note::
Debug mode requires the child-side runtime to be
``trio``-native so that the tty-lock IPC dialog works; it's
currently supported on the ``'trio'`` (default) and
``'main_thread_forkserver'`` spawn backends and raises
``RuntimeError`` for any other ``start_method``.
Your first pause point
----------------------
:func:`tractor.pause` is the SC-aware, multi-process spelling of
the stdlib's ``breakpoint()``. In the root actor it looks almost
boring:
.. literalinclude:: ../../examples/debugging/root_actor_breakpoint.py
:caption: examples/debugging/root_actor_breakpoint.py
:language: python
Run it and you get a ``(Pdb+)`` prompt parked on the ``pause()``
line; type ``c`` (continue) and the program finishes normally.
The exact same call works from *any* subactor, no matter how deep
in the tree:
.. literalinclude:: ../../examples/debugging/subactor_breakpoint.py
:caption: examples/debugging/subactor_breakpoint.py
:language: python
Each loop iteration the child actor requests the terminal from
the root over IPC, REPLs you, then releases it on ``c``. Pause
points are re-entrant-safe: repeat calls from the same task are
no-op'd and other local tasks queue politely for the REPL.
When you get bored, type ``q`` (quit): the resulting
``bdb.BdbQuit`` is boxed and shipped to the parent like any other
remote error XD — causality is preserved even for your debugging
mistakes.
Crash REPLs: errors climb the tree
----------------------------------
Pause points are only half the story. With debug mode armed, any
*uncaught* error anywhere in the tree triggers what we call crash
handling mode:
.. literalinclude:: ../../examples/debugging/subactor_error.py
:caption: examples/debugging/subactor_error.py
:language: python
What happens when the child hits that (very intentional)
``NameError``:
1. a REPL opens **in the crashed child first** — you inspect the
raising frame, its locals, the works, right inside the failed
process,
2. when you quit, the error is boxed into a
:class:`~tractor.RemoteActorError` and relayed to the parent,
3. the parent (here the root) gets *its own* crash REPL with the
rendered remote traceback,
4. quit again and the nursery tears the tree down — errors keep
propagating per SC rules, no zombies left behind.
You debug the failure at every hop of the supervision tree, which
for multi-hop trees means you can chase an error from the leaf
that raised it all the way up to the root that supervises it.
Need to skip REPL entry for certain exceptions? Pass a predicate
via ``open_root_actor(debug_filter=...)``; by default
cancellation-only exception (groups) don't engage the REPL.
One terminal, many actors
-------------------------
So how do N processes share one tty without garbling it? The root
actor owns stdio for the whole tree and guards it with a FIFO
mutex; every subactor REPL entry is an IPC lock request to the
root. Exactly one actor-task in the entire tree can own the
terminal at a time, so prompts never interleave — ever.
.. d2:: diagrams/debug_lock.d2
:caption: Every REPL entry serializes through the root actor's
tty lock; ``continue``-ing one REPL hands the terminal to
the next waiter, FIFO style.
:alt: sequence diagram of two subactors serializing pdb REPL
access through the root actor's tty lock
The runtime's teardown paths cooperate too: a cancelling parent
always waits for any live REPL to release before reaping
children, so the debugger never gets yanked out from under you
mid-keystroke.
.. margin:: Watch the tree live
Run any of these examples with a process-tree watcher in a
second terminal and watch actors come and go::
watch -n 0.1 "pstree -a $$"
Here's the showpiece: one daemon child re-entering
``tractor.pause()`` forever inside a stream, while its sibling
repeatedly raises a ``NameError``:
.. literalinclude:: ../../examples/debugging/multi_daemon_subactors.py
:caption: examples/debugging/multi_daemon_subactors.py
:language: python
What you'll actually see
************************
Running it looks *roughly* like this (uids, tracebacks and source
listings elided; REPL order can vary with who wins the lock
race)::
$ python examples/debugging/multi_daemon_subactors.py
Opening a pdb REPL in paused actor: ('bp_forever', '<uuid>')
<highlighted source around the `await tractor.pause()` line>
(Pdb+) c
Opening a pdb REPL in crashed actor: ('name_error', '<uuid>')
<live traceback: NameError: name 'doggypants' is not defined>
(Pdb+) q
Opening a pdb REPL in crashed actor: ('root', '<uuid>')
<boxed RemoteActorError traceback relayed from 'name_error'>
(Pdb+) q
Two (then three) processes, one terminal, zero confusion:
``c``-ing out of the paused daemon's REPL releases the tty lock,
which immediately hands the prompt to the crashed sibling; quit
that and the error propagates as a fully-rendered
:class:`~tractor.RemoteActorError` to the parent where one final
crash REPL catches it before clean, zombie-free teardown.
For maximum drama run
``multi_nested_subactors_error_up_through_nurseries.py`` (under
``examples/debugging/``) which pulls the same trick across a
*three-deep* process tree — the tty lock keeps every prompt
orderly the whole way up.
Post-mortem, on demand
----------------------
Crash handling is automatic, but you can also enter a REPL on
a live exception *manually* with :func:`tractor.post_mortem`
the actor-aware equivalent of ``pdb.post_mortem()`` — from inside
any ``except`` block in any actor (kwargs: ``tb=`` for an
explicit traceback, plus ``shield=`` and ``hide_tb=``):
.. literalinclude:: ../../examples/debugging/pm_in_subactor.py
:caption: examples/debugging/pm_in_subactor.py
:language: python
This example demos three REPL entries from one error:
- the child's manual ``post_mortem()`` inside its ``except``,
- the runtime's automatic crash handler in the same child once
the error re-raises out of the RPC task,
- a manual ``post_mortem()`` in the parent on the received
:class:`~tractor.RemoteActorError`, whose ``.boxed_type``
faithfully reports the original ``NameError``.
Pausing from sync code
----------------------
No ``await``? No problem. :func:`tractor.pause_from_sync` brings
the same tree-aware REPL to plain synchronous functions — handy
when the suspect code is three helpers deep and decidedly not
async.
It's powered by `greenback`_, which is optional, so you need to:
1. install it (it ships in ``tractor``'s ``sync_pause``
dependency group),
2. enable it at runtime entry:
.. code:: python
async with tractor.open_nursery(
debug_mode=True,
maybe_enable_greenback=True,
) as an:
...
With that armed, sync code can pause from three different caller
environments: the main ``trio`` thread, ``trio.to_thread`` bg
threads, and (see the next section) ``asyncio`` tasks in infected
actors. The greenback "portal" hops back into the ``trio`` loop
to do the lock/REPL dance on your behalf:
.. literalinclude:: ../../examples/debugging/sync_bp.py
:caption: examples/debugging/sync_bp.py (the sync fn, excerpt)
:language: python
:pyobject: sync_pause
.. literalinclude:: ../../examples/debugging/sync_bp.py
:caption: examples/debugging/sync_bp.py (called in a subactor,
excerpt)
:language: python
:pyobject: start_n_sync_pause
The full script also exercises the hairier root-actor bg-thread
cases (and documents their remaining sharp edges) if you want the
deep lore.
The builtin ``breakpoint()`` override
*************************************
When debug mode boots with greenback available, ``tractor`` wires
Python's `PEP 553`_ hook so the *builtin* ``breakpoint()`` becomes
the actor-aware sync pause, by exporting::
PYTHONBREAKPOINT=tractor.devx.debug._sync_pause_from_builtin
That means third-party and legacy code containing bare
``breakpoint()`` calls debugs correctly inside your actor tree
with zero edits (the override even forwards kwargs like
``hide_tb`` to the underlying pause machinery, as shown in the
excerpt above).
.. warning::
Without greenback (or with ``maybe_enable_greenback=False``,
the default), ``debug_mode=True`` instead *blocks* the builtin
``breakpoint()``: ``sys.breakpointhook`` is swapped for a
raiser and ``PYTHONBREAKPOINT=0`` is set. A naive
``breakpoint()`` from some random process would clobber the
shared tty, so we'd rather hand you a loud ``RuntimeError``
with install instructions.
Both the hook and the env var are restored to their prior values
on runtime exit — see
``examples/debugging/restore_builtin_breakpoint.py`` for the
proof.
Breakpoints inside ``asyncio`` tasks
------------------------------------
Yes, even "infected ``asyncio``" actors get the goods. Spawn a
child with ``infect_asyncio=True`` (``trio`` runs as a guest on
the ``asyncio`` loop inside it) and, with debug mode + greenback
armed, every ``asyncio`` task started via ``tractor.to_asyncio``
is automatically granted a greenback portal — so a plain builtin
``breakpoint()`` (or ``tractor.pause_from_sync()``) inside an
``asyncio.Task`` joins the same single-terminal, tree-locked REPL
flow:
.. literalinclude:: ../../examples/debugging/asyncio_bp.py
:caption: examples/debugging/asyncio_bp.py
:language: python
Note the interleave: a ``breakpoint()`` on the ``asyncio`` side,
``tractor.pause()`` on the ``trio`` side of the same actor, and
another pause up in the root — all serialized through the one tty
lock with no cross-actor (or cross-event-loop!) clobbering.
One catch: ``asyncio`` tasks spawned *out-of-band* — i.e. not via
``tractor.to_asyncio``, typically by some third-party aio lib —
have no portal bestowed, so a sync pause from one raises a loud
``RuntimeError`` telling you to ``greenback.ensure_portal()``
first. See :ref:`the caveats <debugging-caveats>` below.
Teardown debugging: the shielded pause
--------------------------------------
`Cancellation`_ is ``trio``'s bread and butter, which raises an
awkward question: how do you REPL inside an *already-cancelled*
scope, say while debugging some teardown sequence? A bare
``pause()`` would itself be cancelled at its next checkpoint.
The answer is ``await tractor.pause(shield=True)``, which wraps
the lock acquisition and REPL session in a shielded cancel scope
(``post_mortem(shield=True)`` works the same way):
.. literalinclude:: ../../examples/debugging/shielded_pause.py
:caption: examples/debugging/shielded_pause.py
:language: python
If you forget, ``tractor`` has your back: an unshielded
``pause()`` from a cancelled scope fails fast with a hint
suggesting ``await tractor.pause(shield=True)`` instead of
silently never REPL-ing.
Go ahead, mash ctrl-c
---------------------
While any REPL is live the runtime installs a custom ``SIGINT``
handler tree-wide so that a reflexive ``ctrl-c`` (or five) can't
nuke your debug session:
- the actor that owns the REPL ignores the interrupt and simply
re-flushes the prompt — keep mashing, it's fine,
- the root actor ignores ``SIGINT`` while a still-IPC-connected
child holds the tty lock, so the supervisor won't tear down the
tree out from under the debugger,
- if the lock state has gone *stale* — the locking child died or
its IPC channel dropped — the root cancels the stale lock scope
and restores ``trio``'s default handler, so ``ctrl-c`` works
again exactly when it should.
The handler is uninstalled and ``trio``'s own ``SIGINT``
semantics restored every time a REPL releases (on ``continue`` /
``quit``).
Live task-tree dumps
--------------------
Sometimes there's no error to catch — the tree is just *hung* and
you want to know where. For that ``tractor`` integrates
`stackscope`_: send a signal, get a full ``trio`` task-tree dump
from every actor in the tree.
Enable it any of three ways:
- ``open_root_actor(enable_stack_on_sig=True)`` (or via
``open_nursery()`` which forwards it),
- set ``TRACTOR_ENABLE_STACKSCOPE=1`` in the env — it's inherited
through the process tree so every (sub)actor arms the handler
at boot,
- call ``tractor.devx.enable_stack_on_sig()`` directly.
It's intentionally *not* gated on ``debug_mode`` so you can leave
it armed in plain runs. Then, when the hang strikes, signal the
tree with ``SIGUSR1``.
.. tip::
No need to hunt down pids — pattern-match the original cmdline
with ``pkill``::
$ pkill --signal SIGUSR1 -f "python example_script.py"
Each actor dumps its entire ``trio`` task tree (full nursery
recursion via ``stackscope.extract()``) to its tty *and* tees it
to ``/tmp/tractor-stackscope-<pid>.log`` — so the trace survives
even under captured-stdio harnesses — then relays the signal on
to its children, parent-before-child, until the whole tree has
reported in.
Try it yourself with the demo script, which deliberately hangs a
subactor in a shielded sleep:
.. literalinclude:: ../../examples/debugging/shield_hang_in_sub.py
:caption: examples/debugging/shield_hang_in_sub.py
:language: python
(That ``trio.CancelScope(shield=True)`` hang also shows off the
zombie reaper: ``ctrl-c`` the root and the un-cancellable child
still gets hard-reaped — if you can create a zombie it **is a
bug**.)
Crash handling for sync and CLI code
------------------------------------
All of the above rides on the actor runtime, but crashes don't
politely wait for ``trio.run()``. For plain sync code — think
``typer``/``click`` CLI endpoints, config parsing, anything
pre-runtime — there's a sync context manager that wraps the same
``pdbp`` post-mortem UX:
.. code:: python
from tractor.devx import open_crash_handler
def main(): # any sync code, no runtime required
with open_crash_handler() as boxed:
run_my_cli_thing()
By default any ``BaseException`` (minus an ``ignore`` set
defaulting to ``KeyboardInterrupt`` and ``trio.Cancelled``)
enters the REPL then re-raises on exit; pass
``raise_on_exit=False`` to suppress instead and introspect the
``boxed.value`` afterward. The ``catch``/``ignore`` sets and a
``repl_fixture`` are all tweakable.
For the classic ``--pdb`` CLI-flag pattern use the conditional
variant:
.. code:: python
from tractor.devx import maybe_open_crash_handler
@app.command() # a `typer` (or `click`) endpoint
def cmd(pdb: bool = False):
with maybe_open_crash_handler(pdb=pdb):
...
REPL niceties and hooks
-----------------------
Every REPL in this guide is a `pdbp`_ instance (the maintained
fork-and-fix of `pdb++`_) pre-configured by ``tractor``:
- pygments syntax highlighting in listings and tracebacks,
- tab completion — including an automatic fixup for
libedit-compiled CPythons (e.g. ``uv``-distributed pythons),
- sticky mode available via the ``sticky`` command (off by
default),
- no long-line truncation (terminal resizes behave),
- the ``(Pdb+)`` prompt, ``ll``, hidden-frames support and the
rest of the ``pdb++`` goodies you may already know.
Internal runtime frames are traceback-hidden so the REPL lands
exactly on *your* ``pause()``-call or crash frame, never on
``tractor`` guts.
Finally, if your app owns the terminal (TUIs, fullscreen
dashboards) pass ``repl_fixture=<your ctx mngr>`` to ``pause()``,
``post_mortem()`` or ``open_crash_handler()``: it's entered just
before the REPL engages (return ``False`` to skip entry entirely)
and exited on release — perfect for suspending and restoring your
screen around a debug session.
.. _debugging-caveats:
Caveats and platform notes
--------------------------
An honest list of the current rough edges:
- **Windows**: the debugger has no CI coverage on windows at all
(the entire test module is skipped there); manual testing has
shown it *can* work, but you're in uncharted territory —
reports welcome!
- **macOS**: supported but with rough edges: special-cased prompt
re-flushing for ``bash``-on-darwin, a few tooling tests skipped
on CI, and the AF_UNIX ~104-char socket-path limit forces some
examples (like the stackscope demo above) to fall back from
``'uds'`` to ``'tcp'`` transport. Wonder if all of it'll work
on OS X? So do we.
- **CPython 3.14**: ``greenback`` (via ``greenlet``) doesn't
support 3.14 yet, so ``pause_from_sync()`` and the builtin
``breakpoint()`` override are effectively 3.13-only for now.
The async APIs — ``pause()`` and ``post_mortem()`` — need no
greenback and work everywhere.
- **out-of-band** ``asyncio`` **tasks**: sync pauses from aio
tasks *not* spawned via ``tractor.to_asyncio`` raise a
``RuntimeError`` (no greenback portal was bestowed); run
``await greenback.ensure_portal()`` inside such a task first.
- **nested-tree ctrl-c edges**: ``SIGINT`` relay through
intermediary parents that aren't themselves in debug mode still
has known rough edges — see `#320`_.
- **captured stdio**: ``pytest``-style output capture can hang a
``pause()``; use a real terminal (or a pty à la ``pexpect``,
which is how ``tractor``'s own suite drives every one of these
examples).
Where to next?
--------------
.. seealso::
- :doc:`/guide/context` — the SC-linked cross-actor task API
that all the crash-propagation semantics above ride on.
- :func:`tractor.pause`, :func:`tractor.post_mortem` and
:func:`tractor.pause_from_sync` in the API reference.
- ``examples/debugging/`` — 20-odd runnable scripts, nearly
every one exercised by the test suite through a real pty.
.. _structured concurrency:
https://en.wikipedia.org/wiki/Structured_concurrency
.. _trio: https://github.com/python-trio/trio
.. _cancellation: https://trio.readthedocs.io/en/latest/
reference-core.html#cancellation-and-timeouts
.. _pdbp: https://github.com/mdmintz/pdbp
.. _pdb++: https://github.com/pdbpp/pdbpp
.. _greenback: https://github.com/oremanj/greenback
.. _stackscope: https://github.com/oremanj/stackscope
.. _PEP 553: https://peps.python.org/pep-0553/
.. _#320: https://github.com/goodboy/tractor/issues/320

Some files were not shown because too many files have changed in this diff Show More