Compare commits
55 Commits
a834487838
...
b1f6ade840
| Author | SHA1 | Date |
|---|---|---|
|
|
b1f6ade840 | |
|
|
2fb6143604 | |
|
|
6d5b145787 | |
|
|
503a3ed766 | |
|
|
d0261787fd | |
|
|
d1ffb436d0 | |
|
|
244e080efe | |
|
|
6dd149df3d | |
|
|
f7bf068d09 | |
|
|
1fd857ce21 | |
|
|
1b0b085667 | |
|
|
790a52ca7a | |
|
|
7abbb2b5f0 | |
|
|
40e96b2991 | |
|
|
b9dca728d9 | |
|
|
be3e78197d | |
|
|
ece3827bdf | |
|
|
35e496c166 | |
|
|
0a0aa64305 | |
|
|
51f4e31ce0 | |
|
|
0a741282af | |
|
|
419ce6a631 | |
|
|
176e0c3e61 | |
|
|
6b8e9ad298 | |
|
|
bcf8f87ea5 | |
|
|
c7451bb2cb | |
|
|
fa8067f79f | |
|
|
768b531662 | |
|
|
4c8188652b | |
|
|
8e39fd8b07 | |
|
|
13cdc43f01 | |
|
|
ba6b181cf5 | |
|
|
cd1e8160fe | |
|
|
3c6bd928b7 | |
|
|
0457e8f3b1 | |
|
|
86417fa19c | |
|
|
a03b52619a | |
|
|
9cdbe36bca | |
|
|
325369af22 | |
|
|
07ba2be07d | |
|
|
f81dd82b89 | |
|
|
e25e018370 | |
|
|
f220b4641d | |
|
|
7796480062 | |
|
|
d799bd6f68 | |
|
|
a95ca7577f | |
|
|
a40fb2ebde | |
|
|
38883dca03 | |
|
|
97aff52050 | |
|
|
96a1838210 | |
|
|
0be872ff97 | |
|
|
319868e92d | |
|
|
a6c1158853 | |
|
|
4c507ea13c | |
|
|
a279cb39a9 |
|
|
@ -0,0 +1,136 @@
|
|||
---
|
||||
model: claude-opus-5
|
||||
service: claude
|
||||
session: 7b9c97c4-fff7-4ac4-97fb-35720453308e
|
||||
timestamp: 2026-08-13T00:11:02Z
|
||||
git_ref: 27c34aeb
|
||||
scope: docs+code
|
||||
substantive: true
|
||||
raw_file: 20260813T001102Z_27c34aeb_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
> draft hyper detailed implementation plans for [three]
|
||||
> prospective new transport (tpt) backends for tractor's `.ipc`
|
||||
> layer, from four GitHub issues: TIPC (gh #378) using built-in
|
||||
> linux socket API w/ `trio` interfacing, leveraging TIPC's
|
||||
> built-in discovery machinery; QUIC (gh #353) using the `iroh`
|
||||
> lib, ideally with the py asyncio support (via ffi) rewritten
|
||||
> for trio; wg (gh #482 and/or #443) with other shuttle-able
|
||||
> tpts, using `pyroute2`, as much trio wrapping as possible
|
||||
> where any other async support can be replaced.
|
||||
|
||||
With constraints: "be only slightly speculative"; realistic given
|
||||
the linked info; the plans must be workable "by another
|
||||
model/provider without much discrepancy on the design or lib
|
||||
selections needed"; do it all in a new `/open-wkt` off main; and
|
||||
be "as modular and modern-async-python-as-nearly-functional-style
|
||||
as possible."
|
||||
|
||||
Key follow-up prompts that changed the design mid-session:
|
||||
|
||||
> wg requires binding a socket on both ends of the tunnel, the
|
||||
> new support we added to `py-multiaddr` in their #108 enables
|
||||
> declaring this in the maddr? […] so the
|
||||
> `/ip4/10.0.0.1/udp/666/ip4/192.168.0.13/tcp/80` bullet example
|
||||
> from #443 isn't correct for a wg tunnel maddr and would
|
||||
> actually end up looking like,
|
||||
> `/ip4/10.0.0.1/udp/666/wg/<pub-key>/ip4/192.168.0.13/tcp/80` no?
|
||||
|
||||
> right, we're likely going to need less `tuple`-formed
|
||||
> `UnwrappedAddress` usage to make them all work and/or to extend
|
||||
> the unwrapped form with a proto-str, likely the key from
|
||||
> `multiaddrs` no?
|
||||
|
||||
> better encapsulating `UnwrappedAddress` for use externally is
|
||||
> really way overdue; we should likely be encouraging (if not
|
||||
> preventing unwrapped) wrapped addrs always much like other
|
||||
> (std) libs, `ipaddress` comes to mind.
|
||||
|
||||
> so iirc netns mgmt becomes a lower-level runtime config-api
|
||||
> rather then an actor-app-code API which can be dynamically
|
||||
> changed once the actor is up ya?
|
||||
|
||||
## Response summary
|
||||
|
||||
A research-and-planning branch, `ng_tpts_planning` — **no runtime
|
||||
code touched**. Five plan docs under `ai/tpt-backends/` plus a
|
||||
corrected `wg` example set under `examples/multihost/wg_lan/`,
|
||||
across 9 commits.
|
||||
|
||||
Three design conclusions came out of the planning that outlive
|
||||
the branch:
|
||||
|
||||
1. **`wg` is a bindspace, not a `MsgTransport`.** The `/wg/`
|
||||
maddr segment is *infix*: bearer (underlay ip+udp
|
||||
`ListenPort`, bound by the kernel) → `/wg/u<key>` (an
|
||||
identity, bound by nothing) → overlay (the only part tractor
|
||||
binds). Verified empirically by installing
|
||||
`baudco/py-multiaddr@wg_support` (py-multiaddr#108) in a
|
||||
throwaway venv and round-tripping each candidate form; gh
|
||||
#482's original suffix form parses but is semantically
|
||||
inverted.
|
||||
2. **Proto-key the `UnwrappedAddress` form**, spelled with the
|
||||
`multiaddr` protocol names, dispatched via
|
||||
`_address_types[addr[0]]`. Kills a whole collision class
|
||||
(TIPC's `(str, int)` ≡ `TCPAddress`; iroh's `(str, str)`
|
||||
swallowed by the UDS case) and is the recommended migration
|
||||
*before* any new backend lands.
|
||||
3. **netns is a runtime/boot-time config API, not an app-code
|
||||
one** — `setns(2)` is per-thread and won't move
|
||||
already-created sockets, so there is deliberately no
|
||||
`await actor.enter_netns(...)`.
|
||||
|
||||
Also verified that `trio.SocketStream`/`SocketListener` are
|
||||
address-family agnostic (no `AF_*` check anywhere), which is what
|
||||
makes TIPC the cheapest of the three backends to add.
|
||||
|
||||
Four related issues were annotated with the results (#378, #353,
|
||||
#482, #443); #443's body was rewritten to reflect the corrected
|
||||
grammar, with no existing checkbox state changed.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `ai/tpt-backends/00_shared_backend_contract.md` — normative
|
||||
backend duck-type contract, registration checklist, §1.1
|
||||
proto-key conclusion
|
||||
- `ai/tpt-backends/01_tipc_backend.md` — TIPC plan; service
|
||||
addressing, `TIPC_TOP_SRV` push registry, instance-collision
|
||||
hazard, step-0 probe
|
||||
- `ai/tpt-backends/02_quic_iroh_backend.md` — `iroh` plan;
|
||||
`uniffi`→`trio` bridge, listener/stream adapters, API-truth
|
||||
table
|
||||
- `ai/tpt-backends/03_wg_tunnel_bindspace.md` — `wg`-as-bindspace
|
||||
plan; verified maddr grammar, 3-owner split, netns reality
|
||||
- `ai/tpt-backends/README.md` — index
|
||||
- `examples/multihost/wg_lan/wg_maddr.py` — frozen `msgspec`
|
||||
tunnelled addr + pure parse/render helpers; impure
|
||||
`verify_wg_peer()` kept separate
|
||||
- `examples/multihost/wg_lan/host_a_srv.py` — host-A actor tree
|
||||
- `examples/multihost/wg_lan/host_b_client.py` — host-B dialer
|
||||
- `examples/multihost/wg_lan/README.md` — grammar, owner table,
|
||||
setup, "what changed vs #482"
|
||||
|
||||
## Human edits
|
||||
|
||||
Substantial human steering rather than post-hoc editing; the
|
||||
corrections were applied by the model in-session after being
|
||||
challenged:
|
||||
|
||||
- rejected an initial claim that `wg` has "nothing to bind at the
|
||||
tunnel layer" and supplied the correct composed maddr form,
|
||||
which forced a rewrite of plan 03 §3.2 and a retraction in the
|
||||
already-posted #443 comment
|
||||
- rejected a supporting claim that `/ip4/../udp/443/quic-v1` was
|
||||
"also composed"
|
||||
- directed the proto-key/`ipaddress`-discipline conclusion and
|
||||
the netns-as-runtime-config framing, both of which were then
|
||||
folded back into the docs
|
||||
- chose the commit boundaries and authored all commits; ran every
|
||||
`git` mutation (commit, rebase, push) themselves
|
||||
|
||||
One model-initiated correction pre-publication: a self-review
|
||||
downgraded two overconfident claims (the `uniffi`/asyncio thesis
|
||||
and TIPC duplicate-binder behaviour) to explicitly-flagged
|
||||
assumptions before the #353/#378 comments were posted.
|
||||
|
|
@ -0,0 +1,165 @@
|
|||
---
|
||||
model: claude-opus-5
|
||||
service: claude
|
||||
timestamp: 2026-08-13T00:11:02Z
|
||||
git_ref: 27c34aeb
|
||||
diff_cmd: git diff main..ng_tpts_planning
|
||||
---
|
||||
|
||||
# Raw output — next-gen tpt-backend implementation plans
|
||||
|
||||
## Generated planning docs
|
||||
|
||||
> `git diff main..ng_tpts_planning -- ai/tpt-backends/`
|
||||
|
||||
Five markdown docs. `00_shared_backend_contract.md` is normative
|
||||
and the other three are written against it so they can be worked
|
||||
independently:
|
||||
|
||||
- **`00_shared_backend_contract.md`** — the backend duck-type
|
||||
(`<Proto>Address(msgspec.Struct, frozen=True)` + module-level
|
||||
`start_listener()`/`close_listener()` + a
|
||||
`Msgpack<Proto>Stream(MsgpackTransport)`), the
|
||||
`inspect.getmodule(self.addr)` reflection in
|
||||
`Endpoint.start_listener()` that forces the Address class and
|
||||
its listener fns to share a module, a 10-item registration
|
||||
checklist, the dep policy, the test-harness shape, and §1.1's
|
||||
proto-key conclusion (below).
|
||||
- **`01_tipc_backend.md`** — service addressing via
|
||||
`TIPC_ADDR_NAMESEQ` (bind/publish) and `TIPC_ADDR_NAME`
|
||||
(connect/lookup), `TIPC_TOP_SRV` topology subscriptions as a
|
||||
push-based registry, the `get_random()` instance-collision
|
||||
hazard, and a step-0 capability-probe spike.
|
||||
- **`02_quic_iroh_backend.md`** — `iroh` over
|
||||
`aioquic`/`quiche`/`trio-asyncio`, a `_uniffi_trio.py` bridge
|
||||
built on `TrioToken.run_sync_soon()`, `trio.abc.Listener`/
|
||||
`HalfCloseableStream` adapters, and an API-truth table to fill
|
||||
in during step 0.
|
||||
- **`03_wg_tunnel_bindspace.md`** — `wg` as a *bindspace* rather
|
||||
than a `MsgTransport`, a `TunnelledAddress` wrapper delegating
|
||||
`.proto_key`/`.unwrap()` to `.inner`, `pyroute2` for layer B,
|
||||
and `@acm`-managed netns/iface for layer C.
|
||||
- **`README.md`** — index.
|
||||
|
||||
## Generated example code
|
||||
|
||||
> `git diff main..ng_tpts_planning -- examples/multihost/wg_lan/`
|
||||
|
||||
- `wg_maddr.py` — `WGTunnelledAddr(msgspec.Struct, frozen=True)`
|
||||
carrying `bearer: tuple[str, int]`, `peer_pubkey: str`,
|
||||
`inner: tuple[str, int]`, `inner_proto: Literal['tcp']`, plus a
|
||||
`.maddr` property that re-renders the canonical form. Pure
|
||||
helpers `mb_pubkey()`, `wg8_pubkey()`, `parse_wg_maddr()`, and
|
||||
`_segments()` (with a marked stopgap for when the `wg` codec
|
||||
isn't installed). `verify_wg_peer()` is impure **by design** and
|
||||
kept out of the parse path.
|
||||
- `host_a_srv.py` / `host_b_client.py` — the two-host runs; both
|
||||
pass only `addr.inner` to `open_nursery()`/`open_root_actor()`.
|
||||
- `README.md` — grammar, owner table, `#108`-branch install line,
|
||||
tunnel setup, "what changed vs #482".
|
||||
|
||||
## Verified findings (non-code, verbatim)
|
||||
|
||||
### `trio` is address-family agnostic
|
||||
|
||||
Read against the installed `trio`. `SocketStream`/`SocketListener`
|
||||
ctor checks are only "is a trio sock object" + `type ==
|
||||
SOCK_STREAM`, plus an `OSError`-**suppressed** `SO_ACCEPTCONN`
|
||||
probe. No `AF_*` check anywhere; `TCP_NODELAY`/`TCP_NOTSENT_LOWAT`
|
||||
are set under `suppress(OSError)`. A TIPC `SOCK_STREAM` sock should
|
||||
therefore drop straight into `trio.serve_listeners()` with the
|
||||
existing `MsgpackTransport` framing, making TIPC mostly
|
||||
table-registration boilerplate w/ zero new deps.
|
||||
|
||||
### the `wg` maddr grammar — `/wg/` is infix, not suffix
|
||||
|
||||
Installed `baudco/py-multiaddr@wg_support` (PR
|
||||
multiformats/py-multiaddr#108) into a throwaway venv and
|
||||
round-tripped every candidate form:
|
||||
|
||||
| maddr | `[p.name for p in m.protocols()]` |
|
||||
| --- | --- |
|
||||
| `/ip4/1.2.3.4/udp/51820/wg/u<k>` | `['ip4','udp','wg']` |
|
||||
| `/ip4/../udp/../wg/u<k>/ip4/../tcp/..` | `['ip4','udp','wg','ip4','tcp']` |
|
||||
| `/ip4/10.0.11.1/tcp/1616/wg/u<k>` | `['ip4','tcp','wg']` |
|
||||
|
||||
```
|
||||
/ip4/192.168.1.50/udp/51820/wg/u<A_pub>/ip4/10.0.11.1/tcp/1616
|
||||
\_______ bearer __________/\__ key __/\______ overlay ______/
|
||||
```
|
||||
|
||||
Segments *before* `/wg/` are the bearer — the underlay
|
||||
`(ip, udp-port)` that `wg(8)` itself listens on (`ListenPort`).
|
||||
Segments *after* are the overlay endpoint, the only part tractor
|
||||
binds. The third row above is #482's original suffix form: it
|
||||
parses, but is semantically inverted.
|
||||
|
||||
Three parts, three owners — and only one is an `Endpoint`:
|
||||
|
||||
| part | bound by | in the runtime? |
|
||||
| --- | --- | --- |
|
||||
| bearer | kernel, via `wg-quick`/`pyroute2` | no |
|
||||
| `/wg/u<key>` | nothing — an identity | no, verified out-of-band |
|
||||
| overlay | `tractor`'s `IPCServer` | yes, as `.inner` |
|
||||
|
||||
### proto-key-tagged `UnwrappedAddress`
|
||||
|
||||
Shape-matching in `wrap_address()` does not survive four backends.
|
||||
TIPC's natural unwrapped form is a `(str, int)`, indistinguishable
|
||||
from `TCPAddress`; iroh's is a `(str, str)`, already swallowed by
|
||||
the existing UDS case (`case (_, filename) if type(filename) is
|
||||
str`). Ordering hacks and prefix-tagging only paper over it.
|
||||
|
||||
Recommended prerequisite for all three backends: carry an explicit
|
||||
proto-key spelled with the `multiaddr` protocol name —
|
||||
`('tcp', host, port)`, `('unix', path)`,
|
||||
`('tipc', stype, inst, scope)` — so `wrap_address()` collapses to
|
||||
`_address_types[addr[0]]` and the collision class stops existing.
|
||||
This also makes the on-wire form agree with
|
||||
`mk_maddr()`/`parse_maddr()` instead of being an independent
|
||||
invention. It is a wire-format change (`SpawnSpec`,
|
||||
`_root_mailbox`, `_registry_addrs`) plus every fixture and
|
||||
downstream config, so it wants its own migration commit landed
|
||||
before any new backend — and it is the moment to stop handing raw
|
||||
tuples to users at all, making `Address` the public currency and
|
||||
`UnwrappedAddress` an internal serialization detail (the
|
||||
discipline `ipaddress` uses).
|
||||
|
||||
### netns is a runtime-level config API
|
||||
|
||||
`setns(2)` affects the calling thread only and does not move
|
||||
already-created sockets. So a netns is a spawn/boot-time input
|
||||
alongside `enable_transports`/`tpt_bind_addrs`, and there is
|
||||
deliberately no `await actor.enter_netns(...)` — a mid-life API
|
||||
would silently leave the IPC server bound in the old namespace.
|
||||
Corollary for layer B: pass `netns=` down to `pyroute2` rather
|
||||
than assuming a `trio.to_thread` worker inherits it.
|
||||
|
||||
### `examples/` collection would have failed CI
|
||||
|
||||
`tests/test_docs_examples.py` walks `examples/` recursively and
|
||||
subproc-runs every collected file asserting `rc == 0`. Its filter
|
||||
never checks the extension, so all four `wg_lan` files were
|
||||
collected — including `README.md`, which would have been run as
|
||||
`python README.md`. `'multihost' not in p[0]` was already in the
|
||||
exclusion list with no directory using it. Moving the set under
|
||||
`examples/multihost/wg_lan/` drops collection 24 → 20 with zero
|
||||
test changes; confirmed via `pytest --collect-only`.
|
||||
|
||||
## Corrections applied during the session
|
||||
|
||||
The human corrected two claims that had been asserted without
|
||||
verification, both since retracted in-place in the docs and in the
|
||||
posted issue comments:
|
||||
|
||||
1. that `wg` has "nothing to bind at the tunnel layer, exactly one
|
||||
bind" — wrong; a wg stack is genuinely composed, and the real
|
||||
axis is *who owns* each layer's endpoint.
|
||||
2. that `/ip4/../udp/443/quic-v1` was "also composed" — wrong;
|
||||
that is one endpoint with a protocol qualifier, not a tunnel.
|
||||
|
||||
A self-review before publication also downgraded two
|
||||
overconfident claims to explicitly-flagged assumptions: the
|
||||
`uniffi`-uses-asyncio-only-as-executor thesis (contradicted that
|
||||
plan's own "do not guess from memory" step 0) and TIPC's
|
||||
duplicate-binder round-robin behaviour (unverified).
|
||||
|
|
@ -0,0 +1,69 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
session: intercepted-claude-7b9c97c4-fff7-4ac4-97fb-35720453308e
|
||||
timestamp: 2026-08-18T02:17:29Z
|
||||
git_ref: d9a6e2e9
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260818T021729Z_d9a6e2e9_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
> Intercept Claude session
|
||||
> `7b9c97c4-fff7-4ac4-97fb-35720453308e`, pick up where it
|
||||
> stopped in its open worktree, finish the tunnelled-address change
|
||||
> and commit plan, and prepare any outstanding context for another
|
||||
> provider.
|
||||
|
||||
The recovered final prompt specifically called out the `uds` versus
|
||||
`unix` boundary decision, absent concrete `Address.namespace`
|
||||
implementations, the corrected namespace test, and the passing
|
||||
focused and wider suites.
|
||||
|
||||
During review, the human further required that tunnelled-maddr work
|
||||
delegate to `multiaddr`'s encapsulation APIs, challenged the premature
|
||||
transport-shaped listener hooks and directed their removal, corrected
|
||||
the long-term bearer provisioning model, and confirmed the intended
|
||||
split between discovery metadata and bindspace lifecycle code.
|
||||
|
||||
## Response summary
|
||||
|
||||
Recovered the transcript and matched it to `wkts/addr_unpacking`,
|
||||
audited the staged implementation, and completed the interrupted
|
||||
verification and commit-plan work. The audit removed premature
|
||||
transport-shaped listener hooks, widened the namespace identifier
|
||||
type, updated stale import documentation, and removed an
|
||||
invalid-escape warning from the maddr diagram. It also preserved the
|
||||
layer-C design where tractor provisions the kernel-owned tunnel
|
||||
bearer without treating it as a message transport.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `tractor/discovery/_tunnel.py` - tunnel specs, address wrapper, and
|
||||
peeling helpers.
|
||||
- `tractor/discovery/_addr.py` - wrapped-address recognition and
|
||||
namespace typing.
|
||||
- `tractor/discovery/__init__.py` - public tunnel API exports.
|
||||
- `tests/discovery/test_tunnelled_addr.py` - delegation and boundary
|
||||
regression coverage.
|
||||
- `ai/tpt-backends/03_wg_tunnel_bindspace.md` - distinguish
|
||||
tractor-owned bindspace provisioning from kernel socket ownership.
|
||||
|
||||
## Human edits
|
||||
|
||||
Substantial human-directed editing occurred over several review turns:
|
||||
|
||||
- required use of `multiaddr`'s `.encapsulate()`/`.decapsulate()`
|
||||
family rather than a hand-rolled tunnel peeler
|
||||
- rejected the premature `start_listener()`/`close_listener()` hooks
|
||||
and directed their removal from this foundational change
|
||||
- corrected the documentation so tractor retains ownership of future
|
||||
bindspace provisioning while the kernel owns the bearer socket
|
||||
- reviewed and accepted the placement of declarative tunnel metadata
|
||||
under `tractor.discovery`, with lifecycle code kept separate
|
||||
|
||||
The final source lines were applied through the coding agents, but
|
||||
these design corrections and deletion decisions came from the human
|
||||
review and materially shaped the patch.
|
||||
|
|
@ -0,0 +1,66 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-18T02:17:29Z
|
||||
git_ref: d9a6e2e9
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
# Raw output - tunnelled-address handoff completion
|
||||
|
||||
Recovered Claude Code session
|
||||
`7b9c97c4-fff7-4ac4-97fb-35720453308e` and continued its
|
||||
interrupted `wkts/addr_unpacking` changes.
|
||||
|
||||
## Generated code
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tractor/discovery/_tunnel.py`
|
||||
|
||||
Added frozen `WGTunnelSpec` and `TunnelledAddress` structs. The
|
||||
wrapper delegates transport identity, validity, bindspace, and wire
|
||||
serialization to its overlay while retaining tunnel metadata locally.
|
||||
Added pure helpers to peel nested wrappers and enumerate their tunnel
|
||||
specs. The module documents why wrappers must be peeled before
|
||||
`Endpoint` selects the overlay transport backend.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tractor/discovery/_addr.py`
|
||||
|
||||
Extended `is_wrapped_addr()` to recognize `TunnelledAddress` without
|
||||
registering tunnels as message transports, and widened the namespace
|
||||
identifier type to cover named network namespaces.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tractor/discovery/__init__.py`
|
||||
|
||||
Exported the tunnel address API from `tractor.discovery` and updated
|
||||
the eager-import documentation.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/discovery/test_tunnelled_addr.py`
|
||||
|
||||
Added focused coverage for delegation, serialization, rewrapping,
|
||||
namespace fallback, nested peeling order, and frozen structs.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- ai/tpt-backends/03_wg_tunnel_bindspace.md`
|
||||
|
||||
Clarified that tractor owns the eventual bindspace lifecycle and may
|
||||
provision the WireGuard iface, routes, and kernel UDP listener through
|
||||
netlink/`pyroute2`. Kernel socket ownership does not make the bearer an
|
||||
application `MsgTransport` endpoint.
|
||||
|
||||
## Verification
|
||||
|
||||
Focused tests:
|
||||
|
||||
```text
|
||||
9 passed in 0.03s
|
||||
```
|
||||
|
||||
Discovery and IPC suites:
|
||||
|
||||
```text
|
||||
67 passed, 2 xpassed
|
||||
```
|
||||
|
||||
The audit removed premature module-level listener hooks. Runtime
|
||||
integration must peel the wrapper explicitly at bind and dial
|
||||
boundaries rather than make `._tunnel` impersonate a transport
|
||||
backend.
|
||||
|
|
@ -0,0 +1,50 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
session: tractor-addr-unpacking-followup
|
||||
timestamp: 2026-08-18T07:50:31Z
|
||||
git_ref: dd02c7c0
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260818T075031Z_dd02c7c0_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The human requested the complete tunnelled-maddr parsing/composition
|
||||
slice as an unattended batch, with every numbered requirement finished
|
||||
and atomic commit plans prepared at the end. Existing human decisions
|
||||
required native `multiaddr` encapsulation APIs, no hand-written peeler,
|
||||
and preservation of tractor's future bindspace lifecycle ownership.
|
||||
|
||||
## Response summary
|
||||
|
||||
Implemented strict WG key codecs and native single/nested tunnel maddr
|
||||
parsing and composition, integrated them into discovery APIs, migrated
|
||||
the multihost example off its duplicate parser, corrected package
|
||||
dependency metadata, and added focused and end-to-end parser
|
||||
regressions. Verified the complete tractor suite and built both package
|
||||
artifacts.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `tractor/discovery/` - WG codecs, parser/composer, wrapper typing,
|
||||
public exports, and discovery dispatch.
|
||||
- `tests/discovery/` - key, grammar, nesting, round-trip, and public
|
||||
boundary regressions.
|
||||
- `examples/multihost/wg_lan/` - production parser migration and
|
||||
updated usage documentation.
|
||||
- `pyproject.toml`, `uv.lock` - reproducible WG codec and multibase
|
||||
dependencies for checkout and package installs.
|
||||
- `ai/tpt-backends/03_wg_tunnel_bindspace.md` - current layer-A state
|
||||
and future bindspace ownership.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human selected the five-step scope and batch execution model,
|
||||
required delegation to `multiaddr`'s encapsulation APIs, rejected
|
||||
transport-shaped listener placeholders in the prerequisite commit, and
|
||||
clarified that tractor will eventually provision the kernel-owned
|
||||
bearer through its bindspace layer. The agent implemented and tested
|
||||
those decisions; no direct manual source edits were observed during
|
||||
this batch.
|
||||
|
|
@ -0,0 +1,61 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-18T07:50:31Z
|
||||
git_ref: dd02c7c0
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
# Raw output - native WireGuard maddr integration
|
||||
|
||||
The human requested completion of the five-step tunnelled-maddr slice:
|
||||
port the proven WireGuard parser, delegate to `py-multiaddr`'s native
|
||||
tunnel APIs, integrate public parse and composition entry points, add
|
||||
regressions, and return atomic commit plans after completing the batch.
|
||||
|
||||
## Generated code
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tractor/discovery/_tunnel.py tractor/discovery/_multiaddr.py tractor/discovery/_addr.py tractor/discovery/__init__.py`
|
||||
|
||||
Added strict WireGuard standard-base64/multibase key codecs and native
|
||||
WG maddr parsing/composition. Nested stacks peel the last `/wg/`
|
||||
repeatedly with `.decapsulate_code()`, isolate segments through
|
||||
`.split()`/`.join()`, and compose recursively with `.encapsulate()`.
|
||||
Public discovery parsing, wrapping, endpoint-table parsing, and maddr
|
||||
composition now preserve `TunnelledAddress` metadata locally.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- pyproject.toml uv.lock`
|
||||
|
||||
Declared `py-multibase` directly and pinned the unreleased
|
||||
py-multiaddr#108 WG codec revision in distribution metadata. Enabled
|
||||
Hatch direct-reference metadata so editable, wheel, and sdist builds
|
||||
use the same dependency contract.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/discovery/test_multiaddr.py tests/discovery/test_tunnelled_addr.py`
|
||||
|
||||
Added key-codec, single/nested WG round-trip, malformed key, invalid
|
||||
bearer/overlay, missing codec, missing bearer, `wrap_address()`, and
|
||||
`parse_endpoints()` coverage while retaining plain TCP/UDS regressions.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- examples/multihost/wg_lan ai/tpt-backends/03_wg_tunnel_bindspace.md`
|
||||
|
||||
Removed the duplicate example parser, migrated callers to production
|
||||
`TunnelledAddress`, retained only explicit `wg(8)` verification, and
|
||||
updated dependency and future bindspace ownership documentation.
|
||||
|
||||
## Verification
|
||||
|
||||
- focused discovery files: `45 passed`
|
||||
- discovery and IPC suites: `81 passed, 2 xpassed`
|
||||
- full suite: `443 passed, 9 skipped, 7 xfailed, 4 xpassed`
|
||||
- collection: `462 tests`
|
||||
- Ruff: clean
|
||||
- sdist and wheel builds: successful
|
||||
|
||||
## Review corrections
|
||||
|
||||
A review caught that `[tool.uv.sources]` alone would not constrain
|
||||
built-package consumers to a WG-capable py-multiaddr revision. The
|
||||
dependency was moved into PEP 621 metadata and the parser now converts
|
||||
missing-codec failures into an actionable tractor error without
|
||||
misclassifying Unix paths containing a `wg` directory.
|
||||
|
|
@ -0,0 +1,37 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
session: tractor-addr-unpacking-followup
|
||||
timestamp: 2026-08-19T21:31:44Z
|
||||
git_ref: f81fc5e5
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260819T213144Z_f81fc5e5_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The human requested runtime boundary integration with the established
|
||||
per-step implementation and commit-plan workflow.
|
||||
|
||||
## Response summary
|
||||
|
||||
Kept `TunnelledAddress` available to callers while peeling it at the
|
||||
last outbound boundary before transport lookup and dialing. Added a
|
||||
regression which captures both transport arguments and confirms plain
|
||||
TCP behavior is unchanged.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `tractor/ipc/_chan.py` - peel tunnel annotations before outbound
|
||||
transport dispatch and connection.
|
||||
- `tests/ipc/test_channel_tunnel_boundary.py` - verify plain and
|
||||
tunnelled channel inputs deliver only TCP overlays.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human chose the runtime-boundary slice, required the existing
|
||||
per-step commit-plan flow, and previously established that wrappers
|
||||
must retain bindspace metadata without impersonating transports. The
|
||||
agent implemented those constraints; no direct manual source edits were
|
||||
observed during this step.
|
||||
|
|
@ -0,0 +1,25 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-19T21:31:44Z
|
||||
git_ref: f81fc5e5
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
# Raw output - outbound tunnel boundary
|
||||
|
||||
The human requested the next tunnelled-address slice using the same
|
||||
per-step commit-plan flow. Existing design decisions require retaining
|
||||
tunnel metadata until the narrow IPC transport boundary and never
|
||||
teaching exact-type transport tables about tunnel wrappers.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tractor/ipc/_chan.py tests/ipc/test_channel_tunnel_boundary.py`
|
||||
|
||||
Extended channel address inputs to accept tunnel declarations, then
|
||||
called `strip_tunnels()` immediately before exact-type transport lookup
|
||||
and `connect_to()`. Added plain/tunnel parameterized coverage proving
|
||||
both operations receive the identical TCP overlay while the original
|
||||
wrapper retains its tunnel spec.
|
||||
|
||||
Verification included focused IPC tests, Ruff, discovery/IPC suites,
|
||||
and the full tractor suite.
|
||||
|
|
@ -0,0 +1,40 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
session: tractor-addr-unpacking-followup
|
||||
timestamp: 2026-08-19T21:31:45Z
|
||||
git_ref: f81fc5e5
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260819T213145Z_f81fc5e5_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The human requested completion of inbound runtime peeling using the
|
||||
same per-step implementation and commit-plan workflow.
|
||||
|
||||
## Response summary
|
||||
|
||||
Preserved tunnel declarations through listener configuration, peeled
|
||||
them immediately before `Endpoint` construction, and used the overlay
|
||||
for backend-specific random listener allocation after registry
|
||||
discovery. Added a real listener regression for the reflection and
|
||||
exact-type boundary.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `tractor/ipc/_server.py` - accept wrapper declarations and peel at
|
||||
`Endpoint` construction.
|
||||
- `tractor/_root.py` - allocate random transport addresses from the
|
||||
contacted registry's overlay.
|
||||
- `tests/ipc/test_server_tunnel_boundary.py` - verify a real listener
|
||||
stores only TCP while preserving the source annotation.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human selected the runtime-boundary work and previously corrected
|
||||
the architecture so tractor retains future bindspace provisioning
|
||||
ownership while `Endpoint` sees only application transports. The agent
|
||||
implemented and tested that direction; no direct manual source edits
|
||||
were observed during this step.
|
||||
|
|
@ -0,0 +1,25 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-19T21:31:45Z
|
||||
git_ref: f81fc5e5
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
# Raw output - inbound tunnel boundary
|
||||
|
||||
The human requested runtime boundary integration while preserving the
|
||||
future tractor-owned bindspace lifecycle.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tractor/ipc/_server.py tractor/_root.py tests/ipc/test_server_tunnel_boundary.py`
|
||||
|
||||
Broadened listener declarations to carry tunnel wrappers until
|
||||
`_serve_ipc_eps()` and peeled immediately before `Endpoint`
|
||||
construction. Also peeled a contacted tunnelled registry before
|
||||
backend-specific random listener allocation. Added a real TCP listener
|
||||
regression proving `Endpoint` stores only the resolved overlay while
|
||||
the original declaration retains bindspace metadata.
|
||||
|
||||
Verification included `465` collected tests, `84` passing
|
||||
discovery/IPC tests with two xpasses, Ruff, and the full suite with
|
||||
`447` passes.
|
||||
|
|
@ -0,0 +1,41 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
session: tractor-addr-unpacking-followup
|
||||
timestamp: 2026-08-20T02:15:16Z
|
||||
git_ref: dfad66a0
|
||||
scope: docs
|
||||
substantive: true
|
||||
raw_file: 20260820T021516Z_dfad66a0_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The human requested that the bindspace plan preserve the agreed
|
||||
capability, spawn-bootstrap, endpoint-role, namespace augmentation,
|
||||
random-address, and teardown semantics, using `github/ns_aware` as
|
||||
prototype input.
|
||||
|
||||
## Response summary
|
||||
|
||||
Updated plan-03 and the shared backend contract to separate serializable
|
||||
bindspace declarations from scoped live capabilities, make namespace
|
||||
entry a pre-runtime spawn operation, keep maddr paths role-neutral, and
|
||||
define listen/dial provisioning plus ownership-sensitive teardown.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `ai/tpt-backends/03_wg_tunnel_bindspace.md` - layer-C capability,
|
||||
bootstrap, role, teardown, test, and risk model.
|
||||
- `ai/tpt-backends/00_shared_backend_contract.md` - distinguish
|
||||
transport bind selectors from process namespace lifecycle.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human supplied the core architecture: structured scoped
|
||||
capabilities, spawn-time namespace entry, orthogonal namespace
|
||||
augmentation, source/destination-dependent provisioning, and
|
||||
role-dependent teardown. They also rejected premature assumptions about
|
||||
`open_bindspace()` returning an address and requested grounding in the
|
||||
existing namespace prototype. The agent translated those decisions into
|
||||
the plan text; no direct manual source edits were observed.
|
||||
|
|
@ -0,0 +1,34 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-20T02:15:16Z
|
||||
git_ref: dfad66a0
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
# Raw output - bindspace capability design
|
||||
|
||||
The human corrected the layer-C design around local network-stack
|
||||
realization. They established that bindspace state should be both
|
||||
structured and a scoped capability; namespace entry belongs in
|
||||
subactor bootstrap; maddrs can describe source or destination network
|
||||
paths while namespace selection augments them orthogonally; random
|
||||
address and teardown behavior depend on operation role and ownership.
|
||||
|
||||
They directed comparison with the prototype on `github/ns_aware` and
|
||||
requested these decisions be preserved in the plan.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- ai/tpt-backends/03_wg_tunnel_bindspace.md ai/tpt-backends/00_shared_backend_contract.md`
|
||||
|
||||
Reworked layer C around serializable `BindspaceSpec`, stable
|
||||
`BindspaceIdentity`, and scoped non-serializable `BindspaceHandle`
|
||||
concepts. Namespace FDs pin identity and lifetime; parent/supervisor
|
||||
provisioning transfers entry capability through spawn; the child enters
|
||||
before runtime, channels, listeners, sockets, or worker threads and then
|
||||
drops authority. Listen/dial roles and owned/borrowed teardown are
|
||||
explicit, while maddrs remain role-neutral network-path declarations.
|
||||
|
||||
The shared backend contract now separates transport-level `.bindspace`
|
||||
selectors from process namespace lifecycle. Added tests/risks for FD
|
||||
identity, bootstrap ordering, privilege drop, role ownership, and
|
||||
shared-resource teardown.
|
||||
|
|
@ -0,0 +1,41 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
session: tractor-addr-unpacking
|
||||
timestamp: 2026-08-20T03:31:07Z
|
||||
git_ref: ba07e09d
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260820T033107Z_ba07e09d_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The human requested canonical tagged transport addresses with a
|
||||
reader-first migration. TCP should decode `('tcp', host, port)`, Unix
|
||||
should decode `('unix', path)`, `uds` should remain an accepted input
|
||||
alias and internal transport key, and legacy tuple/list inputs must keep
|
||||
working before writers switch formats.
|
||||
|
||||
## Response summary
|
||||
|
||||
Introduced canonical and compatibility address aliases, explicit tagged
|
||||
dispatch, transport-specific tagged readers, and focused serialization
|
||||
tests. Kept legacy pair inputs and native IPv6 socket values readable so
|
||||
this boundary can ship before tagged emission.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `tractor/discovery/_addr.py` - address aliases and tagged dispatch.
|
||||
- `tractor/ipc/_tcp.py` - tagged, legacy, and IPv6 TCP decoding.
|
||||
- `tractor/ipc/_uds.py` - canonical Unix and UDS-alias decoding.
|
||||
- `tests/discovery/test_address_serialization.py` - reader compatibility
|
||||
coverage.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human supplied the canonical `tcp` and `unix` forms, chose `uds` as
|
||||
an input-only serialization alias while preserving it as the runtime
|
||||
transport key, and required a reader-first commit boundary. The agent
|
||||
implemented those decisions; no direct manual source edits were
|
||||
observed.
|
||||
|
|
@ -0,0 +1,25 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-20T03:31:07Z
|
||||
git_ref: ba07e09d
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
# Raw output - tagged address readers
|
||||
|
||||
The human requested a migration away from ambiguous untagged transport
|
||||
tuples. They established `('tcp', host, port)` and `('unix', path)` as
|
||||
canonical forms, retained `('uds', path)` as an input alias, and required
|
||||
a reader-first compatibility boundary before changing emitted values.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tractor/discovery/_addr.py tractor/ipc/_tcp.py tractor/ipc/_uds.py tests/discovery/test_address_serialization.py`
|
||||
|
||||
Added explicit tagged address aliases and dispatch, taught TCP and UDS
|
||||
readers to decode tagged tuple/list payloads, preserved legacy pair input,
|
||||
and retained native IPv6 socket-address decoding. Added focused tests for
|
||||
canonical tags, the UDS alias, msgpack-style lists, legacy pairs, and IPv6
|
||||
socket values.
|
||||
|
||||
Focused reader tests and Ruff checks passed before the writer migration
|
||||
was applied.
|
||||
|
|
@ -0,0 +1,45 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
session: tractor-addr-unpacking
|
||||
timestamp: 2026-08-20T03:31:08Z
|
||||
git_ref: ba07e09d
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260820T033108Z_ba07e09d_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The human asked the agent to continue after adding tagged readers,
|
||||
complete canonical address emission without dropping legacy input
|
||||
compatibility, verify the migration, and prepare a complete multi-commit
|
||||
package when the turn was done.
|
||||
|
||||
## Response summary
|
||||
|
||||
Changed `.unwrap()` to emit tagged TCP and Unix addresses, updated direct
|
||||
tuple consumers and spawn payload declarations, and aligned multiaddr,
|
||||
runtime, IPC, and discovery tests with canonical serialized equality.
|
||||
Kept untagged tuples and the `uds` spelling readable at input boundaries.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `tractor/discovery/_addr.py` - canonical output alias.
|
||||
- `tractor/discovery/_multiaddr.py` - tagged address composition.
|
||||
- `tractor/ipc/_tcp.py` - tagged emission and direct socket dialing.
|
||||
- `tractor/ipc/_uds.py` - tagged full-path emission.
|
||||
- `tractor/msg/types.py` - protocol-neutral spawn tuple containers.
|
||||
- `tests/discovery/test_address_serialization.py` - writer assertions.
|
||||
- `tests/discovery/test_multiaddr.py` - canonical round-trip assertions.
|
||||
- `tests/discovery/test_tpt_bind_addrs.py` - tagged bind assertions.
|
||||
- `tests/ipc/test_each_tpt.py` - canonical runtime address assertions.
|
||||
- `tests/ipc/test_server_tunnel_boundary.py` - tagged TCP destructuring.
|
||||
- `tests/test_local.py` - canonical registry comparison.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human established the reader-before-writer sequencing, canonical tag
|
||||
spellings, retained compatibility expectations, and requested final
|
||||
multi-commit packaging. The agent implemented and tested those choices;
|
||||
no direct manual source edits were observed.
|
||||
|
|
@ -0,0 +1,26 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-20T03:31:08Z
|
||||
git_ref: ba07e09d
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
# Raw output - canonical tagged address writers
|
||||
|
||||
After the reader compatibility boundary, the human asked the agent to
|
||||
continue the migration and package the completed work as dependency-
|
||||
ordered commits.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tractor/discovery/_addr.py tractor/discovery/_multiaddr.py tractor/ipc/_tcp.py tractor/ipc/_uds.py tractor/msg/types.py tests/discovery/test_address_serialization.py tests/discovery/test_multiaddr.py tests/discovery/test_tpt_bind_addrs.py tests/ipc/test_each_tpt.py tests/ipc/test_server_tunnel_boundary.py tests/test_local.py`
|
||||
|
||||
Switched TCP and Unix `.unwrap()` output to canonical tagged tuples,
|
||||
updated direct transport and multiaddr consumers, widened spawn message
|
||||
tuple containers for protocol-specific shapes, and migrated runtime and
|
||||
test comparisons to serialized address equality. Legacy inputs remain
|
||||
accepted at `wrap_address()` and backend reader boundaries.
|
||||
|
||||
Ruff and focused tests passed. The complete non-debugger TCP suite passed
|
||||
with 412 tests; the UDS suite reached 80% without failure before the
|
||||
harness timeout, then all 97 remaining tests passed on resume. Debugger
|
||||
PTY coverage was excluded after an unrelated timeout.
|
||||
|
|
@ -0,0 +1,49 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
session: tractor-addr-unpacking
|
||||
timestamp: 2026-08-21T23:32:04Z
|
||||
git_ref: 5d92595f
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260821T233204Z_5d92595f_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The human asked to proceed with WireGuard Layer B from PR #505's
|
||||
head, using the established one-change-per-turn workflow and
|
||||
finishing with a commit plan. This turn was limited to read-only
|
||||
pyroute2 inspection; peer verification and bindspace provisioning
|
||||
remain later changes.
|
||||
|
||||
During review, the human required the generated Python to follow the
|
||||
deployed `/py-codestyle` rules and asked for the medium-term plan to
|
||||
remove pyroute2's asyncio I/O runtime from the Trio read path.
|
||||
|
||||
## Response summary
|
||||
|
||||
Added Linux-only read helpers for WireGuard device and peer public
|
||||
keys. Pyroute2's synchronous API is fully contained in a Trio worker
|
||||
thread, supports named netns reads without creation side effects,
|
||||
validates decoded keys and always closes the netlink client.
|
||||
Follow-up edits added the required local annotations, boolean layout,
|
||||
helper docstrings and 69-column source formatting.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `pyproject.toml` - Linux-only `wg` optional dependency.
|
||||
- `uv.lock` - resolved pyroute2 0.9.6 metadata.
|
||||
- `tractor/discovery/_tunnel.py` - read-only WireGuard helpers.
|
||||
- `tractor/discovery/__init__.py` - public helper exports.
|
||||
- `tests/discovery/test_wg_inspection.py` - fake-backed netlink and
|
||||
worker-thread regressions.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human chose the exact #505 head as a stacked base, required
|
||||
incremental changes ending in commit plans, and limited this turn to
|
||||
read-only pyroute2 inspection. The human then identified that Ruff
|
||||
success had not established `/py-codestyle` compliance and directed
|
||||
the agent to correct the Python-specific annotation, documentation
|
||||
and layout rules. No direct manual source edits were observed.
|
||||
|
|
@ -0,0 +1,31 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-21T23:32:04Z
|
||||
git_ref: 5d92595f
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
# Raw output - read WireGuard state through pyroute2
|
||||
|
||||
The human authorized the first incremental WireGuard Layer B change
|
||||
as a stacked branch from tractor PR #505, with one atomic change and
|
||||
a commit plan at the end of the turn.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- pyproject.toml uv.lock tractor/discovery/__init__.py tractor/discovery/_tunnel.py tests/discovery/test_wg_inspection.py`
|
||||
|
||||
Added a Linux-only `wg` extra using pyroute2 0.9.6, plus
|
||||
asynchronous public helpers for reading one interface's public key
|
||||
and peer keys. The complete synchronous open/read/parse/close
|
||||
sequence runs in a Trio worker thread because pyroute2 owns a private
|
||||
asyncio loop internally.
|
||||
|
||||
Named namespace reads pass `flags=0` to override pyroute2's `O_CREAT`
|
||||
default, ensuring inspection cannot create a missing namespace. Fake
|
||||
netlink messages cover multipart dumps, key validation, stable peer
|
||||
deduplication, worker-thread execution, netns selection and cleanup
|
||||
on success/error.
|
||||
|
||||
Ruff and lock checks passed. Focused tunnel/multiaddr coverage passed
|
||||
47 tests; the complete discovery suite passed 88 tests with 2
|
||||
xpasses.
|
||||
|
|
@ -0,0 +1,46 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
session: tractor-addr-unpacking
|
||||
timestamp: 2026-08-22T02:32:26Z
|
||||
git_ref: 59a8ecfd
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260822T023226Z_59a8ecfd_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
After committing the read-only pyroute2 helpers and `wgman` design
|
||||
update, the human authorized the next isolated Layer B change:
|
||||
explicit `verify_wg_peer()` composition over WireGuard inspection.
|
||||
|
||||
## Response summary
|
||||
|
||||
Added and exported async `verify_wg_peer()` using one validated
|
||||
WireGuard key snapshot. It recognizes local-interface and configured
|
||||
peer identities without coupling kernel inspection to address
|
||||
parsing. Updated the multihost examples to use the production helper
|
||||
and removed their subprocess-based probe.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `tractor/discovery/_tunnel.py` - shared async snapshot reader and
|
||||
explicit verification helper.
|
||||
- `tractor/discovery/__init__.py` - public verification export.
|
||||
- `tests/discovery/test_wg_inspection.py` - local, peer, absent and
|
||||
malformed-key verification coverage.
|
||||
- `examples/multihost/wg_lan/host_a_srv.py` - async local-key check.
|
||||
- `examples/multihost/wg_lan/host_b_client.py` - async peer-key check.
|
||||
- `examples/multihost/wg_lan/wg_maddr.py` - removed obsolete
|
||||
subprocess probe.
|
||||
- `examples/multihost/wg_lan/README.md` - pyroute2 requirements and
|
||||
verification workflow.
|
||||
- `ai/tpt-backends/03_wg_tunnel_bindspace.md` - async API contract.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human selected this pre-agreed verification layer as the next
|
||||
atomic change after reviewing and committing the preceding read and
|
||||
architecture changes. The agent implemented the source changes; no
|
||||
direct manual edits or follow-up corrections were observed.
|
||||
|
|
@ -0,0 +1,31 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-22T02:32:26Z
|
||||
git_ref: 59a8ecfd
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
# Raw output - verify declared WireGuard identities
|
||||
|
||||
The human authorized the next incremental Layer B change after
|
||||
committing the read-only pyroute2 helpers and first-child `wgman`
|
||||
design update.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tractor/discovery/_tunnel.py tractor/discovery/__init__.py tests/discovery/test_wg_inspection.py examples/multihost/wg_lan ai/tpt-backends/03_wg_tunnel_bindspace.md`
|
||||
|
||||
Added async `verify_wg_peer()` over one WireGuard key snapshot. It
|
||||
validates the declared `WGTunnelSpec.peer_pubkey` before I/O, forwards
|
||||
the spec's iface/netns, and accepts either the local interface key for
|
||||
a source/listen declaration or a configured peer key for a
|
||||
destination/dial declaration.
|
||||
|
||||
Refactored worker offload behind one shared async reader so
|
||||
verification cannot compare two different netlink snapshots. Exported
|
||||
the helper, added local/peer/absent/malformed-key coverage, and moved
|
||||
the multihost examples from their local `wg show` subprocess probe to
|
||||
the production API.
|
||||
|
||||
Ruff and lock checks passed. Focused WireGuard/tunnel/multiaddr
|
||||
coverage passed 51 tests; the complete discovery suite passed 92
|
||||
tests with 2 xpasses.
|
||||
|
|
@ -0,0 +1,44 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
session: tractor-addr-unpacking
|
||||
timestamp: 2026-08-22T03:25:20Z
|
||||
git_ref: d35c802b
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260822T032520Z_d35c802b_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The human reported the explicit WireGuard verification commit done,
|
||||
asked the agent to check it, and authorized the next incremental
|
||||
Layer C change.
|
||||
|
||||
## Response summary
|
||||
|
||||
Verified commit `d35c802b`, made plain transport namespace behavior
|
||||
explicit, and retained each original listener declaration beside its
|
||||
peeled transport address. Endpoint and server diagnostics can now
|
||||
report a tunnel's namespace without violating the transport boundary.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `tractor/ipc/_tcp.py` - explicit plain-address namespace property.
|
||||
- `tractor/ipc/_uds.py` - explicit plain-address namespace property.
|
||||
- `tractor/discovery/_tunnel.py` - direct overlay namespace delegation.
|
||||
- `tractor/ipc/_server.py` - retained declaration and namespace
|
||||
diagnostics.
|
||||
- `tests/discovery/test_tunnelled_addr.py` - plain and tunnel namespace
|
||||
behavior.
|
||||
- `tests/ipc/test_server_tunnel_boundary.py` - declaration retention
|
||||
and diagnostic coverage.
|
||||
- `ai/tpt-backends/03_wg_tunnel_bindspace.md` - concrete endpoint
|
||||
boundary contract.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human selected continued incremental implementation after
|
||||
reviewing and committing the preceding verification layer. The agent
|
||||
implemented this dependency-ordered namespace slice; no direct manual
|
||||
edits or follow-up corrections were observed.
|
||||
|
|
@ -0,0 +1,29 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-22T03:25:20Z
|
||||
git_ref: d35c802b
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
# Raw output - retain endpoint namespace declarations
|
||||
|
||||
The human reported the explicit WireGuard verification commit done,
|
||||
asked for it to be checked, and authorized the next incremental
|
||||
change.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tractor/ipc/_tcp.py tractor/ipc/_uds.py tractor/discovery/_tunnel.py tractor/ipc/_server.py tests/discovery/test_tunnelled_addr.py tests/ipc/test_server_tunnel_boundary.py ai/tpt-backends/03_wg_tunnel_bindspace.md`
|
||||
|
||||
Confirmed commit `d35c802b` and a clean worktree, then implemented the
|
||||
smallest dependency-ordered Layer C slice. Plain TCP and UDS addresses
|
||||
now explicitly report no namespace, allowing `TunnelledAddress` to
|
||||
delegate without an attribute fallback.
|
||||
|
||||
Added required `Endpoint.declared_addr` metadata beside the peeled,
|
||||
resolved `Endpoint.addr`. Endpoint and server diagnostics expose the
|
||||
declaration's namespace without passing a tunnel wrapper into
|
||||
transport reflection. Updated the Layer C plan and tests for plain,
|
||||
tunnelled, endpoint and server namespace behavior.
|
||||
|
||||
Ruff passed. Focused namespace tests passed 13 tests; combined
|
||||
discovery and IPC coverage passed 101 tests with 2 xpasses.
|
||||
|
|
@ -0,0 +1,64 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
session: tractor-addr-unpacking
|
||||
timestamp: 2026-08-22T04:20:26Z
|
||||
git_ref: 29141f0b
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260822T042026Z_29141f0b_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
After committing endpoint namespace visibility, the human authorized
|
||||
continued Layer C implementation.
|
||||
|
||||
During staged review, the human requested Literal-derived validation,
|
||||
ownership documentation, stable-inode clarification, explicit
|
||||
non-serialization rationale and consolidated invalid-model tests.
|
||||
|
||||
## Response summary
|
||||
|
||||
Added the foundational bindspace model: serializable declarations and
|
||||
stable identities are separated from a process-local live capability.
|
||||
The handle validates names, ownership and FD identity. A global
|
||||
`ProcessLocal` sentinel blocks default encoding while retaining
|
||||
msgspec struct behavior.
|
||||
Review fixes require a positive inode for every realized netns,
|
||||
derive runtime choices from the Literal aliases and clarify that an FD
|
||||
integer is not transferable capability authority.
|
||||
The human then clarified that msgspec structs are useful generic
|
||||
storage independently of serialization policy, so the live handle now
|
||||
uses a struct while remaining process-local by contract.
|
||||
The human first selected an opaque FD wrapper, then recognized that
|
||||
future process-local handles need the same guard and directed a global
|
||||
marker under `tractor.msg` instead.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `tractor/discovery/_bindspace.py` - declaration, identity and live
|
||||
capability models.
|
||||
- `tractor/discovery/__init__.py` - public bindspace exports.
|
||||
- `tractor/msg/_local.py` - reusable process-local struct marker.
|
||||
- `tractor/msg/__init__.py` - public `ProcessLocal` export.
|
||||
- `tests/discovery/test_bindspace.py` - serialization and capability
|
||||
invariant coverage.
|
||||
- `tests/msg/test_process_local.py` - direct and nested wire rejection.
|
||||
- `ai/tpt-backends/03_wg_tunnel_bindspace.md` - concrete initial model
|
||||
contract.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human selected the foundational Layer C capability model, then
|
||||
reviewed the staged patch and directed all eight follow-up changes:
|
||||
remove distracting ownership variation, consolidate invalid-model
|
||||
tests, derive containment checks from Literal aliases, require a
|
||||
stable inode and explain why the live handle is not a msgspec struct.
|
||||
The human subsequently corrected that last design choice: struct use
|
||||
does not imply IPC authorization, so `BindspaceHandle` should retain
|
||||
msgspec's generic storage benefits. The agent applied those
|
||||
human-directed edits. The human first chose an opaque process-local FD
|
||||
wrapper, then refined that decision to a reusable global marker under
|
||||
`tractor.msg` for future local handles. No direct manual source edits
|
||||
were observed.
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-22T04:20:26Z
|
||||
git_ref: 29141f0b
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
# Raw output - model bindspace capabilities
|
||||
|
||||
The human authorized the next incremental Layer C change after
|
||||
committing endpoint namespace visibility.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tractor/discovery/_bindspace.py tractor/discovery/__init__.py tests/discovery/test_bindspace.py ai/tpt-backends/03_wg_tunnel_bindspace.md`
|
||||
|
||||
Added frozen, serializable `BindspaceSpec` and
|
||||
`BindspaceIdentity` models for the initial Linux netns kind. Added a
|
||||
plain slotted `BindspaceHandle` which carries ownership and an optional
|
||||
namespace FD without being serializable by default msgspec encoding.
|
||||
|
||||
The live handle validates declaration/identity kind and name, ownership
|
||||
values, non-negative FDs and FD/inode correspondence. Serializable
|
||||
models reject empty identities and kinds without implementations.
|
||||
|
||||
Added round-trip, non-serialization, identity-mismatch, empty-identity
|
||||
and unsupported-kind tests. Ruff passed; discovery plus
|
||||
message-serialization coverage passed 125 tests with 2 xpasses.
|
||||
|
|
@ -0,0 +1,48 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
session: tractor-addr-unpacking
|
||||
timestamp: 2026-08-25T04:55:57Z
|
||||
git_ref: fdccfd7e
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260825T045557Z_fdccfd7e_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
After committing the foundational bindspace capability model, the
|
||||
human authorized the next incremental Layer C change.
|
||||
|
||||
## Response summary
|
||||
|
||||
Added a borrow-only existing-netns context manager which pins stable
|
||||
identity through an open FD and always releases that local descriptor.
|
||||
The API does not create, enter or remove namespaces and rejects named
|
||||
path traversal.
|
||||
Staged review renamed the context to `attach_netns()`, introduced the
|
||||
readable `CURRENT_NETNS` alias and made key validation errors identify
|
||||
their exact owning field.
|
||||
Follow-up review made the empty-key error spell out that `None` is
|
||||
written as `CURRENT_NETNS` for `BindspaceSpec.key` declarations.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `tractor/discovery/_bindspace.py` - existing-netns lifecycle and key
|
||||
validation.
|
||||
- `tractor/discovery/__init__.py` - public lifecycle export.
|
||||
- `tests/discovery/test_bindspace.py` - current, named, missing and
|
||||
traversal coverage.
|
||||
- `ai/tpt-backends/03_wg_tunnel_bindspace.md` - borrow-only lifecycle
|
||||
contract.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human selected the previously deferred borrow-only netns lifecycle
|
||||
as the next incremental Layer C change. The agent implemented the
|
||||
source changes. During staged review, the human selected
|
||||
`attach_netns()` terminology, requested explicit
|
||||
`BindspaceSpec.key = CURRENT_NETNS` semantics and field-specific key
|
||||
validation. Follow-up review requested the validation error itself
|
||||
connect `None` to `CURRENT_NETNS`. The agent applied those
|
||||
human-directed edits; no direct manual source edits were observed.
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-25T04:55:57Z
|
||||
git_ref: fdccfd7e
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
# Raw output - borrow existing netns bindspaces
|
||||
|
||||
The human reported the bindspace capability model committed and
|
||||
authorized the next incremental Layer C change.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tractor/discovery/_bindspace.py tractor/discovery/__init__.py tests/discovery/test_bindspace.py ai/tpt-backends/03_wg_tunnel_bindspace.md`
|
||||
|
||||
Added async `open_existing_netns()` as a borrow-only context manager.
|
||||
It opens the current process netns or a named entry under the standard
|
||||
iproute2 run directory, derives stable identity from the opened FD,
|
||||
and yields a borrowed process-local `BindspaceHandle`.
|
||||
|
||||
The context uses `O_CLOEXEC`, never creates, enters or removes a
|
||||
namespace, and synchronously closes only its FD on exit. Netns keys
|
||||
reject paths to keep named lookup beneath the run directory.
|
||||
|
||||
Added current, named, missing and traversal tests. Ruff and lock checks
|
||||
passed; discovery plus message coverage passed 132 tests with 2
|
||||
xpasses.
|
||||
|
|
@ -0,0 +1,37 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
session: tractor-addr-unpacking
|
||||
timestamp: 2026-08-25T19:05:29Z
|
||||
git_ref: e1007547
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260825T190529Z_e1007547_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
After committing the attach-only network namespace lifecycle, the
|
||||
human authorized the next incremental Layer C change.
|
||||
|
||||
## Response summary
|
||||
|
||||
Added owned `open_netns()` creation and teardown through shielded
|
||||
pyroute2 worker calls. The context pins identity through
|
||||
`attach_netns()`, yields an owned handle, closes its FD before removal
|
||||
and cleans up despite caller cancellation.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `tractor/discovery/_bindspace.py` - owned netns lifecycle.
|
||||
- `tractor/discovery/__init__.py` - public lifecycle export.
|
||||
- `tests/discovery/test_bindspace.py` - ownership, ordering,
|
||||
cancellation and name requirements.
|
||||
- `ai/tpt-backends/03_wg_tunnel_bindspace.md` - owned lifecycle and
|
||||
spawn-boundary contract.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human selected owned network namespace creation as the next
|
||||
incremental Layer C change. The agent implemented the source changes;
|
||||
no direct manual edits or follow-up corrections were observed.
|
||||
|
|
@ -0,0 +1,26 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-25T19:05:29Z
|
||||
git_ref: e1007547
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
# Raw output - own created netns bindspaces
|
||||
|
||||
The human reported the attach-only netns lifecycle committed and
|
||||
authorized the next incremental Layer C change.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tractor/discovery/_bindspace.py tractor/discovery/__init__.py tests/discovery/test_bindspace.py ai/tpt-backends/03_wg_tunnel_bindspace.md`
|
||||
|
||||
Added async `open_netns()` as the owned counterpart to
|
||||
`attach_netns()`. It requires a named spec, creates through pyroute2 in
|
||||
a shielded worker call, attaches the resulting namespace FD and yields
|
||||
an owned process-local `BindspaceHandle`.
|
||||
|
||||
FD closure occurs before shielded namespace removal on normal,
|
||||
exceptional and cancelled exits. The context never calls `setns()`;
|
||||
namespace entry remains a spawn/bootstrap responsibility.
|
||||
|
||||
Privileged operations are faked in tests. Ruff and lock checks passed;
|
||||
discovery plus message coverage passed 137 tests with 2 xpasses.
|
||||
|
|
@ -0,0 +1,40 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
session: tractor-addr-unpacking
|
||||
timestamp: 2026-08-25T19:18:45Z
|
||||
git_ref: 5b2a064a
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260825T191845Z_5b2a064a_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The human pointed out that the agent had produced no changes after the
|
||||
previous commit and expected the next incremental implementation.
|
||||
|
||||
## Response summary
|
||||
|
||||
Corrected the sequencing mistake and added explicit serialized
|
||||
bindspace lifecycle policy plus generic `open_bindspace()` dispatch.
|
||||
Borrowed and owned netns contexts share a policy-neutral FD-pinning
|
||||
primitive while validating their public lifecycle before side effects.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `tractor/discovery/_bindspace.py` - lifecycle policy, ownership
|
||||
invariants, shared pinning and dispatch.
|
||||
- `tractor/discovery/__init__.py` - public policy and dispatcher exports.
|
||||
- `tests/discovery/test_bindspace.py` - lifecycle serialization,
|
||||
validation and both dispatcher branches.
|
||||
- `ai/tpt-backends/03_wg_tunnel_bindspace.md` - explicit lifecycle
|
||||
policy independent of transport role.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human identified that the agent had accidentally repeated a
|
||||
summary of already committed work instead of implementing the next
|
||||
slice. That correction directly caused this lifecycle/dispatcher
|
||||
change to be implemented. The agent wrote the source changes; no
|
||||
direct manual edits were observed.
|
||||
|
|
@ -0,0 +1,30 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-25T19:18:45Z
|
||||
git_ref: 5b2a064a
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
# Raw output - dispatch explicit bindspace lifecycle
|
||||
|
||||
The human noticed the agent had re-reported already committed work
|
||||
without changing the worktree. The agent confirmed the sequencing
|
||||
error and proceeded to the actual next Layer C slice.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tractor/discovery/_bindspace.py tractor/discovery/__init__.py tests/discovery/test_bindspace.py ai/tpt-backends/03_wg_tunnel_bindspace.md`
|
||||
|
||||
Added serialized `BindspaceSpec.lifecycle` policy with explicit
|
||||
`attach` and `open` choices. Added `open_bindspace()` dispatch without
|
||||
inferring resource ownership from listen/dial role.
|
||||
|
||||
Refactored FD attachment into policy-neutral `_pin_netns()` so
|
||||
borrowed and owned public lifecycles can share identity pinning while
|
||||
enforcing their own policy before side effects. Handle construction
|
||||
also verifies lifecycle and resulting ownership agree.
|
||||
|
||||
One composition error introduced during implementation was caught by
|
||||
the focused test: owned creation initially called the newly guarded
|
||||
public attach API. `_pin_netns()` fixed that layering error. Ruff and
|
||||
lock checks passed; discovery plus message coverage passed 138 tests
|
||||
with 2 xpasses.
|
||||
|
|
@ -0,0 +1,42 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
session: tractor-addr-unpacking
|
||||
timestamp: 2026-08-25T23:46:31Z
|
||||
git_ref: b973e78c
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260825T234631Z_b973e78c_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The human pointed out that the agent had again produced no worktree
|
||||
changes after the latest commit and expected actual next-layer work.
|
||||
|
||||
## Response summary
|
||||
|
||||
Corrected the sequencing mistake and added a process-local WireGuard
|
||||
interface configuration boundary. Secret/routing policy is separated
|
||||
from serializable maddr-derived tunnel declarations, redacted from
|
||||
repr and blocked from default wire encoding.
|
||||
Staged review clarified that address/network parser calls validate
|
||||
every declaration and do not select or consume a first address.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `tractor/discovery/_tunnel.py` - local WG config and validation.
|
||||
- `tractor/discovery/__init__.py` - public config export.
|
||||
- `tests/discovery/test_wg_config.py` - secrecy and validation coverage.
|
||||
- `ai/tpt-backends/03_wg_tunnel_bindspace.md` - secret/config boundary
|
||||
for future interface provisioning.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human identified the second sequencing failure where the agent
|
||||
summarized an already committed slice instead of implementing the next
|
||||
one. That correction directly caused this new WG configuration layer
|
||||
to be implemented. During staged review, the human requested explicit
|
||||
explanation of the address and allowed-route validation loops. The
|
||||
agent applied that human-directed clarification; no direct manual edits
|
||||
were observed.
|
||||
|
|
@ -0,0 +1,28 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-25T23:46:31Z
|
||||
git_ref: b973e78c
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
# Raw output - separate local WireGuard configuration
|
||||
|
||||
The human again noticed the agent had re-reported already committed
|
||||
work without changing the worktree. The agent confirmed the sequencing
|
||||
mistake and moved to the next unimplemented dependency.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tractor/discovery/_tunnel.py tractor/discovery/__init__.py tests/discovery/test_wg_config.py ai/tpt-backends/03_wg_tunnel_bindspace.md`
|
||||
|
||||
Added process-local `WGInterfaceConfig` for private/preshared keys,
|
||||
local interface CIDRs, peer allowed CIDRs, listen port and persistent
|
||||
keepalive. Secrets are redacted from repr and the global
|
||||
`ProcessLocal` marker blocks default wire encoding.
|
||||
|
||||
Removed the unused serialized `WGTunnelSpec.maybe_allowed_ips`
|
||||
placeholder so maddr-derived declarations retain only public identity,
|
||||
endpoint and interface selection. Added validation for keys, CIDRs,
|
||||
ports and keepalive before future kernel mutation.
|
||||
|
||||
Ruff and lock checks passed. Focused WG coverage passed 25 tests;
|
||||
discovery plus message coverage passed 145 tests with 2 xpasses.
|
||||
|
|
@ -0,0 +1,37 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
session: tractor-addr-unpacking
|
||||
timestamp: 2026-08-26T00:14:42Z
|
||||
git_ref: dcdf4d82
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260826T001442Z_dcdf4d82_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The human selected an explicit peer-list model rather than a
|
||||
dial-only single-peer shortcut before `open_wg_iface()`.
|
||||
|
||||
## Response summary
|
||||
|
||||
Added process-local per-peer configuration and refactored interface
|
||||
configuration to own a unique peer tuple. This supports multi-peer
|
||||
listeners and dial targets without overloading role-dependent tunnel
|
||||
maddr identity.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `tractor/discovery/_tunnel.py` - peer model and interface peer list.
|
||||
- `tractor/discovery/__init__.py` - public peer-config export.
|
||||
- `tests/discovery/test_wg_config.py` - peer secrecy and validation.
|
||||
- `ai/tpt-backends/03_wg_tunnel_bindspace.md` - explicit multi-peer
|
||||
provisioning contract.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human chose explicit peer lists over a dial-only implementation so
|
||||
listeners can represent multiple client public keys and routing
|
||||
policies. The agent implemented that human-selected design; no direct
|
||||
manual source edits were observed.
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-26T00:14:42Z
|
||||
git_ref: dcdf4d82
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
# Raw output - model explicit WireGuard peers
|
||||
|
||||
After committing local WG interface configuration, the human selected
|
||||
an explicit peer-list model before interface provisioning.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tractor/discovery/_tunnel.py tractor/discovery/__init__.py tests/discovery/test_wg_config.py ai/tpt-backends/03_wg_tunnel_bindspace.md`
|
||||
|
||||
Added process-local `WGPeerConfig` with public key, allowed CIDRs,
|
||||
optional endpoint, preshared key and keepalive. Refactored
|
||||
`WGInterfaceConfig` to own a tuple of unique peers beside its private
|
||||
key, local addresses and listen port.
|
||||
|
||||
Peer PSKs remain redacted and nested `ProcessLocal` sentinels prevent
|
||||
default wire encoding. Validation covers keys, routes, endpoints,
|
||||
ports, keepalive and duplicate peers.
|
||||
|
||||
Ruff and lock checks passed. Focused peer/config coverage passed 11
|
||||
tests. A broad unrelated registrar cancellation case timed out after
|
||||
86 passes and 2 xpasses; its isolated rerun passed.
|
||||
|
|
@ -0,0 +1,42 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
session: tractor-addr-unpacking
|
||||
timestamp: 2026-08-26T00:34:30Z
|
||||
git_ref: 6dd39da0
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260826T003430Z_6dd39da0_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
After committing explicit WireGuard peer configuration, the human
|
||||
authorized the next incremental lifecycle change.
|
||||
|
||||
## Response summary
|
||||
|
||||
Added owned WireGuard interface provisioning and teardown through
|
||||
cancellation-shielded pyroute2 worker calls. Role-specific policy is
|
||||
validated before mutation, local addresses and explicit peers are
|
||||
configured, and owned interfaces are removed on exit.
|
||||
Staged review clarified role-specific test assertions, added the
|
||||
second peer's endpoint-port check, included conflicting values in
|
||||
policy errors and aligned annotated assignment with `/py-codestyle`.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `tractor/discovery/_tunnel.py` - role policy and WG iface lifecycle.
|
||||
- `tractor/discovery/__init__.py` - public lifecycle and role exports.
|
||||
- `tests/discovery/test_wg_iface_lifecycle.py` - policy and cancellation
|
||||
coverage with privileged calls faked.
|
||||
- `ai/tpt-backends/03_wg_tunnel_bindspace.md` - initial owned WG iface
|
||||
lifecycle contract.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human selected continuation from explicit peer modeling into the
|
||||
owned WireGuard interface lifecycle. During staged review, the human
|
||||
requested clearer role assertions, complete endpoint checks, concrete
|
||||
error values and inline annotated assignment. The agent applied those
|
||||
human-directed edits; no direct manual source edits were observed.
|
||||
|
|
@ -0,0 +1,29 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-26T00:34:30Z
|
||||
git_ref: 6dd39da0
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
# Raw output - own WireGuard interface lifecycle
|
||||
|
||||
After committing explicit peer configuration, the human authorized the
|
||||
next incremental lifecycle change.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tractor/discovery/_tunnel.py tractor/discovery/__init__.py tests/discovery/test_wg_iface_lifecycle.py ai/tpt-backends/03_wg_tunnel_bindspace.md`
|
||||
|
||||
Added owned async `open_wg_iface()` lifecycle with role-policy
|
||||
validation before side effects. Listen bearer ports configure the local
|
||||
listener; dial bearers fill omitted endpoints only for the selected
|
||||
maddr peer.
|
||||
|
||||
Pyroute2 worker code creates the WireGuard link, assigns local
|
||||
addresses, configures private key/listen port and every explicit peer,
|
||||
raises the link, and removes it on exit. Partial create failures clean
|
||||
up synchronously; async create/remove calls are cancellation-shielded.
|
||||
|
||||
Privileged operations are faked in tests. Ruff and lock checks passed;
|
||||
focused WG lifecycle/config/inspection coverage passed 22 tests. A
|
||||
broad unrelated registrar cancellation parameter timed out after 76
|
||||
passes and 2 xpasses; its isolated rerun passed.
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
session: tractor-addr-unpacking
|
||||
timestamp: 2026-08-26T02:24:34Z
|
||||
git_ref: 2245f094
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260826T022434Z_2245f094_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
After committing the owned WireGuard interface lifecycle, the human
|
||||
asked to continue with the next established incremental step.
|
||||
|
||||
## Response summary
|
||||
|
||||
Added a WG-specific composition context which owns one declared
|
||||
bindspace and an ordered interface stack. It snapshots caller-owned
|
||||
layer order before checkpointing, enters interfaces outermost-first,
|
||||
and guarantees inside-out interface teardown before bindspace release.
|
||||
Its inbound type shows the `(WGTunnelSpec, WGInterfaceConfig)` pairing
|
||||
directly, and its docstring depicts the stack and unwind order.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `tractor/discovery/_tunnel.py` - bindspace/interface stack driver.
|
||||
- `tractor/discovery/__init__.py` - public layer and driver exports.
|
||||
- `tests/discovery/test_wg_iface_lifecycle.py` - mutable-input,
|
||||
cancellation and nested teardown-order coverage.
|
||||
- `ai/tpt-backends/03_wg_tunnel_bindspace.md` - implemented composition
|
||||
and ownership-order contract.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human accepted and committed the preceding reviewed lifecycle
|
||||
boundary, then explicitly requested the next step. That selected the
|
||||
previously established bindspace/WireGuard composition boundary. In
|
||||
staged review, the human requested an explicit interface-stack analogy
|
||||
and a small ASCII diagram explaining entry and exit order. The human
|
||||
then identified `WGInterfaceLayer` as an unnecessary single-use alias
|
||||
which obscured the inbound tuple shape. The agent applied both
|
||||
human-directed edits. The human accepted the fake lifecycle test as a
|
||||
focused control-flow check which avoids pyroute2 overhead, while noting
|
||||
their general preference for end-to-end coverage. No direct manual
|
||||
source edits were observed.
|
||||
|
|
@ -0,0 +1,28 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-26T02:24:34Z
|
||||
git_ref: 2245f094
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
# Raw output - compose WireGuard bindspace lifecycles
|
||||
|
||||
After committing the owned WireGuard interface lifecycle, the human
|
||||
requested the next incremental change.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tractor/discovery/_tunnel.py tractor/discovery/__init__.py tests/discovery/test_wg_iface_lifecycle.py ai/tpt-backends/03_wg_tunnel_bindspace.md`
|
||||
|
||||
Added `open_wg_bindspace()` to compose one declared bindspace with an
|
||||
ordered sequence of WireGuard interface layers. The context enters the
|
||||
bindspace first, enters interfaces outermost-first through
|
||||
`AsyncExitStack`, yields the live `BindspaceHandle`, and removes
|
||||
interfaces inside-out before releasing or removing the namespace.
|
||||
|
||||
The caller's layer sequence is snapshotted before the first checkpoint,
|
||||
so concurrent mutation cannot alter resource entry or unwind order.
|
||||
Fake lifecycle tests clear a mutable input list during bindspace entry,
|
||||
cancel from the yielded scope, and prove the original stack unwinds in
|
||||
dependency-safe order.
|
||||
|
||||
Ruff passed and focused bindspace/WireGuard coverage passed 39 tests.
|
||||
|
|
@ -0,0 +1,54 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
session: tractor-addr-unpacking
|
||||
timestamp: 2026-08-26T03:05:34Z
|
||||
git_ref: d130431c
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260826T030534Z_d130431c_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
After committing the reviewed bindspace/interface stack, the human
|
||||
asked to continue with the next established Layer C step.
|
||||
|
||||
## Response summary
|
||||
|
||||
Retained a serializable realized `BindspaceRef` on frozen tunnelled
|
||||
address declarations without carrying the process-local `Bindspace`
|
||||
capability. The namespace API remains tuple-shaped, using the declared
|
||||
key before realization and stable inode afterward. Existing transport
|
||||
peeling and unrealized serialization remain unchanged.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `tractor/discovery/_tunnel.py` - realized bindspace-ref retention and
|
||||
immutable annotation method.
|
||||
- `tests/discovery/test_tunnelled_addr.py` - ref, compatibility and
|
||||
mismatch coverage.
|
||||
- `tests/ipc/test_server_tunnel_boundary.py` - real listener diagnostic
|
||||
coverage for the realized inode.
|
||||
- `ai/tpt-backends/03_wg_tunnel_bindspace.md` - realized namespace
|
||||
visibility contract.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human accepted and committed the preceding reviewed composition
|
||||
boundary, then requested the next step. Their stated preference for
|
||||
end-to-end tests informed use of the existing real TCP listener path to
|
||||
verify endpoint/server diagnostics, while focused unit coverage handles
|
||||
metadata invariants without pyroute2 overhead. During review, the human
|
||||
chose the final unshipped terminology: `BindspaceRef` for the
|
||||
serializable non-owning reference, `Bindspace` for the live
|
||||
process-local capability, `.ref` for that capability's reference, and
|
||||
`bindspace_ref` at the tunnel declaration API.
|
||||
They then requested the prerequisite model rename as a separate commit
|
||||
before this feature. The human also replaced the module-level helper
|
||||
with `TunnelledAddress.with_bindspace_ref()` and requested inline
|
||||
msgspec encode/decode expressions in the serialization assertions. The
|
||||
human chose not to add a second typed `.namespace` projection, and
|
||||
requested native tagged `TunnelledAddress` decoding remain as a separate
|
||||
design-plan follow-up. The agent applied those human-directed changes;
|
||||
no direct manual source edits were observed.
|
||||
|
|
@ -0,0 +1,31 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-26T03:05:34Z
|
||||
git_ref: d130431c
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
# Raw output - retain realized bindspace identity
|
||||
|
||||
After committing WireGuard bindspace composition, the human requested
|
||||
the next incremental Layer C change.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tractor/discovery/_tunnel.py tractor/discovery/__init__.py tests/discovery/test_tunnelled_addr.py tests/ipc/test_server_tunnel_boundary.py ai/tpt-backends/03_wg_tunnel_bindspace.md`
|
||||
|
||||
Added optional `BindspaceIdentity` metadata to frozen
|
||||
`TunnelledAddress` declarations and a pure
|
||||
`with_bindspace_identity()` annotation helper. Unrealized declarations
|
||||
retain their prior serialized shape. Realized declarations retain only
|
||||
serializable key/inode identity, never the FD-bearing capability.
|
||||
|
||||
The existing `.namespace` tuple contract remains compatible:
|
||||
unrealized declarations report `(kind, key)`, while realized
|
||||
declarations report the stable `(kind, inode)`. Name mismatches between
|
||||
the tunnel declaration and realized bindspace are rejected.
|
||||
|
||||
Unit coverage verifies immutability, serialization, delegation and
|
||||
mismatch handling. The existing real TCP listener test proves endpoint
|
||||
and server diagnostics expose the retained inode without implying that
|
||||
the process entered that namespace. Ruff passed and focused tunnel,
|
||||
listener and bindspace coverage passed 30 tests.
|
||||
|
|
@ -0,0 +1,43 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
session: tractor-addr-unpacking
|
||||
timestamp: 2026-08-27T21:11:15Z
|
||||
git_ref: d130431c
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260827T211115Z_d130431c_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
During review, the human requested concrete names for the existing
|
||||
bindspace abstractions and asked to isolate that rename before realized
|
||||
reference retention.
|
||||
|
||||
## Response summary
|
||||
|
||||
Renamed the serializable resource record to `BindspaceRef`, the live
|
||||
FD-backed capability to `Bindspace`, and the capability's record field
|
||||
to `.ref`. Updated existing lifecycle APIs, tests and active design
|
||||
contracts without compatibility aliases.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `tractor/discovery/_bindspace.py` - renamed models and `.ref` field.
|
||||
- `tractor/discovery/_tunnel.py` - existing lifecycle annotations.
|
||||
- `tractor/discovery/__init__.py` - renamed public model exports.
|
||||
- `tests/discovery/test_bindspace.py` - renamed model contracts.
|
||||
- `tests/discovery/test_wg_iface_lifecycle.py` - live bindspace usage.
|
||||
- `ai/tpt-backends/00_shared_backend_contract.md` - shared terminology.
|
||||
- `ai/tpt-backends/03_wg_tunnel_bindspace.md` - request, ref and live
|
||||
capability terminology.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human identified the earlier identity/handle terminology as too
|
||||
abstract for an IPC-passed non-owning reference and live resource. They
|
||||
selected `BindspaceRef`, `Bindspace`, and `.ref`, then requested this
|
||||
rename as a prerequisite commit separate from the realized-ref feature.
|
||||
The agent applied those human-directed names; no direct manual source
|
||||
edits were observed.
|
||||
|
|
@ -0,0 +1,26 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-27T21:11:15Z
|
||||
git_ref: d130431c
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
# Raw output - simplify bindspace model names
|
||||
|
||||
While reviewing realized bindspace metadata, the human requested more
|
||||
concrete names for the existing bindspace model family and asked that
|
||||
the rename remain separate from the subsequent feature.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tractor/discovery/_bindspace.py tractor/discovery/_tunnel.py tractor/discovery/__init__.py tests/discovery/test_bindspace.py tests/discovery/test_wg_iface_lifecycle.py ai/tpt-backends/00_shared_backend_contract.md ai/tpt-backends/03_wg_tunnel_bindspace.md`
|
||||
|
||||
Renamed `BindspaceIdentity` to `BindspaceRef` for the serializable,
|
||||
non-owning, host-local resource record. Renamed `BindspaceHandle` to
|
||||
`Bindspace` for the live process-local FD-backed capability, and renamed
|
||||
its `.identity` field to `.ref`.
|
||||
|
||||
No compatibility aliases were added because the models are unshipped.
|
||||
Active design contracts now distinguish the serialized request, the
|
||||
wire-safe non-owning ref, and the live capability. Ruff passed, 518
|
||||
tests collected, and 53 focused bindspace/WireGuard/listener tests
|
||||
passed across the final two-commit tree.
|
||||
|
|
@ -0,0 +1,41 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
session: tractor-addr-unpacking
|
||||
timestamp: 2026-08-27T23:25:00Z
|
||||
git_ref: d067505a
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260827T232500Z_d067505a_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
After committing the reviewed bindspace-reference changes, the human
|
||||
asked to continue with the next Layer C increment.
|
||||
|
||||
## Response summary
|
||||
|
||||
Added the private Linux network-namespace entry primitive required by
|
||||
future child bootstrap wiring. It verifies the inherited capability
|
||||
before `setns()`, constrains entry to network namespaces, and verifies
|
||||
the resulting process namespace before runtime initialization proceeds.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `tractor/spawn/_netns.py` - synchronous verified netns entry.
|
||||
- `tests/test_netns_spawn.py` - stale-FD, syscall and post-entry guards.
|
||||
- `ai/tpt-backends/03_wg_tunnel_bindspace.md` - primitive ownership and
|
||||
remaining integration boundary.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human requested the next incremental implementation after accepting
|
||||
the prior split commits. Their established preference for end-to-end
|
||||
tests informed use of real FDs where possible; only privileged `setns()`
|
||||
and procfs observation are replaced to keep the suite unprivileged. In
|
||||
staged review, the human requested the full project license header,
|
||||
explicit comments on both intentional inode mismatches, and independent
|
||||
call traces proving both fake syscall and procfs hooks execute. The
|
||||
agent applied those human-directed revisions; no direct manual source
|
||||
edits were observed.
|
||||
|
|
@ -0,0 +1,25 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-27T23:25:00Z
|
||||
git_ref: d067505a
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
# Raw output - verify pre-runtime netns entry
|
||||
|
||||
After committing realized bindspace references and their deferred codec
|
||||
plan, the human requested the next Layer C increment.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tractor/spawn/_netns.py tests/test_netns_spawn.py ai/tpt-backends/03_wg_tunnel_bindspace.md`
|
||||
|
||||
Added a private synchronous `enter_netns()` bootstrap primitive. It
|
||||
validates the inherited FD and expected inode, constrains `setns()` to
|
||||
`CLONE_NEWNET`, verifies `/proc/self/ns/net` afterward, and leaves FD
|
||||
closure to the future spawn-bootstrap caller.
|
||||
|
||||
Tests use real stand-in FDs while replacing only the privileged syscall
|
||||
and post-entry procfs observation. They prove stale FDs fail before
|
||||
entry, the syscall receives the exact namespace type, and bootstrap
|
||||
rejects an unexpected post-entry namespace. Ruff and all three focused
|
||||
tests passed.
|
||||
|
|
@ -0,0 +1,45 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
session: addr-unpacking-rebase-conflict
|
||||
timestamp: 2026-08-28T00:38:05Z
|
||||
git_ref: e40c39da
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260828T003805Z_e40c39da_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The human asked the agent to resolve the first conflict in the
|
||||
`wkt/addr_unpacking` rebase because the replayed history was primarily
|
||||
agent-generated.
|
||||
|
||||
## Response summary
|
||||
|
||||
Preserved the lazy-import and platform-aware behavior from the new
|
||||
base while restoring the replayed WireGuard address parsing and
|
||||
composition behavior. Reconciled the changed dependency lock with the
|
||||
replayed direct dependency contract and validated the combined result.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `tractor/discovery/_addr.py` - combined address-platform and tunnel
|
||||
typing changes.
|
||||
- `tractor/discovery/_multiaddr.py` - combined lazy imports with WG
|
||||
parse and composition dispatch.
|
||||
- `tractor/discovery/_tunnel.py` - retained cold-import behavior for
|
||||
the newly added WG implementation.
|
||||
- `tests/discovery/test_multiaddr.py` - adapted missing-codec
|
||||
monkeypatching to the lazily imported upstream protocol module.
|
||||
- `uv.lock` - restored the WG-capable multiaddr source and direct
|
||||
multibase metadata.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human chose to abort the first rebase attempt, restart with an
|
||||
explicit old-base boundary, and delegated the conflict resolution to
|
||||
the agent. The human also chose not to preserve the superseded
|
||||
module-level protocol-lookup test seam; the agent redirected the test
|
||||
to the upstream protocol module. No direct manual source edits were
|
||||
observed during this resolution.
|
||||
|
|
@ -0,0 +1,32 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-28T00:38:05Z
|
||||
git_ref: e40c39da
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
# Raw output - addr-unpacking rebase resolution
|
||||
|
||||
The human delegated resolution of the first conflict while rebasing
|
||||
`wkt/addr_unpacking` from `d9a6e2e9` onto `85a44588`.
|
||||
|
||||
## Generated code
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tractor/discovery/_addr.py tractor/discovery/_multiaddr.py tractor/discovery/_tunnel.py`
|
||||
|
||||
Combined the rebased base's platform-aware address registries and lazy
|
||||
optional-dependency imports with the replayed `TunnelledAddress` typing
|
||||
and WireGuard multiaddr dispatch. Adapted the new tunnel module so
|
||||
importing tractor does not eagerly load `multiaddr`.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- uv.lock`
|
||||
|
||||
Regenerated the lockfile to retain the replayed WG-capable
|
||||
`py-multiaddr` revision and direct `py-multibase` dependency after the
|
||||
new base's independently changed lockfile silently omitted that hunk.
|
||||
|
||||
## Verification
|
||||
|
||||
Conflict-marker, syntax, lock consistency, lint, collection, focused
|
||||
discovery, and cold-import checks were selected for the resolution.
|
||||
|
|
@ -0,0 +1,38 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
session: wg-netns-bootstrap-entry
|
||||
timestamp: 2026-08-28T01:22:05Z
|
||||
git_ref: 9ae7cd86
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260828T012205Z_9ae7cd86_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
After completing and auditing the stacked rebase, the human approved
|
||||
the next WireGuard Layer C slice: integrate the verified network
|
||||
namespace entry primitive into child spawn bootstrap.
|
||||
|
||||
## Response summary
|
||||
|
||||
Wire an optional child-owned namespace FD and expected inode into both
|
||||
child entrypoints. Keep namespace entry and FD closure synchronous and
|
||||
ordered before actor runtime initialization, while deliberately
|
||||
deferring backend FD transport and public API plumbing.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `tractor/spawn/_entry.py` - enter and release an optional inherited
|
||||
namespace capability before multiprocessing child runtime startup.
|
||||
- `tractor/_child.py` - consume the capability at the start of the
|
||||
Trio child bootloader, before patching or actor construction.
|
||||
- `tests/test_netns_spawn.py` - prove entry ordering, failure isolation,
|
||||
and exact FD ownership for both child entrypoints.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human selected spawn-time namespace integration as the next work
|
||||
slice and authorized implementation. No direct manual source edits
|
||||
were observed during generation.
|
||||
|
|
@ -0,0 +1,36 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-28T01:22:05Z
|
||||
git_ref: 9ae7cd86
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
# Raw output - enter inherited netns during child bootstrap
|
||||
|
||||
The human asked to continue the WireGuard Layer C implementation after
|
||||
the rebased branch landed the verified `spawn._netns.enter_netns()`
|
||||
primitive.
|
||||
|
||||
## Generated code
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tractor/_child.py tractor/spawn/_entry.py tests/test_netns_spawn.py`
|
||||
|
||||
Add one optional atomic `(namespace_fd, expected_inode)` bootstrap
|
||||
capability to the multiprocessing entrypoint and Trio child bootloader.
|
||||
Enter and verify the namespace before Trio patching, actor construction,
|
||||
runtime state, logging, Trio startup, parent connection, sockets, or
|
||||
worker threads, then close the child-owned inherited FD before
|
||||
continuing.
|
||||
|
||||
Add privilege-free direct-entrypoint tests using real stand-in FDs and
|
||||
a fake namespace syscall boundary. Cover helper-level no-op behavior,
|
||||
successful entry ordering, exact FD closure, malformed-FD isolation,
|
||||
primary-error preservation, and failure cleanup for both child
|
||||
bootstrap paths.
|
||||
|
||||
## Scope boundary
|
||||
|
||||
This increment does not transfer FDs through spawn backends or expose a
|
||||
public actor API. Backend-specific FD duplication, bootstrap failure
|
||||
reporting, and root-process namespace entry remain follow-up work.
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
session: trio-pre-handshake-child-death
|
||||
timestamp: 2026-08-28T04:51:19Z
|
||||
git_ref: fb6d81d3
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260828T045119Z_fb6d81d3_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The human approved the next WireGuard Layer C implementation step after
|
||||
committing child-side network namespace bootstrap.
|
||||
|
||||
## Response summary
|
||||
|
||||
Restore and harden the boot-race design originally implemented by
|
||||
`3b0724eba85b4014170ed95773e1e41a60d5c513` before transferring
|
||||
namespace FDs. Race the initial peer handshake against child-process
|
||||
death so a pre-connection bootstrap failure surfaces instead of
|
||||
parking `ActorNursery.start_actor()` forever.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `tractor/spawn/_spawn.py` - provide the generic handshake/process
|
||||
death race.
|
||||
- `tractor/spawn/_trio.py` - use the race during Trio child startup.
|
||||
- `tests/test_spawning.py` - cover handshake-first and death-first
|
||||
schedules, simultaneous outcomes, waiter errors, and backend cleanup.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human approved continued Layer C implementation. The agent narrowed
|
||||
the next slice to failure detection after analysis showed FD transfer
|
||||
would otherwise introduce a parent hang. The human then identified the
|
||||
earlier `wait_for_peer_or_proc_death()` implementation and required
|
||||
explicit credit; the source docstring and commit message now cite its
|
||||
original commit rather than a later cherry-pick. The human directly
|
||||
reflowed `_trio.py` to the project line limit and normalized nearby
|
||||
quotes. The human also required naming the race helper in its lifecycle
|
||||
comment, explicit `trio.Cancelled` handling in waiter tests, expected
|
||||
cancellation diagnostics, and a `trio.Process`-constrained test double.
|
||||
After mainline portability verification, the human chose to commit the
|
||||
fix on the current WireGuard branch and require its cherry-pick onto
|
||||
`main` as a blocker in the eventual WireGuard pull request.
|
||||
|
|
@ -0,0 +1,34 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-28T04:51:19Z
|
||||
git_ref: fb6d81d3
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
# Raw output - detect child death before parent handshake
|
||||
|
||||
The human asked to continue WireGuard Layer C after child entrypoints
|
||||
learned to consume inherited network namespace capabilities.
|
||||
|
||||
## Generated code
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tractor/spawn/_spawn.py tractor/spawn/_trio.py tests/test_spawning.py`
|
||||
|
||||
Race a spawned child's initial parent handshake against its process
|
||||
wait operation. Return the established channel when the handshake wins;
|
||||
raise `ActorFailure` with the exit status when the process dies first.
|
||||
Use the race in the Trio exec backend instead of waiting indefinitely
|
||||
on `IPCServer.wait_for_peer()`.
|
||||
|
||||
Add deterministic Trio tests for handshake-first, death-first,
|
||||
simultaneous, and waiter-error schedules, controlling readiness with
|
||||
events rather than sleeps. Exercise the full Trio backend with a fake
|
||||
dead process to prove pre-publication event registration and exact
|
||||
failed-startup cleanup.
|
||||
|
||||
## Scope boundary
|
||||
|
||||
This prerequisite prevents pre-connection namespace-entry failures
|
||||
from hanging their parent. Namespace FD transfer itself remains the
|
||||
next commit.
|
||||
|
|
@ -0,0 +1,53 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
session: trio-bindspace-fd-transport
|
||||
timestamp: 2026-08-28T17:29:43Z
|
||||
git_ref: 2ca8c570
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260828T172943Z_2ca8c570_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
After committing generic pre-handshake child-death detection, the human
|
||||
approved the proposed next Layer C slice: transport a live bindspace
|
||||
capability through the Trio spawn backend.
|
||||
|
||||
## Response summary
|
||||
|
||||
Plumb an optional process-local `Bindspace` through actor spawn APIs.
|
||||
Give a Trio exec child its own inherited namespace descriptor, preserve
|
||||
existing caller `pass_fds`, transport the FD/inode pair through the CLI,
|
||||
and close the temporary parent descriptor on every open-process outcome.
|
||||
Reject unsupported multiprocessing transport explicitly. Exercise the
|
||||
complete Trio path with a real actor that enters a distinct network
|
||||
namespace and reports its namespace and inherited-FD inodes over RPC.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `tractor/runtime/_supervise.py` - accept and relay a child bindspace.
|
||||
- `tractor/to_actor/_api.py` - expose bindspace spawn configuration.
|
||||
- `tractor/spawn/_spawn.py` - relay the bindspace to spawn backends.
|
||||
- `tractor/spawn/_trio.py` - duplicate, transfer, and close the FD.
|
||||
- `tractor/spawn/_mp.py` - reject unsupported MP transport.
|
||||
- `tractor/_child.py` - parse and forward the atomic CLI capability.
|
||||
- `tests/test_netns_spawn.py` - cover transport and cleanup schedules.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human approved the Trio-first transport boundary after reviewing
|
||||
the proposed dependency order, then supplied staged-diff comments across
|
||||
multiple review passes. Human-directed revisions replaced abstract
|
||||
ownership wording with concrete FD and process terminology, clarified
|
||||
descriptor cleanup, preserved existing `pass_fds` explicitly, and
|
||||
replaced the mocked success-path test with a real child actor, real
|
||||
`setns()`, handshake, and RPC round trip inside unprivileged user/network
|
||||
namespaces. Follow-up review added direct parent FD-table verification
|
||||
that the temporary child-bound duplicate is closed after successful
|
||||
spawn.
|
||||
|
||||
Multiprocessing FD reduction and root-process namespace entry remain
|
||||
separate follow-up slices. No direct manual source edits were observed;
|
||||
the review-driven source revisions were applied by the agent.
|
||||
|
|
@ -0,0 +1,34 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-28T17:29:43Z
|
||||
git_ref: 2ca8c570
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
# Raw output - transport bindspace FD through Trio spawn
|
||||
|
||||
The human approved the next WireGuard Layer C increment after landing
|
||||
pre-handshake child-death detection.
|
||||
|
||||
## Generated code
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tractor/_child.py tractor/runtime/_supervise.py tractor/spawn/_spawn.py tractor/spawn/_trio.py tractor/spawn/_mp.py tractor/to_actor/_api.py tests/test_netns_spawn.py`
|
||||
|
||||
Thread an optional live `Bindspace` through actor spawn APIs. For the
|
||||
Trio backend, duplicate its namespace FD, append that temporary
|
||||
descriptor to existing `pass_fds`, and send the atomic descriptor/inode
|
||||
pair through the child CLI. Close the parent temporary duplicate as
|
||||
soon as `open_process()` returns, raises, or is cancelled, while the
|
||||
child consumes its inherited descriptor before runtime startup.
|
||||
|
||||
Reject bindspace transport explicitly in multiprocessing backends until
|
||||
their descriptor-reduction slice is implemented. Add privilege-free
|
||||
tests for CLI forwarding, merged pass-FD state, caller immutability,
|
||||
source/duplicate ownership, exec failure, cancellation, absent live
|
||||
authority, death-before-handshake cleanup, and MP rejection.
|
||||
|
||||
## Scope boundary
|
||||
|
||||
Multiprocessing FD reduction and root-process namespace entry remain
|
||||
follow-up work.
|
||||
|
|
@ -0,0 +1,62 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
session: pr512-docs-review
|
||||
timestamp: 2026-08-28T20:08:22Z
|
||||
git_ref: wkt/big_boi_docs_472_follow_ups
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260828T200822Z_0be872ff_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Prime PR #512 for landing by applying the repository Python style and
|
||||
typing policy across changed examples, reviewing and correcting the
|
||||
result with delegated agents, validating it, and auditing documentation
|
||||
changes since PR #460 for broader follow-up work.
|
||||
|
||||
## Response summary
|
||||
|
||||
Completed a branch-wide style, typing, and docstring pass over the
|
||||
changed examples. Reworked the dedicated registrar example so it proves
|
||||
external discovery, handles bind collisions, publishes readiness
|
||||
atomically, and shuts down cleanly across platforms. Staged review moved
|
||||
it under the discovery examples, introduced an async lifecycle manager,
|
||||
and captured public-subsystem, Piker, and pytest follow-ups. Corrected
|
||||
discovery and typed-messaging documentation after cross-checking runtime
|
||||
behavior.
|
||||
Created issue #514 for typed `Start` contracts and linked its planned
|
||||
sender-side argument validation from the messaging guide.
|
||||
Independent review passes were repeated until no actionable findings
|
||||
remained. Static checks, Sphinx, runnable examples, Trio coverage, and
|
||||
focused registrar checks passed; Ctrl-C debugger failures remain
|
||||
unclassified after one case reproduced on untouched `main`.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `examples/discovery/dedicated_registrar.py` - implement robust
|
||||
standalone registrar discovery and teardown.
|
||||
- `examples/**/*.py` - apply style, typing, docstring, and terminology
|
||||
corrections to the 38 changed example modules.
|
||||
- `docs/guide/discovery.rst` - document actual registrar and
|
||||
multi-registrar behavior.
|
||||
- `docs/guide/msging.rst` - document actual validation and codec-hook
|
||||
boundaries.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human selected PR #512, required the `/py-codestyle` policy and
|
||||
typing improvements throughout the examples, required delegated review
|
||||
agents to find additional fixes, authorized agents to own and push their
|
||||
code through the proper review process, and requested a broad audit of
|
||||
documentation changes since PR #460 rendered as an evolving Taken WIP.
|
||||
The human then required the earlier `ParamSpec` proposal to be captured
|
||||
as a dedicated GitHub issue and linked from the current docs patch.
|
||||
The human also required the registrar demo to move under discovery, use
|
||||
structural matching and an async context manager, explain socket ownership,
|
||||
and preserve public API, Piker, and pytest-harness follow-ups.
|
||||
The human also explicitly retracted messages intended for another
|
||||
session; those messages did not direct the result. No direct manual
|
||||
source edits were identified; the agent applied the human-directed
|
||||
changes.
|
||||
|
|
@ -0,0 +1,90 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-28T20:08:22Z
|
||||
git_ref: wkt/big_boi_docs_472_follow_ups
|
||||
diff_cmd: git diff 0be872ff..HEAD
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Prime PR #512 for landing. Apply the Python style policy throughout
|
||||
the changed examples, improve typing, use review agents to find and
|
||||
fix correctness issues, run the proper review process, push the agent's
|
||||
own code, and then audit documentation changes since PR #460 for stale,
|
||||
missing, defunct, or improvable coverage.
|
||||
|
||||
## Response
|
||||
|
||||
> `git diff 0be872ff..HEAD -- examples/`
|
||||
|
||||
Restyled and typed all 38 changed example modules while preserving
|
||||
intentional debugger failures and hanging reproducers. Added required
|
||||
multiline docstrings, corrected async callable annotations, modernized
|
||||
legacy API terminology, and preserved complete debugger process trees.
|
||||
|
||||
Rebuilt `examples/discovery/dedicated_registrar.py` as a real
|
||||
external-discovery example. A standalone registrar process now selects
|
||||
and retries bounded ephemeral addresses, publishes readiness atomically,
|
||||
rejects accidental registrar reuse, serves sibling service and client
|
||||
actors, proves lookup used the registrar instead of a local-peer channel,
|
||||
and performs bounded, validated, cross-platform shutdown.
|
||||
|
||||
Staged review moved the demo into the discovery example group, converted
|
||||
collision classification to structural pattern matching, documented the
|
||||
selector-socket close/rebind race, and extracted process ownership into an
|
||||
async context manager. A source TODO records the future public discovery
|
||||
subsystem, Piker service-management lessons, and pytest registry-isolation
|
||||
use case.
|
||||
|
||||
> `git diff 0be872ff..HEAD -- docs/guide/discovery.rst`
|
||||
|
||||
Corrected registrar, duplicate-name, and multi-registrar discovery
|
||||
guidance, including configured-order and `None` placeholder behavior.
|
||||
|
||||
> `git diff 0be872ff..HEAD -- docs/guide/msging.rst`
|
||||
|
||||
Corrected typed-payload validation boundaries and separated working
|
||||
task-scoped codec encoding from private per-dialog decoding and the
|
||||
incomplete decorator hook parameters.
|
||||
|
||||
## Review and validation
|
||||
|
||||
Multiple independent reviews found and drove fixes for registrar
|
||||
discovery validity, port-selection races, teardown, process diagnostics,
|
||||
shutdown status, inaccurate discovery ordering, async callable typing,
|
||||
missing docstrings, truncated debugger command diagrams, stale APIs, and
|
||||
payload-error relay wording.
|
||||
|
||||
Validation completed:
|
||||
|
||||
- AST parsing, Ruff, 69-column checks, and required-docstring audit for
|
||||
all 38 changed Python files.
|
||||
- Sphinx HTML build succeeded.
|
||||
- Documentation example harness: 24 passed.
|
||||
- Trio coverage: 7 passed, 4 xfailed, 1 xpassed.
|
||||
- Dedicated registrar direct run and focused harness test passed with a
|
||||
clean child exit and no traceback.
|
||||
- Debugger suite: 20 passed, 6 skipped, and 8 reproducible
|
||||
`ctl-c=True` pexpect timeouts. One exact failure reproduced on
|
||||
untouched `main`, confirming a baseline failure; its root cause
|
||||
remains unclassified. No leaked actor processes remained.
|
||||
|
||||
The post-PR-#460 audit identified follow-up work around advertised but
|
||||
inert runtime selectors, platform/backend support, unresolved discovery
|
||||
contracts, cached-context teardown, codec recipes, examples-as-tests
|
||||
coverage, public API exports, broadcast factory contracts, stale examples,
|
||||
README duplication, release notes, and process-title terminology.
|
||||
|
||||
## Follow-up prompt
|
||||
|
||||
Capture the previously proposed `ParamSpec`-based `Start` argument
|
||||
validation work in a focused GitHub issue, then link it from the current
|
||||
typed-messaging docs patch as planned sender-side checking.
|
||||
|
||||
## Follow-up response
|
||||
|
||||
Created https://github.com/goodboy/tractor/issues/514 to track deriving
|
||||
typed `Start` contracts from endpoint signatures, preserving caller-facing
|
||||
signatures, and validating arguments before sending where possible. Added
|
||||
the issue link beside the guide's current `Start` validation boundary.
|
||||
|
|
@ -0,0 +1,409 @@
|
|||
# `tractor.ipc` next-gen transport backends: the shared contract
|
||||
|
||||
Status: design doc / implementation spec.
|
||||
Audience: any model or human implementing one of the three
|
||||
sibling plans in this directory.
|
||||
|
||||
- [`01_tipc_backend.md`](./01_tipc_backend.md) — `AF_TIPC`
|
||||
(gh #378)
|
||||
- [`02_quic_iroh_backend.md`](./02_quic_iroh_backend.md) — QUIC
|
||||
via `iroh` FFI, uniffi-async rewritten onto `trio` (gh #353)
|
||||
- [`03_wg_tunnel_bindspace.md`](./03_wg_tunnel_bindspace.md) —
|
||||
WireGuard (and other shuttle-able) tunnels as a *nested
|
||||
bindspace* layer via `pyroute2` (gh #482, #443)
|
||||
|
||||
This doc is the **normative** description of what a `tractor`
|
||||
transport backend *is* as of `main@83b34884`. Each sibling plan
|
||||
assumes it and only documents its own deltas. Read this first;
|
||||
do not re-derive it from the code.
|
||||
|
||||
---
|
||||
|
||||
## 0. Why a shared contract doc
|
||||
|
||||
The three plans are meant to be implementable *independently and
|
||||
concurrently* by different models/providers without design
|
||||
drift. Everything they share — the backend duck-type, the
|
||||
registration tables, the test harness plumbing, the naming and
|
||||
code-style rules — lives here exactly once. If an implementer
|
||||
finds this doc disagrees with `main`, **the code wins**; fix this
|
||||
doc in the same PR.
|
||||
|
||||
---
|
||||
|
||||
## 1. The backend duck-type (empirical, from `_tcp.py`/`_uds.py`)
|
||||
|
||||
A transport backend is **one module** under `tractor/ipc/`
|
||||
exposing exactly four things. There is no ABC to subclass and no
|
||||
plugin entrypoint; wiring is by explicit table registration
|
||||
(§2) plus one piece of reflection (§1.3).
|
||||
|
||||
### 1.1 `class <Proto>Address(msgspec.Struct, frozen=True)`
|
||||
|
||||
Structurally conforms to the `Address` `Protocol` in
|
||||
`tractor/discovery/_addr.py:82`. Required surface:
|
||||
|
||||
| member | kind | notes |
|
||||
| --- | --- | --- |
|
||||
| `proto_key` | `ClassVar[str]` | the wire/registry key, e.g. `'tcp'`, `'uds'` |
|
||||
| `unwrapped_type` | `ClassVar[type]` | the primitive tuple shape |
|
||||
| `def_bindspace` | `ClassVar` | default bindspace value |
|
||||
| `is_valid` | `@property -> bool` | "is this a *dialable/bindable* addr" |
|
||||
| `bindspace` | `@property` | the "set of hosts"-ish scope (see below) |
|
||||
| `from_addr(cls, addr)` | `@classmethod` | primitive -> wrapped, `match`-based |
|
||||
| `unwrap(self)` | method | wrapped -> primitive (must be msgpack-native!) |
|
||||
| `get_random(cls, bindspace=...)` | `@classmethod` | per-subactor ephemeral addr |
|
||||
| `get_root(cls)` | `@classmethod` | host-singleton default registrar addr |
|
||||
| `__repr__` | method | `f'{type(self).__name__}[{...}]'` house style |
|
||||
|
||||
Hard constraints learned from the existing two:
|
||||
|
||||
- **`frozen=True`.** Addresses are dict keys
|
||||
(`Server.epsdict()`, `Endpoint.peer_tpts`) and are compared by
|
||||
value all over the runtime.
|
||||
- **`.unwrap()` output must round-trip through `msgspec` and
|
||||
through `wrap_address()`.** It is what actually crosses the
|
||||
wire in `SpawnSpec`/`_root_mailbox`/`_registry_addrs`, and it
|
||||
is what `Actor.reg_addrs` and every test compares against. If
|
||||
your unwrapped form is not *uniquely* pattern-matchable
|
||||
against the other backends' forms in
|
||||
`wrap_address()` (`_addr.py:230`), you have a bug that
|
||||
manifests as the wrong transport being loaded — the file's own
|
||||
`XXX NOTE` warns about precisely this.
|
||||
|
||||
⚠️ **and shape-matching does not survive 4 backends.** Adding
|
||||
TIPC and iroh breaks it outright: TIPC's natural form is a
|
||||
`(str, int)` — indistinguishable from `TCPAddress` — and
|
||||
iroh's is a `(str, str)`, which the *existing* UDS case
|
||||
(`case (_, filename) if type(filename) is str`) already
|
||||
swallows. Ordering hacks and prefix-tagging (an earlier
|
||||
revision of plan 01 proposed `('tipc:<stype>:<scope>', inst)`)
|
||||
paper over it at best.
|
||||
|
||||
**The fix, and the recommended prerequisite for all three
|
||||
backends: make the unwrapped form carry an explicit
|
||||
proto-key, using the `multiaddr` protocol name as the
|
||||
canonical spelling** — `('tcp', host, port)`,
|
||||
`('unix', path)`, `('udp', ...)`, `('tipc', stype, inst,
|
||||
scope)`. Then `wrap_address()` collapses from an
|
||||
order-sensitive `match` to `_address_types[addr[0]]`, and the
|
||||
whole collision class stops existing. Note this *also* aligns
|
||||
the on-wire form with `mk_maddr()`/`parse_maddr()`, so the two
|
||||
representations stop being independent inventions.
|
||||
|
||||
Two consequences to plan for:
|
||||
- it's a **wire-format change** (`SpawnSpec`,
|
||||
`_root_mailbox`, `_registry_addrs`) plus every test fixture
|
||||
and downstream config (`piker`'s `[network]` table). It
|
||||
wants its **own migration commit, landed before any new
|
||||
backend**, not smuggled into one.
|
||||
- it's the moment to **stop handing raw unwrapped tuples to
|
||||
users at all.** The long-term shape is: `Address` subtypes
|
||||
are the public currency and `UnwrappedAddress` becomes an
|
||||
internal serialization detail — the same discipline
|
||||
`ipaddress` uses (you pass `IPv4Address`, not a 4-tuple).
|
||||
Public API should accept `Address|maddr-str` and treat bare
|
||||
tuples as legacy-tolerated input, ideally deprecated.
|
||||
- **`.get_random()` must be collision-free without a live
|
||||
runtime.** See the `UDSAddress.get_random()` uuid-token
|
||||
comment (`_uds.py:207-220`): with no `current_actor()` the
|
||||
sockname degenerates to a pure fn of `(prefix, pid)` and two
|
||||
calls in one proc alias. Mix in a `uuid4().hex[:8]` token.
|
||||
- **`.bindspace` semantics**: "the address' bindable space" —
|
||||
ip/host for `tcp`, the socket-file *directory* for `uds`. For
|
||||
the new backends: the TIPC *scope* (§1 of plan 01), the iroh
|
||||
*ALPN + relay/discovery realm* (plan 02). Do not overload this
|
||||
transport-level bind selector with process namespace lifecycle.
|
||||
Plan 03 augments an maddr/address declaration with a serializable
|
||||
`BindspaceSpec` request and host-local `BindspaceRef`. A scoped,
|
||||
non-serializable `Bindspace` carries that ref and owns the FD/lifetime
|
||||
used during spawn bootstrap before a concrete address reaches
|
||||
transport bind.
|
||||
`Address.namespace` is already spec'd in the Protocol as
|
||||
"the if-available OS-specific network namespace key" and is
|
||||
currently unimplemented by both backends — plan 03 is the
|
||||
first real consumer.
|
||||
|
||||
### 1.2 module-level listener lifecycle
|
||||
|
||||
```python
|
||||
async def start_listener(
|
||||
addr: <Proto>Address,
|
||||
**kwargs,
|
||||
) -> trio.SocketListener # or a trio.abc.Listener, see §3
|
||||
...
|
||||
|
||||
def close_listener( # OPTIONAL
|
||||
addr: <Proto>Address,
|
||||
lstnr: trio.abc.Listener,
|
||||
) -> None:
|
||||
...
|
||||
```
|
||||
|
||||
`close_listener()` is optional; `Endpoint.close_listener()`
|
||||
(`_server.py:674`) `getattr`s it and treats absence as "closing
|
||||
is implicit". `uds` needs it (unlinks the sock-file), `tcp`
|
||||
does not.
|
||||
|
||||
### 1.3 the ONE piece of reflection you must not break
|
||||
|
||||
`Endpoint.start_listener()` (`_server.py:656`):
|
||||
|
||||
```python
|
||||
tpt_mod: ModuleType = inspect.getmodule(self.addr)
|
||||
lstnr = await tpt_mod.start_listener(addr=self.addr)
|
||||
```
|
||||
|
||||
The transport module is found by `inspect.getmodule()` **on the
|
||||
`Address` instance**. Therefore: *the `Address` class and its
|
||||
`start_listener()`/`close_listener()` MUST live in the same
|
||||
module.* Do not define the address type in `_types.py` or a
|
||||
`_addrs.py` and the listener elsewhere.
|
||||
|
||||
Immediately after, the same method does:
|
||||
|
||||
```python
|
||||
if (unwrapped := lstnr.socket.getsockname()) != self.addr.unwrap():
|
||||
self.addr = self.addr.from_addr(unwrapped)
|
||||
```
|
||||
|
||||
i.e. it assumes `lstnr.socket.getsockname()` exists and that its
|
||||
return value is a valid `from_addr()` input. This is fine for
|
||||
TIPC (§3 of plan 01) and **is the main integration hazard for
|
||||
iroh** (§3 of plan 02) — plans that break it must say so
|
||||
explicitly and propose the upstream `_server.py` patch.
|
||||
|
||||
### 1.4 `class Msgpack<Proto>Stream(MsgpackTransport)`
|
||||
|
||||
Subclass `tractor.ipc._transport.MsgpackTransport`. You inherit
|
||||
all framing (`<I` 4-byte little-endian length prefix),
|
||||
`msgspec` codec ctx-var lookup, `TransportClosed` normalization,
|
||||
`.drain()`, `__aiter__`. You implement only:
|
||||
|
||||
| member | notes |
|
||||
| --- | --- |
|
||||
| `address_type` | the `<Proto>Address` class |
|
||||
| `layer_key: int` | OSI-ish layer, `4` for both current backends |
|
||||
| `maddr` `@property` | `-> Multiaddr\|str`, via `mk_maddr(self.raddr)` |
|
||||
| `connected(self) -> bool` | `tcp`/`uds` both use `self.stream.socket.fileno() != -1` |
|
||||
| `connect_to(cls, addr, prefix_size=4, codec=None, **kw)` | `@classmethod`, returns an instance |
|
||||
| `get_stream_addrs(cls, stream) -> (laddr, raddr)` | `@classmethod`, called from `MsgpackTransport.__init__` |
|
||||
|
||||
`MsgpackTransport.__init__` requires the object passed as
|
||||
`stream` to satisfy:
|
||||
|
||||
- `await stream.send_all(bytes)`
|
||||
- usable as `tricycle.BufferedReceiveStream(transport_stream=stream)`,
|
||||
i.e. `await stream.receive_some(n)`
|
||||
- `trio.BrokenResourceError` / `trio.ClosedResourceError` /
|
||||
`ValueError('...unclean EOF...')` on the failure paths that
|
||||
`_iter_packets()` and `send()` already `match` on
|
||||
(`_transport.py:221-304`, `:436-499`).
|
||||
|
||||
That is **`trio.abc.Stream`, not `trio.SocketStream`**. The
|
||||
`MsgTransport` Protocol's `stream: trio.SocketStream`
|
||||
annotation (`_transport.py:83`) is a lie of convenience — the
|
||||
actual `MsgpackTransport.__init__` param is typed
|
||||
`trio.abc.Stream` and nothing in the msg path touches
|
||||
`.socket`. Only `connected()` (which each backend defines) and
|
||||
`Endpoint.start_listener()`'s `getsockname()` do.
|
||||
|
||||
### 1.5 verified-good news for socket-family backends
|
||||
|
||||
Both `trio.SocketStream` and `trio.SocketListener` are
|
||||
**address-family agnostic**. Verified against the installed
|
||||
`trio` (`trio/_highlevel_socket.py`): the only constructor
|
||||
checks are
|
||||
|
||||
- `isinstance(socket, trio.socket.SocketType)`
|
||||
- `socket.type == SOCK_STREAM`
|
||||
- (listener) `getsockopt(SOL_SOCKET, SO_ACCEPTCONN)` is truthy,
|
||||
with `OSError` **suppressed** (the macOS carve-out, which
|
||||
also covers exotic families that reject the opt)
|
||||
|
||||
There is no `AF_*` check and no `IPPROTO_TCP` hard dependency
|
||||
(`TCP_NODELAY`/`TCP_NOTSENT_LOWAT` are set under
|
||||
`suppress(OSError)`). Consequence: **any `SOCK_STREAM` family
|
||||
CPython can create — including `AF_TIPC` — drops straight into
|
||||
the existing `trio.SocketStream` + `trio.serve_listeners()`
|
||||
path.** This is why plan 01 is small and plan 02 is not.
|
||||
|
||||
---
|
||||
|
||||
## 2. Registration tables (the full wiring checklist)
|
||||
|
||||
Adding a backend touches these and only these:
|
||||
|
||||
1. `tractor/runtime/_state.py:46`
|
||||
`TransportProtocolKey = Literal['tcp', 'uds', ...]` — add the
|
||||
key. This `Literal` is the canonical set; `_testing/pytest.py`
|
||||
drives `--tpt-proto` validation off `_addr._address_types`,
|
||||
and the spawn-backend fixture already models the
|
||||
"drive-the-set-from-the-Literal" pattern
|
||||
(`pytest.py:870-880`) — do the same rather than hardcoding.
|
||||
2. `tractor/discovery/_addr.py:173` `_address_types: bidict` —
|
||||
`{'<key>': <Proto>Address}`. Note it is a **`bidict`**, so
|
||||
the mapping must stay 1:1.
|
||||
3. `tractor/discovery/_addr.py:181` `_default_lo_addrs` —
|
||||
`'<key>': <Proto>Address.get_root().unwrap()`.
|
||||
⚠️ this dict is built at **import time**, so
|
||||
`get_root()` must not require a live runtime, a loaded kernel
|
||||
module, or network I/O. (`UDSAddress.def_bindspace =
|
||||
get_rt_dir()` is the precedent for "cheap, pure, filesystem-
|
||||
ish".) A backend whose root addr needs I/O must make this
|
||||
entry lazy — propose that refactor explicitly.
|
||||
4. `tractor/discovery/_addr.py:230` `wrap_address()` `match` —
|
||||
add a case iff your `unwrapped_type` isn't already uniquely
|
||||
matched. **Preferably do the proto-key migration in §1.1
|
||||
first**, after which this step becomes a one-line
|
||||
`_address_types` entry instead of an order-sensitive `case`.
|
||||
5. `tractor/ipc/_types.py` — `Address` union alias,
|
||||
`_msg_transports` list, `_key_to_transport[('msgpack', key)]`,
|
||||
`_addr_to_transport[<Proto>Address]`.
|
||||
6. `tractor/ipc/_types.py:92` `transport_from_stream()` — the
|
||||
`sock.family` `match`. For a non-socket stream type (iroh)
|
||||
this needs a different discriminator; see plan 02 §3.3.
|
||||
7. `tractor/discovery/_multiaddr.py` —
|
||||
`_tpt_proto_to_maddr`, and a `case` in both `mk_maddr()` and
|
||||
`parse_maddr()`.
|
||||
8. `tractor/ipc/__init__.py` — re-export if the backend has a
|
||||
public surface.
|
||||
9. `tractor/_testing/addr.py::get_rando_addr()` — per-proto
|
||||
branch so the whole suite can run under `--tpt-proto <key>`.
|
||||
10. `pyproject.toml` — new deps go in an **optional extra**, never
|
||||
in `[project].dependencies`. See §5.
|
||||
|
||||
## 3. Where the `trio.SocketListener` assumption is load-bearing
|
||||
|
||||
`_serve_ipc_eps()` (`_server.py:1041`) annotates
|
||||
`listener: trio.abc.Listener` and hands the list to
|
||||
`trio.serve_listeners(handler=handle_stream_from_peer,
|
||||
listeners=..., handler_nursery=stream_handler_tn)`.
|
||||
`trio.serve_listeners` itself is generic over
|
||||
`trio.abc.Listener`. So the *only* `SocketListener`-specific
|
||||
code in the server path is the `getsockname()` reconciliation in
|
||||
`Endpoint.start_listener()` (§1.3) and the type annotations.
|
||||
|
||||
`handle_stream_from_peer()` (`_server.py:298`) then does
|
||||
`Channel.from_stream(stream)` →
|
||||
`transport_from_stream(stream)` → `sock.family` match (§2.6).
|
||||
|
||||
**Therefore**: a non-socket backend needs (a) a
|
||||
`trio.abc.Listener` subclass, (b) a change to
|
||||
`Endpoint.start_listener()` to not blindly `getsockname()`, and
|
||||
(c) a change to `transport_from_stream()`'s discrimination.
|
||||
All three are small, upstream-able, and *should be landed as
|
||||
their own prep PR* before the backend itself — see plan 02 §3.
|
||||
|
||||
## 4. Handshake / discovery invariants you inherit
|
||||
|
||||
- Every accepted stream immediately does
|
||||
`chan._do_handshake(aid=actor.aid)`; a peer that fails it is
|
||||
logged at `runtime` and dropped, **not** raised
|
||||
(`_server.py:334-365`). Discovery-sys "pings" rely on this,
|
||||
so your `connect_to()` must raise something that normalizes
|
||||
to `TransportClosed`/`ConnectionError` on a dead peer, never
|
||||
a novel exception type.
|
||||
- `_root.py:381-406` fail-fasts when a `registry_addrs` entry's
|
||||
`proto_key` is not in `enable_transports`. Your key must be
|
||||
spellable in both.
|
||||
- `_root.py:256` currently enforces `len(enable_transports) == 1`.
|
||||
Multi-tpt actors are a separate work item; none of these three
|
||||
plans may depend on lifting it.
|
||||
- Sub-actor bind addrs come from
|
||||
`_runtime.py:1600-1610`: for each key in the parent-supplied
|
||||
`enable_transports`, `get_address_cls(key).get_random()`.
|
||||
So `get_random()` runs *in the child, post-fork, pre-listen*.
|
||||
Anything it needs (kernel module, netns membership, an iroh
|
||||
secret key) must already be true at that moment.
|
||||
|
||||
## 5. Dependency policy
|
||||
|
||||
`[project].dependencies` stays lean (see the boot-latency work,
|
||||
gh #470: `import tractor` is budgeted at ~0.145s). Every new
|
||||
backend dep is an extra:
|
||||
|
||||
```toml
|
||||
[project.optional-dependencies]
|
||||
tipc = [] # stdlib-only!
|
||||
quic = ["iroh>=0.35"] # pin per plan 02 §1
|
||||
wg = ["pyroute2>=0.9"] # pin per plan 03 §1
|
||||
```
|
||||
|
||||
and every backend module must be **import-lazy**: a
|
||||
`tractor/ipc/_<proto>.py` that imports its 3rd-party dep at
|
||||
module scope must not be imported by `tractor/__init__.py`,
|
||||
`tractor/ipc/__init__.py`, or `tractor/discovery/_addr.py`'s
|
||||
import-time table construction. The `_addr._default_lo_addrs`
|
||||
eager-dict (§2.3) is the trap: keep the backend's `get_root()`
|
||||
dep-free, or make that table lazy.
|
||||
|
||||
## 6. Test-harness plumbing (identical for all three)
|
||||
|
||||
- `--tpt-proto <key>` (`_testing/pytest.py:409`) selects the
|
||||
session-wide proto; the `tpt_proto` fixture mutates
|
||||
`_state._def_tpt_proto` + `_runtime_vars['_enable_tpts']`
|
||||
(`pytest.py:807-835`). Adding the key to `_address_types` is
|
||||
what makes `--tpt-proto <key>` legal (`pytest.py:795-800`
|
||||
asserts the lookup).
|
||||
- The **acceptance bar** for every backend is: the *entire*
|
||||
existing suite passes under `--tpt-proto <key>`, unmodified.
|
||||
That is the whole point of the abstraction. Backend-specific
|
||||
unit tests go in `tests/ipc/test_each_tpt.py` (the existing
|
||||
`test_uds_bindspace_created_implicitly` /
|
||||
`test_uds_double_listen_raises_connerr` are the model).
|
||||
- Capability gating: each backend needs a **cheap, pure
|
||||
predicate** + a `pytest.mark.skipif`, because these are all
|
||||
environment-dependent. Verified example: on this dev box
|
||||
`socket.socket(AF_TIPC, SOCK_STREAM)` raises
|
||||
`OSError(97, 'Address family not supported by protocol')`
|
||||
because the `tipc` module isn't loaded. Put the predicate in
|
||||
the backend module (so apps can use it too), not in the test.
|
||||
- New pytest marks must be registered in `pyproject.toml`, per
|
||||
the project's fix-warnings-at-source rule (gh #469).
|
||||
|
||||
## 7. Code style (non-negotiable, matches the repo)
|
||||
|
||||
- module header tagline: `# tractor: distributed structured
|
||||
concurrency.` for **new** files (not the legacy
|
||||
`structured concurrent "actors".` form the existing `_tcp.py`
|
||||
carries).
|
||||
- AGPL header block copied verbatim from `_tcp.py`.
|
||||
- `from __future__ import annotations` first.
|
||||
- annotate *everything*, including locals:
|
||||
`sockpath: Path = addr.sockpath`.
|
||||
- `match`/`case` over `isinstance` chains for address and
|
||||
error dispatch.
|
||||
- multi-line call/`import` style with trailing commas.
|
||||
- never emit a whitespace-only line.
|
||||
- error messages are multi-line f-strings ending in `\n`, with
|
||||
the `f'...\n' f'...\n'` implicit-concat layout and the
|
||||
`>[`/`[>`/`<=(` nested-op sigils where a `nest_from_op()` is
|
||||
in play.
|
||||
- prefer pure functions + module-level helpers over methods;
|
||||
keep `Address` types data-only. Where a helper needs
|
||||
scoped setup/teardown, it's an `@acm` — not a class with
|
||||
`.start()`/`.stop()`.
|
||||
- pure getters: no `get_*(..., mutate=True)` flags; split into
|
||||
a read-only getter and an explicit sibling setter.
|
||||
|
||||
---
|
||||
|
||||
## 8. Cross-plan sequencing
|
||||
|
||||
The three are independent *except*:
|
||||
|
||||
- plan 02 (iroh) needs the `_server.py` /
|
||||
`transport_from_stream()` generalization (§3) — plan 01 does
|
||||
**not**, and should therefore land first as the cheap proof
|
||||
that the table-registration story works for a genuinely new
|
||||
proto.
|
||||
- plan 03 (wg) composes *under* whatever L4 tpt is in use and
|
||||
its netns work is what finally implements
|
||||
`Address.namespace`. It can land before or after 02, but its
|
||||
`TunnelledAddress` design must be reviewed against plan 02's
|
||||
address shape so the "tunnelled maddr" grammar (gh #443)
|
||||
covers `/…/quic-v1/p2p/…` inner addrs too.
|
||||
- All three want first-class `wg`/`quic`/`tipc` protos in
|
||||
`py-multiaddr`; that upstream track is gh #483 and
|
||||
multiformats/py-multiaddr#107/#108.
|
||||
|
|
@ -0,0 +1,707 @@
|
|||
# Plan 01 — `TIPC` transport backend (`tractor/ipc/_tipc.py`)
|
||||
|
||||
Tracks gh [#378]. Prereq reading:
|
||||
[`00_shared_backend_contract.md`](./00_shared_backend_contract.md).
|
||||
|
||||
**Thesis**: TIPC is the *cheapest* new backend we can add and
|
||||
simultaneously the only one that gives us cluster-wide service
|
||||
discovery **for free, in the kernel**, replacing (for
|
||||
TIPC-capable deployments) the whole `tractor.discovery`
|
||||
registrar round-trip with a `bind()`/`connect()` on a
|
||||
*service name*. It is stdlib-only: zero new dependencies.
|
||||
|
||||
[#378]: https://github.com/goodboy/tractor/issues/378
|
||||
|
||||
---
|
||||
|
||||
## 1. Why this is small: three verified facts
|
||||
|
||||
1. **CPython already speaks TIPC.** `socket.AF_TIPC` plus 23
|
||||
`TIPC_*` constants are present in the stdlib on Linux
|
||||
(verified on the dev box, py3.13):
|
||||
`AF_TIPC, SOL_TIPC, TIPC_ADDR_ID, TIPC_ADDR_NAME,
|
||||
TIPC_ADDR_NAMESEQ, TIPC_CFG_SRV, TIPC_CLUSTER_SCOPE,
|
||||
TIPC_CONN_TIMEOUT, TIPC_{CRITICAL,HIGH,MEDIUM,LOW}_IMPORTANCE,
|
||||
TIPC_DEST_DROPPABLE, TIPC_IMPORTANCE, TIPC_NODE_SCOPE,
|
||||
TIPC_PUBLISHED, TIPC_SRC_DROPPABLE, TIPC_SUBSCR_TIMEOUT,
|
||||
TIPC_SUB_CANCEL, TIPC_SUB_PORTS, TIPC_SUB_SERVICE,
|
||||
TIPC_TOP_SRV, TIPC_WAIT_FOREVER, TIPC_WITHDRAWN,
|
||||
TIPC_ZONE_SCOPE`.
|
||||
`sock.bind()/connect()/getsockname()` take/return the
|
||||
5-tuple `(addr_type, v1, v2, v3, scope)` — the last element
|
||||
is optional on input and defaults to `0`.
|
||||
2. **`trio` doesn't care about the address family.** Per
|
||||
contract §1.5, `trio.SocketStream` and `trio.SocketListener`
|
||||
only require a trio socket object of type `SOCK_STREAM`.
|
||||
TIPC's `SOCK_STREAM` is a real connection-oriented reliable
|
||||
byte stream. So we reuse `trio.SocketStream`,
|
||||
`trio.SocketListener`, `trio.serve_listeners()`,
|
||||
`MsgpackTransport`'s framing — *all of it*.
|
||||
3. **It is not available by default.** On this box
|
||||
`socket.socket(AF_TIPC, SOCK_STREAM)` →
|
||||
`OSError(97, 'Address family not supported by protocol')`
|
||||
with no `tipc` in `/proc/modules`. `modprobe tipc` is
|
||||
required; cross-node needs a bearer
|
||||
(`tipc bearer enable media eth device <if>` or
|
||||
`media udp name <n> localip <ip>`). Everything about this
|
||||
plan's testability hinges on gating (§7).
|
||||
|
||||
Non-goals: `SOCK_RDM`/`SOCK_DGRAM`/`SOCK_SEQPACKET` message
|
||||
modes, multicast fan-out, and TIPC group messaging. They are
|
||||
genuinely interesting for a future `tractor` broadcast/pubsub
|
||||
transport but they do **not** fit `MsgTransport`'s
|
||||
stream-of-length-prefixed-msgs shape. Note them in the
|
||||
follow-up issue, do not build them here.
|
||||
|
||||
---
|
||||
|
||||
## 2. `TIPCAddress`
|
||||
|
||||
### 2.1 the three TIPC address flavours, and which we use
|
||||
|
||||
| flavour | tuple | meaning |
|
||||
| --- | --- | --- |
|
||||
| `TIPC_ADDR_NAMESEQ` | `(type, lower, upper, scope)` | a *published range* — what a server `bind()`s |
|
||||
| `TIPC_ADDR_NAME` | `(type, instance, domain, scope)` | a *lookup* — what a client `connect()`s |
|
||||
| `TIPC_ADDR_ID` | `(node, ref, 0, scope)` | a concrete port id — the "physical" address |
|
||||
|
||||
The design decision that makes this backend coherent:
|
||||
|
||||
> **A `tractor` actor's TIPC address is a *service name*
|
||||
> `(type, instance)`; `bind()` publishes the singleton range
|
||||
> `(type, instance, instance)`; peers `connect()` by name and
|
||||
> the kernel resolves + load-balances. `TIPC_ADDR_ID` is only
|
||||
> ever an *observed* address (`getpeername()`), never a
|
||||
> user-facing one.**
|
||||
|
||||
This is exactly the "leverage the built-in discovery machinery"
|
||||
ask in #378: publishing a bind *is* registration, and
|
||||
`connect()` on a name *is* a lookup, with no registrar actor in
|
||||
the loop.
|
||||
|
||||
### 2.2 the struct
|
||||
|
||||
```python
|
||||
class TIPCAddress(
|
||||
msgspec.Struct,
|
||||
frozen=True,
|
||||
):
|
||||
_stype: int # TIPC "type" == service class
|
||||
_instance: int # service instance within the type
|
||||
_scope: int = TIPC_CLUSTER_SCOPE
|
||||
# observed-only, never part of identity/equality-by-intent
|
||||
maybe_node: int|None = None # from TIPC_ADDR_ID getpeername()
|
||||
maybe_ref: int|None = None
|
||||
|
||||
proto_key: ClassVar[str] = 'tipc'
|
||||
unwrapped_type: ClassVar[type] = tuple[str, int]
|
||||
def_bindspace: ClassVar[int] = TIPC_CLUSTER_SCOPE
|
||||
```
|
||||
|
||||
**Unwrapped form** (the wire/`SpawnSpec` shape).
|
||||
|
||||
TIPC's natural form is `(stype, instance, scope)` — but a
|
||||
2-tuple squeeze of it is a `(str, int)`, i.e. *the same coarse
|
||||
shape as `TCPAddress`*, so `wrap_address()`'s
|
||||
`case (str(), int())` steals it. This backend is therefore the
|
||||
forcing function for the contract-doc's conclusion (§1.1):
|
||||
|
||||
> **make the unwrapped form carry an explicit proto-key, spelled
|
||||
> with the `multiaddr` protocol name.**
|
||||
|
||||
```python
|
||||
def unwrap(self) -> tuple[str, int, int, int]:
|
||||
return ('tipc', self._stype, self._instance, self._scope)
|
||||
```
|
||||
|
||||
`wrap_address()` then dispatches `_address_types[addr[0]]` and
|
||||
the collision class disappears. **This is a prerequisite
|
||||
migration commit, not part of this backend** — see contract §1.1
|
||||
for its blast radius (wire format + every fixture + `piker`
|
||||
config) and for the follow-on "stop handing raw tuples to users
|
||||
at all, à la `ipaddress`" direction.
|
||||
|
||||
⚠️ an earlier revision of this plan proposed a self-tagging
|
||||
`('tipc:<stype>:<scope>', instance)` string-prefix hack with an
|
||||
ordered `case` guard. **Dropped** — it papers over the problem,
|
||||
keeps `wrap_address()` order-sensitive, and doesn't help iroh's
|
||||
`(str, str)`-vs-UDS collision at all. Do not resurrect it.
|
||||
|
||||
Note `TIPCAddress` is the first backend where `.unwrap()` is
|
||||
**not** a lossless view of the live socket — `maybe_node`/
|
||||
`maybe_ref` are observed metadata, exactly like
|
||||
`UDSAddress.maybe_pid` (which is likewise excluded from
|
||||
`.unwrap()`). Follow that precedent, including its `__repr__`
|
||||
treatment (`_uds.py:242`).
|
||||
|
||||
### 2.3 how to pick `_stype` and `_instance`
|
||||
|
||||
- `_stype` = a `tractor`-reserved service class. TIPC reserves
|
||||
0..63 for internal use (`TIPC_TOP_SRV == 1`,
|
||||
`TIPC_CFG_SRV == 0`). Use a module constant
|
||||
`TRACTOR_STYPE: int = 0x74_72_00_00` ("tr\0\0") as the default
|
||||
and make it overridable via `TIPCAddress._stype` so an app
|
||||
can partition service classes. Document that two `tractor`
|
||||
trees sharing a cluster **and** a `_stype` share a namespace.
|
||||
- `_instance` for `get_root()`: `1616` — mirrors the
|
||||
`TCPAddress.get_root()` port and the `registry@1616.sock`
|
||||
UDS filename, so the "1616 is tractor's registrar" idiom
|
||||
holds across all backends.
|
||||
- `_instance` for `get_random()`: TIPC gives us no
|
||||
kernel-assigned-instance analogue of `port=0`, so we must
|
||||
choose. Use a *pure* fn of the actor identity so it is
|
||||
reproducible and collision-free:
|
||||
```python
|
||||
# 32-bit instance derived from the actor's uuid4 (+ pid when
|
||||
# there's no live runtime, per the UDS precedent).
|
||||
inst: int = int.from_bytes(
|
||||
blake2b(seed.encode(), digest_size=4).digest(),
|
||||
'big',
|
||||
)
|
||||
```
|
||||
where `seed = f'{actor.aid.name}@{pid}'` if
|
||||
`current_actor(err_on_no_runtime=False)` else
|
||||
`f'{prefix}.{uuid4().hex[:8]}@{pid}'`. Must avoid the reserved
|
||||
low range: `inst = 64 + (inst % (2**32 - 64))`.
|
||||
⚠️ *unlike* `port=0`, a collision here surfaces as a
|
||||
successful-but-shared publication (TIPC allows multiple
|
||||
binders on the same name and round-robins!) rather than
|
||||
`EADDRINUSE`. That is a silent-crosstalk failure mode; §7 has
|
||||
the test that proves the 4-byte digest is enough and §9 has
|
||||
the mitigation if it isn't.
|
||||
- `_scope`: `TIPC_NODE_SCOPE` for a same-host-only actor (the
|
||||
UDS-equivalent), `TIPC_CLUSTER_SCOPE` (default) for
|
||||
cluster-visible. **This is `.bindspace`**:
|
||||
```python
|
||||
@property
|
||||
def bindspace(self) -> int:
|
||||
return self._scope
|
||||
```
|
||||
It is the honest analogue of "the set of hosts this bind is
|
||||
reachable from", which is precisely the docstring in
|
||||
`Address.bindspace`. (`TIPC_ZONE_SCOPE` is deprecated/aliased
|
||||
to cluster in modern kernels — accept it on input, normalize
|
||||
to cluster, log at `transport` level.)
|
||||
|
||||
### 2.4 `is_valid`
|
||||
|
||||
```python
|
||||
@property
|
||||
def is_valid(self) -> bool:
|
||||
return (
|
||||
self._instance != 0
|
||||
and
|
||||
self._stype not in _tipc_reserved_stypes # {0, 1, ...}
|
||||
and
|
||||
self._scope in (TIPC_NODE_SCOPE, TIPC_CLUSTER_SCOPE)
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Listener + stream
|
||||
|
||||
### 3.1 `start_listener()`
|
||||
|
||||
```python
|
||||
async def start_listener(
|
||||
addr: TIPCAddress,
|
||||
backlog: int = 128,
|
||||
**kwargs,
|
||||
) -> SocketListener:
|
||||
sock = trio.socket.socket(
|
||||
socket.AF_TIPC,
|
||||
socket.SOCK_STREAM,
|
||||
)
|
||||
# publish the singleton name-range == "register the service"
|
||||
await sock.bind((
|
||||
socket.TIPC_ADDR_NAMESEQ,
|
||||
addr._stype,
|
||||
addr._instance,
|
||||
addr._instance,
|
||||
addr._scope,
|
||||
))
|
||||
sock.listen(backlog)
|
||||
return SocketListener(sock)
|
||||
```
|
||||
|
||||
Notes / hazards:
|
||||
|
||||
- `bind()` on `AF_TIPC` is **not** a filesystem or port-table
|
||||
operation and can't block on DNS, but keep it `await`ed
|
||||
through `trio.socket` anyway for uniformity.
|
||||
- `backlog=128` matching `_uds.start_listener()`'s hard-won
|
||||
value (see its comment at `_uds.py:317-331` re: concurrent
|
||||
deregistration storms). Do not use `1`.
|
||||
- **no `close_listener()` needed** — nothing to unlink. Omit the
|
||||
function entirely (contract §1.2: absence means implicit).
|
||||
Withdrawal of the published name happens on socket close.
|
||||
- ⚠️ `SocketListener.__init__` will try
|
||||
`getsockopt(SOL_SOCKET, SO_ACCEPTCONN)`. If TIPC rejects it,
|
||||
trio's `except OSError: pass` covers us. Assert this in a
|
||||
unit test rather than assuming.
|
||||
- Wrap the bind in a `_reraise_as_connerr()`-style `@cm` (copy
|
||||
the `_uds.py:256` pattern) so `EADDRINUSE`-ish and
|
||||
`EAFNOSUPPORT` become `ConnectionError` with the addr in the
|
||||
message. `EAFNOSUPPORT` here means "kernel module not
|
||||
loaded" and deserves a *specifically actionable* message:
|
||||
`'TIPC unavailable — try `sudo modprobe tipc`\n'`.
|
||||
|
||||
### 3.2 the `getsockname()` reconciliation
|
||||
|
||||
`Endpoint.start_listener()` does
|
||||
`if lstnr.socket.getsockname() != self.addr.unwrap(): self.addr =
|
||||
self.addr.from_addr(unwrapped)`.
|
||||
|
||||
For TIPC, `getsockname()` on a bound-but-listening socket
|
||||
returns a `TIPC_ADDR_ID`-flavoured 5-tuple (the port id), *not*
|
||||
the name-seq we bound. So the `!=` is **always true** and
|
||||
`from_addr()` will be handed a 5-tuple.
|
||||
|
||||
Handle it inside `TIPCAddress.from_addr()` — do **not** patch
|
||||
`_server.py`:
|
||||
|
||||
```python
|
||||
@classmethod
|
||||
def from_addr(cls, addr) -> TIPCAddress:
|
||||
match addr:
|
||||
# our own unwrapped form
|
||||
case (str() as tag, int() as inst) if tag.startswith('tipc:'):
|
||||
_, stype, scope = tag.split(':')
|
||||
return TIPCAddress(int(stype), inst, int(scope))
|
||||
|
||||
# a kernel-observed TIPC_ADDR_ID 5-tuple: keep the
|
||||
# *service* identity we already know and only annotate
|
||||
# the observed port-id.
|
||||
case (int() as atype, *rest) if atype == socket.TIPC_ADDR_ID:
|
||||
...
|
||||
```
|
||||
|
||||
The `TIPC_ADDR_ID` case cannot reconstruct `(stype, instance)`
|
||||
— that info isn't in a port id. So `from_addr()` alone is
|
||||
insufficient for the reconciliation path. **Resolution**: make
|
||||
`from_addr()` raise a clear `ValueError` for the bare
|
||||
`TIPC_ADDR_ID` case, and instead prevent the reconciliation
|
||||
from firing by having `start_listener()` return a listener
|
||||
whose `getsockname()` we never need — i.e. land this two-line
|
||||
upstream fix in `_server.py:664`:
|
||||
|
||||
```python
|
||||
if (
|
||||
(unwrapped := lstnr.socket.getsockname()) != self.addr.unwrap()
|
||||
and
|
||||
self.addr.rebind_from_sockname # ClassVar[bool] = True on tcp/uds
|
||||
):
|
||||
```
|
||||
|
||||
with `TIPCAddress.rebind_from_sockname: ClassVar[bool] = False`
|
||||
(and `True` on `TCPAddress`/`UDSAddress`, preserving today's
|
||||
behaviour exactly). Rationale: the reconciliation exists *only*
|
||||
to learn the kernel-assigned port for `port=0` TCP binds (its
|
||||
own comment says so, `_server.py:662`); TIPC has no such
|
||||
late-binding, so opting out is semantically right rather than a
|
||||
hack. **Land this as its own commit, ahead of the backend**,
|
||||
with a test that `tcp`'s `port=0` behaviour is unchanged.
|
||||
|
||||
Keep the observed port-id available anyway: annotate
|
||||
`ep.addr = ep.addr.with_port_id(*getsockname()[1:3])` (a pure
|
||||
`msgspec.structs.replace()` helper) purely for logging/repr.
|
||||
|
||||
### 3.3 `MsgpackTIPCStream`
|
||||
|
||||
```python
|
||||
class MsgpackTIPCStream(MsgpackTransport):
|
||||
address_type = TIPCAddress
|
||||
layer_key: int = 4
|
||||
|
||||
@property
|
||||
def maddr(self) -> Multiaddr|str:
|
||||
return mk_maddr(self.raddr)
|
||||
|
||||
def connected(self) -> bool:
|
||||
return self.stream.socket.fileno() != -1
|
||||
|
||||
@classmethod
|
||||
async def connect_to(
|
||||
cls,
|
||||
destaddr: TIPCAddress,
|
||||
prefix_size: int = 4,
|
||||
codec: MsgCodec|None = None,
|
||||
**kwargs,
|
||||
) -> MsgpackTIPCStream:
|
||||
sock = trio.socket.socket(AF_TIPC, SOCK_STREAM)
|
||||
with close_on_error(sock):
|
||||
# NOTE: connect by *name* -> kernel does the lookup,
|
||||
# so this is our "discovery" call.
|
||||
await sock.connect((
|
||||
socket.TIPC_ADDR_NAME,
|
||||
destaddr._stype,
|
||||
destaddr._instance,
|
||||
0, # domain: 0 == "anywhere in scope"
|
||||
destaddr._scope,
|
||||
))
|
||||
return cls(
|
||||
trio.SocketStream(sock),
|
||||
prefix_size=prefix_size,
|
||||
codec=codec,
|
||||
)
|
||||
```
|
||||
|
||||
- reuse `trio._highlevel_open_unix_stream.close_on_error` (the
|
||||
UDS backend already imports it) or inline the equivalent
|
||||
`try/except: sock.close(); raise`.
|
||||
- `SO_/TIPC_` opts worth setting and documenting:
|
||||
- `setsockopt(SOL_TIPC, TIPC_IMPORTANCE, TIPC_HIGH_IMPORTANCE)`
|
||||
for the *parent<->child* lifetime channel — this is a real
|
||||
win TIPC gives us that TCP can't: the runtime's
|
||||
supervision channel can outrank bulk app traffic under
|
||||
congestion. Wire it as a `connect_to(..., importance=...)`
|
||||
kwarg defaulted from a module constant, and have
|
||||
`_runtime.py`'s parent-chan path pass the high value **in a
|
||||
follow-up** (don't couple it to this PR).
|
||||
- `TIPC_CONN_TIMEOUT` — the kernel-side connect timeout;
|
||||
leave at default, we have `trio` cancel scopes.
|
||||
- `TIPC_DEST_DROPPABLE = 0` on the connection so undeliverable
|
||||
msgs come back as errors rather than being silently dropped.
|
||||
- **`connect_to()` on a name with no publisher**: TIPC returns
|
||||
`ECONNREFUSED`/`EHOSTUNREACH` promptly (no SYN-timeout wait),
|
||||
which is *better* discovery-ping behaviour than TCP. Confirm
|
||||
the errno and make sure it surfaces as `ConnectionError`
|
||||
(contract §4 — the registrar ping path depends on it).
|
||||
|
||||
### 3.4 `get_stream_addrs()`
|
||||
|
||||
```python
|
||||
@classmethod
|
||||
def get_stream_addrs(cls, stream) -> tuple[TIPCAddress, TIPCAddress]:
|
||||
sock = stream.socket
|
||||
# both return TIPC_ADDR_ID 5-tuples for a connected sock
|
||||
l_id = sock.getsockname()
|
||||
r_id = sock.getpeername()
|
||||
...
|
||||
```
|
||||
|
||||
Problem: neither end's port-id tells us the *service name*. The
|
||||
`laddr`/`raddr` are used for logging, `Channel.raddr`,
|
||||
`Server._peers` keying-adjacent repr, and `maddr`. Design:
|
||||
|
||||
- the **connecting** side knows the destaddr it dialled →
|
||||
`connect_to()` overrides `_raddr` after construction with the
|
||||
known-good `TIPCAddress`, exactly as
|
||||
`MsgpackUDSStream.connect_to()` does for the peer-pid case
|
||||
(`_uds.py:539-543`).
|
||||
- the **accepting** side does not know the peer's service name
|
||||
from the socket. Two honest options:
|
||||
- **(a) accept it: `raddr` carries only `(node, ref)`** via
|
||||
`maybe_node`/`maybe_ref`, `_stype/_instance` set to a
|
||||
sentinel `-1`, and `__repr__` renders
|
||||
`TIPCAddress[<peer-node:0x...>:<ref>]`. The `Aid` from the
|
||||
handshake already gives us the peer's logical identity, so
|
||||
nothing in the runtime actually *needs* the peer's service
|
||||
name. **Recommended.**
|
||||
- (b) piggyback the peer's own bound name in the handshake.
|
||||
Rejected for this PR: touches `Aid`/msg-spec.
|
||||
- `laddr` on the accepting side: the `Endpoint` knows its own
|
||||
`addr`; but `get_stream_addrs()` is a `@classmethod` with only
|
||||
the stream. Use `TIPC_ADDR_ID` for `laddr` too and let
|
||||
`Endpoint.peer_tpts` keying (which is by *peer* addr) still
|
||||
work. Verify nothing asserts `laddr == ep.addr` — grep for
|
||||
`.laddr` uses before committing (`_server.py`'s
|
||||
`con_status` logging, `Channel.pformat()`).
|
||||
|
||||
---
|
||||
|
||||
## 4. Multiaddr representation
|
||||
|
||||
There is no `/tipc` in the multiaddr protocol table. Interim
|
||||
grammar, mirroring how `uds` maps to the spec-legal `/unix`:
|
||||
|
||||
```
|
||||
/tipc/<stype>/<instance> # scope implied = cluster
|
||||
/tipc/<stype>/<instance>/<scope> # explicit
|
||||
```
|
||||
|
||||
- `_tpt_proto_to_maddr['tipc'] = 'tipc'` and a `mk_maddr()`
|
||||
`case 'tipc':` building the above.
|
||||
- `parse_maddr()` gets `case ['tipc']:` — but note
|
||||
`py-multiaddr` will reject an unregistered protocol name
|
||||
outright, so this **requires an upstream registration** (same
|
||||
track as the `wg` work, gh #483 /
|
||||
multiformats/py-multiaddr#107). Until that lands:
|
||||
- `MsgpackTIPCStream.maddr` returns the **`str`** form (the
|
||||
`MsgTransport.maddr` return type is already
|
||||
`Multiaddr|str`, and `MsgpackUDSStream.maddr` already
|
||||
exercises the `str` branch), and
|
||||
- `parse_maddr()` special-cases the `/tipc/` prefix *before*
|
||||
handing the string to `Multiaddr()`.
|
||||
Document this as the reason gh #443's "standardize on
|
||||
returning `Multiaddr` everywhere" item stays blocked.
|
||||
|
||||
Propose `/tipc/` upstream as: name `tipc`, code TBD, size
|
||||
variable, value `<stype>:<instance>:<scope>` — or as three
|
||||
composed protos. Prefer *one* proto with a structured value so
|
||||
the maddr stays 2-segment like `/unix/...`.
|
||||
|
||||
---
|
||||
|
||||
## 5. Discovery: the actually-interesting part
|
||||
|
||||
Two independently-shippable layers. **Layer A is in scope for
|
||||
the first PR; layer B is a fast-follow.**
|
||||
|
||||
### 5.1 Layer A — "discovery by bind" (free)
|
||||
|
||||
Because `bind(TIPC_ADDR_NAMESEQ)` publishes and
|
||||
`connect(TIPC_ADDR_NAME)` resolves, a `tractor` tree whose
|
||||
`registry_addrs` are TIPC service names needs **no registrar
|
||||
liveness at all** for the connect path: `find_actor()`'s
|
||||
"connect to the registrar and ask" becomes "connect to the
|
||||
service name directly". Concretely:
|
||||
|
||||
- `tractor.discovery._api.find_actor()` etc. keep working
|
||||
unchanged (they go through the registrar), *and*
|
||||
- a new, TIPC-only fast path becomes possible: derive an actor's
|
||||
service name from its `(name, uuid)` and dial it without any
|
||||
registrar hop.
|
||||
|
||||
Do **not** build the fast path in PR 1. Instead, prove the
|
||||
property with a test (§7.4) and file the follow-up: it changes
|
||||
`discovery` semantics (name→instance derivation must be a
|
||||
documented, stable, cross-language-able hash) and deserves its
|
||||
own design.
|
||||
|
||||
### 5.2 Layer B — the topology service (`TIPC_TOP_SRV`)
|
||||
|
||||
This is what makes #378's "end game cluster proto" claim real:
|
||||
a *subscription* to name-table events, i.e. push-based
|
||||
`register`/`deregister` for free, replacing the registrar's
|
||||
polled `find_actor()`.
|
||||
|
||||
Mechanics (verify each field against
|
||||
`linux/include/uapi/linux/tipc.h` + `net/tipc/topsrv.c` at
|
||||
implementation time — the struct layout below is from the uapi
|
||||
header and the byte-order caveat is real):
|
||||
|
||||
```python
|
||||
# SOCK_SEQPACKET connected to the topology server
|
||||
sock = trio.socket.socket(AF_TIPC, SOCK_SEQPACKET)
|
||||
await sock.connect((
|
||||
socket.TIPC_ADDR_NAME,
|
||||
socket.TIPC_TOP_SRV, # == 1
|
||||
socket.TIPC_TOP_SRV,
|
||||
0,
|
||||
))
|
||||
|
||||
# struct tipc_subscr {
|
||||
# struct tipc_name_seq seq; /* 3 * __u32: type, lower, upper */
|
||||
# __u32 timeout; /* TIPC_WAIT_FOREVER == ~0 */
|
||||
# __u32 filter; /* TIPC_SUB_{PORTS,SERVICE,CANCEL} */
|
||||
# char usr_handle[8];
|
||||
# } /* == 28 bytes */
|
||||
_SUBSCR_FMT: str = '=IIIII8s' # ⚠ 5*I is 20 -> use '=5I8s'
|
||||
```
|
||||
|
||||
- **byte order**: the topology server historically accepts both
|
||||
host and swapped order and auto-detects; modern kernels are
|
||||
strict-ish. Pack native (`'='`) first, and if the server
|
||||
closes the connection immediately, retry with `'>'`. Encode
|
||||
that as a one-time probe helper
|
||||
`_detect_topsrv_endianness()` cached at module level — and
|
||||
put a `# ?TODO` pointing at `net/tipc/topsrv.c` for someone
|
||||
to make it deterministic.
|
||||
- **events**: `struct tipc_event` is `event: u32`,
|
||||
`found_lower: u32`, `found_upper: u32`,
|
||||
`port: {ref: u32, node: u32}`, then the 28-byte subscription
|
||||
echo → 40 bytes. `event ∈ {TIPC_PUBLISHED, TIPC_WITHDRAWN,
|
||||
TIPC_SUBSCR_TIMEOUT}`.
|
||||
- **trio shape** — this is where the "nearly-functional,
|
||||
modern-async" style pays off; expose it as an `@acm` yielding
|
||||
a `trio` receive-channel of typed events, *not* a class:
|
||||
|
||||
```python
|
||||
@acm
|
||||
async def open_topology_events(
|
||||
stype: int = TRACTOR_STYPE,
|
||||
lower: int = 0,
|
||||
upper: int = 0xFFFFFFFF,
|
||||
filter: int = TIPC_SUB_SERVICE,
|
||||
timeout: int = TIPC_WAIT_FOREVER,
|
||||
buf_size: int = 64,
|
||||
) -> AsyncGenerator[
|
||||
trio.MemoryReceiveChannel[TIPCNameEvent],
|
||||
None,
|
||||
]:
|
||||
...
|
||||
```
|
||||
|
||||
with `TIPCNameEvent(msgspec.Struct, frozen=True)` fields
|
||||
`kind: Literal['published','withdrawn','timeout']`,
|
||||
`addr: TIPCAddress`, `node: int`, `ref: int`. One
|
||||
`trio.lowlevel`-free implementation: a nursery-spawned reader
|
||||
task doing `await sock.recv(40)` in a loop and
|
||||
`send_nowait()`ing decoded events, with the `@acm` closing the
|
||||
socket on exit → reader gets `ClosedResourceError` → cancel
|
||||
scope collapses. Standard `tractor` `@acm` discipline.
|
||||
- **consumer**: `tractor/discovery/_registry.py` gains an
|
||||
optional "watch" mode so a registrar (or any actor) can keep
|
||||
a live view of the actor set without polling. Sketch the
|
||||
integration in the follow-up issue; do not wire it in PR 1.
|
||||
- **`SOCK_SEQPACKET` is fine here** because this socket never
|
||||
goes through `MsgpackTransport` — it's a plain trio socket
|
||||
used with `recv()`. The contract's "`SOCK_STREAM` only"
|
||||
constraint applies to `MsgTransport` streams, not to this.
|
||||
|
||||
---
|
||||
|
||||
## 6. Commit sequencing (each independently reviewable + green)
|
||||
|
||||
1. `_server.py`: add `Address.rebind_from_sockname:
|
||||
ClassVar[bool]`, gate the `getsockname()` reconciliation on
|
||||
it, `True` for tcp/uds. Test: tcp `port=0` unchanged.
|
||||
2. `tractor/ipc/_tipc.py`: `TIPCAddress` + `is_tipc_available()`
|
||||
predicate + `start_listener()`. No transport yet.
|
||||
Tests: address round-trip (`unwrap`/`from_addr`/`wrap_address`),
|
||||
`get_random()` uniqueness, bind/listen + `SO_ACCEPTCONN`
|
||||
tolerance, `EAFNOSUPPORT` → actionable `ConnectionError`.
|
||||
3. `MsgpackTIPCStream` + `connect_to()` + `get_stream_addrs()`.
|
||||
Test: two `trio` tasks in one proc exchange a msg over
|
||||
`Msgpack` framing (no `tractor` runtime).
|
||||
4. registration tables (contract §2 items 1-6, 9) +
|
||||
`pyproject.toml` mark/extra. Test: full suite under
|
||||
`--tpt-proto tipc` (§7.3).
|
||||
5. maddr support (`str` form + prefix special-case) + docs.
|
||||
6. `open_topology_events()` @acm + its tests (layer B).
|
||||
7. docs page + `docs/` example.
|
||||
|
||||
Per project convention, a reproducing/guard test lands in its
|
||||
own commit **before** the fix it guards.
|
||||
|
||||
---
|
||||
|
||||
## 7. Testing
|
||||
|
||||
### 7.1 the capability predicate (in `_tipc.py`, public)
|
||||
|
||||
```python
|
||||
def is_tipc_available() -> bool:
|
||||
'''
|
||||
True iff this kernel can create an `AF_TIPC` socket, i.e.
|
||||
the `tipc` module is loaded.
|
||||
|
||||
'''
|
||||
try:
|
||||
socket.socket(socket.AF_TIPC, socket.SOCK_STREAM).close()
|
||||
return True
|
||||
except OSError:
|
||||
return False
|
||||
```
|
||||
|
||||
Cache it in a module global (it can't change without a
|
||||
`modprobe`, and a cold call costs a syscall). Pure predicate, no
|
||||
side effects, no logging.
|
||||
|
||||
### 7.2 gating
|
||||
|
||||
- `pytest.mark.tipc` registered in `pyproject.toml`.
|
||||
- module-level
|
||||
`pytestmark = pytest.mark.skipif(not is_tipc_available(),
|
||||
reason='`tipc` kernel module not loaded (`modprobe tipc`)')`
|
||||
in `tests/ipc/test_tipc.py`.
|
||||
- `--tpt-proto tipc` with no module must fail **loudly and
|
||||
early** with the actionable message, not with 400 confusing
|
||||
timeouts. Add the check to the `tpt_protos` fixture's existing
|
||||
per-proto validation loop (`_testing/pytest.py:795`): if the
|
||||
chosen `Address` type exposes an `is_available()`-style
|
||||
classmethod, call it and `pytest.fail()` with its reason.
|
||||
Generalize (don't special-case tipc) — plans 02/03 need the
|
||||
same hook.
|
||||
|
||||
### 7.3 CI
|
||||
|
||||
- add a job matrix entry `--tpt-proto tipc` that runs
|
||||
`sudo modprobe tipc` in a `before` step. GH's
|
||||
`ubuntu-latest` runners do allow `modprobe tipc` (the module
|
||||
ships with the standard Ubuntu kernel package); verify in a
|
||||
throwaway workflow before wiring the matrix. If it turns out
|
||||
to be unavailable, fall back to a container job with
|
||||
`--privileged`/`--cap-add NET_ADMIN`, and mark the job
|
||||
`continue-on-error` until it's proven stable.
|
||||
- cross-node TIPC (bearer) cannot be CI'd; cover it with a
|
||||
documented manual smoke test in the docs page, in the style
|
||||
of gh #482's LAN examples.
|
||||
|
||||
### 7.4 backend-specific tests worth writing
|
||||
|
||||
- **name-publication is discovery**: bind a listener on
|
||||
`(stype, inst)`, then from a second task `connect()` by name
|
||||
and assert it lands — *without* any `tractor` registrar.
|
||||
- **`get_random()` collision resistance**: 10k `get_random()`
|
||||
calls with no live runtime → 10k distinct `_instance`s.
|
||||
(This is the silent-crosstalk risk from §2.3; if the 4-byte
|
||||
digest ever collides in this test, escalate to §9.)
|
||||
- **round-robin surprise**: two listeners bound to the *same*
|
||||
`(stype, inst)` both succeed (TIPC allows it) and connects
|
||||
distribute. Assert the observed behaviour and reference it
|
||||
from the `get_random()` docstring so the next reader knows
|
||||
why the hash matters.
|
||||
- **scope isolation**: a `TIPC_NODE_SCOPE` bind is not visible
|
||||
to a cluster-scope lookup from another node (manual/marked).
|
||||
- **importance opt** round-trips via `getsockopt`.
|
||||
- **graceful + abrupt close** produce `TransportClosed` with the
|
||||
same `loglevel` classification as tcp/uds — i.e. re-run the
|
||||
relevant `tests/ipc/test_each_tpt.py` cases parametrized over
|
||||
the new proto rather than writing new ones.
|
||||
|
||||
---
|
||||
|
||||
## 8. Deployment / docs deliverable
|
||||
|
||||
A `docs/` page (and/or an `examples/` script) covering:
|
||||
|
||||
```bash
|
||||
# single host, node-scope only
|
||||
sudo modprobe tipc
|
||||
tipc node get addr
|
||||
|
||||
# multi-host over ethernet (pairs beautifully with plan 03's wg)
|
||||
sudo tipc bearer enable media eth device eth0
|
||||
# ...or over UDP when L2 isn't available:
|
||||
sudo tipc bearer enable media udp name uc localip 10.0.11.1
|
||||
tipc link list
|
||||
tipc nametable show # <- see tractor's published services!
|
||||
```
|
||||
|
||||
`tipc nametable show` displaying live `tractor` actors is the
|
||||
single best demo this backend has; lead with it.
|
||||
|
||||
---
|
||||
|
||||
## 9. Known risks + escalations
|
||||
|
||||
| risk | mitigation |
|
||||
| --- | --- |
|
||||
| `_instance` hash collision → silent crosstalk (two actors share a service name, TIPC round-robins connects between them) | §7.4 test; if it bites, add a post-bind verification handshake, or bump to a 6-byte digest folded into `(stype_low, instance)` |
|
||||
| kernel/module unavailability everywhere (dev boxes, macOS, CI) | hard gating (§7.2); TIPC is explicitly an *opt-in cluster* transport, never a default |
|
||||
| `getsockname()` returns port-id not name | the `rebind_from_sockname` opt-out (§3.2), landed first |
|
||||
| unregistered `/tipc` multiaddr proto | `str` maddr fallback (§4) + upstream track gh #483 |
|
||||
| stale docs (#378 notes tipc.io docs may be out of date) | treat `include/uapi/linux/tipc.h` + `net/tipc/` as the only normative source; cite file+symbol in code comments |
|
||||
| `SOCK_SEQPACKET` topology framing byte-order | probe helper + `?TODO` (§5.2) |
|
||||
|
||||
## 10. Follow-up issue seeds
|
||||
|
||||
- **register `/tipc` in the multiaddr spec**, mirroring the `wg`
|
||||
track (multiformats/py-multiaddr#107/#108 + gh #483). Same
|
||||
shape of work: propose the proto + code, land a codec in
|
||||
`py-multiaddr`, then drop our `str`-maddr fallback (§4). Worth
|
||||
filing *alongside* the `wg` spec-submission issue so both
|
||||
proposals go up together rather than as one-offs.
|
||||
- registrar-less discovery fast path via name derivation (§5.1)
|
||||
- `TIPC_TOP_SRV`-driven push registry in
|
||||
`discovery/_registry.py` (§5.2)
|
||||
- `TIPC_IMPORTANCE` for the parent<->child lifetime channel
|
||||
(§3.3) — genuinely novel supervision QoS, no other backend
|
||||
can do it
|
||||
- TIPC multicast / group messaging as a *broadcast* transport
|
||||
for `tractor.trionics` fan-out (explicitly not `MsgTransport`)
|
||||
- dual-link resiliency / multi-homing (#378's "hybrid dual link")
|
||||
once bearers are scripted in the docs
|
||||
|
|
@ -0,0 +1,566 @@
|
|||
# Plan 02 — QUIC backend via `iroh` FFI, uniffi-async rewritten onto `trio`
|
||||
|
||||
Tracks gh [#353]. Prereq reading:
|
||||
[`00_shared_backend_contract.md`](./00_shared_backend_contract.md).
|
||||
|
||||
**Thesis**: the value of `iroh` over "just QUIC" is
|
||||
`NodeId`-addressed, NAT-traversing, relay-fallback endpoints —
|
||||
i.e. a `tractor` actor tree that spans hosts *without* a
|
||||
reachable listening socket. The cost is that `iroh`'s python
|
||||
surface is `uniffi`-generated **asyncio** and its listener is not
|
||||
a socket. This plan spends its complexity budget in exactly two
|
||||
places: a `trio`-native uniffi future bridge, and a
|
||||
`trio.abc.Listener`/`Stream` adapter pair. Everything else is
|
||||
contract boilerplate.
|
||||
|
||||
[#353]: https://github.com/goodboy/tractor/issues/353
|
||||
|
||||
---
|
||||
|
||||
## 1. Library selection (decided, with the rejected alternatives)
|
||||
|
||||
**Chosen: `iroh` (PyPI, from `n0-computer/iroh-ffi`), pinned to
|
||||
a single minor.** The `iroh` python package is a `uniffi`
|
||||
binding over the rust `iroh` crate (QUIC via `quinn`/`noq`).
|
||||
|
||||
Rejected, and why — record these so the next implementer doesn't
|
||||
relitigate:
|
||||
|
||||
- **`aioquic`** (sans-io + asyncio): genuinely trio-portable
|
||||
(`hypercorn` already pairs its sans-io core with a trio UDP
|
||||
server, see the links in #353) and dependency-light. But it
|
||||
gives us *only* QUIC — no NodeId identity, no hole punching,
|
||||
no relay. We'd be reimplementing iroh's whole reason for
|
||||
existing. **Keep as the documented fallback** if the FFI
|
||||
bridge (§2) proves unmaintainable; the `MsgTransport` and
|
||||
`Listener` adapters from §3 are ~90% reusable against an
|
||||
`aioquic` core, which is a deliberate design property of this
|
||||
plan.
|
||||
- **`quiche` / `quinn` via a hand-rolled PyO3 ext**: strictly
|
||||
more work than reusing `iroh-ffi`, and puts us in the
|
||||
build-wheels business.
|
||||
- **`trio-asyncio`**: viable *shortcut* to run the asyncio-shaped
|
||||
bindings under trio, and `tractor` already ships
|
||||
infected-asyncio machinery (`tractor.to_asyncio`,
|
||||
`tests/test_infected_asyncio.py`). Rejected as the *primary*
|
||||
design because it makes every IPC send/recv cross a
|
||||
loop-boundary shim in the hot path, and because #353 asks
|
||||
explicitly for the asyncio support to be "rewritten for trio".
|
||||
**But**: build it first as the throwaway spike (§6 step 0) to
|
||||
de-risk the iroh API surface before writing the bridge.
|
||||
|
||||
Version pinning: `iroh` moves fast and has had breaking
|
||||
API renames across minors. Pin `iroh>=X.Y,<X.Y+1` in a `quic`
|
||||
extra, and **write down the exact resolved version + the
|
||||
generated `iroh/_uniffi*` module layout** in the module
|
||||
docstring, because §2 depends on generated-code internals.
|
||||
|
||||
**Step 0 of implementation is an API-truth pass**: install the
|
||||
pinned `iroh`, `python -c "import iroh; help(iroh)"`, and record
|
||||
in this doc's §1.1 the real names of: endpoint builder, secret
|
||||
key type, `connect`/`accept`, bi-stream open/accept, the
|
||||
send/recv methods and their exact signatures/return types, and
|
||||
whether they're `async def`. Everything below uses *provisional*
|
||||
names and must be reconciled. Do not skip this; do not guess
|
||||
from memory.
|
||||
|
||||
### 1.1 API-truth table (fill in during step 0)
|
||||
|
||||
| concept | provisional name | actual (fill in) |
|
||||
| --- | --- | --- |
|
||||
| secret key | `iroh.SecretKey.generate()` | |
|
||||
| endpoint builder | `iroh.Endpoint.builder(...).bind()` | |
|
||||
| node id | `endpoint.node_id() -> str` | |
|
||||
| node addr (relay + direct) | `iroh.NodeAddr` | |
|
||||
| dial | `await endpoint.connect(node_addr, alpn)` | |
|
||||
| accept conn | `await endpoint.accept()` | |
|
||||
| open bi-stream | `await conn.open_bi()` | |
|
||||
| accept bi-stream | `await conn.accept_bi()` | |
|
||||
| send | `await send_stream.write_all(b)` | |
|
||||
| recv | `await recv_stream.read(n) -> bytes\|None` | |
|
||||
| half-close | `await send_stream.finish()` | |
|
||||
|
||||
---
|
||||
|
||||
## 2. The `trio`-native uniffi future bridge (`tractor/ipc/_uniffi_trio.py`)
|
||||
|
||||
### 2.1 what uniffi actually generates
|
||||
|
||||
`uniffi`'s async support does not use asyncio *semantically* —
|
||||
it uses asyncio only as the *executor* for a poll loop. The
|
||||
generated python for an `async fn` is, in shape:
|
||||
|
||||
1. call `_uniffi_..._<method>(...)` → returns an opaque
|
||||
`RustFuture` handle (a `void*`/`u64`).
|
||||
2. loop: call
|
||||
`ffi_..._rust_future_poll_<T>(handle, callback, callback_data)`.
|
||||
The callback is a C-ABI fn pointer invoked **from an
|
||||
arbitrary rust thread** with a poll-result code
|
||||
(`READY`/`MAYBE_READY`).
|
||||
3. the generated glue's callback resolves an
|
||||
`asyncio.Future` via `loop.call_soon_threadsafe(...)`; the
|
||||
coroutine awaits it, then re-polls.
|
||||
4. on ready: `ffi_..._rust_future_complete_<T>(handle,
|
||||
&call_status)` → the value; then
|
||||
`ffi_..._rust_future_free_<T>(handle)`.
|
||||
|
||||
**The asyncio dependency is confined to step 3.** That is the
|
||||
whole insight: the bridge is ~40 lines.
|
||||
|
||||
### 2.2 the trio version
|
||||
|
||||
```python
|
||||
async def await_rust_future(
|
||||
poll: Callable, # ffi_..._rust_future_poll_<T>
|
||||
complete: Callable, # ffi_..._rust_future_complete_<T>
|
||||
free: Callable, # ffi_..._rust_future_free_<T>
|
||||
handle: int,
|
||||
lift: Callable[[Any], Any],
|
||||
) -> Any:
|
||||
'''
|
||||
Drive a `uniffi` rust-future to completion on the current
|
||||
`trio` task, bridging rust-thread wakeups via
|
||||
`TrioToken.run_sync_soon()`.
|
||||
|
||||
'''
|
||||
token = trio.lowlevel.current_trio_token()
|
||||
while True:
|
||||
wake = trio.Event()
|
||||
# NOTE, invoked from a *rust* thread!
|
||||
def _cb(_data, poll_code):
|
||||
token.run_sync_soon(wake.set)
|
||||
|
||||
cb = _UNIFFI_FUTURE_CALLBACK(_cb) # keep a strong ref!
|
||||
poll(handle, cb, 0)
|
||||
await wake.wait()
|
||||
if <poll_code was READY>:
|
||||
break
|
||||
try:
|
||||
status = _UniffiRustCallStatus.default()
|
||||
res = complete(handle, status)
|
||||
_uniffi_check_call_status(status) # reuse generated helper
|
||||
return lift(res)
|
||||
finally:
|
||||
free(handle)
|
||||
```
|
||||
|
||||
Critical details, each a real bug if missed:
|
||||
|
||||
- **`token.run_sync_soon()` is the only trio API callable from a
|
||||
foreign thread**, and it is documented as such. Use it; do
|
||||
*not* use `trio.from_thread.run_sync` (requires a trio thread
|
||||
context) and do not touch the `Event` directly from the
|
||||
callback.
|
||||
- **the poll code must reach the trio side.** Capture it in a
|
||||
`nonlocal`/1-slot list written by the callback *before*
|
||||
`run_sync_soon`, since the callback owns the value. Handle
|
||||
`MAYBE_READY` by re-polling (the loop above does).
|
||||
- **keep the `ctypes` callback object alive** across the await —
|
||||
a GC'd `CFUNCTYPE` trampoline is a segfault. Bind it to a
|
||||
local *and* make sure the local outlives the `poll()` call
|
||||
window.
|
||||
- **cancellation.** `await wake.wait()` is a trio checkpoint, so
|
||||
a `Cancelled` can fire while rust still owns the future. On
|
||||
cancel we must still `free(handle)` — and per uniffi, the
|
||||
correct sequence is to call the generated
|
||||
`ffi_..._rust_future_cancel_<T>(handle)` then continue
|
||||
polling to completion before `free`. Wrap the whole thing so
|
||||
the cancel path does:
|
||||
`with trio.CancelScope(shield=True): cancel(handle); <drain
|
||||
poll loop>; free(handle)`. **Bounded** shield (add a
|
||||
`trio.move_on_after()` with a module-level constant) so a
|
||||
wedged rust future can't make an actor un-cancellable —
|
||||
`tractor` is SC-first and an unbounded shield here would
|
||||
violate that.
|
||||
- **`trio.lowlevel.current_trio_token()`** must be captured on
|
||||
the trio side (not in the callback).
|
||||
|
||||
### 2.3 how to apply it to the generated bindings
|
||||
|
||||
Do **not** fork/vendor the generated `iroh` python. Instead ship
|
||||
a *narrow* re-dispatch shim:
|
||||
|
||||
- write `tractor/ipc/_uniffi_trio.py` with `await_rust_future()`
|
||||
plus a `@cm patch_uniffi_for_trio()` that monkey-patches the
|
||||
generated module's single async-driver entrypoint (in current
|
||||
uniffi that's `_uniffi_rust_call_async` / `_rust_call_async`,
|
||||
one function) to the trio implementation.
|
||||
- verify at import time that the expected symbol exists and
|
||||
raise a clear, actionable error naming the pinned `iroh`
|
||||
version if not. A silent fallback to asyncio would be a
|
||||
nightmare to debug.
|
||||
- **plan for this to break on `iroh`/`uniffi` upgrades.** Mitigate
|
||||
with (a) a unit test that drives one trivial `iroh` async call
|
||||
under bare `trio.run()` and asserts no event loop was ever
|
||||
created (`asyncio.get_event_loop_policy()` untouched /
|
||||
`asyncio._get_running_loop() is None`), and (b) a docstring
|
||||
pointing at the uniffi codegen template this mirrors.
|
||||
|
||||
If step 0 reveals the generated code is *structurally* hostile
|
||||
to this (e.g. `asyncio` imported and used at module scope for
|
||||
more than the driver), fall back to option (b): run iroh under
|
||||
`tractor.to_asyncio` infected mode and open the follow-up to
|
||||
revisit. Say so in the PR rather than fighting it.
|
||||
|
||||
---
|
||||
|
||||
## 3. Mapping QUIC onto `MsgTransport`
|
||||
|
||||
### 3.1 the layering decision
|
||||
|
||||
QUIC natively multiplexes streams inside one connection. The
|
||||
mapping that preserves *all* existing `tractor` semantics with
|
||||
the least new code:
|
||||
|
||||
```
|
||||
iroh Endpoint == one per actor (process) -> the "listener"
|
||||
iroh Connection == one per peer actor -> pooled
|
||||
iroh bi-stream == one `Channel`/`MsgTransport` -> 1:1
|
||||
```
|
||||
|
||||
- keep the 4-byte `<I` length-prefix framing **unchanged**. It's
|
||||
redundant-ish over a QUIC stream but it means
|
||||
`MsgpackTransport` is reused verbatim, and framing is cheap.
|
||||
Revisit only after it works.
|
||||
- **one-task-per-stream** falls out naturally, which is exactly
|
||||
the #353 note about QUIC sub-stream QoS/cancellation fitting
|
||||
`trio`.
|
||||
- `layer_key: int = 4` still (QUIC is L4-ish); note in a comment
|
||||
that this backend is really 4+security+multiplex.
|
||||
|
||||
**Connection pooling** is the one place we add state the other
|
||||
backends don't have: dialing the same peer twice should reuse
|
||||
the `Connection` and open a second bi-stream. Implement as a
|
||||
module-level `dict[NodeId, Connection]` guarded by a
|
||||
`trio.Lock`... **no** — that's a per-process cache with
|
||||
lifetime/teardown hazards. Instead reuse the codebase's existing
|
||||
idiom: `tractor.trionics.maybe_open_context()` keyed on the
|
||||
node-id, which already solves exactly this (one-cached-resource-
|
||||
per-key, refcounted, teardown-on-last-exit) and whose teardown
|
||||
semantics were just hardened (gh #488). Use it; do not hand-roll
|
||||
a cache. Anything concurrency-subtle here should get the
|
||||
`conc-anal` skill run over it.
|
||||
|
||||
### 3.2 `IrohAddress`
|
||||
|
||||
```python
|
||||
class IrohAddress(
|
||||
msgspec.Struct,
|
||||
frozen=True,
|
||||
):
|
||||
_node_id: str # 32B ed25519 pubkey, hex or z32
|
||||
_alpn: str = 'tractor/0' # the bindspace!
|
||||
# optional dial hints; NOT part of identity
|
||||
maybe_relay_url: str|None = None
|
||||
maybe_direct_addrs: tuple[str, ...] = ()
|
||||
|
||||
proto_key: ClassVar[str] = 'iroh' # ?or 'quic'; see §3.2.1
|
||||
unwrapped_type: ClassVar[type] = tuple[str, str]
|
||||
def_bindspace: ClassVar[str] = 'tractor/0'
|
||||
```
|
||||
|
||||
- **`.unwrap() -> (node_id_str, alpn_str)`** — a `(str, str)`
|
||||
tuple, which is *unambiguously distinct* from
|
||||
`TCPAddress`'s `(str, int)`. But careful:
|
||||
`wrap_address()`'s UDS case is
|
||||
`case (_, filename) if type(filename) is str` — which
|
||||
**already catches `(str, str)`**. So the iroh `case` MUST be
|
||||
ordered *before* the UDS case and guarded, e.g.
|
||||
`case (str() as nid, str() as alpn) if _is_node_id(nid):`
|
||||
with `_is_node_id()` a cheap length+alphabet check. Add a
|
||||
regression test asserting a UDS `(dir, filename)` pair still
|
||||
wraps to `UDSAddress` — this is the exact "wrong transport
|
||||
loaded" hazard `_addr.py:214` warns about.
|
||||
- `.bindspace` → `self._alpn`. This is the honest analogue:
|
||||
the ALPN is the set of endpoints willing to talk to you, and
|
||||
two `tractor` deployments sharing an iroh network are
|
||||
separated by ALPN exactly as two UDS deployments are
|
||||
separated by directory. Include a `tractor` version/proto
|
||||
epoch in the default ALPN so incompatible runtimes can't
|
||||
handshake.
|
||||
- `.is_valid` → node-id parses, alpn non-empty.
|
||||
- **`get_root()` is the hard one.** There is no
|
||||
well-known-port analogue: an iroh node id is a *keypair*, so
|
||||
"the host's default registrar addr" requires a *persisted
|
||||
secret key*. Design:
|
||||
- the root/registrar's secret key lives at
|
||||
`get_rt_dir() / 'iroh_registrar.key'` (0600), created on
|
||||
first use.
|
||||
- `get_root()` must stay **pure and import-time-safe**
|
||||
(contract §2.3: `_default_lo_addrs` is built at import!).
|
||||
So `get_root()` *reads* the key file if present and
|
||||
otherwise returns an `IrohAddress` with
|
||||
`_node_id=''`/sentinel, and the **generation** happens in
|
||||
an explicit sibling — `ensure_registrar_key() ->
|
||||
IrohAddress` — called from the listen path. Pure getter,
|
||||
explicit setter; do not smuggle key generation into
|
||||
`get_root()`.
|
||||
- this almost certainly means `_default_lo_addrs` must become
|
||||
lazy for this backend. **Land that refactor as its own prep
|
||||
commit** (a `default_lo_addrs()` that computes per-call
|
||||
instead of the import-time dict) — it also unblocks plan
|
||||
03's netns-scoped defaults.
|
||||
- `get_random()`: generate a fresh `SecretKey` per subactor and
|
||||
return its node-id. Note this runs post-fork pre-listen
|
||||
(contract §4) and costs an ed25519 keygen (~µs, fine). The
|
||||
*secret* can't live in a frozen `Address`, so it must be
|
||||
stashed where the listen path can find it: a module-level
|
||||
`dict[node_id, SecretKey]` populated by `get_random()` and
|
||||
consumed+popped by `start_listener()`. Ugly but honest;
|
||||
document it and note the alternative (thread the key through
|
||||
`Endpoint`) as a follow-up.
|
||||
|
||||
#### 3.2.1 `proto_key`: `'iroh'` vs `'quic'`
|
||||
|
||||
Use **`'quic'`** for the `proto_key`/`--tpt-proto` name and
|
||||
name the module `_quic.py`, with `iroh` as the *implementation*.
|
||||
Rationale: it keeps the door open for the `aioquic` fallback
|
||||
(§1) without a user-visible rename, and it matches how `uds` is
|
||||
a proto name rather than a lib name. Put `iroh`-specific bits
|
||||
behind an internal `_iroh` submodule if the file gets big.
|
||||
|
||||
### 3.3 the `trio.abc` adapters — where the real work is
|
||||
|
||||
Contract §3 says a non-socket backend needs three upstream
|
||||
generalizations. Land them **as a prep PR, before any iroh
|
||||
code**, so they can be reviewed on their own merits with
|
||||
tcp/uds still the only backends:
|
||||
|
||||
1. **`Endpoint.start_listener()` must not assume
|
||||
`.socket.getsockname()`.** Use the same
|
||||
`Address.rebind_from_sockname: ClassVar[bool]` gate that
|
||||
plan 01 §3.2 introduces — coordinate so it lands once. (If
|
||||
plan 01 lands first, this is free.)
|
||||
2. **`transport_from_stream()` (`_types.py:92`) must not assume
|
||||
`trio.SocketStream`.** Replace the `sock.family` match with:
|
||||
check `isinstance(stream, trio.SocketStream)` → existing
|
||||
family match; else look for a
|
||||
`stream.tpt_key: ClassVar[MsgTransportKey]` attribute on the
|
||||
adapter and use it. Keeps the existing path byte-identical
|
||||
and makes new stream types self-describing (a much better
|
||||
shape than growing an `isinstance` ladder).
|
||||
3. **type annotations**: `handle_stream_from_peer(stream:
|
||||
trio.SocketStream)` → `trio.abc.Stream`; `Endpoint._listener:
|
||||
SocketListener|None` → `trio.abc.Listener|None`;
|
||||
`MsgTransport.stream: trio.SocketStream` →
|
||||
`trio.abc.Stream`. Annotation-only, zero behaviour change.
|
||||
|
||||
Then the adapters:
|
||||
|
||||
```python
|
||||
class QuicMsgStream(trio.abc.HalfCloseableStream):
|
||||
'''
|
||||
A single `iroh` bi-directional QUIC stream presented as
|
||||
a `trio` byte-stream so `MsgpackTransport` can frame over
|
||||
it unmodified.
|
||||
|
||||
'''
|
||||
tpt_key: ClassVar[MsgTransportKey] = ('msgpack', 'quic')
|
||||
|
||||
def __init__(self, conn, send, recv) -> None: ...
|
||||
async def send_all(self, data: bytes) -> None: ...
|
||||
async def wait_send_all_might_not_block(self) -> None: ...
|
||||
async def receive_some(self, max_bytes: int|None = None) -> bytes: ...
|
||||
async def send_eof(self) -> None: ...
|
||||
async def aclose(self) -> None: ...
|
||||
```
|
||||
|
||||
Non-negotiable behaviours (each maps to a `match` case that
|
||||
already exists in `_transport.py` and must keep working):
|
||||
|
||||
- `receive_some()` returns `b''` at clean EOF →
|
||||
`MsgpackTransport._iter_packets()` sees `header == b''` and
|
||||
raises `TransportClosed(loglevel='transport')`. **This is the
|
||||
graceful-disconnect path the whole runtime relies on**; get it
|
||||
right first.
|
||||
- a reset/aborted stream → raise `trio.BrokenResourceError`.
|
||||
- use after local close → raise `trio.ClosedResourceError`
|
||||
(ideally with `'another task closed this fd'`-equivalent text
|
||||
absent, so the `raise_on_report` branch at
|
||||
`_transport.py:290` stays quiet).
|
||||
- `send_all()` on a closed peer → `trio.BrokenResourceError`.
|
||||
- honour `trio`'s one-task-per-direction rule: guard with
|
||||
`trio._util.ConflictDetector` equivalents (or just document +
|
||||
assert), because `MsgpackTransport` already serializes sends
|
||||
with a `StrictFIFOLock` but recvs are single-task by
|
||||
construction.
|
||||
- **buffering**: if iroh's `read()` doesn't support
|
||||
"read up to n", `receive_some()` must maintain an internal
|
||||
leftover buffer. Note `MsgpackTransport` wraps us in
|
||||
`tricycle.BufferedReceiveStream` anyway, so `receive_some()`
|
||||
just needs *some* nonzero-progress contract.
|
||||
|
||||
```python
|
||||
class QuicListener(trio.abc.Listener):
|
||||
'''
|
||||
Accepts iroh `Connection`s and yields one `QuicMsgStream`
|
||||
per accepted bi-stream, so `trio.serve_listeners()` spawns
|
||||
one `handle_stream_from_peer()` per `Channel`.
|
||||
|
||||
'''
|
||||
async def accept(self) -> QuicMsgStream: ...
|
||||
async def aclose(self) -> None: ...
|
||||
```
|
||||
|
||||
The accept-side subtlety: `trio.abc.Listener.accept()` yields
|
||||
one stream per call, but iroh gives us *connections* which then
|
||||
yield *streams*. So `QuicListener` needs an internal
|
||||
`trio.MemoryReceiveChannel[QuicMsgStream]` fed by a background
|
||||
task-pair (one task accepting connections, one per connection
|
||||
accepting bi-streams). `trio.abc.Listener` has no nursery, so:
|
||||
make the listener **constructed by an `@acm`** that owns the
|
||||
nursery, and have `start_listener()` be that `@acm`'s driver.
|
||||
|
||||
⚠️ this collides with `Endpoint.start_listener()` being a plain
|
||||
`async def` returning a listener. Two options:
|
||||
- **(a)** hang the nursery off the `Endpoint`'s existing
|
||||
`listen_tn` — `_serve_ipc_eps()` already creates `listen_tn`
|
||||
and passes it into every `Endpoint` (`_server.py:1063-1074`),
|
||||
and `Endpoint.listen_tn` is right there. So
|
||||
`start_listener()` can `self.listen_tn.start_soon(...)` the
|
||||
acceptor tasks. **Recommended**: no upstream signature change,
|
||||
correct lifetime (dies with the ep group), and it's why
|
||||
`listen_tn` is on the struct in the first place.
|
||||
- (b) change `start_listener()` to a `@acm`. Bigger blast
|
||||
radius; only if (a) proves insufficient.
|
||||
|
||||
Since `start_listener()` is called via
|
||||
`inspect.getmodule(addr)` with only `addr=` (contract §1.3),
|
||||
option (a) needs the `Endpoint` itself. Either add `ep=` to the
|
||||
module-level `start_listener()` call signature (all backends
|
||||
ignore it except quic → small upstream change, do it as part of
|
||||
the prep PR and make it keyword-only with a default) or have
|
||||
`QuicListener.accept()` lazily spawn via
|
||||
`trio.lowlevel.current_task().parent_nursery` (**rejected** —
|
||||
fragile, implicit). Do the explicit `ep=` kwarg.
|
||||
|
||||
### 3.4 `maddr`
|
||||
|
||||
Multiaddr already standardizes the pieces:
|
||||
|
||||
```
|
||||
/ip4/<h>/udp/<p>/quic-v1 # direct
|
||||
/ip4/<h>/udp/<p>/quic-v1/p2p/<node-id> # direct + identity
|
||||
/dns/<relay-host>/tcp/443/tls/ws/p2p/<node> # relay-ish
|
||||
```
|
||||
|
||||
- primary form: `/p2p/<node-id>` alone is a legal maddr and is
|
||||
the *only* required component for iroh dialling — relay +
|
||||
direct addrs are discovery hints. So `mk_maddr()` emits
|
||||
`/p2p/<node_id>` and, when known, prefixes the direct
|
||||
`/ip4/../udp/../quic-v1/`.
|
||||
- `/p2p/` values are multihash-encoded peer ids; an iroh node-id
|
||||
is a raw ed25519 key. Converting requires the identity
|
||||
multihash + libp2p key protobuf wrapper. **Decide**: emit the
|
||||
raw node-id under a *tractor-local* `/iroh/<node-id>` segment
|
||||
(needs upstream registration, same track as `wg`/`tipc`,
|
||||
gh #483) rather than pretending to be a libp2p peer-id we
|
||||
can't round-trip. Return the `str` form until upstream lands
|
||||
(`MsgTransport.maddr` is `Multiaddr|str`).
|
||||
- this backend is the strongest argument for gh #443's
|
||||
**tunnelled/composed maddr** item: `/ip4/../udp/../quic-v1/..`
|
||||
*is* a composed stack. Cross-reference plan 03 §5 so the two
|
||||
grammars land compatibly.
|
||||
|
||||
---
|
||||
|
||||
## 4. Discovery integration
|
||||
|
||||
- iroh's node-id addressing means the `tractor` registrar can
|
||||
hold `IrohAddress`es that are **reachable from anywhere** with
|
||||
no port-forwarding — that is the headline feature. The
|
||||
registrar itself works unchanged.
|
||||
- iroh has its own discovery (DNS/pkarr/mdns). **Out of scope**;
|
||||
note in the follow-up that `tractor.discovery` could
|
||||
eventually delegate to it, which would be the direct analogue
|
||||
of plan 01's TIPC-topology idea.
|
||||
- relay servers: default to n0's public relays for the demo,
|
||||
document self-hosting (docs.iroh.computer's dedicated-infra
|
||||
page is linked from #353), and make the relay set a
|
||||
`start_listener()` kwarg.
|
||||
|
||||
## 5. Security note
|
||||
|
||||
QUIC is TLS-1.3-always and iroh authenticates by node-id, so
|
||||
this backend is the first `tractor` transport with real
|
||||
transport security and peer authentication. Two things follow:
|
||||
1. an **allowlist hook** — an actor should be able to reject
|
||||
inbound connections from unknown node-ids *before* the
|
||||
`Aid` handshake. Natural home: a predicate kwarg on
|
||||
`start_listener()`, evaluated in `QuicListener`'s connection
|
||||
acceptor task. Sketch it; ship it in PR 1 if cheap (it is).
|
||||
2. do **not** claim any security property for the other
|
||||
backends by association. `tcp`/`uds`/`tipc` remain
|
||||
unauthenticated; that's what plan 03 (wg) is for.
|
||||
|
||||
## 6. Commit sequencing
|
||||
|
||||
0. **spike (throwaway, not committed)**: drive iroh under
|
||||
`trio-asyncio`/`tractor.to_asyncio`, echo bytes over a
|
||||
bi-stream between two procs. Fills in §1.1. Timebox it.
|
||||
1. prep PR: annotation widening + `rebind_from_sockname` gate +
|
||||
`transport_from_stream()` `tpt_key` dispatch + `ep=` kwarg on
|
||||
`start_listener()` + lazy `default_lo_addrs()`. **No new
|
||||
backend.** Full suite green on tcp *and* uds.
|
||||
2. `_uniffi_trio.py` + its tests (drive one iroh async call
|
||||
under bare `trio.run()`; assert no asyncio loop; assert
|
||||
cancellation frees the future).
|
||||
3. `QuicMsgStream` + tests against a *loopback* iroh endpoint
|
||||
pair in one process (no `tractor` runtime): send/recv, clean
|
||||
EOF → `b''`, reset → `BrokenResourceError`, use-after-close
|
||||
→ `ClosedResourceError`.
|
||||
4. `QuicListener` + `start_listener()` + `IrohAddress` +
|
||||
key-file mgmt.
|
||||
5. `MsgpackQuicStream(MsgpackTransport)` + `connect_to()` +
|
||||
`maybe_open_context()` connection pooling.
|
||||
6. registration tables + `--tpt-proto quic` + full suite.
|
||||
7. maddr + docs + a two-host example (pairs with #482's format).
|
||||
|
||||
## 7. Testing
|
||||
|
||||
- capability predicate `is_quic_available()` → `iroh` importable
|
||||
*and* the uniffi driver symbol present at the pinned version.
|
||||
Same `pytest.fail`-early hook as plan 01 §7.2.
|
||||
- **the acceptance bar is the same**: whole suite green under
|
||||
`--tpt-proto quic`. Expect this to shake out real bugs in the
|
||||
adapters (esp. teardown ordering and `TransportClosed`
|
||||
classification) — that's the point.
|
||||
- expect to need **timeout headroom**: iroh endpoint bind +
|
||||
first connect (relay discovery) is orders of magnitude slower
|
||||
than a UDS bind. Before touching any test deadline, rule out
|
||||
the CPU-throttle false-positive (see the project's
|
||||
`env_cpu_throttle_masquerades_as_regression` note); then, if
|
||||
real, add a per-proto timeout multiplier to the test harness
|
||||
rather than editing individual tests.
|
||||
- a no-network test mode: iroh with relays disabled +
|
||||
loopback direct addrs only, so CI doesn't depend on n0's
|
||||
infra. **Make this the default in CI**; mark the relay tests
|
||||
`pytest.mark.net` and keep them out of the default run.
|
||||
- leak checks: assert every `SecretKey`/`Endpoint` is closed on
|
||||
actor teardown (an `Endpoint` left open holds UDP sockets and
|
||||
relay connections; a leak here shows up as hung tests, not
|
||||
errors).
|
||||
|
||||
## 8. Risks
|
||||
|
||||
| risk | mitigation |
|
||||
| --- | --- |
|
||||
| uniffi codegen internals shift on upgrade | pinned minor, symbol assertion at import, the "no asyncio loop" test, documented fallback to `to_asyncio` |
|
||||
| rust-thread callback → trio wakeup mishandled (segfault / lost wakeup / un-cancellable task) | strong ref on the ctypes trampoline; `run_sync_soon` only; **bounded** shielded cancel-drain; run the `conc-anal` skill over the bridge |
|
||||
| `iroh` wheel availability for 3.13/3.14 on linux+macos | verify in step 0; if missing, that alone may force the `aioquic` fallback |
|
||||
| QUIC latency/jitter destabilizes the existing suite's timing assumptions | per-proto timeout multiplier, relay-less CI mode |
|
||||
| `(str, str)` unwrapped form collides with UDS in `wrap_address()` | guarded case ordered first + explicit regression test (§3.2) |
|
||||
| scope creep into iroh's docs/blobs/gossip crates | this backend is `Endpoint`+`Connection`+bi-streams only; anything else is a separate issue |
|
||||
|
||||
## 9. Follow-up issue seeds
|
||||
|
||||
- `tractor.discovery` delegating to iroh discovery (DNS/pkarr/mdns)
|
||||
- per-`Context` QUIC sub-streams: today one `Channel` == one
|
||||
stream; QUIC would let each `tractor.Context` own its own
|
||||
stream with independent flow-control and cancellation — this
|
||||
is the genuinely novel win #353 gestures at, and it's a
|
||||
runtime-layer change, not a transport one
|
||||
- unreliable QUIC datagrams for a lossy-ok broadcast transport
|
||||
(pairs with plan 01's TIPC-multicast seed)
|
||||
- node-id allowlist → a real `tractor` authz story
|
||||
- `aioquic` sans-io backend reusing §3's adapters
|
||||
|
|
@ -0,0 +1,757 @@
|
|||
# Plan 03 — WireGuard (and other tunnels) as a *nested bindspace* via `pyroute2`
|
||||
|
||||
Tracks gh [#482] + the tunnelled-maddr item of [#443].
|
||||
Prereq reading:
|
||||
[`00_shared_backend_contract.md`](./00_shared_backend_contract.md).
|
||||
|
||||
**Thesis**: WireGuard is **not** a `MsgTransport`. It is an
|
||||
interface-layer tunnel that is transparent to `socket(2)`, so
|
||||
the correct abstraction is a *bindspace* — a scoped,
|
||||
`@acm`-managed network context that an existing L4 transport
|
||||
(`tcp`, and later `quic`/`tipc`-over-UDP-bearer) binds *inside*.
|
||||
This plan implements `Address.namespace` (spec'd but unused
|
||||
since day one) and the composed/tunnelled maddr grammar, with
|
||||
`pyroute2` as the netlink codec and as much of the I/O moved
|
||||
onto `trio` as the library's sans-io layer allows.
|
||||
|
||||
[#482]: https://github.com/goodboy/tractor/issues/482
|
||||
[#443]: https://github.com/goodboy/tractor/issues/443
|
||||
|
||||
---
|
||||
|
||||
## 1. What exists today (verified, per #482)
|
||||
|
||||
- `wrap_address()` accepts maddr `str`s (leading-`/` dispatch,
|
||||
`_addr.py:262`). `parse_maddr()` and `mk_maddr()` support plain
|
||||
TCP/UDS addresses plus nested, canonical bearer-first `/wg/`
|
||||
stacks represented locally as `TunnelledAddress` wrappers.
|
||||
- there is no `wg` proto in the multiaddr *spec* yet, but
|
||||
multiformats/py-multiaddr#108 (key form `u<base64url>`) is
|
||||
**merged** as of 2026-07-28 (`f86519da`) — and unreleased, the
|
||||
latest `0.2.0` predating it. Spec registration is still tracked
|
||||
by multiformats/py-multiaddr#107 and gh #483.
|
||||
- **today's deployable story remains declarative**: run `wg-quick`
|
||||
out-of-band, parse the maddr, strip its wrapper to the overlay
|
||||
`(host, port)`, verify the pubkey against the live tunnel,
|
||||
hand the overlay addr to `registry_addrs=`/`tpt_bind_addrs=`.
|
||||
#482 already contains working example code for exactly this.
|
||||
- `Address.namespace` exists in the Protocol
|
||||
(`_addr.py:94-101`, "the if-available OS-specific network
|
||||
namespace key"). `TunnelledAddress` implements it from its spec;
|
||||
no concrete transport backend implements it yet.
|
||||
|
||||
## 2. Three layers, three PRs
|
||||
|
||||
| layer | what | dep | ships |
|
||||
| --- | --- | --- | --- |
|
||||
| **A. declarative** | commit #482's examples; `parse_maddr()` learns `/wg/u<key>` → overlay `Address` + verified pubkey | `multiaddr` (already), `wg(8)` CLI | first |
|
||||
| **B. `pyroute2` read/verify** | replace the `subprocess.run(['sudo','wg','show'])` shelling with netlink queries | `pyroute2` extra | second |
|
||||
| **C. `@acm` lifecycle** | create/configure/tear down wg ifaces + netns *from the runtime*, as nested bindspaces; implement `Address.namespace` | `pyroute2` + `CAP_NET_ADMIN` | third |
|
||||
|
||||
Each is independently valuable and independently reviewable.
|
||||
**Do not attempt C first** — the interesting design (nested
|
||||
bindspace `@acm`s) is only well-posed once A has pinned the
|
||||
address grammar and B has proven the netlink path under trio.
|
||||
|
||||
---
|
||||
|
||||
## 3. Layer A — declarative `wg` maddrs
|
||||
|
||||
### 3.1 the address shape
|
||||
|
||||
The decision: **a wg segment annotates an existing address, it
|
||||
does not create a new address type.** Two candidate encodings;
|
||||
**pick (a)**:
|
||||
|
||||
- **(a) `TunnelledAddress` wrapper** (recommended):
|
||||
```python
|
||||
class TunnelledAddress(
|
||||
msgspec.Struct,
|
||||
frozen=True,
|
||||
):
|
||||
overlay: Address # e.g. TCPAddress
|
||||
tunnel: WGTunnelSpec # proto-specific, frozen
|
||||
```
|
||||
with `.proto_key` **delegating to `overlay.proto_key`** so every
|
||||
existing table lookup (`_addr_to_transport`,
|
||||
`enable_transports` guard at `_root.py:391`,
|
||||
`transport_from_addr()`) keeps working untouched, and
|
||||
`.unwrap()` delegating to `overlay.unwrap()` so **nothing new
|
||||
crosses the wire**. `.namespace` and `.bindspace` come from
|
||||
the tunnel spec. The wrapper is stripped (`→ .overlay`) at the
|
||||
moment of bind/connect.
|
||||
- ⚠️ `is_wrapped_addr()` (`_addr.py:194`) tests
|
||||
`type(addr) in _address_types.values()` — a `bidict` of
|
||||
proto_key→type. `TunnelledAddress` isn't in it and must not
|
||||
be (it's not 1:1 with a proto). So either add an explicit
|
||||
`isinstance(addr, TunnelledAddress)` clause there, or give
|
||||
the wrapper a marker and test structurally. Do the former;
|
||||
it's two lines and honest.
|
||||
- the reflection in `Endpoint.start_listener()`
|
||||
(`inspect.getmodule(self.addr)`) would resolve to the
|
||||
*wrapper's* module, not the transport's. **So the wrapper
|
||||
must be unwrapped before it reaches `Endpoint`** — i.e. by
|
||||
the bindspace `@acm` (layer C) or by `parse_maddr()`
|
||||
(layer A). State this loudly in the docstring; it's the #1
|
||||
way to get this wrong.
|
||||
- (b) add fields to each existing `Address` type. Rejected:
|
||||
duplicates tunnel logic per-backend and pollutes `.unwrap()`.
|
||||
|
||||
```python
|
||||
class WGTunnelSpec(
|
||||
msgspec.Struct,
|
||||
frozen=True,
|
||||
):
|
||||
peer_pubkey: str # std-base64 `wg(8)` form
|
||||
iface: str = 'wg0'
|
||||
netns: str|None = None
|
||||
# layer-C-only fields, unset in layer A
|
||||
maybe_endpoint: tuple[str, int]|None = None
|
||||
maybe_allowed_ips: tuple[str, ...] = ()
|
||||
```
|
||||
|
||||
### 3.2 `parse_maddr()`/`mk_maddr()`
|
||||
|
||||
Grammar — **verified** against py-multiaddr#108, first on the
|
||||
`baudco/py-multiaddr@wg_support` branch and re-verified after it
|
||||
merged upstream (`multiformats/py-multiaddr@f86519da`); all three
|
||||
forms below parse *and* round-trip. Note the codec also validates
|
||||
that the key decodes to exactly 32 bytes, so a truncated key is a
|
||||
`StringParseError`, not a silently-mangled parse:
|
||||
|
||||
```
|
||||
/ip4/192.168.1.50/udp/51820/wg/u<A_pub>/ip4/10.0.11.1/tcp/1616
|
||||
\_______ bearer __________/\__ key __/\______ overlay ______/
|
||||
underlay, wg `ListenPort` the `MsgTransport` bind
|
||||
```
|
||||
|
||||
The `/wg/` segment is **infix, not suffix** — the segments
|
||||
*before* it are the wg **bearer** (the underlay `(ip, udp-port)`
|
||||
that `wg(8)` itself listens on, per the codec docstring's own
|
||||
`/ip4/1.2.3.4/udp/51820/wg/{key}` example), and the segments
|
||||
*after* are the **overlay** endpoint that `tractor` binds.
|
||||
|
||||
⚠️ **CORRECTION** — an earlier revision of this plan (and the
|
||||
examples in gh #482) used a *suffix* form
|
||||
`/ip4/10.0.11.1/tcp/1616/wg/u<key>`. That parses, but it is
|
||||
semantically inverted: it puts the overlay addr where the bearer
|
||||
belongs, `tcp` where wg's `udp` `ListenPort` goes, and declares
|
||||
no overlay endpoint at all. `parse_wg_maddr()` in
|
||||
`examples/multihost/wg_lan/` now rejects it with an actionable
|
||||
error.
|
||||
Observed protocol-name lists, for writing the `match`:
|
||||
|
||||
| maddr | `[p.name for p in m.protocols()]` |
|
||||
| --- | --- |
|
||||
| `/ip4/1.2.3.4/udp/51820/wg/u<k>` | `['ip4','udp','wg']` |
|
||||
| `/ip4/../udp/../wg/u<k>/ip4/../tcp/..` | `['ip4','udp','wg','ip4','tcp']` |
|
||||
|
||||
- so the three parts have **three different owners**, and only the
|
||||
third is an `Endpoint`:
|
||||
|
||||
| part | socket owner / provisioner | runtime role |
|
||||
| --- | --- | --- |
|
||||
| bearer | kernel-owned; externally provisioned in layer A, tractor bindspace-provisioned in layer C | control-plane metadata, never an `Endpoint` |
|
||||
| `/wg/u<key>` | nothing — it's an identity | parsed and explicitly verified |
|
||||
| overlay | `tractor`'s `IPCServer` | application `MsgTransport`, as `.overlay` |
|
||||
|
||||
This owner-split is the real axis of the design, *not* whether
|
||||
the maddr stack is "composed" (it is).
|
||||
- ⚠️ **CORRECTION**, an earlier draft of this section specced a
|
||||
hand-rolled `_peel_tunnel_segs(proto_names) -> (bearer_names,
|
||||
tunnel_specs, overlay_names)`. **Do not write it.**
|
||||
`py-multiaddr` already ships the whole tunnel compose/peel API
|
||||
and it was simply missed here — see its README "En/decapsulate"
|
||||
and "Tunneling" sections, and gh #443's 2nd bullet which links
|
||||
them. Verified against the pinned rev:
|
||||
|
||||
| need | API |
|
||||
| --- | --- |
|
||||
| isolate the bearer | `ma.decapsulate_code(P_WG)` |
|
||||
| drop the overlay, keep bearer+key | `ma.decapsulate(overlay_ma)` |
|
||||
| per-seg maddrs | `ma.split()` |
|
||||
| rejoin a seg tail | `Multiaddr.join(*segs)` |
|
||||
| read the key | `ma.value_for_protocol('wg')` |
|
||||
| recompose | `bearer.encapsulate(key).encapsulate(overlay)` |
|
||||
|
||||
`.decapsulate_code()` handles the infix `/wg/` seg cleanly
|
||||
*because* it cuts on proto-code and never tries to match an
|
||||
addr value — the key seg has no addr of its own. This is the
|
||||
same NIH trap gh #429 existed to close, one layer up.
|
||||
|
||||
- ⚠️ `value_for_protocol('ip4')` on a *full* tunnelled maddr
|
||||
silently returns the **first** match, i.e. the bearer's host.
|
||||
Always call it on a peeled sub-maddr, never the whole stack.
|
||||
|
||||
- `parse_maddr()` gains a case on
|
||||
`[('ip4'|'ip6'), 'udp', 'wg', ('ip4'|'ip6'), <overlay-l4>]` →
|
||||
peel w/ the API above, decode the multibase key to std-base64,
|
||||
and return `TunnelledAddress(overlay=..., tunnel=WGTunnelSpec(
|
||||
...))` w/ the bearer recorded in the spec.
|
||||
- keep the existing 2-proto cases byte-identical; add the new
|
||||
case *after* them.
|
||||
- nesting (wg-in-wg) falls out of `.decapsulate_code()` cutting
|
||||
at the *last* occurrence — peel repeatedly rather than
|
||||
recursing through a bespoke splitter.
|
||||
- `mk_maddr()` inverse for `TunnelledAddress` is just
|
||||
`.encapsulate()` composition; don't rebuild `str`s by hand.
|
||||
- **pending an upstream release**: py-multiaddr#108 is merged, so
|
||||
`Multiaddr('/…/wg/u…')` parses off a PEP 621 direct-revision pin,
|
||||
since no release carries the codec. Gate parser entry on
|
||||
`_wg_proto_code()`, implemented as
|
||||
`protocols.protocol_with_name('wg')` under
|
||||
`except ProtocolNotFoundError`. Do **not** probe by parsing a
|
||||
dummy like `Multiaddr('/wg/uAAAA')` — the codec enforces a
|
||||
32-byte key, so that raises even when the proto *is* known. Do
|
||||
**not** hand-roll a `wg` parser in `tractor` — the whole point
|
||||
of #429 was dropping the NIH parser.
|
||||
|
||||
### 3.3 pure codecs + explicit verification
|
||||
|
||||
Port #482 §2's pure helpers into
|
||||
`tractor/discovery/_tunnel.py`, keeping the impure probe cleanly
|
||||
separated until layer B:
|
||||
|
||||
```python
|
||||
def parse_wg_maddr(maddr: str) -> TunnelledAddress: ... # pure
|
||||
def wg8_pubkey(multibase_key: str) -> str: ... # pure
|
||||
async def verify_wg_peer(spec: WGTunnelSpec) -> bool: ... # layer B
|
||||
```
|
||||
|
||||
Layer A's example-local `verify_wg_peer()` may shell out (`wg show
|
||||
<if> peers`), but layer B replaces that probe with one explicit async
|
||||
function backed by pyroute2. Never call it implicitly from
|
||||
`wrap_address()`/`parse_maddr()` — parsing must stay pure and
|
||||
side-effect-free; verification is the *caller's* explicit step
|
||||
(and later, the bindspace `@acm`'s).
|
||||
|
||||
### 3.4 deliverables
|
||||
|
||||
- `examples/` scripts distilled from #482 §§3-5 (this is the
|
||||
unchecked "commit examples from ^" bullet in #443). They live
|
||||
under `examples/multihost/` — `test_docs_examples.py` walks
|
||||
`examples/` recursively and runs every collected file as a
|
||||
subproc asserting `rc == 0` (it doesn't even filter by
|
||||
extension, so a stray `README.md` would be `python`-run too),
|
||||
and `'multihost' not in p[0]` is already in its exclusion
|
||||
list. Anything needing a real second host or a live tunnel
|
||||
belongs there.
|
||||
- a `docs/` page: tunnel setup, the maddr form, the two-host
|
||||
run. Keep prose in the docs; keep the examples runnable and
|
||||
minimal.
|
||||
- tests: maddr round-trip, `TunnelledAddress` delegation
|
||||
(`proto_key`/`unwrap` identical to overlay), `wrap_address()`
|
||||
regression (a tunnelled maddr `str` → `TunnelledAddress`; a
|
||||
plain one → unchanged), and **a real end-to-end over a
|
||||
locally-created wg pair** gated on `CAP_NET_ADMIN` (see §5.4).
|
||||
|
||||
---
|
||||
|
||||
## 4. Layer B — `pyroute2` under `trio`
|
||||
|
||||
### 4.1 the library situation (verify at implementation time)
|
||||
|
||||
`pyroute2` ≥0.9 rewrote its core onto **asyncio**
|
||||
(`AsyncIPRoute`; the sync `IPRoute` wraps it with its own loop).
|
||||
It also ships a `WireGuard` netlink (generic-netlink) module
|
||||
supporting `.set(iface, private_key=..., peer={...})` and
|
||||
`.info(iface)`, plus `pyroute2.netns` / `NetNS` for namespaces,
|
||||
and `IPRoute.link('add', kind='wireguard', ifname=...)`.
|
||||
|
||||
Three integration options, in increasing trio-nativeness:
|
||||
|
||||
- **(1) `trio.to_thread.run_sync()` around the sync API.**
|
||||
Netlink ops here are one-shot, sub-millisecond, and happen at
|
||||
bind/teardown time only — *not* in the msg hot path. This is
|
||||
the **correct default**: it's ~10 lines, uses a battle-tested
|
||||
API, and costs nothing where it's used.
|
||||
- **(2) sans-io: `trio.socket` + pyroute2's message codecs.**
|
||||
`pyroute2`'s message classes
|
||||
(`pyroute2.netlink.rtnl.*`, `pyroute2.netlink.generic.wireguard.wgmsg`)
|
||||
encode/decode independently of its I/O core. So a
|
||||
`tractor/ipc/_netlink.py` with a small trio `NetlinkSocket`
|
||||
(`trio.socket.socket(AF_NETLINK, SOCK_RAW|SOCK_DGRAM, proto)`,
|
||||
`sendto`/`recv`, seq/pid matching, `NLMSG_DONE`/`NLMSG_ERROR`
|
||||
handling) + pyroute2 codecs is very achievable and is the
|
||||
honest reading of "as much trio wrapping as possible where any
|
||||
other async support can be replaced".
|
||||
**Do this for the paths we actually need** (link add/del,
|
||||
addr add, wg get/set, netns bind) and *only* those — a
|
||||
general netlink client is out of scope.
|
||||
- (3) reimplement the codecs. Never.
|
||||
|
||||
**Recommended split**: ship (1) first so layer B is a small,
|
||||
reviewable, behaviour-preserving swap of `verify_wg_peer()`'s
|
||||
body; then land (2) as a follow-up commit for the read path
|
||||
(`wg get`, `link get`) where the sans-io surface is smallest,
|
||||
and keep (1) for the privileged mutating ops. Measure before
|
||||
converting anything else — there is no perf argument here, only
|
||||
a "no foreign event loop in a trio actor" argument, which (1)
|
||||
already satisfies (a thread is not an event loop).
|
||||
|
||||
Explicitly **do not** pull in `trio-asyncio` for pyroute2 or infect
|
||||
every wg-using actor merely to service one-shot netlink calls. A
|
||||
dedicated `wgman` actor (§5.1) is the one plausible asyncio-hosted
|
||||
shape: it can use tractor's own `.to_asyncio` task linkage while
|
||||
keeping the foreign loop and provisioning authority out of ordinary
|
||||
actor processes.
|
||||
|
||||
### 4.2 API shape
|
||||
|
||||
Pure-ish, functional, `@acm` for anything with teardown:
|
||||
|
||||
```python
|
||||
async def read_wg_peers(
|
||||
iface: str = 'wg0',
|
||||
netns: str|None = None,
|
||||
) -> tuple[str, ...]: ... # base64 pubkeys
|
||||
|
||||
async def read_wg_pubkey(iface: str = 'wg0', ...) -> str: ...
|
||||
```
|
||||
|
||||
and `verify_wg_peer()` becomes a thin composition over one shared key
|
||||
snapshot.
|
||||
Note the pure-getter rule: no `read_wg_peers(..., create=True)`.
|
||||
|
||||
---
|
||||
|
||||
## 5. Layer C — nested bindspace `@acm`s + `Address.namespace`
|
||||
|
||||
This is the part #443 and `multiaddr_declare_eps.md` actually
|
||||
ask for: *"for any tunneled maddr-`str`-entry we deliver a
|
||||
data-structure which can easily be passed to nested `@acm`s
|
||||
which consecutively setup nested net bindspaces for binding the
|
||||
endpoint addrs"*.
|
||||
|
||||
Layer C is where tractor takes ownership of bindspace orchestration.
|
||||
For a fully bootstrapped deployment it may create the netns and wg
|
||||
iface, configure peers/routes, and ask the kernel to establish the
|
||||
bearer's UDP `ListenPort` through netlink/`pyroute2`. "Kernel-owned"
|
||||
describes the data-plane socket, not who provisions it: tractor owns
|
||||
the lifecycle while `Endpoint`/`MsgTransport` remain responsible only
|
||||
for the overlay application socket.
|
||||
|
||||
### 5.1 candidate default: first-child `wgman`
|
||||
|
||||
For a WG-enabled deployment profile, consider eagerly spawning one
|
||||
private **WireGuard manager** (`wgman`) as the root actor's logical
|
||||
first child. It is a narrow network-control-plane service, not a
|
||||
general worker and not an application-visible transport endpoint.
|
||||
The naive profile gets one manager for the actor tree; advanced
|
||||
deployments may disable it for pre-provisioned networking or place
|
||||
one manager in each capability/bindspace security domain.
|
||||
|
||||
"First child" describes supervision and teardown ordering, not a
|
||||
serial startup barrier. Submit the `wgman` spawn in the same startup
|
||||
wave as ordinary children, start its pyroute2 import, generic-netlink
|
||||
discovery and declared-tunnel reconciliation immediately, and publish
|
||||
a readiness signal separately. Sibling processes can boot in parallel;
|
||||
only their first WG-dependent bind/dial waits for manager readiness.
|
||||
This overlaps setup with actor-tree startup and avoids every sibling
|
||||
paying its own pyroute2/loop/socket initialization latency.
|
||||
|
||||
The initial manager can still call the sync helpers from §4.1. A
|
||||
natural follow-up is to spawn it with `infect_asyncio=True` and keep
|
||||
`AsyncWireGuard` clients alive on asyncio's host loop through
|
||||
`tractor.to_asyncio.run_task()`. Tractor then owns cross-loop task
|
||||
linkage, cancellation and error propagation, while normal siblings
|
||||
remain plain Trio actors. Keep one client per realized namespace or
|
||||
other kernel control domain; do not share a pyroute2 socket across
|
||||
domains merely to reduce object count.
|
||||
|
||||
Keep the authority surface deliberately small:
|
||||
|
||||
- accept structured inspect/verify/ensure/release requests derived
|
||||
from `WGTunnelSpec`, `BindspaceSpec` and explicit `role`; never
|
||||
expose arbitrary pyroute2 calls, shell commands or `setns()` RPC;
|
||||
- let the root/supervisor mediate access initially, or hand siblings
|
||||
a scoped manager capability; do not register a privileged `wgman`
|
||||
endpoint for unrestricted cluster-wide discovery;
|
||||
- never return private keys or namespace FDs to application actors;
|
||||
pass secrets and live capabilities into the manager through the
|
||||
supervisor-owned bootstrap path;
|
||||
- grant only the capabilities required for the manager's assigned
|
||||
domain. Prefer a manager already placed in that user/net namespace
|
||||
over one process holding ambient authority across every namespace;
|
||||
- make ensure/release idempotent and reference-count ownership so one
|
||||
sibling cannot tear down a tunnel still borrowed by another.
|
||||
|
||||
The root owns the manager's lifetime. `wgman` must outlive all
|
||||
siblings borrowing its tunnels and exit before the root drops the
|
||||
underlying namespace capabilities. A manager crash fails closed:
|
||||
dependent operations receive an explicit service error; restart, if
|
||||
enabled, reconciles declared state idempotently before advertising
|
||||
readiness again. Do not silently let siblings fall back to privileged
|
||||
local provisioning, since that defeats both the security boundary and
|
||||
the single warm control-plane benefit.
|
||||
|
||||
Treat eager `wgman` as a measured deployment-profile choice. Compare
|
||||
root startup with no WG declarations, pre-provisioned read-only WG,
|
||||
and runtime-managed tunnels before making it unconditional whenever
|
||||
the `wg` extra is installed. The intended invariant is "one warm
|
||||
manager per simple WG actor tree", not "every tractor program spawns
|
||||
a privileged child".
|
||||
|
||||
### 5.2 the composition
|
||||
|
||||
The maddr describes the composed network path and can be used as
|
||||
either a source/listen or destination/dial handle. It does **not**
|
||||
select the local instance of that network stack. A netns, VRF,
|
||||
interface, user namespace, or equivalent platform resource is
|
||||
orthogonal augmentation carried alongside/below the maddr.
|
||||
|
||||
Keep three bindspace representations with deliberately different roles
|
||||
and lifetimes:
|
||||
|
||||
```python
|
||||
class BindspaceSpec(msgspec.Struct, frozen=True):
|
||||
'''Serializable spawn/config declaration.'''
|
||||
kind: str # `netns`, later `vrf`, ...
|
||||
key: str|None # requested name/key, if any
|
||||
lifecycle: Literal['attach', 'open']
|
||||
|
||||
|
||||
class BindspaceRef(msgspec.Struct, frozen=True):
|
||||
'''Wire-safe, non-owning ref to the realized resource.'''
|
||||
kind: str
|
||||
key: str|None # mutable name, absent after unlink
|
||||
inode: int # host-local Linux nsfs fingerprint
|
||||
|
||||
|
||||
class Bindspace(ProcessLocal):
|
||||
'''Scoped, non-serializable capability for one live bindspace.'''
|
||||
spec: BindspaceSpec
|
||||
ref: BindspaceRef
|
||||
namespace_fd: int|None
|
||||
ownership: Literal['owned', 'borrowed']
|
||||
|
||||
|
||||
@acm
|
||||
async def open_bindspace(
|
||||
spec: BindspaceSpec,
|
||||
) -> AsyncGenerator[Bindspace, None]:
|
||||
'''
|
||||
Provision/borrow one bindspace and yield its live capability.
|
||||
|
||||
'''
|
||||
```
|
||||
|
||||
The initial model limits `BindspaceKind` to `netns` while preserving
|
||||
the required role split. `BindspaceSpec` is the requested resource and
|
||||
lifecycle policy. `BindspaceRef` is a serializable, non-owning,
|
||||
host-local record of the resource that was actually opened; it can be
|
||||
compared or logged, but cannot reopen, pin or enter that resource.
|
||||
`Bindspace` is the live capability and uses msgspec's generic struct
|
||||
storage by inheriting the global `tractor.msg.ProcessLocal` marker. Its
|
||||
hidden unsupported sentinel blocks direct and nested default msgspec
|
||||
encoding without a recursive IPC hot-path scan. The live bindspace
|
||||
validates any supplied FD against `BindspaceRef.inode`; explicit FD
|
||||
transfer belongs to the supervisor bootstrap path. An FD avoids
|
||||
name-resolution TOCTOU,
|
||||
survives rename/unlink, and identifies the exact namespace the parent
|
||||
provisioned. Extend the kind/field union only when a second platform
|
||||
resource is implemented.
|
||||
|
||||
`BindspaceSpec.lifecycle` is explicit serialized policy:
|
||||
`'attach'` borrows an existing resource and `'open'` creates/owns one.
|
||||
`open_bindspace()` dispatches that policy by bindspace kind. Never
|
||||
infer it from a listen/dial role: either role may use pre-provisioned
|
||||
or locally owned networking.
|
||||
|
||||
The first lifecycle implementation is deliberately borrow-only:
|
||||
`attach_netns()` opens either `/proc/self/ns/net` when
|
||||
`BindspaceSpec.key = CURRENT_NETNS`, or a named entry beneath
|
||||
`/var/run/netns`. It derives a `BindspaceRef` from the opened FD, yields
|
||||
`ownership='borrowed'`, and closes only that FD on exit. "Attach" does
|
||||
not call `setns()`; it never creates, enters or removes a namespace.
|
||||
Future `open_netns()` creation and owned teardown remain a separate
|
||||
privileged supervisor change.
|
||||
|
||||
`open_netns()` is that owned counterpart: it requires a named spec,
|
||||
creates through pyroute2 in a shielded worker call, attaches the live
|
||||
FD, and yields `ownership='owned'`. FD closure precedes another
|
||||
shielded pyroute2 removal call on every post-creation exit, including
|
||||
cancellation. It still never calls `setns()`; process entry remains a
|
||||
spawn/bootstrap operation.
|
||||
|
||||
`open_bindspace()` is **not** an address factory and does not return a
|
||||
`TunnelledAddress`. At the declaration layer, listener allocation can
|
||||
use the live bindspace to replace an overlay while preserving every
|
||||
tunnel:
|
||||
|
||||
```python
|
||||
async with open_bindspace(
|
||||
bindspace_spec,
|
||||
) as bindspace:
|
||||
listen_decl = declared_addr.get_random(
|
||||
bindspace=bindspace,
|
||||
)
|
||||
transport_addr = strip_tunnels(listen_decl)
|
||||
```
|
||||
|
||||
That sketch intentionally leaves the `.get_random()`/bindspace value
|
||||
contract open. A concrete transport call returns a concrete overlay;
|
||||
a declaration-level call may replace the overlay and return a new
|
||||
`TunnelledAddress`. In either case wrappers remain until the final
|
||||
transport bind/dial boundary, where `strip_tunnels()` is mandatory.
|
||||
At the listener boundary, keep the split explicit:
|
||||
`Endpoint.addr` is the peeled concrete address used for transport
|
||||
reflection, while `Endpoint.declared_addr` retains the original
|
||||
wrapper for namespace diagnostics and later bindspace orchestration.
|
||||
|
||||
Per-platform provisioning still composes one resource context per
|
||||
tunnel/bindspace layer:
|
||||
|
||||
```python
|
||||
@acm
|
||||
async def open_netns(
|
||||
spec: BindspaceSpec,
|
||||
) -> AsyncGenerator[Bindspace, None]: ...
|
||||
|
||||
@acm
|
||||
async def open_wg_iface(
|
||||
spec: WGTunnelSpec,
|
||||
config: WGInterfaceConfig,
|
||||
bindspace: Bindspace,
|
||||
role: Literal['listen', 'dial'],
|
||||
) -> AsyncGenerator[WGTunnelSpec, None]: ...
|
||||
```
|
||||
|
||||
`WGInterfaceConfig` and each `WGPeerConfig` are process-local and
|
||||
rejected by the global `ProcessLocal` wire guard. The interface config
|
||||
owns its private key, local addresses and listen port; each peer owns
|
||||
its public key, allowed CIDRs, optional endpoint, preshared key and
|
||||
keepalive. Reprs redact private/preshared keys. `WGTunnelSpec` remains
|
||||
serializable public maddr-derived identity/endpoint data. This split
|
||||
supports multi-peer listeners without overloading the tunnel maddr.
|
||||
|
||||
The initial `open_wg_iface()` lifecycle is owned and Linux-only. It
|
||||
validates role-dependent bearer policy before side effects, creates the
|
||||
iface and addresses through `IPRoute`, configures keys/peers through
|
||||
`WireGuard`, raises the link, and removes it on every post-creation
|
||||
exit. Creation/removal run in shielded Trio worker calls. A listen
|
||||
bearer supplies the local listen port; a dial bearer supplies an
|
||||
omitted endpoint only for the selected maddr peer.
|
||||
|
||||
The composition driver folds a list of specs into nested contexts with
|
||||
`contextlib.AsyncExitStack` for the N-deep case. The
|
||||
`parse_endpoints()` API (`_multiaddr.py:153`) is the front door:
|
||||
it already returns
|
||||
`dict[name, list[Address|TunnelledAddress]]` and the
|
||||
`multiaddr_declare_eps.md` sketch anticipates the recursive
|
||||
`dict[str, list[Address]]|dict[...]` return for tunnelled
|
||||
entries. Extend it to carry the tunnel stack, not to *enter* it.
|
||||
|
||||
`open_wg_bindspace()` is the initial driver for one bindspace and an
|
||||
ordered sequence of `(WGTunnelSpec, WGInterfaceConfig)` layers. It
|
||||
opens the bindspace first, enters WG interfaces outermost-first through
|
||||
`AsyncExitStack`, and yields the live `Bindspace` for endpoint
|
||||
allocation. Exit is inside-out, so every interface is removed while the
|
||||
namespace FD remains pinned; only then can an owned namespace be
|
||||
removed. Endpoint/channel lifetimes belong inside the yielded scope.
|
||||
|
||||
The caller supplies `role` to tunnel-resource contexts such as
|
||||
`open_wg_iface()`; do not infer it from maddr shape. Bindspace
|
||||
lifecycle remains the independent explicit policy above. The same
|
||||
composed maddr can name a server source or client destination (§5.4).
|
||||
|
||||
### 5.3 `Address.namespace`, at last
|
||||
|
||||
- an unrealized `TunnelledAddress.namespace` reports its declared name
|
||||
as `(kind, key)`, e.g. `('netns', 'tractor-wg0')`;
|
||||
- `TunnelledAddress.with_bindspace_ref()` returns a frozen declaration
|
||||
annotated with `bindspace.ref`, never the FD-bearing `Bindspace`.
|
||||
Its `.namespace` reports `(kind, inode)` so the
|
||||
realized ref remains stable across rename or unlink;
|
||||
- existing plain backends implement it explicitly as `None`, so the
|
||||
Protocol does not lie and tunnel delegation needs no `getattr()`
|
||||
fallback.
|
||||
- `Endpoint.namespace` reads the retained declaration rather than its
|
||||
peeled transport addr; both `Endpoint.pformat()` and
|
||||
`Server.pformat()` expose that value as the cheapest proof the layer
|
||||
is wired.
|
||||
|
||||
Deferred follow-ups:
|
||||
|
||||
- add native tagged encoding for the complete `TunnelledAddress` graph,
|
||||
including its concrete overlay-address union, tunnel-spec union and
|
||||
optional `BindspaceRef`. Once that codec exists, tests should perform
|
||||
typed roundtrips instead of inspecting an untyped decoded payload.
|
||||
|
||||
Use `github/ns_aware@e4688cad` as prototype evidence, not code to
|
||||
cherry-pick unchanged. Its `/proc/<pid>/ns/<type>` inode reader and
|
||||
`ip netns identify` probe establish the useful `(key, inode)` reference
|
||||
record. Layer C should move that shape into `BindspaceRef`, avoid a
|
||||
subprocess where netlink/procfs suffices, and hold the namespace FD in
|
||||
`Bindspace` to pin the referenced resource.
|
||||
|
||||
### 5.4 the netns/process reality — read this before designing
|
||||
|
||||
**The headline consequence, stated up front**: netns is a
|
||||
**runtime-level config API, not an actor-app-code API.** It is
|
||||
declared as part of how an actor process is *brought up* — a
|
||||
spawn-time/boot-time input alongside `enable_transports` and
|
||||
`tpt_bind_addrs` — and it is **not** dynamically re-enterable by
|
||||
app code once the actor is live. There is deliberately no
|
||||
`await actor.enter_netns(...)`. Two hard reasons, both below:
|
||||
`setns(2)` doesn't retroactively move existing sockets, and it's
|
||||
per-thread rather than per-process. Anything that *looks* like a
|
||||
mid-life API here would be a footgun that silently leaves the IPC
|
||||
server bound in the old namespace.
|
||||
|
||||
- `setns(2)` with `CLONE_NEWNET` affects **the calling thread
|
||||
only**, and sockets already created keep their original netns.
|
||||
A trio actor is effectively single-threaded for our purposes,
|
||||
so "enter the netns, *then* bind" works — but any
|
||||
`to_thread` worker (§4.1 option 1!) is in the **original**
|
||||
netns unless it also `setns`. Concretely: a wg query issued
|
||||
via `trio.to_thread` will hit the wrong namespace. Either
|
||||
pass `netns=` down to `pyroute2` (which does the
|
||||
fork/setns dance itself) or pin a dedicated worker. **This is
|
||||
the single subtlest bug in this plan — write the test first.**
|
||||
- entering a netns is *process-global-ish and irreversible-ish*
|
||||
in practice. Therefore: **netns membership belongs to the
|
||||
actor process, decided before the runtime binds**, not to a
|
||||
mid-life actor API. Design:
|
||||
- the root/parent decides the `BindspaceSpec`, provisions or
|
||||
borrows it, and passes the spec plus an inherited/transferred
|
||||
namespace-FD capability through the spawn backend (there's already
|
||||
`enable_transports`/`accept_addrs` plumbing at
|
||||
`_runtime.py:1595-1615` — the netns rides alongside).
|
||||
- the child spawn/bootstrap trampoline calls `setns()` **before**
|
||||
`_runtime.async_main()`, `IPCServer.listen_on()`, parent-channel
|
||||
connection, or creation of any worker thread/socket.
|
||||
- `spawn._netns.enter_netns()` is the first private bootstrap
|
||||
primitive: it checks the inherited FD against the expected inode,
|
||||
calls `setns(fd, CLONE_NEWNET)`, and verifies
|
||||
`/proc/self/ns/net` before returning. It deliberately does not own
|
||||
or close the FD; spawn propagation and status reporting remain the
|
||||
caller's next integration boundary.
|
||||
- only after successful entry does the child drop namespace-entry
|
||||
privileges and initialize the actor runtime.
|
||||
- a root/single-actor process follows the same ordering: enter during
|
||||
root bootstrap, never after actor runtime startup.
|
||||
- iface/route/WG provisioning is genuinely scoped and remains under
|
||||
the parent/supervisor's `Bindspace` context.
|
||||
- document the constraint rather than hiding it; a
|
||||
`RuntimeError` if namespace entry is attempted after bootstrap.
|
||||
- capabilities: iface/netns creation/config needs `CAP_NET_ADMIN`;
|
||||
entering an existing Linux namespace normally requires
|
||||
`CAP_SYS_ADMIN` in the owning user namespace. Never `sudo` from
|
||||
inside the runtime. A privileged parent/helper should provision the
|
||||
stack and open the namespace FD; the child receives only the scoped
|
||||
capability and temporary authority needed to enter it, then drops
|
||||
that authority before actor code runs. This separates create/config
|
||||
authority from enter/use authority and fits user-namespace/capability
|
||||
deployments without granting every actor broad ambient caps.
|
||||
Two supported modes remain:
|
||||
(i) pre-provisioned out-of-band (layers A/B — the default,
|
||||
and what #482 documents), (ii) runtime-managed when the supervising
|
||||
process/helper holds the required caps. Probe exact required caps and
|
||||
*fail loudly with an actionable message* otherwise.
|
||||
- role semantics are explicit:
|
||||
- `listen`: may create/own the local bindspace, iface, routes, WG
|
||||
peer/listener state, and random local overlay; lifetime normally
|
||||
extends through all listeners and the actor process.
|
||||
- `dial`: may borrow an actor-wide bindspace or ensure local routing
|
||||
and tunnel state reaches the remote stack; it does not own the
|
||||
remote maddr and may need no new local resource at all.
|
||||
- source/destination use is an operation property, never permanently
|
||||
encoded into the maddr or inferred from segment ordering.
|
||||
- teardown follows capability ownership, not just address type:
|
||||
- owned listener bindspaces tear down after endpoints/channels and
|
||||
the actor process have exited;
|
||||
- borrowed dial/actor-wide bindspaces only release their capability;
|
||||
- nested resources exit inside-out, but shared resources remain until
|
||||
their owning supervisor drops the final capability.
|
||||
- teardown must be idempotent and tolerant: an iface/netns
|
||||
already gone must not strand the rest of the teardown — the
|
||||
exact lesson `_uds.close_listener()`'s `FileNotFoundError`
|
||||
tolerance and `_serve_ipc_eps()`'s per-ep `try/except`
|
||||
encode. Mirror both.
|
||||
|
||||
### 5.5 tests for layer C
|
||||
|
||||
- unit: fold-N-tunnel-specs-into-nested-`@acm`s, with fakes; assert
|
||||
enter/exit ordering (outermost-last-out) via a trace list.
|
||||
- integration, gated on `CAP_NET_ADMIN` (skip otherwise, and in
|
||||
CI run it in a `--cap-add NET_ADMIN` container job): create two
|
||||
netns + a wg pair entirely in-process, boot a `tractor` root in
|
||||
one and a subactor in the other, `find_actor()` across the
|
||||
tunnel. This is a *fantastic* test to have and is fully
|
||||
self-contained — no second host, no `sudo` in the test body.
|
||||
- the `to_thread`-netns-mismatch regression from §5.4, written
|
||||
**first** (red), then the fix (green), per project convention.
|
||||
- bootstrap ordering: assert the child reports the expected namespace
|
||||
inode before parent-channel connect and listener creation.
|
||||
- FD capability: rename/unlink the namespace name after opening its FD
|
||||
and prove child entry still selects the pinned inode.
|
||||
- privilege drop: prove actor code lacks provisioning caps after entry.
|
||||
- role/ownership: fake listen/dial resources and assert owned listener
|
||||
teardown versus borrowed dial-handle release.
|
||||
- `wgman` bootstrap: prove sibling process startup overlaps manager
|
||||
reconciliation while the first WG operation still waits for its
|
||||
readiness signal.
|
||||
- `wgman` authority: reject arbitrary callers/operations and prove an
|
||||
unprivileged sibling cannot receive secrets, FDs or provisioning
|
||||
authority through the manager API.
|
||||
- `wgman` lifetime: prove it outlives tunnel borrowers, fails pending
|
||||
requests explicitly on crash and reconciles before restart-ready.
|
||||
|
||||
---
|
||||
|
||||
## 6. "Other shuttle-able tpts"
|
||||
|
||||
The generalization the #482 follow-up gestures at: once
|
||||
`TunnelledAddress` + `open_bindspace()` exist, the same
|
||||
machinery covers any iface-layer tunnel `pyroute2` can drive —
|
||||
`ipip`/`gre`/`sit`/`vxlan`/`geneve`/`bridge`/`veth`. Keep
|
||||
`WGTunnelSpec` as *one* frozen struct among a
|
||||
`TunnelSpec = WGTunnelSpec|VxlanTunnelSpec|...` union with a
|
||||
`kind: ClassVar[str]`, and dispatch `open_*` by `match` on it.
|
||||
Design for it now (union + `match`), implement only `wg` +
|
||||
`netns`. `veth`-pairs-in-netns is the natural second one because
|
||||
it makes the §5.5 integration test possible without wg at all —
|
||||
consider doing it *first* for exactly that reason.
|
||||
|
||||
## 7. Non-goals
|
||||
|
||||
- no wg userspace implementation, no key exchange, no
|
||||
`wg-quick` reimplementation (config-file parsing is
|
||||
out of scope; take structured input).
|
||||
- no persistence of private keys beyond what layer C's iface
|
||||
creation needs (and that stays in `get_rt_dir()`, 0600).
|
||||
- macOS/Windows: layers B/C are Linux-only. Layer A (declarative)
|
||||
works anywhere `wg` does. Gate accordingly and say so in the
|
||||
docs — do not silently no-op.
|
||||
|
||||
## 8. Risks
|
||||
|
||||
| risk | mitigation |
|
||||
| --- | --- |
|
||||
| `to_thread` worker runs in the wrong netns | §5.4; pass `netns=` to pyroute2 or pin a worker; test-first |
|
||||
| namespace name is renamed/replaced between provision and spawn | pass an open namespace FD; verify `(key, inode)` after child entry |
|
||||
| child starts sockets/threads before `setns()` | enter in the spawn bootstrap trampoline before `_runtime.async_main()`; assert inode ordering |
|
||||
| ambient capabilities leak into actor app code | split provision/enter authority and drop caps before runtime initialization |
|
||||
| dial path tears down a shared actor bindspace | encode ownership in `Bindspace`; borrowed bindspaces never remove resources |
|
||||
| py-multiaddr#108 merged but unreleased | PEP 621 direct-revision pin + `_wg_proto_code()` gate; replace with a release floor once published |
|
||||
| `TunnelledAddress` leaks into transport reflection/type dispatch | keep wrappers through declaration/bindspace handling, call `strip_tunnels()` at channel/endpoint boundaries, and retain the boundary regressions |
|
||||
| privileged ops in a library | never `sudo`; explicit cap probe + actionable error; pre-provisioned is the default |
|
||||
| pyroute2 0.9 asyncio core drags a loop into every actor | use a worker for one-shots; confine persistent asyncio to an infected `wgman` (§4.1, §5.1) |
|
||||
| eager `wgman` serializes or slows root bootstrap | spawn it in parallel; gate only WG-dependent operations on readiness; measure before making the profile unconditional |
|
||||
| `wgman` becomes a cluster-wide privilege oracle | keep it private/scoped, expose structured verbs only and split managers by capability domain |
|
||||
| netns teardown strands actor teardown | idempotent/tolerant teardown mirroring `_uds.close_listener()` |
|
||||
|
||||
## 9. Follow-up issue seeds
|
||||
|
||||
- `veth`-in-netns bindspace (unblocks capless-ish integration
|
||||
testing, and is a great local multi-"host" test rig)
|
||||
- composed/tunnelled maddr grammar shared with plan 02's
|
||||
`/…/quic-v1/…` stacks (gh #443)
|
||||
- `wg` proto into the multiaddr **spec** (gh #483), then flip
|
||||
`MsgTransport.maddr` to always return `Multiaddr` (the third
|
||||
#443 bullet)
|
||||
- first-child `wgman` prototype: concurrent bootstrap, scoped sibling
|
||||
access, infected-asyncio pyroute2 ownership and restart reconciliation
|
||||
- runtime-managed wg key rotation / peer add-remove through `wgman` —
|
||||
the natural "actor that owns the network" demo
|
||||
|
|
@ -0,0 +1,54 @@
|
|||
# next-gen `tractor.ipc` transport backend plans
|
||||
|
||||
Implementation specs for three prospective `.ipc` transport
|
||||
backends, written so each can be worked independently (by a
|
||||
different model/provider) without design or lib-selection drift.
|
||||
|
||||
**Read [`00_shared_backend_contract.md`](./00_shared_backend_contract.md)
|
||||
first** — it is the normative description of what a `tractor`
|
||||
transport backend *is* as of `main@83b34884` (the backend
|
||||
duck-type, the 10-item registration checklist, the test-harness
|
||||
plumbing, the code-style rules). The three plans assume it and
|
||||
document only their own deltas.
|
||||
|
||||
| plan | issue | dep | size | lands |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| [01 — TIPC](./01_tipc_backend.md) | [#378] | **none** (stdlib) | small | first |
|
||||
| [02 — QUIC/`iroh`](./02_quic_iroh_backend.md) | [#353] | `iroh` (uniffi FFI) | large | needs a prep PR |
|
||||
| [03 — `wg` bindspace](./03_wg_tunnel_bindspace.md) | [#482], [#443] | `pyroute2` | medium, 3 layers | layer A now |
|
||||
|
||||
Headline conclusions:
|
||||
|
||||
- **TIPC is the cheap win.** Verified: `trio.SocketStream` and
|
||||
`trio.SocketListener` are address-family agnostic (only
|
||||
`SOCK_STREAM` + a trio socket), and CPython ships `AF_TIPC` +
|
||||
23 `TIPC_*` constants. So the backend is ~one module of
|
||||
contract boilerplate, zero new deps, and it buys
|
||||
*kernel-native* service discovery: `bind()` publishes,
|
||||
`connect()`-by-name resolves — no registrar in the loop.
|
||||
(`modprobe tipc` is required; hard-gate everything.)
|
||||
- **QUIC's cost is entirely in two adapters**, not in QUIC. The
|
||||
`iroh` python bindings are `uniffi`-generated asyncio, but the
|
||||
asyncio dependency is confined to *one* future-poll callback —
|
||||
a ~40-line `trio` bridge (`TrioToken.run_sync_soon`) replaces
|
||||
it. The second cost is that an iroh listener isn't a socket,
|
||||
which needs a small, independently-reviewable prep PR to
|
||||
`_server.py`/`_types.py`.
|
||||
- **WireGuard is not a transport.** It's an iface-layer tunnel,
|
||||
so it belongs as a *nested bindspace* (`TunnelledAddress` +
|
||||
`open_bindspace()` `@acm`s) wrapping whatever L4 tpt is in
|
||||
use — which is also what finally implements the long-spec'd
|
||||
`Address.namespace`, and what generalizes to
|
||||
`veth`/`vxlan`/`gre`.
|
||||
|
||||
Ordering rationale: plan 01 first as the cheap proof the
|
||||
table-registration story generalizes to a genuinely new proto;
|
||||
plan 03 layer A is already deployable-today doc/example work;
|
||||
plan 02 last (and gated on its prep PR). Plans 01 and 02 both
|
||||
want the same `Address.rebind_from_sockname` gate — whichever
|
||||
lands first ships it.
|
||||
|
||||
[#378]: https://github.com/goodboy/tractor/issues/378
|
||||
[#353]: https://github.com/goodboy/tractor/issues/353
|
||||
[#482]: https://github.com/goodboy/tractor/issues/482
|
||||
[#443]: https://github.com/goodboy/tractor/issues/443
|
||||
|
|
@ -30,9 +30,9 @@ the registry tracks the live tree as it grows and shrinks.
|
|||
|
||||
.. note::
|
||||
Actor names are **not** enforced unique — the registry is keyed
|
||||
by the full ``(name, uuid)`` pair. Name-based lookups simply
|
||||
resolve to the *last* registered match, so if you boot five
|
||||
actors all named ``'bob'``, you get the freshest ``'bob'`` B)
|
||||
by the full ``(name, uuid)`` pair. A name lookup returns one
|
||||
matching registration, but the API does not promise which match
|
||||
wins. Use unique service names when selection matters.
|
||||
|
||||
First boot: who's the registrar?
|
||||
--------------------------------
|
||||
|
|
@ -65,6 +65,43 @@ the one-and-only registrar; boot then fails loudly with a
|
|||
``RuntimeError`` if some other process already bound the registry
|
||||
socket(s).
|
||||
|
||||
A dedicated registrar
|
||||
---------------------
|
||||
That second rule — *"if a registrar answers, boot as a plain
|
||||
root"* — is all you need to run the registry as its own
|
||||
**standalone process**, decoupled from any app tree's root. In the
|
||||
daemon process, enter ``open_root_actor()`` with an explicit
|
||||
``registry_addrs`` and ``ensure_registry=True``; the latter makes
|
||||
startup fail instead of silently joining a registrar that won the
|
||||
address. Point each app tree at the address that daemon actually
|
||||
bound:
|
||||
|
||||
.. literalinclude:: ../../examples/discovery/dedicated_registrar.py
|
||||
:caption: examples/discovery/dedicated_registrar.py
|
||||
:language: python
|
||||
|
||||
The example's selector socket binds but deliberately never listens.
|
||||
It owns the kernel-selected local address only long enough to read it,
|
||||
then closes so Tractor's actual listener can bind the same address.
|
||||
This is not a socket transfer: the close/rebind handoff is non-atomic,
|
||||
so the example retries with a fresh candidate only when registrar
|
||||
startup reports that another process claimed the released address.
|
||||
Retries are bounded, and other startup failures remain visible. It
|
||||
publishes the selected address only after the actor context enters.
|
||||
It also performs the lookup inside a separate ``client`` actor. The
|
||||
service is its sibling, not its child, so the client has no spawn-time
|
||||
service channel to satisfy the local-peer fast path. The
|
||||
``query_actor()`` assertion verifies that a registrar portal handled
|
||||
the lookup before ``find_actor()`` makes the service RPC.
|
||||
|
||||
This is the "registrar as a subsystem, not the app root actor"
|
||||
shape. Two caveats today (both tracked as #472 follow-ups):
|
||||
``enable_transports`` is single-proto per runtime, so a registrar
|
||||
can't yet serve multiple backends at once; and there's no way to
|
||||
spawn a registrar as a *sub*-actor of a shared tree (only as its
|
||||
own root), since ``start_actor()`` has no custom-``actor_cls``
|
||||
hook.
|
||||
|
||||
Looking up actors
|
||||
-----------------
|
||||
|
||||
|
|
@ -92,13 +129,22 @@ Knobs worth knowing:
|
|||
- ``registry_addrs=[...]``: query specific (possibly multiple,
|
||||
possibly remote) registrars instead of your tree's default,
|
||||
|
||||
- ``only_first=False``: deliver a ``list[Portal]`` of *all*
|
||||
matches found across the queried registrars instead of just the
|
||||
first,
|
||||
- ``only_first=True``: after all configured registrars are queried
|
||||
concurrently, yield the result in the first ``registry_addrs``
|
||||
position. This is configured order, not first-reachable order, so
|
||||
the result can be ``None`` even when a later registrar returned a
|
||||
portal,
|
||||
|
||||
- ``raise_on_none=True``: raise a ``RuntimeError`` instead of
|
||||
yielding ``None`` when no match is found — for when absence is
|
||||
a hard error in your app.
|
||||
- ``only_first=False``: when any query succeeds, yield an ordered
|
||||
``list[Portal | None]`` with one result per ``registry_addrs``
|
||||
position; misses remain ``None`` placeholders. When every query
|
||||
misses, yield ``None`` instead of a list. This does not enumerate
|
||||
every duplicate name in one registrar,
|
||||
|
||||
- ``raise_on_none=True``: raise a ``RuntimeError`` when every
|
||||
registrar query returns ``None``. With ``only_first=True`` it does
|
||||
not raise merely because the first ordered result is ``None`` when
|
||||
a later result is a portal.
|
||||
|
||||
``wait_for_actor()``
|
||||
********************
|
||||
|
|
@ -131,10 +177,11 @@ Yields a portal straight to the registrar actor itself — or a
|
|||
Fast paths and address preference
|
||||
---------------------------------
|
||||
|
||||
Before doing any RPC to the registrar, every lookup first scans
|
||||
the calling actor's *already-connected peers*: if you have a live
|
||||
channel to an actor named ``name`` you get a portal over it
|
||||
immediately, no registrar round-trip at all.
|
||||
Before doing any RPC to the registrar, ``query_actor()``,
|
||||
``wait_for_actor()``, and the default ``find_actor()`` lookup first
|
||||
scan the calling actor's *already-connected peers*. If the caller
|
||||
has a live channel to an actor named ``name``, it gets a portal over
|
||||
that channel immediately, with no registrar round-trip.
|
||||
|
||||
When a registry entry holds *multiple* addresses (a multihomed
|
||||
actor) the "best" one is chosen by locality:
|
||||
|
|
@ -201,8 +248,8 @@ the existing registrar:
|
|||
|
||||
Per the bootstrap rules above, if those addrs are absent this process
|
||||
becomes its own registrar root, so the same code works standalone and
|
||||
as a tree-joiner. An occupied address that does not complete a Tractor
|
||||
registrar handshake fails startup instead of being rebound.
|
||||
as a tree-joiner. An occupied address that does not complete a
|
||||
Tractor registrar handshake fails startup instead of being rebound.
|
||||
|
||||
"Arbiter"? A legacy naming note
|
||||
-------------------------------
|
||||
|
|
@ -226,10 +273,11 @@ Very naive, very honest
|
|||
-----------------------
|
||||
|
||||
To be clear, this is a **very naive** discovery system: one
|
||||
process-tree-local registrar holding a dict, no replication, no
|
||||
re-election when it dies, no cross-host propagation. That's
|
||||
intentional (for now); it covers the "wire up my services on this
|
||||
host" case without dragging in a consensus protocol.
|
||||
in-memory registrar holding a dict, no replication, no re-election
|
||||
when it dies, and no automatic cross-host propagation. Separate
|
||||
programs can use the same reachable registrar, as above, but must be
|
||||
configured with its address. That's intentional (for now); it covers
|
||||
the "wire up my services" case without a consensus protocol.
|
||||
|
||||
On the roadmap (issue `#216`_ tracks a chunk of it):
|
||||
|
||||
|
|
@ -254,6 +302,7 @@ to hear from you.
|
|||
:class:`tractor.Registrar`.
|
||||
|
||||
.. _gossip protocol: https://en.wikipedia.org/wiki/Gossip_protocol
|
||||
.. _modern protocol: https://en.wikipedia.org/wiki/Rendezvous_protocol
|
||||
.. _modern protocol:
|
||||
https://en.wikipedia.org/wiki/Rendezvous_protocol
|
||||
.. _discovery: https://zguide.zeromq.org/docs/chapter8/#Discovery
|
||||
.. _#216: https://github.com/goodboy/tractor/issues/216
|
||||
|
|
|
|||
|
|
@ -153,14 +153,16 @@ the high-rate stream path.
|
|||
never even hits the wire. (You can opt out per-call with
|
||||
``ctx.started(..., validate_pld_spec=False)`` if you measure
|
||||
a real cost.)
|
||||
- ``Yield`` payloads are **never** checked inside
|
||||
``MsgStream.send()``; they're validated receiver-side on each
|
||||
``MsgStream.receive()``. A violation raises a ``MsgTypeError``
|
||||
in the receiver *and* relays an ``Error`` msg back so the
|
||||
offending sender gets one raised too.
|
||||
- the remaining control msgs (``Start``, ``Return``) are likewise
|
||||
validated such that violations raise in the **sending** actor,
|
||||
pointing the traceback at the code that actually goofed.
|
||||
- ``Yield`` and ``Return`` payloads are not checked before sending;
|
||||
they're decoded against the dialog's spec by the receiver. A
|
||||
violation raises a ``MsgTypeError`` there and terminates that
|
||||
dialog. The peer then observes the resulting protocol teardown;
|
||||
it is not guaranteed to receive the same ``MsgTypeError``.
|
||||
- ``Start`` arguments are dispatched through the RPC endpoint's
|
||||
Python signature. They are not payloads covered by the dialog's
|
||||
``pld_spec``. A planned follow-up will derive a typed ``Start``
|
||||
contract from endpoint annotations and validate arguments
|
||||
sender-side; see `#514`_.
|
||||
|
||||
Anatomy of a ``MsgTypeError``
|
||||
-----------------------------
|
||||
|
|
@ -177,21 +179,21 @@ a msg fails to decode against the active spec. The useful bits:
|
|||
``.src_uid``, ``.ipc_msg`` and the fancy ``.pformat()`` tb-box
|
||||
rendering.
|
||||
|
||||
Practical reading guide: a *sender-side* MTE (``Started``,
|
||||
``Return``) points straight at your offending ``await
|
||||
ctx.started()`` or ``return`` statement, while a *receiver-side*
|
||||
MTE (``Yield``) surfaces from the consumer's ``receive()`` call
|
||||
with the relay copy delivered back to the producer. Either way
|
||||
the failure is scoped to that one dialog; sibling contexts on the
|
||||
same channel keep right on trucking.
|
||||
Practical reading guide: a *sender-side* MTE for ``Started`` points
|
||||
straight at the offending ``await ctx.started()`` call. A
|
||||
*receiver-side* MTE for ``Yield`` or ``Return`` surfaces while the
|
||||
peer decodes the payload. Either way the failure is scoped to that
|
||||
one dialog; sibling contexts on the same channel keep right on
|
||||
trucking.
|
||||
|
||||
Custom wire types: ``mk_codec()`` and friends
|
||||
---------------------------------------------
|
||||
msgspec covers a wide set of `builtin types`__ natively; for
|
||||
anything else you teach the codec via extension hooks. The
|
||||
easiest path is per-endpoint: ``@tractor.context()`` accepts
|
||||
``enc_hook``/``dec_hook`` params right alongside ``pld_spec``.
|
||||
For full control build and apply a codec yourself; encode-side:
|
||||
anything else you teach the codec via extension hooks. The complete
|
||||
public path currently available is task-scoped encoding:
|
||||
``tractor.msg.mk_codec()`` builds a codec with an ``enc_hook``, and
|
||||
``tractor.msg.apply_codec()`` installs it for the current task. To
|
||||
build and apply that transport codec:
|
||||
|
||||
__ https://jcristharif.com/msgspec/supported-types.html
|
||||
|
||||
|
|
@ -206,8 +208,9 @@ __ https://jcristharif.com/msgspec/supported-types.html
|
|||
with apply_codec(codec): # ContextVar-scoped override
|
||||
... # msgs sent by this task now encode NSPs
|
||||
|
||||
and decode-side, scoped to an open context (note the import from
|
||||
``tractor.msg._ops``, not yet re-exported):
|
||||
The context manager which temporarily installs payload-decoder
|
||||
settings on an open context is separate and still private (note
|
||||
the ``tractor.msg._ops`` import):
|
||||
|
||||
.. code:: python
|
||||
|
||||
|
|
@ -220,11 +223,15 @@ and decode-side, scoped to an open context (note the import from
|
|||
):
|
||||
... # this dialog's payloads decode as NSPs
|
||||
|
||||
``apply_codec()`` is ``ContextVar``-scoped: it overrides the
|
||||
codec for the current task (and only that task), not the whole
|
||||
process. For complete working flows, including hook pairing rules
|
||||
and roundtrip cases, see ``tests/msg/test_ext_types_msgspec.py``
|
||||
and ``tests/msg/test_pldrx_limiting.py``.
|
||||
``apply_codec()`` is ``ContextVar``-scoped: it overrides the codec
|
||||
for the current task (and only that task), not the whole process.
|
||||
``@tractor.context()`` accepts ``enc_hook`` and ``dec_hook``
|
||||
parameters, but their runtime wiring is not yet a symmetric,
|
||||
end-to-end public hook pair: the encode hook is not consumed and
|
||||
the decode hook is not applied on both peers. For the working flows
|
||||
and their current boundaries, see
|
||||
``tests/msg/test_ext_types_msgspec.py`` and
|
||||
``tests/msg/test_pldrx_limiting.py``.
|
||||
|
||||
The runtime dogfoods this pattern with
|
||||
:class:`tractor.msg.NamespacePath`: a ``str``-subtype shaped like
|
||||
|
|
@ -238,9 +245,33 @@ Toward capability-based msging
|
|||
The ``pld_spec`` + codec-hook layer is the foundation for the
|
||||
long-game: **capability-based msging** where each dialog's
|
||||
type contract doubles as a capability grant, negotiated as part
|
||||
of the protocol itself. That work is tracked in `#196`_ (with the
|
||||
original typed-proto epic in `#36`_); if strongly-typed
|
||||
distributed systems get you going, we'd love your input.
|
||||
of the protocol itself. The epic is tracked in `#196`_ (evolving
|
||||
the original typed-proto work in `#36`_), and the most recent
|
||||
concrete step is `#365`_ — driving the whole ``pld_spec`` off
|
||||
plain type-annotations (e.g. annotating a context's
|
||||
``open_stream()`` with ``msgspec.Struct`` subtypes) instead of
|
||||
explicit ``pld_spec=`` kwargs.
|
||||
|
||||
You don't have to wait for that, though: the decorator-level
|
||||
``@tractor.context(pld_spec=...)`` shown above is already the
|
||||
*higher-level* way to pin a dialog's payload contract, while
|
||||
``tractor.msg._ops.limit_plds()`` is the lower-level, per-block
|
||||
escape hatch. Both are exercised end-to-end in
|
||||
``tests/msg/test_pldrx_limiting.py`` and
|
||||
``tests/msg/test_ext_types_msgspec.py``.
|
||||
|
||||
The codec constructor and task-scoped override are public; the
|
||||
per-dialog decoder override remains private, and the decorator hook
|
||||
parameters remain incomplete. `#376`_ (from
|
||||
`@guilledk <https://github.com/guilledk>`_, on the
|
||||
`auto_codecs <https://github.com/goodboy/tractor/tree/auto_codecs>`_
|
||||
branch) instead drafts pair-building factories which derive
|
||||
matching ``enc_hook``/``dec_hook`` functions and encoder/decoder
|
||||
pairs from a type spec. That automation, not hook availability,
|
||||
is the proposed long-term home for custom-type (de)serialization.
|
||||
|
||||
If strongly-typed distributed systems get you going, we'd love
|
||||
your input on any of the above.
|
||||
|
||||
Where to next?
|
||||
--------------
|
||||
|
|
@ -258,3 +289,6 @@ Where to next?
|
|||
.. _(un)protocol: https://zguide.zeromq.org/docs/chapter7/#Unprotocols
|
||||
.. _#196: https://github.com/goodboy/tractor/issues/196
|
||||
.. _#36: https://github.com/goodboy/tractor/issues/36
|
||||
.. _#365: https://github.com/goodboy/tractor/issues/365
|
||||
.. _#376: https://github.com/goodboy/tractor/pull/376
|
||||
.. _#514: https://github.com/goodboy/tractor/issues/514
|
||||
|
|
|
|||
|
|
@ -1,28 +1,40 @@
|
|||
import trio
|
||||
import tractor
|
||||
|
||||
_this_module = __name__
|
||||
the_line = 'Hi my name is {}'
|
||||
_this_module: str = __name__
|
||||
the_line: str = 'Hi my name is {}'
|
||||
|
||||
|
||||
tractor.log.get_console_log("INFO")
|
||||
tractor.log.get_console_log('INFO')
|
||||
|
||||
|
||||
async def hi():
|
||||
async def hi() -> str:
|
||||
'''
|
||||
Return a greeting naming the current actor.
|
||||
|
||||
'''
|
||||
return the_line.format(tractor.current_actor().name)
|
||||
|
||||
|
||||
async def say_hello(other_actor):
|
||||
async def say_hello(other_actor: str) -> str:
|
||||
'''
|
||||
Ask another actor to return its greeting.
|
||||
|
||||
'''
|
||||
portal: tractor.Portal
|
||||
async with tractor.wait_for_actor(other_actor) as portal:
|
||||
return await portal.run(hi)
|
||||
|
||||
|
||||
async def main():
|
||||
"""Main tractor entry point, the "master" process (for now
|
||||
async def main() -> None:
|
||||
'''
|
||||
Main tractor entry point, the "master" process (for now
|
||||
acts as the "director").
|
||||
"""
|
||||
|
||||
'''
|
||||
an: tractor.ActorNursery
|
||||
async with tractor.open_nursery() as an:
|
||||
print("Alright... Action!")
|
||||
print('Alright... Action!')
|
||||
|
||||
# both actors wait on (then dial!) the *other*, so each
|
||||
# must outlive both hellos: spawn as daemons, run the
|
||||
|
|
@ -39,6 +51,10 @@ async def main():
|
|||
name: str,
|
||||
other_actor: str,
|
||||
) -> None:
|
||||
'''
|
||||
Print a greeting fetched through a named actor.
|
||||
|
||||
'''
|
||||
print(
|
||||
# RPC through an existing actor's `Portal`.
|
||||
await portals[name].run(
|
||||
|
|
@ -47,13 +63,14 @@ async def main():
|
|||
)
|
||||
)
|
||||
|
||||
tn: trio.Nursery
|
||||
async with trio.open_nursery() as tn:
|
||||
tn.start_soon(run_and_print, 'donny', 'gretchen')
|
||||
tn.start_soon(run_and_print, 'gretchen', 'donny')
|
||||
|
||||
await an.cancel()
|
||||
|
||||
print("CUTTTT CUUTT CUT!!! Donny!! You're supposed to say...")
|
||||
print('CUTTTT CUUTT CUT!!! Donny!! You\'re supposed to say...')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
|
|
|
|||
|
|
@ -2,14 +2,20 @@ import trio
|
|||
import tractor
|
||||
|
||||
|
||||
async def cellar_door():
|
||||
async def cellar_door() -> str:
|
||||
'''
|
||||
Return a phrase from a spawned actor.
|
||||
|
||||
'''
|
||||
assert not tractor.is_root_process()
|
||||
return "Dang that's beautiful"
|
||||
return 'Dang that\'s beautiful'
|
||||
|
||||
|
||||
async def main():
|
||||
"""The main ``tractor`` routine.
|
||||
"""
|
||||
async def main() -> None:
|
||||
'''
|
||||
The main ``tractor`` routine.
|
||||
|
||||
'''
|
||||
# spawn a subactor, run ``cellar_door()`` as its lone task,
|
||||
# block until its result arrives and the subactor is reaped.
|
||||
print(
|
||||
|
|
|
|||
|
|
@ -2,19 +2,24 @@ import trio
|
|||
import tractor
|
||||
|
||||
|
||||
async def movie_theatre_question():
|
||||
"""A question asked in a dark theatre, in a tangent
|
||||
async def movie_theatre_question() -> str:
|
||||
'''
|
||||
A question asked in a dark theatre, in a tangent
|
||||
(errr, I mean different) process.
|
||||
"""
|
||||
|
||||
'''
|
||||
return 'have you ever seen a portal?'
|
||||
|
||||
|
||||
async def main():
|
||||
"""The main ``tractor`` routine.
|
||||
"""
|
||||
async def main() -> None:
|
||||
'''
|
||||
The main ``tractor`` routine.
|
||||
|
||||
'''
|
||||
an: tractor.ActorNursery
|
||||
async with tractor.open_nursery() as an:
|
||||
|
||||
portal = await an.start_actor(
|
||||
portal: tractor.Portal = await an.start_actor(
|
||||
'frank',
|
||||
# enable the actor to run funcs from this current module
|
||||
enable_modules=[__name__],
|
||||
|
|
@ -24,9 +29,8 @@ async def main():
|
|||
# call the subactor a 2nd time
|
||||
print(await portal.run(movie_theatre_question))
|
||||
|
||||
# the async with will block here indefinitely waiting
|
||||
# for our actor "frank" to complete, but since it's an
|
||||
# "outlive_main" actor it will never end until cancelled
|
||||
# the async with will wait indefinitely for "frank" because
|
||||
# its runtime remains active until explicitly cancelled
|
||||
await portal.cancel_actor()
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -1,33 +1,46 @@
|
|||
from typing import AsyncIterator
|
||||
from itertools import repeat
|
||||
from typing import AsyncIterator
|
||||
|
||||
import trio
|
||||
import tractor
|
||||
|
||||
|
||||
async def stream_forever() -> AsyncIterator[int]:
|
||||
async def stream_forever() -> AsyncIterator[str]:
|
||||
'''
|
||||
Stream the same message indefinitely.
|
||||
|
||||
for i in repeat("I can see these little future bubble things"):
|
||||
# each yielded value is sent over the ``Channel`` to the parent actor
|
||||
yield i
|
||||
'''
|
||||
message: str
|
||||
for message in repeat(
|
||||
'I can see these little future bubble things',
|
||||
):
|
||||
# each yielded value is sent over the ``Channel`` to the
|
||||
# parent actor
|
||||
yield message
|
||||
await trio.sleep(0.01)
|
||||
|
||||
|
||||
async def main():
|
||||
async def main() -> None:
|
||||
'''
|
||||
Print messages streamed from a subactor.
|
||||
|
||||
'''
|
||||
an: tractor.ActorNursery
|
||||
async with tractor.open_nursery() as an:
|
||||
|
||||
portal = await an.start_actor(
|
||||
portal: tractor.Portal = await an.start_actor(
|
||||
'donny',
|
||||
enable_modules=[__name__],
|
||||
)
|
||||
|
||||
# this async for loop streams values from the above
|
||||
# async generator running in a separate process
|
||||
stream: tractor.MsgStream
|
||||
async with portal.open_stream_from(stream_forever) as stream:
|
||||
count = 0
|
||||
async for letter in stream:
|
||||
print(letter)
|
||||
count: int = 0
|
||||
message: str
|
||||
async for message in stream:
|
||||
print(message)
|
||||
count += 1
|
||||
|
||||
if count > 50:
|
||||
|
|
|
|||
|
|
@ -13,7 +13,11 @@ import tractor
|
|||
@tractor.context
|
||||
async def sleep(
|
||||
ctx: tractor.Context,
|
||||
):
|
||||
) -> None:
|
||||
'''
|
||||
Start a context after a brief initialization delay.
|
||||
|
||||
'''
|
||||
await trio.sleep(0.5)
|
||||
await ctx.started()
|
||||
await trio.sleep_forever()
|
||||
|
|
@ -21,10 +25,13 @@ async def sleep(
|
|||
|
||||
async def open_ctx(
|
||||
n: tractor.runtime._supervise.ActorNursery
|
||||
):
|
||||
) -> None:
|
||||
'''
|
||||
Spawn a sleeper and open a context with it.
|
||||
|
||||
'''
|
||||
# spawn both actors
|
||||
portal = await n.start_actor(
|
||||
portal: tractor.Portal = await n.start_actor(
|
||||
name='sleeper',
|
||||
enable_modules=[__name__],
|
||||
)
|
||||
|
|
@ -35,8 +42,11 @@ async def open_ctx(
|
|||
assert first is None
|
||||
|
||||
|
||||
async def main():
|
||||
async def main() -> None:
|
||||
'''
|
||||
Fail the root while a subactor context is still starting.
|
||||
|
||||
'''
|
||||
async with tractor.open_nursery(
|
||||
debug_mode=True,
|
||||
loglevel='runtime',
|
||||
|
|
|
|||
|
|
@ -1,9 +1,14 @@
|
|||
from collections.abc import AsyncIterator
|
||||
|
||||
import tractor
|
||||
import trio
|
||||
|
||||
|
||||
async def breakpoint_forever():
|
||||
"Indefinitely re-enter debugger in child actor."
|
||||
async def breakpoint_forever() -> AsyncIterator[str]:
|
||||
'''
|
||||
Indefinitely re-enter debugger in child actor.
|
||||
|
||||
'''
|
||||
try:
|
||||
while True:
|
||||
yield 'yo'
|
||||
|
|
@ -15,12 +20,15 @@ async def breakpoint_forever():
|
|||
raise
|
||||
|
||||
|
||||
async def name_error():
|
||||
"Raise a ``NameError``"
|
||||
async def name_error() -> None:
|
||||
'''
|
||||
Raise a ``NameError``.
|
||||
|
||||
'''
|
||||
getattr(doggypants) # noqa
|
||||
|
||||
|
||||
async def main():
|
||||
async def main() -> None:
|
||||
'''
|
||||
Test breakpoint in a streaming actor.
|
||||
|
||||
|
|
@ -28,8 +36,14 @@ async def main():
|
|||
async with tractor.open_nursery(
|
||||
debug_mode=True,
|
||||
) as an:
|
||||
p0 = await an.start_actor('bp_forever', enable_modules=[__name__])
|
||||
p1 = await an.start_actor('name_error', enable_modules=[__name__])
|
||||
p0: tractor.Portal = await an.start_actor(
|
||||
'bp_forever',
|
||||
enable_modules=[__name__],
|
||||
)
|
||||
p1: tractor.Portal = await an.start_actor(
|
||||
'name_error',
|
||||
enable_modules=[__name__],
|
||||
)
|
||||
|
||||
# retreive results
|
||||
async with p0.open_stream_from(breakpoint_forever) as stream:
|
||||
|
|
@ -40,6 +54,7 @@ async def main():
|
|||
except tractor.RemoteActorError as rae:
|
||||
assert rae.boxed_type is NameError
|
||||
|
||||
i: str
|
||||
async for i in stream:
|
||||
|
||||
# a second time try the failing subactor and this tie
|
||||
|
|
|
|||
|
|
@ -4,13 +4,19 @@ import trio
|
|||
import tractor
|
||||
|
||||
|
||||
async def name_error():
|
||||
"Raise a ``NameError``"
|
||||
async def name_error() -> None:
|
||||
'''
|
||||
Raise a ``NameError``.
|
||||
|
||||
'''
|
||||
getattr(doggypants) # noqa
|
||||
|
||||
|
||||
async def breakpoint_forever():
|
||||
"Indefinitely re-enter debugger in child actor."
|
||||
async def breakpoint_forever() -> None:
|
||||
'''
|
||||
Indefinitely re-enter debugger in child actor.
|
||||
|
||||
'''
|
||||
while True:
|
||||
await tractor.pause()
|
||||
|
||||
|
|
@ -20,9 +26,13 @@ async def breakpoint_forever():
|
|||
# await trio.sleep(0)
|
||||
|
||||
|
||||
async def spawn_until(depth=0):
|
||||
""""A nested nursery that triggers another ``NameError``.
|
||||
"""
|
||||
async def spawn_until(
|
||||
depth: int = 0,
|
||||
) -> None:
|
||||
'''
|
||||
A nested nursery that triggers another ``NameError``.
|
||||
|
||||
'''
|
||||
async with (
|
||||
tractor.open_nursery() as an,
|
||||
trio.open_nursery() as tn,
|
||||
|
|
@ -37,7 +47,8 @@ async def spawn_until(depth=0):
|
|||
)
|
||||
)
|
||||
|
||||
# Let the background one-shot enter `breakpoint_forever()`
|
||||
# Let the background one-shot enter
|
||||
# `breakpoint_forever()`
|
||||
# before its sibling raises and cancellation propagates.
|
||||
await trio.sleep(0.5)
|
||||
# rx and propagate error from child
|
||||
|
|
@ -48,9 +59,9 @@ async def spawn_until(depth=0):
|
|||
)
|
||||
|
||||
else:
|
||||
# recusrive call to spawn another process branching layer of
|
||||
# the tree; blocks (up) each level until the leaf's
|
||||
# `name_error` relays through.
|
||||
# recusrive call to spawn another process branching
|
||||
# layer of the tree; blocks (up) each level until the
|
||||
# leaf's `name_error` relays through.
|
||||
depth -= 1
|
||||
await tractor.to_actor.run(
|
||||
partial(
|
||||
|
|
@ -63,25 +74,35 @@ async def spawn_until(depth=0):
|
|||
|
||||
|
||||
# TODO: notes on the new boxed-relayed errors through proxy actors
|
||||
async def main():
|
||||
"""The main ``tractor`` routine.
|
||||
async def main() -> None:
|
||||
'''
|
||||
The main ``tractor`` routine.
|
||||
|
||||
The process tree should look as approximately as follows when the debugger
|
||||
first engages:
|
||||
The process tree should look approximately as follows when the
|
||||
debugger first engages:
|
||||
|
||||
python examples/debugging/multi_nested_subactors_bp_forever.py
|
||||
├─ python -m tractor._child --uid ('spawner1', '7eab8462 ...)
|
||||
│ └─ python -m tractor._child --uid ('spawn_until_3', 'afcba7a8 ...)
|
||||
│ └─ python -m tractor._child --uid ('spawn_until_2', 'd2433d13 ...)
|
||||
│ └─ python -m tractor._child --uid ('spawn_until_1', '1df589de ...)
|
||||
│ └─ python -m tractor._child --uid ('spawn_until_0', '3720602b ...)
|
||||
├─ python -m tractor._child --uid
|
||||
│ ('spawner1', '7eab8462 ...')
|
||||
│ └─ python -m tractor._child --uid
|
||||
│ ('spawn_until_3', 'afcba7a8 ...')
|
||||
│ └─ python -m tractor._child --uid
|
||||
│ ('spawn_until_2', 'd2433d13 ...')
|
||||
│ └─ python -m tractor._child --uid
|
||||
│ ('spawn_until_1', '1df589de ...')
|
||||
│ └─ python -m tractor._child --uid
|
||||
│ ('spawn_until_0', '3720602b ...')
|
||||
│
|
||||
└─ python -m tractor._child --uid ('spawner0', '1d42012b ...)
|
||||
└─ python -m tractor._child --uid ('spawn_until_2', '2877e155 ...)
|
||||
└─ python -m tractor._child --uid ('spawn_until_1', '0502d786 ...)
|
||||
└─ python -m tractor._child --uid ('spawn_until_0', 'de918e6d ...)
|
||||
└─ python -m tractor._child --uid
|
||||
('spawner0', '1d42012b ...')
|
||||
└─ python -m tractor._child --uid
|
||||
('spawn_until_2', '2877e155 ...')
|
||||
└─ python -m tractor._child --uid
|
||||
('spawn_until_1', '0502d786 ...')
|
||||
└─ python -m tractor._child --uid
|
||||
('spawn_until_0', 'de918e6d ...')
|
||||
|
||||
"""
|
||||
'''
|
||||
async with (
|
||||
tractor.open_nursery(
|
||||
debug_mode=True,
|
||||
|
|
|
|||
|
|
@ -7,14 +7,19 @@ import trio
|
|||
import tractor
|
||||
|
||||
|
||||
async def name_error():
|
||||
"Raise a ``NameError``"
|
||||
async def name_error() -> None:
|
||||
'''
|
||||
Raise a ``NameError``.
|
||||
|
||||
'''
|
||||
getattr(doggypants) # noqa
|
||||
|
||||
|
||||
async def spawn_error():
|
||||
""""A nested nursery that triggers another ``NameError``.
|
||||
"""
|
||||
async def spawn_error() -> None:
|
||||
'''
|
||||
A nested nursery that triggers another ``NameError``.
|
||||
|
||||
'''
|
||||
async with tractor.open_nursery() as an:
|
||||
return await tractor.to_actor.run(
|
||||
name_error,
|
||||
|
|
@ -23,8 +28,9 @@ async def spawn_error():
|
|||
)
|
||||
|
||||
|
||||
async def main():
|
||||
"""The main ``tractor`` routine.
|
||||
async def main() -> None:
|
||||
'''
|
||||
The main ``tractor`` routine.
|
||||
|
||||
The process tree should look as approximately as follows:
|
||||
|
||||
|
|
@ -37,7 +43,8 @@ async def main():
|
|||
- nested name_error sub-sub-actor
|
||||
- root actor should then fail on assert
|
||||
- program termination
|
||||
"""
|
||||
|
||||
'''
|
||||
async with (
|
||||
tractor.open_nursery(
|
||||
debug_mode=True,
|
||||
|
|
@ -46,11 +53,11 @@ async def main():
|
|||
trio.open_nursery() as tn,
|
||||
):
|
||||
# spawn both actors..
|
||||
portal = await an.start_actor(
|
||||
portal: tractor.Portal = await an.start_actor(
|
||||
'name_error',
|
||||
enable_modules=[__name__],
|
||||
)
|
||||
portal1 = await an.start_actor(
|
||||
portal1: tractor.Portal = await an.start_actor(
|
||||
'spawn_error',
|
||||
enable_modules=[__name__],
|
||||
)
|
||||
|
|
|
|||
|
|
@ -1,22 +1,32 @@
|
|||
from collections.abc import Awaitable, Callable
|
||||
|
||||
import tractor
|
||||
import trio
|
||||
|
||||
|
||||
async def breakpoint_forever():
|
||||
"Indefinitely re-enter debugger in child actor."
|
||||
async def breakpoint_forever() -> None:
|
||||
'''
|
||||
Indefinitely re-enter debugger in child actor.
|
||||
|
||||
'''
|
||||
while True:
|
||||
await trio.sleep(0.1)
|
||||
await tractor.pause()
|
||||
|
||||
|
||||
async def name_error():
|
||||
"Raise a ``NameError``"
|
||||
async def name_error() -> None:
|
||||
'''
|
||||
Raise a ``NameError``.
|
||||
|
||||
'''
|
||||
getattr(doggypants) # noqa
|
||||
|
||||
|
||||
async def spawn_error():
|
||||
""""A nested nursery that triggers another ``NameError``.
|
||||
"""
|
||||
async def spawn_error() -> None:
|
||||
'''
|
||||
A nested nursery that triggers another ``NameError``.
|
||||
|
||||
'''
|
||||
async with tractor.open_nursery() as an:
|
||||
return await tractor.to_actor.run(
|
||||
name_error,
|
||||
|
|
@ -25,8 +35,9 @@ async def spawn_error():
|
|||
)
|
||||
|
||||
|
||||
async def main():
|
||||
"""The main ``tractor`` routine.
|
||||
async def main() -> None:
|
||||
'''
|
||||
The main ``tractor`` routine.
|
||||
|
||||
The process tree should look as approximately as follows:
|
||||
|
||||
|
|
@ -35,7 +46,8 @@ async def main():
|
|||
|-python -m tractor._child --uid ('bp_forever', '1f787a7e ...)
|
||||
`-python -m tractor._child --uid ('spawn_error', '52ee14a5 ...)
|
||||
`-python -m tractor._child --uid ('name_error', '3391222c ...)
|
||||
"""
|
||||
|
||||
'''
|
||||
errors: list[BaseException] = []
|
||||
|
||||
async with tractor.open_nursery(
|
||||
|
|
@ -43,12 +55,14 @@ async def main():
|
|||
# loglevel='runtime',
|
||||
) as an:
|
||||
|
||||
async def run_and_collect(fn):
|
||||
async def run_and_collect(
|
||||
fn: Callable[[], Awaitable[object]],
|
||||
) -> None:
|
||||
'''
|
||||
One-shot whose (boxed) error is stashed instead of
|
||||
raised so a sibling's crash never cancels the others
|
||||
before they've had their own debugger sessions (the
|
||||
"collect all errors" the legacy `run_in_actor()` API
|
||||
'collect all errors' the legacy `run_in_actor()` API
|
||||
did implicitly at nursery teardown).
|
||||
|
||||
'''
|
||||
|
|
|
|||
|
|
@ -1,19 +1,28 @@
|
|||
import trio
|
||||
import tractor
|
||||
|
||||
async def die():
|
||||
|
||||
async def die() -> None:
|
||||
'''
|
||||
Deliberately crash the calling actor.
|
||||
|
||||
'''
|
||||
raise RuntimeError
|
||||
|
||||
|
||||
async def main():
|
||||
async def main() -> None:
|
||||
'''
|
||||
Crash actors with different debugger settings concurrently.
|
||||
|
||||
'''
|
||||
async with tractor.open_nursery() as an:
|
||||
|
||||
debug_actor = await an.start_actor(
|
||||
debug_actor: tractor.Portal = await an.start_actor(
|
||||
'debugged_boi',
|
||||
enable_modules=[__name__],
|
||||
debug_mode=True,
|
||||
)
|
||||
crash_boi = await an.start_actor(
|
||||
crash_boi: tractor.Portal = await an.start_actor(
|
||||
'crash_boi',
|
||||
enable_modules=[__name__],
|
||||
# debug_mode=True,
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ import tractor
|
|||
@tractor.context
|
||||
async def name_error(
|
||||
ctx: tractor.Context,
|
||||
):
|
||||
) -> None:
|
||||
'''
|
||||
Raise a `NameError`, catch it and enter `.post_mortem()`, then
|
||||
expect the `._rpc._invoke()` crash handler to also engage.
|
||||
|
|
@ -18,7 +18,7 @@ async def name_error(
|
|||
raise
|
||||
|
||||
|
||||
async def main():
|
||||
async def main() -> None:
|
||||
'''
|
||||
Test 3 `PdbREPL` entries:
|
||||
- one in the child due to manual `.post_mortem()`,
|
||||
|
|
@ -49,7 +49,9 @@ async def main():
|
|||
await tractor.post_mortem()
|
||||
raise
|
||||
else:
|
||||
raise RuntimeError('IPC ctx should have remote errored!?')
|
||||
raise RuntimeError(
|
||||
'IPC ctx should have remote errored!?'
|
||||
)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
|
|
|
|||
|
|
@ -2,8 +2,11 @@ import trio
|
|||
import tractor
|
||||
|
||||
|
||||
async def main():
|
||||
async def main() -> None:
|
||||
'''
|
||||
Pause in the root actor to exercise its debugger REPL.
|
||||
|
||||
'''
|
||||
async with tractor.open_root_actor(
|
||||
debug_mode=True,
|
||||
):
|
||||
|
|
|
|||
|
|
@ -2,7 +2,11 @@ import trio
|
|||
import tractor
|
||||
|
||||
|
||||
async def main():
|
||||
async def main() -> None:
|
||||
'''
|
||||
Raise an assertion error from the debug-enabled root actor.
|
||||
|
||||
'''
|
||||
async with tractor.open_root_actor(
|
||||
debug_mode=True,
|
||||
):
|
||||
|
|
|
|||
|
|
@ -4,14 +4,21 @@ import trio
|
|||
import tractor
|
||||
|
||||
|
||||
async def name_error():
|
||||
"Raise a ``NameError``"
|
||||
async def name_error() -> None:
|
||||
'''
|
||||
Raise a ``NameError``.
|
||||
|
||||
'''
|
||||
getattr(doggypants) # noqa
|
||||
|
||||
|
||||
async def spawn_until(depth=0):
|
||||
""""A nested nursery that triggers another ``NameError``.
|
||||
"""
|
||||
async def spawn_until(
|
||||
depth: int = 0,
|
||||
) -> None:
|
||||
'''
|
||||
A nested nursery that triggers another ``NameError``.
|
||||
|
||||
'''
|
||||
async with tractor.open_nursery() as an:
|
||||
if depth < 1:
|
||||
await tractor.to_actor.run(name_error, an=an)
|
||||
|
|
@ -27,18 +34,23 @@ async def spawn_until(depth=0):
|
|||
)
|
||||
|
||||
|
||||
async def main():
|
||||
async def main() -> None:
|
||||
'''
|
||||
The process tree should look as approximately as follows when the
|
||||
debugger first engages:
|
||||
|
||||
python examples/debugging/multi_nested_subactors_bp_forever.py
|
||||
├─ python -m tractor._child --uid ('spawner1', '7eab8462 ...)
|
||||
│ └─ python -m tractor._child --uid ('spawn_until_0', '3720602b ...)
|
||||
│ └─ python -m tractor._child --uid ('name_error', '505bf71d ...)
|
||||
├─ python -m tractor._child --uid
|
||||
│ ('spawner1', '7eab8462 ...')
|
||||
│ └─ python -m tractor._child --uid
|
||||
│ ('spawn_until_0', '3720602b ...')
|
||||
│ └─ python -m tractor._child --uid
|
||||
│ ('name_error', '505bf71d ...')
|
||||
│
|
||||
└─ python -m tractor._child --uid ('spawner0', '1d42012b ...)
|
||||
└─ python -m tractor._child --uid ('name_error', '6c2733b8 ...)
|
||||
└─ python -m tractor._child --uid
|
||||
('spawner0', '1d42012b ...')
|
||||
└─ python -m tractor._child --uid
|
||||
('name_error', '6c2733b8 ...')
|
||||
|
||||
'''
|
||||
async with (
|
||||
|
|
|
|||
|
|
@ -2,7 +2,11 @@ import trio
|
|||
import tractor
|
||||
|
||||
|
||||
async def main():
|
||||
async def main() -> None:
|
||||
'''
|
||||
Enter shielded debugging after root cancellation, then fail.
|
||||
|
||||
'''
|
||||
async with tractor.open_root_actor(
|
||||
debug_mode=True,
|
||||
loglevel='cancel',
|
||||
|
|
@ -18,16 +22,19 @@ async def main():
|
|||
try:
|
||||
await tractor.pause()
|
||||
except trio.Cancelled as _taskc:
|
||||
assert (root_cs := _root._root_tn.cancel_scope).cancel_called
|
||||
root_cs: trio.CancelScope
|
||||
assert (
|
||||
root_cs := _root._root_tn.cancel_scope
|
||||
).cancel_called
|
||||
# NOTE^^ above logic but inside `open_root_actor()` and
|
||||
# passed to the `shield=` expression is effectively what
|
||||
# we're testing here!
|
||||
await tractor.pause(shield=root_cs.cancel_called)
|
||||
|
||||
# XXX, if shield logic *is wrong* inside `open_root_actor()`'s
|
||||
# crash-handler block this should never be interacted,
|
||||
# instead `trio.Cancelled` would be bubbled up: the original
|
||||
# BUG.
|
||||
# XXX, if shield logic *is wrong* inside
|
||||
# `open_root_actor()`'s crash-handler block this should never
|
||||
# be interacted, instead `trio.Cancelled` would be bubbled
|
||||
# up: the original BUG.
|
||||
assert 0
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -2,12 +2,15 @@ import trio
|
|||
import tractor
|
||||
|
||||
|
||||
async def key_error():
|
||||
"Raise a ``NameError``"
|
||||
async def key_error() -> None:
|
||||
'''
|
||||
Raise a ``KeyError``.
|
||||
|
||||
'''
|
||||
return {}['doggy']
|
||||
|
||||
|
||||
async def main():
|
||||
async def main() -> None:
|
||||
'''
|
||||
Root is fail-after-cancelled while blocking and child RPC fails
|
||||
simultaneously.
|
||||
|
|
@ -21,7 +24,7 @@ async def main():
|
|||
trio.open_nursery() as tn,
|
||||
):
|
||||
# spawn the actor..
|
||||
portal = await an.start_actor(
|
||||
portal: tractor.Portal = await an.start_actor(
|
||||
'key_error',
|
||||
enable_modules=[__name__],
|
||||
)
|
||||
|
|
@ -32,9 +35,9 @@ async def main():
|
|||
# root blocks below.
|
||||
tn.start_soon(portal.run, key_error)
|
||||
|
||||
# XXX: originally a bug caused by this is where root would enter
|
||||
# the debugger and clobber the tty used by the repl even though
|
||||
# child should have it locked.
|
||||
# XXX: originally a bug caused by this is where root would
|
||||
# enter the debugger and clobber the tty used by the repl
|
||||
# even though child should have it locked.
|
||||
with trio.fail_after(1):
|
||||
await trio.Event().wait()
|
||||
|
||||
|
|
|
|||
|
|
@ -3,8 +3,15 @@ import tractor
|
|||
|
||||
|
||||
async def cancellable_pause_loop(
|
||||
task_status: trio.TaskStatus[trio.CancelScope] = trio.TASK_STATUS_IGNORED
|
||||
):
|
||||
task_status: trio.TaskStatus[
|
||||
trio.CancelScope
|
||||
] = trio.TASK_STATUS_IGNORED,
|
||||
) -> None:
|
||||
'''
|
||||
Exercise shielded debugger pauses under cancellation.
|
||||
|
||||
'''
|
||||
cs: trio.CancelScope
|
||||
with trio.CancelScope() as cs:
|
||||
task_status.started(cs)
|
||||
for _ in range(3):
|
||||
|
|
@ -30,7 +37,11 @@ async def cancellable_pause_loop(
|
|||
await trio.lowlevel.checkpoint()
|
||||
|
||||
|
||||
async def pm_on_cancelled():
|
||||
async def pm_on_cancelled() -> None:
|
||||
'''
|
||||
Compare shielded and unshielded post-mortem entry.
|
||||
|
||||
'''
|
||||
async with trio.open_nursery() as tn:
|
||||
tn.cancel_scope.cancel()
|
||||
try:
|
||||
|
|
@ -56,7 +67,7 @@ async def pm_on_cancelled():
|
|||
|
||||
|
||||
async def cancelled_before_pause(
|
||||
):
|
||||
) -> None:
|
||||
'''
|
||||
Verify that using a shielded pause works despite surrounding
|
||||
cancellation called state in the calling task.
|
||||
|
|
@ -71,7 +82,11 @@ async def cancelled_before_pause(
|
|||
await pm_on_cancelled()
|
||||
|
||||
|
||||
async def main():
|
||||
async def main() -> None:
|
||||
'''
|
||||
Exercise shielded debugger entry in subactor and root tasks.
|
||||
|
||||
'''
|
||||
async with tractor.open_nursery(
|
||||
debug_mode=True,
|
||||
) as an:
|
||||
|
|
|
|||
|
|
@ -1,10 +1,15 @@
|
|||
import platform
|
||||
from collections.abc import AsyncIterator
|
||||
|
||||
import tractor
|
||||
import trio
|
||||
|
||||
|
||||
async def gen():
|
||||
async def gen() -> AsyncIterator[str]:
|
||||
'''
|
||||
Yield values around debugger pauses.
|
||||
|
||||
'''
|
||||
yield 'yo'
|
||||
await tractor.pause()
|
||||
yield 'yo'
|
||||
|
|
@ -15,11 +20,15 @@ async def gen():
|
|||
async def just_bp(
|
||||
ctx: tractor.Context,
|
||||
) -> None:
|
||||
'''
|
||||
Pause repeatedly before deliberately breaking the context.
|
||||
|
||||
'''
|
||||
await ctx.started()
|
||||
await tractor.pause()
|
||||
|
||||
# TODO: bps and errors in this call..
|
||||
val: str
|
||||
async for val in gen():
|
||||
print(val)
|
||||
|
||||
|
|
@ -34,15 +43,18 @@ async def just_bp(
|
|||
|
||||
|
||||
|
||||
async def main():
|
||||
async def main() -> None:
|
||||
'''
|
||||
Run the breakpoint context over a supported transport.
|
||||
|
||||
'''
|
||||
# !TODO, parametrize the --tpt-proto={key} with osenv vars just
|
||||
# like we do for loglevel/spawn-backend!
|
||||
# - [ ] run on both tpts for all such debugger tests?
|
||||
# - [ ] special skip for macos!
|
||||
#
|
||||
if platform.system() != 'Darwin':
|
||||
tpt = 'uds'
|
||||
tpt: str = 'uds'
|
||||
else:
|
||||
# XXX, precisely we can't use pytest's tmp-path generation
|
||||
# for tests.. apparently because:
|
||||
|
|
@ -59,7 +71,7 @@ async def main():
|
|||
enable_transports=[tpt],
|
||||
loglevel='devx',
|
||||
) as an:
|
||||
p = await an.start_actor(
|
||||
p: tractor.Portal = await an.start_actor(
|
||||
'bp_boi',
|
||||
enable_modules=[__name__],
|
||||
)
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@ import trio
|
|||
import tractor
|
||||
|
||||
|
||||
async def breakpoint_forever():
|
||||
async def breakpoint_forever() -> None:
|
||||
'''
|
||||
Indefinitely re-enter debugger in child actor.
|
||||
|
||||
|
|
@ -12,8 +12,11 @@ async def breakpoint_forever():
|
|||
await tractor.pause()
|
||||
|
||||
|
||||
async def main():
|
||||
async def main() -> None:
|
||||
'''
|
||||
Run a subactor that repeatedly pauses in the debugger.
|
||||
|
||||
'''
|
||||
async with tractor.open_nursery(
|
||||
debug_mode=True,
|
||||
loglevel='cancel',
|
||||
|
|
|
|||
|
|
@ -2,11 +2,19 @@ import trio
|
|||
import tractor
|
||||
|
||||
|
||||
async def name_error():
|
||||
async def name_error() -> None:
|
||||
'''
|
||||
Deliberately raise a ``NameError`` in a subactor.
|
||||
|
||||
'''
|
||||
getattr(doggypants) # noqa (on purpose)
|
||||
|
||||
|
||||
async def main():
|
||||
async def main() -> None:
|
||||
'''
|
||||
Surface a subactor `NameError` at the waiting root task.
|
||||
|
||||
'''
|
||||
async with tractor.open_nursery(
|
||||
debug_mode=True,
|
||||
) as an:
|
||||
|
|
|
|||
|
|
@ -0,0 +1,394 @@
|
|||
'''
|
||||
Run a dedicated registrar in a standalone process.
|
||||
|
||||
The service and discovery client are sibling actors. The client has
|
||||
no pre-existing channel to the service, so its lookup must use the
|
||||
external registrar instead of the local-peer fast path.
|
||||
|
||||
'''
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import AsyncIterator
|
||||
from contextlib import asynccontextmanager as acm
|
||||
import errno
|
||||
from pathlib import Path
|
||||
import signal
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
|
||||
import trio
|
||||
import tractor
|
||||
|
||||
|
||||
MAX_BIND_ATTEMPTS: int = 5
|
||||
|
||||
|
||||
def _is_addr_collision(exc: BaseException) -> bool:
|
||||
'''
|
||||
Return whether registrar startup lost the selected TCP address.
|
||||
|
||||
Tractor can notice the collision while probing the address or
|
||||
later when its listener binds. Exception groups are retryable
|
||||
only when every contained failure reports the same collision.
|
||||
|
||||
'''
|
||||
match exc:
|
||||
case BaseExceptionGroup(exceptions=exceptions):
|
||||
return bool(exceptions) and all(
|
||||
_is_addr_collision(child)
|
||||
for child in exceptions
|
||||
)
|
||||
|
||||
case OSError() as os_error:
|
||||
return (
|
||||
os_error.errno in {errno.EADDRINUSE, 10048}
|
||||
or getattr(os_error, 'winerror', None) == 10048
|
||||
)
|
||||
|
||||
case RuntimeError() as runtime_error:
|
||||
message: str = str(runtime_error)
|
||||
return (
|
||||
'Registry address(es) are occupied' in message
|
||||
or 'registry socket(s) already bound' in message
|
||||
)
|
||||
|
||||
case _:
|
||||
return False
|
||||
|
||||
|
||||
def run_registrar(ready_path: str) -> None:
|
||||
'''
|
||||
Serve as the required registrar and report its selected address.
|
||||
|
||||
The kernel selects ephemeral loopback candidates in this process.
|
||||
If another process claims a released candidate first, retry with
|
||||
a fresh candidate up to `MAX_BIND_ATTEMPTS`. Other startup errors
|
||||
and the final collision remain visible. `ensure_registry=True`
|
||||
prevents silently joining a registrar that won the address.
|
||||
|
||||
'''
|
||||
ready_file: Path = Path(ready_path)
|
||||
|
||||
async def serve() -> None:
|
||||
'''
|
||||
Open the registrar, publish readiness, and serve forever.
|
||||
|
||||
'''
|
||||
for attempt in range(1, MAX_BIND_ATTEMPTS + 1):
|
||||
# This selector socket reserves and reports a
|
||||
# kernel-selected candidate; it never listens and is
|
||||
# not transferred to Tractor. Closing it lets
|
||||
# `open_root_actor()` create its own listener on the
|
||||
# same addr. The close/rebind handoff is non-atomic,
|
||||
# hence the bounded collision retries.
|
||||
sock: socket.socket
|
||||
with socket.socket(
|
||||
socket.AF_INET,
|
||||
socket.SOCK_STREAM,
|
||||
) as sock:
|
||||
sock.bind(('127.0.0.1', 0))
|
||||
selected: tuple[str, int] = sock.getsockname()
|
||||
registry_addr: tuple[str, int] = (
|
||||
selected[0],
|
||||
selected[1],
|
||||
)
|
||||
|
||||
try:
|
||||
actor: tractor.Actor
|
||||
async with tractor.open_root_actor(
|
||||
name='dedicated_registrar',
|
||||
registry_addrs=[registry_addr],
|
||||
enable_transports=['tcp'],
|
||||
enable_modules=[],
|
||||
ensure_registry=True,
|
||||
loglevel='error',
|
||||
) as actor:
|
||||
if not actor.is_registrar:
|
||||
raise RuntimeError(
|
||||
'daemon did not become registrar'
|
||||
)
|
||||
|
||||
tmp_file: Path = ready_file.with_suffix('.tmp')
|
||||
tmp_file.write_text(
|
||||
str(registry_addr[1]),
|
||||
encoding='ascii',
|
||||
)
|
||||
tmp_file.replace(ready_file)
|
||||
await trio.sleep_forever()
|
||||
|
||||
except BaseException as exc:
|
||||
if (
|
||||
not _is_addr_collision(exc)
|
||||
or attempt == MAX_BIND_ATTEMPTS
|
||||
):
|
||||
raise
|
||||
await trio.sleep(.05 * attempt)
|
||||
|
||||
try:
|
||||
trio.run(serve)
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
|
||||
|
||||
def _registrar_command(ready_path: Path) -> list[str]:
|
||||
'''
|
||||
Build a child command that loads without running `main()`.
|
||||
|
||||
`runpy.run_path()` also works when the docs test copies and
|
||||
renames this example before executing it.
|
||||
|
||||
'''
|
||||
module_path: str = repr(str(Path(__file__).resolve()))
|
||||
function_name: str = repr('run_registrar')
|
||||
ready_arg: str = repr(str(ready_path))
|
||||
code: str = (
|
||||
f'import runpy; module = runpy.run_path({module_path}); '
|
||||
f'module[{function_name}]({ready_arg})'
|
||||
)
|
||||
return [sys.executable, '-c', code]
|
||||
|
||||
|
||||
def _wait_registrar_ready(
|
||||
ready_path: Path,
|
||||
proc: subprocess.Popen,
|
||||
deadline: float = 10.0,
|
||||
) -> tuple[str, int]:
|
||||
'''
|
||||
Wait until the child has entered its registrar actor context.
|
||||
|
||||
The child atomically publishes its selected port only after
|
||||
`open_root_actor()` completes. Fail early if startup crashes.
|
||||
|
||||
'''
|
||||
end: float = time.monotonic() + deadline
|
||||
while time.monotonic() < end:
|
||||
if proc.poll() is not None:
|
||||
returncode: int|None = proc.returncode
|
||||
raise RuntimeError(
|
||||
f'registrar exited during startup: {returncode=}'
|
||||
)
|
||||
|
||||
try:
|
||||
port: int = int(
|
||||
ready_path.read_text(encoding='ascii')
|
||||
)
|
||||
except (
|
||||
OSError,
|
||||
ValueError,
|
||||
):
|
||||
time.sleep(.05)
|
||||
continue
|
||||
|
||||
if not 0 < port < 2**16:
|
||||
raise RuntimeError(f'invalid registrar port: {port!r}')
|
||||
if proc.poll() is not None:
|
||||
raise RuntimeError(
|
||||
'registrar exited after reporting ready'
|
||||
)
|
||||
return ('127.0.0.1', port)
|
||||
|
||||
raise TimeoutError('registrar did not report ready')
|
||||
|
||||
|
||||
def _stop_registrar(
|
||||
proc: subprocess.Popen,
|
||||
graceful_timeout: float = 5.0,
|
||||
) -> None:
|
||||
'''
|
||||
Stop and reap the registrar, escalating after a bounded wait.
|
||||
|
||||
Windows children receive `CTRL_C_EVENT` in their new process
|
||||
group; POSIX children receive `SIGINT`. A child that ignores
|
||||
graceful shutdown is killed, and every path finishes with
|
||||
`wait()`. A non-zero child exit remains visible to the caller.
|
||||
|
||||
'''
|
||||
if proc.poll() is None:
|
||||
graceful_signal: int = (
|
||||
signal.CTRL_C_EVENT
|
||||
if sys.platform == 'win32'
|
||||
else signal.SIGINT
|
||||
)
|
||||
try:
|
||||
proc.send_signal(graceful_signal)
|
||||
except OSError:
|
||||
if proc.poll() is None:
|
||||
proc.terminate()
|
||||
|
||||
try:
|
||||
proc.wait(timeout=graceful_timeout)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
proc.wait()
|
||||
|
||||
if proc.returncode:
|
||||
raise RuntimeError(
|
||||
'registrar shutdown failed: '
|
||||
f'returncode={proc.returncode}'
|
||||
)
|
||||
|
||||
|
||||
async def greet() -> str:
|
||||
'''
|
||||
Return a greeting identifying the actor serving the RPC.
|
||||
|
||||
'''
|
||||
actor_name: str = tractor.current_actor().name
|
||||
return f'hello from {actor_name}!'
|
||||
|
||||
|
||||
async def discover_and_greet(
|
||||
registry_addr: tuple[str, int],
|
||||
) -> tuple[str, str, str]:
|
||||
'''
|
||||
Prove registrar lookup from a client without a service channel.
|
||||
|
||||
The parent spawns this actor as `greeter`'s sibling. A non-`None`
|
||||
registry portal from `query_actor()` proves that discovery did
|
||||
not take the existing-peer fast path, which returns no registry
|
||||
portal.
|
||||
|
||||
'''
|
||||
service_addr: tuple[str, int]|None
|
||||
registry_portal: tractor.Portal|None
|
||||
async with tractor.query_actor(
|
||||
'greeter',
|
||||
regaddr=registry_addr,
|
||||
) as (service_addr, registry_portal):
|
||||
if registry_portal is None:
|
||||
raise RuntimeError('lookup used a local service channel')
|
||||
if service_addr is None:
|
||||
raise RuntimeError('greeter was not registered')
|
||||
|
||||
service_portal: tractor.Portal|None
|
||||
async with tractor.find_actor(
|
||||
'greeter',
|
||||
registry_addrs=[registry_addr],
|
||||
) as service_portal:
|
||||
if service_portal is None:
|
||||
raise RuntimeError('greeter disappeared before RPC')
|
||||
greeting: str = await service_portal.run(greet)
|
||||
|
||||
client_name: str = tractor.current_actor().name
|
||||
return client_name, repr(service_addr), greeting
|
||||
|
||||
|
||||
async def app(registry_addr: tuple[str, int]) -> None:
|
||||
'''
|
||||
Use sibling service and client actors with an external registrar.
|
||||
|
||||
Only the parent receives both spawn-time portals. The `client`
|
||||
actor performs discovery in its own process and has no direct
|
||||
`greeter` channel before the lookup.
|
||||
|
||||
'''
|
||||
actor_nursery: tractor.ActorNursery
|
||||
async with tractor.open_nursery(
|
||||
registry_addrs=[registry_addr],
|
||||
enable_transports=['tcp'],
|
||||
) as actor_nursery:
|
||||
await actor_nursery.start_actor(
|
||||
'greeter',
|
||||
enable_modules=[__name__],
|
||||
)
|
||||
client_portal: tractor.Portal = (
|
||||
await actor_nursery.start_actor(
|
||||
'client',
|
||||
enable_modules=[__name__],
|
||||
)
|
||||
)
|
||||
result: tuple[str, str, str] = await client_portal.run(
|
||||
discover_and_greet,
|
||||
registry_addr=registry_addr,
|
||||
)
|
||||
client_name: str
|
||||
service_addr: str
|
||||
greeting: str
|
||||
(
|
||||
client_name,
|
||||
service_addr,
|
||||
greeting,
|
||||
) = result
|
||||
print(
|
||||
f'{client_name!r} found `greeter` through registrar '
|
||||
f'{registry_addr!r}; service address: {service_addr}\n'
|
||||
f'{greeting}'
|
||||
)
|
||||
await actor_nursery.cancel()
|
||||
|
||||
|
||||
# TODO: Promote this lifecycle into an OTB `tractor.discovery`
|
||||
# registrar subsystem. Reuse attach-or-create ownership from
|
||||
# `piker.service.maybe_open_pikerd()` and named service supervision
|
||||
# from `piker.service.Services`; replace the file readiness
|
||||
# handshake, then use the API from `tractor._testing.pytest` to
|
||||
# isolate remaining hard-coded `reg_addr` cases.
|
||||
@acm
|
||||
async def _open_registrar(
|
||||
) -> AsyncIterator[tuple[str, int]]:
|
||||
'''
|
||||
Start, publish, and reap one dedicated registrar process.
|
||||
|
||||
The Windows child gets a distinct console process group so the
|
||||
graceful control event targets it without interrupting this
|
||||
process.
|
||||
|
||||
'''
|
||||
temp_dir: str
|
||||
with tempfile.TemporaryDirectory(
|
||||
prefix='tractor-registrar-',
|
||||
) as temp_dir:
|
||||
ready_path: Path = Path(temp_dir) / 'ready'
|
||||
creationflags: int = (
|
||||
subprocess.CREATE_NEW_PROCESS_GROUP
|
||||
if sys.platform == 'win32'
|
||||
else 0
|
||||
)
|
||||
registrar: subprocess.Popen = subprocess.Popen(
|
||||
_registrar_command(ready_path),
|
||||
stdout=subprocess.DEVNULL,
|
||||
creationflags=creationflags,
|
||||
)
|
||||
primary_error: BaseException|None = None
|
||||
try:
|
||||
registry_addr: tuple[str, int] = _wait_registrar_ready(
|
||||
ready_path,
|
||||
registrar,
|
||||
)
|
||||
print(
|
||||
f'dedicated registrar ready at {registry_addr!r} '
|
||||
f'(pid {registrar.pid})'
|
||||
)
|
||||
yield registry_addr
|
||||
except BaseException as error:
|
||||
primary_error = error
|
||||
raise
|
||||
finally:
|
||||
try:
|
||||
_stop_registrar(registrar)
|
||||
except BaseException as cleanup_error:
|
||||
if primary_error is None:
|
||||
raise
|
||||
cleanup_note: str = (
|
||||
'registrar cleanup also failed: '
|
||||
f'{cleanup_error!r}'
|
||||
)
|
||||
primary_error.add_note(cleanup_note)
|
||||
print('dedicated registrar shut down')
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
'''
|
||||
Run the external registrar and sibling discovery actors.
|
||||
|
||||
'''
|
||||
registry_addr: tuple[str, int]
|
||||
async with _open_registrar() as registry_addr:
|
||||
await app(registry_addr)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
trio.run(main)
|
||||
|
|
@ -1,4 +1,6 @@
|
|||
import time
|
||||
from typing import AsyncIterator
|
||||
|
||||
import trio
|
||||
import tractor
|
||||
from tractor import (
|
||||
|
|
@ -9,14 +11,19 @@ from tractor import (
|
|||
|
||||
|
||||
# this is the first 2 actors, streamer_1 and streamer_2
|
||||
async def stream_data(seed):
|
||||
async def stream_data(seed: int) -> AsyncIterator[int]:
|
||||
'''
|
||||
Stream integers up to a seed value.
|
||||
|
||||
'''
|
||||
i: int
|
||||
for i in range(seed):
|
||||
yield i
|
||||
await trio.sleep(0.0001) # trigger scheduler
|
||||
|
||||
|
||||
# this is the third actor; the aggregator
|
||||
async def aggregate(seed):
|
||||
async def aggregate(seed: int) -> AsyncIterator[int]:
|
||||
'''
|
||||
Ensure that the two streams we receive match but only stream
|
||||
a single set of values to the parent.
|
||||
|
|
@ -25,30 +32,47 @@ async def aggregate(seed):
|
|||
an: ActorNursery
|
||||
async with tractor.open_nursery() as an:
|
||||
portals: list[Portal] = []
|
||||
i: int
|
||||
for i in range(1, 3):
|
||||
|
||||
# fork/spawn call
|
||||
portal = await an.start_actor(
|
||||
portal: Portal = await an.start_actor(
|
||||
name=f'streamer_{i}',
|
||||
enable_modules=[__name__],
|
||||
)
|
||||
|
||||
portals.append(portal)
|
||||
|
||||
send_chan: trio.MemorySendChannel[int]
|
||||
recv_chan: trio.MemoryReceiveChannel[int]
|
||||
send_chan, recv_chan = trio.open_memory_channel(500)
|
||||
|
||||
async def push_to_chan(portal, send_chan):
|
||||
async def push_to_chan(
|
||||
portal: Portal,
|
||||
send_chan: trio.MemorySendChannel[int],
|
||||
) -> None:
|
||||
'''
|
||||
Forward one remote stream into a local channel.
|
||||
|
||||
'''
|
||||
# TODO: https://github.com/goodboy/tractor/issues/207
|
||||
async with send_chan:
|
||||
async with portal.open_stream_from(stream_data, seed=seed) as stream:
|
||||
stream: MsgStream
|
||||
async with portal.open_stream_from(
|
||||
stream_data,
|
||||
seed=seed,
|
||||
) as stream:
|
||||
value: int
|
||||
async for value in stream:
|
||||
# leverage trio's built-in backpressure
|
||||
await send_chan.send(value)
|
||||
|
||||
print(f"FINISHED ITERATING {portal.channel.uid}")
|
||||
uid: tuple[str, str] = portal.chan.uid
|
||||
print(f'FINISHED ITERATING {uid}')
|
||||
|
||||
# spawn 2 trio tasks to collect streams and push to a local queue
|
||||
# spawn 2 trio tasks to collect streams and push to a local
|
||||
# queue
|
||||
n: trio.Nursery
|
||||
async with trio.open_nursery() as n:
|
||||
|
||||
for portal in portals:
|
||||
|
|
@ -61,8 +85,9 @@ async def aggregate(seed):
|
|||
# close this local task's reference to send side
|
||||
await send_chan.aclose()
|
||||
|
||||
unique_vals = set()
|
||||
unique_vals: set[int] = set()
|
||||
async with recv_chan:
|
||||
value: int
|
||||
async for value in recv_chan:
|
||||
if value not in unique_vals:
|
||||
unique_vals.add(value)
|
||||
|
|
@ -71,11 +96,11 @@ async def aggregate(seed):
|
|||
|
||||
assert value in unique_vals
|
||||
|
||||
print("FINISHED ITERATING in aggregator")
|
||||
print('FINISHED ITERATING in aggregator')
|
||||
|
||||
await an.cancel()
|
||||
print("WAITING on `ActorNursery` to finish")
|
||||
print("AGGREGATOR COMPLETE!")
|
||||
print('WAITING on `ActorNursery` to finish')
|
||||
print('AGGREGATOR COMPLETE!')
|
||||
|
||||
|
||||
async def main() -> list[int]:
|
||||
|
|
@ -94,8 +119,8 @@ async def main() -> list[int]:
|
|||
# debug_mode=True,
|
||||
) as an:
|
||||
|
||||
seed = int(1e3)
|
||||
pre_start = time.time()
|
||||
seed: int = int(1e3)
|
||||
pre_start: float = time.time()
|
||||
|
||||
portal: Portal = await an.start_actor(
|
||||
name='aggregator',
|
||||
|
|
@ -108,23 +133,27 @@ async def main() -> list[int]:
|
|||
seed=seed,
|
||||
) as stream:
|
||||
|
||||
start = time.time()
|
||||
start: float = time.time()
|
||||
# the portal call returns exactly what you'd expect
|
||||
# as if the remote "aggregate" function was called locally
|
||||
# as if the remote "aggregate" function was called
|
||||
# locally
|
||||
result_stream: list[int] = []
|
||||
value: int
|
||||
async for value in stream:
|
||||
result_stream.append(value)
|
||||
|
||||
cancelled: bool = await portal.cancel_actor()
|
||||
assert cancelled
|
||||
|
||||
stream_time: float = time.time() - start
|
||||
total_time: float = time.time() - pre_start
|
||||
print(
|
||||
f"STREAM TIME = {time.time() - start}\n"
|
||||
f"STREAM + SPAWN TIME = {time.time() - pre_start}\n"
|
||||
f'STREAM TIME = {stream_time}\n'
|
||||
f'STREAM + SPAWN TIME = {total_time}\n'
|
||||
)
|
||||
assert result_stream == list(range(seed))
|
||||
return result_stream
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
final_stream = trio.run(main)
|
||||
final_stream: list[int] = trio.run(main)
|
||||
|
|
|
|||
|
|
@ -13,7 +13,10 @@ import tractor
|
|||
async def aio_echo_server(
|
||||
chan: tractor.to_asyncio.LinkedTaskChannel,
|
||||
) -> None:
|
||||
'''
|
||||
Echo messages received through an asyncio task channel.
|
||||
|
||||
'''
|
||||
# a first message must be sent **from** this ``asyncio``
|
||||
# task or the ``trio`` side will never unblock from
|
||||
# ``tractor.to_asyncio.open_channel_from():``
|
||||
|
|
@ -28,9 +31,15 @@ async def aio_echo_server(
|
|||
@tractor.context
|
||||
async def trio_to_aio_echo_server(
|
||||
ctx: tractor.Context,
|
||||
):
|
||||
) -> None:
|
||||
'''
|
||||
Bridge an actor stream to the asyncio echo server.
|
||||
|
||||
'''
|
||||
# this will block until the ``asyncio`` task sends a "first"
|
||||
# message.
|
||||
chan: tractor.to_asyncio.LinkedTaskChannel
|
||||
first: str
|
||||
async with tractor.to_asyncio.open_channel_from(
|
||||
aio_echo_server,
|
||||
) as (chan, first):
|
||||
|
|
@ -38,39 +47,49 @@ async def trio_to_aio_echo_server(
|
|||
assert first == 'start'
|
||||
await ctx.started(first)
|
||||
|
||||
stream: tractor.MsgStream
|
||||
async with ctx.open_stream() as stream:
|
||||
|
||||
msg: int
|
||||
async for msg in stream:
|
||||
await chan.send(msg)
|
||||
|
||||
out = await chan.receive()
|
||||
out: int = await chan.receive()
|
||||
# echo back to parent actor-task
|
||||
await stream.send(out)
|
||||
|
||||
|
||||
async def main():
|
||||
async def main() -> None:
|
||||
'''
|
||||
Run the infected asyncio echo-server example.
|
||||
|
||||
'''
|
||||
an: tractor.ActorNursery
|
||||
async with tractor.open_nursery() as an:
|
||||
p = await an.start_actor(
|
||||
portal: tractor.Portal = await an.start_actor(
|
||||
'aio_server',
|
||||
enable_modules=[__name__],
|
||||
infect_asyncio=True,
|
||||
)
|
||||
async with p.open_context(
|
||||
ctx: tractor.Context
|
||||
first: str
|
||||
async with portal.open_context(
|
||||
trio_to_aio_echo_server,
|
||||
) as (ctx, first):
|
||||
|
||||
assert first == 'start'
|
||||
|
||||
count = 0
|
||||
count: int = 0
|
||||
stream: tractor.MsgStream
|
||||
async with ctx.open_stream() as stream:
|
||||
|
||||
delays = []
|
||||
send = time.time()
|
||||
delays: list[float] = []
|
||||
send: float = time.time()
|
||||
|
||||
await stream.send(count)
|
||||
msg: int
|
||||
async for msg in stream:
|
||||
recv = time.time()
|
||||
recv: float = time.time()
|
||||
delays.append(recv - send)
|
||||
assert msg == count
|
||||
count += 1
|
||||
|
|
@ -81,7 +100,7 @@ async def main():
|
|||
break
|
||||
|
||||
print(f'mean round trip rate (Hz): {1/mean(delays)}')
|
||||
await p.cancel_actor()
|
||||
await portal.cancel_actor()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
|
|
|
|||
|
|
@ -1,9 +1,10 @@
|
|||
"""
|
||||
'''
|
||||
Integration test: spawning tractor actors from an MPI process.
|
||||
|
||||
When a parent is launched via ``mpirun``, Open MPI sets ``OMPI_*`` env
|
||||
vars that bind ``MPI_Init`` to the ``orted`` daemon. Tractor children
|
||||
inherit those env vars, so if ``inherit_parent_main=True`` (the default)
|
||||
When a parent is launched via ``mpirun``, Open MPI sets
|
||||
``OMPI_*`` env vars that bind ``MPI_Init`` to the ``orted``
|
||||
daemon. Tractor children inherit those env vars, so if
|
||||
``inherit_parent_main=True`` (the default)
|
||||
the child re-executes ``__main__``, re-imports ``mpi4py``, and
|
||||
``MPI_Init_thread`` fails because the child was never spawned by
|
||||
``orted``::
|
||||
|
|
@ -12,13 +13,15 @@ the child re-executes ``__main__``, re-imports ``mpi4py``, and
|
|||
--> Returned value No permission (-17) instead of ORTE_SUCCESS
|
||||
|
||||
Passing ``inherit_parent_main=False`` and placing RPC functions in a
|
||||
separate importable module (``_child``) avoids the re-import entirely.
|
||||
separate importable module (``_child``) avoids the re-import
|
||||
entirely.
|
||||
|
||||
Usage::
|
||||
|
||||
mpirun --allow-run-as-root -np 1 python -m \
|
||||
examples.integration.mpi4py.inherit_parent_main
|
||||
"""
|
||||
|
||||
'''
|
||||
|
||||
from mpi4py import MPI
|
||||
|
||||
|
|
@ -30,21 +33,27 @@ from ._child import child_fn
|
|||
|
||||
|
||||
async def main() -> None:
|
||||
rank = MPI.COMM_WORLD.Get_rank()
|
||||
print(f"[parent] rank={rank} pid={os.getpid()}", flush=True)
|
||||
'''
|
||||
Spawn an MPI-safe child without replaying the parent main.
|
||||
|
||||
'''
|
||||
rank: int = MPI.COMM_WORLD.Get_rank()
|
||||
pid: int = os.getpid()
|
||||
print(f'[parent] rank={rank} pid={pid}', flush=True)
|
||||
|
||||
an: tractor.ActorNursery
|
||||
async with tractor.open_nursery(start_method='trio') as an:
|
||||
portal = await an.start_actor(
|
||||
portal: tractor.Portal = await an.start_actor(
|
||||
'mpi-child',
|
||||
enable_modules=[child_fn.__module__],
|
||||
# Without this the child replays __main__, which
|
||||
# re-imports mpi4py and crashes on MPI_Init.
|
||||
inherit_parent_main=False,
|
||||
)
|
||||
result = await portal.run(child_fn)
|
||||
print(f"[parent] got: {result}", flush=True)
|
||||
result: str = await portal.run(child_fn)
|
||||
print(f'[parent] got: {result}', flush=True)
|
||||
await portal.cancel_actor()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if __name__ == '__main__':
|
||||
trio.run(main)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,172 @@
|
|||
# `tractor` over a WireGuard tunnel, declared as one maddr
|
||||
|
||||
A two-host LAN setup: a `tractor` actor tree on host A, dialed
|
||||
from host B, with the endpoint declared as a single `wg`
|
||||
multiaddr.
|
||||
|
||||
Supersedes the example set in gh
|
||||
[#482](https://github.com/goodboy/tractor/issues/482) — see
|
||||
[what changed](#what-changed-vs-482).
|
||||
|
||||
> **Why `examples/multihost/`?** `tests/test_docs_examples.py`
|
||||
> walks `examples/` recursively and runs everything it collects
|
||||
> as a subproc, asserting `rc == 0`. These need a real second
|
||||
> host and a live `wg` tunnel, so they can't satisfy that;
|
||||
> `'multihost' not in p[0]` is already in the test's exclusion
|
||||
> list, which is what keeps them out of CI.
|
||||
|
||||
## the maddr form
|
||||
|
||||
```
|
||||
/ip4/192.168.1.50/udp/51820/wg/u<A_pub>/ip4/10.0.11.1/tcp/1616
|
||||
\____ wg bearer ___________/\__ key __/\____ tractor ep _____/
|
||||
underlay, wg `ListenPort` overlay, on the wg iface
|
||||
(kernel owns the socket) (`MsgTransport` binds this)
|
||||
```
|
||||
|
||||
Three parts, three different owners:
|
||||
|
||||
| part | socket owner / provisioner | runtime role |
|
||||
| --- | --- | --- |
|
||||
| `/ip4/../udp/51820` bearer | kernel-owned; `wg-quick` now, tractor bindspace later | control-plane metadata |
|
||||
| `/wg/u<key>` | nothing — it's an identity | parsed, verified explicitly |
|
||||
| `/ip4/../tcp/1616` overlay | `tractor`'s `IPCServer` | application `MsgTransport` |
|
||||
|
||||
Verified against py-multiaddr
|
||||
[#108](https://github.com/multiformats/py-multiaddr/pull/108):
|
||||
this composed form parses and round-trips
|
||||
(`['ip4','udp','wg','ip4','tcp']`).
|
||||
|
||||
## requirements
|
||||
|
||||
py-multiaddr #108 is **merged** (2026-07-28) but ships in no
|
||||
release yet — the latest `0.2.0` (2026-03-17) predates it and has
|
||||
no `wg` codec. So `pyproject.toml` temporarily pins the merge commit
|
||||
in its PEP 621 dependency metadata, and a plain
|
||||
|
||||
```bash
|
||||
uv sync --extra wg
|
||||
```
|
||||
|
||||
gets you a `wg`-aware `multiaddr` plus pyroute2's Linux netlink API.
|
||||
The multiaddr pin goes away once a release carries the codec.
|
||||
`py-multibase` is a direct dependency.
|
||||
|
||||
Without the codec `parse_wg_maddr()` raises immediately with an
|
||||
actionable message — there is deliberately **no** degraded
|
||||
hand-split fallback. `_wg_proto_code()` performs the capability
|
||||
check before parsing.
|
||||
|
||||
Every peel and re-compose here goes through `py-multiaddr`'s own
|
||||
tunnel API (`.decapsulate_code()`, `.split()`, `.join()`,
|
||||
`.encapsulate()`, `.value_for_protocol()`) rather than any
|
||||
bespoke segment slicing — see its README "En/decapsulate" and
|
||||
"Tunneling" sections. gh #429 was about *dropping* our NIH
|
||||
parser, and that applies to peeling a tunnel stack just as much
|
||||
as to decoding one proto.
|
||||
|
||||
## 0. tunnel setup (out-of-band, both hosts)
|
||||
|
||||
Host A is the service host (underlay e.g. `192.168.1.50`), host B
|
||||
your workstation. Overlay net `10.0.11.0/24`.
|
||||
|
||||
```bash
|
||||
umask 077
|
||||
wg genkey | tee wg_priv.key | wg pubkey > wg_pub.key
|
||||
```
|
||||
|
||||
`/etc/wireguard/wg0.conf` on **host A**:
|
||||
|
||||
```ini
|
||||
[Interface]
|
||||
PrivateKey = <A_priv>
|
||||
Address = 10.0.11.1/24
|
||||
ListenPort = 51820
|
||||
```
|
||||
```ini
|
||||
[Peer]
|
||||
PublicKey = <B_pub>
|
||||
AllowedIPs = 10.0.11.2/32
|
||||
```
|
||||
|
||||
on **host B**:
|
||||
|
||||
```ini
|
||||
[Interface]
|
||||
PrivateKey = <B_priv>
|
||||
Address = 10.0.11.2/24
|
||||
```
|
||||
```ini
|
||||
[Peer]
|
||||
PublicKey = <A_pub>
|
||||
Endpoint = 192.168.1.50:51820
|
||||
AllowedIPs = 10.0.11.1/32
|
||||
PersistentKeepalive = 25
|
||||
```
|
||||
|
||||
Note how `ListenPort` and `Endpoint` are exactly the maddr's
|
||||
bearer segment, and `[Interface] Address` is its overlay host.
|
||||
|
||||
```bash
|
||||
sudo wg-quick up wg0 # both hosts
|
||||
ping -c1 10.0.11.1 # from B
|
||||
```
|
||||
|
||||
## 1. get your pubkey into the maddr
|
||||
|
||||
```bash
|
||||
python -c "
|
||||
from tractor.discovery import mb_pubkey
|
||||
key = open('wg_pub.key').read().strip()
|
||||
print(mb_pubkey(key))
|
||||
"
|
||||
```
|
||||
|
||||
Paste the `u...` output into `WG_MADDR` in both scripts (they use
|
||||
the same string — A's bearer, A's key, A's overlay ep).
|
||||
|
||||
## 2. run
|
||||
|
||||
```bash
|
||||
# host A
|
||||
python host_a_srv.py
|
||||
|
||||
# host B
|
||||
python host_b_client.py
|
||||
```
|
||||
|
||||
`host_a_srv.py` must be importable on host B too, since
|
||||
`portal.run()` refs the fn by module path — standard `tractor`
|
||||
RPC semantics.
|
||||
|
||||
## what changed vs #482
|
||||
|
||||
Four corrections, all from
|
||||
`ai/tpt-backends/03_wg_tunnel_bindspace.md`:
|
||||
|
||||
1. **the maddr semantics were inverted.** #482 used
|
||||
`/ip4/10.0.11.1/tcp/1616/wg/u<key>` — that parses, but it puts
|
||||
the *overlay* addr where the bearer belongs and `tcp` where
|
||||
wg's `udp` `ListenPort` goes, and it declares no overlay ep at
|
||||
all. `parse_wg_maddr()` now rejects it with an actionable
|
||||
error.
|
||||
2. **parsing is pure.** #482's helper had the key-check adjacent
|
||||
to the parse; async `verify_wg_peer()` is now a separate,
|
||||
explicitly composed step that the caller invokes. Implicit
|
||||
kernel inspection from a parser is a nasty surprise.
|
||||
3. **no `sudo` or subprocess.** #482 ran `sudo wg show`; tractor's
|
||||
helper reads generic netlink through pyroute2 and never attempts
|
||||
privilege escalation or namespace creation.
|
||||
4. **no new `Address` proto-type.** The tunnel rides *beside* the
|
||||
overlay addr in a frozen `TunnelledAddress`, and only `.overlay`
|
||||
crosses into `open_nursery()`. #482 §6 floated a `WGAddress`
|
||||
registered in `_address_types` — that table is a `bidict`
|
||||
(1:1 proto-key↔type) and `_addr_to_transport` wants a
|
||||
`MsgTransport` per addr-type, which `wg` doesn't have.
|
||||
|
||||
## next
|
||||
|
||||
Layer A's `TunnelledAddress` and native maddr parser plus Layer B's
|
||||
explicit pyroute2 verification now live in `tractor.discovery`. Next,
|
||||
add `open_bindspace()` `@acm`s which create/tear down the iface and
|
||||
netns.
|
||||
|
|
@ -0,0 +1,61 @@
|
|||
# tractor: distributed structured concurrency.
|
||||
'''
|
||||
Host A: the service host, reachable over a `wg` tunnel.
|
||||
|
||||
Binds `tractor`'s registrar + an `echo_srv` sub-actor on the
|
||||
tunnel's *overlay* addr, declared as a single `wg` maddr.
|
||||
|
||||
'''
|
||||
from __future__ import annotations
|
||||
|
||||
import tractor
|
||||
import trio
|
||||
from tractor.discovery import (
|
||||
TunnelledAddress,
|
||||
mk_maddr,
|
||||
parse_wg_maddr,
|
||||
verify_wg_peer,
|
||||
)
|
||||
|
||||
# bearer = host A's underlay `(ip, wg ListenPort)`
|
||||
# key = host A's OWN tunnel pubkey
|
||||
# overlay = the ep `tractor` binds, on the wg iface's addr
|
||||
WG_MADDR: str = (
|
||||
'/ip4/192.168.1.50/udp/51820'
|
||||
'/wg/u<A_pub_b64url>'
|
||||
'/ip4/10.0.11.1/tcp/1616'
|
||||
)
|
||||
|
||||
|
||||
async def echo(msg: str) -> str:
|
||||
actor = tractor.current_actor()
|
||||
return f'{actor.aid.name!r} echoes: {msg}'
|
||||
|
||||
|
||||
async def main():
|
||||
addr: TunnelledAddress = parse_wg_maddr(WG_MADDR)
|
||||
assert await verify_wg_peer(addr.tunnel), (
|
||||
f'wg pubkey from maddr not active on wg0 !\n'
|
||||
f'maddr: {WG_MADDR}\n'
|
||||
f'key: {addr.tunnel.peer_pubkey}\n'
|
||||
)
|
||||
print(
|
||||
f'wg bearer (kernel-owned): {addr.tunnel.bearer}\n'
|
||||
f'tractor overlay ep: {addr.overlay}\n'
|
||||
)
|
||||
async with tractor.open_nursery(
|
||||
# XXX only `.overlay` crosses into the runtime; the bearer
|
||||
# + key are bindspace metadata, never `Endpoint` addrs.
|
||||
registry_addrs=[addr.overlay],
|
||||
enable_transports=[addr.overlay.proto_key],
|
||||
) as an:
|
||||
await an.start_actor(
|
||||
'echo_srv',
|
||||
enable_modules=[__name__],
|
||||
)
|
||||
print(f'echo_srv up on\n {mk_maddr(addr)}\n')
|
||||
await trio.sleep_forever()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
trio.run(main)
|
||||
|
|
@ -0,0 +1,51 @@
|
|||
# tractor: distributed structured concurrency.
|
||||
'''
|
||||
Host B: workstation dialing host A's actor tree through the
|
||||
`wg` tunnel.
|
||||
|
||||
'''
|
||||
from __future__ import annotations
|
||||
|
||||
import tractor
|
||||
import trio
|
||||
from tractor.discovery import (
|
||||
TunnelledAddress,
|
||||
parse_wg_maddr,
|
||||
verify_wg_peer,
|
||||
)
|
||||
|
||||
from host_a_srv import echo # noqa: F401 (RPC refs it by mod path)
|
||||
# same maddr as host A: A's bearer, A's key, A's overlay ep
|
||||
WG_MADDR: str = (
|
||||
'/ip4/192.168.1.50/udp/51820'
|
||||
'/wg/u<A_pub_b64url>'
|
||||
'/ip4/10.0.11.1/tcp/1616'
|
||||
)
|
||||
|
||||
|
||||
async def main():
|
||||
addr: TunnelledAddress = parse_wg_maddr(WG_MADDR)
|
||||
assert await verify_wg_peer(addr.tunnel), (
|
||||
f'wg pubkey from maddr not a peer on wg0 !\n'
|
||||
f'maddr: {WG_MADDR}\n'
|
||||
)
|
||||
async with (
|
||||
tractor.open_root_actor(
|
||||
name='wg_client',
|
||||
registry_addrs=[addr.overlay],
|
||||
enable_transports=[addr.overlay.proto_key],
|
||||
),
|
||||
tractor.find_actor(
|
||||
'echo_srv',
|
||||
registry_addrs=[addr.overlay],
|
||||
) as portal,
|
||||
):
|
||||
res: str = await portal.run(
|
||||
echo,
|
||||
msg='hello over wg!',
|
||||
)
|
||||
print(res)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
trio.run(main)
|
||||
|
|
@ -1,3 +1,5 @@
|
|||
from typing import AsyncIterator
|
||||
|
||||
import trio
|
||||
import tractor
|
||||
|
||||
|
|
@ -5,17 +7,30 @@ import tractor
|
|||
log = tractor.log.get_logger('multiportal')
|
||||
|
||||
|
||||
async def stream_data(seed=10):
|
||||
log.info("Starting stream task")
|
||||
async def stream_data(seed: int = 10) -> AsyncIterator[int]:
|
||||
'''
|
||||
Stream a finite sequence of integers.
|
||||
|
||||
'''
|
||||
log.info('Starting stream task')
|
||||
|
||||
i: int
|
||||
for i in range(seed):
|
||||
yield i
|
||||
await trio.sleep(0) # trigger scheduler
|
||||
|
||||
|
||||
async def stream_from_portal(p, consumed):
|
||||
async def stream_from_portal(
|
||||
portal: tractor.Portal,
|
||||
consumed: list[int],
|
||||
) -> None:
|
||||
'''
|
||||
Consume one stream and toggle each value in a shared list.
|
||||
|
||||
async with p.open_stream_from(stream_data) as stream:
|
||||
'''
|
||||
stream: tractor.MsgStream
|
||||
async with portal.open_stream_from(stream_data) as stream:
|
||||
item: int
|
||||
async for item in stream:
|
||||
if item in consumed:
|
||||
consumed.remove(item)
|
||||
|
|
@ -23,20 +38,33 @@ async def stream_from_portal(p, consumed):
|
|||
consumed.append(item)
|
||||
|
||||
|
||||
async def main():
|
||||
async def main() -> None:
|
||||
'''
|
||||
Consume two concurrent streams through one portal.
|
||||
|
||||
'''
|
||||
an: tractor.ActorNursery
|
||||
async with tractor.open_nursery(loglevel='info') as an:
|
||||
|
||||
p = await an.start_actor('stream_boi', enable_modules=[__name__])
|
||||
portal: tractor.Portal = await an.start_actor(
|
||||
'stream_boi',
|
||||
enable_modules=[__name__],
|
||||
)
|
||||
|
||||
consumed = []
|
||||
consumed: list[int] = []
|
||||
|
||||
n: trio.Nursery
|
||||
async with trio.open_nursery() as n:
|
||||
for i in range(2):
|
||||
n.start_soon(stream_from_portal, p, consumed)
|
||||
for _ in range(2):
|
||||
n.start_soon(
|
||||
stream_from_portal,
|
||||
portal,
|
||||
consumed,
|
||||
)
|
||||
|
||||
# both streaming consumer tasks have completed and so we should
|
||||
# have nothing in our list thanks to single threadedness
|
||||
# both streaming consumer tasks have completed and so we
|
||||
# should have nothing in our list thanks to single
|
||||
# threadedness
|
||||
assert not consumed
|
||||
|
||||
await an.cancel()
|
||||
|
|
|
|||
|
|
@ -41,6 +41,7 @@ async def fan_out_squares(
|
|||
aggregated squares to our parent.
|
||||
|
||||
'''
|
||||
an: tractor.ActorNursery
|
||||
async with tractor.open_nursery() as an:
|
||||
portals: list[tractor.Portal] = []
|
||||
for i in (1, 2):
|
||||
|
|
@ -52,21 +53,28 @@ async def fan_out_squares(
|
|||
)
|
||||
# unblock the parent's `.open_context()` entry and
|
||||
# report which leaves came up.
|
||||
await ctx.started(
|
||||
[p.chan.aid.name for p in portals]
|
||||
)
|
||||
leaf_names: list[str] = [
|
||||
portal.chan.aid.name
|
||||
for portal in portals
|
||||
]
|
||||
await ctx.started(leaf_names)
|
||||
squares: dict[int, int] = {}
|
||||
|
||||
async def run_in_leaf(
|
||||
portal: tractor.Portal,
|
||||
x: int,
|
||||
) -> None:
|
||||
'''
|
||||
Run one square calculation in a leaf actor.
|
||||
|
||||
'''
|
||||
squares[x] = await portal.run(
|
||||
compute_square,
|
||||
x=x,
|
||||
)
|
||||
|
||||
# fan out one sub-RPC per input val, concurrently.
|
||||
tn: trio.Nursery
|
||||
async with trio.open_nursery() as tn:
|
||||
for i, x in enumerate(vals):
|
||||
tn.start_soon(
|
||||
|
|
@ -83,11 +91,18 @@ async def fan_out_squares(
|
|||
|
||||
|
||||
async def main() -> None:
|
||||
'''
|
||||
Run the nested actor-tree example.
|
||||
|
||||
'''
|
||||
an: tractor.ActorNursery
|
||||
async with tractor.open_nursery() as an:
|
||||
portal = await an.start_actor(
|
||||
portal: tractor.Portal = await an.start_actor(
|
||||
'supervisor',
|
||||
enable_modules=[__name__],
|
||||
)
|
||||
ctx: tractor.Context
|
||||
leaf_names: list[str]
|
||||
async with portal.open_context(
|
||||
fan_out_squares,
|
||||
vals=[1, 2, 3, 4],
|
||||
|
|
|
|||
|
|
@ -1,18 +1,23 @@
|
|||
"""
|
||||
'''
|
||||
Demonstration of the prime number detector example from the
|
||||
``concurrent.futures`` docs:
|
||||
|
||||
https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor-example
|
||||
https://docs.python.org/3/library/concurrent.futures.html\
|
||||
#processpoolexecutor-example
|
||||
|
||||
This uses no extra threads, fancy semaphores or futures; all we need
|
||||
is ``tractor``'s channels.
|
||||
|
||||
"""
|
||||
'''
|
||||
from contextlib import (
|
||||
asynccontextmanager as acm,
|
||||
aclosing,
|
||||
)
|
||||
from typing import Callable
|
||||
from typing import (
|
||||
AsyncIterator,
|
||||
Awaitable,
|
||||
Callable,
|
||||
)
|
||||
import itertools
|
||||
import math
|
||||
import time
|
||||
|
|
@ -21,7 +26,12 @@ import tractor
|
|||
import trio
|
||||
|
||||
|
||||
PRIMES = [
|
||||
type ActorMap = Callable[
|
||||
[Callable[[int], Awaitable[bool]], list[int]],
|
||||
AsyncIterator[tuple[int, bool]],
|
||||
]
|
||||
|
||||
PRIMES: list[int] = [
|
||||
112272535095293,
|
||||
112582705942171,
|
||||
112272535095293,
|
||||
|
|
@ -31,7 +41,11 @@ PRIMES = [
|
|||
]
|
||||
|
||||
|
||||
async def is_prime(n):
|
||||
async def is_prime(n: int) -> bool:
|
||||
'''
|
||||
Return whether ``n`` is prime.
|
||||
|
||||
'''
|
||||
if n < 2:
|
||||
return False
|
||||
if n == 2:
|
||||
|
|
@ -47,23 +61,32 @@ async def is_prime(n):
|
|||
|
||||
|
||||
@acm
|
||||
async def worker_pool(workers=4):
|
||||
"""Though it's a trivial special case for ``tractor``, the well
|
||||
async def worker_pool(
|
||||
workers: int = 4,
|
||||
) -> AsyncIterator[ActorMap]:
|
||||
'''
|
||||
Though it's a trivial special case for ``tractor``, the well
|
||||
known "worker pool" seems to be the defacto "but, I want this
|
||||
process pattern!" for most parallelism pilgrims.
|
||||
|
||||
Yes, the workers stay alive (and ready for work) until you close
|
||||
the context.
|
||||
"""
|
||||
|
||||
'''
|
||||
an: tractor.ActorNursery
|
||||
async with tractor.open_nursery() as an:
|
||||
|
||||
portals = []
|
||||
portals: list[tractor.Portal] = []
|
||||
snd_chan: trio.MemorySendChannel[tuple[int, bool]]
|
||||
recv_chan: trio.MemoryReceiveChannel[tuple[int, bool]]
|
||||
snd_chan, recv_chan = trio.open_memory_channel(len(PRIMES))
|
||||
|
||||
i: int
|
||||
for i in range(workers):
|
||||
|
||||
# this starts a new sub-actor (process + trio runtime) and
|
||||
# stores it's "portal" for later use to "submit jobs" (ugh).
|
||||
# this starts a new sub-actor (process + trio
|
||||
# runtime) and stores it's "portal" for later use to
|
||||
# "submit jobs" (ugh).
|
||||
portals.append(
|
||||
await an.start_actor(
|
||||
f'worker_{i}',
|
||||
|
|
@ -72,17 +95,36 @@ async def worker_pool(workers=4):
|
|||
)
|
||||
|
||||
async def _map(
|
||||
worker_func: Callable[[int], bool],
|
||||
sequence: list[int]
|
||||
) -> list[bool]:
|
||||
worker_func: Callable[[int], Awaitable[bool]],
|
||||
sequence: list[int],
|
||||
) -> AsyncIterator[tuple[int, bool]]:
|
||||
'''
|
||||
Dispatch values across workers and yield their results.
|
||||
|
||||
# define an async (local) task to collect results from workers
|
||||
async def send_result(func, value, portal):
|
||||
await snd_chan.send((value, await portal.run(func, n=value)))
|
||||
'''
|
||||
# define an async (local) task to collect results from
|
||||
# workers
|
||||
async def send_result(
|
||||
func: Callable[[int], Awaitable[bool]],
|
||||
value: int,
|
||||
portal: tractor.Portal,
|
||||
) -> None:
|
||||
'''
|
||||
Run one remote worker call and send its result.
|
||||
|
||||
'''
|
||||
result: bool = await portal.run(func, n=value)
|
||||
await snd_chan.send((value, result))
|
||||
|
||||
tn: trio.Nursery
|
||||
async with trio.open_nursery() as tn:
|
||||
|
||||
for value, portal in zip(sequence, itertools.cycle(portals)):
|
||||
value: int
|
||||
portal: tractor.Portal
|
||||
for value, portal in zip(
|
||||
sequence,
|
||||
itertools.cycle(portals),
|
||||
):
|
||||
tn.start_soon(
|
||||
send_result,
|
||||
worker_func,
|
||||
|
|
@ -101,21 +143,30 @@ async def worker_pool(workers=4):
|
|||
await an.cancel()
|
||||
|
||||
|
||||
async def main():
|
||||
async def main() -> None:
|
||||
'''
|
||||
Report primality results from a pool of actors.
|
||||
|
||||
'''
|
||||
actor_map: ActorMap
|
||||
async with worker_pool() as actor_map:
|
||||
|
||||
start = time.time()
|
||||
start: float = time.time()
|
||||
|
||||
results: AsyncIterator[tuple[int, bool]]
|
||||
async with aclosing(actor_map(is_prime, PRIMES)) as results:
|
||||
number: int
|
||||
prime: bool
|
||||
async for number, prime in results:
|
||||
|
||||
print(f'{number} is prime: {prime}')
|
||||
|
||||
print(f'processing took {time.time() - start} seconds')
|
||||
elapsed: float = time.time() - start
|
||||
print(f'processing took {elapsed} seconds')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
start = time.time()
|
||||
start: float = time.time()
|
||||
trio.run(main)
|
||||
print(f'script took {time.time() - start} seconds')
|
||||
elapsed: float = time.time() - start
|
||||
print(f'script took {elapsed} seconds')
|
||||
|
|
|
|||
|
|
@ -1,30 +1,36 @@
|
|||
"""
|
||||
'''
|
||||
Run with a process monitor from a terminal using::
|
||||
|
||||
$TERM -e watch -n 0.1 "pstree -a $$" \
|
||||
& python examples/parallelism/single_func.py \
|
||||
&& kill $!
|
||||
|
||||
"""
|
||||
'''
|
||||
import os
|
||||
|
||||
import tractor
|
||||
import trio
|
||||
|
||||
|
||||
async def burn_cpu():
|
||||
async def burn_cpu() -> int:
|
||||
'''
|
||||
Burn CPU briefly and return the current process ID.
|
||||
|
||||
pid = os.getpid()
|
||||
'''
|
||||
pid: int = os.getpid()
|
||||
|
||||
# burn a core @ ~ 50kHz
|
||||
for _ in range(50000):
|
||||
await trio.sleep(1/50000/50)
|
||||
await trio.sleep(1 / 50000 / 50)
|
||||
|
||||
return pid
|
||||
|
||||
|
||||
async def main():
|
||||
async def main() -> None:
|
||||
'''
|
||||
Run ``burn_cpu()`` in the parent and a subactor.
|
||||
|
||||
'''
|
||||
async with trio.open_nursery() as tn:
|
||||
|
||||
# burn rubber in the parent too
|
||||
|
|
@ -32,9 +38,9 @@ async def main():
|
|||
|
||||
# run the same func as the lone task in a subactor,
|
||||
# block on and collect its PID as the caller-side result
|
||||
pid = await tractor.to_actor.run(burn_cpu)
|
||||
pid: int = await tractor.to_actor.run(burn_cpu)
|
||||
|
||||
print(f"Collected subproc {pid}")
|
||||
print(f'Collected subproc {pid}')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
|
|
|
|||
|
|
@ -1,20 +1,24 @@
|
|||
|
||||
import trio
|
||||
import tractor
|
||||
|
||||
|
||||
async def sleepy_jane() -> None:
|
||||
uid: tuple = tractor.current_actor().uid
|
||||
'''
|
||||
Identify the current actor and sleep forever.
|
||||
|
||||
'''
|
||||
uid: tuple[str, str] = tractor.current_actor().uid
|
||||
print(f'Yo i am actor {uid}')
|
||||
await trio.sleep_forever()
|
||||
|
||||
|
||||
async def main():
|
||||
async def main() -> None:
|
||||
'''
|
||||
Spawn a flat actor cluster, with one process per detected core.
|
||||
|
||||
'''
|
||||
portal_map: dict[str, tractor.Portal]
|
||||
tn: trio.Nursery
|
||||
|
||||
# look at this hip new syntax!
|
||||
async with (
|
||||
|
|
@ -27,7 +31,9 @@ async def main():
|
|||
trio.open_nursery() as tn,
|
||||
):
|
||||
|
||||
for (name, portal) in portal_map.items():
|
||||
name: str
|
||||
portal: tractor.Portal
|
||||
for name, portal in portal_map.items():
|
||||
tn.start_soon(
|
||||
portal.run,
|
||||
sleepy_jane,
|
||||
|
|
|
|||
|
|
@ -2,18 +2,30 @@ import trio
|
|||
import tractor
|
||||
|
||||
|
||||
async def assert_err():
|
||||
async def assert_err() -> None:
|
||||
'''
|
||||
Raise an assertion error in the current actor.
|
||||
|
||||
'''
|
||||
assert 0
|
||||
|
||||
|
||||
async def main():
|
||||
async def main() -> None:
|
||||
'''
|
||||
Propagate a failing one-shot task from a subactor.
|
||||
|
||||
'''
|
||||
an: tractor.ActorNursery
|
||||
async with tractor.open_nursery() as an:
|
||||
real_actors = []
|
||||
real_actors: list[tractor.Portal] = []
|
||||
i: int
|
||||
for i in range(3):
|
||||
real_actors.append(await an.start_actor(
|
||||
f'actor_{i}',
|
||||
enable_modules=[__name__],
|
||||
))
|
||||
real_actors.append(
|
||||
await an.start_actor(
|
||||
f'actor_{i}',
|
||||
enable_modules=[__name__],
|
||||
)
|
||||
)
|
||||
|
||||
# run one one-shot task actor that will fail immediately;
|
||||
# its error raises right here in the caller's task..
|
||||
|
|
@ -28,4 +40,4 @@ if __name__ == '__main__':
|
|||
# also raises
|
||||
trio.run(main)
|
||||
except tractor.RemoteActorError:
|
||||
print("Look Maa that actor failed hard, hehhh!")
|
||||
print('Look Maa that actor failed hard, hehhh!')
|
||||
|
|
|
|||
|
|
@ -9,16 +9,19 @@ async def simple_rpc(
|
|||
data: int,
|
||||
|
||||
) -> None:
|
||||
'''Test a small ping-pong 2-way streaming server.
|
||||
'''
|
||||
Test a small ping-pong 2-way streaming server.
|
||||
|
||||
'''
|
||||
# signal to parent that we're up much like
|
||||
# ``trio.TaskStatus.started()``
|
||||
await ctx.started(data + 1)
|
||||
|
||||
stream: tractor.MsgStream
|
||||
async with ctx.open_stream() as stream:
|
||||
|
||||
count = 0
|
||||
count: int = 0
|
||||
msg: str
|
||||
async for msg in stream:
|
||||
|
||||
assert msg == 'ping'
|
||||
|
|
@ -30,15 +33,22 @@ async def simple_rpc(
|
|||
|
||||
|
||||
async def main() -> None:
|
||||
'''
|
||||
Exercise bidirectional streaming with a remote actor.
|
||||
|
||||
'''
|
||||
an: tractor.ActorNursery
|
||||
async with tractor.open_nursery() as an:
|
||||
|
||||
portal = await an.start_actor(
|
||||
portal: tractor.Portal = await an.start_actor(
|
||||
'rpc_server',
|
||||
enable_modules=[__name__],
|
||||
)
|
||||
|
||||
# XXX: syntax requires py3.9
|
||||
ctx: tractor.Context
|
||||
sent: int
|
||||
stream: tractor.MsgStream
|
||||
async with (
|
||||
|
||||
portal.open_context(
|
||||
|
|
@ -52,10 +62,11 @@ async def main() -> None:
|
|||
|
||||
assert sent == 11
|
||||
|
||||
count = 0
|
||||
count: int = 0
|
||||
# receive msgs using async for style
|
||||
await stream.send('ping')
|
||||
|
||||
msg: str
|
||||
async for msg in stream:
|
||||
assert msg == 'pong'
|
||||
await stream.send('ping')
|
||||
|
|
|
|||
|
|
@ -35,11 +35,13 @@ async def client_task() -> None:
|
|||
|
||||
'''
|
||||
# a lookup miss yields `None` (not an error).
|
||||
async with tractor.find_actor('no_such_svc') as portal:
|
||||
assert portal is None
|
||||
maybe_portal: tractor.Portal|None
|
||||
async with tractor.find_actor('no_such_svc') as maybe_portal:
|
||||
assert maybe_portal is None
|
||||
print('client: "no_such_svc" is not registered')
|
||||
# block until the service shows up in the registry,
|
||||
# then call into it through the delivered portal.
|
||||
portal: tractor.Portal
|
||||
async with tractor.wait_for_actor('quote_svc') as portal:
|
||||
quote: float = await portal.run(
|
||||
get_quote,
|
||||
|
|
@ -49,13 +51,19 @@ async def client_task() -> None:
|
|||
|
||||
|
||||
async def main() -> None:
|
||||
'''
|
||||
Run a discoverable quote service and its client.
|
||||
|
||||
'''
|
||||
an: tractor.ActorNursery
|
||||
async with tractor.open_nursery() as an:
|
||||
portal = await an.start_actor(
|
||||
portal: tractor.Portal = await an.start_actor(
|
||||
'quote_svc',
|
||||
enable_modules=[__name__],
|
||||
)
|
||||
# run the client in a separate task which discovers
|
||||
# the daemon purely by its registered name.
|
||||
tn: trio.Nursery
|
||||
async with trio.open_nursery() as tn:
|
||||
tn.start_soon(client_task)
|
||||
# explicit graceful teardown of the daemon.
|
||||
|
|
|
|||
|
|
@ -1,19 +1,29 @@
|
|||
import trio
|
||||
import tractor
|
||||
|
||||
tractor.log.get_console_log("INFO")
|
||||
tractor.log.get_console_log('INFO')
|
||||
|
||||
|
||||
async def main(service_name):
|
||||
async def main(service_name: str) -> None:
|
||||
'''
|
||||
Discover one actor and inspect its registrar connection.
|
||||
|
||||
'''
|
||||
an: tractor.ActorNursery
|
||||
async with tractor.open_nursery() as an:
|
||||
await an.start_actor(service_name)
|
||||
|
||||
async with tractor.get_registry() as portal:
|
||||
print(f"Registrar is listening on {portal.channel}")
|
||||
async with tractor.get_registry() as reg_portal:
|
||||
print(
|
||||
f'Registrar is listening on {reg_portal.channel}'
|
||||
)
|
||||
|
||||
async with tractor.wait_for_actor(service_name) as sockaddr:
|
||||
print(f"my_service is found at {sockaddr}")
|
||||
actor_portal: tractor.Portal
|
||||
async with tractor.wait_for_actor(
|
||||
service_name,
|
||||
) as actor_portal:
|
||||
service_addr = actor_portal.chan.raddr
|
||||
print(f'my_service is found at {service_addr}')
|
||||
|
||||
await an.cancel()
|
||||
|
||||
|
|
|
|||
|
|
@ -28,6 +28,7 @@ async def tick_stream(
|
|||
# wait for the go-signal ensuring every parent-side
|
||||
# subscriber is attached before any tick is sent.
|
||||
assert await stream.receive() == 'go'
|
||||
i: int
|
||||
for i in range(count):
|
||||
await stream.send(i)
|
||||
# falling out gracefully closes our stream side;
|
||||
|
|
@ -37,15 +38,17 @@ async def tick_stream(
|
|||
async def consume(
|
||||
name: str,
|
||||
stream: tractor.MsgStream,
|
||||
task_status: trio.TaskStatus = trio.TASK_STATUS_IGNORED,
|
||||
task_status: trio.TaskStatus[None] = trio.TASK_STATUS_IGNORED,
|
||||
) -> None:
|
||||
'''
|
||||
Consume a private broadcast-copy of the IPC stream.
|
||||
|
||||
'''
|
||||
bcaster: tractor.trionics.BroadcastReceiver
|
||||
async with stream.subscribe() as bcaster:
|
||||
task_status.started()
|
||||
ticks: list[int] = []
|
||||
tick: int
|
||||
async for tick in bcaster:
|
||||
print(f'{name}: rx {tick}')
|
||||
ticks.append(tick)
|
||||
|
|
@ -54,11 +57,19 @@ async def consume(
|
|||
|
||||
|
||||
async def main() -> None:
|
||||
'''
|
||||
Fan one remote stream out to local subscribers.
|
||||
|
||||
'''
|
||||
an: tractor.ActorNursery
|
||||
async with tractor.open_nursery() as an:
|
||||
portal = await an.start_actor(
|
||||
portal: tractor.Portal = await an.start_actor(
|
||||
'ticker',
|
||||
enable_modules=[__name__],
|
||||
)
|
||||
ctx: tractor.Context
|
||||
first: int
|
||||
stream: tractor.MsgStream
|
||||
async with (
|
||||
portal.open_context(
|
||||
tick_stream,
|
||||
|
|
@ -67,9 +78,11 @@ async def main() -> None:
|
|||
ctx.open_stream() as stream,
|
||||
):
|
||||
assert first == 5
|
||||
tn: trio.Nursery
|
||||
async with trio.open_nursery() as tn:
|
||||
# use `.start()` so each consumer is known
|
||||
# to be subscribed before the ticks flow.
|
||||
i: int
|
||||
for i in range(3):
|
||||
await tn.start(
|
||||
consume,
|
||||
|
|
|
|||
|
|
@ -1,3 +1,4 @@
|
|||
from collections.abc import AsyncIterator
|
||||
from contextlib import (
|
||||
asynccontextmanager as acm,
|
||||
)
|
||||
|
|
@ -16,7 +17,11 @@ _lock: trio.Lock|None = None
|
|||
|
||||
@acm
|
||||
async def acquire_singleton_lock(
|
||||
) -> None:
|
||||
) -> AsyncIterator[trio.Lock]:
|
||||
'''
|
||||
Acquire and yield the process-wide lock.
|
||||
|
||||
'''
|
||||
global _lock
|
||||
if _lock is None:
|
||||
log.info('Allocating LOCK')
|
||||
|
|
@ -32,8 +37,15 @@ async def acquire_singleton_lock(
|
|||
|
||||
|
||||
async def hold_lock_forever(
|
||||
task_status=trio.TASK_STATUS_IGNORED
|
||||
):
|
||||
task_status: trio.TaskStatus[
|
||||
trio.Lock,
|
||||
] = trio.TASK_STATUS_IGNORED,
|
||||
) -> None:
|
||||
'''
|
||||
Hold the singleton lock until cancellation.
|
||||
|
||||
'''
|
||||
lock: trio.Lock
|
||||
async with (
|
||||
tractor.trionics.maybe_raise_from_masking_exc(),
|
||||
acquire_singleton_lock() as lock,
|
||||
|
|
@ -46,7 +58,12 @@ async def main(
|
|||
ignore_special_cases: bool,
|
||||
loglevel: str = 'info',
|
||||
debug_mode: bool = True,
|
||||
):
|
||||
) -> None:
|
||||
'''
|
||||
Exercise lock acquisition while cancellation is masked.
|
||||
|
||||
'''
|
||||
tn: trio.Nursery
|
||||
async with (
|
||||
trio.open_nursery() as tn,
|
||||
|
||||
|
|
@ -58,7 +75,7 @@ async def main(
|
|||
from tractor.trionics import _taskc
|
||||
_taskc._mask_cases.clear()
|
||||
|
||||
_lock = await tn.start(
|
||||
_held_lock: trio.Lock = await tn.start(
|
||||
hold_lock_forever,
|
||||
)
|
||||
with trio.move_on_after(0.2):
|
||||
|
|
@ -74,8 +91,8 @@ if __name__ == '__main__':
|
|||
tractor.log.get_console_log(level='info')
|
||||
for case in [True, False]:
|
||||
log.info(
|
||||
f'\n'
|
||||
f'------ RUNNING SCRIPT TRIAL ------\n'
|
||||
'\n'
|
||||
'------ RUNNING SCRIPT TRIAL ------\n'
|
||||
f'ignore_special_cases: {case!r}\n'
|
||||
)
|
||||
trio.run(partial(
|
||||
|
|
|
|||
|
|
@ -1,3 +1,4 @@
|
|||
from collections.abc import Iterator
|
||||
from contextlib import (
|
||||
contextmanager as cm,
|
||||
# TODO, any diff in async case(s)??
|
||||
|
|
@ -17,7 +18,7 @@ log = tractor.log.get_logger(
|
|||
@cm
|
||||
def teardown_on_exc(
|
||||
raise_from_handler: bool = False,
|
||||
):
|
||||
) -> Iterator[None]:
|
||||
'''
|
||||
You could also have a teardown handler which catches any exc and
|
||||
does some required teardown. In this case the problem is
|
||||
|
|
@ -30,7 +31,7 @@ def teardown_on_exc(
|
|||
except BaseException as _berr:
|
||||
berr = _berr
|
||||
log.exception(
|
||||
f'Handling termination teardown in child due to,\n'
|
||||
'Handling termination teardown in child due to,\n'
|
||||
f'{berr!r}\n'
|
||||
)
|
||||
if raise_from_handler:
|
||||
|
|
@ -54,14 +55,18 @@ def teardown_on_exc(
|
|||
|
||||
|
||||
async def finite_stream_to_rent(
|
||||
tx: trio.abc.SendChannel,
|
||||
tx: trio.abc.SendChannel[int],
|
||||
child_errors_mid_stream: bool,
|
||||
raise_unmasked: bool,
|
||||
|
||||
task_status: trio.TaskStatus[
|
||||
trio.CancelScope,
|
||||
trio.CancelScope|None,
|
||||
] = trio.TASK_STATUS_IGNORED,
|
||||
):
|
||||
) -> None:
|
||||
'''
|
||||
Stream values while reproducing exception masking on close.
|
||||
|
||||
'''
|
||||
async with (
|
||||
# XXX without this unmasker the mid-streaming RTE is never
|
||||
# reported since it is masked by the `tx.aclose()`
|
||||
|
|
@ -134,19 +139,26 @@ async def main(
|
|||
|
||||
raise_unmasked: bool = False,
|
||||
loglevel: str = 'info',
|
||||
):
|
||||
) -> None:
|
||||
'''
|
||||
Reproduce cancellation masking a child-stream failure.
|
||||
|
||||
'''
|
||||
tractor.log.get_console_log(level=loglevel)
|
||||
|
||||
# the `.aclose()` being checkpoints on these
|
||||
# is the source of the problem..
|
||||
tx: trio.MemorySendChannel[int]
|
||||
rx: trio.MemoryReceiveChannel[int]
|
||||
tx, rx = trio.open_memory_channel(1)
|
||||
|
||||
tn: trio.Nursery
|
||||
async with (
|
||||
tractor.trionics.collapse_eg(),
|
||||
trio.open_nursery() as tn,
|
||||
rx as rx,
|
||||
):
|
||||
_child_cs = await tn.start(
|
||||
_child_cs: trio.CancelScope|None = await tn.start(
|
||||
partial(
|
||||
finite_stream_to_rent,
|
||||
child_errors_mid_stream=child_errors_mid_stream,
|
||||
|
|
@ -154,6 +166,7 @@ async def main(
|
|||
tx=tx,
|
||||
)
|
||||
)
|
||||
msg: int
|
||||
async for msg in rx:
|
||||
log.debug(
|
||||
f'Rent rx {msg!r}\n'
|
||||
|
|
@ -162,12 +175,13 @@ async def main(
|
|||
# simulate some external cancellation
|
||||
# request **JUST BEFORE** the child errors.
|
||||
if msg == 65:
|
||||
log.cancel(
|
||||
f'Cancelling parent on,\n'
|
||||
f'msg={msg}\n'
|
||||
f'\n'
|
||||
f'Simulates OOB cancel request!\n'
|
||||
)
|
||||
cancel_msg: str = (
|
||||
'Cancelling parent on,\n'
|
||||
'msg={msg}\n'
|
||||
'\n'
|
||||
'Simulates OOB cancel request!\n'
|
||||
).format(msg=msg)
|
||||
log.cancel(cancel_msg)
|
||||
tn.cancel_scope.cancel()
|
||||
|
||||
|
||||
|
|
@ -176,8 +190,8 @@ if __name__ == '__main__':
|
|||
tractor.log.get_console_log(level='info')
|
||||
for case in [True, False]:
|
||||
log.info(
|
||||
f'\n'
|
||||
f'------ RUNNING SCRIPT TRIAL ------\n'
|
||||
'\n'
|
||||
'------ RUNNING SCRIPT TRIAL ------\n'
|
||||
f'child_errors_midstream: {case!r}\n'
|
||||
)
|
||||
try:
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue