Add the `tipc_cluster` example set
Plan 01 §8's deployment deliverable, under `examples/multihost/` (like the `wg_lan` set) since these need the `tipc` kernel module — and, for the 2-host pair, a live bearer — so they can't satisfy `test_docs_examples.py`'s "walk `examples/` and assert rc == 0". `'multihost'` is already in that test's exclusion list. - `single_host.py` — boots a 4-actor tree and shells out to `tipc nametable show` before/during/after. Watching 4 service names appear in the KERNEL's table and vanish on teardown, entirely outside any `tractor` API, is the single best demo this backend has. - `watch_nametable.py` — the same story push-based, via `open_topology_events()`: live `[+] published` / `[-] withdrawn` as actors come and go. - `host_a_srv.py` + `host_b_client.py` — the cross-node pair. Note what's absent from both: any IP, hostname or port. Both sides name the same *service* and the kernel routes it. - `README.md` — the manual smoke test (bearer setup, `tipc link list` verify) per §7.3, plus the gotchas: silent crosstalk, graceful-close-looks-like-`ECONNRESET`, the interim maddr. Both single-host scripts were RUN against a live kernel and their real output is what's pasted in the README. (this patch was generated in some part by `claude-code` using `claude-opus-5` (`anthropic`))wkt/pr493_review
parent
33a040b312
commit
b14332017d
|
|
@ -0,0 +1,213 @@
|
|||
# `tractor` over `AF_TIPC`, where the address *is* the service name
|
||||
|
||||
TIPC is a linux-kernel cluster IPC protocol whose service names
|
||||
live in a **cluster-wide name table maintained by the kernel**.
|
||||
For `tractor` that means:
|
||||
|
||||
- an actor's IPC address is a service name `(stype, instance)`,
|
||||
not a host/port,
|
||||
- `.bind()`ing it **is** service registration,
|
||||
- a peer's `.connect()`-by-name **is** the lookup.
|
||||
|
||||
So the discovery machinery `tractor.discovery` normally
|
||||
implements with a registrar actor comes for free, in-kernel —
|
||||
which is the ask in gh
|
||||
[#378](https://github.com/goodboy/tractor/issues/378).
|
||||
|
||||
> **Why `examples/multihost/`?** `tests/test_docs_examples.py`
|
||||
> walks `examples/` recursively and runs everything it collects
|
||||
> as a subproc, asserting `rc == 0`. These need the `tipc`
|
||||
> kernel module (and, for the two-host pair, a live bearer), so
|
||||
> they can't satisfy that; `'multihost' not in p[0]` is already
|
||||
> in the test's exclusion list, which is what keeps them out of
|
||||
> CI. See "CI" below for the separate matrix-entry plan.
|
||||
|
||||
## the single best demo
|
||||
|
||||
```bash
|
||||
sudo modprobe tipc
|
||||
python single_host.py
|
||||
```
|
||||
|
||||
Four actors boot, four service names appear in the kernel's
|
||||
table, and all four are withdrawn on teardown — observed with
|
||||
`tipc(8)`, entirely outside `tractor`:
|
||||
|
||||
```
|
||||
--- `tipc nametable show` :: root + 3 subactors ---
|
||||
Type Lower Upper Scope Port
|
||||
1953628160 1616 1616 cluster 3161982128
|
||||
1953628160 1219427151 1219427151 cluster 1587358717
|
||||
1953628160 2641339936 2641339936 cluster 1864021571
|
||||
1953628160 3344505866 3344505866 cluster 3816483388
|
||||
|
||||
--- `tipc nametable show` :: after teardown (all withdrawn) ---
|
||||
Type Lower Upper Scope Port
|
||||
```
|
||||
|
||||
`1953628160` is `0x74720000` — `tractor`'s reserved service
|
||||
type, ascii `tr` in the high half. `1616` is the host-singleton
|
||||
registrar, the same idiom as the TCP port and the
|
||||
`registry@1616.sock` UDS filename. The other three instances are
|
||||
per-actor digests (see "silent crosstalk" below).
|
||||
|
||||
## push-based discovery
|
||||
|
||||
```bash
|
||||
python watch_nametable.py
|
||||
```
|
||||
|
||||
Subscribes to the kernel's *topology service* and prints name
|
||||
table transitions as they happen — no polling, no registrar
|
||||
round-trip:
|
||||
|
||||
```
|
||||
watching the TIPC name table..
|
||||
[+] published instance=1616 port=0x00000000:2375440573
|
||||
spawning subactors..
|
||||
[+] published instance=186947472 port=0x00000000:3960753074
|
||||
[+] published instance=2191362136 port=0x00000000:2263898853
|
||||
[+] published instance=3484369663 port=0x00000000:2126817956
|
||||
tearing down..
|
||||
[-] withdrawn instance=186947472 port=0x00000000:3960753074
|
||||
...
|
||||
```
|
||||
|
||||
This is the groundwork for a push registry in
|
||||
`tractor.discovery._registry` (gh
|
||||
[#184](https://github.com/goodboy/tractor/issues/184),
|
||||
[#216](https://github.com/goodboy/tractor/issues/216)) — a
|
||||
registrar that *never polls* `find_actor()`.
|
||||
|
||||
## two hosts
|
||||
|
||||
Everything above is single-node (`modprobe` is enough). To span
|
||||
hosts you need a **bearer** on both, which is the one thing that
|
||||
can't be CI'd.
|
||||
|
||||
```bash
|
||||
# on BOTH hosts
|
||||
sudo modprobe tipc
|
||||
|
||||
# over ethernet (L2) — simplest when the hosts share a segment
|
||||
sudo tipc bearer enable media eth device eth0
|
||||
|
||||
# ..or over UDP when L2 isn't available (pairs nicely with the
|
||||
# `wg` tunnel examples in ../wg_lan/)
|
||||
sudo tipc bearer enable media udp name uc localip 10.0.11.1
|
||||
|
||||
# verify BEFORE running anything: this must list the peer
|
||||
tipc link list
|
||||
tipc node list
|
||||
```
|
||||
|
||||
Then:
|
||||
|
||||
```bash
|
||||
# host A
|
||||
python host_a_srv.py
|
||||
|
||||
# host B
|
||||
python host_b_client.py
|
||||
```
|
||||
|
||||
Note what's absent from both scripts: any IP, hostname or port.
|
||||
Both sides name the *same service*, and the kernel routes it.
|
||||
Move `host_a_srv.py` to a third node and host B's dial keeps
|
||||
working, unchanged.
|
||||
|
||||
### scope
|
||||
|
||||
`TIPCAddress._scope` is the backend's `.bindspace` — literally
|
||||
"the set of hosts this published name is reachable from":
|
||||
|
||||
| scope | meaning |
|
||||
| --- | --- |
|
||||
| `TIPC_NODE_SCOPE` | same host only — the UDS analogue |
|
||||
| `TIPC_CLUSTER_SCOPE` | cluster-visible (the default) |
|
||||
|
||||
`TIPC_ZONE_SCOPE` is deprecated and aliased to cluster by modern
|
||||
kernels; `tractor` accepts it on input and folds it, logging at
|
||||
`transport` level.
|
||||
|
||||
## gotchas worth knowing before you deploy
|
||||
|
||||
**Silent crosstalk.** Unlike every other backend, a duplicate
|
||||
bind does **not** raise `EADDRINUSE` — TIPC happily accepts
|
||||
multiple publishers of one name and *round-robins* connects
|
||||
between them (verified: 6 dials alternated `b,a,b,a,b,a`). So an
|
||||
instance collision is silent traffic-splitting, not an error.
|
||||
That's why `TIPCAddress.get_random()` derives the instance from
|
||||
a `blake2b` digest of the actor identity rather than a counter.
|
||||
Two `tractor` trees sharing both a cluster **and** an `_stype`
|
||||
share a name space; partition them by passing a distinct
|
||||
`_stype`.
|
||||
|
||||
**Graceful close looks like a reset.** A peer closing cleanly
|
||||
surfaces as `BrokenResourceError`/`ECONNRESET` rather than the
|
||||
clean 0-byte EOF you get from TCP/UDS. It's benign — the
|
||||
transport layer already classifies it as a normal disconnect —
|
||||
but it does look alarming in `transport`-level logs.
|
||||
|
||||
**Dialing an unpublished name** answers `EHOSTUNREACH`
|
||||
*instantly* (no SYN-timeout wait), which is much better
|
||||
discovery-ping behaviour than TCP. `tractor` normalizes it to
|
||||
`ConnectionError`.
|
||||
|
||||
**It's opt-in, never a default.** The module isn't loaded on
|
||||
most boxes and doesn't exist off-linux, so
|
||||
`enable_transports=['tipc']` is always explicit. Check
|
||||
`tractor.ipc._tipc.is_tipc_available()` before assuming.
|
||||
|
||||
## maddr form
|
||||
|
||||
There is no registered `/tipc` protocol in the multiaddr table
|
||||
yet (upstream track: gh
|
||||
[#483](https://github.com/goodboy/tractor/issues/483) +
|
||||
multiformats/py-multiaddr#107), so the grammar is interim and
|
||||
`str`-only:
|
||||
|
||||
```
|
||||
/tipc/<stype>/<instance>/<scope>
|
||||
```
|
||||
|
||||
`parse_maddr()` special-cases this prefix *before* handing
|
||||
anything to `Multiaddr()`, which would otherwise reject the
|
||||
unregistered name outright. Registering it upstream is what
|
||||
would unblock gh
|
||||
[#443](https://github.com/goodboy/tractor/issues/443)'s
|
||||
"return `Multiaddr` everywhere" item.
|
||||
|
||||
## running the suite over TIPC
|
||||
|
||||
The whole test suite runs under the backend:
|
||||
|
||||
```bash
|
||||
sudo modprobe tipc
|
||||
pytest --tpt-proto tipc
|
||||
```
|
||||
|
||||
Without the module that fails loudly and immediately with an
|
||||
actionable message rather than a few hundred connect timeouts.
|
||||
Backend-specific unit tests live in `tests/ipc/test_tipc.py` and
|
||||
self-skip when the module is absent.
|
||||
|
||||
## CI
|
||||
|
||||
Single-host TIPC *is* CI-able — the module ships with the
|
||||
standard Ubuntu kernel package, so a `sudo modprobe tipc` step
|
||||
plus a `--tpt-proto tipc` matrix entry should work. That's not
|
||||
wired up yet; verify in a throwaway workflow first, and fall
|
||||
back to a container job with `--cap-add NET_ADMIN` if the
|
||||
runners refuse. Cross-node (bearer) testing stays manual — this
|
||||
README is that smoke test.
|
||||
|
||||
## normative refs
|
||||
|
||||
The tipc.io docs are stale in places (gh #378 says as much).
|
||||
Treat the kernel sources as the only normative reference:
|
||||
|
||||
- `include/uapi/linux/tipc.h` — address flavours, sockopts, the
|
||||
topology `struct`s
|
||||
- `net/tipc/socket.c`, `net/tipc/topsrv.c`
|
||||
- `man 8 tipc`
|
||||
|
|
@ -0,0 +1,69 @@
|
|||
'''
|
||||
HOST A — publish a `tractor` service on a cluster-scoped TIPC
|
||||
service name.
|
||||
|
||||
Note what is NOT in this file: any IP address, hostname or port.
|
||||
The actor's address IS the service name `(stype, instance)`, and
|
||||
the kernel routes it over whatever bearer you enabled. Move this
|
||||
process to another node and host B's dial keeps working,
|
||||
unchanged.
|
||||
|
||||
Prereqs on BOTH hosts (see README.md),
|
||||
|
||||
sudo modprobe tipc
|
||||
sudo tipc bearer enable media eth device <iface>
|
||||
tipc link list # must show a link to the peer
|
||||
|
||||
Then here,
|
||||
|
||||
python host_a_srv.py
|
||||
|
||||
'''
|
||||
from __future__ import annotations
|
||||
|
||||
import trio
|
||||
import tractor
|
||||
from tractor.ipc._tipc import (
|
||||
TIPCAddress,
|
||||
is_tipc_available,
|
||||
)
|
||||
|
||||
|
||||
@tractor.context
|
||||
async def echo(
|
||||
ctx: tractor.Context,
|
||||
) -> None:
|
||||
await ctx.started()
|
||||
async with ctx.open_stream() as stream:
|
||||
async for msg in stream:
|
||||
print(f'host-a <- {msg!r}')
|
||||
await stream.send(f'{msg} (from host A)')
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
# the host-singleton registrar name, `instance=1616` —
|
||||
# the same "1616 is tractor's registrar" idiom as the tcp
|
||||
# port and the `registry@1616.sock` UDS filename.
|
||||
reg: TIPCAddress = TIPCAddress.get_root()
|
||||
print(f'host A publishing {reg}')
|
||||
|
||||
async with tractor.open_root_actor(
|
||||
name='host_a',
|
||||
enable_transports=['tipc'],
|
||||
registry_addrs=[reg.unwrap()],
|
||||
enable_modules=[__name__],
|
||||
):
|
||||
print(
|
||||
'registrar up — `tipc nametable show` on EITHER host\n'
|
||||
'should now list this service. ctrl-c to stop.'
|
||||
)
|
||||
await trio.sleep_forever()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
if not is_tipc_available():
|
||||
raise RuntimeError(
|
||||
'The `tipc` kernel module is not loaded!\n'
|
||||
' |_try: `sudo modprobe tipc`\n'
|
||||
)
|
||||
trio.run(main)
|
||||
|
|
@ -0,0 +1,59 @@
|
|||
'''
|
||||
HOST B — dial host A's service *by name*, across the cluster.
|
||||
|
||||
The `.connect()` on a TIPC service name IS the discovery lookup:
|
||||
the kernel resolves the published name to whichever node serves
|
||||
it. So this client needs no IP, no port and no idea where host A
|
||||
actually is.
|
||||
|
||||
Prereqs: same bearer setup as `host_a_srv.py`, and that script
|
||||
already running on the other node.
|
||||
|
||||
python host_b_client.py
|
||||
|
||||
'''
|
||||
from __future__ import annotations
|
||||
|
||||
import trio
|
||||
import tractor
|
||||
from tractor.ipc._tipc import (
|
||||
TIPCAddress,
|
||||
is_tipc_available,
|
||||
)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
reg: TIPCAddress = TIPCAddress.get_root()
|
||||
print(f'host B dialling {reg} (by NAME, not address)')
|
||||
|
||||
async with tractor.open_root_actor(
|
||||
name='host_b',
|
||||
enable_transports=['tipc'],
|
||||
registry_addrs=[reg.unwrap()],
|
||||
):
|
||||
async with tractor.find_actor('host_a') as ptl:
|
||||
if ptl is None:
|
||||
raise RuntimeError(
|
||||
'No `host_a` in the cluster name table!\n'
|
||||
' |_is `host_a_srv.py` running?\n'
|
||||
' |_does `tipc link list` show the peer?\n'
|
||||
)
|
||||
|
||||
async with (
|
||||
ptl.open_context(
|
||||
'host_a_srv:echo',
|
||||
) as (ctx, _),
|
||||
ctx.open_stream() as stream,
|
||||
):
|
||||
for msg in ('hello', 'from', 'the other node'):
|
||||
await stream.send(msg)
|
||||
print(f'host-b <- {await stream.receive()!r}')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
if not is_tipc_available():
|
||||
raise RuntimeError(
|
||||
'The `tipc` kernel module is not loaded!\n'
|
||||
' |_try: `sudo modprobe tipc`\n'
|
||||
)
|
||||
trio.run(main)
|
||||
|
|
@ -0,0 +1,104 @@
|
|||
'''
|
||||
`tractor` over `AF_TIPC` on a single host.
|
||||
|
||||
Every actor's IPC address is a TIPC *service name*, and binding
|
||||
one publishes it into the kernel's cluster-wide name table. So
|
||||
`tipc nametable show` lists your live actor tree — no registrar
|
||||
query, no `tractor` API, just the kernel telling you what's up.
|
||||
|
||||
Run,
|
||||
|
||||
sudo modprobe tipc
|
||||
python single_host.py
|
||||
|
||||
'''
|
||||
from __future__ import annotations
|
||||
import subprocess
|
||||
|
||||
import trio
|
||||
import tractor
|
||||
from tractor.ipc._tipc import (
|
||||
TRACTOR_STYPE,
|
||||
is_tipc_available,
|
||||
)
|
||||
|
||||
|
||||
def show_nametable(tag: str) -> None:
|
||||
'''
|
||||
Dump the kernel name-table rows belonging to `tractor`.
|
||||
|
||||
'''
|
||||
print(f'\n--- `tipc nametable show` :: {tag} ---')
|
||||
out = subprocess.run(
|
||||
['tipc', 'nametable', 'show'],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
for line in out.stdout.splitlines():
|
||||
# header, or one of *our* service-type rows
|
||||
if (
|
||||
line.startswith('Type')
|
||||
or
|
||||
line.startswith(str(TRACTOR_STYPE))
|
||||
):
|
||||
print(f' {line}')
|
||||
|
||||
|
||||
@tractor.context
|
||||
async def wait_until_cancelled(
|
||||
ctx: tractor.Context,
|
||||
) -> None:
|
||||
await ctx.started()
|
||||
await trio.sleep_forever()
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
async with tractor.open_nursery(
|
||||
enable_transports=['tipc'],
|
||||
) as an:
|
||||
|
||||
show_nametable('root only')
|
||||
|
||||
portals: list[tractor.Portal] = []
|
||||
for name in ('donny', 'walter', 'dude'):
|
||||
portals.append(
|
||||
await an.start_actor(
|
||||
name,
|
||||
enable_modules=[__name__],
|
||||
)
|
||||
)
|
||||
|
||||
async with trio.open_nursery() as tn:
|
||||
for ptl in portals:
|
||||
tn.start_soon(
|
||||
_hold_open,
|
||||
ptl,
|
||||
)
|
||||
await trio.sleep(0.5)
|
||||
|
||||
# XXX the money shot: 4 actors, 4 published names
|
||||
show_nametable('root + 3 subactors')
|
||||
|
||||
tn.cancel_scope.cancel()
|
||||
|
||||
await an.cancel()
|
||||
|
||||
show_nametable('after teardown (all withdrawn)')
|
||||
|
||||
|
||||
async def _hold_open(
|
||||
ptl: tractor.Portal,
|
||||
) -> None:
|
||||
async with ptl.open_context(
|
||||
wait_until_cancelled,
|
||||
) as (ctx, _):
|
||||
await trio.sleep_forever()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
if not is_tipc_available():
|
||||
raise RuntimeError(
|
||||
'The `tipc` kernel module is not loaded!\n'
|
||||
' |_try: `sudo modprobe tipc`\n'
|
||||
)
|
||||
trio.run(main)
|
||||
|
|
@ -0,0 +1,93 @@
|
|||
'''
|
||||
Watch `tractor` actors (de)register themselves, live, via TIPC's
|
||||
topology service.
|
||||
|
||||
`open_topology_events()` subscribes to the kernel's name table
|
||||
and yields a `trio` receive-channel of `publish`/`withdraw`
|
||||
events. That's **push-based** service discovery: no registrar
|
||||
round-trip, no polling — the kernel tells you the instant any
|
||||
actor anywhere in the cluster comes or goes.
|
||||
|
||||
Run,
|
||||
|
||||
sudo modprobe tipc
|
||||
python watch_nametable.py
|
||||
|
||||
'''
|
||||
from __future__ import annotations
|
||||
|
||||
import trio
|
||||
import tractor
|
||||
from tractor.ipc._tipc import (
|
||||
TIPCNameEvent,
|
||||
is_tipc_available,
|
||||
open_topology_events,
|
||||
)
|
||||
|
||||
|
||||
@tractor.context
|
||||
async def wait_until_cancelled(
|
||||
ctx: tractor.Context,
|
||||
) -> None:
|
||||
await ctx.started()
|
||||
await trio.sleep_forever()
|
||||
|
||||
|
||||
async def print_events(
|
||||
events: trio.MemoryReceiveChannel[TIPCNameEvent],
|
||||
) -> None:
|
||||
glyphs: dict[str, str] = {
|
||||
'published': '[+]',
|
||||
'withdrawn': '[-]',
|
||||
'timeout': '[!]',
|
||||
}
|
||||
async for ev in events:
|
||||
print(
|
||||
f' {glyphs.get(ev.kind, "[?]")} {ev.kind:<10} '
|
||||
f'instance={ev.addr._instance:<12} '
|
||||
f'port=0x{ev.node:08x}:{ev.ref}'
|
||||
)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
# NOTE, subscribe BEFORE booting the runtime so we catch the
|
||||
# root actor's own publication too.
|
||||
async with open_topology_events() as events:
|
||||
async with trio.open_nursery() as tn:
|
||||
tn.start_soon(print_events, events)
|
||||
|
||||
print('watching the TIPC name table..\n')
|
||||
async with tractor.open_nursery(
|
||||
enable_transports=['tipc'],
|
||||
) as an:
|
||||
await trio.sleep(0.3)
|
||||
|
||||
print('\nspawning subactors..')
|
||||
portals: list[tractor.Portal] = []
|
||||
for name in ('donny', 'walter', 'dude'):
|
||||
portals.append(
|
||||
await an.start_actor(
|
||||
name,
|
||||
enable_modules=[__name__],
|
||||
)
|
||||
)
|
||||
await trio.sleep(0.2)
|
||||
|
||||
print('\ntearing down..')
|
||||
for ptl in portals:
|
||||
await ptl.cancel_actor()
|
||||
await trio.sleep(0.2)
|
||||
|
||||
await an.cancel()
|
||||
|
||||
await trio.sleep(0.5)
|
||||
tn.cancel_scope.cancel()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
if not is_tipc_available():
|
||||
raise RuntimeError(
|
||||
'The `tipc` kernel module is not loaded!\n'
|
||||
' |_try: `sudo modprobe tipc`\n'
|
||||
)
|
||||
trio.run(main)
|
||||
Loading…
Reference in New Issue