diff --git a/examples/multihost/tipc_cluster/README.md b/examples/multihost/tipc_cluster/README.md new file mode 100644 index 00000000..e1b888f7 --- /dev/null +++ b/examples/multihost/tipc_cluster/README.md @@ -0,0 +1,213 @@ +# `tractor` over `AF_TIPC`, where the address *is* the service name + +TIPC is a linux-kernel cluster IPC protocol whose service names +live in a **cluster-wide name table maintained by the kernel**. +For `tractor` that means: + +- an actor's IPC address is a service name `(stype, instance)`, + not a host/port, +- `.bind()`ing it **is** service registration, +- a peer's `.connect()`-by-name **is** the lookup. + +So the discovery machinery `tractor.discovery` normally +implements with a registrar actor comes for free, in-kernel — +which is the ask in gh +[#378](https://github.com/goodboy/tractor/issues/378). + +> **Why `examples/multihost/`?** `tests/test_docs_examples.py` +> walks `examples/` recursively and runs everything it collects +> as a subproc, asserting `rc == 0`. These need the `tipc` +> kernel module (and, for the two-host pair, a live bearer), so +> they can't satisfy that; `'multihost' not in p[0]` is already +> in the test's exclusion list, which is what keeps them out of +> CI. See "CI" below for the separate matrix-entry plan. + +## the single best demo + +```bash +sudo modprobe tipc +python single_host.py +``` + +Four actors boot, four service names appear in the kernel's +table, and all four are withdrawn on teardown — observed with +`tipc(8)`, entirely outside `tractor`: + +``` +--- `tipc nametable show` :: root + 3 subactors --- + Type Lower Upper Scope Port + 1953628160 1616 1616 cluster 3161982128 + 1953628160 1219427151 1219427151 cluster 1587358717 + 1953628160 2641339936 2641339936 cluster 1864021571 + 1953628160 3344505866 3344505866 cluster 3816483388 + +--- `tipc nametable show` :: after teardown (all withdrawn) --- + Type Lower Upper Scope Port +``` + +`1953628160` is `0x74720000` — `tractor`'s reserved service +type, ascii `tr` in the high half. `1616` is the host-singleton +registrar, the same idiom as the TCP port and the +`registry@1616.sock` UDS filename. The other three instances are +per-actor digests (see "silent crosstalk" below). + +## push-based discovery + +```bash +python watch_nametable.py +``` + +Subscribes to the kernel's *topology service* and prints name +table transitions as they happen — no polling, no registrar +round-trip: + +``` +watching the TIPC name table.. + [+] published instance=1616 port=0x00000000:2375440573 +spawning subactors.. + [+] published instance=186947472 port=0x00000000:3960753074 + [+] published instance=2191362136 port=0x00000000:2263898853 + [+] published instance=3484369663 port=0x00000000:2126817956 +tearing down.. + [-] withdrawn instance=186947472 port=0x00000000:3960753074 + ... +``` + +This is the groundwork for a push registry in +`tractor.discovery._registry` (gh +[#184](https://github.com/goodboy/tractor/issues/184), +[#216](https://github.com/goodboy/tractor/issues/216)) — a +registrar that *never polls* `find_actor()`. + +## two hosts + +Everything above is single-node (`modprobe` is enough). To span +hosts you need a **bearer** on both, which is the one thing that +can't be CI'd. + +```bash +# on BOTH hosts +sudo modprobe tipc + +# over ethernet (L2) — simplest when the hosts share a segment +sudo tipc bearer enable media eth device eth0 + +# ..or over UDP when L2 isn't available (pairs nicely with the +# `wg` tunnel examples in ../wg_lan/) +sudo tipc bearer enable media udp name uc localip 10.0.11.1 + +# verify BEFORE running anything: this must list the peer +tipc link list +tipc node list +``` + +Then: + +```bash +# host A +python host_a_srv.py + +# host B +python host_b_client.py +``` + +Note what's absent from both scripts: any IP, hostname or port. +Both sides name the *same service*, and the kernel routes it. +Move `host_a_srv.py` to a third node and host B's dial keeps +working, unchanged. + +### scope + +`TIPCAddress._scope` is the backend's `.bindspace` — literally +"the set of hosts this published name is reachable from": + +| scope | meaning | +| --- | --- | +| `TIPC_NODE_SCOPE` | same host only — the UDS analogue | +| `TIPC_CLUSTER_SCOPE` | cluster-visible (the default) | + +`TIPC_ZONE_SCOPE` is deprecated and aliased to cluster by modern +kernels; `tractor` accepts it on input and folds it, logging at +`transport` level. + +## gotchas worth knowing before you deploy + +**Silent crosstalk.** Unlike every other backend, a duplicate +bind does **not** raise `EADDRINUSE` — TIPC happily accepts +multiple publishers of one name and *round-robins* connects +between them (verified: 6 dials alternated `b,a,b,a,b,a`). So an +instance collision is silent traffic-splitting, not an error. +That's why `TIPCAddress.get_random()` derives the instance from +a `blake2b` digest of the actor identity rather than a counter. +Two `tractor` trees sharing both a cluster **and** an `_stype` +share a name space; partition them by passing a distinct +`_stype`. + +**Graceful close looks like a reset.** A peer closing cleanly +surfaces as `BrokenResourceError`/`ECONNRESET` rather than the +clean 0-byte EOF you get from TCP/UDS. It's benign — the +transport layer already classifies it as a normal disconnect — +but it does look alarming in `transport`-level logs. + +**Dialing an unpublished name** answers `EHOSTUNREACH` +*instantly* (no SYN-timeout wait), which is much better +discovery-ping behaviour than TCP. `tractor` normalizes it to +`ConnectionError`. + +**It's opt-in, never a default.** The module isn't loaded on +most boxes and doesn't exist off-linux, so +`enable_transports=['tipc']` is always explicit. Check +`tractor.ipc._tipc.is_tipc_available()` before assuming. + +## maddr form + +There is no registered `/tipc` protocol in the multiaddr table +yet (upstream track: gh +[#483](https://github.com/goodboy/tractor/issues/483) + +multiformats/py-multiaddr#107), so the grammar is interim and +`str`-only: + +``` +/tipc/// +``` + +`parse_maddr()` special-cases this prefix *before* handing +anything to `Multiaddr()`, which would otherwise reject the +unregistered name outright. Registering it upstream is what +would unblock gh +[#443](https://github.com/goodboy/tractor/issues/443)'s +"return `Multiaddr` everywhere" item. + +## running the suite over TIPC + +The whole test suite runs under the backend: + +```bash +sudo modprobe tipc +pytest --tpt-proto tipc +``` + +Without the module that fails loudly and immediately with an +actionable message rather than a few hundred connect timeouts. +Backend-specific unit tests live in `tests/ipc/test_tipc.py` and +self-skip when the module is absent. + +## CI + +Single-host TIPC *is* CI-able — the module ships with the +standard Ubuntu kernel package, so a `sudo modprobe tipc` step +plus a `--tpt-proto tipc` matrix entry should work. That's not +wired up yet; verify in a throwaway workflow first, and fall +back to a container job with `--cap-add NET_ADMIN` if the +runners refuse. Cross-node (bearer) testing stays manual — this +README is that smoke test. + +## normative refs + +The tipc.io docs are stale in places (gh #378 says as much). +Treat the kernel sources as the only normative reference: + +- `include/uapi/linux/tipc.h` — address flavours, sockopts, the + topology `struct`s +- `net/tipc/socket.c`, `net/tipc/topsrv.c` +- `man 8 tipc` diff --git a/examples/multihost/tipc_cluster/host_a_srv.py b/examples/multihost/tipc_cluster/host_a_srv.py new file mode 100644 index 00000000..7f586166 --- /dev/null +++ b/examples/multihost/tipc_cluster/host_a_srv.py @@ -0,0 +1,69 @@ +''' +HOST A — publish a `tractor` service on a cluster-scoped TIPC +service name. + +Note what is NOT in this file: any IP address, hostname or port. +The actor's address IS the service name `(stype, instance)`, and +the kernel routes it over whatever bearer you enabled. Move this +process to another node and host B's dial keeps working, +unchanged. + +Prereqs on BOTH hosts (see README.md), + + sudo modprobe tipc + sudo tipc bearer enable media eth device + tipc link list # must show a link to the peer + +Then here, + + python host_a_srv.py + +''' +from __future__ import annotations + +import trio +import tractor +from tractor.ipc._tipc import ( + TIPCAddress, + is_tipc_available, +) + + +@tractor.context +async def echo( + ctx: tractor.Context, +) -> None: + await ctx.started() + async with ctx.open_stream() as stream: + async for msg in stream: + print(f'host-a <- {msg!r}') + await stream.send(f'{msg} (from host A)') + + +async def main() -> None: + # the host-singleton registrar name, `instance=1616` — + # the same "1616 is tractor's registrar" idiom as the tcp + # port and the `registry@1616.sock` UDS filename. + reg: TIPCAddress = TIPCAddress.get_root() + print(f'host A publishing {reg}') + + async with tractor.open_root_actor( + name='host_a', + enable_transports=['tipc'], + registry_addrs=[reg.unwrap()], + enable_modules=[__name__], + ): + print( + 'registrar up — `tipc nametable show` on EITHER host\n' + 'should now list this service. ctrl-c to stop.' + ) + await trio.sleep_forever() + + +if __name__ == '__main__': + if not is_tipc_available(): + raise RuntimeError( + 'The `tipc` kernel module is not loaded!\n' + ' |_try: `sudo modprobe tipc`\n' + ) + trio.run(main) diff --git a/examples/multihost/tipc_cluster/host_b_client.py b/examples/multihost/tipc_cluster/host_b_client.py new file mode 100644 index 00000000..966445a3 --- /dev/null +++ b/examples/multihost/tipc_cluster/host_b_client.py @@ -0,0 +1,59 @@ +''' +HOST B — dial host A's service *by name*, across the cluster. + +The `.connect()` on a TIPC service name IS the discovery lookup: +the kernel resolves the published name to whichever node serves +it. So this client needs no IP, no port and no idea where host A +actually is. + +Prereqs: same bearer setup as `host_a_srv.py`, and that script +already running on the other node. + + python host_b_client.py + +''' +from __future__ import annotations + +import trio +import tractor +from tractor.ipc._tipc import ( + TIPCAddress, + is_tipc_available, +) + + +async def main() -> None: + reg: TIPCAddress = TIPCAddress.get_root() + print(f'host B dialling {reg} (by NAME, not address)') + + async with tractor.open_root_actor( + name='host_b', + enable_transports=['tipc'], + registry_addrs=[reg.unwrap()], + ): + async with tractor.find_actor('host_a') as ptl: + if ptl is None: + raise RuntimeError( + 'No `host_a` in the cluster name table!\n' + ' |_is `host_a_srv.py` running?\n' + ' |_does `tipc link list` show the peer?\n' + ) + + async with ( + ptl.open_context( + 'host_a_srv:echo', + ) as (ctx, _), + ctx.open_stream() as stream, + ): + for msg in ('hello', 'from', 'the other node'): + await stream.send(msg) + print(f'host-b <- {await stream.receive()!r}') + + +if __name__ == '__main__': + if not is_tipc_available(): + raise RuntimeError( + 'The `tipc` kernel module is not loaded!\n' + ' |_try: `sudo modprobe tipc`\n' + ) + trio.run(main) diff --git a/examples/multihost/tipc_cluster/single_host.py b/examples/multihost/tipc_cluster/single_host.py new file mode 100644 index 00000000..a4ee92b9 --- /dev/null +++ b/examples/multihost/tipc_cluster/single_host.py @@ -0,0 +1,104 @@ +''' +`tractor` over `AF_TIPC` on a single host. + +Every actor's IPC address is a TIPC *service name*, and binding +one publishes it into the kernel's cluster-wide name table. So +`tipc nametable show` lists your live actor tree — no registrar +query, no `tractor` API, just the kernel telling you what's up. + +Run, + + sudo modprobe tipc + python single_host.py + +''' +from __future__ import annotations +import subprocess + +import trio +import tractor +from tractor.ipc._tipc import ( + TRACTOR_STYPE, + is_tipc_available, +) + + +def show_nametable(tag: str) -> None: + ''' + Dump the kernel name-table rows belonging to `tractor`. + + ''' + print(f'\n--- `tipc nametable show` :: {tag} ---') + out = subprocess.run( + ['tipc', 'nametable', 'show'], + capture_output=True, + text=True, + ) + for line in out.stdout.splitlines(): + # header, or one of *our* service-type rows + if ( + line.startswith('Type') + or + line.startswith(str(TRACTOR_STYPE)) + ): + print(f' {line}') + + +@tractor.context +async def wait_until_cancelled( + ctx: tractor.Context, +) -> None: + await ctx.started() + await trio.sleep_forever() + + +async def main() -> None: + async with tractor.open_nursery( + enable_transports=['tipc'], + ) as an: + + show_nametable('root only') + + portals: list[tractor.Portal] = [] + for name in ('donny', 'walter', 'dude'): + portals.append( + await an.start_actor( + name, + enable_modules=[__name__], + ) + ) + + async with trio.open_nursery() as tn: + for ptl in portals: + tn.start_soon( + _hold_open, + ptl, + ) + await trio.sleep(0.5) + + # XXX the money shot: 4 actors, 4 published names + show_nametable('root + 3 subactors') + + tn.cancel_scope.cancel() + + await an.cancel() + + show_nametable('after teardown (all withdrawn)') + + +async def _hold_open( + ptl: tractor.Portal, +) -> None: + async with ptl.open_context( + wait_until_cancelled, + ) as (ctx, _): + await trio.sleep_forever() + + +if __name__ == '__main__': + if not is_tipc_available(): + raise RuntimeError( + 'The `tipc` kernel module is not loaded!\n' + ' |_try: `sudo modprobe tipc`\n' + ) + trio.run(main) diff --git a/examples/multihost/tipc_cluster/watch_nametable.py b/examples/multihost/tipc_cluster/watch_nametable.py new file mode 100644 index 00000000..5d18d81c --- /dev/null +++ b/examples/multihost/tipc_cluster/watch_nametable.py @@ -0,0 +1,93 @@ +''' +Watch `tractor` actors (de)register themselves, live, via TIPC's +topology service. + +`open_topology_events()` subscribes to the kernel's name table +and yields a `trio` receive-channel of `publish`/`withdraw` +events. That's **push-based** service discovery: no registrar +round-trip, no polling — the kernel tells you the instant any +actor anywhere in the cluster comes or goes. + +Run, + + sudo modprobe tipc + python watch_nametable.py + +''' +from __future__ import annotations + +import trio +import tractor +from tractor.ipc._tipc import ( + TIPCNameEvent, + is_tipc_available, + open_topology_events, +) + + +@tractor.context +async def wait_until_cancelled( + ctx: tractor.Context, +) -> None: + await ctx.started() + await trio.sleep_forever() + + +async def print_events( + events: trio.MemoryReceiveChannel[TIPCNameEvent], +) -> None: + glyphs: dict[str, str] = { + 'published': '[+]', + 'withdrawn': '[-]', + 'timeout': '[!]', + } + async for ev in events: + print( + f' {glyphs.get(ev.kind, "[?]")} {ev.kind:<10} ' + f'instance={ev.addr._instance:<12} ' + f'port=0x{ev.node:08x}:{ev.ref}' + ) + + +async def main() -> None: + # NOTE, subscribe BEFORE booting the runtime so we catch the + # root actor's own publication too. + async with open_topology_events() as events: + async with trio.open_nursery() as tn: + tn.start_soon(print_events, events) + + print('watching the TIPC name table..\n') + async with tractor.open_nursery( + enable_transports=['tipc'], + ) as an: + await trio.sleep(0.3) + + print('\nspawning subactors..') + portals: list[tractor.Portal] = [] + for name in ('donny', 'walter', 'dude'): + portals.append( + await an.start_actor( + name, + enable_modules=[__name__], + ) + ) + await trio.sleep(0.2) + + print('\ntearing down..') + for ptl in portals: + await ptl.cancel_actor() + await trio.sleep(0.2) + + await an.cancel() + + await trio.sleep(0.5) + tn.cancel_scope.cancel() + + +if __name__ == '__main__': + if not is_tipc_available(): + raise RuntimeError( + 'The `tipc` kernel module is not loaded!\n' + ' |_try: `sudo modprobe tipc`\n' + ) + trio.run(main)