11 KiB

Raw Blame History

`tractor` over `AF_TIPC`, where the address is the service name

TIPC is a linux-kernel cluster IPC protocol whose service names live in a cluster-wide name table maintained by the kernel. It is also described as Cluster Domain Sockets: the Unix-domain socket model extended from one kernel to a cluster. That name is a useful explanation for new users, while the code keeps tipc as its protocol key to match Linux’s AF_TIPC, kernel module and tooling. For tractor that means:

an actor’s IPC address is a service name (stype, instance), not a host/port,
.bind()ing it is service registration,
a peer’s .connect()-by-name is the lookup.

So the discovery machinery tractor.discovery normally implements with a registrar actor comes for free, in-kernel — which is the ask in gh #378.

Why examples/multihost/? tests/test_docs_examples.py walks examples/ recursively and runs everything it collects as a subproc, asserting rc == 0. These need the tipc kernel module (and, for the two-host pair, a live bearer), so they can’t satisfy that; 'multihost' not in p[0] is already in the test’s exclusion list, which is what keeps them out of CI. See “CI” below for the separate matrix-entry plan.

the single best demo

sudo modprobe tipc
python single_host.py

Four actors boot, four service names appear in the kernel’s table, and all four are withdrawn on teardown — observed with tipc(8), entirely outside tractor:

--- `tipc nametable show` :: root + 3 subactors ---
  Type       Lower      Upper      Scope    Port
  1953628160 1616       1616       cluster  3161982128
  1953628160 1219427151 1219427151 cluster  1587358717
  1953628160 2641339936 2641339936 cluster  1864021571
  1953628160 3344505866 3344505866 cluster  3816483388

--- `tipc nametable show` :: after teardown (all withdrawn) ---
  Type       Lower      Upper      Scope    Port

1953628160 is 0x74720000 — tractor’s reserved service type, ascii tr in the high half. 1616 is the host-singleton registrar, the same idiom as the TCP port and the registry@1616.sock UDS filename. The other three instances are per-actor digests (see “silent crosstalk” below).

push-based discovery

python watch_nametable.py

Subscribes to the kernel’s topology service and prints name table transitions as they happen — no polling, no registrar round-trip:

watching the TIPC name table..
  [+] published  instance=1616         port=0x00000000:2375440573
spawning subactors..
  [+] published  instance=186947472    port=0x00000000:3960753074
  [+] published  instance=2191362136   port=0x00000000:2263898853
  [+] published  instance=3484369663   port=0x00000000:2126817956
tearing down..
  [-] withdrawn  instance=186947472    port=0x00000000:3960753074
  ...

This is the groundwork for a push registry in tractor.discovery._registry (gh #184, #216) — a registrar that never polls find_actor().

two hosts

Everything above is single-node (modprobe is enough). To span hosts you need a bearer on both, which is the one thing that can’t be CI’d.

For the first physical test, use two wired Linux hosts on the same L2 segment. Prefer a direct cable or uncomplicated switch; avoid Wi-Fi, guest VLANs and port isolation until the basic link works. Use the same checkout and Python environment on both hosts:

# on BOTH hosts
git rev-parse HEAD             # must match on A and B
uv sync --all-extras --dev
sudo modprobe tipc

# choose the real wired iface; do not assume `eth0`
ip -br link
IFACE=enp3s0

# inspect existing cluster identity before changing anything
tipc node get address          # must differ between hosts
tipc node get netid            # must match between hosts

# use one private test netid on BOTH hosts, before enabling bearers
sudo tipc node set netid 37801

# ethernet is simplest when the hosts share an L2 segment
sudo tipc bearer enable media eth device "$IFACE"

# ..or over UDP when L2 isn't available — and MANDATORY over a
# `wg` mesh, see below
sudo tipc bearer enable media udp name uc localip 10.0.11.1

# verify BEFORE running anything: this must list the peer
tipc bearer list
tipc link list
tipc node list

If the link does not appear, first verify carrier, a common TIPC network ID, distinct node addresses, a common VLAN and compatible MTUs. Ethernet TIPC uses EtherType traffic rather than IP routing, so a successful ping alone does not prove the bearer can work.

Run each command from examples/multihost/tipc_cluster/:

# host A
uv run python host_a_srv.py

# host B
watch -n 0.5 tipc nametable show  # optional second terminal
uv run python host_b_client.py

Note what’s absent from both scripts: any IP, hostname or port. Both sides name the same service, and the kernel routes it. Move host_a_srv.py to a third node and host B’s dial keeps working, unchanged.

For a first resilience pass, use a local console or separate management link so the test does not cut off your own SSH session:

# host B: record the healthy baseline
tipc link list
tipc link statistics show

# either host: withdraw and recreate the bearer
sudo tipc bearer disable media eth device "$IFACE"
tipc link list
sudo tipc bearer enable media eth device "$IFACE"
tipc link list

# prove name withdrawal/republication and RPC recovery
tipc nametable show
uv run python host_b_client.py

Capture uname -a, both node addresses, tipc bearer list, tipc link list, tipc link statistics show, the name table and both Python transcripts. Those artifacts distinguish an actor bug from bearer discovery, cluster identity or switch configuration.

Clean up a disposable Ethernet test on both hosts with:

sudo tipc bearer disable media eth device "$IFACE"
tipc link list

Restore any pre-existing network ID only after all bearers are disabled. The first useful automation target is a two-node network namespace fixture that asserts link-up, remote publication, RPC, withdrawal and republication in that order; physical hardware then remains the validation layer for real NIC and switch behaviour.

The commands above use iproute2’s tipc frontend, which speaks the kernel’s TIPCv2 generic-netlink family. The planned pyroute2 dependency already manages WireGuard, interfaces and namespaces and provides generic-netlink primitives, but it does not currently ship a TIPC message codec. Adding one upstream would let tractor replace these manual commands with one Python netlink stack instead of shelling out; until then, tipc(8) remains the canonical frontend.

over a `wg` mesh

TIPC over WireGuard is the intended reference multihost deployment (gh #502). One hard constraint: a wg interface is L3/tun — POINTOPOINT,NOARP, link/none, no L2 address — so TIPC’s eth media cannot bind it. The udp bearer is mandatory there, bound to the wg overlay IP, and wg’s typical 1420 MTU sits under ethernet’s 1500 so link MTU wants checking.

Composed, the deployment address is:

/ip4/<pub>/udp/51820/wg/u<key>/tipc/<stype>/<inst>/<scope>
\____ wg bearer ________/\_key_/\______ tractor ep ________/

Note the tipc segment has no locative part, unlike tcp’s inner /ip4/../tcp/.. — a service name is location-independent, so wg carries the routing and tipc carries identity.

wg here is not about confidentiality. TIPC ships AES-GCM crypto of its own (tipc node set key, linux 5.9+) with cluster/master/per-node keys and rekeying, so “wg adds the encryption TIPC lacks” is wrong.

The motivation is different but real: TIPC’s keys are symmetric and pre-shared — distribution, rotation and revocation are all on you — whereas wg gives public-key identity and a handshake, NAT traversal, and one overlay shared by every transport instead of a TIPC-only mechanism. Worth benchmarking either way; native crypto skips the tunnel hop.

Note too that the ethernet-bearer pairing #378 imagined does not apply over wg: on a given link the L2 path and the wg path are mutually exclusive.

scope

TIPCAddress._scope is the backend’s .bindspace — literally “the set of hosts this published name is reachable from”:

scope	meaning
`TIPC_NODE_SCOPE`	same host only — the UDS analogue
`TIPC_CLUSTER_SCOPE`	cluster-visible (the default)

TIPC_ZONE_SCOPE is deprecated and aliased to cluster by modern kernels; tractor accepts it on input and folds it, logging at transport level.

gotchas worth knowing before you deploy

Silent crosstalk. Unlike every other backend, a duplicate bind does not raise EADDRINUSE — TIPC happily accepts multiple publishers of one name and round-robins connects between them (verified: 6 dials alternated b,a,b,a,b,a). So an instance collision is silent traffic-splitting, not an error. That’s why TIPCAddress.get_random() derives the instance from a blake2b digest of the actor identity rather than a counter. Two tractor trees sharing both a cluster and an _stype share a name space; partition them by passing a distinct _stype.

Graceful close looks like a reset. A peer closing cleanly surfaces as BrokenResourceError/ECONNRESET rather than the clean 0-byte EOF you get from TCP/UDS. It’s benign — the transport layer already classifies it as a normal disconnect — but it does look alarming in transport-level logs.

Dialing an unpublished name answers EHOSTUNREACH instantly (no SYN-timeout wait), which is much better discovery-ping behaviour than TCP. tractor normalizes it to ConnectionError.

It’s opt-in, never a default. The module isn’t loaded on most boxes and doesn’t exist off-linux, so enable_transports=['tipc'] is always explicit. Check tractor.ipc._tipc.is_tipc_available() before assuming.

maddr form

There is no registered /tipc protocol in the multiaddr table yet (upstream track: gh #483 + multiformats/py-multiaddr#107), so the grammar is interim and str-only:

/tipc/<stype>/<instance>/<scope>

parse_maddr() special-cases this prefix before handing anything to Multiaddr(), which would otherwise reject the unregistered name outright. Registering it upstream is what would unblock gh #443’s “return Multiaddr everywhere” item.

running the suite over TIPC

The whole test suite runs under the backend:

sudo modprobe tipc
pytest --tpt-proto tipc

Without the module that fails loudly and immediately with an actionable message rather than a few hundred connect timeouts. Backend-specific unit tests live in tests/ipc/test_tipc.py and self-skip when the module is absent.

CI

Single-host TIPC is CI-able — the module ships with the standard Ubuntu kernel package. CI loads it with sudo modprobe tipc and runs the suite with --tpt-proto tipc as a blocking matrix leg. Cross-node bearer testing stays manual — this README is that smoke test.

normative refs

The tipc.io docs are stale in places (gh #378 says as much). Treat the kernel sources as the only normative reference:

include/uapi/linux/tipc.h — address flavours, sockopts, the topology structs
net/tipc/socket.c, net/tipc/topsrv.c
man 8 tipc

11 KiB Raw Blame History Unescape Escape

tractor over AF_TIPC, where the address is the service name