Traversal & relays

Reachability becomes another measured path.

The traversal stack adopted STUN, replaced TURN with an Atlas-native relay, and replaced ICE with rendezvous over that relay — so a relayed path and a punched direct path are just links in the bond, scheduled like everything else. A node behind translation that calls a node with a public address needs none of it when it joins with a pairing code, or when the public node already holds an address for it, which roaming moves to where the caller’s authenticated packets come from. This page is the mechanics; the managed service around them lives at nexusatlas.net.

Opt-in ([traversal], default off)Hosted relays live · EU + NA
The whole sequence

Watch two NATs open — and one that won’t.

Relayed, then rendezvous, then direct. The beat worth waiting for is the third: the punched path is not a mode and not a new link — it is wwan0, the underlay link that was already there, with its endpoint for this peer discovered at runtime. From that instant the ordinary probe loop measures it and the ordinary scheduler ranks it, beside the relay that carried the tunnel until then. All of it is opt-in: [traversal] enabled defaults to false, and with it off no netcheck task is spawned at all.

Two nodes, two NATs — relayed, then direct, then the pair that stays relayedSimulation
Phase
Mapping behaviour at NAT-B

One mapping serves every destination, so the public address netcheck read off the relay is the address node A can aim at. Both ends sort their candidate lists by the same total order — server-reflexive first, then host, then by address — which is what makes them try the same pair at the same moment. Without that agreement two nodes can trade candidates forever and still never have their packets cross.

relay-eulink carrying the traffic
74 msend-to-end RTT
1 linklinks in the bond
—netcheck: NAT-B mapping
What you’re watching: a simulation of the real sequence — netcheck out of the link’s own socket, candidates carried as a rendezvous payload inside the relay session both ends already hold, then a trial in which the candidate is written into the link’s per-peer endpoint slot. The timings and addresses are invented for legibility; the mechanism, the vocabulary and both outcomes are the daemon’s. The relay is never torn down — after a successful punch it simply loses on score and stays warm.
From the data socket

STUN out of the same 5-tuple

Each underlay link probes the STUN servers out of its own socket — the mapping discovered is the mapping data gets, not the mapping some separate test socket got. Results (public endpoints + RFC 4787 classification per link) land in the stats sidecar, GET /api/netcheck, and atlasd status.

Two IPs, always

Why every relay answers STUN on two addresses

Distinguishing endpoint-independent from address-dependent mapping requires replies from two different server IPs — two ports on one IP cannot do it. Every Nexus Atlas relay carries two public IPv4 addresses and answers STUN on both, so one relay is a complete observation pair. Config validation rejects single-IP STUN lists for exactly this reason.

Ordinary member

[[link]] type = "relay" — probed, scored, scheduled

Peer traffic rides encrypted envelopes addressed to the peer’s static key through a relay server — but the link is otherwise ordinary: its own socket, its own probes, its own RTT in the same scheduler. That is what lets the bond hold a relayed path and a direct path at once and shift between them without any reconnect. Peers need no endpoint on a relay link: identity is the public key; addresses are discovered at runtime.

66 bytes

Envelope overhead, accounted for globally

Adding any relay link shrinks the global fragment payload by the 66-byte envelope so relayed datagrams never exceed the path MTU. And a relay link sharing a modem with a direct link needs its own listen_port — otherwise SO_REUSEPORT silently load-balances the two sockets against each other; validation catches it.

Under a second

Outbound-only bring-up

A node behind carrier-grade NAT becomes reachable through its relay session in under a second, with no inbound rule and no public IP — the relay session both ends already hold doubles as the rendezvous channel for everything that follows.

Rendezvous

Candidates over the relay you already share

Peers exchange server-reflexive and local candidates through the established relay session — no signalling service exists or is needed — then aim at each other simultaneously. The crossing probes open both NATs.

Not a new link

A punched path is a discovered endpoint

The punch writes the peer’s public endpoint into an existing configured link’s slot for that peer. The probe loop then probes it out of that link’s socket, a probe-ack returns, the monitor marks it alive with a real RTT, and it enters the scheduler beside the relay path — which stays in the bond as a warm standby. The relay→direct switchover is an ordinary scheduler decision; a failed punch costs nothing.

Measured

Proven behind two CGNATs

The namespace harness builds two daemons each behind its own stateful NAT (with real latency — zero-delay namespaces let connection tracking steal the port tuple and deadlock any punch): tunnel up via relay, correct NAT classification, direct path punched under endpoint-independent mapping, relay-only under endpoint-dependent mapping (the CGNAT case), traffic never dropping across the transition.

Signed map

Ed25519 discovery with rollback rejection

A relay link that omits its server resolves it from a relay map signed by an offline key whose public half is compiled into the daemon — HTTPS is transport, not trust. Clients verify, cache the last good map on disk, and reject older signed maps as rollbacks: a compromised control plane can’t forge membership, and a directory outage never becomes a tunnel outage. Static per-link config always wins over discovery.

The relay itself

One binary, zero disk state

nexus-atlas-relay: STUN on both public addresses, a session registry keyed by client static keys, opaque-envelope forwarding with per-session crypto, keepalives that hold the NAT binding, and session telemetry to the control plane under salted hashes, never client keys. It authenticates clients; it cannot read the tunneled payload it forwards.

Relay network

Attach near home, reach anyone.

Each node holds a session to a relay close to it — a short first hop, which is where NAT bindings, mobile uplinks and lossy radios do best — and still reaches any peer, wherever that peer is attached. Three mechanisms build on each other: a joint election when the pair shares relays, a directory of sealed records when it does not, and one-hop federation between relays so neither end has to reach across the world. None of them keeps a list of which node is on which relay.

Shipped

Relay auto-election, make-before-break: both ends meet at the best relay they share

Each end STUN-sweeps its relay candidates and advertises to the peer, over the relay control channel, its RTT to every relay that is still answering — a relay that stopped answering leaves the vector, so the peer stops ranking it too. Both then rank the relays they hold in common by the combined round trip — my leg plus your leg, the cost a relayed packet actually pays — with a hash tie-break, and arrive at the same answer independently, with no negotiation. Exactly relay_paths sessions stay active. When a relay stops answering, both sides drop it within two sweeps and elect the same next-best relay; when one is replaced while still working, the outgoing relay keeps carrying for a 20 s grace, make-before-break. With a punched direct path up, killing both relays cost 0% loss over 123 seconds — relays are warm standbys, not the path. Measured

Directory-assisted rendezvous Shipped · opt-in

A pair with no relay in common has nothing to elect. So while a node holds a relay session it posts, for each of its configured peers, a sealed presence record to the control plane — a record only that peer can read. A node that cannot find its peer on any relay it holds looks the record up, opens it, and attaches to the relay the peer is on; the ordinary handshake and the ordinary election take over from there.

1 · A PUBLISHES 2 · B LOOKS UP Node Aattached to Zurich Control planeopaque id → sealed blobno listing · no search Node Battached to New York · A absent THE RECORD id · a hash of both keys, opaque to the server blob · A’s relays, sealed so only B opens it signed by A · checked before it is stored expires in ten minutes · re-posted every five THE LOOKUP B derives the same id from the two keys fetched by that exact id, in the request body B opens the blob and reads “Zurich” a 404 for anyone without the id Relay · ZurichA and B meet here 3 · NEW RELAY LINK the server stores an id it cannot reverse and a blob it cannot open — never a relay name, never a peer
A PUBLISHESa record under an id made from both keys, its relays sealed so only B can open them, signed, ten-minute expiry
B LOOKS UPderives the same id, fetches that exact record, opens it and reads “Zurich”
B ATTACHESadds a relay link to Zurich and meets A there — the server never saw a relay name
The id

Computable only by someone holding both keys

The record id is a SHA-256 over a fixed label, the peer’s public key and the poster’s public key, so only the pair can compute it, and A’s record for B is a different id from B’s record for A. The server stores it as an opaque key and can do nothing with it but match an exact lookup.

The blob

Sealed to the peer, authenticated by the sender

The blob names the relays the poster holds an established session with. It is sealed to the peer with an ephemeral X25519 exchange plus a static-to-static contribution, so only the peer can open it and only the real sender can produce one the peer accepts, then padded to a fixed size so its length gives nothing away. The poster signs id, blob and expiry with a key derived from its node key; the server verifies the signature before it stores anything and rate-limits per poster, so nobody can plant a record for someone else.

Ten minutes

Records live only while the relay session does

A record expires within ten minutes. A node re-posts every five minutes while it holds a relay session, within seconds of its relay set changing, and not at all without one — so a stale record ages out on its own, and a node that moves is findable at its new relay almost at once.

The lookup

Exact id, once a minute, only while the peer is missing

A peer absent from every relay this node holds for 30 seconds is looked up once a minute until it is found. A peer never seen since start-up waits only 5 seconds for its first lookup, and a node posts its own record two seconds after its first relay session, so a cold bring-up through the directory takes seconds; the 30-second hold is for a peer that was present and has gone. Each relay its record names becomes a runtime relay link, its key pinned from the record and cross-checked against the signed relay map; a relay the map does not list is refused. A slot whose relay has carried no peer for ten minutes is released, so a node follows its peer to a third relay without a restart. There is no listing, no search and no query by key — an unknown id is simply a 404.

One-hop relay federation Shipped · opt-in

The directory lets the far end attach to the near end’s relay — which still gives one of them a long first hop. Federation removes it: each node stays on its home relay, and the relays carry the frame between them. The sending node already knows from the peer’s record which relay the peer is on, so it adds a via hint naming that relay; its home relay, which does not hold the destination, forwards the frame to the named relay over an inter-relay session, and that relay delivers it exactly as if it had arrived locally.

EUROPE NORTH AMERICA Node Ahome: Zurich Relay · Zurichdoes not hold B Relay · New Yorkholds B Node Bhome: New York FRAME FOR B via: New York ONE HOP, NEVER TWO authenticated relay-to-relay session relay keys from the signed map DELIVERED FROM A unchanged for B New York does not hold B → the error returns through Zurich, New York named ONE COST SCALE FOR THE ELECTION common relay R: A’s leg to R + B’s leg to R federated pair: A’s leg to Zurich + Zurich ⇢ New York, from the relay map + B’s leg to New York
NODE Astays on Zurich and sends the frame for B with a via hint: New York
ZURICH ⇢ NEW YORKone hop over an authenticated relay-to-relay session — never a second
NODE Breceives an ordinary frame from A, and answers through New York with via: Zurich
The via hint

The sender names the far relay; the relays keep no directory

A frame for a peer that is not on this node’s relay, but whose record names a relay this node does not hold, goes out with the far relay’s key as a via hint. The home relay forwards a frame for a key it does not hold only when the hint names a relay in the signed map. The far relay hands it to the peer as an ordinary frame from the sender, and the reply takes the mirror path with the peer’s own hint. No relay keeps a directory of clients: the far relay is learned from the hint on each frame, by the sender’s home relay alone.

Exactly one hop

A frame from a relay is never forwarded to another relay

Relays talk over Noise IK sessions keyed with the static relay keys from the signed map, opened on first use, kept alive like client sessions and closed when idle — one per relay pair. The inter-relay envelope carries a hop flag, and a frame that arrived from a relay, or from a key the map does not list, is dropped and counted. Loops cannot form by construction.

Errors and quotas

Failures come back; both ends pay

A hint naming a relay that is not in the map, unreachable or not federating is refused to the sender at once. A far relay that does not hold the peer answers with an error that the home relay relays back with the far relay named, so the sender’s presence evidence works exactly as it does on a single relay. Every frame is charged to the sender’s session quota and to a per-pair, per-direction rate limit at both relays, under a global cap on the number of federated pairs.

One scale

A relay-to-relay RTT matrix beside the signed map

Each federating relay measures its round trip to every other relay and reports it, and the relay map carries the resulting matrix alongside its signed part. The signature covers the membership — every relay’s key, addresses and region — while the RTT figures are live, unsigned measurements: they can change how a path is ranked, never which relays a node accepts. The election then ranks a federated pair — A’s leg to its relay, plus the relay-to-relay RTT, plus B’s leg to its relay — against every common relay on the same scale, and both ends, holding the same vectors and the same map, rank identically. relay_paths counts paths of either kind; make-before-break is unchanged; the dashboard draws a federated path as two relays in series.

End to end

Requires the end-to-end payload layer

A federated path puts a second relay on the path, so the daemon takes one only when end-to-end sealing ([crypto] e2e) seals every packet from source to destination — unless an operator explicitly allows plain forwarding (federation_allow_plain). Otherwise no via hint is sent, the dashboard says why, and the pair keeps using the directory to meet at a common relay.

No central list of which node is on which relay.

The question a reviewer asks first is the one this design answers by construction: the control plane holds no list of node-to-relay assignments. Relays report their sessions only as salted hashes, and presence records are sealed to the one peer they are for and fetched only by exact id — so neither the control plane nor the console can say which node is on which relay.

The control plane holds

  • Opaque record ids it cannot reverse
  • Fixed-size blobs it cannot open, with their expiry
  • Per relay, session counts, ages and traffic under salted hashes

It never holds

  • A relay name inside any record
  • A client public key in any relay report
  • A record id beside the caller’s address in a request path
Hashed sessions

Relays report sessions, not clients

A relay’s heartbeat lists its sessions under an 8-byte HMAC of the client key, salted with a secret the relay generates at start and never reports. The console keeps counts, ages and traffic per session for capacity planning, and cannot name a single client from them.

Sealed records

Records reveal nothing to the server

The server checks a record’s signature, keeps the id, the blob, the expiry and a hash of the poster for rate limiting, and nothing else. Even someone who knows both public keys and computes the id learns only that a record exists — the relay names open only with the peer’s private key.

Lookups

The id never travels in a path

Nodes look records up with the id in the body of the request, so request logs never pair a record id with the address that asked for it. There is no endpoint that lists, searches or queries records by key.

Nexus Atlas: Relay Election1:46 · plays from YouTube
Gateways & exits

When the far end is the internet itself.

Relays make two of your nodes reachable. Hosted gateways are the other case: an Atlas daemon you terminate a bonded tunnel on — published in a second signed map (same offline key, its own canonical form and expiry, so neither map can impersonate the other), with region, endpoints, co-located STUN addresses and a coarse load hint.

Nexus Atlas: Private Exit1:37 · plays from YouTube

Hosted relays Live service

Hosted relays run in five regions on four continents — Los Angeles (na-west), New York (na-east), Zurich (eu-central), Hong Kong (ap-east) and Sydney (au-east) — each listed in the signed relay map with its region and co-located STUN addresses. A client reads the gateway map the same way: it ranks the listed gateways by measured RTT, selects one, and brings its bonded links up to it.

Full-tunnel exit

A gateway can carry the whole device: default route through the bond, NAT at the gateway’s side, DNS pinned through the tunnel — the “reach the internet through a bonded exit” mode the Android wizard configures.

Self-publish

Gateways self-register and heartbeat through a registry sidecar reading the daemon’s stats snapshot; publication requires outside-in STUN verification, and a dead or firewalled gateway un-publishes itself. Private estates simply stay out of the public map. The service → nexusatlas.net

The metadata boundary

What a hosted relay can and cannot see.

“Encrypted” is not a substitute for a precise boundary. Here it is.

A relay observes

  • Outer source addresses and ports
  • Authenticated node identities (static public keys)
  • Session timing, liveness and forwarded byte counts
  • On a federated path, the second relay sees the same, plus which relay handed it the frame

A relay cannot observe

  • Inner IP packets or application payloads
  • Video, sensor or command content
  • Anything requiring the tunnel’s keys — it holds none of them
Scope note: this boundary is the hosted traversal model. Multi-hop mesh forwarding through your own fleet is a different mode: per-hop re-encryption by trusted member nodes, or, with end-to-end sealing on, sealed envelopes the forwarding nodes cannot open — all four trust models stated precisely on the security page, because your review will ask.
Request an evaluation

Behind CGNAT is not offline.

Two nodes, two carrier NATs, zero port forwarding: reachable in under a second, direct when the NATs allow it. Bring your worst SIM.