What running Atlas costs.
Atlas is measured against the bare link and kernel WireGuard on the same links, on eight tiers of hardware and a phone: a Raspberry Pi Zero 2 W, a Radxa Zero 3W companion computer over USB 3 gigabit Ethernet, a Pixel 10 Pro XL phone over its own Wi-Fi 6 radio with what a gigabyte costs its battery, a GL.iNet Beryl AX Wi-Fi 6 travel router, a small edge computer over 2.5 GbE, laptops, a desktop core, one-vCPU virtual machines and a two-socket AMD EPYC server over 100 GbE, on bare metal and as a Proxmox guest; and on five cloud machine types, from a shared-core e2-medium to a c4-standard-4, with a cross-region pair. Every figure is the shipped build, 2026-11, and where a tier ran the data path with its offloads switched off, that figure is beside it. Where Atlas costs more than WireGuard, or carries less than its links, this page says so before you find it.
One link, eight hardware tiers, a phone and five cloud machine types.
Where the shaped link sets the rate, Atlas carries the same goodput on every x86 tier, on every cloud machine type and on a Beryl AX router; a Pi Zero 2 W and a Radxa Zero 3W as deployed carry it too, and so does a phone over its own radio, 90.7 Mbit/s down and 91.1 up. What changes from tier to tier is what that rate costs the processor, and where the data path’s offloads were switched off in the same pass, what they save.
| Tier | Atlas TCP Mbit/s | Bare link | WireGuard | atlasd CPU-s/GB | Offloads off, CPU-s/GB | Added latency ms |
|---|---|---|---|---|---|---|
| SBC, Pi Zero 2 W, Cortex-A53, one core | 80.5 | 94.4 | 87.7 | 138.4 | 179.6 ⁵ | +0.61 ¹ |
| SBC, Pi Zero 2 W, Cortex-A53, unpinned (as deployed) | 90.3 | 92.8 | – | 191.2 | – | – |
| SBC, Radxa Zero 3W, Cortex-A55, one core ⁶ | 89.1 | 95.5 | 91.1 | 64.5 | – | +0.55 ¹ |
| SBC, Radxa Zero 3W, Cortex-A55, unpinned (as deployed) ⁶ | 90.5 | 95.4 | 91.2 | 132.3 | – | +0.55 ¹ |
| SBC, Raspberry Pi CM4, Cortex-A72 | Pending | Pending | Pending | Pending | Pending | Pending |
| Phone, Pixel 10 Pro XL, Tensor G5, not pinned (as deployed) | 90.7 / 91.1 ⁹ | – | – | 31.7 / 55.4 ⁹ | – | – |
| Router, Beryl AX, Cortex-A53, one core ⁷ | 90.8 | 95.6 | 91.2 | 45.9 | – | +0.007 ¹ |
| Router, Beryl AX, Cortex-A53 ×2, not pinned (as deployed) ⁷ | 90.8 | 95.6 | 91.3 | 44.8 | – | +0.27 ¹ |
| Jetson, Orin Nano, Cortex-A78AE | Pending | Pending | Pending | Pending | Pending | Pending |
| Mini PC, NUC 14 Pro, Core 3 100U P-core ² | 90.8 | 95.6 | 91.3 | 6.87 | – | none |
| Mini PC, NUC 14 Pro, Core 3 100U E-core ² | 90.8 | 95.6 | 91.3 | 19.96 | – | as WireGuard ³ |
| Mini PC, NUC 14 Pro, Core 3 100U, unpinned (as deployed) ² | 90.8 | 95.6 | 91.3 | 11.71 | – | as WireGuard ³ |
| Laptop, Ryzen 5 7535U core | 90.8 | 95.6 | 91.5 | 27.2 | 30.3 ⁵ | +0.006 ¹ |
| Desktop, Ryzen 9 9950X3D core, network namespaces | 90.8 | 95.6 | 91.6 | 6.12 | 10.97 | +0.015 |
| Desktop VM, Ryzen 9 9950X3D, one vCPU, dgram links | 90.7 | 95.5 | 91.5 | 17.7 | – | none ¹ |
| Desktop VM, Ryzen 9 9950X3D, one vCPU, tap + vhost-net | 90.7 | 95.6 | 91.5 | 14.3 | – | none ¹ |
| Server, EPYC 7763 core, NICs’ socket | 90.8 | 95.7 | 91.6 | 18.3 | – | none |
| Server VM, Proxmox guest on the EPYC 7763, one vCPU ⁴ | 90.8 | 95.6 | 91.5 | 22.7 | – | none |
| Cloud VM, e2-medium, Broadwell vCPU (shared core) ⁸ | 90.6 | 95.3 | – | 19.6 / 19.5 | 32.1 / 45.3 | +0.28 / +0.29 ¹ |
| Cloud VM, t2a-standard-2, Neoverse-N1 vCPU ⁸ | 90.5 | 95.5 | – | 11.7 / 13.5 | 26.8 / 29.2 | none / none ¹ |
| Cloud VM, n2-standard-2, Cascade Lake vCPU ⁸ | 90.5 / 90.6 | 95.5 | – | 13.9 / 15.6 | 22.5 / 32.5 | none / +0.28 ¹ |
| Cloud VM, c3-standard-4, Sapphire Rapids vCPU ⁸ | 90.5 | 95.5 | – | 6.5 / 7.4 | 13.1 / 17.1 | none / none ¹ |
| Cloud VM, c4-standard-4, Emerald Rapids vCPU ⁸ | 90.5 | 95.5 | – | 5.7 / 6.7 | 9.1 / 12.5 | none / none ¹ |
One TCP flow over one link shaped to 100 Mbit/s with 5 ms of delay, median of three 20-second runs. Rows are ordered by the platform’s typical power draw, lowest first, in this and every table on this page that lines up the tiers; virtual machines follow the host they run on. Cloud VMs follow with no power basis (cloud, billed per vCPU), by on-demand hourly price, lowest first (0.037–0.23 USD/h). Power basis: Raspberry Pi Zero 2 W about 3 W and Radxa Zero 3W about 4 W (the whole board under full load); Raspberry Pi CM4 about 5 W; the phone, an endpoint device over its own radio, 2–4 W for the whole phone under traffic, screen off, after the single-board and companion computers, whose class it does not share, and before the router; GL.iNet Beryl AX about 5 W, the whole board; NVIDIA Jetson Orin Nano 7–15 W (power modes); Intel Core 3 100U (NUC 14 Pro) 15 W processor base power; AMD Ryzen 5 7535U (laptops) 28 W default TDP; AMD Ryzen 9 9950X3D (desktop) 170 W TDP; AMD EPYC 7763 (server) 280 W TDP per socket. Labels start with the device type; SBC = single-board computer. Pending rows are tiers scheduled for a later edition, not yet measured (this table and the one-core ceiling table only). atlasd CPU-seconds per GB count both ends. Latency is the unloaded 64 B probe: one way over the bare link on the desktop, half the round trip where marked ¹ (kernel WireGuard: +0.277 on the laptops, +0.28 on the Pi and on the Radxa, +0.003 on one Beryl AX core), the round trip p50 against the bare link’s elsewhere. Radxa Zero 3W: four Cortex-A55 cores at 1.8 GHz; Beryl AX: a GL.iNet Beryl AX (GL-MT3000) Wi-Fi 6 travel router, two Cortex-A53 cores at 1.3 GHz; NUC 14 Pro: an ASUS NUC 14 Pro; laptop: a Ryzen 5 7535U core; desktop: a Ryzen 9 9950X3D core per node; server: an AMD EPYC 7763 core over 100 GbE. ² The receiving NUC only; traffic flows from the desktop to the NUC. ³ On this rig the probe reads 11.12 ms for both tunnels against 10.56 bare on the E-core, 11.11 against 10.58 unpinned. ⁴ atlasd inside the guest; on the host core Atlas costs 49 CPU-s per GB against kernel WireGuard’s 100. ⁵ The same cell with the tunnel offload and the link’s segmentation offload switched off, run back to back with an as-shipped twin, which read 26.8 on the laptops and 139.4 on the Pi; on the desktop the twin read 6.12. ⁶ The receiving Radxa only; traffic flows from the desktop to the board over its USB 3 gigabit Ethernet adapter. ⁷ The receiving Beryl AX only; traffic flows from the desktop to the router over its 2.5 GbE port, linked at gigabit, and the router keeps its own firewall, with connection tracking on every packet. ⁸ Cloud VMs: one pinned vCPU / not pinned (as deployed) where a cell gives two figures; the receiving VM only; the VM’s own NIC shaped on both VMs; Atlas as shipped, its tunnel MTU 1376 sized from the provider’s 1460 B path (see the cloud section); the offloads-off figures measured with the tunnel MTU set by hand (1388), both cells of that pair at the same MTU; no WireGuard cell shaped; latency half the round trip against bare. ⁹ Phone: down / up (the far end to the phone / the phone to the far end); the far end shapes the tunnel’s datagrams, so there is no bare or WireGuard twin; the CPU figure is the app’s engine, its process minus the traffic generator’s threads, not atlasd; sealed end to end it carries 86.3 / 86.6 Mbit/s, exactly the smaller tunnel MTU.
And a phone, over its own radio
A different kind of tier: not a core the harness can pin, but an endpoint device. A Pixel 10 Pro XL (Google Tensor G5) runs the Android app as the measured endpoint over its own Wi-Fi 6 radio, screen off and on battery. Its radio and receive path set the rate, so here it is read as a share of its bare radio link in the same pass; the table above has only its shaped-link row, at the link’s rate like every tier. The phone section has the detail.
| Phone, Pixel 10 Pro XL, Tensor G5 | Mbit/s down / up | Share of the bare link | Engine CPU-s/GB | Whole phone, W | J per GB |
|---|---|---|---|---|---|
| Bare Wi-Fi link, one TCP flow | 453 / 363 | 1 | – | 2.05 / 2.61 | 36 / 56 |
| Atlas, one TCP flow | 309 / 319 | 0.68 / 0.88 | 16.9 / 27.1 | 2.33 / 3.58 | 61 / 92 |
| Atlas, end-to-end sealing, one TCP flow | 277 / 325 | 0.61 / 0.89 | 21.4 / 35.0 | 2.53 / 4.27 | 73 / 105 |
| Atlas, one link shaped to 100 Mbit/s + 5 ms ¹ | 90.7 / 91.1 | – ¹ | 31.7 / 55.4 | 1.53 / 2.48 | 134 / 217 |
| Atlas, end-to-end sealing, shaped ¹ | 86.3 / 86.6 | – ¹ | 40.0 / 66.6 | 1.76 / 2.67 | 162 / 246 |
Down = far end to the phone, up = the phone to the far end; one TCP flow, median of 3. The engine figure is the app’s process minus its traffic generator, not atlasd’s; power is the whole phone’s, from its battery’s fuel gauge, screen off. ¹ The far end shapes the tunnel’s datagrams, so bare traffic, which does not cross that shaper, has no shaped cell: the shaped rows carry the link’s rate as every tier above does, and sealing’s 86.3 / 86.6 is exactly its smaller tunnel MTU.
Fewer crossings per packet; the cipher is half of what remains.
Atlas reads TCP from the tunnel as superpackets, sends what waits in batches, segments runs of full-size datagrams in the kernel and writes coalesced segments back into the tunnel. Each of these is a mode of the data path, measured with it and without it, and the two offloads can be switched off at run time (NEXUS_ATLAS_TUN_OFFLOAD=off, NEXUS_ATLAS_LINK_GSO=off). With every mode on, the per-packet path is small enough that on x86 the cipher is about half of what atlasd spends at the ceiling.
| Mode | In the data path | Without → with, one desktop core per node |
|---|---|---|
| Statistics written off the data path | on | worker tick 452 → 61 µs idle; idle atlasd 0.53 → 0.23 % of a core; goodput and latency unchanged |
| Link queues sized in time, 50 ms, with backpressure; a 50 ms receive buffer | on | sender queue drops about 20,000 → 0 per window; retransmits 35,316 → 507; one flow 2,566 → 2,633 Mbit/s |
| Tunnel offload: TCP superpackets read from the tunnel, coalesced writes into it | on; NEXUS_ATLAS_TUN_OFFLOAD=off turns it off | 1,656 → 2,566 Mbit/s one flow; 7.24 → 5.25 CPU-s/GB at the ceiling; about 20 segments per tunnel write |
| Batched sends, with segmentation offload (UDP GSO) for runs of full-size datagrams | on; NEXUS_ATLAS_LINK_GSO=off turns the segmentation off, the batching stays | 2,633 → 3,202 Mbit/s one flow; one shaped link 11.63 → 7.53 CPU-s/GB, two links 13.05 → 9.37 |
| Segmentation runs on bonds that move or mirror a flow | off on lowest_delivery_time and broadcast bonds and on mesh relays | two links under broadcast: 4,909 retransmits per window with runs, 5 without; lowest_delivery_time 4,306 against 2,023 |
| Tunnel read and write batching | measured, left out | CPU ±4 %, retransmits +40–66 % |
| Segmentation for every datagram size, or runs of 64 at any link rate | measured, left out | 64 B UDP over two shaped links 93.8 → 37.6 thousand packets per second; eight shaped flows 732 retransmits against 90 |
| Both offloads off against on, same pass | Both offloads off | Both on |
|---|---|---|
| SBC, Pi Zero 2 W, Cortex-A53, one core, one shaped link | 60.8 Mbit/s, 179.6 CPU-s/GB | 79.2, 139.4 |
| SBC, Radxa Zero 3W, Cortex-A55, one core, one flow, unshaped | 68.5 Mbit/s (core saturated), 75.6 CPU-s/GB | 139.5 (core 88–91 %), 33.8 |
| SBC, Radxa Zero 3W, Cortex-A55, unpinned, one flow, unshaped | 200 Mbit/s, 73.3 CPU-s/GB | 300, 35.0 |
| Router, Beryl AX, Cortex-A53, one core, one flow, unshaped | 137 Mbit/s (core 97–98 %), 45.0 CPU-s/GB | 228 (core 81–86 %), 25.7 |
| Router, Beryl AX, Cortex-A53 ×2, not pinned, one flow, unshaped | 208 Mbit/s, 44.4 CPU-s/GB | 260, 29.0 |
| Mini PC, NUC 14 Pro, Core 3 100U P-core, one flow, unshaped | 1,566 (core 99 %) | 2,234 (core 77 %) |
| Laptop, Ryzen 5 7535U core, atlasd CPU-s/GB at 100 Mbit/s | 30.3 (17.0 µs per packet sent) | 26.8 (11.6) |
| Desktop, Ryzen 9 9950X3D core, one flow, unshaped | 1,987 Mbit/s | 3,259 |
| Desktop, Ryzen 9 9950X3D core, atlasd CPU-s/GB, unshaped | 6.72 | 4.51 |
| Desktop, Ryzen 9 9950X3D core, atlasd CPU-s/GB at 100 Mbit/s | 10.97 | 6.12 |
| Cloud VM, e2-medium, Broadwell vCPU (shared core), one flow, unshaped, tunnel MTU set by hand (1388) | 326.8 Mbit/s, 12.04 receiver CPU-s/GB | 907.5, 7.29 |
| Cloud VM, t2a-standard-2, Neoverse-N1 vCPU, one flow, unshaped, tunnel MTU set by hand (1388) | 508.4 Mbit/s, 12.00 receiver CPU-s/GB | 1,077, 6.88 |
| Cloud VM, n2-standard-2, Cascade Lake vCPU, one flow, unshaped, tunnel MTU set by hand (1388) | 678.8 Mbit/s, 8.94 receiver CPU-s/GB | 1,199, 5.68 |
| Cloud VM, c3-standard-4, Sapphire Rapids vCPU, one flow, unshaped, tunnel MTU set by hand (1388) | 1,025 Mbit/s, 6.26 receiver CPU-s/GB | 1,628, 4.22 |
| Cloud VM, c4-standard-4, Emerald Rapids vCPU, one flow, unshaped, tunnel MTU set by hand (1388) | 1,371 Mbit/s, 4.75 receiver CPU-s/GB | 2,207, 3.14 |
The first table measures each mode on its own against the data path without it, one desktop core per node, same-day runs. The second switches off the tunnel offload and the link’s segmentation offload together and runs the cell back to back with an as-shipped twin; batched sends, the time-sized queues and the statistics writer stay on. On the Pi the two offloads add 30 percent of goodput and take 22 percent off the CPU per GB; on one A55 core of a Radxa Zero 3W they double one flow and take 55 percent off the CPU per GB, the largest gain of any tier (1.50 times unpinned, against 1.66 on one A53 of a Beryl AX router, 1.43 on the NUC’s P-core and 1.30 on the Pi); on the Beryl AX they take 43 percent off the CPU per GB on one core and 35 percent on two, 1.25 times one flow; on the NUC they take the P-core off its ceiling to the wire; on a laptop they move CPU at the sender, not goodput or the pinned cores’ total (48.0 against 47.7 CPU-s/GB); on a desktop core they take 44 percent off the CPU per GB at 100 Mbit/s and lift the one-core ceiling from 1.99 to 3.26 Gbit/s; on the cloud VMs, measured with the tunnel MTU set by hand (1388), both cells of each pair at the same MTU, they are worth 1.6–1.8 times one flow on one Intel vCPU, 2.12 times on the Arm core and 2.78 times on e2-medium’s shared core, and 32–43 percent of the receiver’s CPU per GB, 1.19–1.35 times one flow not pinned; they work on every cloud NIC measured because the segmentation is the kernel’s. Switching off only the tunnel offload makes one shaped TCP flow retransmit about 1,000 times per window (1,023, against 0 as shipped and 6 with both off): switch both or neither. With runs, a bond that moves or mirrors a flow makes one TCP flow retransmit thousands of times per window, so those bonds send without runs, and so does a mesh relay, which has two links. No mode changes the wire format.
One harness, the same matrix on every tier.
Two Atlas nodes over one to three shaped links, the bare path and kernel WireGuard over the same link beside them, and chains of three and five nodes on the desktop and of three on the server. Each cell runs three times on the shipped build from a cold start of its data path; each test is a 20-second window after a 3-second warm-up, and tables report the median.
Raspberry Pi Zero 2 W: two Pis over one dedicated 802.11s radio link on 5 GHz. Radxa Zero 3W: a Rockchip RK3566 board ( four Cortex-A55 cores at 1.8 GHz, 2 GB) receiving from the desktop over its USB 3 gigabit Ethernet adapter, which offloads nothing. Beryl AX: a GL.iNet Beryl AX (GL-MT3000) Wi-Fi 6 travel router (MediaTek MT7981B, two Cortex-A53 cores at a fixed 1.3 GHz, 512 MB, OpenWrt) receiving from the desktop over its 2.5 GbE WAN port, linked at gigabit through two switches, with the router’s own firewall and its connection tracking left in place; also measured as the served-LAN gateway for a plain host on its LAN and as a Wi-Fi 6 access point. NUC 14 Pro: an ASUS NUC 14 Pro (Intel Core 3 100U: two P-cores with SMT, four E-cores) receiving from the desktop over one 2.5 GbE cable (Intel I226-V). Laptops: two Ryzen 5 7535U laptops over four VLANs of one direct cable through each laptop’s USB 2.0 Ethernet adapter. Desktop: each node a Linux network namespace joined by veth pairs, on a Ryzen 9 9950X3D, Debian 13, Linux 6.12. Virtual machines: two KVM guests with one vCPU and 1 GiB each, the whole QEMU process of a guest pinned to one host core, links over QEMU’s dgram backend and over tap + vhost-net. Phone: a Pixel 10 Pro XL (Google Tensor G5, Android 17) running the Android app’s bench build, the shipped app with a per-second sampler and an iperf3 client, as the measured endpoint over its own Wi-Fi 6 radio (5 GHz, 80 MHz) to a Beryl AX access point and on to the desktop, screen off and on battery; nothing reaches the phone, which fetches its tests from a collector on the LAN and reports there. Server: two-socket AMD EPYC 7763 hosts (64 cores per socket, 256 CPUs) running Proxmox, over 100 GbE ports through switches, a relay chain across three hosts, and one-vCPU Proxmox guests with the whole QEMU process on one host core and virtio-net with vhost. Cloud VMs: pairs of Google Compute Engine VMs of one machine type in one zone, e2-medium, t2a-standard-2, n2-standard-2, c3-standard-4 and c4-standard-4, on the provider’s network, MTU 1460, with Atlas as shipped, its tunnel MTU sized from the path, and the offload, encryption-off and WireGuard cells from a pass with the tunnel MTU set by hand (1388); a cross-region n2-standard-2 pair; a hub on 8-vCPU VMs with eight clients; and one relay server on an n2-standard-2, -4 and -8 and on a c3-standard-8 with 8 to 256 sessions from five client VMs, each node in its own network namespace behind NAT.
Every link is shaped per direction with an htb rate and a netem delay: 100 Mbit/s + 5 ms, 50 Mbit/s + 10 ms, 25 Mbit/s + 20 ms, and a twin of the first for the equal-link cell. Mesh hops are 100 Mbit/s + 2 ms each. Unshaped cells have no queueing discipline and show what one core, or the link itself, can move. No link fails or fades on its own.
Every process of a node, atlasd, iperf3 and the probes, is pinned to that node’s one core: one core of the Pi, one A55 core of the Radxa with the USB controller’s interrupt on it, one A53 core of the Beryl AX with the Ethernet controller’s interrupts on it, a P-core with its sibling offline or an E-core on the NUC, one core of each laptop, a physical core with its SMT sibling idle on the desktop, the guest’s single vCPU, one core on the NICs’ socket or on the other socket of the server, one vCPU of a cloud VM with the NIC’s interrupts on it and its hyperthread sibling idle. The Pi, the Radxa, the Beryl AX, the NUC, the server and the cloud VMs are also measured unpinned, as they are installed; the phone only unpinned, since an app cannot pin its threads. Kernel WireGuard’s crypto runs in per-CPU kernel workers that cannot be pinned, so only the virtual machines hold it to one core.
TCP goodput with one and eight flows; UDP at 64, 512 and 1388 B payloads offered at the cell’s nominal capacity, and unthrottled on unshaped links; CPU of each process from its own counters and of each pinned core; peak resident memory; latency from 200 unloaded 64 B probes per second for 10 seconds; bytes on the wire per packet from the underlay counters; 60 seconds of idle cost. The Radxa Zero 3W, Beryl AX, NUC 14 Pro, server and cloud tiers add a per-second sampler on every node: per-CPU busy and softirq, NIC queues, per-thread CPU and the daemon’s own counters. The phone samples itself once a second: the app’s CPU with its traffic generator apart, the whole phone’s power from the battery’s fuel gauge, the charge counter, the thermal status, the Wi-Fi signal and the engine’s own counters; its traffic tests run 75 seconds and are read over their last 60.
÷ bare holds a bond against perfect aggregation of its links. CPU-s per GB is CPU time per gigabyte delivered, both ends unless a row says otherwise. µs per packet divides atlasd’s CPU time by the full-size TCP packets delivered, however the tunnel or the socket batched them. Each Atlas cell also runs with encryption off, which splits Atlas’s cost into the cipher and the rest of the data path.
The same cells, the same result, on three tiers.
The shaped links set these rates, not the processor: moving from a desktop core to a virtual machine or a laptop changes the CPU figures, not the goodput. With equal links the bond carries twice one link. With unequal links, how much it carries depends on the strategy and the number of flows; on the desktop one TCP flow over three unequal links carries what eight do.
| TCP Mbit/s, one flow, median of 3 | Laptops, Ryzen 5 7535U | Desktop, Ryzen 9 9950X3D | Desktop VM, Ryzen 9 9950X3D |
|---|---|---|---|
| Bare link, 100 Mbit/s | 95.6 | 95.6 | 95.5 |
| Kernel WireGuard | 91.5 | 91.6 | 91.5 |
| Atlas, one link | 90.8 | 90.8 | 90.7 |
| Atlas, one link, encryption off | 91.8 | 91.8 | 91.7 |
| Two links (100 + 50), wrr | 136.1 | 136.2 | 136.1 |
| Three links (100 + 50 + 25), wrr, one flow | 137.7 ¹ | 158.2 | 158.6 |
| Three links, wrr, eight flows | 158.5 | 158.5 | 158.7 |
| Two equal links (2 × 100), wrr | 181.4 | 181.6 | 181.5 |
| Two links, wrr, QoS on | 129.2 | 129.3 | 129.2 |
| Two links, lowest_delivery_time | 84.2 | 83.9 | 85.0 |
| Three links, lowest_delivery_time | 85.0 | 84.0 | 85.2 |
| Two links, broadcast | 90.7 | 90.8 | 90.6 |
¹ Bimodal per run on the laptops: 136.5–158.6 Mbit/s; with eight flows the cell carries 158.5. wrr = weighted_round_robin, with weights in proportion to declared capacity. QoS on paces every declared link to 95 percent of its wire, so the two links carry 0.95 of their unpaced rate by design. One TCP flow retransmits 2,069 times per 20 s window over two links under lowest_delivery_time and 5 times under broadcast. The Raspberry Pi, the Radxa Zero 3W, the Beryl AX and the edge computer ran one link only; two links on the server are in the server section.
UDP at full size, desktop tier+
| Cell | Offered Mbit/s | 1388 B goodput | Loss % |
|---|---|---|---|
| Bare link | 100 | 97.1 | 2.9 |
| Kernel WireGuard | 100 | 92.9 | 7.1 |
| Atlas, one link | 100 | 92.4 | 7.6 |
| Two links, wrr | 150 | 138.6 | 7.6 |
| Three links, wrr | 175 | 161.7 | 7.6 |
| Two equal links, wrr | 200 | 184.8 | 7.6 |
| Two links, lowest_delivery_time | 100 | 81.1 | 18.9 |
| Two links, broadcast | 50 | 50.0 | 0.0 |
Offered at the cell’s nominal capacity: the sum of the links for weighted_round_robin, the fastest link for lowest_delivery_time, the slowest for broadcast. The loss at the offered rate is each path’s header overhead: the link is full before the payload is. At 64 B one shaped 100 Mbit/s link carries 70.2 thousand packets per second through Atlas, the wire’s limit with 72 B added per packet.
What Atlas adds beside the throughput.
Atlas adds 72.0 B to every packet: 28 B of outer IPv4 and UDP and 44 B of its own header and tag, at every payload size. WireGuard adds 64 B. With encryption off Atlas adds 56 B, with end-to-end sealing 139.0 B, the 67 B seal on top.
| Tier | Added latency ms (WireGuard) | Idle, per link and node | Idle atlasd CPU | atlasd RSS MiB | RSS with QoS on, MiB |
|---|---|---|---|---|---|
| SBC, Pi Zero 2 W, Cortex-A53 | 0.61 (0.28) ¹ | 8.1 packets/s | 2.4 % of an A53 core | 24.5–29.0; 39.3 peak unshaped, not pinned | – |
| SBC, Radxa Zero 3W, Cortex-A55 | 0.55 (0.28) ¹ | – | 4.1–4.5 % of an A55 core | 22–31; 35.8 peak unshaped, not pinned | – |
| Router, Beryl AX, Cortex-A53 | 0.007 (0.003) ¹; not pinned 0.27 | – | 1.7–1.9 % of an A53 core | 17.7–26.0; 35.5 peak unshaped, not pinned | – |
| Mini PC, NUC 14 Pro, Core 3 100U | none on the P-core ² | – | 0.40–0.70 % of a core | 25–31 | – |
| Laptops, Ryzen 5 7535U | 0.006 (0.277) ¹ | 745–750 B/s, 8.1 packets/s | 0.34–0.69 % of a core | 25.3–25.8; 32.4 unshaped | 28.6–35.0 |
| Desktop, Ryzen 9 9950X3D | 0.015 (0.026) | 745 B/s, 8.1 packets/s | 0.30 % of a core | 26.1; 30.9 unshaped | 39.6 |
| Desktop VM, Ryzen 9 9950X3D | 0.00 (0.00) ¹ | 751 B/s, 8.1 packets/s | 0.35–0.39 % of a vCPU | 26.1–26.2; 28.0–28.2 unshaped | 35.0 |
| Server, EPYC 7763 core | none ² | 8.2 packets/s | 0.35 % of a core | 25.1–31.3; 49.7 not pinned | – |
| Server VM, Proxmox guest on the EPYC 7763 | none ² | – | 0.35–0.40 % of the vCPU | 25.2–31.8 | – |
| Cloud VM, e2-medium, Broadwell vCPU (shared core) | 0.30 (0.02) ¹; not pinned 0.30 | – | 0.94 % of a vCPU | 25.0–31.4; 35.8 not pinned | – |
| Cloud VM, t2a-standard-2, Neoverse-N1 vCPU | none (none) ¹ | – | 0.69 % of a vCPU | 24.5–30.1; 35.3 not pinned | – |
| Cloud VM, n2-standard-2, Cascade Lake vCPU | none (none) ¹ | – | 0.79 % of a vCPU | 24.9–31.3; 35.0 not pinned | – |
| Cloud VM, c3-standard-4, Sapphire Rapids vCPU | none (none) ¹ | – | 0.49 % of a vCPU | 24.7–31.1; 40.5 not pinned | – |
| Cloud VM, c4-standard-4, Emerald Rapids vCPU | none (none) ¹ | – | 0.50 % of a vCPU | 24.8–26.6; 40.0 not pinned | – |
Latency is unloaded, 64 B probes: one way over the bare link on the desktop, ¹ half the round trip, ² the round trip p50 equal to the bare link’s within 0.01 ms. On the laptops kernel WireGuard adds 0.277 ms in every pass, likely the wake-up of its idle per-CPU workers. On the cloud VMs the probe runs over the unshaped NIC, Atlas as shipped; “none” there is within 0.005 ms of bare, and on e2-medium’s shared core Atlas adds 0.30 ms in both placements. The WireGuard figures in brackets and the idle cost and memory, the receiving VM’s, are from the pass with the tunnel MTU set by hand (1388), where e2-medium VMs on another host added 0.02 ms pinned. Idle traffic is the link probes every 250 ms and their replies plus a keepalive every 25 s. Resident memory is a few MiB higher with an unshaped sender (30.9 MiB against 26.1 on the desktop, 32.4 against 25.3–25.8 on a laptop): each link’s buffers are sized in time, and an unshaped link fills them. Unpinned on the 256-CPU server, with one worker per CPU, atlasd holds 49.7 MiB (52.1 MB) at the ceiling. RSS is in MiB here; the memory section below gives each figure in MB as well.
About 25 MiB (26 MB), set by the configuration, not the processor.
atlasd’s resident memory is 17.5–25.5 MiB (18.3–26.7 MB) idle on every tier and 17.7–29.0 MiB (18.5–30.4 MB) at a 100 Mbit/s link, the Beryl AX router lowest. Sending one flow at an x86 core’s ceiling adds 2.0–6.6 MiB (2.1–6.9 MB) at the sender, the tunnel and link offloads’ batch buffers filling at full rate; with the offloads off the sender stays where it was. QoS holds a paced link’s backlog in its per-class queues, 39.6 MiB (41.6 MB) at the sender on two links, and an unpinned many-core host runs one data-path worker per CPU, 49.7 MiB (52.1 MB) on 256 CPUs. The cloud VMs hold what the other x86 tiers hold, as shipped: 24.4–25.0 MiB (25.6–26.2 MB) idle, 25.1–26.7 MiB (26.3–28.0 MB) at a shaped link and 26.6–31.4 MiB (27.8–32.9 MB) at one vCPU’s ceiling.
| Tier | Idle, MiB (MB) | 100 Mbit/s shaped, MiB (MB) | One-core ceiling, MiB (MB) | Peak, MiB (MB) |
|---|---|---|---|---|
| SBC, Pi Zero 2 W, Cortex-A53, one core | 23.9 (25.1) | 29.0 (30.4) | 29.5 (31.0) | 30.3 (31.7) |
| SBC, Radxa Zero 3W, Cortex-A55, one core | 18.6 (19.5) | 25.0 (26.2) | 22.7 (23.8) | 28.3 (29.7) |
| Router, Beryl AX, Cortex-A53, one core | 17.5 (18.3) | 17.7 (18.5) | 23.5 (24.7) | 26.0 (27.3) |
| Mini PC, NUC 14 Pro, Core 3 100U P-core | 25.5 (26.7) | 25.6 (26.9) | 28.9 (30.3) | 31.7 (33.2) |
| Mini PC, NUC 14 Pro, Core 3 100U E-core | 25.1 (26.4) | 25.3 (26.6) | 28.8 (30.2) | 31.3 (32.9) |
| Laptop, Ryzen 5 7535U core | 25.0 (26.2) | 25.8 (27.0) | 32.4 (34.0) | 36.1 (37.9) |
| Desktop, Ryzen 9 9950X3D core, network namespaces | 24.6 (25.8) | 26.1 (27.3) | 30.9 (32.4) | 41.0 (43.0) |
| Desktop VM, Ryzen 9 9950X3D, one vCPU, dgram links | 24.6 (25.8) | 26.1 (27.4) | 28.1 (29.5) | 36.7 (38.5) |
| Desktop VM, Ryzen 9 9950X3D, one vCPU, tap + vhost-net | 24.5 (25.7) | 26.2 (27.4) | 28.0 (29.4) | 29.4 (30.8) |
| Server, EPYC 7763 core, NICs’ socket | 25.2 (26.4) | 26.7 (28.0) | 31.3 (32.8) | 31.6 (33.1) |
| Server VM, Proxmox guest on the EPYC 7763, one vCPU | 25.1 (26.3) | 26.0 (27.3) | 31.8 (33.3) | 32.0 (33.5) |
| Cloud VM, e2-medium, Broadwell vCPU (shared core) | 25.0 (26.2) | 26.7 (28.0) | 31.4 (32.9) | 31.5 (33.1) |
| Cloud VM, t2a-standard-2, Neoverse-N1 vCPU | 24.5 (25.7) | 25.1 (26.3) | 30.1 (31.6) | 30.1 (31.6) |
| Cloud VM, n2-standard-2, Cascade Lake vCPU | 24.9 (26.1) | 26.2 (27.5) | 31.3 (32.8) | 31.5 (33.0) |
| Cloud VM, c3-standard-4, Sapphire Rapids vCPU | 24.7 (25.9) | 26.6 (27.9) | 31.1 (32.6) | 31.6 (33.1) |
| Cloud VM, c4-standard-4, Emerald Rapids vCPU | 24.8 (26.0) | 26.5 (27.8) | 26.6 (27.8) | 28.3 (29.7) |
VmRSS of each atlasd at the end of the run (idle: after the idle window that opens it), median of three runs, the larger of the two ends; on the NUC 14 Pro, the Radxa Zero 3W and the Beryl AX the receiving board. The Beryl AX runs a static build with musl’s memory allocator, a difference from the other tiers that was not separated; not pinned, on its two cores, it peaks at 35.5 MiB (37.2 MB), with encryption off. On the cloud VMs both VMs of a pair count, Atlas as shipped, idle from the pass with the tunnel MTU set by hand (1388); on the c4-standard-4 pair, where the receiving end sets the pace, the sender stays at 26.6 MiB (27.8 MB) at the ceiling; not pinned, with one data-path worker per vCPU, the sender holds 35.0–35.8 MiB (36.7–37.5 MB) on 2 vCPUs and 40.0–40.5 MiB (41.9–42.5 MB) on 4. Peak: the highest VmHWM of any cell the tier ran, QoS included. The number of links, the strategy, end-to-end sealing, encryption and a mesh relay’s role move it by less than a megabyte; a declared link capacity asks the kernel, not atlasd, for its larger receive buffer. Every figure is given twice: MiB, what the kernel reports (1 MiB = 1,048,576 bytes), and in brackets MB, decimal (1 MB = 1,000,000 bytes); throughput figures are megabits per second and unaffected. Kernel WireGuard runs in the kernel and has no process figure to set beside these. Not measured: growth over hours and the Android app. The relay server holds 5.9 MiB (6.2 MB) fresh and 6.6 MiB (6.9 MB) with 256 sessions, about 2.8 KiB a session, and a hub 26–29 MiB taking traffic in, 39–42 MiB sending out (sizing). The per-cell table, the knobs measured as pairs and the method are section 3.6 of the performance report.
The processor sets the price of a gigabyte.
At 100 Mbit/s a gigabyte costs atlasd 138 CPU-seconds on a Pi’s A53, and on one A55 core of a Radxa Zero 3W the receiving end alone costs 64.5, on one A53 core of a Beryl AX router 45.9; 27.2 on a laptop core, 6.1 on a desktop core, 14–18 on one vCPU, 18.3 on a server core and 5.7–19.6 at the receiving end on one cloud vCPU, the newest core cheapest. The tunnel offload and the link’s segmentation offload take 44 percent off the desktop core’s figure and 22 percent off the Pi’s. The cipher is about half of what remains on x86 at the ceiling; at 100 Mbit/s on a laptop or a Pi it is a fifth.
| Tier, core | atlasd CPU-s/GB | Offloads off | Encryption off | atlasd per packet, µs, sender / receiver |
|---|---|---|---|---|
| SBC, Pi Zero 2 W, Cortex-A53, one core | 138.4 | 179.6 ¹ | 109.0 | 106 / 81; offloads off 136 / 106 ¹ |
| SBC, Pi Zero 2 W, Cortex-A53, unpinned | 191.2 | – | 172.1 | 134 / 124 |
| SBC, Radxa Zero 3W, Cortex-A55, one core, receiving end | 64.5 | – | – | receiver 95.8 |
| SBC, Radxa Zero 3W, Cortex-A55, unpinned, receiving end | 132.3 | – | – | receiver 201.7 |
| Router, Beryl AX, Cortex-A53, one core, receiving end | 45.9 | – | – | receiver 63.3 |
| Router, Beryl AX, Cortex-A53 ×2, not pinned, receiving end | 44.8 | – | – | receiver 61.4 |
| Mini PC, NUC 14 Pro, Core 3 100U P-core, receiving end | 6.87 | – | – | receiver 9.65 |
| Mini PC, NUC 14 Pro, Core 3 100U E-core, receiving end | 19.96 | – | – | receiver 28.03 |
| Mini PC, NUC 14 Pro, Core 3 100U, unpinned, receiving end | 11.71 | – | – | receiver 16.24 |
| Laptop, Ryzen 5 7535U core | 27.2 | 30.3 ¹ | 22.0 | 12.2 / 24.7; offloads off 17.0 / 24.3 ¹ |
| Desktop, Ryzen 9 9950X3D core | 6.12 | 10.97 | 3.01 | – |
| Desktop VM, Ryzen 9 9950X3D, one vCPU, dgram | 17.7 | – | 13.25 | – |
| Desktop VM, Ryzen 9 9950X3D, one vCPU, vhost-net | 14.3 | – | – | – |
| Server, EPYC 7763 core | 18.3 | – | – | 10.6 / 14.4 |
| Server VM, Proxmox guest on the EPYC 7763, in the guest | 22.7 | – | – | – |
| Cloud VM, e2-medium, Broadwell vCPU (shared core), receiving end | 19.6 | 32.1 ² | – | – |
| Cloud VM, e2-medium, Broadwell, not pinned, receiving end | 19.5 | 45.3 ² | – | – |
| Cloud VM, t2a-standard-2, Neoverse-N1 vCPU, receiving end | 11.7 | 26.8 ² | – | – |
| Cloud VM, t2a-standard-2, Neoverse-N1, not pinned, receiving end | 13.5 | 29.2 ² | – | – |
| Cloud VM, n2-standard-2, Cascade Lake vCPU, receiving end | 13.9 | 22.5 ² | – | – |
| Cloud VM, n2-standard-2, Cascade Lake, not pinned, receiving end | 15.6 | 32.5 ² | – | – |
| Cloud VM, c3-standard-4, Sapphire Rapids vCPU, receiving end | 6.5 | 13.1 ² | – | – |
| Cloud VM, c3-standard-4, Sapphire Rapids, not pinned, receiving end | 7.4 | 17.1 ² | – | – |
| Cloud VM, c4-standard-4, Emerald Rapids vCPU, receiving end | 5.7 | 9.1 ² | – | – |
| Cloud VM, c4-standard-4, Emerald Rapids, not pinned, receiving end | 6.7 | 12.5 ² | – | – |
One TCP flow at 100 Mbit/s over one link. ¹ The control pair with the tunnel offload and the link’s segmentation offload switched off; its as-shipped twin read 26.8 CPU-s/GB and 11.6 / 24.6 µs on the laptops, 139.4 and 108.7 / 79.9 on the Pi. At link rate a gigabyte costs more than at the ceiling, because fewer segments arrive together to batch: on the server 18.3 against 9.3 CPU-s per GB. At 100 Mbit/s atlasd wakes a server core once per two packets, so it costs about 2 percent of one server core per 100 Mbit/s at each end. Quote a per-GB cost at the deployment’s rate. The edge figures are the receiving NUC’s, per datagram received; at 100 Mbit/s the NUC coalesces almost nothing, the segments arriving one at a time. The Radxa figures are the receiving board’s, per datagram received: at 100 Mbit/s it coalesces 6.8 segments per tunnel write against 13.6 at the ceiling, so one A55 core spends 64.5 CPU-seconds per GB there against 33.8 at the ceiling; unpinned, four workers wake for a flow one worker carries, 132.3 CPU-seconds per GB on 1.5 of the board’s four cores. The Beryl AX figures are the receiving router’s, per datagram received: at 100 Mbit/s it coalesces 7.3 segments per tunnel write against 31 at the ceiling, so one A53 spends 45.9 CPU-seconds per GB there against 25.7, with a quarter of the core to spare; not pinned costs no more per byte, 44.8, because with two cores there is little to spread over. The cloud figures are the receiving VM’s, Atlas as shipped: the shaped link costs a byte 1.6–2.8 times the ceiling’s figure there too. ² Measured with the tunnel MTU set by hand (1388), both cells of a pair at the same MTU; the offloads-on twin there read 18.0 / 20.1, 12.0 / 13.2, 14.3 / 15.7, 6.5 / 7.6 and 4.8 / 5.8, and on one vCPU at the ceiling the cipher is 53–56 percent of the receiving atlasd on the x86 shapes, 39 on e2-medium’s Broadwell and 44 on the Neoverse-N1, measured the same way.
Held to one core, Atlas and kernel WireGuard carry about the same.
Where both tunnels are confined to the same single core, Atlas carries one TCP flow at 1.86 Gbit/s against WireGuard’s 1.60 on vhost-net, 1.30 against 1.35 on dgram links and 0.81 against 0.78 in a Proxmox guest. On bare metal WireGuard’s crypto spreads over the host’s cores, so those rows are not single-core comparisons. On one cloud vCPU Atlas carries one flow at 1.02–2.10 Gbit/s as shipped, one end’s vCPU the limit; there too kernel WireGuard runs on every vCPU of the VM.
| Tier | Atlas, one core | Kernel WireGuard | Atlas, encryption off | Cipher share at the ceiling | What bounds it |
|---|---|---|---|---|---|
| SBC, Pi Zero 2 W, Cortex-A53, one core | 87.4 Mbit/s | 102.0 ¹ | 103.7 | 32 % ² | the A53 core, 98 % busy; bare 108.1 |
| SBC, Radxa Zero 3W, Cortex-A55, one core | 139.5 Mbit/s; offloads off 68.5 | 342 ¹ | 236 | 39 % | the A55 core, 88–91 % busy, saturated with the offloads off; bare 479 |
| SBC, Raspberry Pi CM4, Cortex-A72 | Pending | Pending | Pending | Pending | Pending |
| Phone, Pixel 10 Pro XL, Tensor G5, not pinned (as deployed) | 309 / 319 Mbit/s ⁵ | – | – | – | the radio and the phone’s receive path; bare 453 / 363 |
| Router, Beryl AX, Cortex-A53, one core | 228 Mbit/s; offloads off 137 | 550 ¹ | 371 | 41 % | atlasd’s one worker, the core 81–86 % busy; bare 678, the switched gigabit path |
| Router, Beryl AX, Cortex-A53 ×2, not pinned | 260 Mbit/s on two cores; offloads off 208 | 564 | 408 | 39 % | one flow’s worker path, the busier core 74 %; bare 705 |
| Jetson, Orin Nano, Cortex-A78AE | Pending | Pending | Pending | Pending | Pending |
| Mini PC, NUC 14 Pro, Core 3 100U P-core | 2.23 Gbit/s; offloads off 1.57 | 2.25 | 2.26 | 55 % | the 2.5 GbE wire; the core 77 % busy, 99 % with the offloads off |
| Mini PC, NUC 14 Pro, Core 3 100U E-core | 1.58 Gbit/s | 2.25 | 2.26 | 44 % | the core, 98 % busy |
| Laptop, Ryzen 5 7535U core | 238.8 Mbit/s | 241.8 ¹ | 238.9 | 21 % | the USB 2.0 adapter, for all three; bare 258.2 |
| Desktop, Ryzen 9 9950X3D core | 3.26 Gbit/s; offloads off 1.99 | 6.64 ¹ | 7.80 | 64 % | the sender’s core, atlasd at 99 % |
| Desktop VM, Ryzen 9 9950X3D, one vCPU, vhost-net | 1.86 Gbit/s | 1.60 | 3.14 | about 45 % | the vCPU; eight flows: Atlas 1.70, WireGuard 1.81 |
| Desktop VM, Ryzen 9 9950X3D, one vCPU, dgram | 1.30 Gbit/s | 1.35 | 1.70 | – | QEMU’s dgram I/O, for both tunnels: a lower bound |
| Server, EPYC 7763 core, NICs’ socket | 1.59 Gbit/s | 1.77 ¹ | 3.10 | 51 % | the core, 90–100 % busy at both ends |
| Server, EPYC 7763 core, other socket | 1.33 Gbit/s | 1.73 ¹ | 2.35 | 48 % | the core, plus the softirq on the NICs’ socket |
| Server VM, Proxmox guest on the EPYC 7763, one vCPU | 0.81 Gbit/s | 0.78 | – | – | the host core, QEMU and vhost on it |
| Cloud VM, e2-medium, Broadwell vCPU (shared core) | 1.02 Gbit/s; offloads off 0.33 ⁴ | 1.22 ¹ | 1.13 | 39 % | one vCPU at each end, 94–96 % busy; bare 1.96, the shape’s egress cap |
| Cloud VM, e2-medium, Broadwell, shared core, not pinned | 0.62 Gbit/s; offloads off 0.51 ⁴ | 1.16 ³ | 0.52 | – | a shared core, erratic run to run (602–680 Mbit/s) |
| Cloud VM, t2a-standard-2, Neoverse-N1 vCPU | 1.06 Gbit/s; offloads off 0.51 ⁴ | 2.22 ¹ | 1.83 | 44 % | the receiving vCPU, 98 % busy; bare 9.89 |
| Cloud VM, t2a-standard-2, Neoverse-N1, not pinned | 1.06 Gbit/s; offloads off 0.80 ⁴ | 2.06 ³ | 1.87 | 43 % | one flow’s worker path over two cores, level with one vCPU |
| Cloud VM, n2-standard-2, Cascade Lake vCPU | 1.22 Gbit/s; offloads off 0.68 ⁴ | 2.23 ¹ | 2.23 | 53 % | the sending vCPU, 99 % busy; bare 9.89 |
| Cloud VM, n2-standard-2, Cascade Lake, not pinned | 1.06 Gbit/s; offloads off 0.78 ⁴ | 1.42 ³ | 1.89 | 45 % | two workers on one physical core’s two hyperthreads |
| Cloud VM, c3-standard-4, Sapphire Rapids vCPU | 1.58 Gbit/s; offloads off 1.03 ⁴ | 5.02 ¹ | 3.21 | 54 % | the sending vCPU, 100 % busy; bare 21.7 |
| Cloud VM, c3-standard-4, Sapphire Rapids, not pinned | 1.38 Gbit/s; offloads off 1.17 ⁴ | 2.82 ³ | 2.60 | 51 % | one flow’s worker path over four vCPUs |
| Cloud VM, c4-standard-4, Emerald Rapids vCPU | 2.10 Gbit/s; offloads off 1.37 ⁴ | 5.73 ¹ | 4.32 | 56 % | the receiving vCPU, 98 % busy; bare 22.0 |
| Cloud VM, c4-standard-4, Emerald Rapids, not pinned | 1.91 Gbit/s; offloads off 1.55 ⁴ | 3.61 ³ | 3.48 | 52 % | one flow’s worker path over four vCPUs |
One TCP flow, unshaped link, median of 3. Rows by the platform’s power draw, as in the table across the tiers, the cloud VMs last by hourly price; Pending rows are tiers scheduled for a later edition. ¹ Not a single-core figure: kernel WireGuard’s crypto workers run on unpinned CPUs. ² Of the pinned cores over the bare link. ³ Kernel WireGuard at its 1420 default, which fragments on the cloud network’s 1460 B MTU; its 1380 cell ran in the one-vCPU placement only. Cloud VMs: Atlas as shipped, its tunnel MTU 1376 sized from the path; ⁴ the offloads-off, kernel WireGuard, encryption-off and cipher-share cells measured with the tunnel MTU set by hand (1388), both cells of a pair at the same MTU, kernel WireGuard at 1380, the MTU wg-quick picks on a 1460 B route, unless marked ³; CPU figures the receiving VM’s; bare is the VM’s egress cap, 2, 10 and 22 Gbit/s, not a CPU figure. ⁵ Phone: an endpoint device over its own radio, not a core ceiling, one TCP flow down / up over Wi-Fi 6, screen off, on battery; bare is the same radio link with no VPN in the same pass; the WireGuard app’s tunnel was not run; Android has no data-path switches; Atlas carries 0.68 of the bare link downstream and 0.88 upstream, the engine’s threads at 54 % of a core downstream; the app’s engine costs 16.9 / 27.1 CPU-s per GB, not atlasd’s figure. Like for like on one vCPU with vhost-net, Atlas costs 7.0 guest vCPU-s per GB at the ceiling against WireGuard’s 6.2, and the whole host core 8.3 against 9.8. Across the whole server node, both ends, every process and softirq counted, Atlas costs 13.9 CPU-s per GB at the one-flow ceiling against WireGuard’s 28.3, within ±10 percent on live hosts. Offloads off means the tunnel offload and the link’s segmentation offload switched off in the same pass. With end-to-end sealing on, one desktop core carries 1.83 Gbit/s. On the Beryl AX both tunnels unpinned on its two cores are like for like: there kernel WireGuard, with the kernel’s arm64 assembly cipher on both cores and no crossing into user space, carries 564 Mbit/s against Atlas’s 260, at 26.1 against 37.6 CPU-s per GB of the whole board.
One vCPU carries what its virtual NIC lets it.
Three placements of a one-vCPU guest: a desktop KVM guest over QEMU’s dgram backend and over tap + vhost-net, and a Proxmox guest on a server host. In each, the whole QEMU process sits on one host core. At 100 Mbit/s Atlas costs the guest or the host core about half of what kernel WireGuard costs; at the ceiling one flow through Atlas carries about what one through WireGuard does, more on vhost-net.
| One vCPU per guest | Atlas one flow, Gbit/s | WireGuard one flow | Eight flows, Atlas / WireGuard | CPU-s/GB at the ceiling, Atlas / WireGuard | CPU-s/GB at 100 Mbit/s, Atlas / WireGuard |
|---|---|---|---|---|---|
| Desktop VM, Ryzen 9 9950X3D, dgram links | 1.30 | 1.35 | 1.24 / 1.54 | 6.3 / 3.2 | 28.1 / 51.6 |
| Desktop VM, Ryzen 9 9950X3D, tap + vhost-net | 1.86 | 1.60 | 1.70 / 1.81 | 7.0 / 6.2 | 21.3 / 40.9 |
| Server VM, Proxmox guest on the EPYC 7763, virtio-net with vhost ¹ | 0.81 | 0.78 | 0.81 / 0.70 | 18.7 / 20.1 | 49 / 100 |
Both ends, one TCP flow, median of 3. CPU per GB is the guests’ vCPU on the desktop and ¹ the whole host core under the guest on the Proxmox host, QEMU and its vhost worker included. On the desktop’s host cores Atlas costs 56.5 against WireGuard’s 134.9 CPU-s per GB at 100 Mbit/s over dgram links, and over vhost-net 31.7 against 104.1 at 100 Mbit/s and 8.3 against 9.8 at the ceiling. The dgram backend holds both tunnels at QEMU’s own I/O, its process 91–98 percent of each host core, so that row’s ceiling is a lower bound for both. In the Proxmox guest the guest’s own counters under-read under steal, so its figures are the host core’s. Read none of these against the bare-metal core figures as if a vCPU were a host core: the vhost worker runs on the same host core.
One P-core reaches the 2.5 GbE wire.
On an ASUS NUC 14 Pro with an Intel Core 3 100U, one P-core receives one TCP flow through Atlas at 2,234 Mbit/s, against 2,245 for kernel WireGuard, with the core 77 percent busy: the wire is the limit, not the processor. An E-core is the limit, at 1,576 Mbit/s. Unpinned, as installed, one flow carries 2,194. With the tunnel offload and the link’s segmentation offload switched off, the P-core is the limit again: 1,566 Mbit/s with the core 99 percent busy.
| NUC receiving | One flow Mbit/s | Eight flows | Kernel WireGuard | Atlas, encryption off | atlasd CPU-s/GB | atlasd per datagram, µs | Core busy |
|---|---|---|---|---|---|---|---|
| P-core, SMT sibling offline | 2,234 | 2,234 | 2,245 | 2,257 | 2.62 | 3.60 | 77 % |
| P-core, offloads off | 1,566 | 1,489 | 2,245 | – | 4.40 | 6.04 | 99 % |
| E-core | 1,576 | 1,446 | 2,245 | 2,256 | 4.85 | 6.66 | 98 % |
| Not pinned, as installed | 2,194 | 2,227 | 2,245 | 2,247 | 3.13 | 4.29 | 0.93 cores |
One TCP flow from the desktop to the NUC over one unshaped 2.5 GbE cable, median of 3; the bare link carries 2,354 Mbit/s. CPU figures are the receiving NUC’s; offloads off is the same cell with the tunnel offload and the link’s segmentation offload switched off, run back to back with it. Unpinned, atlasd spreads the flow over its eight workers, P- and E-cores alike, at 0.93 cores in all. Declaring the wire rate as the link’s capacity changes nothing for TCP on the P-core or unpinned and costs 9 percent on the saturated E-core, so declare a capacity where the link’s real rate is known and shaped. Shaped to 100 Mbit/s every placement carries 90.8 Mbit/s, against 95.6 bare and 91.3 with WireGuard.
One server core carries 1.59 Gbit/s. Pin it.
On one core of an AMD EPYC 7763 over 100 GbE, atlasd carries one TCP flow at 1.59 Gbit/s, at 9.3 CPU-seconds per GB. Kernel WireGuard carries 1.77, spreading its crypto over several cores, and costs twice what Atlas costs per byte across the whole node. The core is the limit; the link’s send queue drops nothing.
| One 100 GbE link | One flow Gbit/s | Eight flows | Kernel WireGuard | Bare link | atlasd CPU-s/GB | atlasd per packet, µs, sender / receiver |
|---|---|---|---|---|---|---|
| One core on the NICs’ socket | 1.59 | 1.44 | 1.77 | 8.14 | 9.3 | 6.2 / 6.6 |
| One core on the other socket | 1.33 | 1.20 | 1.73 | 5.37 | 10.5 | 6.4 / 7.9 |
| Not pinned, as installed ¹ | 0.26 | 0.24 | 1.74 | 11.19 | 39.7 | 44.4 / 10.2 |
| NICs’ socket, encryption off | 3.10 | 2.75 | – | – | 4.5 | 2.9 / 3.3 |
| NICs’ socket, end-to-end sealing | 0.93 | 0.90 | – | – | 16.5 | 11.4 / 11.1 |
One TCP flow, unshaped, median of 3; bare and WireGuard capped at 40 Gbit/s per test. ¹ atlasd starts one data-path worker per CPU, 261 threads on a 256-CPU host, and one flow’s datagrams reach the tunnel through several of them, which breaks the batches: 6.1 times slower than pinned at four times the CPU per byte. Pin atlasd to a core on the NIC’s NUMA node; on the other socket it carries 16 percent less, because the receive softirq stays on the NICs’ socket. Hosts with few CPUs are not affected the same way: the edge computer unpinned carries 2,194 Mbit/s, and the Pi unpinned beats pinned.
| Two 100 GbE links, one core on the NICs’ socket | One flow Gbit/s | Retransmits per 20 s | atlasd CPU-s/GB |
|---|---|---|---|
| One link | 1.59 | 188 | 9.3 |
| Two links, weighted_round_robin | 0.93 | 2,699 | 14.6 |
| Two links, lowest_delivery_time | 0.95 | 0 | 14.1 |
| Two links, broadcast | 0.65 | 0 | 23.3 |
The receiver’s coalescing sets the rate, and it needs one flow’s datagrams in order on one socket: over one link it writes 33 segments per tunnel write, over two links under weighted_round_robin 3.7, and the two ports reorder the flow. lowest_delivery_time and broadcast send single datagrams, so the sender’s core is the limit. On links faster than a core, bond for redundancy; on links slower than a core, every radio tier, the bonding gains hold.
| Relay chain over 100 GbE | One hop, Gbit/s | Two hops, Gbit/s | Forwarder CPU-s/GB | Forwarder per packet, µs |
|---|---|---|---|---|
| Kernel IP forwarding | 9.50 | 6.55 | – | – |
| Atlas mesh | 1.59 | 0.70 | 11.4 | 15.5 |
| Atlas mesh, encryption off | 3.16 | 1.22 | 6.5 | 8.8 |
One TCP flow, unshaped. The forwarder is the limit at two hops: its atlasd at 99.7 percent of its core while the two ends run at 48–60 percent. A forwarder never touches the tunnel, so the tunnel offload does not reach it, and it sends one datagram per send; a relay hop costs about 1.4 cores per Gbit/s forwarded. Over one 10 GbE hop one flow carries 1.32 Gbit/s: those ports sit on the other socket, and the figure is that placement’s.
A relay hop costs 7 to 9.5 CPU-seconds per GB.
Chains of three and five nodes on the desktop, one link per hop shaped to 100 Mbit/s + 2 ms, traffic over one, two and four hops against kernel IP forwarding along the same chain. Shaped hops carry 0.94 to 0.95 of the bare chain, and each one adds a few hundredths of a millisecond.
| Chain of five, shaped | Hops | TCP Mbit/s | ÷ bare | UDP 1388 B Mbit/s | Forwarder CPU-s/GB | Added one-way p50 ms | One-way p99 ms |
|---|---|---|---|---|---|---|---|
| Bare forwarding | 1 / 2 / 4 | 95.6 / 95.6 / 95.6 | – | 97.1 / 97.1 / 97.1 | – | – | – |
| Atlas | 1 | 90.8 | 0.95 | 92.4 | – | +0.021 | 2.647 |
| Atlas | 2 | 90.1 | 0.94 | 91.7 | 7.0 | +0.036 | 5.787 |
| Atlas | 4 | 90.0 | 0.94 | 91.6 | 9.5 | +0.124 | 11.430 |
Added one-way latency is the p50 over the bare chain’s, unloaded. Unshaped, one TCP flow over the chain carries 3.24, 1.82 and 1.49 Gbit/s over one, two and four hops. A relay has two links, so it sends its forwarded datagrams without segmentation runs, and the p99 over four hops is 11.43 ms. The desktop runs chains of three and five, the server a chain of three.
On a Pi Zero 2 W, Atlas runs at the radio’s rate.
As deployed, unpinned on its four A53 cores, a Raspberry Pi Zero 2 W carries 90.3 Mbit/s of one TCP flow through Atlas over a radio link that carries 92.8 bare, 0.97 of it. Pinned to one core, as on the other tiers, it carries 80.5 Mbit/s, where kernel WireGuard carries 87.7. Memory fits with room to spare: under 40 MB per node.
| One radio link of 100 Mbit/s | TCP one flow, not pinned | TCP eight flows, not pinned | UDP 1388 B, not pinned | TCP one flow, one core | atlasd CPU-s/GB, not pinned / one core |
|---|---|---|---|---|---|
| Bare radio link | 92.8 | 95.1 | 97.1 | 94.4 | – |
| Kernel WireGuard | – | – | – | 87.7 | in kernel workers |
| Atlas | 90.3 | 89.9 | 92.4 | 80.5 | 191.2 / 138.4 |
| Atlas, encryption off | 89.4 | 91.8 | 93.4 | 88.5 | 172.1 / 109.0 |
| Atlas, offloads off ¹ | – | – | – | 60.8 | – / 179.6 |
Mbit/s, median of 3, over one 802.11s radio link shaped to 100 Mbit/s + 5 ms; UDP offered at 100. Kernel WireGuard was measured pinned. Unshaped and pinned, Atlas reaches 87.4 Mbit/s against WireGuard’s 102.0 and the radio’s 108.1, and with encryption off 103.7. On one pinned core eight flows carry less than one, 67.9 against 80.5; unpinned they carry the same. atlasd uses 106 µs per packet at the sender and 81 at the receiver pinned. ¹ The tunnel offload and the link’s segmentation offload switched off, run back to back with an as-shipped twin that carried 79.2 Mbit/s at 139.4 CPU-s/GB; with both off one flow also retransmits 475 times per window instead of none. Atlas adds 0.61 ms, half the round trip, WireGuard 0.28.
On a Radxa Zero 3W, the A55 cores set the rate, not the gigabit link.
On a Radxa Zero 3W, a Rockchip RK3566 with four Cortex-A55 cores at 1.8 GHz and 2 GB, receiving from the desktop over its USB 3 gigabit Ethernet adapter, one TCP flow through Atlas carries 300 Mbit/s as deployed, unpinned on its four cores, against 421 for kernel WireGuard and 710 for the bare link. atlasd’s per-packet path is the limit, not the link and not one core: no core is above 61 percent busy. Held to one A55 core with the USB controller’s interrupt on it, one flow carries 139.5 Mbit/s with the core 88–91 percent busy; with the tunnel offload and the link’s segmentation offload switched off it carries 68.5 and the core is saturated.
| One TCP flow into the Radxa Zero 3W, unshaped | Cortex-A55 core, pinned | Cortex-A55, not pinned, four cores |
|---|---|---|
| Bare link | 479 | 710 |
| Kernel WireGuard | 342 ¹ | 421 |
| Atlas | 139.5 | 300 |
| Atlas, offloads off | 68.5 | 200 |
| Atlas, encryption off | 236 | 388 |
| Atlas, end-to-end sealing | 107 | 211 |
| atlasd CPU-s/GB, as shipped / offloads off | 33.8 / 75.6 | 35.0 / 73.3 |
| atlasd per datagram received, µs | 50.8 | 52.0 |
| Cipher share of atlasd | 39 % | 32 % |
| What bounds it | the A55 core, 88–91 % busy | atlasd’s per-packet path; no core above 61 % |
Mbit/s, median of 3, one TCP flow from the desktop over the board’s USB 3 gigabit adapter, unshaped. CPU figures are the receiving Radxa’s own. ¹ Not a single-core figure: kernel WireGuard’s crypto workers run on unpinned CPUs. The adapter offloads nothing, so the bare link alone costs the board 1.5 cores at 710 Mbit/s. Offloads off is the same cell with the tunnel offload and the link’s segmentation offload switched off, run back to back with it. Shaped to 100 Mbit/s + 5 ms, one flow carries 89.1 Mbit/s on one A55 core, with the core saturated, at 64.5 CPU-s per GB, and 90.5 unpinned at 132.3, on 1.5 cores, against 95.5 bare and 91.1 with WireGuard. Atlas adds 0.55 ms, half the round trip, WireGuard 0.28. Per packet at the ceiling an A55 at 1.8 GHz is about 1.6 times as fast as the Pi Zero 2 W’s A53 at 1 GHz, 46 against 73 µs per 1364 B packet; an edge E-core does a datagram 7.6 times as fast, a P-core 14 times.
On a Beryl AX travel router, one receive path sets the rate.
A GL.iNet Beryl AX (GL-MT3000) Wi-Fi 6 travel router, a MediaTek MT7981B with two Cortex-A53 cores at 1.3 GHz and 512 MB, was measured in three roles: as an Atlas node receiving from the desktop over its 2.5 GbE WAN port, linked at gigabit; as the served-LAN gateway for a plain host on its LAN; and as a Wi-Fi 6 access point. As deployed, on its two cores, one TCP flow through Atlas carries 260 Mbit/s, against 564 for kernel WireGuard and 705 for the bare path, which the switched gigabit path sets, not the router. Held to one A53 it carries 228 Mbit/s with the core 81–86 percent busy. A 100 Mbit/s link it carries at 90.8 Mbit/s in both placements, with a quarter of one core to spare.
| One TCP flow into the Beryl AX | Cortex-A53, one core | Cortex-A53 ×2, not pinned, as deployed |
|---|---|---|
| Bare path | 678 | 705 |
| Kernel WireGuard | 550 ¹ | 564 |
| Atlas | 228 | 260 |
| Atlas, offloads off | 137 | 208 |
| Atlas, encryption off | 371 | 408 |
| Atlas, end-to-end sealing | 168 | 179 |
| atlasd CPU-s/GB, as shipped / offloads off | 25.7 / 45.0 | 29.0 / 44.4 |
| atlasd per datagram received, µs | 35.1 | 39.6 |
| Cipher share of atlasd | 41 % | 39 % |
| Shaped 100 Mbit/s + 5 ms: Mbit/s, atlasd CPU-s/GB | 90.8, 45.9 (core 77 %) | 90.8, 44.8 (0.83 cores) |
| Added latency on the shaped link, one way | none measurable ² | +0.27 ms |
| What bounds it | atlasd’s one worker, the core 81–86 % busy | one flow’s worker path; the busier core 74 % |
Mbit/s, median of 3, one TCP flow from the desktop over the router’s 2.5 GbE port linked at gigabit, unshaped unless marked. CPU figures are the receiving router’s own, and every figure includes the router’s own firewall, with connection tracking on every packet. ¹ Not a single-core figure: kernel WireGuard’s crypto workers run on both cores. ² 5.566 ms one way against 5.559 bare and 5.562 with WireGuard, half the round trip, the far end’s relay hop included. The bare path is the switched gigabit path, not the router: at 678 Mbit/s one A53 is 28 percent busy. Offloads off is the same cell with the tunnel offload and the link’s segmentation offload switched off, run back to back with it: on one A53 the two are worth 1.66 times one flow and 43 percent of the CPU per GB, on two cores 1.25 times and 35 percent. At 100 Mbit/s a byte costs 1.8 times what it costs at the ceiling, because the receiver coalesces 7.3 segments per tunnel write there against 31. Per packet one A53 at 1.3 GHz is 2.1 times as fast as the Pi Zero 2 W’s A53 at 1 GHz, 35.1 against 73 µs per 1364 B packet, and 1.3 times one A55 of the Radxa Zero 3W at 1.8 GHz (46.1).
The router’s Ethernet controller has one receive queue, and OpenWrt’s default packet steering runs the receive softirq on one core whichever core takes the interrupt. One tunnel flow is one stream of datagrams on one socket, handled by one data-path worker at a time: not pinned, the second worker shares the same flow’s work, two workers at about 47 percent each, instead of carrying a flow of its own. So the second core adds 14 percent, 228 to 260 Mbit/s, at 13 percent more CPU per byte. The Radxa Zero 3W gains 2.15 times from four cores because its pinned core also carried its USB adapter’s software receive path, which unpinning moved away; on the Beryl AX that softirq is 5.6 percent of a core, so there is little for a second core to take. On this board pinning gains nothing and needs root and a custom service: run atlasd as deployed. Not measured: a second peer or link, which would give the second worker a flow of its own.
The SoC’s own Ethernet controller verifies and fills checksums, gathers fragments and segments in hardware; the Radxa Zero 3W’s USB adapter does none of it. Measured: the receive softirq costs about 12 percent of a core per Gbit/s of bare traffic here against about 115 percent on the Radxa, and one A53 at 28 percent carries the bare path where the Radxa’s takes 70 percent of an A55. atlasd’s own time per datagram is lower too, 35 against 51 µs, on a core with 0.72 of the clock: likely because the controller does the checksums that the Radxa’s kernel computes in software, partly in atlasd’s own system time. That part is an inference, not a measurement: no profile split atlasd’s system time, and the router’s build runs a different memory allocator, musl’s, which was not isolated either. The same offloads let one pinned A53 receive a bare UDP flood at 790 Mbit/s, where the Radxa’s pinned A55 livelocked.
Atlas’s wire format fixes the cipher at ChaCha20-Poly1305. On 64-bit Arm ChaCha20 runs on NEON, which the in-order A53 executes as two 64-bit halves per 128-bit operation, and Poly1305 runs as portable scalar code: together they are 41 percent of atlasd’s CPU on one A53 and 39 percent on two. Encryption off carries 371 and 408 Mbit/s, which bounds what any faster cipher code could give on this board. The A53s have the ARMv8 cryptography extensions, but using them would mean a different cipher, a change of the wire format, and Atlas offers none. Kernel WireGuard, with the same cipher, carries 550–564 Mbit/s here: the kernel’s arm64 assembly ChaCha20 and Poly1305 on both cores, and no crossing into user space.
The served-LAN gateway
With [[serve]] the router serves its LAN into the tunnel without masquerade, so the far end sees the LAN host’s own address; a plain host behind it, on the router’s gigabit LAN port, ran nothing of Atlas. This measures the forwarding ceiling for one plain LAN host, against plain routing with NAT and kernel WireGuard on the same board in the same pass.
| One TCP flow through the router, unshaped | One A53: Mbit/s | atlasd CPU-s/GB | Not pinned: Mbit/s | atlasd CPU-s/GB | Board busy, not pinned |
|---|---|---|---|---|---|
| Downstream, far end to LAN host: routed with NAT, no tunnel | 666 | – | 700 | – | 0.61 cores |
| Downstream: kernel WireGuard | 470 ¹ | – | 464 | – | 1.49 cores |
| Downstream: Atlas served LAN | 201 (core 87 %) | 29.9 | 214 | 32.8 | 1.12 cores |
| Upstream, LAN host to far end: routed with NAT, no tunnel | 711 | – | 732 | – | 0.67 cores |
| Upstream: kernel WireGuard | 436 ¹ | – | 442 | – | 1.66 cores |
| Upstream: Atlas served LAN | 146 (core 100 %) | 46.8 | 175 | 53.3 | 1.52 cores |
| Atlas served LAN, WAN shaped to 100 Mbit/s + 5 ms, down / up | 90.8 / 90.4 | 43.6 / 57.7 | 90.8 / 90.6 | 45.6 / 68.5 | 0.78 / 1.16 cores |
| For comparison: the router as the endpoint, unshaped | 228 | 25.7 | 260 | 29.0 | – |
Median of 3, one TCP flow, the LAN port linked at gigabit. Routed with NAT runs at the rate of the switched path between the test machines, not the router’s: at 666 Mbit/s one A53 is 32 percent busy. ¹ Not a single-core figure: kernel WireGuard’s crypto workers run on both cores. Downstream, the ceiling is one receive worker, as when the router terminates: 201 Mbit/s on one A53 and 214 as deployed, 0.88 and 0.82 of the 228 / 260 it carries as the endpoint, at 29.9 / 32.8 atlasd CPU-s per GB, 13–16 percent more per byte than terminating. Measured: atlasd’s coalesced tunnel writes carry 14.7 segments each (14.1 not pinned) against 31 / 23 when the router terminates, so it makes about twice the writes per byte; likely the difference, not separated. Upstream is the expensive direction: one A53 is saturated at 146 Mbit/s, and as deployed the board carries 175 Mbit/s on 1.52 of its two cores, at 46.8 / 53.3 CPU-s per GB, about 1.6 times the downstream cost. The LAN port’s receive offload does merge the host’s frames, but OpenWrt’s default fraglist GRO builds frame lists, which a tunnel interface cannot take, so the kernel splits them again and the host’s segments reach the tunnel one per read. That is not the limit: with fraglist GRO switched off as a diagnostic, the tunnel reads 7 segments at a time, yet goodput moves only from 146 to 153 Mbit/s at the same CPU per byte. Sealing every datagram is the upstream cost, as for the router’s own traffic below; an inference, no profile split atlasd’s send path. OpenWrt’s default stays: with fraglist GRO off, plain UDP routing saturates the core. Through the router Atlas adds 0.28 ms one way to the LAN host’s path, 2.806 ms round trip against 2.248 routed and 2.253 with WireGuard, in both placements. Eight flows carry no more than one, and atlasd stays at 18.1–32.2 MiB resident. The router’s served-LAN counters show the host’s traffic forwarded and returned with nothing refused in every window.
The Wi-Fi 6 access point as a bearer
The router’s 5 GHz radio as an access point for a laptop’s two-stream Wi-Fi 6 adapter: HE, 80 MHz, two spatial streams at MCS 9–11, 960–1,200 Mbit/s PHY in both directions. The adapter sits on a USB 2.0 hub, which most likely limits the bare link, not the router’s radio; no cell isolated the station. The router’s own Wi-Fi 6 capacity is not measured here: these figures are a lower bound and what Atlas costs on top of it.
| One TCP flow over the radio, nothing pinned | Router to station | Station to router |
|---|---|---|
| Bare Wi-Fi link | 181 | 242 |
| Atlas over the link | 170 | 226 |
| Atlas ÷ bare | 0.94 | 0.93 |
| Router busy over idle, Atlas / bare | 1.30 / 0.40 cores | 1.41 / 0.56 cores |
| Router atlasd CPU-s/GB | 47.1 | 33.7 |
| Shaped 100 Mbit/s + 5 ms: bare / Atlas | 95.6 / 90.8 | 95.6 / 90.8 |
| Added latency, one way, unshaped | +0.38 ms | +0.28 ms |
Mbit/s, median of 3; signal −51 to −54 dBm at both ends; added latency is half the round trip over the bare radio link’s. Atlas carries 0.93–0.94 of the bare radio link, at 1.30–1.41 of the router’s two cores: close to the router’s own Atlas ceiling of 260 Mbit/s wired, so with a faster station the router’s processor, not its radio, would become the limit, at about 230–260 Mbit/s of Atlas. Sending its own traffic costs the router 47.1 CPU-s per GB, with the tunnel handing it TCP superpackets to split, seal and send in runs; a forwarded LAN upload, one segment per read, costs the same, 46.8 (above): sealing into the tunnel is the expensive direction either way. Observed, not explained: receiving over Wi-Fi the router wrote no coalesced segments into its tunnel, where the wired path coalesces 31 per write; Atlas still carried 0.93 of bare.
On a cloud VM, one vCPU sets the rate.
The cloud end of a deployment, a hub, a relay server or the far end of a site’s tunnel, runs on rented vCPUs billed by the hour. Five Google Compute Engine machine types, from a shared-core e2-medium to a c4-standard-4 on Emerald Rapids, carry one TCP flow through Atlas between two VMs of one zone at 1.02–2.10 Gbit/s on one vCPU and at 0.62–1.91 not pinned, as deployed: as shipped, with no configuration, each node sizing its tunnel MTU from the path. One vCPU at one end of the flow is the limit every time. Per dollar the 2-vCPU shapes lead, because one flow uses one vCPU. Every shape carries a 100 Mbit/s link at 90.5–90.6 Mbit/s, and across the Atlantic Atlas carries 0.89 of the bare flow.
| One TCP flow between two VMs, unshaped | Atlas Mbit/s, as shipped | Both offloads off ² | Encryption off ² | e2e | WireGuard 1380 ² | Bare | Receiver atlasd CPU-s/GB | Per billed vCPU | Per USD/h of one VM |
|---|---|---|---|---|---|---|---|---|---|
| Cloud VM, e2-medium, Broadwell vCPU (shared core) | 1,018 | 326.8 | 1,126 | 644.7 | 1,220 | 1,962 | 6.98 | 508.8 | 27,503 |
| Cloud VM, e2-medium, Broadwell, shared core, not pinned (as deployed) | 622.0 | 510.0 | 524.2 | 483.4 | – | 1,962 | 10.86 | 311.0 | 16,811 |
| Cloud VM, t2a-standard-2, Neoverse-N1 vCPU | 1,058 | 508.4 | 1,834 | 668.9 | 2,218 | 9,889 | 7.04 | 529.2 | 12,453 |
| Cloud VM, t2a-standard-2, Neoverse-N1, not pinned (as deployed) | 1,055 | 796.0 | 1,872 | 678.4 | – | 9,890 | 7.72 | 527.7 | 12,416 |
| Cloud VM, n2-standard-2, Cascade Lake vCPU | 1,216 | 678.8 | 2,230 | 730.1 | 2,234 | 9,890 | 5.68 | 608.1 | 11,367 |
| Cloud VM, n2-standard-2, Cascade Lake, not pinned (as deployed) | 1,061 | 782.9 | 1,894 | 646.4 | – | 9,871 | 7.54 | 530.4 | 9,915 |
| Cloud VM, c3-standard-4, Sapphire Rapids vCPU | 1,583 | 1,025 | 3,211 | 932.3 | 5,021 | 21,745 | 4.19 | 395.8 | 7,196 |
| Cloud VM, c3-standard-4, Sapphire Rapids, not pinned (as deployed) | 1,377 | 1,168 | 2,598 | 859.7 | – | 21,749 | 4.38 | 344.2 | 6,258 |
| Cloud VM, c4-standard-4, Emerald Rapids vCPU | 2,099 | 1,371 | 4,324 | 1,262 | 5,729 | 21,969 | 3.64 | 524.6 | 9,124 ¹ |
| Cloud VM, c4-standard-4, Emerald Rapids, not pinned (as deployed) | 1,912 | 1,546 | 3,482 | 1,254 | – | 21,969 | 4.24 | 478.0 | 8,313 ¹ |
Mbit/s, median of 3, one TCP flow between two VMs of the same machine type in europe-west4-a. Rows by on-demand hourly price, lowest first; the labels without a placement are one pinned vCPU, every bench process and the NIC’s interrupts on it and its hyperthread sibling idle; not pinned is as deployed, one data-path worker per vCPU. As shipped, with no setting, every node sizes its tunnel MTU from the path: 1376 B on the provider’s 1460 B network and 1309 with e2e, the same on relay links, so no tunnel packet is larger than the path and nothing fragments. ² Measured with the tunnel MTU set by hand (1388), both cells of a pair at the same MTU: both offloads off is the tunnel offload and the link’s segmentation offload switched off, encryption off the same cell without the cipher, and WireGuard 1380 kernel WireGuard at the MTU wg-quick picks on a 1460 B route, in the one-vCPU placement only, its crypto on every vCPU of the VM. Bare is the VM’s egress cap, not a CPU figure. Receiver CPU: the receiving VM’s atlasd per GB delivered. Per billed vCPU: Atlas ÷ the VM’s vCPU count; per USD/h: one VM’s goodput ÷ its hourly price, the provider’s on-demand list of 2026-11: e2-medium 0.037, t2a-standard-2 0.085, n2-standard-2 0.107, c3-standard-4 0.22 and c4-standard-4 0.23 USD/h. ¹ The c4-standard-4 price is estimated: it is not on the list.
In every cell one end’s vCPU is 96–100 percent busy with its hyperthread sibling idle, the sender’s on n2 and c3, the receiver’s on t2a, c4 and e2-medium, and the NIC never comes near its cap: the bare path carries 1.96, 9.89 and 21.7–22.0 Gbit/s. The receiving atlasd spends 3.6–7.0 CPU-seconds per GB and 4.8–9.4 µs per received datagram, the newest core, c4, the cheapest. Kernel WireGuard at 1380 carries 1.22–5.73 Gbit/s with its crypto spread over every vCPU of the VM, where Atlas’s one flow is one data-path worker’s, so the cloud pairs have no like-for-like WireGuard cell. Eight flows carry no more than one: 0.97–1.99 Gbit/s on one vCPU, and not pinned 1.04 and 1.46 against 1.38 and 1.91 on c3 and c4.
One TCP flow is one stream of datagrams through one tunnel queue and one link socket, handled in order, one step at a time. Not pinned, the data-path workers take turns on it, the same work over two to four vCPUs and never in parallel, so atlasd spends 5–33 percent more CPU per byte for up to 13 percent less goodput, and on e2-medium’s shared core 56 percent more for 39 percent less. On the Intel shapes a vCPU is a hyperthread: pinned, the flow’s vCPU ran on a whole physical core with its sibling idle, and not pinned on an n2-standard-2 the two workers share that one core. On the t2a every vCPU is a core, and not pinned carries the same, 1,055 against 1,058 Mbit/s. Leave atlasd unpinned, as deployed: the pinned figure is the per-core ceiling, the unpinned one what an install as deployed carries. Choose the shape by the number of flows or link sockets a node carries at once, not by vCPUs alone; the sizing section below does that for hubs and relays.
On the x86 shapes ChaCha20-Poly1305 is 53–56 percent of the receiving atlasd at one vCPU’s ceiling, n2 53, c3 54 and c4 56 percent, and encryption off doubles one flow: n2 2.23, c3 3.21, c4 4.32 Gbit/s. On e2-medium’s Broadwell it is 39 percent, and on the Neoverse-N1 44 percent, the NEON ChaCha20 with a portable Poly1305 on a core that is slower per packet, 9.37 µs per datagram against the n2’s 7.56. The tunnel and link offloads are worth most where a packet is expensive: 2.12 times one flow on the Arm core, 1.6–1.8 times on one Intel vCPU and 2.78 times on e2-medium; they work on every NIC here because the segmentation is the kernel’s. These shares and ratios were measured with the tunnel MTU set by hand (1388), both cells of each pair at the same MTU. End-to-end sealing as shipped carries 0.6 of what Atlas carries without it on one vCPU, a second pass of the cipher that is already half the cost.
Two vCPUs on half a physical core, Broadwell at 2.20 GHz, the cheapest shape. Pinned, one flow carries 1,018 Mbit/s (959–1,031) at 6.98 receiver CPU-seconds per GB, the best figure per dollar of the five, and 12 percent more than other e2-medium VMs carried with the tunnel MTU set by hand (908): a shared core moves with the host it lands on. Its cells vary run to run, not pinned one flow 602–680 Mbit/s and e2e 236–506, and the latency rises: 0.30 ms one way unloaded in both placements and 0.28–0.29 ms on a shaped link, where kernel WireGuard on e2-medium stays near bare, 0.68–0.70 ms round trip against 0.64–0.66. A user-space daemon that sleeps between packets waits for its vCPU to be scheduled again on a shared core; that is inferred from the latency and the variance, not measured: steal reads 0.2 percent at most. For steady figures, choose a dedicated shape.
One shaped link
The VM’s own NIC shaped to 100 Mbit/s with 5 ms of delay on both VMs, Atlas as shipped: every machine type carries the link’s rate, and what changes is what a byte costs the receiving VM.
| Shaped 100 Mbit/s + 5 ms, one vCPU / not pinned | Atlas Mbit/s | Receiver atlasd CPU-s/GB | Receiving VM, CPU-s/GB | Both offloads off, atlasd CPU-s/GB ¹ | Added latency, one way |
|---|---|---|---|---|---|
| Cloud VM, e2-medium, Broadwell vCPU (shared core) | 90.6 / 90.6 | 19.6 / 19.5 | 26.7 / 30.7 | 32.1 / 45.3 | +0.28 / +0.29 ms |
| Cloud VM, t2a-standard-2, Neoverse-N1 vCPU | 90.5 / 90.5 | 11.7 / 13.5 | 17.0 / 17.8 | 26.8 / 29.2 | none / none |
| Cloud VM, n2-standard-2, Cascade Lake vCPU | 90.5 / 90.6 | 13.9 / 15.6 | 18.5 / 26.2 | 22.5 / 32.5 | none / +0.28 ms |
| Cloud VM, c3-standard-4, Sapphire Rapids vCPU | 90.5 / 90.5 | 6.5 / 7.4 | 8.7 / 12.7 | 13.1 / 17.1 | none / none |
| Cloud VM, c4-standard-4, Emerald Rapids vCPU | 90.5 / 90.5 | 5.7 / 6.7 | 7.9 / 8.1 | 9.1 / 12.5 | none / none |
Median of 3, one TCP flow; each cell gives one pinned vCPU / not pinned. Bare carries 95.3–95.5 Mbit/s and eight flows through Atlas 90.4–90.8. Added latency is half the unloaded probe’s round trip over the bare link’s. A byte costs 1.6–2.8 times what it costs at the ceiling, because fewer segments arrive together per tunnel write at 100 Mbit/s. Pinned, the dedicated shapes add no measurable latency, within 0.01 ms of bare; n2 not pinned adds 0.28 ms one way, as the Beryl AX does not pinned (0.27): the two workers hand the flow between them; e2-medium’s shared core adds 0.28–0.29 ms in both placements. ¹ Measured with the tunnel MTU set by hand (1388), both cells of the pair at the same MTU: there the offloads-on twin read 18.0 / 20.1, 12.0 / 13.2, 14.3 / 15.7, 6.5 / 7.6 and 4.8 / 5.8 CPU-s/GB, so with both offloads off the link is carried at the same 90.6 Mbit/s at 1.5–2.3 times the CPU per byte; in that pass paced UDP carried 92.2–92.3 Mbit/s.
Across the Atlantic
An n2-standard-2 pair, one VM in europe-west4 in the Netherlands and one in us-east4 in northern Virginia, not pinned, one TCP flow over the provider’s backbone, the one-way delay measured on both clocks, disciplined by the provider’s time service.
| One TCP flow, europe-west4 ↔ us-east4 | Mbit/s | RTT p50 ms | One way eu → us, ms | One way us → eu, ms |
|---|---|---|---|---|
| Bare path | 287 (277–290) | 87.99 | 43.70 | 43.94 |
| Kernel WireGuard, MTU 1380 ¹ | 195 (195–195) | 88.48 | 44.31 | 43.86 |
| Kernel WireGuard, MTU 1420 (fragmented) ¹ | 0.5 | 88.52 | 44.29 | 43.86 |
| Atlas, as shipped (tunnel MTU 1376) | 256 (255–256) | 88.46 | 43.77 | 44.42 |
| Atlas, e2e, as shipped (tunnel MTU 1309) | 253 (252–253) | 88.53 | 43.80 | 44.44 |
10-second windows, median of 3 (min–max). The one-way figures are good to about 0.1 ms. The path drifts during the pass: bare’s own three runs read 87.38, 87.99 and 88.04 ms round trip and Atlas’s runs after them 88.45–88.54, so Atlas’s 0.5 ms over bare’s median is within that drift and not separated from it; against bare’s last run Atlas adds 0.4 ms round trip, 0.1 ms us → eu. Atlas carries 0.89 of the bare flow, which the default TCP receive buffer itself holds: 6 MiB at 88 ms is about 300 Mbit/s for one flow. With the tunnel MTU set by hand Atlas carried 256 Mbit/s too, at 87.95 ms round trip against bare’s 88.03 on a steadier path, at most 0.2 ms one way. ¹ From that pass, on the same regions and machine type: kernel WireGuard at 1380 carried 0.69 of its bare flow (283 Mbit/s); why it carries less than Atlas here was not isolated (one flow, one pass). At its 1420 default kernel WireGuard fragments every packet on the 1460 B network, and across regions its fragments carried 0.5 Mbit/s.
Sizing a relay deployment.
A relay server carries the sessions that cannot take a direct path, and a hub is one Atlas node many sites connect to. One relay server was measured on an n2-standard-2, -4 and -8 (Cascade Lake) and on a c3-standard-8 (Sapphire Rapids) with 8, 32, 128 and 256 sessions from five client VMs in its zone, each node in its own network namespace behind NAT, so that the two nodes of a pair reach each other only through the relay: thin telemetry-like sessions, keepalive-only ones, 5 Mbit/s video-like TCP sessions and unthrottled ones, on one and two relay addresses, with e2e, in ramps and in two storms. The relay runs out of packets, not sessions: one relay address forwards a fixed number of packets per second whatever its session count. The figures turn into a method: the packets each session puts through the relay, how to count addresses and VMs, the headroom for bursts and for the loss of a relay, and what it costs. A hub’s link socket takes about 2 Gbit/s in, whatever the VM’s size.
| What one relay address forwards | USD/h | One address, packets/s | CPU per packet | Two addresses, packets/s | Goodput at the ceiling, 256 fat sessions, one / two addresses |
|---|---|---|---|---|---|
| Relay VM, n2-standard-2, Cascade Lake | 0.107 | 64,000 | 20.2 µs | 115,000 ¹ | 287 / 496 Mbit/s |
| Relay VM, n2-standard-4, Cascade Lake | 0.214 ³ | 72,000 | 17.4 µs | 121,000 | 326 / 530 |
| Relay VM, n2-standard-8, Cascade Lake | 0.428 ³ | 80,000 | 15.4 µs | 139,000 | 346 / 615 |
| Relay VM, c3-standard-8, Sapphire Rapids | 0.44 | 119,000 | 10.0 µs | ≥ 220,000 ² | 530 / 948 |
Median of 3, 20-second windows, every daemon restarted per run. Packets per second as the relay VM’s NIC receives them, each forwarded once; CPU per packet is the relay daemon’s CPU time over the packets it receives, its two system calls included. Fat sessions: one 5 Mbit/s TCP flow per session, both nodes of each pair sending, so the data and the far end’s acknowledgements cross the relay, the mean relayed datagram 700–720 B; one-way bulk carries about twice the goodput at the same packet rate. The nodes as shipped: tunnel MTU 1376 on the relay links, 1309 with e2e. The n2 relays use virtio-net and the c3-standard-8 gVNIC, part of its per-packet lead. ¹ The VM’s two vCPUs full. ² A lower bound: the client VMs were near their own limit. ³ Estimated, not on the provider’s list: twice and four times the n2-standard-2’s price.
On one address the forwarded packet rate is flat from 8 to 256 sessions: on an n2-standard-2, 60,000, 62,000, 64,000 and 63,000 packets per second at 8, 32, 128 and 256 unthrottled sessions. Fat 5 Mbit/s sessions are carried in full to 32, 159.6 of 160 Mbit/s, and flatten between 32 and 128. One receive loop per address does the work, receiving a datagram, decrypting the client’s envelope, re-encrypting it for the destination’s session and sending it, one datagram at a time: more vCPUs add 12–25 percent, a faster core 1.9 times and a second address about 1.8 times. Per USD/h the cheapest packets are a 2-vCPU VM with two addresses, about a million packets per second per USD/h.
256 telemetry sessions, 200 B at 50 packets per second each, take about 0.3 of a vCPU, lose 0 of 294,400 datagrams and add 0.03–0.22 ms of round trip over the bare path; 256 keepalive-only sessions take 0.07–0.08 of a vCPU.
About 2.8 KiB per session: the relay holds 5.9 MiB fresh and 6.6 MiB with 256 sessions. 256 nodes started at once are admitted at 640–1,180 sessions per second, as fast as the clients start: every session within 0.03–0.07 s of the last node’s start, at most 16 percent of one vCPU. With 256 sessions every session is back 7.3–7.8 s after a relay restart and 3.2–3.6 s after a 15-second cut of every client’s link, none lost, the relay’s CPU peaking at 6–13 percent of a vCPU: the clients’ loss detection and retry set those times, not the relay.
End-to-end sealing costs the relay nothing: the same packet ceiling and the same CPU per packet. The goodput is a little lower at the same packet rate, each datagram carrying 67 bytes less of payload.
The relay runs on the kernel’s default socket receive buffer (208 KiB on these VMs). A burst that arrives faster than one address drains it is dropped even when the average is well below the ceiling: 8 fat sessions, 40 Mbit/s in all, lose 3,500–8,000 datagrams per 20 s window (0.3–1 percent of probe pings). Past the ceiling every session on the address loses 11–19 percent at 1.7–2.9 ms round trip. Size below the ceiling; bursts above it are dropped.
| Relay server, c3-standard-8, one one-way TCP flow per relayed path, tunnel MTU set by hand | Forwarded Mbit/s | Per USD/h | Relay daemon, vCPUs | Relay CPU-s per GB | RSS MiB |
|---|---|---|---|---|---|
| One address, 1 relayed path (2 sessions) | 640 (632–644) | 1,454 | 0.65 | 8.10 | 5.9 |
| One address, 2 paths (4 sessions) | 674 (667–675) | 1,532 | 0.74 | 8.80 | 6.0 |
| One address, 4 paths (8 sessions) | 724 (722–725) | 1,646 | 0.82 | 9.01 | 6.0 |
| One address, 4 paths, e2e on | 734 (733–734) | 1,668 | 0.88 | 9.59 | 6.0 |
| Two addresses, 4 paths (8 sessions) | 1,005 (992–1,011) | 2,284 | 1.12 | 8.93 | 6.0 |
An earlier pass, measured with the tunnel MTU set by hand (1310 on the relay links): median of 3 (min–max), 20-second windows, every daemon restarted per run; a relayed path between two clients is two sessions on the relay. One one-way TCP flow per path sends full-size datagrams, so the same address carries about twice the goodput of the two-way sessions above at a similar packet rate: the same ceiling, counted in bytes. A second address took it from 724 to 1,005 Mbit/s, 1.39 times, on 1.12 vCPUs; with end-to-end sealing the relay forwarded the sealed payload at no extra cost; memory 6.0 MiB (6.3 MB) with 8 sessions. These byte figures hold for one-way bulk only: size a relay by packets per second, the table above.
| One atlasd as a hub, one TCP flow per peer | Mbit/s | atlasd, vCPUs | Whole VM, vCPUs | RSS MiB |
|---|---|---|---|---|
| c3-standard-8: 1 peer → hub (the peer’s limit) | 1,111 | 0.61 | 0.70 | 26.3 |
| c3-standard-8: 4 / 8 peers → hub | 2,003 / 1,948 | 1.29 / 1.34 | 1.90 / 2.08 | 27.3 / 27.5 |
| c3-standard-8: 8 peers → hub, atlasd on 4 vCPUs | 1,883 | 1.31 | 2.00 | 26.5 |
| c3-standard-8: 8 peers → hub, two link sockets | 3,143 | 2.91 | 3.58 | 28.7 |
| c3-standard-8: hub → 8 peers | 1,050 | 2.01 | 2.16 | 41.9 |
| c3-standard-8: kernel WireGuard hub, 8 peers → hub | 11,431 | – | 7.21 | – |
| t2a-standard-8: 4 / 8 peers → hub | 1,233 / 1,151 | 1.24 / 1.26 | 1.49 / 1.61 | 26.4 / 26.8 |
| t2a-standard-8: 8 peers → hub, two link sockets | 1,903 | 2.81 | 3.32 | 28.1 |
| t2a-standard-8: hub → 8 peers | 800 | 2.18 | 2.36 | 39.0 |
| t2a-standard-8: kernel WireGuard hub, 8 peers → hub | 8,224 | – | 6.70 | – |
Median of 3; the hub one atlasd, measured with the tunnel MTU set by hand (1388); the t2a-standard-8 (8 Neoverse-N1 cores) at an estimated 0.34 USD/h. A hub takes no more from 8 peers than from 4: every peer’s datagrams arrive on the hub’s one link socket, one [[link]], drained by one task, and at 2 Gbit/s its receive buffer overflows; pinned to 4 vCPUs it takes the same, the extra vCPUs never used. A second [[link]] on another port, the peers split between them, takes 3.1 Gbit/s on the c3, 1.61 times one, and 1.9 on the t2a. Out of the hub the ceiling is about 1.05 Gbit/s on the c3 and 0.8 on the t2a, with one link socket or two. Kernel WireGuard spreads its crypto over every vCPU of the VM: per GB of the whole VM the Atlas hub costs 8.5 CPU-seconds against WireGuard’s 5.0 on the c3.
[[link]] sockets; above that, more hubs. Memory: 26–29 MiB taking traffic in, 39–42 MiB sending out to 4–8 peers.The method, in nine steps
Count what each session puts through the relay, as measured: a TCP stream, video or bulk, about 234 packets per second per Mbit/s, its data and the far end’s acknowledgements, so 1,872 for 8 Mbit/s and 2,808 for 12; a telemetry or control session its own packet rate plus about 8 for the link’s probes and keepalives, 58 for 50 packets per second; a keepalive-only session about 9.
The sum over the fleet of sessions × packets per session. Unless direct paths are certain, plan for every path relayed, which is what a site behind two layers of NAT, or a direct path that fails, puts on the relay. A relayed path is two sessions on the relay: count each stream once, at its sender.
Video and bulk × 1.5, for keyframes, a second stream on demand and TCP probing for more; telemetry × 1. Keep each address under about 70 percent of its measured ceiling: bursts drop earlier.
The busy hour’s packets per second ÷ (0.7 × the shape’s per-address ceiling), rounded up. Two addresses on one VM give the two-address ceiling, 1.7–1.8 times one, not twice; at most two per VM were measured, and on an n2-standard-2 two fill both vCPUs.
The relayed Mbit/s against what an address carried at its ceiling, 287–352 Mbit/s of two-way TCP on an n2-standard-2 and 530–577 on a c3-standard-8, one-way bulk about twice that, and against the VM’s egress cap. Session count and memory are not a constraint: 2.8 KiB a session.
The loss of one relay must not overload the rest: with N relays, the busy hour must fit N − 1 of them. With two relays, each carries the whole load alone.
Put relays in the regions their nodes are in: a relayed path takes the path’s latency. Europe to the US east is 88 ms round trip, and Atlas adds at most 0.2 ms one way. One TCP flow at 88 ms with default buffers stops near 256 Mbit/s: a single fast flow across an ocean needs larger TCP buffers, not a larger relay.
Nothing to set: as shipped, the node sizes it from the path, 1376 on a 1460 B cloud network and 1309 with e2e, relay links too. A tunnel MTU set by hand still works and is kept as set.
VMs × the hourly price × 730 hours a month, plus an external IPv4 per relay address at 0.005 USD/h and a 10 GB boot disk at 0.0014 USD/h. Then egress: every byte a relay forwards leaves its VM, and the provider bills it per GB by network tier and destination; the examples below assume 0.08 USD/GB for the arithmetic, not a quote. At 100 Mbit/s a relay sends 45 GB an hour; at a deployment’s scale egress, not the VMs, is the larger cost line.
The per-address ceiling is a per-core figure: one receive loop works one datagram at a time, so a faster core raises it and more vCPUs barely do. On one address the c3-standard-8’s Sapphire Rapids forwards 1.9 times what an n2-standard-2 does and 1.5 times an n2-standard-8, which has the same vCPU count. As measured, the cheapest packets are an n2-standard-2 with two addresses, about a million packets per second per USD/h; the most per VM, a c3-standard-8.
Three worked examples
Three fleets sized by the method, every path relayed, the worst case; each node assumed by two figures, its video rate and its telemetry packet rate; the busy hour on the mean; prices from the provider’s on-demand list of 2026-11, europe-west4, and us-east4 for the US VMs.
| Step | 1. One region, 12 nodes | 2. 40 nodes, failover | 3. Two regions, 30 nodes |
|---|---|---|---|
| Fleet (assumed) | 12 nodes, each one 8 Mbit/s video stream and 50 packets per second of telemetry to one operations centre | 40 nodes, each one 12 Mbit/s video stream and 50 packets per second of telemetry to one operations centre | 20 nodes in Europe and 10 in the US east, each one 8 Mbit/s video stream and 50 packets per second of telemetry to its region’s operations centre, plus 6 of the European nodes’ video, 8 Mbit/s each, also to the US centre, relayed in Europe |
| Relayed packets per second, mean | Video 12 × 1,872 = 22,464, telemetry 12 × 58 = 696: 23,200 | Video 40 × 2,808 = 112,320, telemetry 2,320: 114,600 | Europe: video (20 + 6) × 1,872 = 48,672, telemetry 1,160: 49,800. US: 18,720 + 580: 19,300 |
| Busy hour | 22,464 × 1.5 + 696 = 34,400 | 170,800 | Europe 74,200, US 28,700 |
| Addresses and shape | 54 % of one n2-standard-2 address (64,000), under 70 %: one address | Two relays cannot carry it on a survivor, even with two c3-standard-8 addresses: 78 % of ≥ 220,000. n2-standard-2 with two addresses (115,000) | Europe: 65 % of an n2-standard-2 with two addresses (115,000). US: 45 % of one n2-standard-2 address |
| Relays, N + 1 | 2 × n2-standard-2, one address each, either carrying all | 4 × n2-standard-2, two addresses each: any three carry 56,900 each, 49 % of the two-address ceiling; in normal operation 42,700 each (37 %) | Europe 2 × n2-standard-2, two addresses each; US 2 × n2-standard-2 in us-east4 (0.110 USD/h), one address each |
| Bytes, the second check | 97 Mbit/s mean, 145 busy, far below the 287–352 an address carries | On a survivor of a failure 161 Mbit/s mean, 241 busy, against 496–551 | Europe 210 Mbit/s mean (314 busy) against 496–551; US 81 (121) against 287–352 |
| VMs per month | 2 × 0.1134 × 730 = 166 USD | 4 × 0.1184 × 730 = 346 USD | 2 × 0.1184 × 730 + 2 × 0.1164 × 730 = 343 USD |
| Alternatives | – | 3 × n2-standard-4 with two addresses each: 71 % on a survivor, over the 70 % line, 494 USD; 3 × c3-standard-8 with two addresses each: 989 USD | – |
| Egress per operating hour | 43.6 GB | 217 GB | 131 GB, 21.6 GB of it across regions |
| Egress at an assumed 0.08 USD/GB | 3.49 USD/h; 614 USD for 176 h a month | 17.40 USD/h; 3,062 USD for 176 h | 10.45 USD/h; 1,840 USD for 176 h |
Per node: 8 Mbit/s of video is 1,872 packets per second through the relay, 12 Mbit/s 2,808, and 50 packets per second of telemetry 58. Hourly rates: the VM’s on-demand price plus 0.005 USD/h for each relay address’s external IPv4 and 0.0014 USD/h for a 10 GB boot disk, so 0.1134 for an n2-standard-2 with one address, 0.1184 with two and 0.1164 in us-east4 with one; a month is 730 hours. 176 operating hours are 8 hours a day on 22 days; egress scales with the hours the fleet actually sends. The per-GB rate is an assumption for the arithmetic, the cross-region transfer price, not a quote: internet egress is billed by the provider’s network tier and destination, and is often higher. In the first example failover alone sets the count. In the second, at this scale egress costs about nine times the VMs. In the third, a European node’s video to the US centre is relayed in Europe and arrives in 44 ms one way, the path’s own latency.
At most 256 sessions on one relay, at most two addresses per relay VM and one flow per session; the c3-standard-8 at 256 sessions only, and the hub with at most 8 peers, one TCP flow each. A mixed cell, thin sessions beside saturating fat ones on one address, was not measured: the loss figures imply the thin ones would see the loss. Sessions crossing regions through one relay were not measured. The relay server runs on x86 only, its build being x86_64; the hub ran on Arm too. WAN links slower than the relay are not modelled: every client sat in the relay’s zone, and a field node’s own uplink, radio, LTE or satellite, sets its paths’ rates and is the first limit in practice. The per-address and per-socket ceilings are those of the current data path: a relay address’s one receive loop and a hub’s one link socket per [[link]]. One cloud, one zone, and the relay on a static configuration, without its control plane or federation.
On a phone, its own radio sets the rate, and receiving is the hard direction.
A Pixel 10 Pro XL (Google Tensor G5, Android 17) runs the Android app as the measured endpoint over its own Wi-Fi 6 radio, screen off and on battery, against the desktop through a Beryl AX access point. One TCP flow through Atlas carries 0.68 of the bare radio link downstream, 309 against 453 Mbit/s, and 0.88 upstream, 319 against 363; with end-to-end sealing about 10 percent less downstream. A 100 Mbit/s link it carries at 90.7 and 91.1 Mbit/s for 1.5 and 2.5 W of the whole phone, and in two hours of tests the phone reached no thermal state.
| Between the phone and the desktop, unshaped | Bare Wi-Fi link | Atlas | Atlas, end-to-end sealing |
|---|---|---|---|
| One TCP flow down, Mbit/s (÷ bare) | 453 | 309 (0.68) | 277 (0.61) |
| One TCP flow up, Mbit/s (÷ bare) | 363 | 319 (0.88) | 325 (0.89) |
| Eight TCP flows down, Mbit/s (÷ bare) | 655 | 417 (0.64) | 380 (0.58) |
| Eight TCP flows up, Mbit/s (÷ bare) | 445 | 407 (0.92) | 309 (0.70) |
| Engine CPU-s/GB, one flow down / up | – | 16.9 / 27.1 | 21.4 / 35.0 |
| Whole phone, mW, one flow down / up | 2,051 / 2,607 | 2,327 / 3,584 | 2,527 / 4,269 |
| Net mW per Mbit/s, one flow down / up ¹ | 3.27 / 5.47 | 5.46 / 9.55 | 6.67 / 11.04 |
| Joules per GB, one flow down / up | 36 / 56 | 61 / 92 | 73 / 105 |
| Joules per GB, eight flows down / up | 43 / 55 | 69 / 86 | 76 / 112 |
| Idle, screen off, mW | 564 (no VPN) | 680 (tunnel up) | – |
Down = the desktop to the phone, up = the phone to the desktop; median of 3, each a 75-second test read over its last 60 seconds. The radio: Wi-Fi 6, 5 GHz, 80 MHz, two streams, a link rate of 1,134 Mbit/s up and 1,200 down at −25 dBm throughout. The engine figure is the app’s process minus its traffic generator, the runtime and bridge included, so it is not comparable with atlasd’s figures on the other tiers. Power is the whole phone’s, from the battery’s fuel gauge, screen off and on battery. ¹ Power above the same mode’s idle, per Mbit/s; sealing is read against Atlas’s idle.
Downstream nothing in the app is busy: the engine’s threads use 54 percent of a core for one flow and the test client 16 percent, against 96 percent for the engine sending. The limit sits below the app, in the phone’s path for the tunnel’s datagrams from the radio through the kernel to the tunnel interface, which an app cannot sample; the bare link receives TCP with the radio’s and the kernel’s receive offloads, which a stream of datagrams into a user-space engine does not get. That is an inference, not a measurement.
With end-to-end sealing every packet is sealed and opened on the phone: 27–29 percent more engine CPU per GB for one flow, about 10 percent less downstream, one flow upstream level, eight flows upstream 0.76 of Atlas (309 against 407 Mbit/s), and 11–31 percent more energy per GB. On the shaped link sealing carries 86.3 and 86.6 Mbit/s, exactly what its tunnel MTU of 1349 leaves of 100 Mbit/s.
A 100 Mbit/s download through Atlas draws 1.5 W of the whole phone, about 6 percent of the battery an hour, an upload 2.5 W, about 12 percent; the busiest tests, sealed uploads, about 21 percent an hour. Per GB a shaped link costs more than an unshaped one, 134 against 61 J downstream: the radio and the processor stay awake for a third of the bytes. The thermal status stayed at none in every sample of two hours of back-to-back tests, its forecast headroom at most 0.75.
With the tunnel up and nothing to carry the phone draws 680 mW against 564 with no VPN, screen off, and the app takes 6.6 percent more of a core: the link probes, every 250 ms, keep the radio and a core awake. About 2.8 Wh a day.
Below each mode’s ceiling, at 0.7 times its one-flow TCP figure, Atlas delivers UDP with 0.02 percent loss each way. Bare UDP downstream at 315 Mbit/s lost 4.8 percent: the phone’s kernel handed every datagram to the app, whose test client, one thread at 98 percent of a core, could not read them all. That is the test client at a higher offered rate, not the radio and not a difference between the modes.
The test adapter stops all three paths.
On the laptops every unshaped path stops at the test adapter’s USB 2.0 limit: 258 Mbit/s bare, 242 with WireGuard, 239 with Atlas. Atlas reaches it with its receiving core 81 percent busy. With the tunnel and link offloads on, the sender spends a third less per packet than with them off, 11.6 against 17.0 µs.
| Unshaped, median of 3 | TCP one flow | TCP eight flows | Core busy % (sender / receiver) | atlasd CPU-s/GB |
|---|---|---|---|---|
| Bare link | 258.2 | 257.9 | 11 / 24 | – |
| Kernel WireGuard | 241.8 | 242.0 | 13 / 29 ¹ | – |
| Atlas | 238.8 | 238.7 | 31 / 81 | 26.1 |
| Atlas, encryption off | 238.9 | 238.7 | 24 / 74 | 20.7 |
Mbit/s. ¹ The pinned core under-counts WireGuard’s unpinned crypto workers. Encryption off and eight flows change nothing, so these cells show the adapter’s wall, about 275 Mbit/s of Ethernet frames, not a processor’s ceiling. Shaped to 100 Mbit/s, atlasd costs 27.2 CPU-s per GB (30.3 with both offloads off, against 26.8 for the as-shipped twin of that pair), 12.2 µs per packet at the sender and 24.7 at the receiver; the cipher is 19 percent of it. Small packets are CPU-bound on the sender’s core: 64 B UDP delivers 58.7 thousand packets per second unshaped. With end-to-end sealing on, one flow carries 224.2 Mbit/s unshaped.
Sealing runs at MTU 1349 by itself.
A node with [crypto] e2e on and no mtu line runs its tunnel at 1349, so a sealed full-size packet stays one datagram per hop. TCP over a sealed tunnel is carried whole on every tier, and at the ceiling one desktop core carries 1.83 Gbit/s sealed, where it carries 3.26 unsealed.
| Sealing on | Test | Sealed default MTU 1349 |
|---|---|---|
| SBC, Pi Zero 2 W, Cortex-A53, one core, one link | TCP one flow | 56.7 |
| SBC, Radxa Zero 3W, Cortex-A55, one core, unshaped | TCP one flow | 107 |
| SBC, Radxa Zero 3W, Cortex-A55, unpinned, unshaped | TCP one flow | 211 |
| Router, Beryl AX, Cortex-A53, one core, unshaped | TCP one flow | 168 |
| Router, Beryl AX, Cortex-A53 ×2, not pinned, unshaped | TCP one flow | 179 |
| Mini PC, NUC 14 Pro, Core 3 100U P-core, unshaped | TCP one flow | 1,712 |
| Laptops, Ryzen 5 7535U, one link | TCP one flow | 86.3 |
| Laptops, Ryzen 5 7535U, three links, wrr | TCP one flow | 150.9 |
| Desktop, Ryzen 9 9950X3D, one link | TCP one flow | 86.3 |
| Desktop, Ryzen 9 9950X3D, three links, wrr | TCP one flow | 151.0 |
| Desktop, Ryzen 9 9950X3D core, unshaped | TCP one flow | 1,833.7 |
| Server, EPYC 7763 core, unshaped | TCP one flow | 0.93 Gbit/s |
| Laptops, Ryzen 5 7535U, one link | UDP, 1321 B datagrams at link rate | 87.9 |
| Desktop, Ryzen 9 9950X3D, one link | UDP, 1388 B datagrams at link rate | 79.7 ¹ |
| Desktop, Ryzen 9 9950X3D, three links, wrr | UDP, 1388 B datagrams at 175 Mbit/s | 0.0 ¹ |
Mbit/s, median of 3. ¹ A 1388 B datagram does not fit the 1349 tunnel and is fragmented by the sender’s own kernel; one link delivers 79.7 of 100 Mbit/s with 20.4 percent loss, three links, the NUC 14 Pro and the Radxa Zero 3W nothing, the Beryl AX 65.7 on one core and 77.4 not pinned through a shaped 100 Mbit/s link, against 87.8 with 1321 B payloads. The second seal costs atlasd 10.29 against 6.12 CPU-s per GB on a desktop core and 37.0 against 27.2 on the laptops at 100 Mbit/s; on the Beryl AX’s one A53 39.6 against 25.7 at the ceiling, where the cipher is already 41 percent of atlasd.
Where Atlas falls short, and what to do about it.
The cells a careful reader would find in the data, stated here with the figures and the guidance.
With [crypto] e2e on, the tunnel runs at MTU 1349 and an application that writes 1388 B UDP datagrams is IP-fragmented by its own kernel: at link rate one shaped link delivers 79.7 of 100 Mbit/s (20.4 percent loss), three links, the NUC 14 Pro and the Radxa Zero 3W deliver nothing, a Beryl AX router 65.7 and 77.4 (one core, not pinned). Send UDP payloads of 1321 B or less over a sealed tunnel; TCP and senders that size from the interface MTU are unaffected.
Unpinned on a 256-CPU server, atlasd starts one worker per CPU and carries 0.26 Gbit/s of one flow, against 1.59 pinned, at 39.7 against 9.3 CPU-s per GB. Do not read 1.59 as the figure of a default install on a big server: pin atlasd to a core on the NIC’s NUMA node. Hosts with few CPUs are not affected the same way.
On 100 GbE one flow over two links carries less than over one: 0.93 (weighted_round_robin), 0.95 (lowest_delivery_time) and 0.65 Gbit/s (broadcast) against 1.59. Bond such links for redundancy. On links slower than a core, every radio tier, bonding adds capacity as the bonding section shows.
Held to one A55 core with the NIC’s interrupt on the same core, a Radxa Zero 3W cannot receive an unthrottled UDP flood, bare or through Atlas: the interrupt starves the receiver (receive livelock). Unpinned, the same senders deliver, 297 Mbit/s of 1388 B datagrams through Atlas and 815 bare. One pinned A55 is also saturated by one 100 Mbit/s link: it carries one TCP flow at 89.1 Mbit/s, but paced full-size UDP at 35.4 of 100, where unpinned delivers 91.6. Run atlasd unpinned on such a board, as it is installed.
On a Beryl AX one TCP flow is one stream on one receive path, and more flows do not raise it: eight carry 227 against one flow’s 228 Mbit/s on one A53, 243 against 260 not pinned, so the per-packet path is the limit, as on the Radxa Zero 3W. An unthrottled 1388 B UDP flood delivers 150 Mbit/s through Atlas on one core and 138 not pinned with 99.4–99.7 percent lost, held at a drop rate rather than a ceiling (WireGuard 340, bare 790): pace UDP senders to the link. Unlike the Radxa, one pinned core with the interrupts on it is not livelocked: the router’s Ethernet controller keeps the receive softirq small, and bare UDP arrives at 790 Mbit/s. Not pinned, a shaped link costs 0.27 ms one way, where one core adds none: the two data-path workers hand the flow between them.
A plain LAN host’s upload through a Beryl AX into the tunnel saturates one A53 at 146 Mbit/s and carries 175 Mbit/s on 1.52 of the router’s two cores as deployed, at 46.8 / 53.3 CPU-s per GB, about 1.6 times what forwarding downstream costs (201 / 214 Mbit/s). Sealing every datagram is the cost, not how the segments arrive: handing the tunnel 7 merged segments per read, a diagnostic, moves one A53 only from 146 to 153 Mbit/s. A 100 Mbit/s WAN is carried in both directions, 90.4–90.8 Mbit/s. Size a router serving a LAN by the LAN’s upload into the tunnel.
The Wi-Fi 6 station sat behind USB 2.0, so the router’s Wi-Fi 6 capacity could not be measured: the Wi-Fi figures are a lower bound. The served-LAN ceiling is for one LAN host and one flow per cell, and no profile split atlasd’s send path from its cipher and system time. The router had one peer and one link; whether its second core would carry a second flow of its own was not measured.
On a Pixel 10 Pro XL over its own Wi-Fi 6 radio, Atlas carries 0.68 of the bare link downstream with one TCP flow and 0.64 with eight, sealed 0.61 and 0.58, against 0.88–0.92 upstream, with the app’s threads not busy: the phone’s receive path below the app is the limit. Plan a phone’s downloads at about two thirds of what its radio carries bare. An idle tunnel costs the phone 117 mW, a fifth more than the idle phone, screen off.
One phone over Wi-Fi only: not yet measured are the WireGuard app’s tunnel as a reference on the same phone, LTE and 5G, use with the screen on, an hour-long transfer or a warm pocket, and sealing through a relay. The phone’s cores were neither pinned nor named, and the radio’s power is not separated from the rest of the phone.
On the cloud VMs one flow is one data-path worker: not pinned is not faster for one flow on any machine type, level on the t2a, 9 percent slower on c4, 13 percent on n2 and c3 and 39 percent on e2-medium’s shared core, and eight flows carry no more than one, 1.04 and 1.46 against 1.38 and 1.91 Gbit/s not pinned on c3 and c4. e2-medium varies run to run, not pinned one flow 602–680 and e2e 236–506 Mbit/s, and adds about 0.3 ms one way: quote it with its ranges, or choose a dedicated shape for steady figures. Size a cloud node by the flows or link sockets it carries at once, not by its vCPUs.
A hub takes no more from 8 peers than from 4, one link socket being one receive path: 2.0 Gbit/s in on a c3-standard-8, about 1.05 Gbit/s out. A relay runs out of packets, not sessions: one relay address forwards 64,000 packets per second on an n2-standard-2 to 119,000 on a c3-standard-8 whatever its session count, two addresses on one VM 1.7–1.8 times that, and past the ceiling every session on the address loses packets. Size a relay by its packets per second, each address under about 70 percent of its ceiling, and spread a hub’s peers over more [[link]] sockets; measured with up to 256 sessions per relay and 8 peers per hub, as the sizing section sets out.
Unpinned at 100 Mbit/s, atlasd’s workers each wake for a flow one worker carries, so a gigabyte costs twice what one pinned worker spends: 132.3 against 64.5 CPU-s per GB on a Radxa Zero 3W, 1.5 of its four cores for one 100 Mbit/s link. The NUC (11.71 against 6.87) and the Pi (191.2 against 138.4) show the same, less strongly. At the ceiling the two placements cost the same per byte.
Idle atlasd, its link probes every 250 ms and its statistics, costs 4.1–4.5 percent of an A55 core on the Radxa, 2.4 percent of the Pi’s A53 and 1.7–1.9 percent of a Beryl AX’s A53, against under 1 percent of an x86 core.
The laptop tier is bound by its USB 2.0 test adapter, about 275 Mbit/s of frames, for the bare link, WireGuard and Atlas alike, so it measures no processor ceiling; the receiving core is 81 percent busy at that rate.
On one link Atlas sends full-size datagrams in runs of up to eight. A bottleneck queue that counts packets rather than time, a fixed-length netem or pfifo queue, holds such a run as one entry, so under a bulk TCP flow everything behind it waits longer than behind single datagrams. Queues limited by time, fq_codel and the airtime queues of Wi-Fi drivers, are not affected, and the unloaded latency figures on this page do not see it.
lowest_delivery_time, the default, is built for continuity. With a much slower link beside a fast one it carries one TCP flow at 83.9 Mbit/s, where the fast link alone carries 90.8, and full-size UDP loses 18.9 percent instead of 7.6. Use weighted_round_robin when throughput is the goal.
broadcast delivers the fastest link’s rate and pays per copy: 17.14 atlasd CPU-s per GB over two links on a desktop core, against 7.55 for weighted_round_robin over the same links.
A relay sends its forwarded datagrams one by one, so at 90 Mbit/s it costs 7.0 to 9.5 CPU-s per GB on a desktop core, and on a server two hops carry 0.70 Gbit/s where one carries 1.59.
Linux never pushes back on a sender writing into a tunnel interface: a local application that sends UDP faster than the bond carries loses the excess at the tunnel’s queue, 94 percent of an unthrottled flood on one desktop core. Pace UDP senders to the link, or put a tbf shaper on atlas0 below the bond’s rate, which blocks the sender instead. Into one saturated edge core an unthrottled 1388 B flood delivers 1,795 Mbit/s, the loss at the receiving application’s socket, and neither offload switch moves it; TCP and paced senders are unaffected.
Declaring a link’s wire rate as its capacity sizes its receive buffer for that rate. On a saturated edge E-core it costs 9 percent of one flow; elsewhere it changes nothing. Declare the capacity of links whose real rate is known.
Kernel WireGuard runs its crypto in unpinned per-CPU workers, so the pinned-core tiers (Pi Zero 2 W, Radxa Zero 3W, Beryl AX, NUC 14 Pro, laptops, desktop, server) under-count its CPU, and on the cloud VMs its crypto runs on every vCPU of the VM, so the cloud pairs have no like-for-like WireGuard cell. The virtual machines hold both tunnels to one vCPU and are the like-for-like comparison, with the server’s whole-node figure and the Beryl AX’s whole board not pinned, where kernel WireGuard costs less than Atlas: 26.1 against 37.6 CPU-s per GB at the one-flow ceiling, 48.5 against 72.9 at 100 Mbit/s.
Every machine and link the figures ran on.
Every host and network element of the tiers, read in 2026-11 with read-only commands: the CPU and its clocks, memory, storage, the bench NIC with its driver, the link medium and what sits at its other end. Nothing here changes a figure on this page. Where a fact could not be read, it says not recorded.
| Tier | Device | CPU or SoC, cores, clocks | Memory | Storage | Bench NIC, driver | Link medium, other end |
|---|---|---|---|---|---|---|
| SBC, Raspberry Pi Zero 2 W, Cortex-A53 | Raspberry Pi Zero 2 W Rev 1.0, two of five units | Broadcom BCM2710A1 in the RP3A0 package: four Cortex-A53, no SMT, 600–1000 MHz | 512 MB LPDDR2 in the package, no ECC | 64 GB microSD | USB Wi-Fi adapter, MediaTek MT7612U, mt76x2u, on the USB 2.0 port | 802.11s on 5 GHz channel 36, 20 MHz; the two boards side by side, −10 / −11 dBm, 173.3 Mbit/s PHY |
| SBC, Radxa Zero 3W, Cortex-A55 | Radxa ZERO 3W, Wi-Fi variant | Rockchip RK3566: four Cortex-A55, no SMT, 408–1800 MHz | 2 GB LPDDR4, its clock scaled 324–1056 MHz by the board’s memory governor; no ECC | 128 GB microSD, boot and root | ASIX AX88179B USB 3 gigabit adapter, cdc_ncm; no segmentation or checksum offload | UTP Cat 5e, about 10 m, through the office LAN path to the desktop’s gigabit port |
| Router, Beryl AX, Cortex-A53 | GL.iNet GL-MT3000 (Beryl AX), OpenWrt | MediaTek MT7981B: two Cortex-A53, no SMT, fixed 1.3 GHz | 512 MB DDR4, 485 MiB to Linux, no ECC | 256 MB SPI-NAND | 2.5 GbE WAN port: the SoC’s MAC, mtk_soc_eth, to a MaxLinear GPY211C PHY, linked at 1000 Mb/s; TSO, GSO, GRO and checksums on | UTP Cat 5e, about 2 m, at 1 GbE through the office LAN path to the desktop’s gigabit port; the Wi-Fi 6 cells on the 5 GHz radio (MT7976, mt7915e), HE80 |
| Mini PC, NUC 14 Pro, Core 3 100U | ASUS NUC 14 Pro (NUC14RVK-B) | Intel Core 3 100U: two P-cores with SMT up to 4.7 GHz, four E-cores up to 3.3 GHz | 2 × 8 GB DDR5 SODIMM at 5200 MT/s (rated 5600), dual channel, no ECC | A live system run from RAM | Intel I226-V, igc, PCIe x1 | One direct UTP Cat 5e cable, 26 AWG, about 3 m, to the desktop’s Realtek RTL8126 5 GbE port, linked at 2.5 GbE |
| Laptop, Ryzen 5 7535U (two units) | HP 255R G10 notebook, two units | AMD Ryzen 5 7535U (Zen 3+): 6 cores / 12 threads, up to 4.63 GHz | One 16 GB DDR5 SODIMM at 4800 MT/s (rated 5600), single channel, no ECC | NVMe, 512 GB | ASIX AX88179B USB gigabit adapter, cdc_ncm, at 480 Mbit/s behind USB 2.0 hubs on one laptop | One direct copper cable between the two adapters, four 802.1Q VLANs; type and length not recorded |
| Desktop, Ryzen 9 9950X3D | Gigabyte X870E AORUS MASTER | AMD Ryzen 9 9950X3D: 16 cores / 32 threads on two CCDs, one with 3D V-Cache, up to 5.75 GHz | 2 × 32 GB DDR5 UDIMM at 4800 MT/s (JEDEC, EXPO off), dual channel, no ECC | NVMe | Namespaces over veth pairs; to the NUC a Realtek RTL8126 5 GbE controller, r8169; to the office LAN a Realtek RTL8111E gigabit, r8169 | The far end of the edge, companion and router tiers |
| Desktop VM, Ryzen 9 9950X3D | KVM guests under QEMU 10.0 on the desktop | One vCPU (CPU type host), the whole QEMU process on one host core | 1 GiB | A virtio disk | virtio-net over QEMU’s dgram backend, or tap with vhost-net | The loopback address or host bridges on the desktop |
| Server, EPYC 7763 cluster | The three EPYC nodes: two Supermicro H12DSi-N6, one H12DSi-NT6 | Two AMD EPYC 7763 per node, 64 cores / 128 threads each, SMT on; 1.5–2.45 GHz, boost to 3.525 GHz | Registered DDR4 with ECC, 0 corrected errors; eight channels per socket, populated differently per node (below) | Two SATA SSDs for the system, two NVMe for the storage cluster | NVIDIA (Mellanox) ConnectX-5 Ex MCX516A-CDAT, mlx5_core, PCIe Gen4 x16 on the NICs’ NUMA node; the 10 GbE hop on a ConnectX-3 MCX312A-XCBT, mlx4_en | 100 GbE through two switches, every node to both (below) |
| Server VM, Proxmox guests on the EPYC 7763 | Linked clones on node a and node b, destroyed after the run | One vCPU (CPU type host), the whole QEMU process on one host core | 1 GiB in 2 MiB hugepages on the NICs’ NUMA node | A virtio-scsi disk | virtio-net with vhost-net | A host bridge over the second ConnectX-5 port |
| Phone, Pixel 10 Pro XL, Tensor G5 | Google Pixel 10 Pro XL, Android 17 | Google Tensor G5; core layout and clocks not recorded (nothing reaches the phone) | Not recorded | Not recorded | Its own Wi-Fi 6 radio, two streams; cellular off | 5 GHz, 80 MHz, to the Beryl AX above a few metres away, −25 dBm, 1,134 / 1,200 Mbit/s link rate; the Beryl AX’s routing on to the desktop |
| Cloud VMs, Google Compute Engine | Five machine types, the relay VMs and their clients (below) | Broadwell, Neoverse-N1, Cascade Lake, Sapphire Rapids, Emerald Rapids | As the shape gives | A 10 GB boot disk | virtio-net or gVNIC, MTU 1460 | The provider’s network, one zone; a cross-region pair |
Tier labels as in the tables above. Each tier ran its own pass; where the hardware is known to differ from the day a tier ran, the performance report says so. The desktop’s memory is a kit sold as DDR5-6000 with EXPO, run at the JEDEC 4800 MT/s.
Reading the hardware corrected three earlier tier notes. The Beryl AX’s 2.5 GbE WAN port runs on a MaxLinear GPY211C PHY, not an RTL8221B. The Radxa Zero 3W boots from a 128 GB microSD card, not eMMC, and the board’s memory governor scales its LPDDR4 clock over 324–1056 MHz; it was not pinned during the pass. The desktop’s port to the NUC is a Realtek RTL8126 5 GbE controller, linked at 2.5 GbE, not a 2.5 GbE NIC.
Two CRS504-4XQ-IN switches, RouterOS 7.15.2, four QSFP28 100G ports each: one switch carries every node’s first 100 GbE port, the other every node’s second. Every node reaches both by a 5 m 100G active optical cable (Ubiquiti AOC-QSFP28-5M, 850 nm, four lanes), at 100G with RS-FEC. Node side, per lane, transmit +1.35 to +1.61 dBm and receive −0.31 to +0.35 dBm; switch side transmit +1.43 to +1.46 and receive −0.39 to +0.23 dBm, so the lowest receive figure leaves about 9–10 dB of margin against a 100GBASE-SR4 receiver. Zero FCS and RS-FEC symbol errors on every port of both switches. The NICs are ConnectX-5 Ex MCX516A-CDAT at PCIe Gen4 x16.
The EPYC 7763 has eight DDR4 channels per socket, and the three nodes populate them differently: the relay forwarder 7 × 64 GB DDR4-3200 RDIMM on 4 + 3 channels (448 GB), node a 8 × 64 GB DDR4-3200 RDIMM on 4 + 4 (512 GB), node b 12 × 32 GB DDR4-2133 LRDIMM on 6 + 6 (384 GB). ECC throughout, with 0 corrected and 0 uncorrected errors on every memory controller.
The 10 GbE relay hop runs over the nodes’ backup ports: single-mode 10GBASE-LR SFP+ optics, 1310 nm, LC duplex, into a CRS317-1G-16S+ switch. Host-side receive power −2.69 to −6.17 dBm, the lowest still well inside a 10GBASE-LR receiver’s range of about −14 dBm; the switch side was not read.
The Radxa Zero 3W’s and the Beryl AX’s bench links reach the desktop’s office port through the office switch, a TL-SG105PE (five 1 GbE ports, four of them PoE+, web-managed), and the office router, a hAP ax lite (L41G-2axD: two ARM cores at 800 MHz, 256 MiB RAM, 128 MiB flash, RouterOS 7.24.4), the office LAN’s gateway and its 2.4 GHz Wi-Fi 6 access point. The desktop is cabled to the router directly, not to the switch. The cables are UTP Cat 5e: the Beryl AX to the switch about 2 m, the Radxa Zero 3W’s adapter to the switch about 10 m (four pairs of 24 AWG), the desktop to the router about 3 m. Not recorded: the switch’s uplink cable and which router port the desktop uses. Those tiers’ bare cells measured the path as it stands, 710 Mbit/s for the Radxa Zero 3W and 678–705 for the Beryl AX, which is why the Beryl AX’s bare figure is the limit of the switched path, not of the Beryl AX.
The cloud machine types
The cloud VMs as the provider describes them and as each guest reported itself while it ran the bench.
| Machine type | CPU platform, model as reported | vCPUs (threads per core) | Guest memory, MiB | NIC, driver | Zone | USD/h |
|---|---|---|---|---|---|---|
| e2-medium | Intel Broadwell; Xeon at 2.20 GHz; half a shared physical core | 2 (2) | 3,919 | virtio-net, virtio_net | europe-west4-a | 0.037 |
| t2a-standard-2 | Ampere Altra; Neoverse-N1 | 2 (1) | 7,936 | gVNIC, gve | europe-west4-a | 0.085 |
| t2a-standard-8, the hub | Ampere Altra; Neoverse-N1 | 8 (1) | 32,044 | gVNIC, gve | europe-west4-a | 0.34 ¹ |
| n2-standard-2, also a relay server and the hub’s eight clients | Intel Cascade Lake; Xeon at 2.80 GHz | 2 (2) | 7,945 | virtio-net, virtio_net | europe-west4-a | 0.107 |
| n2-standard-2, the cross-region far end | Intel Cascade Lake; Xeon at 2.80 GHz | 2 (2) | 7,945 | virtio-net, virtio_net | us-east4-a | 0.110 |
| n2-standard-4, a relay server and the five session-scaling clients | Intel Cascade Lake; Xeon at 2.80 GHz | 4 (2) | 15,996 | virtio-net, virtio_net | europe-west4-a | 0.214 ¹ |
| n2-standard-8, a relay server | Intel Cascade Lake; Xeon at 2.80 GHz | 8 (2) | 32,099 | virtio-net, virtio_net | europe-west4-a | 0.428 ¹ |
| c3-standard-4 | Intel Sapphire Rapids; Xeon Platinum 8481C at 2.70 GHz | 4 (2) | 15,995 | gVNIC, gve | europe-west4-a | 0.22 |
| c3-standard-8, the hub and the relay server | Intel Sapphire Rapids; Xeon Platinum 8481C at 2.70 GHz | 8 (2) | 32,098 | gVNIC, gve | europe-west4-a | 0.44 |
| c4-standard-4 | Intel Emerald Rapids; Xeon Platinum 8581C at 2.30 GHz | 4 (2) | 14,987 | gVNIC, gve | europe-west4-a | 0.23 ¹ |
Every VM: MTU 1460, the provider network’s; Linux 6.12.113, the Debian 13 cloud kernel for x86_64 or arm64, from the Debian 13 cloud image of 2026-11; a 10 GB boot disk; on-demand scheduling. No guest reports a turbo or maximum clock: the model string’s base clock is what the guest echoes. The vCPUs are hyperthreads on the Intel shapes and whole cores on the t2a. Every NIC offers TCP segmentation, GSO, GRO and checksums, none UDP segmentation in hardware. Prices: on demand per VM, the provider’s list of 2026-11, plus 0.005 USD/h for an external IPv4 and 0.0014 for the disk. ¹ Estimated, not on the list.
The performance report.
Every table and figure of this page, the method in full, the memory review, the data-path modes compared on every tier where they were measured, the relay sizing method, the phone tier and the hardware as measured. Host names and addresses are not in it. Tell us who you are and what you are looking at, and the download starts.