Bonding & scheduling

Measured links. Per-packet decisions.

Every link in the bond is an isolated socket with its own queue, its own watchdog and its own probe clock. The scheduler never trusts configuration optimism — it trusts the last few hundred milliseconds of measurements. This page is the mechanics, with the demos inline.

Field-proven core8 strategies · per packet
Probes

Every link, on its own clock

Default cadence 250 ms per (peer, link) — RTT smoothed by EWMA (α = 0.2), jitter as windowed deviation, loss over a sliding window, dead after 5 consecutive misses. And a link is never alive before its first completed round-trip: recovery requires evidence, not hope.

Shipped · opt-in

Per-link probe intervals

[[link]] probe_interval_ms overrides the global clock per link, with timeout detection at 2× that link’s own interval. The reason it exists: probing a geostationary VSAT (~600 ms RTT) at 250 ms stacks probes inside the path latency and reads phantom loss — so the 4G link keeps 250 ms while the VSAT beside it checks at 1000 ms.

Data loss

Measured on real traffic, not just probes

Sequence numbers piggyback on every Nth data fragment (default 10), so under bursty interference the loss the actual traffic experiences is measured directly — sparse control probes systematically underestimate it.

Radio truth

Telemetry from the hardware itself

Adapters feed live RSSI, noise floor, buffer depth and error counters straight from the radio — signal a UDP probe can’t see. RSSI is the one the scheduler acts on, as an EWMA level and slope; the rest are there to be read, on the dashboard and in the API. adapters ↗

Budgets

A metered link that knows what it has left

[[link]] quota = "20 GB", with quota_period, quota_reset_day and quota_warn_pct, gives a link a byte budget. Once it is spent the scheduler stops promoting that link while any other link is usable — but it is still used when it is the last path left, because a budget is an economic constraint, not a hard-down. Usage, remaining budget, reset time and an over-budget flag are exported per link.

The timing truth, once and clearly. At the defaults (a probe every 250 ms, five misses), a link that goes silent is declared dead about 1.5 to 1.75 s later: a probe counts as missed once it has gone two intervals unanswered, then one more per interval. The sub-second headline — typically configured 0.1–0.75 s — is the reweighting cadence: how fast load shifts as quality degrades, long before anything is declared dead. Faster profiles (e.g. 50 ms × 3 ≈ 200–250 ms detection) are real but tuned. Tuned profile
Scheduling

Eight strategies, one contract: decide per packet.

Strategy is deployment policy, not vendor magic — chosen in one config line, observable in the decision log, and fed by the operator’s adaptive policy where the mission optimises for more than delivery time. Five of the eight are in the simulation below.

Strategy comparison — one stream, three dissimilar linksSimulation
Strategy

One copy per packet, on the path with the lowest expected delivery time, (size/capacity + RTT/2 + queue) × 1/(1 − loss) — so a bulky fragment prefers the fat link while a small control packet can take the low-latency one.

lowest_delivery_timestrategy in force
0%redundancy overhead
0packets in flight
Reading the lanes: the bar is each link’s live share of the stream and the shaded area under it is that link’s share over the last twelve seconds — so changing the strategy shows up as a step in all three at once. Latency-adaptive is the live demo mode: inject delay across the paths and watch redundancy escalate 1 → 2 → 3 simultaneous paths, then step back down only after the dwell — escalation is instant, de-escalation is deliberate.

The full set

lowest_delivery_timedefault

Per fragment, minimise the expected delivery time, (size/capacity + RTT/2 + queue) × 1/(1 − loss) — a bulky fragment prefers the fat link, a tiny control packet may take the low-latency one. A lossy link, or one the prediction says is fading, counts 1/(1 − loss) times as slow, so it hands the primary to a clean link before it fails. Capacity-, size-, queue- and loss-aware bonding without configuration.

weighted_round_robin

Spread packets across alive links proportionally to per-link weight, every fragment of a packet on the link that packet was placed on — true aggregation when the links deserve it: two 100 Mbit links carry 181.6 Mbit/s of one TCP flow, twice one link’s 90.8 and 95% of their summed capacity (one desktop core per node, network namespaces), where the default strategy keeps to the best single link. The demo fleet weights its mesh : ad-hoc : Bluetooth ≈ 70 : 28 : 1.4 so the thin link never bottlenecks the bond.

broadcast

Every fragment on every alive link; the receiver’s anti-replay window deduplicates. Costs N× bandwidth, survives N−1 failures — and turns loss multiplicative: three independent links at 2% each yield 0.0008% effective loss; the scripted demo delivers ~2.7% through three links each forced to 30%. Measured

priority

Strict failover through an explicit priority_order — the primary carries everything until it dies. For pairs like a preferred serial radio with a second one as pure standby.

shortest_path

Switches multi-hop SPF to hop-count costs; per-fragment link choice falls back to delivery time. For meshes where hop count is the honest metric.

adaptiveescalating

Cost-ratio-driven redundancy with Schmitt-trigger hysteresis: mirror when the second link’s cost approaches the best (enter 0.9 / exit 0.5), triplicate deeper (0.5 / 0.25), 5 s dwell blocking de-escalation only. Cost inflates geometrically with loss (1/(1−loss)) and jitter.

loss_adaptiveescalating

The same ladder driven purely by measured loss: two paths at 20% loss, three at 50%, de-escalation at 0.7× the entry threshold.

latency_adaptiveescalating · live demo mode

Escalation driven by the best link’s RTT: mirror at 40 ms, triplicate at 100 ms, shared dwell. This is the interference-adaptive mode demonstrated live — inject delay, watch redundancy engage, watch it stand down.

Plus the redundancy_factor modifier — “send on the best N links” — for topping up any non-adaptive strategy.

What the strategies rank is the operator’s to shape. The strategies optimise delivery time; a mission may care about cost, emissions, energy, a fading signal or the onset of jamming. Rules in the node’s signed configuration read the live link table — round trip, loss, the prediction’s risk, RSSI and its trend, noise, utilisation, quota use, bearer kind, the peer’s role and distance, our speed, the time of day — and exclude a link, weight it, keep it in reserve, prefer it, or keep a traffic class off it, with a sandboxed script for what rules cannot say. They are evaluated twice a second, never per packet; the strategy then decides per packet among what is left, and the dashboard names the rule behind every link that is not carrying. adaptive policy ↗
Delay, not length Shipped

A queue that never drains is a cost the moment it stands.

Thirty packets in a queue are nothing on Ethernet and several seconds on a serial radio. Counting packets compares bearers of different rates by a number that means something different on each, so every link measures time instead: how long its packets wait.

WAIT ↑ QUEUE DRAINS QUEUE STANDS target shortest wait standing queue declared ONE INTERVAL TIME → target = max(5 ms, RTT / 10) · interval = max(100 ms, 2 × RTT) declared when even the shortest wait stays above target for a whole interval
THE MEASUREhow long each packet waited in the link’s queue, not how many were waiting
THE TESTthe shortest wait over an interval, against a target that follows the link’s round trip
THE RESULTa standing queue adds its delay to the link’s delivery time — nothing is dropped
The test

The shortest wait, over an interval, against a target

A queue doing its job drains: in any short stretch some packet finds it nearly empty and passes almost without waiting. A standing queue never drains, so even its luckiest packet waits. Atlas applies the controlled-delay idea to each link’s own send queue: it records how long every packet waited and keeps the shortest wait over an interval. When that minimum stays above the target for a whole interval, the queue is standing, and the minimum is the delay it adds. Target and interval follow the link — a tenth of its round trip, never under 5 ms, and twice its round trip, never under 100 ms — so a satellite link and a cable are each judged on their own terms. The same test runs on the kernel’s figure from the queues below the tunnel, so a backlog standing in a device queue is caught as well.

The response

The delay becomes part of the link’s cost

Nothing is dropped by the test; packets past their budget are already discarded by the deadline check. The queue delay enters each link’s delivery time — half the round trip, plus serialisation, plus the time in the queue — weighted by 1/(1 − loss) for the default and adaptive strategies — which is what the default strategy, the adaptive modes and the deadline feasibility check rank links by. They steer away from a standing queue without a new setting, before any packet is lost. Per-class escalation also holds back while any link to the peer waits more than 250 ms, because extra copies on a congested link only deepen the queue.

Boundaries

The device’s own buffer is inferred, not read

The test sees Atlas’s queue and the kernel’s. A buffer inside the radio or modem is invisible to the host; its sign is a probe round trip that climbs while every queue above it reads empty.

Graceful degradation

Not binary. Continuous.

Real links don’t die instantly; they rot. Threshold failover keeps trusting a rotting link, then drops everything at once. Continuous scoring moves the traffic out while the link still carries.

Nexus Atlas: Links Lost One by One1:56 · plays from YouTube
A VSAT link degrading and recovering — gradual reweighting vs threshold failoverSimulation
steady state — VSAT 580 ms RTT · 0.2% loss · quality 92/100
t+0.0 sscenario clock — the 44-second cycle repeats
92 / 100measured quality of the link
30%share — Nexus Atlas, continuous
30%share — threshold failover
Why the share never quite reaches zero: a trickle of probes keeps measuring the degraded link, so recovery is detected the moment it begins — and rebuilt sample by sample, which is why there is no flapping on the way back up.
Below the tunnel Shipped

Six queues between a packet and the air.

A packet can wait in six places on its way out. Atlas owns two of them. The kernel holds the next three, and Atlas reads those as well — the serial line through the radio’s adapter. The last belongs to the radio or modem and stays dark unless its firmware is integrated with Atlas: a companion adapter, or the native serial link, reports the device’s own buffer fill in-line with the data, and then that queue is read too. A link can answer every probe and still bury each packet behind a full buffer, so Atlas reads every queue the host can show it and reports the wait per link, in milliseconds.

APPLICATION AIR ATLAS KERNEL DEVICE class queueAtlas · read when QoS is on backlog per class link send queueAtlas · read how long each packet waited UDP socket bufferkernel · read bytes not yet sent device queue (qdisc)kernel · read backlog and drops, over netlink serial line (tty)kernel · read by the adapter bytes waiting, and the baud rate radio or modem bufferdevice · not readable probe round trip is the tell-tale read today read by the adapter not readable
READ TODAYAtlas’s class and link queues, the socket’s unsent bytes, the device queue’s backlog and drops
READ BY THE ADAPTERa serial radio’s line queue, in bytes, with the baud rate that turns it into time
NOT READABLEthe radio or modem’s own buffer — a rising probe round trip is its tell-tale
Atlas

Our two queues

With QoS on, the class queue orders traffic by lane; below it, each link’s send queue holds what the scheduler gave that link. Both belong to the daemon, so what they hold and how long each packet waits in them is measured directly, not inferred.

Kernel

The kernel’s two

Under the daemon, each link’s UDP socket keeps the bytes the kernel has not sent yet, and the device’s queueing discipline keeps its own backlog and drop count. Atlas samples both every 200 ms by default — the backlog and drops over netlink, with no shell-out to tc — and turns the backlog into time at the rate the queue is actually draining. A link whose real rate has collapsed below its declared capacity reads its true wait. The same figure gates the low band. pacing ↗

Serial

The serial case

On a serial telemetry radio the last queue the host can see is the serial line itself. The radio’s adapter reads the bytes still waiting in the tty and reports the baud rate beside them, so the daemon knows the exact wait: ten bit times per byte, plus whatever the PPP device queue holds above it. The native serial link goes further: it reads the line itself and the radio’s transmit buffer as the radio’s adapter reports it, and paces on both. adapters ↗

Dark

What stays invisible

A modem’s own buffer, and a radio’s behind PPP, cannot be read from the host, and Atlas does not pretend otherwise. The exception is a serial radio on the native serial link, whose buffer fill its adapter reports over the telemetry socket. A queue inside the device still shows itself, as a probe round trip that climbs while every queue above it reads empty. That remains its tell-tale.

The figures reach the link card on the dashboard, the per-second snapshot and the metrics export, per link.

The low band, paced Shipped

Handed to the kernel only as fast as the air takes it.

Priority inside Atlas is not enough on a slow bearer. Video fragments already handed to the kernel sit in the device queue in front of the next voice frame, and at 64 kbit/s each 256-byte fragment is about 43 ms of airtime. So the low band is released to the kernel only as fast as the link drains it, with a few milliseconds of airtime in hand, and the high band is never held.

HIGH BAND LOW BAND Control · Voice · Positionnever held Video · Telemetry · Bulkpaced pacing gatereleased while the kernel holds ≤ 5 ms of airtime waits in Atlas’s queue meanwhile: a voice frame still overtakes it device queue (qdisc)kernel · backlog read over netlink backlog and drain rate, measured, feed the gate AIR target = 5 ms of airtime at the measured drain rate (2–100 ms) · the high band is never held voice one-way p95 ≈ 30 ms beside 1 400-byte video at 64 kbit/s — 70–90 ms unpaced
HIGH BANDControl, Voice and Position go straight to the kernel — never held
LOW BANDVideo, Telemetry and Bulk are released only while the kernel holds at most 5 ms of airtime at the link’s measured drain rate; meanwhile they wait in Atlas’s queue, where a voice frame still overtakes them
MEASUREDvoice one-way p95 ≈ 30 ms beside 1 400-byte video at 64 kbit/s — 70–90 ms unpaced; a TCP transfer keeps its goodput
The gate

A few milliseconds of airtime in hand

Before a low-band datagram goes to the socket, the bytes the kernel already holds for that link and has not sent must be at or under a small target — [qos] low_band_target_ms, 5 ms by default, 2–100 — of airtime at the link’s measured drain rate. Until then it waits in Atlas’s own queue, where a voice frame still overtakes it, and if its deadline passes there it is dropped in Atlas instead of spending airtime it would not have survived. The high band — Control, Voice and Position — is never held. The target is at least one datagram, so the link never idles for want of a packet.

The rate

The drain rate is measured, not declared

What is in flight is the device queue’s backlog from the queues below the tunnel, read over netlink and modelled between readings — bytes sent, drained at the measured rate. The rate itself comes from those readings, bytes the queue sent over at least 100 ms and 8 kB while it stayed busy; it starts at the declared capacity and is never left below what a sometimes-idle queue actually carried, so a link declared slower than it is runs at its true speed and is not throttled to the figure in its config. A held datagram is released only on a fresh reading, at most one per millisecond.

Measured

Voice beside video on a 64 kbit/s link

Fifty frames a second of 60-byte voice beside 1 400-byte video on a 64 kbit/s link with 5 % loss and the 20 ms fragment ceiling: voice one-way p95 of about 30 ms paced, across repeated runs, against 70–90 ms unpaced. On fast links the gate costs nothing: a TCP transfer keeps its goodput with the gate and without it, 90.8 against 90.8 Mbit/s on a 100 Mbit/s link and 181.1 against 181.2 Mbit/s on a two-link 100 Mbit/s bond. With QoS on, every declared link is also paced to 95 % of its wire, so one TCP flow over a paced 100 + 50 Mbit/s bond carries 129.3 Mbit/s of the 136.2 it carries unpaced, by design (one desktop core per node, network namespaces; 129.2 on the laptops).

Capacity

Every declared link, held at 95 % of its wire

With QoS on, every link whose capacity is declared is also paced by a token bucket to 95 % of what its wire carries. Each datagram is charged its full size on that bearer, the IP packet plus the link-layer framing the link’s own queue counts: the 14-byte Ethernet header on Ethernet, Wi-Fi, mesh and cellular links, PPP’s framing and escaping over a serial radio. The margin therefore holds for voice frames as well as full-size video. The 5 % kept in hand is what makes the queue that fills under load Atlas’s own, where it sheds by class and counts what it sheds, rather than the modem’s or the radio’s buffer. A bearer that frames differently, behind a VLAN tag or over a tunnelled underlay, states its own figure with pace_framing_bytes, and the dashboard’s link card reads “Paced · 95 % of wire”.

Planning

The deadline check reads the paced queue

Because the low band now queues in Atlas, deadline planning reads that queue: a low-band packet on a paced link counts what pacing holds plus the pacer’s estimate of the kernel queue, read live as the packet is planned, so a slow mirror leg is skipped instead of filling with copies that expire. Voice, Control and Position plan against the kernel queue alone — they skip the low band’s queue and are not held infeasible behind its standing estimate. the classes ↗

Boundaries

Paced where the queue can be read

A device whose queue cannot be read is not paced: a link with no bound device, a noqueue root such as a veth and most Wi-Fi drivers, and a netem root that emulates delay, whose backlog is the emulated path rather than a queue. The native serial link paces on its own figures, the line and the radio’s reported buffer. On by default with QoS; [qos] low_band_pacing = false turns it off. Per link in the snapshot (links[].pacing: active, target, rate, what is held and for how long), two metrics in the export, and a “paced” chip on the link card.

Range

The split follows the geometry.

Nothing is cut and no policy changes here — the aircraft simply flies further out. Each bearer’s usable rate falls at its own rate, so the scheduler reweights continuously; the short-reach fat link fades out of the bond long before the long-reach narrow one does.

Ground station to UAV — three bearers, one bond, distance the only variableSimulation
Stand-off
2.0 kmdistance from the ground station
3 of 3bearers carrying traffic
99.6 Mbpsaggregate usable rate
What you’re watching: reach and capacity run in opposite directions — the 5 GHz link is the fat one and the first to run out of range, the 900 MHz link is the thin one still working at the far end. The curves are each bearer’s usable rate against distance (log scale); the bar underneath is the share that rate earns it. All three carry at once, and no moment in the sweep is a failover.
config.toml — two links, adaptive scheduling
# A 4G modem beside a geostationary VSAT — each probed on its own clock.
[[link]]
name = "wwan0"
bind_device = "wwan0"
capacity = "50 Mbps"

[[link]]
name = "vsat0"
bind_device = "eth1"
capacity = "10 Mbps"
probe_interval_ms = 1000   # don’t stack probes inside 600 ms of path latency

[scheduler]
strategy = "latency_adaptive"
latency_mirror_ms = 40
latency_triplicate_ms = 100

[link_monitor]
probe_interval_ms = 250
dead_probe_count = 5

Then atlasd config check proves it loadable before any restart window, and atlasd monitor shows the decisions live.

Request an evaluation

Your worst link deserves better company.

Bring the bearer mix you actually have — VSAT beside LTE beside a 64 kbps serial radio — and watch one daemon make sense of it.