Esc

    Operate

    Latency

    The node's latency budget, the measured latency from the venues to the SDK, what each measure means, and how to run the measurement.

    The internet path from the venue dominates the latency. It is 100 to 1,000 times the node's own time. To place nodes near the venues is the only large gain, and operator diversity is more important than that gain.

    Latency budget

    The budget covers the node only: from the exchange TCP receipt to the signed frame on the node's socket. It excludes internet transit. These are design targets.

    Stage p50 p99
    WebSocket frame decode and normalization 15 µs 80 µs
    Channel hop to the aggregator 5 µs 50 µs
    Aggregation 20 µs 100 µs
    Signature (secp256k1) 40 µs 90 µs
    Serialization and fanout enqueue 10 µs 60 µs
    Total about 90 µs about 400 µs

    The signature takes the most time. If that ever changes, batched signing is the answer. See Batching is reserved.

    Measured latency

    Measured with make latency on 2026-10-07. The node, the registry canister on PocketIC and the SDK ran on one macOS machine. The node served the 14 feeds of node/all-feeds.toml from the live venues for 60 s.

    Stage p50 p99
    Venue timestamp to node receipt 37–146 ms 98–568 ms
    Venue data to signed update (node) 0.12 ms 52 ms
    Signed update to SDK receipt 0.28 ms 1.5 ms
    SDK receipt to resolved price 0.12 ms 0.57 ms

    The venue row comes from an earlier run. The other rows come from the run below.

    text
    venue to node (venue clocks)  p50/p99 binance 0.0/215.6 ms (97% ahead), bybit 0.0/0.0 ms (100% ahead), ...
    venue data to signed (node)   p50 0.12 ms, p99 51.74 ms
    signed to SDK (delivery)      p50 0.28 ms, p90 0.59 ms, p99 1.51 ms, max 6.33 ms (n=1059)
    venue data to SDK (data age)  p50 0.46 ms, p90 136.57 ms, p99 1073.15 ms, max 1994.75 ms (n=945)
    SDK processing                p50 0.12 ms, p90 0.25 ms, p99 0.57 ms, max 4.87 ms (n=1059)

    What each measure means

    Measure From To Notes
    Venue to node The venue's own timestamp The node's receipt Compares the venue clock with the node clock. Valid only with a synced host clock. A lag below zero reads as zero.
    Venue data to signed (node) The newest venue data at the first evaluation that fired The signed update Counts only updates that a price move or a flag change caused. A wait in the rate cap counts. A heartbeat repeats old data, so it does not count.
    delivery The node signs The client receives Compares the node clock with the client clock, so it holds their offset.
    data_age The node receives the newest venue data The client receives Includes heartbeats, which send old data on purpose. Thus it is the age of the data, not only the delay.
    processing The client receives next() returns the resolved price Decryption, the queue, verification and the quorum.

    The SDK stamps each frame when it arrives, before it decrypts the frame.

    The node row has a p99 of about 50 ms because of the rate cap. A feed publishes at most once for each min_interval, 50 ms by default. A move that arrives inside the cap waits, and that wait counts. See Aggregation.

    Run the measurement

    bash
    make latency
    LATENCY_SECS=300 make latency

    make latency runs cargo test --release -p oracle-node --locked --test latency -- --ignored --nocapture. The test needs the internet, so make test skips it. It:

    1. installs the registry canister on PocketIC;
    2. makes a node with oracle-node init, and adds it through the canister;
    3. starts the node on every feed in node/all-feeds.toml, against the live venues;
    4. lists a client and runs the SDK as that client;
    5. waits 15 s for the venue connections and the local books to settle;
    6. measures for LATENCY_SECS seconds (default 60), and prints the rows.

    The node rows come from the node's second status and venue lag lines, because the first one includes the startup.

    Latency in production

    You do not need the test to see latency on a live system:

    • The node logs the venue-data-to-signed p50 and p99 in its status line each minute, and the venue lag in its venue lag line. See The status line.
    • The SDK reports p50, p90, p99 and max of delivery, data_age and processing through OracleClient::latency. reset_latency starts the report again after a warm-up. See Rust SDK.

    Caveats

    • One machine. The measured run put the node, the canister and the SDK on one machine. delivery compares two clocks, so it is exact only on one machine or with clocks in sync. Between machines, it also holds the offset between the two clocks.
    • The host clock. The venue row compares each venue's clock with the host clock. In the last run, the machine was 144 ms behind NTP (sntp), and 69–100% of the venue timestamps were ahead of its clock. Those lags read as zero. Check the host clock with chronyc tracking (Linux) or sntp time.apple.com (macOS) before you read the venue line.
    • Clock drift. Before the node re-anchored its clock to wall time each second, the delivery row read 1.33 ms at p50. Most of that was drift. See The clock.
    • The venue path dominates. The internet path from the venue is 100 to 1,000 times the node's own time. Place nodes near the venues only where that does not cost failure-domain diversity. A 40 ms difference between nodes is absorbed by the median. A correlated outage is not.