Run a node

A node connects to the venues, aggregates their quotes, signs each update, and serves listed clients over /v2/ws. Nodes do not talk to each other. Each node is an independent signer. See Architecture.

The short path, on the node's Linux machine:

curl -fsSL https://raw.githubusercontent.com/markoutlabs/oracle-node/main/release/install.sh | sh
sudo oracle-node init

1. The install script builds the oracle-node binary. See Install oracle-node.
2. sudo oracle-node init asks for the endpoint. Then it sets up the service and starts it. See The system install.
3. Send the add_node call that init prints to governance. The node serves when the registry lists it.

Install oracle-node

On Linux, run the install script. It builds the node from source on your machine:

curl -fsSL https://raw.githubusercontent.com/markoutlabs/oracle-node/main/release/install.sh | sh

The script does these steps:

1. If there is no C compiler, it installs one with the system's package manager: apt, dnf, yum, apk, zypper or pacman.
2. If there is no Rust 1.85 or newer, it installs Rust with rustup. It does not change your shell profile.
3. It builds oracle-node with cargo install --locked. This takes about 5 to 10 minutes. With less than 3.5 GB of memory, it builds one job at a time.
4. It installs the binary to /usr/local/bin, with sudo if necessary. Without root and sudo, it uses ~/.local/bin.

Put options after sh -s --:

curl -fsSL https://raw.githubusercontent.com/markoutlabs/oracle-node/main/release/install.sh | sh -s -- --version v0.1.0 --to /opt/oracle/bin

The build needs about 4 GB of memory. With 2 GB, add swap; the build is then slower.

What a node needs

The deployment needs at least 5 nodes for a quorum of 3. That leaves room for one planned and one unplanned outage. Diversity is more important than latency: the median absorbs a 40 ms difference between nodes, but not a correlated outage.

The node serves plain WebSocket on its listen address. It does not terminate TLS. To publish a wss:// endpoint, put a TLS-terminating reverse proxy in front of the node.

Create the node with init

oracle-node init generates the node's secret seed and derives its two keys. It prints:

• the secp256k1 address, which is the node's id in the registry;
• the ed25519 public key, for venues that verify ed25519, such as tapray-perps;
• the registry entry for governance, which holds no secret.

Nothing that the node signs verifies for anyone until governance adds the entry to a new registry version.

The system install

As root, oracle-node init installs the node as a systemd service:

sudo oracle-node init

It asks one question: the endpoint, the WebSocket URL that clients dial, such as ws://203.0.113.7:8791/v2/ws. After you confirm, it does these steps:

1. It creates the system user oracle-node, with no login shell, if the user does not exist.
2. It makes /var/lib/oracle-node, mode 0700, owned by oracle-node. It writes the key, a generated passphrase, the config and the registry entry there.
3. It installs /etc/systemd/system/oracle-node.service, enables it at boot, and starts it.
4. If ufw or firewalld is active, it opens the listen port for TCP. For any other firewall, such as the cloud provider's, it tells you to open the port.
5. It prints the node's address, the add_node call for governance, and the ed25519 publisher key.

The node starts at once, and waits until the registry lists it. See Start the node.

For a script, give the endpoint with --endpoint. Without a terminal, the flag is necessary:

sudo oracle-node init --endpoint ws://203.0.113.7:8791/v2/ws

--canister, --governance, --gateway and --root-key work as in From flags.

The system install stops before it writes anything in these cases:

• systemd does not run the machine;
• /var/lib/oracle-node is not empty, or the unit exists: the machine already has a node;
• the binary is under a home folder.

If a later step fails, the node is not in the registry yet, so nothing is lost. The error shows the two commands that remove what the install made. Then run init again.

The wizard

Without root, at a terminal, oracle-node init is a wizard. Its first note names the registry canister and the governance address. It then asks one question at a time:

1. Where the files go. The default shows as an absolute path. A leading ~ is your home directory. The wizard refuses a directory that already holds a node key.
2. The public endpoint. The WebSocket URL that clients dial. On mainnet, a loopback address such as 127.0.0.1 gets a warning at once, because no client elsewhere can dial it.
3. How the node opens its key at start.
   • A generated passphrase in a 0600 file. systemd needs this, because nobody can type at a service.
   • A passphrase that you type at each start. Nothing to steal lies next to the key.
4. Whether to write a systemd unit, and the user that runs it. The wizard asks this only for a passphrase file.

The wizard writes nothing until you confirm the summary. Esc or Ctrl-C leaves nothing behind. The wizard checks each answer with the canister's own rules, so the registry accepts the entry that it writes.

The wizard runs only when stdin, stdout and stderr are all terminals.

From flags

With --out DIR, every value comes from flags, and the output is plain text that a script can read.

oracle-node init --out /srv/oracle-node \
  --endpoint wss://node1.example.com/v2/ws \
  --passphrase-file /srv/oracle-node/passphrase \
  --service-user oracle

--out and --passphrase-file expand a leading ~.

init never writes over an existing key. A new key would drop the node out of the registry while it kept running.

What init writes

The registry canister

A node always fetches the registry and the client list from the registry canister. init does not ask where they come from. The mainnet values are built in:

The raw domain hands over the canister's response unchanged. The node checks the IC certificate itself. See Registry.

For a local network, pass --canister and --governance together, with --gateway and --root-key. icp network status --json prints the gateway URL and the root key:

oracle-node init --canister <CANISTER_ID> --governance <GOVERNANCE> \
  --gateway http://127.0.0.1:8000 --root-key <root_key from icp network status --json>

Register with governance

init prints the call that adds the node:

icp canister call <CANISTER_ID> add_node '("{\"address\":\"0x…\",\"endpoints\":[\"wss://…/v2/ws\"],\"weight\":1}")' -n ic

Send node-entry.json to governance. Governance runs the call. Governance is the only caller of the canister. The node never calls it and holds no IC identity.

The config file

One file describes the whole node. The node derives its venue subscriptions from the feed sources, so there is no separate instrument list. The node refuses unknown keys.

This is the shape of the config that init writes. The binary holds the network's feed set, about 225 feeds, and init writes all of them. Every node then serves the same feeds, so a quorum can form on each one.

tick_interval_ms = 25

[server]
listen = "0.0.0.0:8791"
max_pending_feeds = 512

[signing]
key_file = "/srv/oracle-node/signing.key"
passphrase_file = "/srv/oracle-node/passphrase"

[registry]
url = "https://<canister>.raw.icp0.io/registry.json"
canister_id = "<canister>"
governance_address = "0x…"

[clients]
url = "https://<canister>.raw.icp0.io/clients.json"
poll_secs = 30

# The network's 225 feeds, from oracle-node 0.1.0. …

[[feeds]]
symbol = "CRYPTO.BTC/USD"
impact_notional = "20000"
peg = "CRYPTO.USDT/USD"
min_sources = 2
sources = [
  { venue = "coinbase", market = "spot", symbol = "BTC-USD" },
  { venue = "kraken", market = "spot", symbol = "BTC/USD" },
  { venue = "binance", market = "spot", symbol = "BTCUSDT", converted = true },
  { venue = "bybit", market = "spot", symbol = "BTCUSDT", converted = true },
  { venue = "okx", market = "spot", symbol = "BTC-USDT", converted = true },
]

# … and the other feeds

The listen port is the port of the endpoint, or 8791 when the endpoint names no port. init sets max_pending_feeds to twice the feed count, rounded up to a power of two, and at least 512.

Give exactly one of url and path in [registry] and in [clients].

The pinned governance address lives in the config, not in the registry. A document that named its own author would vouch for itself.

[[endpoints]] replaces the built-in URL of one venue market: a regional host, a proxy, or a local feed for tests. The node refuses an endpoint that no feed reads. For Binance, the URL is the combined-stream base, and the node adds the stream names:

[[endpoints]]
venue = "binance"
market = "spot"
url = "wss://data-stream.binance.vision/stream"

Start the node

oracle-node start /srv/oracle-node/config.toml

At startup the node proves everything that would make it useless, before it opens its port:

1. It reads the config and resolves the feeds.
2. It opens the key with the passphrase file, or with a prompt.
3. It fetches the registry, checks the IC certificate, and verifies the governance signature.
4. It checks that the registry lists its own address. If not, it waits, and fetches the registry again every 30 seconds.
5. It fetches and verifies the client list.
6. It opens the listen port and starts to serve.

The node stops cleanly on SIGTERM or SIGINT.

Console, status lines and logs

At a terminal, the node prints a banner once it serves. The banner shows:

• the node's address and the governance address;
• the registry version;
• the number of listed clients, and the client list version;
• the feeds, and the number of venue connections.

Log formats

Logs go to stderr. --log-format auto picks the format from where stderr goes:

RUST_LOG filters the logs as usual. The default level is info.

The status line

Once a minute the node logs a status line. The first one comes one minute after startup.

The latency fields count only updates that a price move or a flag change caused. The time runs from the newest data at the first evaluation that fired. A wait in the rate cap counts. A heartbeat repeats old data, so it does not count. See Latency.

A venue lag line follows. For each venue it shows p50 and p99 from the venue's own timestamp to the node's receipt. This lag also holds the offset between the venue clock and the node clock. The line shows the share of venue timestamps ahead of the node clock. A large share means that the node clock is behind. A lag below zero reads as zero.

Other log lines

See what a venue sends

--tail is the fastest way to see what a venue sends. It shows the data after normalisation, and before any weight, filter or aggregation:

12:03:04  INFO observation venue=deribit market=inverse_perp symbol=BTC-PERPETUAL
               kind="book_top" price=64972.25 bid="64972" ask="64972.5"
               bid_qty="2.292679923659" ask_qty="1.610373619608" lag_ms="49"
12:03:04  INFO published status="ok" price="64971.86250214" conf="6.49718625"
               seq=1 sources=2 venues={Kraken, Deribit}

lag_ms is local_recv − exchange_time. It is for diagnosis only. Staleness always uses the node clock, never the venue clock.

Run under systemd

sudo oracle-node init installs and starts the unit. Without root, init writes oracle-node.service on request: --service-user USER, or yes in the wizard. The unit needs a passphrase file, because nobody can type at a service.

To install and start a unit that init wrote without root:

sudo install -m 0644 /srv/oracle-node/oracle-node.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now oracle-node
journalctl -u oracle-node -f            # add -p warning for warnings only

Under journald, the node writes plain lines with a syslog priority prefix. journalctl -p thus filters by level.

The node tells systemd its state through sd-notify:

init warns about three cases:

• a loopback endpoint on mainnet;
• a unit that starts a binary from a target build folder. Install the binary, for example in /usr/local/bin, and edit ExecStart;
• a port below 1024. Uncomment the two capability lines in the unit.

start --record FILE writes outside the read-only system. Add ReadWritePaths= for that folder. The unit has a commented example.

Keep the passphrase encrypted

The 0600 passphrase file protects the key against someone who copies the key file alone. For more protection, encrypt the passphrase with systemd-creds encrypt and give it to the unit:

LoadCredentialEncrypted=passphrase:/etc/oracle-node/passphrase.cred

Then set passphrase_file = "/run/credentials/oracle-node.service/passphrase" in the config. oracle-node rotate then needs --passphrase-file with the plain passphrase.

The clock

The node uses one clock for receipt times and publish times. The clock never goes back. It starts from the wall clock and then follows the monotonic system clock.

The monotonic clock drifts from wall time: about 1.4 ms a minute on one macOS machine. That is about 2 s a day, more than the SDK's 2 s max_age. Thus the node re-anchors its clock to wall time every second:

• When the node clock is behind wall time, it steps forward.
• When the node clock is ahead, it runs at 90% speed until it matches. A step back would date the next update before the last one.

When one correction is larger than 100 ms, the host clock stepped, and the node logs a warning:

Run the host with a synced clock, such as chrony or systemd-timesyncd. Consumers compare publish_time_ns with their own clocks. The venue lag means nothing without a synced clock. Check the host clock with chronyc tracking (Linux) or sntp time.apple.com (macOS).

The sequence numbers start from the clock in milliseconds at startup. A restart thus continues above the previous run.

Health and metrics

GET /v2/health returns the node's address, the number of feeds, the number of open subscribers, and the number of feeds that published at least once. See REST routes.

The node has no metrics endpoint. The metrics export interface is not chosen yet. The status line, the venue lag line and /v2/health are the node's own figures today.

Run an internal consumer that subscribes to all nodes and watches the divergence between nodes. It catches a subtly wrong node that every node-local figure reports as healthy.