Run the web proxy

oracle-proxy gives web apps verified oracle data. It is one listed client of the nodes. It runs the Rust SDK, with the signature checks and the quorum, and serves the result as JSON over one plain WebSocket, /v1/ws. A reverse proxy in front of it terminates TLS. Web proxy API describes what web clients send and receive.

Install

The proxy runs as the Docker image brinzi/oracle-proxy on Docker Hub, for Linux amd64. Each tag is a commit of the markoutlabs/oracle-proxy repository. That repository's deploy/README.md runs the image on Docker Swarm, behind a Cloudflare tunnel.

The image holds only the binary. It reads its config, its client key and its database URLs from files that you mount, such as Swarm configs and secrets.

To build the binary on your own machine:

git clone https://github.com/markoutlabs/oracle-proxy && cd oracle-proxy
make build        # cargo build --release --locked

The binary is target/release/oracle-proxy. It needs Rust 1.85 or newer. The build fetches the oracle crates from the price-oracle repository with git.

Get a client key listed

The proxy connects to the nodes as one client, with its own client API key. All web clients share it. The two scripts are in the price-oracle repository:

scripts/client-key.sh --out proxy.key          # the proxy's own client key
scripts/add-client.sh --identity <controller> --address <its address> --expires <date>

• Run the first command on the proxy machine. It prints the address of the key.
• Governance runs the second command. It lists the address on the registry canister.
• The key file must have mode 0600. The proxy refuses to start when the group or others can read it.
• When the expiry passes, the nodes close the sessions of the proxy. Every price then goes halted for every web client. Ask governance for set_client_expiry before that date.

Client access explains the key, the client list and the transport.

Configure

The proxy reads one TOML file. proxy.example.toml in the oracle-proxy repository is a commented example:

listen = "127.0.0.1:8792"

[upstream]
# The mainnet canister and governance address when absent.
# canister_id = "xb5gv-diaaa-aaaaf-qg5qa-cai"
# governance = "0x7FB990Cc1111a00931B6396E5780C79574e36897"

# This proxy's client API key, as hex, readable only by its owner (chmod 600).
client_key_file = "proxy.key"

# Nodes that must agree on a price.
quorum = 3

[feeds]
prices = ["CRYPTO.BTC/USD", "CRYPTO.ETH/USD", "CRYPTO.BTC/USDT"]
venues = ["CRYPTO.BTC/USDT"]
books = ["CRYPTO.BTC/USDT"]
funding = ["*"]

[limits]
# Behind the reverse proxy every socket comes from 127.0.0.1.
trust_forwarded_for = true

The proxy refuses an unknown setting in every section.

Top level

[upstream]

How the proxy reaches the nodes.

The mainnet values come from oracle_client::mainnet in the build.

[feeds]

What the proxy pulls from the nodes and serves. Each list takes feed symbols. At least one list must not be empty.

books takes feed symbols, not book names. For CRYPTO.BTC/USDT, the proxy serves the book of each source of that feed, as the node config defines the sources. Web clients find the book names in /v1/catalog.

[limits]

Limits on web clients. The socket is open to anyone, so these limits are the whole defence against a client that asks for too much. Every value except trust_forwarded_for must be above zero.

A socket over a connection limit gets HTTP 429, or 503 when the proxy is full, with a Retry-After header. Limits and errors lists what a web client sees for each limit.

Local network

For a registry canister on a local network, set all four trust values. The certificate check needs the canister id even with an explicit URL. icp network status --json prints the gateway URL and the root key.

[upstream]
canister_id = "<canister id>"
registry_url = "http://127.0.0.1:8000/registry.json?canisterId=<canister id>"
root_key = "<root_key from icp network status --json>"
governance = "0x…"
client_key_file = "proxy.key"
# One local node: no protection against a bad node.
quorum = 1
allow_low_quorum = true

Quorum

The proxy applies the same quorum rules as the SDK:

• The default is 3.
• A quorum below 3 needs allow_low_quorum = true. One node proves who is talking, not that the price is right.
• The proxy does not start when the registry lists fewer nodes than the quorum.
• With fewer than three nodes listed, set quorum to the number that exists and allow_low_quorum = true.

Five nodes give a quorum of 3 room for one planned and one unplanned outage. Quorum has the full rules.

Run

target/release/oracle-proxy proxy.toml

At startup the proxy does these steps:

1. It reads and checks the config file.
2. It reads the client key file and checks its mode.
3. It fetches the registry and verifies it against the governance address. It logs registry verified.
4. It connects to every node and subscribes to the configured feeds.
5. It listens for web clients. It logs serving web clients on /v1/ws.

The proxy does not start when:

• the config file does not parse, or a value is out of range;
• the key file is missing, is not valid hex, or the group or others can read it;
• the registry does not verify against the governance address;
• the registry lists fewer nodes than the quorum, or no node endpoint.

Logs go to stderr. RUST_LOG filters them, and the default level is info. The proxy stops cleanly on SIGTERM, which systemd sends to stop a service, and on SIGINT.

TLS reverse proxy

The proxy speaks plain ws on loopback. Put a reverse proxy in front of it that terminates TLS and passes WebSocket upgrades. Caddy does both:

oracle.example.com {
    reverse_proxy 127.0.0.1:8792
}

Web clients then connect to wss://oracle.example.com/v1/ws. The same host serves /v1/catalog and /health.

trust_forwarded_for

Behind the reverse proxy, every socket comes from 127.0.0.1. Without this setting, the per-IP limits count all web clients as one address. With the defaults, only 8 sockets can then be open in total, and only 30 new sockets can open each minute.

With trust_forwarded_for = true, the proxy takes the client address from the last X-Forwarded-For entry. That is the entry that your reverse proxy added. A client can forge the entries before it, but not the last one. When the header is absent, the proxy uses the address of the TCP peer.

Health checks

GET /health shows the state of the SDK client inside the proxy. It answers 200 while the nodes deliver, and 503 when they do not, so a load balancer or an orchestrator can use it as it is:

curl -s http://127.0.0.1:8792/health

{
  "status": "ok",
  "registry_version": 7,
  "quorum": 3,
  "nodes": 3,
  "connected": 3,
  "last_frame_ns": "1791299498663252000",
  "clients": 12,
  "accepted": 10452,
  "venue_reports": 881,
  "book_updates": 20311,
  "rejected": 0
}

The values update once a second. The status values:

• ok (200): a quorum of nodes is connected, and the newest frame is at most 10 s old.
• degraded (200): the newest frame is fresh, but fewer nodes than the quorum are connected. Prices can halt for want of nodes.
• starting (503): no frame has arrived yet.
• down (503): no frame has arrived in the last 10 s.

Also:

• Watch rejected. A steady rise means that something sits between the proxy and a node, or that a stolen key is in use.
• Subscribe to one price on /v1/ws and check that status is not halted. The proxy resolves each price feed again every second, so a silent upstream shows as halted.

Archive

The archive stores the past prices that /v1/history serves to charts. It writes to PostgreSQL with TimescaleDB. An [archive] section in the config turns it on. The proxy repository's README describes the tables and the setup.

The history is for charts, not for proof. The archive keeps no signed frame and no signature. It stores:

• the SDK's result for each feed: each second, and at each change of status;
• the entries of one venue report of each feed in [feeds] venues each second: bid, ask, sizes, mark, index and funding.

Each feed takes one node's venue reports: the first node in market_nodes that sent a report in the last 3 s, or 30 s for a slow feed. Each row names its node.

• Run the archive as its own proxy instance with web = false. It then serves /health alone. Web load and web deploys then cause no gaps, and the database password stays out of the process that faces the internet.
• The archive never slows the prices. While the database is down, a journal on local disk keeps the records, up to journal_max_bytes.
• retention_days (default 7) deletes older rows. Each table compresses one day after a chunk ends. With 225 feeds and 3 nodes, the archive uses about 2.6 GB a day after compression, and the database stays below about 80 GB with the default.
• /health gets an archive object. lost counts every record that the archive dropped, for any reason. unloaded_age_ms is the age of the oldest record that is not in the database yet.

History for charts

The web instance can serve the archive's past prices on /v1/history. A [history] section turns the route on:

[history]
# The URL of a role that can only read, in a file that only its owner reads.
database_url_file = "history.db-url"
# max_range_hours = 24
# requests_per_minute = 30
# max_concurrent = 4

1. Make the role as the database owner, with the proxy repository's script, after the archive has started:

   shCopy

   psql "$ARCHIVE_URL" -v ON_ERROR_STOP=1 -v role=history \
     -v password="$(openssl rand -hex 24)" -f scripts/history-role.sql

   If the script says that the archive has no tables yet, wait one minute and run it again. A second run only sets the password again.

2. Put postgres://history:<password>@<host>/archive in the URL file, and chmod 600 it.

The role can read only the tables that the route needs. The proxy refuses a role that can write, sets each connection to read only, and cancels a query after 5 s. A leaked URL thus shows only what the dashboard shows already. The archive instance keeps the only credential that can write.