Skip to main content

Sui Archival System

The Sui Archival example application archives Sui checkpoint data on Walrus. It is not the official Sui Archival service. The example uses that service as a checkpoint source and stores the archived data on Walrus, and it shows how to archive blockchain data in a reliable, deterministic, and resilient manner. The application continuously downloads Sui checkpoints, the sequential batches of finalized transactions that the Sui network produces, bundles them into compressed blobs, and uploads the blobs to Walrus. Because Walrus erasure-codes every blob across a large committee of storage nodes, the archive stays readable even when individual nodes fail. You can browse the deployed archive.

How does the Sui archival example work?​

Walrus stores data as blobs, immutable units of data that storage nodes keep available for a paid number of epochs. This example application using the Sui Archival service runs the following components concurrently:

  1. Checkpoint downloader: Downloads checkpoints from a configurable checkpoint source, such as the Sui data ingestion service, using multiple parallel workers.
  2. Checkpoint monitor: Tracks downloaded checkpoints, handles out-of-order delivery, and triggers blob building when the pending checkpoints hit a configured threshold: total size, elapsed time, or the end of a Sui epoch. A backpressure mechanism pauses the downloader when too many checkpoints await bundling.
  3. Checkpoint blob publisher: Bundles a contiguous range of checkpoints into a single compressed blob and uploads it to Walrus. The bundling follows a deterministic algorithm: a given checkpoint range, cut at the configured size, time, or end-of-epoch boundaries, always produces the same blob.
  4. Checkpoint blob extender: Watches the expiration epoch of every archived blob and extends it automatically, so the archive stays available without manual renewals.
  5. Archival state snapshot creator: Uploads snapshots of the archive's metadata and maintains an onchain pointer to the latest snapshot, so a fresh instance can rebuild its database for disaster recovery.
  6. REST API server: Serves the web interface and the JSON endpoints listed in How to query the API.

The service records blob metadata (the Walrus blob ID, the Sui object ID, the checkpoint range, and the size of each archived blob) in a RocksDB database, and can optionally mirror it to PostgreSQL for fast indexed queries.

How to build and run the example​

  • A Rust toolchain. Install Rust.

  • A Walrus client configuration and a wallet funded with SUI and WAL. See Getting Started.

  • Optional: a PostgreSQL database if you want indexed queries alongside the local RocksDB store. You configure it through the optional PostgreSQL connection URL in the example configuration file reviewed in the next step.

Then build and run the example:

  1. Clone the repository and build all crates:

    git clone https://github.com/MystenLabs/walrus-sui-archival.git
    cd walrus-sui-archival
    cargo build --release

  2. Review the example configuration, which selects the checkpoint source, the blob building thresholds, the path to your Walrus client configuration, the local database path, the REST API bind address (default 0.0.0.0:9185), and the optional PostgreSQL connection URL:

  3. Run the example application with your configuration:

    cargo run --release -p walrus-sui-archival -- run --config config/testnet_local_config.yaml

  4. Open the web interface at the REST API bind address from your configuration (http://localhost:9185 when you use the default 0.0.0.0:9185) to view archival statistics, browse archived blobs, and look up checkpoints. To confirm that the service is running, send a request to the GET /v1/health endpoint, which returns 200 OK. See How to query the API.

How to query the API​

The REST API server exposes JSON endpoints alongside the web pages:

EndpointDescription
GET /v1/healthHealth check that returns 200 OK.
GET /v1/checkpoint?checkpoint=<number>Returns the metadata of the blob that contains the given checkpoint. Add show_content=true to include the full checkpoint data.
GET /v1/blobsLists all archived blobs with their metadata.

For example, to find which Walrus blob holds checkpoint 12345 on a local instance:

curl "http://localhost:9185/v1/checkpoint?checkpoint=12345"

The example application also provides inspection subcommands to examine its local database and archived blobs:

# List all archived blobs recorded in the local database
cargo run --release -p walrus-sui-archival -- inspect-db --db-path archival_db list-blobs

# Fetch a blob from Walrus by blob ID and inspect its bundled checkpoints
cargo run --release -p walrus-sui-archival -- inspect-blob \
  --blob-id "<BLOB_ID>" \
  --client-config config/local_testnet_client_config.yaml \
  --context testnet

How to read the archival code​

Running the example needs nothing beyond the steps in How to build and run the example. Read the source when you want to adapt the pattern to your own data: archival.rs wires together the downloader, monitor, publisher, extender, snapshot creator, and REST API server described in How does the Sui archival example work?, and is the file to start from.

View the application's full code on GitHub.