Skip to main content

Repairing Uncertified Blobs

An upload is three steps in order: register the PooledBlob in the account's StoragePool, push the slivers to the storage nodes, and submit the certify transaction. Walrus serves a blob only once it is certified. If the certify transaction fails after register landed, the PooledBlob stays onchain with certified_epoch: null and the data is unreadable.

Before 0.14.4 this state could become permanent. The failed upload answered 502 with no blobs row. The client's retry re-registered, which aborted with EFieldAlreadyExists, and the self-heal path answered 201 with the existing PooledBlob without certifying it. The row it wrote then short-circuited every later PUT of the same content, so the client had no way to repair it, and had already discarded the bytes. The upload path now finishes the certify on the retry (see Blob Lifecycle) and on any later dedup hit (see Content Addressing). The bulk pass for the blobs nobody re-uploads is oysterd blobs repair.

What it does​

A candidate is one (account, blob) with its onchain PooledBlob. Candidates come from the database (blobs rows that reference a PooledBlob but have no certified_epoch) and from --ids-file, a list of PooledBlob object IDs, which also reaches orphans with no row at all: a 502'd upload the client never retried.

For each candidate:

  1. Read the PooledBlob from chain. Gone: MISSING-ON-CHAIN.
  2. Already certified: stamp certified_epoch on the rows (stamped). Rows from before migration 026 land here.
  3. Otherwise ask the storage nodes for a confirmation certificate without uploading anything. The failed upload did push the slivers before its certify failed, and nodes keep uncertified slivers for a while. With a certificate, submit certify and stamp the rows (certified). The blob is readable again.
  4. No certificate: the bytes are gone from Walrus and Oyster never kept a copy. By default the tool reports this as UNRECOVERABLE. With --delete-unrecoverable the tool deletes the PooledBlob (returning its capacity to the pool), removes the rows, and prints the keys so you can tell the customer to re-upload (deleted).

Every step is idempotent. Re-running after a crash or a transient failure picks up where it left off. --dry-run reads the chain and the storage nodes and reports what each candidate would get (would-stamp, would-certify, would-delete, UNRECOVERABLE) without submitting or writing anything.

Prerequisites​

  • Oyster ≥ 0.14.4 deployed, so new uploads stamp certified_epoch and the upload path no longer creates these blobs.
  • The same environment as oysterd serve: DATABASE_URL, PEARL_GRPC_URL, PEARL_SERVICE_SECRET, SUI_RPC_URL, WALRUS_SYSTEM_OBJECT, WALRUS_STAKING_OBJECT. Pearl signs certify and delete transactions under each account's own key, and the account's wallet pays for them (SUI gas only; no WAL).
  • The tool skips an account under oysterd keys migrate (FAILED: signer: key migration in progress); re-run after the rotation.

Procedure​

  1. Dry run, scoped to one account first if you have a known case:

    oysterd blobs repair --dry-run
    oysterd blobs repair --dry-run --account <ACCOUNT_ID>

    Output is TSV: ACCOUNT, BLOB_ID, POOLED_BLOB, REFERENCES (bucket/key,...), OUTCOME. A summary line goes to stderr. The would-certify count is what the real run recovers.

  2. Run it:

    oysterd blobs repair

    Exit code 1 means some candidates need attention. Read their OUTCOME:

    • UNRECOVERABLE: decide with the customer (step 3).
    • MISSING-ON-CHAIN: the rows point at a PooledBlob that no longer exists. The data is gone; treat like unrecoverable.
    • FAILED: ...: transient (RPC, a concurrent transaction on the account's pool). Re-run.

  3. For the unrecoverable ones, once the affected customers know the tool deletes their keys:

    oysterd blobs repair --delete-unrecoverable

    Each deleted line lists the keys that the tool removed. Deleting is what lets a future PUT of the same content register afresh; until then the PooledBlob blocks re-registration and the rows would make the dedup path try (and fail) to certify it again.

  4. With a list of object IDs from an incident (one per line, # comments allowed), include orphans that have no rows:

    oysterd blobs repair --ids-file pooled-blobs.txt
    oysterd blobs repair --ids-file pooled-blobs.txt --skip-db

    --skip-db repairs only the listed IDs. The tool reports entries whose StoragePool matches no account as UNKNOWN-POOL.

Verifying​

  • oysterd blobs repair --dry-run prints nothing (no candidates).
  • GET /api/v1/buckets/{bucket}/blobs/{key} returns the data for a certified key.
  • oyster_dedup_certify_verify_total{certified="completed"} and oyster_register_dedup_self_heal_total{certified="completed"} stay flat afterward; a rising count means certify transactions are still failing on first attempt (see oyster_certify_stale_version_retries_total).

Repair output format​

In the oysterd blobs repair output, a tab separates each column. The REFERENCES column lists every bucket/key pair for the blob, joined by a comma with no space. The tool writes bucket names and keys as stored, without quoting or escaping. A key that contains a comma or a tab therefore makes that line ambiguous when you split it automatically. Because the column order is fixed, parse such lines positionally instead: take ACCOUNT, BLOB_ID, and POOLED_BLOB as the first three tab-separated fields and OUTCOME as the last field, and treat everything between them as REFERENCES. To act on one of those blobs, identify it by its POOLED_BLOB object ID, which oysterd blobs repair --ids-file accepts as a list of PooledBlob object IDs, rather than by its bucket/key text.