Scanning
How a device reads the whole Log, finds the entries it can open, checks their writers and files them.
A scan is reading the Log on your device to find what is yours. No entry names its reader, so no server can hand a reader their messages. Every device reads every entry and tries its own keys on each.
What a device downloads#
| Request | What comes back |
|---|---|
GET /log/head | the number of entries, the number of full segments, the segment size, and the tail: every entry after the last full segment |
GET /log/segments/:n | one full segment, for every n from 0 to the segment count minus one |
GET /pool/leaves | every leaf of the pool, from leaf 0 |
GET /inboxes/head | the inbox directory's event count, sealed segment count, segment size and tail |
GET /inboxes/segments/:n | one sealed segment of 256 directory events, for every n from 0 to the segment count minus one |
The segments and the tail together are the whole Log, in order. The leaves let the device find its own passes and build its own Merkle path for a proof. The inbox directory is every open_inbox and rotate_inbox_key in chain order, one { index, wallet, inboxKey, keyVersion, slot, blockTime } event each; the device folds it into every wallet's Inbox and key history for the checks below. When a writer it needs is missing from its copy and the copy is more than two seconds old, it downloads the new events and looks again. Inboxes and keys gives the fold. Every device makes the same requests whoever it belongs to, so what it downloads says nothing about which entries, which pass or which writers are its own.
Trying each entry#
Each entry's payload has two key slots, and each slot holds an ephemeral public key, a one-byte view tag and a wrapped content key. Log entry format gives the bytes.
The device tries every key it holds on both slots of every entry:
- every version of its inbox key, from 1 to the current one
- the key of every private link it holds and has not revoked
- the keys of link indices it doesn't hold yet: every index from 0 up to the number of links it holds plus seven, so a message to a link whose note this device hasn't read yet is still found
- the key of every room it holds
For each key and slot, it computes the X25519 shared secret with the slot's ephemeral key, hashes it with SHA-256 and compares the first byte with the view tag. A different byte means the slot is not for this key, and the device moves on without decrypting anything. That skips almost every entry: an entry meant for someone else passes the view tag by chance for about one key and slot in 256.
Decryption#
When the view tag matches, the device unwraps the content key from the slot and decrypts the ciphertext with XChaCha20-Poly1305, which authenticates the ciphertext and the whole key-slot header together. A chance view-tag match fails here and the entry is skipped. Cryptography gives the key derivation for each step.
Writer verification#
A decrypted body names its writer's wallet, the writer's inbox key and that key's version. The device accepts it only if every check passes, in this order:
- The body is a well-formed version 1 message.
- The writer is not blocked on this device. A blocked wallet's messages are dropped here, before anything else is read.
- The writer has an Inbox in the directory, and the key and version the body names were the writer's registered key at the slot of this Log entry. The device checks this against the writer's key history, so a message sent before the writer rotated still verifies. See Inboxes and keys.
- The body's writer MAC for this reader is valid. The writer computes it from the X25519 shared secret between their inbox key and the reader's key; the reader computes the same secret from its own secret key and the writer's public key. Only the holder of the writer's inbox secret key could have made it.
- A message addressed to a wallet is addressed to this one, unless this wallet wrote it.
An entry that fails any check is skipped as not this device's. Nothing is shown and nothing is reported.
Multi-part reassembly#
A text longer than one part is sent as numbered parts, one post each. Each part carries the message's id, its own number from 0 and the total. The device collects parts by message id, keeps only parts with the same writer, total and target as the first one found, and joins the texts in order once every number from 0 to the total minus one is present. The message then appears once, with one Log entry per part.
A message whose parts are not all in yet is held back. The device saves its scan position as the earliest such part, so the next scan reads those entries again, after a restart too.
Filing#
| What opened | Where it goes |
|---|---|
| a message to your inbox key | the thread with its writer; a writer new to this device makes a request |
| your own message, through the self slot | the thread with the wallet you wrote to |
| a message under a room key | that room |
| a message under a private link's key | the thread with its writer, marked with the link's name when it has one; a link index this device didn't hold joins its links, named from the link URL inside the message |
| a room invitation | the room, waiting to be accepted; joined at once when this wallet sent it |
| a room removal notice from a member | the room, marked removed, and its key dropped from this device |
| your own room key update that leaves you out | the room, marked left, and its key dropped from this device |
| your own private link note | the link joins this device's private links, or is marked revoked; no thread |
The self slot#
Every message has a second key slot encrypted to its writer's own inbox key. The writer's devices open it like any other entry, and the message is filed as sent. This is how sent messages come back on another device or after a restore: no copy of them is stored anywhere else.
Follow mode#
A scan reads up to the head of the Log and stops, or, in follow mode, waits 1.5 seconds, reads the Log again and continues from the next position. The app scans in follow mode while it is open, which is how a reply arrives without reloading.
The device saves how far it has read in its encrypted history after each pass, and the next scan starts there. If the indexer can't be reached, the scan stops with INDEXER_UNREACHABLE, and the app says the indexer can't be reached and offers to try again. The messages themselves stay on chain. Errors lists the code.
Restore from position 0#
Restore in Keys scans the whole Log again from position 0, without following, and then finds the wallet's passes again. A device with no history already starts its first scan at 0. With the same wallet's signature, a restore brings back every thread, every sent message through the self slot, every room through its invitation, every private link through its note, revoked ones marked revoked, and every pass. Inboxes and keys gives the steps.
Progress#
Each scan reports its progress as it goes:
- stage'downloading' | 'scanning' | 'following' | 'complete'
downloadingwhile the Log is fetched,scanningafter each entry,followingwhen a follow-mode pass reaches the head,completewhen a scan without follow ends.Required
- nextLogIndexstring
- The next zero-based Log position to scan, as a decimal string.
Required
- headLogIndexstring
- The end of the Log as last downloaded, exclusive: the number of entries.
Required
- scannedEntriesstring
- Entries this scan has tried, as a decimal string.
Required
- foundMessagesnumber
- Whole messages this scan has found.
Required
The app shows these as the entry reached, the Log's length and how many are yours.
From the SDK#
for await (const message of talaria.scan({
follow: true,
onProgress: (progress) => console.log(progress.stage, progress.nextLogIndex, progress.headLogIndex),
})) {
console.log(message.direction, message.writer, message.text)
}
const restored = await talaria.restore()
console.log(restored.messages, restored.threads, restored.rooms, restored.nextLogIndex)
scan yields each whole message once. Pass an AbortSignal as signal to stop following. SDK reference gives every option.