On-Demand Tail Read Architecture & Design

ChronoLog 3.1 introduces On-Demand Keeper Tail Read (playback), allowing client applications to query the most recent $N$ events of a story directly from ChronoKeeper in-memory buffers with millisecond-scale latency (~0.4 ms median, 1–5 ms at p90–p99), bypassing the persistent HDF5 tier and ChronoPlayer query service.

---

1. Motivation & Architecture Overview

Historically, retrieving logged events required querying the ChronoPlayer via Client::ReplayStory. ReplayStory returns a whole time range — reading what is already archived from HDF5 and the rest from the keepers — but it incurs indexing, disk I/O, and cross-tier network latencies (~1–2 seconds per query).

Many real-time use cases — such as live dashboards, monitoring alerts, stream processing, and rapid failover recovery — only need the last $N$ events produced by a story.

On-Demand Tail Read provides:

• Direct Client $\leftrightarrow$ Keeper Communication: Clients communicate directly with the story's assigned ChronoKeepers via lightweight Thallium RPCs. No ChronoPlayer or ChronoVisor is in the read path.
• Zero-Copy In-Memory Indexing: Events are indexed directly by their memory pointers in the sealed chunks the keeper retains until they are persisted.
• Two-Phase Scatter/Gather Protocol: Minimal network transfer — exactly $N$ payloads cross the wire across the entire keeper group.
• Writer-Mode Compatibility: Clients instantiated in writer-only mode (ClientPortalServiceConf only) can perform tail reads without configuring query endpoints.

Trying it without writing code

The interactive CLI exposes this path as -p <num_events>, which tail-reads the Story you currently hold:

-a -s my_chronicle my_story
-p 10

See the CLI API Reference. Remember that an Event is only visible once its chunk seals, so allow ~25–30 s after writing (or enable live_tail_read).

---

2. In-Memory Retention & Single Payload Model

In ChronoKeeper, when a StoryChunk seals (past chunk_duration + acceptance_window), KeeperChunkRetentionStore takes ownership of it: it indexes the chunk's events for tail reads and sends the chunk to ChronoGrapher at once.

• Single Payload Footprint: KeeperChunkRetentionStore maintains a sorted map EventSequence -> StoryChunk*. The actual LogEvent payload resides solely in the StoryChunk's map node accessed via StoryChunk::findEvent().
• Freed once persisted: A chunk is sent to ChronoGrapher when it seals, but stays in keeper memory until ChronoGrapher confirms its events are written to HDF5 and no tail-index entry points at it. See Durable Chunk Retention.

---

3. Two-Phase Scatter/Gather Protocol

A story's events are striped across its assigned ChronoKeepers. To gather the global last $N$ events without transferring redundant payloads, the client executes a two-phase protocol:

Phase 1 — Scatter (Keys Only)

1. The client issues asynchronous tail_get_sequences(story_id, n) RPCs concurrently to all keepers assigned to the story.
2. Each keeper queries KeeperChunkRetentionStore::getTailSequences and returns up to $n$ newest EventSequence keys (each a tuple of (chrono_time, clientId, index)).
3. Total Phase 1 latency is bounded by the slowest keeper rather than the sum, and per-RPC timeouts (kTailReadRpcTimeoutMs = 5000ms) isolate failed keepers.

Selection

The client merges all returned keys into a single globally-ordered collection, picks the largest $N$ sequences, and groups them by their owning keeper.

Phase 2 — Gather (Payloads for Winners Only)

1. The client issues concurrent tail_get_events(story_id, selected_seqs) RPCs to each keeper that holds winning keys.
2. Each keeper retrieves the specific LogEvent payloads using KeeperChunkRetentionStore::getTailEvents.
3. The client inserts returned events into a std::map<EventSequence, Event> to ensure cross-keeper deduplication and ascending ordering, returning a std::vector<Event>.

Network Guarantee: Exactly $N$ event payloads are transmitted across the network, independent of the number of keepers in the recording group.

---

4. Retention and Memory Bounds

Archival no longer waits for the tail: every chunk is sent to ChronoGrapher as soon as it seals, and stays in keeper memory until ChronoGrapher confirms it is written to HDF5 (see Durable Chunk Retention). The tail index is bounded on its own:

1. Capacity Eviction (tail_capacity, default 65536): When a story's tail index holds more events than tail_capacity, the oldest entries leave the index. That only shortens the tail playback can return; the chunk itself is freed once it is also persisted and archive_visibility_delay_secs has passed (see Durable Chunk Retention).
2. Release at Retirement: When a story retires on the keeper (no client has it acquired and its acceptance window has passed), its whole tail index is released and playback returns nothing more from this keeper. A low-volume story that never fills tail_capacity therefore keeps its last events in memory only while it is recorded.
3. No Time-Based Age-Out: tail_retention_secs is removed. A low-volume story is archived anyway, because its chunks are sent at seal.
4. Between the Two Phases: Sequences returned by Phase 1 are not pinned. If new events push a sequence out of the tail index before Phase 2 arrives, Phase 2 skips it and the read returns fewer than $N$ events.
5. Shutdown Wait (waitUntilDurable): During clean shutdown, before the extraction threads stop, the keeper sends again every chunk ChronoGrapher has not acknowledged and waits until ChronoGrapher confirms every chunk written, or shutdown_confirm_timeout_secs passes.

---

5. Live Tail from Unsealed Chunks (live_tail_read)

By default, events become visible in playback() once their chunk seals (chunk_duration + acceptance_window, ~25–30s). For latency-critical streaming applications, ChronoKeeper provides the optional live_tail_read setting.

• Mechanism: KeeperStoryPipeline implements the ActiveTailSource interface, registering with KeeperChunkRetentionStore. When enabled, getTailSequences unions the sealed tail with the unsealed active timeline (storyTimelineMap), and getTailEvents falls back to findActiveEvent() under sequencingMutex.
• Visibility Latency: Send-to-visible latency drops from ~20 seconds to ~0.5 s on average, bounded by ~1 s. An event becomes readable only once collectIngestedEvents() merges it into storyTimelineMap, and that maintenance tick runs once per second, so the tick interval — not the RPC — sets this floor.
• Provisional Ordering: Because events inside the active acceptance window arrive out of order from distributed clients, active window reads are provisional (a late event may sort behind an already-seen event until sealed).

---

6. Orphan Event & Chunk Recovery

To prevent data loss when stories are released, retired, or unhooked:

• Keeper Orphan Recovery (sealOrphanedEvents): Late client events arriving after a story pipeline has decayed are rescued from IngestionQueue's orphan queue, sealed into a recovery StoryChunk with retained chronicle/story names, and sent for archival.
• Grapher Orphan Adoption (adoptOrphanChunks): If ChronoGrapher receives an orphan chunk for an unregistered story, it adopts the chunk, instantiates a temporary pipeline using the chunk's embedded metadata, archives it to HDF5, and schedules graceful retirement.

---

7. Performance Characteristics

Metric	Persistent Replay (ReplayStory)	Sealed Tail Read (playback)	Live Tail Read (live_tail_read = true)
Query Latency	~1000–2500 ms (disk / HDF5)	~0.4 ms median, 1–5 ms p90–p99 (keeper RAM)	~0.4 ms median, 1–5 ms p90–p99 (keeper RAM)
Visibility Delay	After chunk seal (~25–30 s); recent events come from keeper memory	After chunk seal (~25–30 s)	~0.5 s mean, ~1 s max (one ingestion tick)
Network Overhead	Full range transfer via Player	Exactly $N$ payloads	Exactly $N$ payloads
Consistency	Strictly ordered / final	Strictly ordered / final	Provisional within acceptance window