On-Demand Tail Read Architecture & Design

ChronoLog 3.1 introduces On-Demand Keeper Tail Read (playback), allowing client applications to query the most recent $N$ events of a story directly from ChronoKeeper in-memory buffers with millisecond-scale latency (~0.4 ms median, 1–5 ms at p90–p99), bypassing the persistent HDF5 tier and ChronoPlayer query service.

---

1. Motivation & Architecture Overview

Historically, retrieving logged events required querying the ChronoPlayer via Client::ReplayStory. While ReplayStory provides complete historical replays across persistent HDF5 files, it incurs indexing, disk I/O, and cross-tier network latencies (~1–2 seconds per query).

Many real-time use cases — such as live dashboards, monitoring alerts, stream processing, and rapid failover recovery — only need the last $N$ events produced by a story.

On-Demand Tail Read provides:

• Direct Client $\leftrightarrow$ Keeper Communication: Clients communicate directly with the story's assigned ChronoKeepers via lightweight Thallium RPCs. No ChronoPlayer or ChronoVisor is in the read path.
• Zero-Copy In-Memory Indexing: Events are indexed directly by their memory pointers in sealed chunks retained before extraction.
• Two-Phase Scatter/Gather Protocol: Minimal network transfer — exactly $N$ payloads cross the wire across the entire keeper group.
• Writer-Mode Compatibility: Clients instantiated in writer-only mode (ClientPortalServiceConf only) can perform tail reads without configuring query endpoints.

Trying it without writing code

The interactive CLI exposes this path as -p <num_events>, which tail-reads the Story you currently hold:

-a -s my_chronicle my_story
-p 10

See the CLI API Reference. Remember that an Event is only visible once its chunk seals, so allow ~25–30 s after writing (or enable live_tail_read).

---

2. In-Memory Retention & Single Payload Model

In ChronoKeeper, when a StoryChunk seals (past chunk_duration + acceptance_window), ownership is handed to KeeperTailStore rather than directly to the extraction queue.

• Single Payload Footprint: KeeperTailStore maintains a sorted map EventSequence -> StoryChunk*. The actual LogEvent payload resides solely in the StoryChunk's map node accessed via StoryChunk::findEvent().
• Deferred Archival: A retained chunk is forwarded to the StoryChunkExtractionQueue for RDMA transfer to ChronoGrapher only after its events age out of the tail window, are evicted by capacity limits, or are handed over by the shutdown flush (see §4).

---

3. Two-Phase Scatter/Gather Protocol

A story's events are striped across its assigned ChronoKeepers. To gather the global last $N$ events without transferring redundant payloads, the client executes a two-phase protocol:

Phase 1 — Scatter (Keys Only)

1. The client issues asynchronous tail_get_sequences(story_id, n) RPCs concurrently to all keepers assigned to the story.
2. Each keeper queries KeeperTailStore::getTailSequences and returns up to $n$ newest EventSequence keys (each a tuple of (chrono_time, clientId, index)).
3. Total Phase 1 latency is bounded by the slowest keeper rather than the sum, and per-RPC timeouts (kTailReadRpcTimeoutMs = 5000ms) isolate failed keepers.

Selection

The client merges all returned keys into a single globally-ordered collection, picks the largest $N$ sequences, and groups them by their owning keeper.

Phase 2 — Gather (Payloads for Winners Only)

1. The client issues concurrent tail_get_events(story_id, selected_seqs) RPCs to each keeper that holds winning keys.
2. Each keeper retrieves the specific LogEvent payloads using KeeperTailStore::getTailEvents.
3. The client inserts returned events into a std::map<EventSequence, Event> to ensure cross-keeper deduplication and ascending ordering, returning a std::vector<Event>.

Network Guarantee: Exactly $N$ event payloads are transmitted across the network, independent of the number of keepers in the recording group.

---

4. Retention Lifecycle & Starvation Prevention

Retaining sealed chunks in keeper memory introduces the need for robust memory bounds and archival guarantees:

Archival Safety Mechanisms

1. Time-Based Age-Out (tail_retention_secs, default 60s): Sealed chunks age out after tail_retention_secs beyond their end time and are forwarded to the extraction queue. This ensures low-volume stories that never fill tail_capacity are still archived to persistent storage.
2. Capacity Eviction (tail_capacity, default 65536): When total retained events for a story exceed tail_capacity, oldest events are evicted immediately. Once a chunk has zero remaining indexed events, it is stashed to the extraction queue.
3. Pin Protection (kPinTicks) with Anti-Starvation (pinDeferredTicks):
   • When Phase 1 returns sequences from a chunk, that chunk is pinned for kPinTicks (~3 maintenance ticks / ~3 seconds) so a concurrent age-out does not delete the payload before Phase 2 arrives.
   • To prevent rapid polling from permanently deferring archival, pins defer age-out for at most kPinTicks consecutive ticks per story (pinDeferredTicks). Archival proceeds regardless of active reads.
4. Shutdown Flush (flushRetainedChunks): During clean shutdown, flushRetainedChunks() explicitly hands all retained chunks to the extraction queue before extraction threads are stopped, preventing data loss.

---

5. Live Tail from Unsealed Chunks (live_tail_read)

By default, events become visible in playback() once their chunk seals (chunk_duration + acceptance_window, ~25–30s). For latency-critical streaming applications, ChronoKeeper provides the optional live_tail_read setting.

• Mechanism: KeeperStoryPipeline implements the ActiveTailSource interface, registering with KeeperTailStore. When enabled, getTailSequences unions the sealed tail with the unsealed active timeline (storyTimelineMap), and getTailEvents falls back to findActiveEvent() under sequencingMutex.
• Visibility Latency: Send-to-visible latency drops from ~20 seconds to ~0.5 s on average, bounded by ~1 s. An event becomes readable only once collectIngestedEvents() merges it into storyTimelineMap, and that maintenance tick runs once per second, so the tick interval — not the RPC — sets this floor.
• Provisional Ordering: Because events inside the active acceptance window arrive out of order from distributed clients, active window reads are provisional (a late event may sort behind an already-seen event until sealed).

---

6. Orphan Event & Chunk Recovery

To prevent data loss when stories are released, retired, or unhooked:

• Keeper Orphan Recovery (sealOrphanedEvents): Late client events arriving after a story pipeline has decayed are rescued from IngestionQueue's orphan queue, sealed into a recovery StoryChunk with retained chronicle/story names, and sent for archival.
• Grapher Orphan Adoption (adoptOrphanChunks): If ChronoGrapher receives an orphan chunk for an unregistered story, it adopts the chunk, instantiates a temporary pipeline using the chunk's embedded metadata, archives it to HDF5, and schedules graceful retirement.

---

7. Performance Characteristics

Metric	Persistent Replay (ReplayStory)	Sealed Tail Read (playback)	Live Tail Read (live_tail_read = true)
Query Latency	~1000–2500 ms (disk / HDF5)	~0.4 ms median, 1–5 ms p90–p99 (keeper RAM)	~0.4 ms median, 1–5 ms p90–p99 (keeper RAM)
Visibility Delay	After HDF5 archival (~60–240 s)	After chunk seal (~25–30 s)	~0.5 s mean, ~1 s max (one ingestion tick)
Network Overhead	Full range transfer via Player	Exactly $N$ payloads	Exactly $N$ payloads
Consistency	Strictly ordered / final	Strictly ordered / final	Provisional within acceptance window