ChronoKeeper

ChronoKeeper is the component responsible for fast ingestion of the log events coming from the client processes, efficiently grouping them into partial StoryChunks and moving them down to the lower tier of ChronoLog system.

Most ChronoLog deployments would have ChronoKeeper processes running on the majority of the compute nodes and each Recording Group is expected to have multiple ChronoKeeper processes.

Component	Description
Keeper Data Store	The main ChronoKeeper module. A collection of Story pipelines for all actively recorded Stories in the Recording Group. Instantiates a Story pipeline when ChronoKeeper receives a StartStoryRecording notification and dismantles it after StopStoryRecording. Sequencing threads are responsible for ordering ingested log events and grouping them into time-range bound StoryChunks. (See ChronoLog Story Pipeline Data Model for reference.)
Keeper Recording Service	Listens to incoming streams of log events from client applications (record_event) and serves on-demand tail reads directly from keeper memory via tail_get_sequences and tail_get_events RPCs.
Keeper Tail Store (KeeperTailStore)	Maintains an in-memory index of recent sealed-but-not-yet-extracted StoryChunk pointers per story. Serves low-latency playback(N) queries directly from memory with zero payload copy. Manages chunk retention, capacity bounds (tail_capacity), time-based age-out (tail_retention_secs), and pin protection (kPinTicks). Optionally unions with active unsealed chunks when live_tail_read is enabled.
Ingestion Queue	A collection of IngestionHandles for actively recorded stories the ChronoKeeper is expecting events for. Receives log events from the Keeper Recording Service and attributes them to the appropriate Story Ingestion Handles. Also recovers late orphaned events for retired stories before archival.
StoryChunk Extraction Queue	A mutex-protected deque of StoryChunk pointers, serving as the communication boundary between the Keeper Data Store / Tail Store and the ExtractionModule.
ExtractionModule	A configurable pipeline of extractors that consumes retired StoryChunks from the extraction queue and forwards each chunk through every extractor in the chain. The module owns a fixed-size pool of extraction-stream threads (extraction_stream_count) that drain the queue in parallel. Available extractor types: csv_extractor writes chunks to a local POSIX directory; single_endpoint_rdma_extractor drains chunks to a single RDMA endpoint (typically a ChronoGrapher); dual_endpoint_rdma_extractor fans chunks out to two RDMA endpoints simultaneously (typically a ChronoGrapher and a ChronoPlayer). The chain is assembled from JSON at startup — see Server Configuration → ExtractionModule.
KeeperRegistryClient	The client side of RPC communication between the ChronoKeeper process and ChronoVisor's Recording Process Registry Service. Sends register/unregister and heartbeat/statistics messages to ChronoVisor.
DataStoreAdminService	Listens to Start/Stop Story recording notifications from ChronoVisor. These notifications trigger instantiation or dismantling of the appropriate StoryPipelines based on client data access requests.

---

On-Demand Tail Reading

In ChronoLog 3.1, ChronoKeeper directly serves tail-read queries from client applications (StoryHandle::playback):

1. tail_get_sequences(story_id, n): Returns up to $n$ newest EventSequence keys for a story held in keeper memory.
2. tail_get_events(story_id, seqs): Returns the full LogEvent payloads for the specific sequences requested by the client after client-side global merging.

For full architectural details on zero-copy indexing, pinning, capacity bounds, and live unsealed reads, see On-Demand Tail Read Architecture.