ChronoKeeper

ChronoKeeper is the component responsible for fast ingestion of the log events coming from the client processes, efficiently grouping them into partial StoryChunks and moving them down to the lower tier of ChronoLog system.

Most ChronoLog deployments would have ChronoKeeper processes running on the majority of the compute nodes and each Recording Group is expected to have multiple ChronoKeeper processes.

Component	Description
Keeper Data Store	The main ChronoKeeper module. A collection of Story pipelines for all actively recorded Stories in the Recording Group. Instantiates a Story pipeline when ChronoKeeper receives a StartStoryRecording notification and dismantles it after StopStoryRecording. Sequencing threads are responsible for ordering ingested log events and grouping them into time-range bound StoryChunks. (See ChronoLog Story Pipeline Data Model for reference.)
Keeper Recording Service	Listens to incoming streams of log events from client applications (record_event), serves on-demand tail reads directly from keeper memory via tail_get_sequences and tail_get_events, and serves ChronoPlayer's replay range fetches (story_range_fetch).
Keeper Chunk Retention Store (KeeperChunkRetentionStore)	Owns every sealed StoryChunk until ChronoGrapher confirms it is written to HDF5, and indexes the most recent sealed events per story to serve playback(N) from memory with no payload copy. See Durable Chunk Retention.
Ingestion Queue	A collection of IngestionHandles for actively recorded stories the ChronoKeeper is expecting events for. Receives log events from the Keeper Recording Service and attributes them to the appropriate Story Ingestion Handles. Also recovers late orphaned events for retired stories before archival.
StoryChunk Extraction Queue	A mutex-protected deque of StoryChunk pointers, serving as the communication boundary between the Chunk Retention Store and the ExtractionModule.
ExtractionModule	A configurable pipeline of extractors that consumes retired StoryChunks from the extraction queue and forwards each chunk through every extractor in the chain. The module owns a fixed-size pool of extraction-stream threads (extraction_stream_count) that drain the queue in parallel. Available extractor types: csv_extractor writes chunks to a local POSIX directory; single_endpoint_rdma_extractor drains chunks to a single RDMA endpoint (typically a ChronoGrapher); dual_endpoint_rdma_extractor fans chunks out to two RDMA endpoints simultaneously (typically a ChronoGrapher and a ChronoPlayer). The chain is assembled from JSON at startup — see Server Configuration → ExtractionModule.
KeeperRegistryClient	The client side of RPC communication between the ChronoKeeper process and ChronoVisor's Recording Process Registry Service. Sends register/unregister and heartbeat/statistics messages to ChronoVisor.
DataStoreAdminService	Listens to Start/Stop Story recording notifications from ChronoVisor, which trigger instantiation or dismantling of the appropriate StoryPipelines, and to ChronoGrapher's persisted-watermark reports (report_story_watermarks), which let the retention store free persisted chunks.

---

On-Demand Tail Reading

In ChronoLog 3.1+, ChronoKeeper directly serves tail-read queries from client applications (StoryHandle::playback):

1. tail_get_sequences(story_id, n): Returns up to $n$ newest EventSequence keys for a story held in keeper memory.
2. tail_get_events(story_id, seqs): Returns the full LogEvent payloads for the specific sequences requested by the client after client-side global merging.

For full architectural details on zero-copy indexing, capacity bounds, and live unsealed reads, see On-Demand Tail Read Architecture. How sealed chunks are kept until they are persisted is described in Durable Chunk Retention.