Skip to main content
Version: 3.4.0

ChronoKeeper

ChronoKeeper is the component responsible for fast ingestion of the log events coming from the client processes, efficiently grouping them into partial StoryChunks and moving them down to the lower tier of ChronoLog system.

Most ChronoLog deployments would have ChronoKeeper processes running on the majority of the compute nodes and each Recording Group is expected to have multiple ChronoKeeper processes.

CHRONOKEEPER COMPONENTSCHRONOKEEPERIndividual Events are streamed from theClient processes to the ChronoKeepersKeeper Recording Serviceaccepted writes (record_event)durable writes (record_event_durable)Keeper Durable WriterAppends the event to the WAL, answers CL_SUCCESSafter the sync, then passes it to the queueKeeper WALGroup commit and fdatasync to segmentson a local drive; released once ChronoGrapherconfirms, replayed into chunks at a restartEvent Ingestion QueueKEEPER DATA STOREKeeper Data StoreStory 1 PipelineStory 2 PipelineStory 3 PipelineChronoKeeper Registrationand Heartbeat MsgsKeeper Registry ClientRegistration & Heartbeat msgsDataStore Admin ServiceStart/Stop Story recording notificationsNotifications from the ChronoVisorto Start & Stop Story RecordingStoryChunk Extraction QueueCSV File ExtractorRDMA ExtractorPartial StoryChunks are drained to the ChronoGrapher or extracted to the local files
ComponentDescription
Keeper Data StoreThe main ChronoKeeper module. A collection of Story pipelines for all actively recorded Stories in the Recording Group. Instantiates a Story pipeline when ChronoKeeper receives a StartStoryRecording notification and dismantles it after StopStoryRecording. Sequencing threads are responsible for ordering ingested log events and grouping them into time-range bound StoryChunks. (See ChronoLog Story Pipeline Data Model for reference.)
Keeper Recording ServiceListens to incoming streams of log events from client applications (record_event for accepted writes, record_event_durable for durable ones), serves on-demand tail reads directly from keeper memory via tail_get_sequences and tail_get_events, and serves ChronoPlayer's replay range fetches (story_range_fetch).
Keeper Durable Writer (KeeperDurableWriter)Answers record_event_durable. It checks that the story id matches the chronicle and story names, refuses the write with CL_ERR_DURABILITY_FULL above durable_cap_mb of chunks ChronoGrapher has not confirmed, appends the event to the WAL, and answers CL_SUCCESS only after the WAL's sync; then it passes the event to the Ingestion Queue. A retry of an event the WAL already holds gets the first attempt's answer. See Keeper WAL.
Keeper WAL (KeeperWal)The write-ahead log, used only when WAL.enabled is set. It writes every durable event queued while the previous fdatasync ran as one group to pre-zeroed segments on a local drive, and answers the whole group after the sync. The retention store tells it when a chunk is confirmed, and a segment is deleted once none of its events are pending. At startup the keeper recovers the WAL and replays its events into sealed chunks before it takes requests. See Keeper WAL.
Keeper Chunk Retention Store (KeeperChunkRetentionStore)Owns every sealed StoryChunk until ChronoGrapher confirms it is written to HDF5, and indexes the most recent sealed events per story to serve playback(N) from memory with no payload copy. See Durable Chunk Retention.
Ingestion QueueA collection of IngestionHandles for actively recorded stories the ChronoKeeper is expecting events for. Receives log events from the Keeper Recording Service and attributes them to the appropriate Story Ingestion Handles. Also recovers late orphaned events for retired stories before archival.
StoryChunk Extraction QueueA mutex-protected deque of StoryChunk pointers, serving as the communication boundary between the Chunk Retention Store and the ExtractionModule.
ExtractionModuleA configurable pipeline of extractors that consumes retired StoryChunks from the extraction queue and forwards each chunk through every extractor in the chain. The module owns a fixed-size pool of extraction-stream threads (extraction_stream_count) that drain the queue in parallel. Available extractor types: csv_extractor writes chunks to a local POSIX directory; single_endpoint_rdma_extractor drains chunks to a single RDMA endpoint (typically a ChronoGrapher); dual_endpoint_rdma_extractor fans chunks out to two RDMA endpoints simultaneously (typically a ChronoGrapher and a ChronoPlayer). The chain is assembled from JSON at startup — see Server Configuration → ExtractionModule.
KeeperRegistryClientThe client side of RPC communication between the ChronoKeeper process and ChronoVisor's Recording Process Registry Service. Sends register/unregister and heartbeat/statistics messages to ChronoVisor.
DataStoreAdminServiceListens to Start/Stop Story recording notifications from ChronoVisor, which trigger instantiation or dismantling of the appropriate StoryPipelines, and to ChronoGrapher's persisted-watermark reports (report_story_watermarks), which let the retention store free persisted chunks.

On-Demand Tail Reading​

In ChronoLog 3.1+, ChronoKeeper directly serves tail-read queries from client applications (StoryHandle::playback):

  1. tail_get_sequences(story_id, n): Returns up to $n$ newest EventSequence keys for a story held in keeper memory.
  2. tail_get_events(story_id, seqs): Returns the full LogEvent payloads for the specific sequences requested by the client after client-side global merging.

For full architectural details on zero-copy indexing, capacity bounds, and live unsealed reads, see On-Demand Tail Read Architecture. How sealed chunks are kept until they are persisted is described in Durable Chunk Retention.