Skip to main content
Version: 2.5.0

ChronoKeeper

ChronoKeeper is the component responsible for fast ingestion of the log events coming from the client processes, efficiently grouping them into partial StoryChunks and moving them down to the lower tier of ChronoLog system.

Most ChronoLog deployments would have ChronoKeeper processes running on the majority of the compute nodes and each Recording Group is expected to have multiple ChronoKeeper processes.

CHRONOKEEPER COMPONENTSCHRONOKEEPERIndividual Events are streamed from theClient processes to the ChronoKeepersKeeper Recording ServiceEvent Ingestion QueueKEEPER DATA STOREKeeper Data StoreStory 1 PipelineStory 2 PipelineStory 3 PipelineChronoKeeper Registrationand Heartbeat MsgsKeeper Registry ClientRegistration & Heartbeat msgsDataStore Admin ServiceStart/Stop Story recording notificationsNotifications from the ChronoVisorto Start & Stop Story RecordingStoryChunk Extraction QueueCSV File ExtractorRDMA ExtractorPartial StoryChunks are drained to the ChronoGrapher or extracted to the local files
ComponentDescription
Keeper Data StoreThe main ChronoKeeper module. A collection of Story pipelines for all actively recorded Stories in the Recording Group. Instantiates a Story pipeline when ChronoKeeper receives a StartStoryRecording notification and dismantles it after StopStoryRecording. Sequencing threads are responsible for ordering ingested log events and grouping them into time-range bound StoryChunks. (See ChronoLog Story Pipeline Data Model for reference.)
Keeper Recording ServiceListens to incoming streams of log events from client applications and passes the events to the Ingestion Queue module for processing.
Ingestion QueueA collection of IngestionHandles for actively recorded stories the ChronoKeeper is expecting events for. Receives log events from the Keeper Recording Service and attributes them to the appropriate Story Ingestion Handles. Story Ingestion Handle is part of the Story Pipeline object exposed to the Ingestion Queue at instantiation time.
StoryChunk Extraction QueueA mutex-protected deque of StoryChunk pointers, serving as the communication boundary between the Keeper Data Store and StoryChunk Extractor modules.
StoryChunk ExtractorsChronoKeeper has two extractor types: CSVFileStoryChunkExtractor writes retired StoryChunks to a configurable local POSIX directory; StoryChunkExtractorRDMA drains retired StoryChunks to ChronoGrapher over an RDMA communication channel. Extractors can be chained if needed.
KeeperRegistryClientThe client side of RPC communication between the ChronoKeeper process and ChronoVisor's Recording Process Registry Service. Sends register/unregister and heartbeat/statistics messages to ChronoVisor.
DataStoreAdminServiceListens to Start/Stop Story recording notifications from ChronoVisor. These notifications trigger instantiation or dismantling of the appropriate StoryPipelines based on client data access requests.