Exploring the Scalable and Decoupled Architecture of InfluxDB 3.0

2023-08-16

InfluxData recently announced the system architecture of its latest time series database, InfluxDB 3.0. The architecture consists of four main components responsible for data ingestion, querying, compression, and garbage collection, and includes two primary storage types. It is also suitable for running the database locally on-site and on major cloud service providers. The central pillar of the InfluxDB 3.0 architecture is the decoupling design of the main components by engineers. These components do not directly communicate with each other but communicate through directories and object storage. For example, the data ingester and query engine are unaware of the existence of the compressor and garbage collector. As a result, these components can be independently scaled and deployed in different ways. The data ingestion component (blue section) is responsible for the process of inputting data into the database. Users write data to the ingest router, which shards the data to an ingester. The number of ingesters can be scaled based on the data workload. Each ingester identifies tables, validates data schemas, partitions data based on the "time" column, deduplicates it, and persists it as Parquet files. The ingester also updates the directory and notifies other components of new data arrivals. InfluxDB optimizes the write path, keeping the write latency in the millisecond range. The data querying component (green section) handles user queries using SQL or InfluxQL. Users send queries to the query router, which forwards them to the querier. The querier reads the required data, builds a query plan, executes the query, and returns the results. Queriers can be scaled based on the query workload. They perform tasks such as caching metadata, reading and caching data, communicating with the ingester to fetch unpersisted data, and building and executing optimal query plans. Data compression (red section) is the process of addressing the performance impact of storing many small files in object storage. The compressor runs as a background job, reading newly ingested files and compressing them into fewer, larger, and non-overlapping files. This process improves query performance significantly by reducing IO during queries. The number of compressors can be scaled based on the compression workload, considering factors such as the number of tables with new data files, the size of the files, and the overlap between new and existing files. The garbage collection component (pink section) is responsible for managing data retention and space reclamation within the database. It performs soft and hard deletions as background jobs. The directory includes metadata about the database, tables, columns, and files. InfluxDB uses a Postgres-compatible database to manage this directory. On the other hand, object storage only contains Parquet files, which can be stored on local disks, Amazon S3, Azure Blob Storage, Google Cloud Storage, and other platforms.

RECENT AI NEWS

RECENT AI TOOLS