GeminiDB Time Series IoV Solution

Application Scenarios

Fueled by immense popularity of intelligent new energy vehicles, time series data generated in real time experienced tremendous growth. There were urgent demands for vehicle enterprises and owners to query the real-time status of vehicles, but the traditional HBase-based vehicle monitoring platform cannot meet the requirements.

Solution Overview

The GeminiDB time series IoV solution is designed for real-time queries of vehicle data through the dedicated Influx API, which parses, sorts, merges, analyzes, and writes millions of time series data of vehicles in real time. This solution supports high compression ratio and separation of cold and hot data, effectively reducing costs.

• Figure 1 shows the technical architecture of HBase.
  • Vehicle data is collected and then reported to Kafka.
  • Real-time data is written to HBase through Flink. All monitoring data is written to HBase as a character string.
  • HBase supports point queries.
  • Offline data is parsed by Flink and written to Hive using Spark for analysis.
  Figure 1 HBase technical architecture
  

• Figure 2 shows the technical architecture of GeminiDB Influx API.
  • Vehicle data is collected and then reported to Kafka.
  • Real-time data is parsed by Flink and written to GeminiDB Influx API.
  • GeminiDB Influx API supports real-time queries and data analysis.
  • GeminiDB Influx API can convert data into Parquet format, which can be directly read and analyzed by Hive. For details, see Converting Data into a Parquet file and Exporting the Data to OBS.
  Figure 2 Technical architecture of GeminiDB Influx API
  

Advantages

• Parsing and writing massive volumes of data in real time; simplifying application development

HBase: Thousands of monitoring metrics reported by vehicles are written into HBase as character strings. When an application reads a metric, it needs to read and parse all character strings. This process is complex and inefficient.

GeminiDB Influx API: Thousands of monitoring metrics reported by vehicles are directly written into GeminiDB Influx instances as thousands of columns. Metrics can be directly queried and without being parsed again.

• Automatically sorting and combining data; simplifying the intermediate process

Multi-dimensional metric data reported at the same time point by vehicles is processed by different components under different network delays, so the data cannot be reported and written at a time in sequence.

HBase: Applications need to use Spark to combine and sort HBase data, which is complex and cannot meet real-time query requirements.

GeminiDB Influx API: When time series data is written, it is automatically merged and sorted. Applications can directly access GeminiDB Influx instances to obtain the result.

• Real-time analysis

Traditional HBase: Raw data consists of unresolved character strings and cannot be used directly. It must be read and resolved, or it cannot meet requirements for efficiently analyzing and processing large volumes. Typically, data analysis involves synchronizing data to platforms like Hive, which lacks real-time capabilities.

GeminiDB Influx API: Data can be queried and analyzed based on metrics at a time. Just one database is enough for effective real-time query and analysis.

• High compression ratio

HBase: The compression algorithm can be set only by column family. Only the GZIP, Snappy, LZO, and LZ4 algorithms are supported.

GeminiDB Influx API: Different compression algorithms are used for data types of each column. Multiple compression algorithms, such as Simple8b, Delta, Delta-Of-Delta, RLE, ZigZag, ZSTD, Snappy, and bit-packing, are supported. The compression ratio is 10 times that of HBase.

• Separation of hot and cold data

Users can configure hot and cold data policies to automatically dump data to cold storage without changing applications, which effectively reducing the overall cost.