Help Center/ Document Database Service/ Best Practices/ Using DDS to Store and Analyze Log Data
Updated on 2025-03-26 GMT+08:00
View PDF
Share

Using DDS to Store and Analyze Log Data

Log data plays an important role in the O&M and analysis of modern applications. It not only helps monitor the health status of applications, but also provides valuable data for service insight, user experience optimization and business decision-making. However, as the data volume increases sharply, traditional log storage and analysis become insufficient. This document describes how to use Document Database Service (DDS) to efficiently store and analyze log data. DDS 4.2 and later versions use RocksDB as the underlying storage engine. RocksDB has good performance in scenarios where write operations are more than read operations, such as log storage and analysis.

By properly designing a log storage structure, optimizing write and query policies, and using data sharding, DDS can efficiently store and analyze massive amounts of log data, providing strong support for application O&M and service analysis. In addition to powerful data storage, DDS provides flexible configuration and optimization policies to help users easily cope with challenges and mine infinite data value in the era of data explosion.

Log Data Storage

In the Internet of Things (IoT) field, device log data plays an important role. It not only helps monitor the status of devices, but also provides detailed information about device usage modes and fault prediction. For example, a log record of a smart home security system may include information such as a device ID, a timestamp, an event type (such as lock opening status or motion detection), a device status, and a possible error code.

  • Log data example
    The following is a typical IoT device log record: 
    DeviceID: 001, Timestamp: 2023-04-05T14:30:00Z, Event: DoorLockOpened, DeviceStatus: Active, Error: None
  • Storage mode optimization
    When data is stored in DDS, logs are converted into structured documents to extract key fields. For example: 
    {   _id: ObjectId('5f442120eb03305789000000'),
    device_id: "001",
    timestamp: ISODate("2023-04-05T14:30:00Z"),
    event: "DoorLockOpened",
    device_status: "Active",
    error: "None"
}

    Irrelevant fields are filtered out based on analysis requirements to save storage space. For example, if you want to analyze the error status of a device that is not concerned, you can omit the error field.

Log Writing and Performance Optimization

DDS replica set instances provide data redundancy and highly reliable storage. To support high-concurrency log writing, you can customize the writeConcern parameter. For example, using {w: 0} can achieve the highest write throughput, but at the cost of data durability. For key device logs, a more secure level, such as {w: 1} or {w: "majority"}, is recommended. This setting allows writing multiple logs in batches and can significantly improve efficiency. 
replica:PRIMARY> log = {
...     "device_id": "001",
...     "timestamp": ISODate("2023-04-05T14:30:00Z"),
...     "event": "DoorLockOpened",
...     "device_status": "Active",
...     "error": "None" ... };
// Write using {w: 0}.
replica:PRIMARY> db.getSiblingDB("iot_logs").events.insert(log, {w: 0})
WriteResult({ "nInserted" : 1 })
// Write using {w: 1}.
replica:PRIMARY> db.getSiblingDB("iot_logs").events.insert(log, {w: 1})
WriteResult({ "nInserted" : 1 })
// Write using {w: "majority"}.
replica:PRIMARY> db.getSiblingDB("iot_logs").events.insert(log, {w: "majority"})
WriteResult({ "nInserted" : 1 })
// Batch write at a time:
replica:PRIMARY> logs = [
... { "device_id": "002", "timestamp": ISODate("2023-04-06T09:45:00Z"), "event": "MotionDetected", "device_status": "Active", "error": "None"},
... { "device_id": "003", "timestamp": ISODate("2023-04-07T16:15:00Z"), "event": "TemperatureAlarm", "device_status": "Active", "error": "None"},
... {"device_id": "004", "timestamp": ISODate("2023-04-08T20:30:00Z"), "event": "WindowOpened", "device_status": "Active", "error": "None"}
... ]
replica:PRIMARY> db.getSiblingDB("iot_logs").events.insertMany(logs, {w: 0})

Log Query and Analysis

  • Log query example
    Query all events of a specified device in a specified time range. 
    db.getSiblingDB("iot_logs").events.find({"device_id": "001", "timestamp": {"$gte": ISODate("2023-04-05T00:00:00Z"), "$lt": ISODate("2023-04-06T00:00:00Z")}})
  • Create indexes to improve query efficiency.
    Creating indexes for the device_id and timestamp fields can accelerate the query in the preceding scenario. 
    replica:PRIMARY> db.getSiblingDB("iot_logs").events.createIndex({"device_id": 1, "timestamp": 1})
  • Complex analysis.

    The DDS aggregation framework can be used for more complex query and analysis. For details, see Aggregation Operations.

Data Sharding and Scaling

As logs spike, the log data volume increases exponentially. A single database node cannot meet the storage and query requirements of massive data. DDS uses the sharding technology to provide the horizontal scalability, effectively share data storage and query stress, and ensure high availability and high performance of the system.

  • Data sharding

    Data sharding means dividing a dataset into multiple parts and store them on different database nodes (shards). Each shard stores a part of a dataset, and data is distributed based on the shard key. A shard key policy determines the data distribution mode and affects the write and query performance.

  • Shard key policies

    When selecting a shard key, consider the following points:

    • Even distribution: Shard keys must ensure that data is evenly distributed among shards to avoid hotspots.
    • Query mode: Shard keys must match common query modes to improve query efficiency.
    • Data access mode: If the data access mode is time-based, you can use timestamps or time-based fields as shard keys.

    For details, see Sharding.

  • Device ID-based sharding
    Assume that there are a large number of IoT devices and each device generates a large number of logs. You can use device_id as the shard key to ensure that the log data of each device is stored on different shards, achieving even data distribution. Example code: 
    db.getSiblingDB("iot_logs").events.createIndex({"device_id": 1})
sh.enableSharding("iot_logs")
sh.shardCollection("iot_logs.events", {"device_id": 1 })
    • When creating a sharded collection, ensure that the shard key is unique in the data to achieve even data distribution.
    • When selecting a shard key, consider the data access mode to optimize the query performance.
    • Sharded cluster management, such as adding or deleting shards, should be adjusted based on data growth and query requirements.

Automatically Deleting Expired Data

  • TTL indexes: Expired documents are deleted automatically, for example, you can set the period to 30 days. 
    db.eventlog.createIndex( { "timestamp": 1 }, { expireAfterSeconds: 30 * 24 * 60 * 60 } )
  • Capped collections: The collection size is limited, and the earliest document is automatically deleted. In capped collections, documents are inserted and retrieved according to the insertion order. Capped collections work similarly to circular buffers: Once a collection is full, it makes space for new documents by overwriting the earliest document in the collection. 
    // Create a capped collection when creating a collection.
db.getSiblingDB("iot_logs").dropDatabase()
db.getSiblingDB("iot_logs").createCollection( "events", { capped: true, size: 5242880 } )
  • Periodic archiving: Log data is archived by month to facilitate historical data management and query. 
    db.getSiblingDB("iot_logs").events.renameCollection("events202301")

    In a DB instance, the total number of databases cannot exceed 200, and the total number of collections cannot exceed 500. If there are too many collections, the memory may be overloaded. In addition, the performance for restarting a DB instance and performing a primary/standby switchover may deteriorate due to too many collections, which affects the high availability performance in emergencies. You are advised to periodically delete unnecessary collections.