DDS Aggregation

DDS aggregation is an advanced data processing capability delivered by Huawei Cloud Document Database Service (DDS) based on the MongoDB aggregation framework. By utilizing pipeline operations, it executes complex transformations, filtering, grouping, and calculations on documents, providing advanced query functions similar to GROUP BY and JOIN in SQL. While fully compatible with the aggregation syntax of MongoDB Community Edition, DDS aggregation enhances performance and stability in distributed environments through cloud-native optimization.

Why You Need DDS Aggregation

DDS aggregation addresses the following core requirements:

Complex data analysis: Complex analysis, such as multi-dimensional statistical analysis (for example, grouping by time, region, and user behavior) and data pivoting (like expanding nested fields)
Data conversion: Conversion of raw documents into structured formats (for example, flattening array fields) to feed service reports or machine learning pipelines
Performance optimization: Offloading complex computational tasks (such as cross-collection associations) directly to the database layer, which minimizes application-level overhead and prevents network transmission bottlenecks
Real-time analysis: Real-time aggregation of high-frequency data, such as IoT device logs and user behavior, as well as second-level responses

Advantages

Although both DDS and MongoDB Community Edition support aggregation operations, DDS outperforms MongoDB Community Edition as a Huawei Cloud managed service in terms of cloud native capabilities, performance optimization, and support for advanced features. It is particularly suitable for high-concurrency, large-scale real-time enterprise analytics, as described in Table 1.

**Table 1** Advantages of DDS aggregation
Dimension	Description	Technical Support
High-performance processing	Efficiently aggregates terabytes of data with parallel processing across sharded clusters.	Combines the MongoDB sharding architecture with dynamic Huawei Cloud compute resource allocation.
Flexible data models	Supports embedded document aggregation, which eliminates complex JOIN dependencies, while maintaining full compatibility with nested JSON structures.	Uses an optimized native MongoDB document model backed by Huawei Cloud's decoupled storage and compute.
Horizontal scalability	Dynamically scales cluster instances to support configurations ranging from 2 to 32 mongos and shard nodes.	Features automated shard management and migration of jumbo chunks.
Abundant aggregation operators	Supports over 50 aggregation functions, such as sin, regexFind, and $graphLookup.	Fully compatible with the MongoDB 5.0 aggregation framework, with extra support for distributed transactions.
Enhanced cloud native	Delivers enterprise-grade features like hedge queries, wildcard indexes, and automated failover.	Integrates proprietary Huawei Cloud distributed database middleware with a multi-replica set HA architecture.

Use Cases

**Table 2** Use cases of DDS aggregation
Scenario	Description	Implementation and DDS Feature
IoT smart device data analysis	Multi-dimensional aggregation analysis of device logs (with details such as temperature, humidity, and location).	Utilizes the aggregation framework to process diverse inputs, employing the $unwind operator to flatten nested document arrays.
Gaming user behavior analysis	Real-time statistics and analysis of user metrics (such as inventory equipment, bonus credits, session duration).	Leverages the $lookup operator for cross-collection associations (for example, between user profiles and equipment inventory) and the $facet operator for multi-dimensional statistical breakdowns. Uses the replica set architecture to support high-concurrency read and write operations and to ensure high availability via automatic failover during primary node disruption.
Real-time service monitoring	Real-time aggregation analysis based on time-series metrics like the numbers of orders and API calls.	Integrates seamlessly with features in MongoDB 5.0 and later versions, supporting time-series collections. This feature provides specialized aggregation optimization for time-series data, effectively reducing data preprocessing costs and saving storage space.

How DDS Aggregation Works

Pipeline processing model
Aggregation operations process data through a multi-stage pipeline. Documents flow through a pipeline and are processed by each stage sequentially, with the output of one stage serving as the input to the next.
- Stage sequencing: For example, a typical pipeline may follow this sequence: $match (filtering) → $group (grouping) → $sort (sorting) → $project (reshaping). Every stage handles a specific transformation or calculation.
- Memory management: Pipelines execute in-memory by default. For massive datasets exceeding memory limits, enabling allowDiskUse: true safely spills intermediate results to temporary disk storage to prevent out-of-memory errors.
Distributed execution (sharded clusters)
In a sharded cluster architecture, DDS accelerates aggregation workloads using query routing and parallel execution:
1. Query routing: mongos (the router process) receives the aggregation command and pushes eligible pipeline stages (such as an initial $match) down to the targeted shards based on the shard key.
2. Parallel execution: Individual shards process data subsets independently and concurrently (supported natively since MongoDB 3.2).
3. Result aggregation: mongos gathers intermediate results from each shard to complete the final pipeline computation, such as sorting and grouping. Operations like $out (outputting results to a collection) are typically executed by the primary shard.
Optimization strategies
The built-in DDS optimizer automatically applies multiple policies to improve aggregation efficiency.
- Index utilization: Early pipeline stages featuring $match and $sort prioritize available indexes to quickly filter and sort datasets, significantly reducing the amount of data to be processed.
- Pipeline optimization: The optimizer restructures the aggregation pipeline to improve performance. For example, consecutive $projection or $addFields stages can be merged to minimize intermediate data passed between stages.

Types of DDS Aggregation

By aggregation stage

**Table 3** Aggregation types by stage
Type	Description	Typical Operator
Data filtering	Documents are filtered based on specified rules.	match and redact
Data grouping	Data is grouped and consolidated by specified fields.	group and bucket
Data transformation	Document structure or fields are reshaped.	project, addFields, and $replaceRoot
Data association	Cross-collection queries are performed, similar to JOIN in SQL.	lookup and graphLookup
Result output	Results are written to a target collection or returned to the client.	out and merge

By version feature

**Table 4** Aggregation types by version feature
Version Feature	Supported Version	Description
Distributed transactions	4.2+	Cross-shard transactions remove the standard 16 MB single-document constraint for transactional operations.
Wildcard indexes	4.2+	Optimization of dynamic field queries minimizes the overhead of index creation.
Hedge queries	4.4+	Multiple shards can be queried in parallel for faster responses.
Time-series optimization	5.0+	A specialized storage engine is applied to optimize time-series data aggregation.

DDS Aggregation vs. MongoDB Aggregation

Although both DDS and MongoDB aggregation are based on the MongoDB aggregation framework, DDS, a managed database service on Huawei Cloud, has different aggregation capabilities from MongoDB aggregation in terms of protocol compatibility, cloud native enhancements, and performance. as outlined in Table 5.

**Table 5** Differences between DDS aggregation and MongoDB aggregation
Category	Item	DDS Aggregation	MongoDB Aggregation (Community Edition)	Key Difference
Protocol compatibility	Version support	Fully compatible with 4.0, 4.2, 4.4, and 5.0.	Version features need to be managed using specific commands.	DDS provides better compatibility.
Function enhancements	Hedge queries (4.4+)	Supported	Not supported	DDS supports parallel shard queries, improving speed on high-latency networks.
	Auto migration of jumbo chunks (4.4+)	Supported	Not supported	DDS automatically balances data chunks, minimizing the need for manual operations.
	Wildcard indexes (4.2+)	Supported (performance optimized)	Supported	DDS optimizes performance for support of complex dynamic field matching.
	Distributed transactions (4.2+)	Supported (sharded cluster)	Supported (typically restricted to replica sets)	DDS extends transaction support to sharded clusters.