What Is DataArts Fabric?

DataArts Fabric (DataArtsFabric) is Huawei Cloud's comprehensive, one-stop platform for data and AI development. It offers full lifecycle management, encompassing data processing, analysis, model fine-tuning, inference, deployment, and rollout. Data engineers, data scientists, and AI application developers can collaborate efficiently using familiar tools within a unified workbench, accelerating workflows from development to production. DataArtsFabric scales automatically to meet demanding application requirements, expanding resources incrementally based on actual needs. It saves up to 50% of costs compared to services with resource pools preset for peak loads.

This serverless approach, leveraging shared resource pools for diverse data and AI workloads (including CPU and NPU heterogeneous resources, and shared development/production environments), optimizes resource investment. It enables hybrid offline/online deployment and integrates training with inference, smoothing out peak and trough resource demands and significantly improving utilization. Customers benefit from a frictionless experience with zero resource thresholds, eliminating the need for cluster management and facilitating low-cost experimentation in dynamic business environments.

Architecture

DataArts Fabric offers a high-performance, highly reliable, low-latency, and cost-effective mass storage system. When integrated with Huawei Cloud big data services, it significantly reduces costs and simplifies big data management for enterprises.

• SQL engine

  DataArtsFabric's distributed SQL engine features layered decoupling of metadata services, computing, caching, and storage, enabling elastic resource allocation at each layer without impacting performance or availability. Statement-level elastic scaling and high-performance distributed analysis engines facilitate TB-level data queries in seconds and PB-level queries in minutes.

• Distributed Ray

  To overcome distributed computing challenges in data processing and ML/DL workloads, DataArtsFabric supports the Ray framework. This integration offers a unified workflow for data and machine learning engineering. DataArtsFabric Ray's Ray-Data, Ray-Train, and Ray-Serve modules facilitate distributed data preprocessing, model training, and inference services.

• Online inference

  DataArtsFabric includes a proprietary, high-performance elastic inference engine. Users can deploy inference jobs via the default inference service or by independently deploying custom models.

• Heterogeneous resource management

  DataArtsFabric offers unified management and allocation of CPU and NPU resources. Resource scheduling is supported at container-level or actor-level granularity. Furthermore, DataArtsFabric provides secure sandboxes for resource isolation and robust fault tolerance.

• Multi-semantic cache acceleration

  DataArtsFabric delivers cross-engine, multi-modal, and multi-semantic acceleration through various caching mechanisms, including data, model, and checkpoint caches.

Figure 1 Product architecture

a web-based management console, HTTPS-based APIs, and SDK clients for seamless compute engine integration.

• Management console

  DataArtsFabric can be accessed via its management console for managing Ray jobs, SQL jobs, model deployment, and model inference. This enables end-to-end data and AI development directly from the console.

• APIs

  For integrating DataArtsFabric into third-party systems or for secondary development, use the provided HTTPS APIs.

• SDKs

  To integrate DataArtsFabric functionalities into third-party systems for secondary development, utilize the SDKs. DataArtsFabric SDKs encapsulate the REST APIs in Python and Java, simplifying development.