Data for AI Overview
Background
In the enterprise-level AI era, with breakthroughs in inference models and domain-specific models, AI is evolving from a developer tool to a core productivity driver for enterprises. However, traditional data warehouses face three major challenges: data redundancy and O&M burden caused by the multi-engine architecture, inefficient development due to the separation of AI and data platforms, and difficulty in fully unleashing the value of unstructured data. Enterprises no longer need just a faster query engine; instead, they need an intelligent data partner that can understand data, perform inference, and explore the value of data.
To address these challenges and maximize the value of data, Huawei Cloud DWS 9.1.1.200 provides Data for AI converged analysis. It integrates the Model Context Protocol (MCP) and supports one-stop online analytical processing (OLAP), point query, full-text search, in-database inference, and AI inference. It redefines the paradigm of intelligent application development and enables data engineers to easily leverage AI to unlock the full value of data and provide real-time, intelligent, and efficient decision-making support for customers.
Why You Need AI-Native Data Warehouses
In enterprise-level data analysis, the traditional converged analysis architecture relies on multiple independent engines. For example, Elasticsearch is used for full-text search, Milvus for vector search, Doris for OLAP, and ClickHouse for point queries. However, this architecture has the following pain points:
- Storage redundancy: Data is repeatedly stored across engines, resulting in low resource utilization.
- Complex O&M: Independent maintenance of multiple systems makes fault locating and performance optimization difficult.
- Inefficient development: Data format conversion and link adaptation consume a large number of R&D resources.
- Insufficient real-time performance: Batch data transfer cannot provide the millisecond-level response times required by AI applications.
To address these challenges and improve data analysis efficiency and timeliness, DWS provides an AI-native one-stop analytics engine, which greatly simplifies the original multi-engine collaborative architecture. This engine integrates MCP to streamline interactions with large model platforms. It also incorporates vector search and provides AI functions for calling large models, enabling end-to-end in-database inference. This creates an AI-ready data analytics infrastructure and eliminates data silos. In addition, binlog-based materialized views enable integrated stream and batch incremental computing. This provides developers with a new paradigm of declarative pipeline processing to achieve near-real-time analysis. The overall architecture is more lightweight, with more efficient development and easier O&M management. This meets your core requirements for deploying one-stop, multimodal converged analytics on AI platforms.
Four Core Capabilities of Data for AI
Data for AI provides the following core capabilities.
Native Integration of MCP: Streamlining the Neural Network of the AI Ecosystem
MCP is an open standard launched by Anthropic, often referred to as the "USB-C of AI". It aims to unify the communication protocol between large models and external data sources and tools. It unleashes the true value of AI models that are restricted by data silos. MCP enables AI applications to securely access and operate any local and remote data.
DWS supports MCP, one-click configuration, and plug-and-play. Through standardized protocols and interfaces, you can seamlessly access mainstream AI platforms such as Claude and Cursor. Eight built-in APIs for data analysis and O&M monitoring enable you to connect everything out of the box.
AI Functions: Pluggable In-database AI Inference
DWS integrates the open-source plugin pgai and can connect to open-source large models and Huawei Cloud MaaS. By using in-database SQL statements, you can call large models for inference. DWS provides 24 AI functions, including text analysis (ai.classify), vectorization (ai.embed), text summarization (ai.summarize), and sentiment analysis (ai.sentiment). By integrating the inference capability of large models into the database kernel, DWS offers more intelligent data analytics and enables data inference and processing in real-time, helping you quickly gain insights and make better decisions more efficiently.
Vector Computing: Necessary for Unstructured Data Analysis
80% of enterprise data is unstructured, such as documents, images, audio, and videos. Such data has long been overlooked in data mining, and its value has been underestimated. The multimodal unified vector storage and representation technology eliminates the barrier to multimodal search.
DWS integrates the open-source plugin pgvector for reconstruction of the distributed architecture. It supports vector data types, IVFFlat and HNSW index structures, and advanced algorithms such as similarity calculation and nearest neighbor search, which provide basic vector search capabilities.
Built-in Large Model Feature Operators: Unleashing Unlimited Creativity in AI Analysis
DWS supports PL/Python and provides 32 Huawei-developed large model operators, which incorporate the Python data science ecosystem into SQL workflows. Developers can directly run complex AI algorithms in databases, eliminating the overhead of cross-system data transfer and achieving an end-to-end closed loop from feature engineering to model inference.
DWS AI Data Warehouse Application Scenarios
Scenario 1 Retail: Real-time Operations, Intelligent Product Selection, and Precision Marketing Improve Consumer Experience
- Pain points
- Traditional data warehouses can only store structured data such as transaction orders and member information, making it difficult to analyze unstructured data such as user browsing logs, product reviews, and bullet comments.
- Offline analysis cannot support real-time promotion decision-making. For example, if a product suddenly becomes popular in a livestream, the inventory and pricing cannot be quickly adjusted.
- User profiles mainly rely on rule-based tags, making it difficult to mine users' real pain points or needs from sources such as product reviews and bullet comments through semantic inference.
- Benefits
- Online reports: Annual, monthly, weekly, and daily reports can be generated efficiently to collect statistics on product categories and sales volumes, helping you make informed decisions in operations.
- Real-time operations: Products whose negative comments exceed 30% of all comments are collected in real time. Alerts are automatically generated for these products so that operations personnel can promptly adjust their scripts or issue coupons.
- Intelligent product selection: High-potential products are accurately identified based on sales data, sentiment of reviews, and the popularity of similar products. This provides scientific guidance for procurement decisions.
- Precision marketing: Coupons for gift boxes are pushed to users who are very likely to repurchase products and mention gifts in their reviews.
Scenario 2 Manufacturing: Device-Process-Quality Collaboration Facilitates Real-Time Decision-Making and Quality Improvement
- Pain points
- IoT device data, process parameter tables, and quality inspection reports are stored in different systems (IoT platform, ERP, and MES), leading to redundant data synchronization during cross-system analysis.
- Quality defect analysis relies on human experience, making it difficult to quickly identify correlations such as abnormal device parameters, process deviation, and defect occurrence.
- Offline analysis does not support real-time process adjustment. For example, when the vibration value of a device exceeds the threshold, an alarm cannot be generated in time to prevent the production of defective products in batches.
- Benefits
- Integrated stream and batch processing: IoT device data is imported to the database in real time, and batch addition, deletion, and modification operations are performed periodically. Offline computing of the association rules between historical process parameters and defect rates, and real-time monitoring of IoT data trigger alarms when parameters deviate from the optimal range.
- Quality tracing: By entering the batch number of defective products, you can query the IoT data of production devices, process parameters, and quality inspection records to quickly locate the root causes of defects.
- Predictive maintenance: Based on the data from device sensors and historical fault records, the system predicts the possibility of device faults, and you can arrange maintenance in advance to reduce the downtime.
- Process optimization: SQL statements can be used to analyze the qualification rates of different process parameter combinations. Regression algorithms can be used to model the relationship between the temperature, pressure, and qualification rate, and the system provides the optimal range of process parameters, improving the product qualification rate by 5% to 10%.
Scenario 3 Finance: Enabling Real-Time Decision-Making for Financial Risk Control Through Transactions, Behavior, and Public Opinions
- Pain points
- Traditional risk control only relies on transaction data (such as the amount, time, and merchants), making it difficult to identify abnormal behavior and public opinion risks. For example, even if an account is stolen, the transaction amount may remain normal, but the login device is abnormal.
- The rule engine also struggles to cope with new fraud methods, such as scammers who register accounts in batches and simulate normal behavior to obtain benefits.
- Public opinion risks (for example, a company is exposed for financial fraud) cannot be synchronized to the risk control system in real time, resulting in delayed credit decision-making.
- Benefits
- Real-time risk control: When a transaction occurs, multi-dimensional risk assessment (including account and device exceptions) is performed in real time to intercept fraud behavior such as unauthorized transactions and scalpers, reducing the loss.
- Real-time anti-fraud: Users whose single transaction amount exceeds $50,000 USD and whose monthly transactions increase by 10 times are filtered and analyzed in real time. Vector search is used to match the similarity between the user behavior and fraud sample.
- Intelligent credit decision-making: Before a loan is granted to an enterprise, suggestions on the amount and interest rate of the loan are provided based on information such as the financial data, public opinion risks, and industry trends.
- Public opinion risk warning: When negative news of an enterprise is disclosed, related credit customers are automatically marked, and a risk check is initiated.
Scenario 4 Government: Smart City Construction Drives Intelligent City Management
- Pain points
- Government data is scattered across departments and in different formats, making it difficult to support cross-department collaborative analysis.
- Most feedback on people's livelihood (for example, from the mayor's hotline or government apps) is unstructured text, making it difficult to quantitatively analyze residents' needs.
- City governance relies on human inspections, making it impossible to respond to emergencies in real time.
- Benefits
- Intelligent traffic scheduling: The system dynamically generates traffic congestion heat maps based on spatiotemporal and real-time traffic data, and intelligently recommends alternative routes. During peak hours, the system adjusts traffic light timing in real time to alleviate congestion.
- Precise response to public needs: Resources are preferentially allocated to address high-frequency issues based on the semantic classification and regional distribution of residents' needs.
- Warning for sudden public incidents: When waterlogging exceeds the threshold and heavy rainfalls are expected, the system automatically triggers flood response and provides suggestions on flood control plans to assist functional departments in emergency decision-making.
What is your overall rating for this page?
Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot