Configuring the Matrix Framework

Service Framework

To maximize the computing power of Ascend 310 chips, Huawei provides the Matrix framework to migrate inference services. The Matrix framework provides the following functions:

• Process orchestration
  1. An engine is defined as a basic functional unit in the process, and its implementation can be customized (for example, inputting image data, classifying images, and outputting the predicted class of images). By default, each engine corresponds to a thread on Ascend 310.
  2. A graph is defined to manage multiple engines. Each graph corresponds to a process on Ascend 310. Figure 1 shows the relationship between the graph and engines.
     Figure 1 Relationship between the graph and engines
     

  In the graph configuration file, configure the serial connections of engine nodes and node properties (parameters required for node running). The actual data flow direction is implemented on the nodes based on the specific service. The entire engine computation process is started by inputting data to the start node of the service.

• Media pre-processing

  Engines running on Ascend 310 can directly call the APIs provided by the DVPP to implement the media pre-processing capability.

• Offline model loading and running

  Engines running on Ascend 310 can directly call the APIs provided by the model manager (AIModelManger) to load offline models and perform inference.

  The following figure shows a user's service software structure based on the Matrix framework. The user creates a service flow (graph) that consists of custom engines. Engines running on the device side can call the APIs of the DVPP and AIModelManager to utilize Ascend 310-enabled media pre-processing and the hardware acceleration function of model inference. Engines on the host side are used to implement the service software logic and exchange data with the engines on the device side.

  Figure 2 Service software framework