Help Center/ ModelArts/ Best Practices/ Model Inference/ Creating an AI Application Using a Custom Engine

Updated on 2024-01-09 GMT+08:00

View PDF

Creating an AI Application Using a Custom Engine

When you use a custom engine to create an AI application, you can select your image stored in SWR as the engine and specify a file directory in OBS as the model package. In this way, bring-your-own images can be used to meet your dedicated requirements.

Before deploying such an AI application as a service, ModelArts downloads the SWR image to the cluster and starts the image as a container as the user whose UID is 1000 and GID is 100. Then, ModelArts downloads the OBS file to the /home/mind/model directory in the container and runs the boot command preset in the SWR image. The service available to port 8080 in the container is automatically registered with APIG. You can access the service through the APIG URL.

Specifications for Using a Custom Engine to Create an AI Application

To use a custom engine to create an AI application, ensure the SWR image, OBS model package, and file size comply with the following requirements:

SWR image specifications
- A common user named ma-user in group ma-group must be built in the SWR image. Additionally, the UID and GID of the user must be 1000 and 100, respectively. The following is the dockerfile command for the built-in user:
```
groupadd -g 100 ma-group && useradd -d /home/ma-user -m -u 1000 -g 100 -s /bin/bash ma-user
```
- Specify a command for starting the image. In the dockerfile, specify cmd. The following shows an example:
```
CMD sh /home/mind/run.sh
```
  Customize the startup entry file run.sh. The following is an example.
```
#!/bin/bash

# User-defined script content
...

# run.sh calls app.py to start the server. For details about app.py, see "HTTPS Example".
python app.py
```
- The service must be HTTPS enabled, and it is available on port 8080. For details, see the HTTPS example.
- (Optional) On port 8080, enable health check with URL /health. (The health check URL must be /health.)

OBS model package specifications
The name of the model package must be model. For details about model package specifications, see Introduction to Model Package Specifications.
File size specifications
When a public resource pool is used, the total size of the downloaded SWR image (not the compressed image displayed on the SWR page) and the OBS model package cannot exceed 30 GB.

HTTPS Example

Use Flask to start HTTPS. The following is an example of the web server code:

from flask import Flask, request
import json 

app = Flask(__name__)

@app.route('/greet', methods=['POST'])
def say_hello_func():
    print("----------- in hello func ----------")
    data = json.loads(request.get_data(as_text=True))
    print(data)
    username = data['name']
    rsp_msg = 'Hello, {}!'.format(username)
    return json.dumps({"response":rsp_msg}, indent=4)

@app.route('/goodbye', methods=['GET'])
def say_goodbye_func():
    print("----------- in goodbye func ----------")
    return '\nGoodbye!\n'


@app.route('/', methods=['POST'])
def default_func():
    print("----------- in default func ----------")
    data = json.loads(request.get_data(as_text=True))
    return '\n called default func !\n {} \n'.format(str(data))

@app.route('/health', methods=['GET'])
def healthy():
    return "{\"status\": \"OK\"}"

# host must be "0.0.0.0", port must be 8080
if __name__ == '__main__':
    app.run(host="0.0.0.0", port=8080, ssl_context='adhoc')

Debugging on a Local Computer

Perform the following operations on a local computer with Docker installed to check whether a custom engine complies with specifications:

Download the custom image, for example, custom_engine:v1 to the local computer.
Copy the model package folder model to the local computer.
Run the following command in the same directory as the model package folder to start the service:
```
docker run --user 1000:100 -p 8080:8080 -v model:/home/mind/model    custom_engine:v1
```
This command is used for simulation only because the directory mounted to -v is assigned the root permission. In the cloud environment, after the model file is downloaded from OBS to /home/mind/model, the file owner will be changed to ma-user.
Start another terminal on the local computer and run the following command to obtain the expected inference result:
```
curl https://127.0.0.1:8080/${Request path to the inference service}
```

Deployment Example

The following section describes how to use a custom engine to create an AI application.

Create an AI application and viewing its details.
Log in to the ModelArts console, choose AI Application Management > AI Applications, and click Create. On the page that is displayed, configure the following parameters:
- Meta Model Source: OBS
- Meta Model: a model package selected from OBS
- AI Engine: Custom
- Engine Package: an SWR image
  Retain the default settings for other parameters.
Click Create Now. In the AI application list that is displayed, check the AI application status. When its status changes to Normal, the AI application has been created.

Figure 1 Creating an AI application

Click the AI application name. On the page that is displayed, view details about the AI application.

Figure 2 Viewing details about an AI application
Deploy the AI application as a service and view service details.
On the AI application details page, choose Deploy > Real-Time Services in the upper right corner. On the Deploy page, select a proper compute node specification, retain the default settings for other parameters, and click Next. When the service status changes to Running, the service has been deployed.

Figure 3 Deploying a service

Click the service name. On the page that is displayed, view the service details. Click the Logs tab to view the service logs.

Figure 4 Logs
Use the service for prediction.
On the service details page, click the Prediction tab to use the service for prediction.

Figure 5 Prediction