(Optional) Creating an API
The API management module can manage APIs of the evaluation services. You can select an evaluation service when creating an evaluation job. You can create, edit, and delete APIs.
- Log in to ModelArts Studio. In the My Spaces area, click the required workspace.
Figure 1 My Spaces
- In the navigation pane, choose Evaluation Center > API Management. Click Create API in the upper right corner.
- Set the API name and organization, select a model type, and click submit.
Figure 2 Creating an API
- After an API is created, add versions on the right to define the API information. You can add multiple versions.
- Click New Version and configure the model version, Concurrency Level, QPM, URL, Param, Body, and Response.
- model version: user-defined.
- Concurrency Level: Set it based on the model performance. The value ranges from 1 to 32.
- QPM: Set it based on the model performance. The value ranges from 1 to 5000.
- URL: Set a model inference address.
- Param: You can set the parameters based on the API requirements. You can define the header and request parameters.
- Body: inference request body. You can customize the body or select the preset OpenAI or TGI structure.
- Response: defines how to obtain the content in the response body in JsonPath format. The JsonPath syntax is used to extract required data from the JSON field in the response body.
Figure 3 Adding a version for an API - After creating an API version, you need to verify whether the API is available. Only verified APIs can be evaluated. On the API Management page, click the name of the created API task, click Interface verification, enter a question, and check whether the API response content is displayed.
Figure 4 Interface verification
Creating an API Service
In this example, we use a deployed and properly running Pangu NLP model to create an API service.
- On the Model Development > Model Deployment page, find the deployed NLP model, click the service name, and copy the API URL on the details page.
- In the navigation pane, choose Evaluation Center > API Management. Click Create API in the upper right corner. Define the API name and organization structure, and set the model type to NLP.
- After an API is created, add versions on the right to define the API information. You can add multiple versions. Click New Version and set parameters based on Table 1.
Table 1 Configuring the API service Parameter
Description
model version
The value can be customized.
Concurrency Level
Set this parameter to 1.
QPM
Set this parameter to 1.
URL
Enter the obtained API URL.
Param
Set two parameters of the header type.
Parameter 1: The parameter name is X-AUTH-TOKEN, and the parameter value is the token value. For details about how to obtain the token, see "Calling REST APIs" > "Authentication" in API Reference.
Parameter 2: The parameter name is Content-Type and the parameter value is application/json.
Body
Select custom and enter the following content:
{"model": "model","messages": [{"role": "user","content": "{{message}}"}],"max_tokens": 1024, "temperature": 0.7,"top_p": 0.95,"presence_penalty": 0.5,"frequency_penalty": 0.5,"stream": false}
Response
Enter the following content:
$.choices[0].message.content
Figure 5 Configuring the API service
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot