Smart Refining Templates
Description
Smart refining templates on ModelArts are best practices for data processing. They combine complex operations into simple workflows you can use with one click.
Core functions:
- You do not need to start from scratch. You can reuse tested solutions from experienced engineers to ensure your data processing is professional and logical.
- You only need to select a template that fits your service scenario when creating a task. The system will automatically load the full operator link. You can run the workflow directly or fine-tune it based on data characteristics, reducing data preparation time from hours to minutes.
- The template-based system standardizes data processing, prevents quality issues from personal settings, and ensures high-quality training sets.
Built-in Templates
ModelArts provides the following built-in templates, covering the processing of text, image, and video data. For details about how to use these templates, see Table 1.
| Template Name | Supported Dataset Modality | Supported Dataset Type | Use Case | Involved Operators |
|---|---|---|---|---|
| Word processing workflow | Text | Document | Use built-in operators to extract and process Word files and generate pre-trained text data. | Word Document Content Extraction |
| Personal Data Anonymization | ||||
| Switching Between Simplified Chinese and Traditional Chinese | ||||
| Symbol Standardization | ||||
| Sensitive Word Filtering | ||||
| Filtering of the Incomplete Sentence at the End of a Paragraph | ||||
| N-gram Feature Filtering | ||||
| Text Length Filtering | ||||
| Pre-trained Text Classification | ||||
| PDF processing workflow | Text | Document | Use built-in operators to extract and process PDF files and generate pre-trained text data. | PDF Content Extraction |
| Personal Data Anonymization | ||||
| Switching Between Simplified Chinese and Traditional Chinese | ||||
| Symbol Standardization | ||||
| Sensitive Word Filtering | ||||
| Filtering of the Incomplete Sentence at the End of a Paragraph | ||||
| N-gram feature filtering | ||||
| Text Length Filtering | ||||
| Pre-trained Text Classification | ||||
| Pre-trained text processing workflow | Text | Pre-trained text | Use built-in operators to process pre-trained text and generate cleaned pre-trained text data. | Switching between simplified Chinese and traditional Chinese |
| Symbol Standardization | ||||
| N-gram feature filtering | ||||
| Text Length Filtering | ||||
| Pre-trained Text Classification | ||||
| Image processing workflow | Image | Image | Use built-in operators to extract, deduplicate, label, and filter images, and generate summaries to produce processed image and text data. | Image and Text Extraction |
| Image Metadata Filtering | ||||
| Image Deduplication | ||||
| Dangerous Situation Image Detection | ||||
| Pornographic Image Detection | ||||
| Violent and Terrorism Image Detection | ||||
| Video processing workflow | Video | Video | Use built-in operators to split, deduplicate, label, and filter video clips, and generate processed video data. | Video Clipping |
| Video Metadata Filtering | ||||
| Video Aspect Ratio Filtering | ||||
| Pornographic Video Detection | ||||
| Terrorism Video Detection | ||||
| Political Video Detection | ||||
| Motion Range Scoring | ||||
| Aesthetics Scoring | ||||
| Single-turn Q&A process | Text | Single-turn Q&A | Use built-in operators to process single-turn Q&A data and generate cleaned single-turn Q&A data. | Personal Data Anonymization |
| Switching between simplified Chinese and traditional Chinese | ||||
| Symbol Standardization | ||||
| Sensitive Word Filtering | ||||
| Filtering of the Incomplete Sentence at the End of a Paragraph | ||||
| N-gram feature filtering | ||||
| Text Length Filtering | ||||
| Q&A ranking process | Text | Q&A sorting | Use built-in operators to process Q&A ranking data and generate cleaned Q&A ranking data. | Personal Data Anonymization |
| Switching between simplified Chinese and traditional Chinese | ||||
| Symbol Standardization | ||||
| Sensitive Word Filtering | ||||
| Filtering of the Incomplete Sentence at the End of a Paragraph | ||||
| Text Length Filtering | ||||
| Multi-turn Q&A process | Text | Multi-turn Q&A | Use built-in operators to process multi-turn Q&A data and generate cleaned multi-turn Q&A data. | Personal Data Anonymization |
| Switching between simplified Chinese and traditional Chinese | ||||
| Symbol Standardization | ||||
| Sensitive Word Filtering | ||||
| Filtering of the Incomplete Sentence at the End of a Paragraph | ||||
| Text Length Filtering | ||||
| Preference optimization process | Text | Preference optimization | Use built-in operators to process preference optimization data and generate cleaned preference optimization data. | Personal Data Anonymization |
| Switching between simplified Chinese and traditional Chinese | ||||
| Symbol Standardization | ||||
| Sensitive Word Filtering | ||||
| Removing incomplete sentences at the end of a paragraph | ||||
| Text Length Filtering |
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot
