Scenario

This section describes how to search for the most appropriate data augmentation policy, learning rate, and network architecture in the search space of the specified data augmentation policy, hyperparameter, and ResNet50-like architecture based on the MNIST dataset when BatchSize is set to 100 and only 5000 steps are trained.

In the preceding scenario, the following functions are used:

• A script that can:
  1. Parse the data augmentation policy.
  2. Parse a NAS code of ResNet50.
  3. Parse learning rate parameters on the CLI.
• YAML configuration for searching for auto data augmentation policies
• YAML configuration for searching for learning rate hyperparameters
• YAML configuration for searching for the ResNet50 architecture

Combine the preceding three YAML configurations and code to complete a multisearch.

Sample Code

The complete code is as follows. The following describes the configuration of some main functions in the code.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100

import argparse 
import time 
import os

import tensorflow as tf 
from autosearch.client.augment.offline_search.preprocessor_builder import ( 
    ImageClassificationTensorflowBuilder, 
) 
from autosearch.client.nas.backbone.resnet import ResNet50 
from tensorflow.examples.tutorials.mnist import input_data 

import autosearch 

parser = argparse.ArgumentParser() 
parser.add_argument( 
    "--max_steps", type=int, default=100, help="Number of steps to run trainer." 
) 
parser.add_argument("--data_url", type=str, default="MNIST_data") 

parser.add_argument( 
    "--learning_rate", 
    type=float, 
    default=0.01,   
    help="Number of steps to run trainer.", 
) 
FLAGS, unparsed = parser.parse_known_args() 

def train(): 
    if is_yundao():
        data_url = yundao_init(FLAGS.data_url)
    mnist = input_data.read_data_sets(data_url, one_hot=True) 
    with tf.Graph().as_default(): 
        sess = tf.InteractiveSession() 
        with tf.name_scope("input"): 
            x = tf.placeholder(tf.float32, [None, 784], name="x-input") 
            y_ = tf.placeholder(tf.int64, [None, 10], name="y-input") 
        x_aug = ImageClassificationTensorflowBuilder("offline")(x) 
        image_shaped_input = tf.reshape(x_aug, [-1, 28, 28, 1]) 
        y = ResNet50(image_shaped_input, include_top=True, mode="train") 
        with tf.name_scope("cross_entropy"): 
            y = tf.reduce_mean(y, [1, 2]) 
            y = tf.layers.dense(y, 10) 
            with tf.name_scope("total"): 
                cross_entropy = tf.losses.softmax_cross_entropy(y_, y) 

        with tf.name_scope("train"): 
            train_step = tf.train.AdamOptimizer(FLAGS.learning_rate).minimize(  # st2 
                cross_entropy 
            ) 

        with tf.name_scope("accuracy"): 
            with tf.name_scope("correct_prediction"): 
                correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1)) 
            with tf.name_scope("accuracy"): 
                accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32)) 
        tf.global_variables_initializer().run() 

        def feed_dict(train): 
            if train: 
                xs, ys = mnist.train.next_batch(100) 
            else: 
                xs, ys = mnist.test.images, mnist.test.labels 
            return {x: xs, y_: ys} 

        max_acc = 0 
        latencys = [] 
        for i in range(FLAGS.max_steps): 
            if i % 10 == 0:  # Record summaries and test-set accuracy 
                loss, acc = sess.run( 
                    [cross_entropy, accuracy], feed_dict=feed_dict(False) 
                ) 
                # print('loss at step %s: %s' % (i, loss)) 
                print("Accuracy at step %s: %s" % (i, acc)) 
                if acc > max_acc: 
                    max_acc = acc 
                # autosearch.reporter(loss=loss) 
                autosearch.reporter(mean_accuracy=acc) 
            else: 
                start = time.time() 
                loss, _ = sess.run( 
                    [cross_entropy, train_step], feed_dict=feed_dict(True) 
                ) 
                end = time.time() 
                if i % 10 != 1: 
                    latencys.append(end - start) 
        latency = sum(latencys) / len(latencys) 
        autosearch.reporter(mean_accuracy=max_acc, latency=latency) 
        sess.close()

def is_yundao():
    return True if os.path.exists("/home/work/user-job-dir") else False

def yundao_init(data_url):
    local_data_dir = "/cache/mnist"

    import moxing as mox

    mox.file.copy_parallel(data_url, local_data_dir)

    return local_data_dir

• In the import statement, import a module for parsing augmentation policies and a module for parsing NAS architecture code.

  1 2 3 4

  # Parse the augmentation policies and convert the policy code into data augmentation operations. from autosearch.client.augment.offline_search.preprocessor_builder import ImageClassificationTensorflowBuilder # Parse NAS architecture code. The architecture code such as 111-2111-211111-2111 is translated into the TensorFlow network architecture. from autosearch.client.nas.backbone.resnet import ResNet50

  For details about the two preset decoder modules used in the code, see Using a Preset Decoder.

Compiling the Configuration File

• YAML configuration for data augmentation
  • Use the single-node, single-CPU/GPU configuration for a single training.
  • Use the built-in image_classification_auto_augment. For details about the built-in space, see Using a Preset Decoder.
  • Use GridSearch as the search algorithm. mean_accuracy is used as the metric. A larger value indicates better performance.

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

  general: gpu_per_instance: 1 search_space: - type: discrete params: - name: image_classification_auto_augment values: [['14-5-5', '14-5-5'], ['14-6-6', '14-6-6'], ['14-7-7', '14-7-7'], ['14-8-8', '14-8-8'], ['14-9-9', '14-9-9'] ] search_algorithm: type: grid_search reward_attr: mean_accuracy

• YAML configuration for hyperparameter search

  Use the YAML file to search for the learning rate. Try 10 to the power of negative 1, 2, 3, 4, and 5.

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

  general: gpu_per_instance: 1 search_space: - type: continuous params: - name: learning_rate start: -1 stop: -5 num: 5 base: 10 search_algorithm: type: grid_search reward_attr: mean_accuracy

• YAML configuration for searching for the NAS architecture

  Based on the YAML configuration, the GridSearch algorithm of AutoSearch traverses the 11 architectures to be provided.

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

  general: gpu_per_instance: 1 search_space: - type: discrete params: - name: resnet50 values: ["1-11111111-2111121111-211111", "1-1112-1111111111121-11111112111", "1-11111121-12-11111211", "11-111112-112-11111211", "1-1-111111112-11212", "1-1-1-2112112", "1-111211-1111112-21111111", "1-1111111111-21112112-11111", "1-111111111112-121111111121-11", "11-211-121-11111121", "111-2111-211111-211" ] search_algorithm: type: grid_search reward_attr: mean_accuracy scheduler: type: FIFOScheduler

Combine the script of the sample code and the three YAML files in Compiling the Configuration File to start and run AutoSearch.

Starting a Search Job

The MNIST dataset is required in this example. Upload and configure the dataset by following instructions in Example: Replacing the Original ResNet-50 with a Better Network Architecture, upload the Python script and YAML configuration file, and start the search job. For details, see the section of creating an auto search job. If multiple YAML files are configured, set this parameter to the full paths of the YAML files and separate them with commas (,), for example, obs://foo/bar.yaml,obs://bar/foo.yaml.