Create Experiments (AutoML)

Create Experiments (AutoML) is used to create new experiment (AutoML) on the Deka MLOps Portal Service. On the Endpoints page press the button “+ New Experiment”.

The Create Experiments page appears, there are 8 steps required to be able to create new Experiments, including the following.

1. Metadata, type name of your Experiment.

   
2. Trial Thresholds, choose how many Trials you want to run.

   1. Parallel Trials

   2. Max Trials

   3. Max Failed Trials

   4. Resume Policy: Never, Long Running, and From Volume

   

3. Objective, add metrics that you want to optimize and type of optimization.

   1. Type: Maximize, Minimize

   2. Metric and Goal, example is Validation-accuracy and 0,99. It means your goal is to achieve validation accuracy metric into 99%. You can also add another metric in this point.

   3. Set Metric Strategies: When you check it, you can choose options in “Strategy for: Validation-accuracy”: Max, Min, Latest.

   
4. Search Algorithm, select hyperparameter tuning algorithm and configure algorithm settings.

   1. Hyper Parameter Tuning

      • Name: Bayesian Optimization, Covariance Matrix Adaption, Evolution Strategy, Grid, Hyperband, Multivariate Tree of parzen Estimators, Population Based Training, Random, Sobol Quasirandom Sequence, Tree of Parzen Estimators.

      • random_state

   2. Neural Architecture Search

      • Differentiable Architecture Search: num_epochs, w_lr, w_lr_min, w_momentum, w_weight_decay, w_grad_clip, alpa_lr, alpha_weight_decay, batch_size, num_workers, init_channels, print_step, num_nodes, stem_multiplier.

      • Efficient Neural Architecture Search: controller_hidden_size, controller_temperature, controller_tanh_const, controller_entropy_weight, controller_baseline_decay, controller_learning_rate, controller_skip_target, controller_skip_weight, controller_train_steps, controller_log_every_steps.

      

5. Early Stopping, add early stopping algorithm if that is required.

   1. Median Stopping Rule: min_trials_required, start_step.

   2. None

1. Hyper Parameters, add hyperparameters and search space that you want to optimize.

1. Metrics Collector, modify metrics collector type if that is required.

   1. Kind: Stdout, File, TensorFlow Event, Prometheus, Custom, and None.

   
2. Trial Template

If you want to modify Experiment YAML, you can click edit and submit YAML at the bottom.

Click the “Create” button when you have finished setting up the Experiments.