Create Experiments (AutoML)

Create Experiments (AutoML) is used to create new experiment (AutoML) on the Deka MLOps Portal Service. On the Endpoints page press the button “+ New Experiment”.

Create Experiments

The Create Experiments page appears, there are 8 steps required to be able to create new Experiments, including the following.

1. Metadata, type name of your Experiment.

   

   Create Experiments-Metadata
2. Trial Thresholds, choose how many Trials you want to run.

   1. Parallel Trials

   2. Max Trials

   3. Max Failed Trials

   4. Resume Policy: Never, Long Running, and From Volume

   

   Create Experiments-Trial Thresholds

3. Objective, add metrics that you want to optimize and type of optimization.

   1. Type: Maximize, Minimize

   2. Metric and Goal, example is Validation-accuracy and 0,99. It means your goal is to achieve validation accuracy metric into 99%. You can also add another metric in this point.

   3. Set Metric Strategies: When you check it, you can choose options in “Strategy for: Validation-accuracy”: Max, Min, Latest.

   

   Create Experiments-Objective
4. Search Algorithm, select hyperparameter tuning algorithm and configure algorithm settings.

   1. Hyper Parameter Tuning

      • Name: Bayesian Optimization, Covariance Matrix Adaption, Evolution Strategy, Grid, Hyperband, Multivariate Tree of parzen Estimators, Population Based Training, Random, Sobol Quasirandom Sequence, Tree of Parzen Estimators.

      • random_state

   2. Neural Architecture Search

      • Differentiable Architecture Search: num_epochs, w_lr, w_lr_min, w_momentum, w_weight_decay, w_grad_clip, alpa_lr, alpha_weight_decay, batch_size, num_workers, init_channels, print_step, num_nodes, stem_multiplier.

      • Efficient Neural Architecture Search: controller_hidden_size, controller_temperature, controller_tanh_const, controller_entropy_weight, controller_baseline_decay, controller_learning_rate, controller_skip_target, controller_skip_weight, controller_train_steps, controller_log_every_steps.

      

      Create Experiments-Algorithm

5. Early Stopping, add early stopping algorithm if that is required.

   1. Median Stopping Rule: min_trials_required, start_step.

   2. None

Create Experiment – Early Stopping

1. Hyper Parameters, add hyperparameters and search space that you want to optimize.

Create Experiment – Hyper Parameters

1. Metrics Collector, modify metrics collector type if that is required.

   1. Kind: Stdout, File, TensorFlow Event, Prometheus, Custom, and None.

   

   Create Experiment – Metrics Collector
2. Trial Template

Create Experiments-Trial Template

Create Experiments-Trial Template

If you want to modify Experiment YAML, you can click edit and submit YAML at the bottom.

Modify YAML

Modify YAML

Click the “Create” button when you have finished setting up the Experiments.

Create Experiments (AutoML)