Create Experiments (AutoML)

Create Experiments (AutoML) is used to create a new AutoML experiment in the Cloudeka Deka MLOps Service Portal. On the Experiments (AutoML) page, click the + New Experiment button.

The Create Experiment page appears. There are eight steps required to create a new experiment, as described below.

1. Metadata, enter the name of your experiment.

   
2. Trial Thresholds, configure the number of trials to run.

   

   1. Parallel Trials

   2. Max Trials

   3. Max Failed Trials

   4. Resume Policy: Never, Long Running, and From Volume

3. Objective, add the metrics that you want to optimize and select the optimization type.

   

   1. Type: Maximize, Minimize

   2. Metric and Goal. For example, Validation Accuracy and 0.99. This means the target validation accuracy is 99%. You can add additional metrics if required.

   3. Set Metric Strategies: When enabled, you can select a strategy for each metric, such as Max, Min, or Latest.

4. Search Algorithm, select a hyperparameter tuning algorithm and configure its settings.

   

   1. Hyperparameter Tuning

      • Name: Bayesian Optimization, Covariance Matrix Adaption, Evolution Strategy, Grid, Hyperband, Multivariate Tree of parzen Estimators, Population Based Training, Random, Sobol Quasirandom Sequence, Tree of Parzen Estimators.

      • random_state

   2. Neural Architecture Search

      • Differentiable Architecture Search: num_epochs, w_lr, w_lr_min, w_momentum, w_weight_decay, w_grad_clip, alpa_lr, alpha_weight_decay, batch_size, num_workers, init_channels, print_step, num_nodes, stem_multiplier.

      • Efficient Neural Architecture Search: controller_hidden_size, controller_temperature, controller_tanh_const, controller_entropy_weight, controller_baseline_decay, controller_learning_rate, controller_skip_target, controller_skip_weight, controller_train_steps, controller_log_every_steps.

5. Early Stopping, add an early stopping algorithm if required.

   

   1. Median Stopping Rule: min_trials_required, start_step.

   2. None

6. Hyper Parameters, add the hyperparameters and search space that you want to optimize.

   
7. Metrics Collector, modify the metrics collector type if required.

   Kind: Stdout, File, TensorFlow Event, Prometheus, Custom, and None.

   
8. Trial Template, configure the trial template used for the experiment.

   

   Review the trial template YAML configuration and define the parameter references used in the experiment.

   

   If you want to modify the experiment YAML, click Edit and submit the YAML configuration at the bottom of the page.

   

   Review and modify the experiment YAML configuration before creating the experiment.

   
9. Click the Create button after you have finished configuring the experiment.