GaussianHyperparameterSearch¶

class
getml.hyperopt.
GaussianHyperparameterSearch
(param_space, pipeline, score='rmse', n_iter=100, seed=5483, ratio_iter=0.8, optimization_algorithm='nelderMead', optimization_burn_in_algorithm='latinHypercube', optimization_burn_ins=500, surrogate_burn_in_algorithm='latinHypercube', gaussian_kernel='matern52', gaussian_optimization_burn_in_algorithm='latinHypercube', gaussian_optimization_algorithm='nelderMead', gaussian_optimization_burn_ins=500, gaussian_nugget=50)¶ Bases:
getml.hyperopt.hyperopt._Hyperopt
Bayesian hyperparameter optimization using a Gaussian process.
After a burnin period, a Gaussian process is used to pick the most promising parameter combination to be evaluated next based on the knowledge gathered throughout previous evaluations. Accessing the quality of potential combinations will be done using the expected information (EI).
 Parameters
param_space (dict) –
Dictionary containing numerical arrays of length two holding the lower and upper bounds of all parameters which will be altered in pipeline during the hyperparameter optimization.
If we have two feature learners and one predictor, the hyperparameter space might look like this:
param_space = { "feature_learners": [ { "num_features": [10, 50], }, { "max_depth": [1, 10], "min_num_samples": [100, 500], "num_features": [10, 50], "reg_lambda": [0.0, 0.1], "shrinkage": [0.01, 0.4] }], "predictors": [ { "reg_lambda": [0.0, 10.0] } ] }
If we only want to optimize the predictor, then we can leave out the feature learners.
pipeline (
Pipeline
) – Base pipeline used to derive all models fitted and scored during the hyperparameter optimization. Be careful when constructing it since only the parameters present in param_space will be overwritten. It defines the data schema and any hyperparameters that are not optimized.score (str, optional) – The score to optimize. Must be from
scores
.n_iter (int, optional) – Number of iterations in the hyperparameter optimization and thus the number of parameter combinations to draw and evaluate. Range: [1, ]
seed (int, optional) – Seed used for the random number generator that underlies the sampling procedure to make the calculation reproducable. Due to nature of the underlying algorithm, this is only the case if the fit is done without multithreading. To reflect this, a seed of None is only allowed to be set to an actual integer if both
num_threads
andn_jobs
instance variables of thepredictor
andfeature_selector
in model  if they are instances of eitherXGBoostRegressor
orXGBoostClassifier
 are set to 1. Internally, a seed of None will be mapped to 5543. Range: [0, ]ratio_iter (float, optional) –
Ratio of the iterations used for the burnin. For a ratio_iter of 1.0, all iterations will be spent in the burnin period resulting in an equivalence of this class to
LatinHypercubeSearch
orRandomSearch
 depending on surrogate_burn_in_algorithm. Range: [0, 1]As a rule of thumb at least 70 percent of the evaluations should be spent in the burnin phase. The more comprehensive the exploration of the param_space during the burnin, the less likely it is that the Gaussian process gets stuck in local minima.
optimization_algorithm (string, optional) – Determines the optimization algorithm used for the local search in the optimization of the expected information (EI). Must be from
optimization
.optimization_burn_in_algorithm (string, optional) – Specifies the algorithm used to draw initial points in the burnin period of the optimization of the expected information (EI). Must be from
burn_in
.optimization_burn_ins (int, optional) – Number of random evaluation points used during the burnin of the minimization of the expected information (EI). After the surrogate model  the Gaussian process  was successfully fitted to the previous parameter combination, the algorithm is able to calculate the EI for a given point. In order to get to the next combination, the EI has to be maximized over the whole parameter space. Much like the GaussianProcess itself, this requires a burnin phase. Range: [3, ]
surrogate_burn_in_algorithm (string, optional) – Specifies the algorithm used to draw new parameter combinations during the burnin period. Must be from
burn_in
.gaussian_kernel (string, optional) – Specifies the 1dimensional kernel of the Gaussian process which will be used along each dimension of the parameter space. All of the choices below will result in continuous sample paths and their main difference is the degree of smoothness of the results with ‘exp’ yielding the least and ‘gauss’ yielding the most smooth paths. Must be from
kernels
.gaussian_optimization_algorithm (string, optional) – Determines the optimization algorithm used for the local search in the fitting of the Gaussian process to the previous parameter combinations. Must be from
optimization
.gaussian_optimization_burn_in_algorithm (string, optional) – Specifies the algorithm used to draw new parameter combinations during the burnin period of the optimization of the Gaussian process. Must be from
burn_in
.gaussian_optimization_burn_ins (int, optional) – Number of random evaluation points used during the burnin of the fitting of the Gaussian process. Range: [3, ]
 Raises
KeyError – If an unsupported instance variable is encountered (via
validate()
).TypeError – If any instance variable is of wrong type (via
validate()
).ValueError – If any instance variable does not match its possible choices (string) or is out of the expected bounds (numerical) (via
validate()
).
Note
A Gaussian hyperparameter search works like this:
It begins with a burnin phase, usually about 70% to 90% of all iterations. During that burnin phase, the hyperparameter space is sampled more or less at random. You can control this phase using
ratio_iter
andsurrogate_burn_in_algorithm
.Once enough information has been collected, it fits a Gaussian process on the hyperparameters with the
score
we want to maximize or minimize as the predicted variable. Note that the Gaussian process has hyperparameters itself, which are also optimized. You can control this phase usinggaussian_kernel
,gaussian_optimization_algorithm
,gaussian_optimization_burn_in_algorithm
andgaussian_optimization_burn_ins
.It then uses the Gaussian process to predict the expected information (EI), which is how much additional information it might get from evaluating a particular point in the hyperparameter space. The expected information is to be maximized. The point in the hyperparameter space with the maximum expected information is the next point that is actually evaluated (meaning a new pipeline with these hyperparameters is trained). You can control this phase using
optimization_algorithm
,optimization_burn_ins
andoptimization_burn_in_algorithm
.
In a nutshell, the GaussianHyperparameterSearch behaves like human data scientists:
At first, it picks random hyperparameter combinations.
Once it has gained a better understanding of the hyperparameter space, it starts evaluating hyperparameter combinations that are particularly interesting.
References
Carl Edward Rasmussen and Christopher K. I. Williams, MIT Press, 2006
Julien Villemonteix, Emmanuel Vazquez, and Eric Walter, 2009
Example
from getml import data from getml import datasets from getml import engine from getml import feature_learning from getml.feature_learning import aggregations from getml.feature_learning import loss_functions from getml import hyperopt from getml import pipeline from getml import predictors #  engine.set_project("examples") #  population_table, peripheral_table = datasets.make_numerical() #  # Construct placeholders population_placeholder = data.Placeholder("POPULATION") peripheral_placeholder = data.Placeholder("PERIPHERAL") population_placeholder.join(peripheral_placeholder, "join_key", "time_stamp") #  # Base model  any parameters not included # in param_space will be taken from this. fe1 = feature_learning.MultirelModel( aggregation=[ aggregations.Count, aggregations.Sum ], loss_function=loss_functions.SquareLoss, num_features=10, share_aggregations=1.0, max_length=1, num_threads=0 ) #  # Base model  any parameters not included # in param_space will be taken from this. fe2 = feature_learning.RelboostModel( loss_function=loss_functions.SquareLoss, num_features=10 ) #  # Base model  any parameters not included # in param_space will be taken from this. predictor = predictors.LinearRegression() #  pipe = pipeline.Pipeline( population=population_placeholder, peripheral=[peripheral_placeholder], feature_learners=[fe1, fe2], predictors=[predictor] ) #  # Build a hyperparameter space. # We have two feature learners and one # predictor, so this is how we must # construct our hyperparameter space. # If we only wanted to optimize the predictor, # we could just leave out the feature_learners. param_space = { "feature_learners": [ { "num_features": [10, 50], }, { "max_depth": [1, 10], "min_num_samples": [100, 500], "num_features": [10, 50], "reg_lambda": [0.0, 0.1], "shrinkage": [0.01, 0.4] }], "predictors": [ { "reg_lambda": [0.0, 10.0] } ] } #  # Wrap a GaussianHyperparameterSearch around the reference model gaussian_search = hyperopt.GaussianHyperparameterSearch( pipeline=pipe, param_space=param_space, n_iter=30, score=pipeline.scores.rsquared ) gaussian_search.fit( population_table_training=population_table, population_table_validation=population_table, peripheral_tables=[peripheral_table] ) #  # We want 5 additional iterations. gaussian_search.n_iter = 5 # We do not want another burninphase, # so we set ratio_iter to 0. gaussian_search.ratio_iter = 0.0 # This widens the hyperparameter space. gaussian_search.param_space["feature_learners"][1]["num_features"] = [10, 100] # This narrows the hyperparameter space. gaussian_search.param_space["predictors"][0]["reg_lambda"] = [0.0, 0.0] # This continues the hyperparameter search using the previous iterations as # prior knowledge. gaussian_search.fit( population_table_training=population_table, population_table_validation=population_table, peripheral_tables=[peripheral_table] ) #  all_hyp = hyperopt.list_hyperopts() best_pipeline = gaussian_search.best_pipeline
Attributes Summary
The best pipeline that is part of the hyperparameter optimization.
Name of the hyperparameter optimization.
Returns the ID of the hyperparameter optimization.
The score to be optimized.
The algorithm used for the hyperparameter optimization.
Methods Summary
fit
(population_table_training, …[, …])Launches the hyperparameter optimization.
refresh
()Reloads the hyperparameter optimization from the engine.
validate
()Validate the parameters of the hyperparameter optimization.
Attributes Documentation

best_pipeline
¶ The best pipeline that is part of the hyperparameter optimization.
This is always based on the validation data you have passed even if you have chosen to score the pipeline on other data afterwards.

id
¶ Name of the hyperparameter optimization. This is used to uniquely identify it on the engine.

name
¶ Returns the ID of the hyperparameter optimization. The name property is kept for backward compatibility.

score
¶ The score to be optimized.

type
¶ The algorithm used for the hyperparameter optimization.
Methods Documentation

fit
(population_table_training, population_table_validation, peripheral_tables=None)¶ Launches the hyperparameter optimization.
The provided
DataFrame
population_table_training, population_table_validation and peripheral_tables must be consistent with thePlaceholders
provided when constructing the base model. Parameters
population_table_training (
DataFrame
) – The population table that pipelines will be trained on.population_table_validation (
DataFrame
) – The population table that pipelines will be evaluated on.peripheral_tables (
DataFrame
) – The peripheral tables used to provide additional information for the population tables.
 Raises
TypeError – If any of population_table_training, population_table_validation or peripheral_tables is not of type
DataFrame
.KeyError – If an unsupported instance variable is encountered (via
validate()
).TypeError – If any instance variable is of wrong type (via
validate()
).ValueError – If any instance variable does not match its possible choices (string) or is out of the expected bounds (numerical) (via
validate()
).

refresh
()¶ Reloads the hyperparameter optimization from the engine.
 Raises
IOError – If the engine did not send a proper model.
 Returns
Current instance
 Return type

validate
()¶ Validate the parameters of the hyperparameter optimization.