TUNING
I’ve learned that the predictions machine-learning models make, vary widely depending on the hyperparameters. However, manually testing these hyperparameters can be quite inefficient. In order to resolve that, I found a software framework to automate this process called Optuna, so I gave it a try.
PARAMETERS vs HYPERPARAMETERS
First of all, let me explain the difference between parameters and hyperparameters.
Let’s say there is a model predicting a student’s test score. So in this case, you will be using 4 parameters(Sleep Time, Study Hours, Previous Score, and Excersize Time) to predict the score and 2 hyperparameters(Use Method A and iterate that 5 times) to assign how you are going to do that.
In a nutshell, parameters are the elements used for prediction, and hyperparameters are used to help the learning process.
OPTUNA
What I thought was cool about Optuna was that it looks through multiple samplers and gets the best one for you. (I’m hoping I’m understanding this correctly)
For example, if you use GridSearch like the code below, you’ll get the best hyperparameters only considering this specific sampler.
if __name__ == "__main__":
"""Import Random Data"""
X, y = get_data()
"""Declare params you want to test out"""
classifier = ensemble.RandomForestClassifier(n_jobs=-1)
param_grid = {
"n_estimators": [100, 200, 300, 400],
"max_depth": [1, 3, 5, 7],
"criterion": ["gini", "entropy"],
}
"""Set params you want to test out"""
# GridSearchCV: Grid Search with Cross-Validation
model = model_selection.GridSearchCV(
estimator=classifier,
param_grid=param_grid,
scoring="accuracy",
verbose=10,
n_jobs=1,
cv=5
)
"""Train Model"""
model.fit(X, y)"""Results"""
best_score = 0.5356589147286822
best_param =
{'bootstrap': True, 'ccp_alpha': 0.0, 'class_weight': None, 'criterion': 'entropy', 'max_depth': 7, 'max_features': 'auto', 'max_leaf_nodes': None, 'max_samples': None, 'min_impurity_decrease': 0.0, 'min_samples_leaf': 1, 'min_samples_split': 2, 'min_weight_fraction_leaf': 0.0, 'n_estimators': 300, 'n_jobs': -1, 'oob_score': False, 'random_state': None, 'verbose': 0, 'warm_start': False}Here is the code for Optuna.
After you import the data, you create an “optimize” function assigning the hyperparameters, then you start the optimization.
It’s kind of obvious but the scores were higher than the GridSearch code I wrote above.
if __name__ == "__main__":
"""Import Random Data"""
X, y = get_data()
"""Create Optimization Function"""
optimization_function = partial(optimize, x=X, y=y)
"""Start Study"""
# Direction: Where you want to maximize or minimize the cross_val_score
study = optuna.study.create_study(direction="minimize")
study.optimize(optimization_function, n_trials=50)Note that compared to GridSearch, you’re not assigning a specific number for the hyperparameter. Instead, you assign a range of numbers.
def optimize(trial, x, y):
"""Declare range of hyper-parameter to consider"""
# Function to measure quality of split
criterion = trial.suggest_categorical("criterion", ["gini", "entropy"])
# The number of trees in the forest.
n_estimators = trial.suggest_int("n_estimators", 100, 1500)
max_depth = trial.suggest_int("max_depth", 3, 15)
# The number of features to consider when looking for the best split:
max_features = trial.suggest_uniform("max_features", 0.01, 1.0)
"""Set hyper-parameter"""
model = ensemble.RandomForestClassifier(
n_estimators=n_estimators,
max_depth=max_depth,
max_features=max_features,
criterion=criterion
)
"""Assign Split Counts you Desire"""
kf = model_selection.StratifiedKFold(n_splits=5)
accuracies = []
"""Get accuracy score for each train_set"""
for idx in kf.split(X=x, y=y):
train_idx, test_idx = idx[0], idx[1]
xtrain = x[train_idx]
ytrain = y[train_idx]
xtest = x[test_idx]
ytest = y[test_idx]
model.fit(xtrain, ytrain)
preds = model.predict(xtest)
fold_acc = metrics.accuracy_score(ytest, preds)
accuracies.append(fold_acc)
return -1.0*np.mean(accuracies)"""Results"""
Trial 21 finished with value: -0.5471899224806202
parameters:
{'criterion': 'entropy', 'n_estimators': 612, 'max_depth': 9, 'max_features': 0.6471869396923406}.
Best is trial 21 with value: -0.5471899224806202.RESULTS
Out of 50 trials, trial 21 was the best. I found out more doesn’t mean better.
I’m still not sure about the optimal numbers for iterations so I’ll try to experiment more.
For my next step, I want to be able to tune CNN(Image Recognition) then someday GAN(Image Generation).
