Photo by editor
# Introduction
Tuning hyperparameters Modeling machine learning is, to some extent, an art or craft, requiring the right skill to balance experience, intuition and considerable experimentation. In practice, this process can sometimes appear difficult because sophisticated models have a large search space, interactions between hyperparameters are complex, and the performance gain due to their adjustment is sometimes subtle.
Below, we compile a list of 7 SkyLearn tricks to take your hyperparameter tuning skills of machine learning models to the next level.
# 1. Limiting the search space with domain knowledge
Not forcing an otherwise extensive search space means looking for a needle in the middle of a (big) haystack! Enlist domain knowledge—or, if necessary—a domain expert to first define a set of well-chosen bounds for some of the relevant hyperparameters in your model. This will help to reduce the complexity and increase the feasibility of the operation process, which will rule out impractical settings.
An example grid for two common hyperparameters in random forest instances might look like:
param_grid = {"max_depth": (3, 5, 7), "min_samples_split": (2, 10)}
# 2. Starting with a random search
For low-budget contexts, try to exploit random search, an efficient approach to search large search spaces that sample some hyperparameter value ranges, by incorporating a distribution-driven sampling process. Just like in this example for sampling Ci.e., the hyperparameter that controls the stiffness in the SVM models’ bounds:
param_dist = {"C": loguniform(1e-3, 1e2)}
RandomizedSearchCV(SVC(), param_dist, n_iter=20)
# 3. Local optimization with grid search
After finding associated regions with a random search, it is sometimes a good idea to apply a narrow focus grid search to further explore these regions to identify marginal gains. Research first, exploitation second.
GridSearchCV(SVC(), {"C": (5, 10), "gamma": (0.01, 0.1)})
# 4. Encapsulating preprocessing pipelines within hyperparameter tuning
SkyLearn pipelines are a great way to simplify and optimize end-to-end machine learning workflows and prevent such problems. Data leakage. If we pass a pipeline as a search instance, both preprocessing and model hyperparameters can be combined together, like this:
param_grid = {
"scaler__with_mean": (True, False), # Scaling hyperparameter
"clf__C": (0.1, 1, 10), # SVM model hyperparameter
"clf__kernel": ("linear", "rbf") # Another SVM hyperparameter
}
grid_search = GridSearchCV(pipeline, param_grid, cv=5)
grid_search.fit(X_train, y_train)
# 5. Trade speed for reliability with cross-validation
Although the application of cross-validation is routine in SkyLearn-powered hyperparameter tuning, it is worth noting that omitting it means that a single-train validation distribution is used: it is faster but produces more variable and sometimes less reliable results. Increasing the number of cross-validation layers—eg cv=5 – Increases performance stability for comparison between models. Find a price that strikes the right balance for you:
GridSearchCV(model, params, cv=5)
# 6. Optimizing multiple metrics
When there are multiple performance trade-offs, monitoring your tuning process helps reveal trade-offs across multiple metrics that may be inadvertent when applying single-score optimizations. Also, you can use refit Specifying the primary objective to determine the final, “best” model.
from sklearn.model_selection import GridSearchCV
param_grid = {
"C": (0.1, 1, 10),
"gamma": (0.01, 0.1)
}
scoring = {
"accuracy": "accuracy",
"f1": "f1"
}
gs = GridSearchCV(
SVC(),
param_grid,
scoring=scoring,
refit="f1", # metric used to select the final model
cv=5
)
gs.fit(X_train, y_train)# 7. Interpret the results wisely
Once your tuning process is finished, and the best score model is found, go the extra mile by using cv_results_ To better understand parameter interactions, trends, etc., or visualize the results if you prefer. This example creates a report and a hierarchy of results called the GridSearch object gsafter completing the search and training process:
import pandas as pd
results_df = pd.DataFrame(gs.cv_results_)
# Target columns for our report
columns_to_show = (
'param_clf__C',
'mean_test_score',
'std_test_score',
'mean_fit_time',
'rank_test_score'
)
print(results_df(columns_to_show).sort_values('rank_test_score'))
# wrap up
Hyperparameter tuning is most effective when it is both systematic and deliberate. By combining smart search strategies, proper validation, and careful interpretation of results, you can make meaningful performance gains without computing or overfitting. Treat tuning as an iterative learning process, not an optimization checkbox.
Ivan Palomares Carrascosa Is a leader, author, speaker, and consultant in AI, Machine Learning, Deep Learning, and LLMS. He trains and guides others in real-world applications of AI.