Hyperparameter Tuning in Python: GridSearchCV, Optuna, and Bayesian Optimization Compared

Why Hyperparameter Tuning Matters

Every machine learning model has two kinds of settings: parameters that the model learns from data during training (like weights in a neural network or split points in a decision tree) and hyperparameters that you set before training begins. Hyperparameters control the learning process itself — the depth of a tree, the learning rate of a gradient boosting model, or the regularization strength of a logistic regression.

Getting hyperparameters wrong leads to one of two failure modes. Values that are too permissive let the model memorize noise in the training set (overfitting), while values that are too restrictive prevent it from capturing real patterns (underfitting). Tuning is the systematic process of finding the sweet spot between the two.

And honestly, it can make a bigger difference than most people expect — I've routinely seen 5–15% improvements in model performance just from proper tuning on real-world datasets.

In this guide, you'll learn how to use every major tuning strategy available in 2026 — from scikit-learn's built-in search methods to Optuna 4.7's Bayesian optimization engine — with working code you can drop straight into a Jupyter notebook.

Prerequisites

This tutorial assumes you have Python 3.10 or later. You'll also need these packages:

pip install scikit-learn==1.8.0 optuna==4.7.0 pandas numpy matplotlib

We'll use the California Housing dataset throughout so you can compare every strategy on the same data. Nothing fancy here — just a clean, well-known regression benchmark:

import numpy as np
import pandas as pd
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error, r2_score

data = fetch_california_housing(as_frame=True)
X, y = data.data, data.target
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

Strategy 1 — Grid Search with GridSearchCV

How Grid Search Works

GridSearchCV is the brute-force baseline. You hand it a dictionary of every hyperparameter value you want to test, and it evaluates every single combination using k-fold cross-validation. With three hyperparameters and five values each, that's 5 × 5 × 5 = 125 fits multiplied by the number of CV folds.

Yeah, it's not subtle.

Complete Code Example

from sklearn.model_selection import GridSearchCV

param_grid = {
    "n_estimators": [100, 200, 300],
    "max_depth": [10, 20, 30, None],
    "min_samples_split": [2, 5, 10],
    "min_samples_leaf": [1, 2, 4],
}

grid_search = GridSearchCV(
    estimator=RandomForestRegressor(random_state=42),
    param_grid=param_grid,
    cv=5,
    scoring="neg_mean_squared_error",
    n_jobs=-1,
    verbose=1,
    return_train_score=True,
)

grid_search.fit(X_train, y_train)

print(f"Best parameters: {grid_search.best_params_}")
print(f"Best CV RMSE:    {np.sqrt(-grid_search.best_score_):.4f}")

best_model = grid_search.best_estimator_
y_pred = best_model.predict(X_test)
print(f"Test RMSE:       {np.sqrt(mean_squared_error(y_test, y_pred)):.4f}")
print(f"Test R²:         {r2_score(y_test, y_pred):.4f}")

When to Use Grid Search

• Your search space is small (fewer than ~200 total combinations).
• You need an exhaustive guarantee that no combination was missed.
• You're tuning a fast model like logistic regression or a small SVM.

When to Avoid Grid Search

• Large parameter spaces — runtime grows exponentially with the number of hyperparameters.
• Expensive models like deep ensembles or neural networks where each fit takes minutes. Trust me, you don't want to wait.

Strategy 2 — Randomized Search with RandomizedSearchCV

How Randomized Search Works

Instead of testing every combination, RandomizedSearchCV samples a fixed number of candidates from the parameter space at random. Research by Bergstra and Bengio (2012) showed that random search finds equally good hyperparameters as grid search in a fraction of the time. The reason is kind of intuitive once you think about it — most hyperparameters have little effect on performance, and random sampling is more likely to explore the dimensions that actually matter.

Complete Code Example

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint, uniform

param_distributions = {
    "n_estimators": randint(100, 500),
    "max_depth": randint(5, 50),
    "min_samples_split": randint(2, 20),
    "min_samples_leaf": randint(1, 10),
    "max_features": uniform(0.1, 0.9),
}

random_search = RandomizedSearchCV(
    estimator=RandomForestRegressor(random_state=42),
    param_distributions=param_distributions,
    n_iter=100,
    cv=5,
    scoring="neg_mean_squared_error",
    n_jobs=-1,
    random_state=42,
    verbose=1,
)

random_search.fit(X_train, y_train)

print(f"Best parameters: {random_search.best_params_}")
print(f"Best CV RMSE:    {np.sqrt(-random_search.best_score_):.4f}")

y_pred = random_search.best_estimator_.predict(X_test)
print(f"Test RMSE:       {np.sqrt(mean_squared_error(y_test, y_pred)):.4f}")

Key Advantage: Continuous Distributions

Unlike grid search, which requires a discrete list of values, RandomizedSearchCV accepts probability distributions. This lets the sampler explore the continuous space of real-valued hyperparameters (things like learning rate or regularization strength) far more effectively than picking a handful of grid points ever could.

Strategy 3 — Successive Halving with HalvingGridSearchCV

The Tournament Analogy

Scikit-learn 1.8 ships HalvingGridSearchCV and HalvingRandomSearchCV as experimental APIs. Both use the successive halving algorithm, which works like a tournament: all candidates start with a small resource budget (by default, a fraction of the training samples), and only the top performers survive to the next round where they get more resources.

It's a clever idea — why waste compute on obviously bad configurations?

Complete Code Example

from sklearn.experimental import enable_halving_search_cv  # noqa
from sklearn.model_selection import HalvingRandomSearchCV
from scipy.stats import randint, uniform

param_distributions = {
    "n_estimators": randint(50, 500),
    "max_depth": randint(3, 50),
    "min_samples_split": randint(2, 20),
    "min_samples_leaf": randint(1, 10),
    "max_features": uniform(0.1, 0.9),
}

halving_search = HalvingRandomSearchCV(
    estimator=RandomForestRegressor(random_state=42),
    param_distributions=param_distributions,
    factor=3,
    cv=5,
    scoring="neg_mean_squared_error",
    n_jobs=-1,
    random_state=42,
    verbose=1,
)

halving_search.fit(X_train, y_train)

print(f"Best parameters: {halving_search.best_params_}")
print(f"Best CV RMSE:    {np.sqrt(-halving_search.best_score_):.4f}")

y_pred = halving_search.best_estimator_.predict(X_test)
print(f"Test RMSE:       {np.sqrt(mean_squared_error(y_test, y_pred)):.4f}")

Choosing the factor Parameter

The factor parameter controls how aggressively candidates are eliminated. A value of 3 (the default) means only the top third survive each round. Higher values finish faster but risk eliminating good candidates too early. In my experience, sticking with the default of 3 is a safe bet for most workloads.

Strategy 4 — Bayesian Optimization with Optuna

Why Bayesian Optimization Wins

Here's where things get really interesting.

Grid search and random search are uninformed — each trial is independent and ignores the results of previous trials. Bayesian optimization takes a fundamentally different approach. It maintains a probabilistic model (a surrogate) of the relationship between hyperparameters and model performance. After each trial, the surrogate gets updated, and the next set of hyperparameters is chosen to maximize the expected improvement.

The practical upshot? Bayesian methods typically find better hyperparameters in 20–50 trials than random search finds in 200. That's not a typo.

Optuna 4.7 — What's New in 2026

Optuna 4.7.0, released January 2026, brings some genuinely useful upgrades:

• Faster GPSampler — parallelized multi-start acquisition via PyTorch batching and optimized NumPy operations.
• AutoSampler — automatic sampler selection for multi-objective and constrained optimization problems.
• SMAC3 Integration — the SMAC3 sampler from AutoML.org is now available through OptunaHub.
• LLM-powered Dashboard — natural language trial filtering and automatic Plotly chart generation in Optuna Dashboard v0.20.
• Python 3.13 support (and Python 3.8 has been dropped).

Complete Code Example

import optuna
from sklearn.model_selection import cross_val_score

def objective(trial):
    params = {
        "n_estimators": trial.suggest_int("n_estimators", 100, 500),
        "max_depth": trial.suggest_int("max_depth", 3, 50),
        "min_samples_split": trial.suggest_int("min_samples_split", 2, 20),
        "min_samples_leaf": trial.suggest_int("min_samples_leaf", 1, 10),
        "max_features": trial.suggest_float("max_features", 0.1, 1.0),
    }

    model = RandomForestRegressor(**params, random_state=42, n_jobs=-1)
    scores = cross_val_score(
        model, X_train, y_train,
        cv=5, scoring="neg_mean_squared_error"
    )
    return scores.mean()

# Suppress Optuna info logs for cleaner output
optuna.logging.set_verbosity(optuna.logging.WARNING)

study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=50)

print(f"Best parameters: {study.best_params}")
print(f"Best CV RMSE:    {np.sqrt(-study.best_value):.4f}")

# Train the final model with optimal parameters
best_model = RandomForestRegressor(**study.best_params, random_state=42, n_jobs=-1)
best_model.fit(X_train, y_train)
y_pred = best_model.predict(X_test)
print(f"Test RMSE:       {np.sqrt(mean_squared_error(y_test, y_pred)):.4f}")
print(f"Test R²:         {r2_score(y_test, y_pred):.4f}")

Visualizing Optimization History

Optuna includes built-in visualization functions that are genuinely helpful for understanding what the optimizer is doing under the hood:

from optuna.visualization import (
    plot_optimization_history,
    plot_param_importances,
    plot_slice,
)

# Show how the objective improved over trials
fig1 = plot_optimization_history(study)
fig1.show()

# Which hyperparameters had the most impact?
fig2 = plot_param_importances(study)
fig2.show()

# How does each hyperparameter affect the objective?
fig3 = plot_slice(study)
fig3.show()

Pruning Unpromising Trials Early

One of Optuna's killer features is trial pruning. When a trial reports intermediate values (say, validation loss at each epoch of a neural network), Optuna can stop the trial early if it's clearly underperforming. This is especially useful for models with expensive training loops — it can save you hours of wasted compute:

from sklearn.model_selection import StratifiedKFold

def objective_with_pruning(trial):
    params = {
        "n_estimators": trial.suggest_int("n_estimators", 50, 500, step=50),
        "max_depth": trial.suggest_int("max_depth", 3, 30),
        "min_samples_split": trial.suggest_int("min_samples_split", 2, 20),
    }

    model = RandomForestRegressor(**params, random_state=42, warm_start=True)
    kf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

    fold_scores = []
    for step, (train_idx, val_idx) in enumerate(kf.split(X_train, pd.qcut(y_train, 5, labels=False))):
        X_fold_train, X_fold_val = X_train.iloc[train_idx], X_train.iloc[val_idx]
        y_fold_train, y_fold_val = y_train.iloc[train_idx], y_train.iloc[val_idx]

        model.fit(X_fold_train, y_fold_train)
        score = -mean_squared_error(y_fold_val, model.predict(X_fold_val))
        fold_scores.append(score)

        # Report intermediate value and check for pruning
        trial.report(np.mean(fold_scores), step)
        if trial.should_prune():
            raise optuna.TrialPruned()

    return np.mean(fold_scores)

pruning_study = optuna.create_study(
    direction="maximize",
    pruner=optuna.pruners.MedianPruner(n_startup_trials=5, n_warmup_steps=2),
)
pruning_study.optimize(objective_with_pruning, n_trials=50)

Strategy 5 — Tuning Gradient Boosting Models

XGBoost with Optuna

Gradient boosting models like XGBoost, LightGBM, and CatBoost have larger hyperparameter spaces than random forests, which makes Bayesian optimization especially valuable here. Below is a production-ready template for tuning XGBoost that I've used on multiple projects:

import xgboost as xgb

def xgb_objective(trial):
    params = {
        "n_estimators": trial.suggest_int("n_estimators", 100, 1000),
        "max_depth": trial.suggest_int("max_depth", 3, 12),
        "learning_rate": trial.suggest_float("learning_rate", 0.01, 0.3, log=True),
        "subsample": trial.suggest_float("subsample", 0.5, 1.0),
        "colsample_bytree": trial.suggest_float("colsample_bytree", 0.5, 1.0),
        "reg_alpha": trial.suggest_float("reg_alpha", 1e-8, 10.0, log=True),
        "reg_lambda": trial.suggest_float("reg_lambda", 1e-8, 10.0, log=True),
        "min_child_weight": trial.suggest_int("min_child_weight", 1, 10),
        "gamma": trial.suggest_float("gamma", 1e-8, 1.0, log=True),
    }

    model = xgb.XGBRegressor(
        **params,
        random_state=42,
        n_jobs=-1,
        tree_method="hist",
    )

    scores = cross_val_score(
        model, X_train, y_train,
        cv=5, scoring="neg_mean_squared_error"
    )
    return scores.mean()

xgb_study = optuna.create_study(direction="maximize")
xgb_study.optimize(xgb_objective, n_trials=100)
print(f"Best XGBoost RMSE: {np.sqrt(-xgb_study.best_value):.4f}")

Using log=True for Scale-Sensitive Hyperparameters

Notice the log=True argument for learning rate and regularization parameters. This tells Optuna to sample on a logarithmic scale, which is essential when the hyperparameter spans several orders of magnitude (for example, 0.001 to 1.0). Without log-scaling, the sampler would spend most of its budget testing values near the upper end of the range — which is almost never where the good values live.

Integrating Tuning into a Scikit-Learn Pipeline

A common mistake (and one I've definitely made myself early on) is tuning hyperparameters on pre-processed data without including the preprocessing in the cross-validation loop. This causes data leakage — information from the validation fold leaks into the training fold via shared scaling or encoding.

The fix is straightforward: wrap everything in a Pipeline.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA

pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("pca", PCA()),
    ("model", RandomForestRegressor(random_state=42)),
])

param_grid = {
    "pca__n_components": [5, 6, 7, 8],
    "model__n_estimators": [100, 200, 300],
    "model__max_depth": [10, 20, None],
}

pipe_search = GridSearchCV(pipe, param_grid, cv=5, scoring="neg_mean_squared_error", n_jobs=-1)
pipe_search.fit(X_train, y_train)
print(f"Best pipeline params: {pipe_search.best_params_}")

Head-to-Head Comparison

So, how do all these strategies stack up against each other? Here's a quick reference table:

Strategy	Search Type	Learns from Past Trials	Typical Trials Needed	Best For
GridSearchCV	Exhaustive	No	All combinations	Small search spaces, guarantees
RandomizedSearchCV	Random sampling	No	50–200	Medium spaces, continuous params
HalvingRandomSearchCV	Tournament	No	Many candidates, few rounds	Large spaces, fast models
Optuna (TPE)	Bayesian	Yes	20–100	Expensive models, large spaces
Optuna (GP)	Bayesian (Gaussian Process)	Yes	10–50	Very expensive models, few params

Best Practices for Production Tuning

1. Always Hold Out a Final Test Set

Cross-validation during tuning estimates generalization performance, but here's the thing — the entire tuning process still makes decisions based on the training data. Always evaluate your final model on a held-out test set that was never seen during any tuning step. No exceptions.

2. Set Random Seeds Everywhere

Reproducibility matters more than most people realize. Set random_state in your model, in the train-test split, and in the search object itself. For Optuna, use a fixed sampler seed:

sampler = optuna.samplers.TPESampler(seed=42)
study = optuna.create_study(direction="maximize", sampler=sampler)

3. Use Log-Scale Sampling for Rate Parameters

Learning rates, regularization strengths, and other parameters that span multiple orders of magnitude should be sampled on a log scale. In scikit-learn use loguniform; in Optuna use suggest_float(..., log=True). This single change can dramatically improve search efficiency.

4. Start Coarse, Then Refine

Run a broad randomized search first to identify the promising region of the search space, then zoom in with a tighter grid or Bayesian search. This two-stage approach is far more efficient than starting with a dense grid — and it's what most experienced ML engineers actually do in practice.

5. Monitor for Overfitting on Validation Folds

Enable return_train_score=True in scikit-learn search objects or log both training and validation metrics in Optuna. A large gap between training and validation scores is a red flag for overfitting to the CV folds.

6. Use Optuna's Dashboard for Complex Searches

For studies with 50+ trials, the Optuna Dashboard is worth setting up for interactive visualization:

pip install optuna-dashboard
optuna-dashboard sqlite:///optuna_study.db

The dashboard shows optimization history, hyperparameter importance, parallel coordinate plots, and — new in v0.20 — lets you query trials using natural language. It's surprisingly handy when you're trying to figure out why certain parameter regions keep getting explored.

Frequently Asked Questions

What is the difference between a parameter and a hyperparameter?

Parameters are learned from data during model training (for example, the coefficients in linear regression). Hyperparameters are set by you before training starts (for example, the number of trees in a random forest). You tune hyperparameters; the algorithm tunes parameters.

How many trials should I run for hyperparameter tuning?

It depends on the strategy. Grid search must evaluate every combination — there's no shortcut there. For randomized search, 100–200 trials is a reasonable starting point. With Bayesian optimization (Optuna), 30–100 trials often suffice because each trial is informed by previous results. A practical tip: monitor the optimization history plot, and when the best value plateaus for 20–30 consecutive trials, you're probably done.

Is Optuna better than GridSearchCV?

For small, well-understood search spaces, GridSearchCV is simple and exhaustive — and there's real value in simplicity. For larger spaces or expensive models, Optuna is almost always more efficient. It finds comparable or better hyperparameters with far fewer trials because its Bayesian sampler learns from past evaluations.

Can I use Optuna with deep learning frameworks like PyTorch?

Absolutely. Optuna integrates with PyTorch, TensorFlow, Keras, XGBoost, LightGBM, CatBoost, and many other libraries. Its pruning feature is especially powerful for neural networks because it can stop underperforming training runs after just a few epochs — saving potentially hours of GPU time.

How do I avoid overfitting during hyperparameter tuning?

Use k-fold cross-validation instead of a single train/validation split. Always evaluate the final model on a held-out test set that was never used during tuning. And enable return_train_score=True to monitor the gap between training and validation metrics — if that gap starts growing, you've got a problem.