Machine Learning Interactive

Hyperparameter Tuning

Find optimal model settings. Grid search is exhaustive, random search is efficient, Bayesian optimization learns from past trials.

📊 Parameters vs Hyperparameters

Parameters

Learned from data during training

• Neural network weights
• Regression coefficients
• Tree split thresholds

Hyperparameters

Set before training, control learning

• Learning rate, regularization
• Number of trees, max depth
• Number of layers, neurons

Search Parameters

5 20

3 15

5 50

📊 Search Results

Grid Search 0.868

5 trials

Random Search 0.872

20 trials

Bayesian 0.844

10 trials (most efficient)

Validation Curve

Training score: Always improves (overfitting)

Validation score: Peaks at optimal

Search Method Comparison

Grid Search

✓ Exhaustive, reproducible

✗ Exponential in dims

Best for: ≤3 hyperparams

Random Search

✓ Efficient in high dims

✗ No learning

Best for: Many hyperparams

Bayesian

✓ Learns from trials

✗ Overhead, sequential

Best for: Expensive evaluations

🏈 Hyperparameters in Betting Models

XGBoost

max_depth, learning_rate, n_estimators, reg_lambda

Neural Net

layers, neurons, dropout, batch_size

Elo System

K-factor, home advantage, initial rating

Kelly

Fraction (full/half/quarter), bankroll

R Code Equivalent

# Hyperparameter tuning with caret
library(caret)

# Grid search
grid <- expand.grid(
  nrounds = c(50, 100, 150),
  max_depth = c(3, 5, 7),
  eta = c(0.01, 0.1, 0.3),
  gamma = 0,
  colsample_bytree = 0.8,
  min_child_weight = 1,
  subsample = 0.8
)

ctrl <- trainControl(
  method = "cv",
  number = 5,
  search = "grid"
)

# Train with grid search
model <- train(
  y ~ ., data = train_data,
  method = "xgbTree",
  trControl = ctrl,
  tuneGrid = grid
)

# Random search alternative
ctrl_random <- trainControl(
  method = "cv",
  number = 5,
  search = "random"
)

model_random <- train(
  y ~ ., data = train_data,
  method = "xgbTree",
  trControl = ctrl_random,
  tuneLength = 20
)

# Best hyperparameters
print(model$bestTune)

✅ Key Takeaways

• Hyperparameters control the learning process
• Grid search: exhaustive but slow
• Random search: efficient in high dimensions

• Bayesian: learns from past trials
• Always use validation set, not training
• Cross-validation for robust estimates