Machine Learning Interactive

Cross-Validation

Estimate model performance on unseen data. Use all data for both training and testing by rotating through folds.

📊 How K-Fold CV Works

1. Split: Divide data into K equal folds
2. Rotate: Use K-1 folds for training, 1 for testing
3. Repeat: Train/test K times, each fold as test once
4. Average: Report mean ± std of K scores

Fold 1

Fold 2

Fold 3

Fold 4

Fold 5

■ Train ■ Test

CV Settings

2 10

50 500

📊 Data Split

Training samples 80

Test samples 20

Test % 20.0%

Accuracy Per Fold

Cross-Validation Summary

Mean Accuracy

63.3%

Std Dev

±2.7%

95% CI

58.0-68.6%

Report as: 63.3% ± 2.7% (5-fold CV)

📋 CV Methods

K-Fold

Split into K equal parts, rotate test set

Best for: General purpose

✓ Balanced, uses all data

Leave-One-Out

K = N, one sample per test fold

Best for: Small datasets

✓ Max training data

Time Series

Train on past, test on future

Best for: Sequential data

✓ Prevents leakage

Stratified

Maintain class proportions in folds

Best for: Imbalanced data

✓ Representative folds

⚠️ Time Series Data Warning

Standard K-Fold CV shuffles data randomly. For sports betting (sequential games):

• Problem: Future games could appear in training, past in test (leakage)
• Solution: Use time-series CV: always train on past, test on future
• Expanding window: Train on all data before test period
• Rolling window: Fixed training window, slides forward

R Code Equivalent

# K-Fold Cross-Validation
library(caret)

# Create folds
folds <- createFolds(y, k = 5, returnTrain = TRUE)

# Manual CV loop
cv_results <- sapply(folds, function(train_idx) { 
  model <- train_model(X[train_idx, ], y[train_idx])
  predictions <- predict(model, X[-train_idx, ])
  accuracy <- mean(predictions == y[-train_idx])
  return(accuracy)
})

cat(sprintf("CV Accuracy: %.1f%% ± %.1f%%\n", 
            mean(cv_results) * 100, sd(cv_results) * 100))

# Time-series CV (expanding window)
library(rsample)
ts_cv <- rolling_origin(data, initial = 80, assess = 20, cumulative = TRUE)

# Or using caret trainControl
ctrl <- trainControl(method = "timeslice", 
                     initialWindow = 80, horizon = 20, fixedWindow = FALSE)

✅ Key Takeaways

• CV uses all data for training AND testing
• Report mean ± std across folds
• K=5 or K=10 is standard

• Use time-series CV for sequential data
• Stratified CV for imbalanced classes
• LOOCV for small datasets