Machine Learning Fundamentals

Learning a mapping from inputs to outputs using labeled data.

Discovering structure or patterns in data without labeled outcomes.

Error from simplifying assumptions that cause systematic underfitting.

Error from sensitivity to the training data; small data changes cause large model changes.

Increasing model flexibility typically lowers bias but increases variance; generalization depends on balancing both.

When a model fits training noise and performs poorly on unseen data.

When a model is too simple to capture real structure in the data.

• Train: fit parameters
• Validation: tune hyperparameters/select models
• Test: final generalization estimate

Typical: 70–80% train, 10–15% validation, 10–15% test (adjust based on dataset size).

Repeatedly training/evaluating on multiple splits to estimate performance more reliably.

When random splitting causes leakage (e.g., time series, grouped data without grouped splits).

When information from validation/test influences training, inflating performance.

Some methods depend on distances or gradients, so unscaled features can dominate learning.

Penalizing complexity to reduce overfitting.

L2 regularization penalizes large coefficients by adding a squared magnitude penalty to the objective, encouraging small, smoothly distributed weights and reducing variance:

$$\lambda \sum_{j} \beta_j^2$$

It improves stability but does not produce sparse solutions.

L1 regularization penalizes the absolute value of coefficients, encouraging sparsity and implicit feature selection:

$$\lambda \sum_{j} |\beta_j|$$

It can set some coefficients exactly to zero but may be unstable with highly correlated predictors.

• Parameters: learned from data
• Hyperparameters: set before training; control learning/model complexity

• Selection: choose using validation
• Assessment: evaluate once on locked test set