Linear Regression Fundamentals

Linear regression assumes a linear relationship between features and targets. You can train quickly with sklearn.

Common metrics include MSE and R².

Despite its conceptual simplicity, linear regression remains a baseline standard in data science due to its interpretability, computational efficiency, and analytical tractability.

Mathematical Formulation

The model expresses the target y as a weighted linear combination of independent input features X plus an intercept bias term b:

y = w_1*x_1 + w_2*x_2 + ... + w_n*x_n + b

The coefficients w are fitted by minimizing the sum of squared residuals (Ordinary Least Squares, OLS).

Implementation with Scikit-Learn

Training a model in Python requires only a few lines with sklearn.linear_model:

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
model = LinearRegression()
model.fit(X_train, y_train)

predictions = model.predict(X_test)
mse = mean_squared_error(y_test, predictions)
r2 = r2_score(y_test, predictions)
print(f"Test MSE: {mse:.2f}, R2 Score: {r2:.3f}")

Diagnostic Metrics Explained

  • Mean Squared Error (MSE): Averages squared prediction discrepancies. Heavily penalizes large outlier mistakes.
  • R² Score (Coefficient of Determination): Quantifies the proportion of variance in the dependent variable explained by the features (1.0 is perfect, 0.0 equals baseline mean prediction).

Combatting Overfitting with Regularization

When dealing with multicollinearity or high feature counts, apply L2 regularization (Ridge Regression) or L1 regularization (Lasso Regression) which drives negligible feature weights toward zero for automatic feature selection.