Below is a ready-to-use lab-based notebook structure.

Copy + paste each block into a Jupyter notebook, keeping the order.

🟦 Notebook Title: End-to-End Linear Regression on Messy Boston Housing Data

🟩 Markdown Cell

Lab 0 — Setup & Imports

In this notebook, we will learn machine learning step-by-step:

Load a real dataset
Introduce realistic problems (nulls, outliers, scaling issues)
Explore with Pandas
Visualize correlations
Clean & preprocess
Split train-test
Train a Linear Regression model
Evaluate and predict

🟧 Code Cell

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score
from sklearn.preprocessing import StandardScaler

plt.style.use("ggplot")

🟩 Markdown Cell

Lab 1 — Load the Boston Housing Dataset

If load_boston is unavailable, use a CSV version.

🟧 Code Cell

# Try Scikit-Learn version
try:
    from sklearn.datasets import load_boston
    boston = load_boston()
    X = pd.DataFrame(boston.data, columns=boston.feature_names)
    y = pd.Series(boston.target, name="MEDV")
    df = X.copy()
    df["MEDV"] = y
    print("Loaded from sklearn.")
except:
    print("load_boston unavailable. Loading CSV...")
    df = pd.read_csv("boston.csv")
    df.rename(columns={"medv": "MEDV"}, inplace=True)

df.head()

🟩 Markdown Cell

Lab 2 — Explore the Data with Pandas

We check:

column structure
missing values
statistical summary
potential anomalies

🟧 Code Cell

df.info()

🟧 Code Cell

df.describe()

------------------------------------

Lab 3 — Inject Real-World Mess

------------------------------------

🟩 Markdown Cell

In real machine learning projects, data is rarely clean.

So we intentionally add:

Missing values
Outliers
Scaling mistakes

🟧 Code Cell

df_with_issues = df.copy()
n_rows = df_with_issues.shape[0]

# 3.1 — Add Missing Values in RM
np.random.seed(42)
missing_indices = np.random.choice(n_rows, size=int(0.01*n_rows), replace=False)
df_with_issues.loc[missing_indices, "RM"] = np.nan

# 3.2 — Add Outliers in LSTAT
outlier_indices = np.random.choice(n_rows, size=5, replace=False)
df_with_issues.loc[outlier_indices, "LSTAT"] = df_with_issues["LSTAT"].max() * 5

# 3.3 — Incorrect Scaling (CRIM)
scale_indices = np.random.choice(n_rows, size=10, replace=False)
df_with_issues.loc[scale_indices, "CRIM"] *= 1000

df_with_issues.head()

----------------------------------------

Lab 4 — Detect Problems Using Pandas

----------------------------------------

🟩 Markdown Cell

Check for:

Nulls
Outliers
Wrong scales

🟧 Code Cell

df_with_issues.isnull().sum()

🟧 Code Cell

df_with_issues["LSTAT"].describe()

🟧 Code Cell

df_with_issues["LSTAT"].hist(bins=30)
plt.title("LSTAT Distribution (with outliers)")
plt.show()

---------------------------------------

Lab 5 — Visualize Linear Correlations

---------------------------------------

🟩 Markdown Cell

Scatter plots help us see whether a feature correlates with the target.

🟧 Code Cell

plt.scatter(df_with_issues["RM"], df_with_issues["MEDV"])
plt.xlabel("RM")
plt.ylabel("MEDV")
plt.title("RM vs MEDV")
plt.show()

🟧 Code Cell

plt.scatter(df_with_issues["LSTAT"], df_with_issues["MEDV"])
plt.xlabel("LSTAT")
plt.ylabel("MEDV")
plt.title("LSTAT vs MEDV")
plt.show()

------------------------------------

Lab 6 — Clean & Preprocess Data

------------------------------------

🟩 Markdown Cell

Steps:

Impute missing values
Handle outliers
Fix scaling issues
Standardize features

🟧 Code Cell

6.1 — Impute

df_clean = df_with_issues.copy()
df_clean["RM"].fillna(df_clean["RM"].mean(), inplace=True)

🟧 Code Cell

6.2 — Clip Outliers

Q1 = df_clean["LSTAT"].quantile(0.25)
Q3 = df_clean["LSTAT"].quantile(0.75)
IQR = Q3 - Q1
upper = Q3 + 1.5*IQR
lower = Q1 - 1.5*IQR

df_clean["LSTAT"] = df_clean["LSTAT"].clip(lower=lower, upper=upper)

🟧 Code Cell

6.3 — Fix CRIM scale

Q1c = df_clean["CRIM"].quantile(0.25)
Q3c = df_clean["CRIM"].quantile(0.75)
IQRc = Q3c - Q1c
upper_c = Q3c + 1.5 * IQRc

df_clean["CRIM"] = df_clean["CRIM"].clip(upper=upper_c)

🟧 Code Cell

6.4 — Standardize All Features

X = df_clean.drop(columns=["MEDV"])
y = df_clean["MEDV"]

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
X_scaled = pd.DataFrame(X_scaled, columns=X.columns)

-----------------------------------------

Lab 7 — Train-Test Split

-----------------------------------------

🟩 Markdown Cell

We evaluate the model on unseen test data.

🟧 Code Cell

X_train, X_test, y_train, y_test = train_test_split(
    X_scaled, y, test_size=0.2, random_state=42
)

-----------------------------------------

Lab 8 — Build Linear Regression Model

-----------------------------------------

🟩 Markdown Cell

Linear Regression attempts to fit a straight hyperplane through the data.

🟧 Code Cell

lin_reg = LinearRegression()
lin_reg.fit(X_train, y_train)

print("Intercept:", lin_reg.intercept_)
pd.Series(lin_reg.coef_, index=X_train.columns)

-----------------------------------------

Lab 9 — Evaluate the Model

-----------------------------------------

🟩 Markdown Cell

Use MSE and R².

🟧 Code Cell

y_pred = lin_reg.predict(X_test)

mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)

mse, r2

🟧 Code Cell

Plot Predicted vs Actual

plt.scatter(y_test, y_pred)
plt.xlabel("Actual MEDV")
plt.ylabel("Predicted MEDV")
plt.title("Actual vs Predicted")
plt.show()

-----------------------------------------

Lab 10 — Predict New Data

-----------------------------------------

🟩 Markdown Cell

Remember: Always apply the same pre-processing as training.

🟧 Code Cell

new_house = pd.DataFrame([{
    "CRIM": 0.1, "ZN": 0, "INDUS": 8, "CHAS": 0, "NOX": 0.5,
    "RM": 6, "AGE": 60, "DIS": 4, "RAD": 4, "TAX": 300,
    "PTRATIO": 18, "B": 390, "LSTAT": 10
}])

new_scaled = scaler.transform(new_house)
lin_reg.predict(new_scaled)

-----------------------------------------

End of Notebook

-----------------------------------------

-----------------------------------------------------------

B) Mini-Exercises for Students (After Each Lab)

-----------------------------------------------------------

✅ Lab 1 Exercises

Print the dataset shape. How many rows and columns?
What is the meaning of the target variable MEDV?

✅ Lab 2 Exercises

Which features contain the largest range between min & max?
Do any columns look like they might already be scaled?

✅ Lab 3 Exercises

Add missing values to another column.
Try multiplying INDUS by 100 for some rows—what happens in describe()?

✅ Lab 4 Exercises

Plot a histogram of CRIM. What do you notice?
Use df_with_issues.quantile([0.01, 0.99]) to inspect extreme values.

✅ Lab 5 Exercises

Plot scatter plots for
- PTRATIO vs MEDV
- TAX vs MEDV
Which seems more linear?

✅ Lab 6 Exercises

Try replacing nulls with median instead of mean.
Try removing outlier rows instead of clipping. Does accuracy change?

✅ Lab 7 Exercises

Change test_size to 0.3 and observe effect on performance.

✅ Lab 8 Exercises

Sort model coefficients from most positive to most negative.
Which features most strongly increase MEDV? Which decrease it?

✅ Lab 9 Exercises

Plot residuals = actual – predicted.
Does the model systematically overestimate or underestimate?

✅ Lab 10 Exercises

Try predicting 5 new houses at once.
Modify RM and LSTAT to see how they affect predicted price.

-------------------------------------

✔️ Ready to export as full notebook

✔️ Clean, step-by-step ML curriculum

✔️ Perfect for fresh ML students

-------------------------------------