Machine Learning · NLP · Data Quality

Email Classification
& Intelligent Triage

Can a model catch spam while avoiding false alerts that could hide legitimate email? I built and evaluated a text-classification workflow designed around that tradeoff.

Tools & Skills Python pandas scikit-learn TF-IDF Logistic Regression Linear SVC Grouped Cross-Validation

01 · Final Result

Held-out evaluation

I selected character-based TF-IDF with balanced Logistic Regression using development data, then evaluated the fixed configuration on 1,251 held-out emails.

98.24% Accuracy
94.36% Spam Precision
99.07% Spam Recall
96.66% Spam F1
Development target: 95% spam precision.
The held-out result reached 94.36%, missing the target by 0.64 percentage points. I did not change the selected model or its 0.50 threshold after seeing this result.

02 · Data & Evaluation Design

Protecting the evaluation from matching copies

The labeled dataset contains 8,348 emails. Because matching subject-and-body content appears in multiple rows, I grouped matching content before splitting the data. This prevents defined exact matches from appearing across training, validation, and held-out test sets.

8,348 Labeled Emails
5,844 Training
1,253 Validation
1,251 Held-Out Test
Email content grouping
Matching normalized email content was grouped before the split to reduce evaluation leakage from exact copies.

03 · Project Workflow

From raw email data to final evaluation

I developed the project in six stages so that data quality, model selection, and final evaluation remained separate.

01 Data Quality Review Missing values, class balance, duplicates, overlap, and email length.
02 Prepare & Split Combined subject and body and kept matching content groups together.
03 Baseline Model Compared a simple reference with word TF-IDF and Logistic Regression.
04 Model Comparison Compared word and character features, class weighting, thresholds, and grouped folds.
05 Held-Out Evaluation Froze the selected configuration and evaluated it once on the held-out test set.
06 Follow-Up Investigation Tested Linear SVC on development data and documented what should be evaluated next.

04 · Model Development

Turning email text into model features

Models cannot work directly with raw email text. I used TF-IDF to convert text patterns into numerical features, then compared Logistic Regression configurations using development data.

INPUT Email Text Subject + body
→
FEATURES Character TF-IDF 3–5 character patterns
→
CLASSIFIER Logistic Regression Balanced class weighting

TF-IDF

TF-IDF gives a text pattern more weight when it is useful in a particular email but less common across the overall training data.

The selected model used character patterns of 3–5 characters, allowing it to learn pieces of words and modified text rather than relying only on complete words.

TFIDF(t,d) = TF(t,d) · IDF(t)

Logistic Regression

Logistic Regression combines the TF-IDF features using learned weights and produces an estimated probability that an email is spam.

The selected model used a 0.50 classification threshold.

z = b + w₁x₁ + w₂x₂ + ··· + wₙxₙ
P(spam) = 1 / (1 + e−z)

Comparing candidate configurations

I compared multiple TF-IDF and Logistic Regression configurations using validation data before selecting the final model.

Model comparison
Validation comparison used during model selection. The final configuration was selected before evaluating the held-out test set.

05 · Final Evaluation

What happened on unseen email

After selecting the model using development data, I kept the configuration fixed and evaluated it once on 1,251 held-out emails. The confusion matrix shows exactly where the model was correct and where it made mistakes.

Final confusion matrix
Final held-out confusion matrix for the selected character TF-IDF and balanced Logistic Regression model.
911 Legitimate Protected
318 Spam Detected
19 False Spam Alerts
3 Missed Spam

Reading the result

The model caught 318 of 321 spam emails, leaving only 3 spam emails undetected. The more important tradeoff for this project was protecting legitimate email: 19 legitimate emails were incorrectly flagged as spam. That produced 94.36% spam precision, slightly below the 95% development target.

06 · Follow-Up Investigation

Could another classifier improve the tradeoff?

After the original held-out evaluation was complete, I tested Linear SVC using the saved development data while keeping the same character TF-IDF features. The goal was to identify a promising candidate for future evaluation — not to replace the original held-out result.

SELECTED FINAL MODEL

Logistic Regression

Validation spam precision 96.32% Validation spam recall 97.82%
FOLLOW-UP CANDIDATE

Linear SVC

Validation spam precision 97.55% Validation spam recall 99.38%

Development error comparison

12 LR False Alerts
8 SVC False Alerts
7 LR Missed Spam
2 SVC Missed Spam

Was the improvement consistent?

I also compared spam precision across five grouped training folds. Logistic Regression averaged 95.53%, while Linear SVC averaged 97.69%. The lowest Linear SVC fold was 95.42%.

Important evaluation boundary
Linear SVC was explored after the original held-out results were already known. Therefore, these development results are not a new blind final test and do not replace the Logistic Regression held-out result. The next step would be to freeze the Linear SVC configuration and evaluate it on fresh, independently labeled email.

07 · Conclusion

What this project demonstrated

This project went beyond training a classifier. The main challenge was designing a trustworthy evaluation process, choosing a model around a practical error tradeoff, and keeping later experimentation separate from the original held-out result.

WHAT I LEARNED Evaluation design matters Grouping matching email content before splitting helped prevent exact-copy leakage and made the evaluation more trustworthy.
LIMITATION One test set is not enough The held-out set is one historical sample. Near duplicates, changing spam patterns, and future email distributions can still affect performance.
NEXT STEP Evaluate on fresh email Freeze the strongest development candidate and evaluate it on fresh, independently labeled email with checked overlap and provenance.

Final takeaway: The selected Logistic Regression model caught 318 of 321 spam emails, but its 94.36% spam precision remained slightly below the 95% development target. The later Linear SVC results provide a promising direction for future testing, not a replacement for the original final result.

← Back to Portfolio Streamlit Demo ↗