Fine-Tuning a Spam Detector with LoRA

Overview The Full Training Pipeline

The entire workflow runs as a numbered script sequence. Each script hands off to the next, and every artifact — the data, the adapters, the fused model, the eval metrics — is written to disk so we can inspect and reproduce any step in isolation.

Step 03 Data Preparation

Downloading the Dataset

We use the sms_spam dataset from Hugging Face. The load_dataset function pulls the full corpus — 5,574 messages — and caches it locally. We request the full "train" split so we can manually slice our own training and validation sets rather than relying on Hugging Face's default splits.

from datasets import load_dataset

ds = load_dataset("sms_spam", split="train")
# → 747 spam messages, 4827 ham messagespython

The Majority-Class Problem — and How We Fix It

The raw dataset is roughly 85% ham / 15% spam. If we train on that imbalance, the model discovers a shortcut: say "not spam" for every message and score 85% accuracy without learning anything useful. We call this Majority-Class Bias, and the fix is a balanced 50/50 split.

spam = [row for row in ds if row["label"] == 1]   # 747 total
ham  = [row for row in ds if row["label"] == 0]   # 4827 total

num_train_per_class = 400
num_val_per_class   = 80

train_rows = spam[:400] + ham[:400]   # 400 spam + 400 ham
val_rows   = spam[400:480] + ham[400:480]

random.seed(42)
random.shuffle(train_rows)
random.shuffle(val_rows)

# Each row → JSONL formatted prompt
# "Classify the following SMS message as spam or not spam.\nMessage: TEXT\nClassification: LABEL"python

The Locked Seed

Setting random.seed(42) before every shuffle means the data split is identical on every run. When we change a hyperparameter and retrain, the only variable that changes is our configuration — the training data and validation data stay exactly the same. This is what makes results comparable across experiments.

Step 04 The Training Configuration

The LoRA Config Dictionary

We define all training parameters in a single Python dictionary. This translates directly into the JSON file that MLX requires, and it makes every experiment parameter visible and diffable — we never tweak a command-line flag without recording it here first.

LORA_CONFIG = {
    "model"          : str(BASELINE_DIR),      # models/baseline
    "train"          : True,                   # "Learning Mode" — updates weights
    "data"           : str(DATA_DIR),           # data/train.jsonl + data/valid.jsonl
    "seed"           : 42,                     # locked shuffle
    "lora_layers"    : 16,                     # how many transformer layers we adapt
    "batch_size"     : 4,                      # messages per gradient update
    "iters"          : 400,                    # total training steps
    "learning_rate"  : 5e-5,                   # "gentleness" of each weight update
    "adapter_path"   : str(RETRAINED_DIR / "adapters"),
}python

Understanding Each Parameter

Understanding Loss and the Three Failure Modes

Loss is the number we watch during training. It represents the model's total prediction error — lower is better. There are three distinct patterns to recognize:

Launching the Fine-Tune

python3 scripts/04_retrain.pybash

Internally, the script constructs and executes the following MLX command:

python3 -m mlx_lm lora \
  --model models/baseline \
  --train \
  --data data \
  --batch-size 4 \
  --iters 400 \
  --val-batches 10 \
  --learning-rate 5e-05 \
  --steps-per-eval 50 \
  --save-every 150 \
  --adapter-path models/retrained/adapters \
  --num-layers 16 \
  --seed 42bash

Step 04 — Results Reading the Training Logs

What the Terminal Shows Us

Every 50 iterations MLX pauses to run a validation check. We watch the Val loss column to understand whether the model is learning, plateauing, or overfitting. Here is the log from our final successful run:

Why This Run Succeeded

The key diagnostic is that the validation loss continued improving all the way to iteration 400. In our first run (8 layers, 1e-5 learning rate), the loss stalled at 2.52 after 150 steps and never recovered. Here, with 16 layers and 5e-5, the model had both the capacity (more layers) and the signal strength (higher learning rate) to keep learning past that plateau.

Step 05 Evaluation — Before vs. After

The Evaluation Script

The eval script (05_retrained_eval.py) does more than check whether answers are right. It measures how confident the model is, which gives us a much richer picture of what the fine-tuning actually changed.

The Results

What Each Improvement Means

Perplexity dropped 83% — from 29.57 down to 5.09. Before training, the model was genuinely confused by SMS-style text and the spam/not-spam framing. After training, it "expects" the word spam after seeing a prize announcement — that kind of anticipatory confidence is exactly what low perplexity measures.

Accuracy went from 48% to 69% — essentially moving from "coin flip" territory to a functional classifier in 47 seconds of training. The 20-point jump is the concrete payoff of fixing the majority-class bias in the dataset.

Step 05+ The Stress Test — Adversarial Inputs

A model that only works on clean, "textbook" examples is not a useful model. We add two adversarial messages to the evaluation set to test whether the model has learned a genuine concept of spam or is just pattern-matching on trigger words like "FREE" and "WINNER."

The Two Stress Test Cases

Stress Test Results

Accuracy barely moved — from 68.9% down to 68.42% — across the two adversarial cases. This is the result we want. A "brittle" model that had only memorized its training examples would have shown a sharp accuracy drop when confronted with novel phrasing. The near-identical scores prove the model is generalizing the underlying concept rather than relying on a lookup table of spam words.

Step 06 Fusing the Model for Production

After training, we have two separate artifacts: the original base model and the adapter file (adapters.safetensors). For production use, we want a single self-contained folder that includes everything. The fuse step mathematically bakes the adapter weights into the base model's layers — the equivalent of permanently installing a lens attachment rather than clipping it on each time.

Running the Fusion

python3 -m mlx_lm fuse \
    --model models/baseline \
    --adapter-path models/retrained/adapters \
    --save-path models/retrained/fusedbash

Verifying the Output

ls -l models/retrained/fused

# model.safetensors     ← the "brain" with adapters baked in
# config.json           ← model architecture and settings
# tokenizer.json        ← how to convert text ↔ tokens
# tokenizer_config.json ← tokenizer settingsbash

Validating Fusion Accuracy

To confirm that fusing did not degrade performance, we rename the adapter folder (forcing the eval script to fall back to the fused model) and re-run the evaluation:

mv models/retrained/adapters models/retrained/adapters_hidden
python3 scripts/05_retrained_eval.pybash

The difference between adapter mode and fused mode is 0.0009 perplexity — effectively zero. The model is production-ready. The models/retrained/fused folder is now a fully self-contained spam classifier that requires no external adapter files to operate.

Step 09 Live Interactive Testing

Starting the Chat Interface

python3 scripts/09_chat_test.pybash

The Final Boss Test Suite

We run the fused model against four progressively harder classification tasks. Each tests a different aspect of what the model learned:

Why the "Trojan Horse" and "Service Alert" Results Matter Most

The "Trojan Horse" message opens with "Hey friend, I tried calling" — the exact kind of friendly opener designed to disarm spam filters. The model looked past the social framing and caught the high-value monetary incentive ($1,000) plus shortened URL pattern that signals phishing.

The "Service Alert" message is arguably harder: it contains no prize words, no "FREE," no "WINNER." It impersonates a utility company using a fear-of-loss angle. The model recognized the authority tone + urgency + external link pattern as phishing — a pattern that trips up simpler keyword-based filters entirely.

Lessons What We Learned — Diagnosing Failures

Three distinct failures occurred before we reached the final working model. Each one taught something important about how LLM fine-tuning actually behaves in practice.

Failure 1 — Loss Explosion (learning_rate: 1e-3)

The first training attempt used a learning rate of 1e-3. Within a few iterations, the loss shot to NaN. Think of steering a car: 1e-3 is like jerking the wheel 90 degrees. The model's internal math destabilizes completely. The fix was dropping the learning rate to 2e-5 — a gentle 2-degree correction instead.

Failure 2 — Overfitting (600 iterations, 8 layers)

The second run was stable but too long. Validation loss improved through iteration 150, then started creeping back up. The model had finished learning the general patterns and was starting to memorize the specific 800 training examples — which made it perform worse on new messages. We cut iterations to 200 and lowered the learning rate further to 1e-5. The model stopped overfitting, but a new problem emerged.

Failure 3 — "Too Nice" (Majority-Class Bias)

With only 8 layers and 1e-5 learning rate, the model passed the automated eval (68.9% accuracy) but failed every live test: it classified every message — even "WINNER! Claim your prize!" — as not spam. The automated eval used a prompt format that included the label at the end; the model had learned to output "not spam" reflexively regardless of input. The root cause was the imbalanced dataset. With 85% of examples labeled "not spam," the model discovered that guessing "not spam" was statistically the safest path to a low loss.

The Iteration History

Summary The Complete ML Lifecycle

What we built here is not just a spam classifier — it is a complete, documented demonstration of the machine learning engineering process. From raw data to a production-ready fused model, every decision is traceable, every result is reproducible, and every failure was diagnosed and fixed scientifically.