Fine-Tuning LLMs on Apple Silicon

Part 00 Understanding Model Size

When people say "8 GB model," they almost always mean 8 billion parameters — not 8 gigabytes of storage. Here is how those numbers actually map to real-world hardware requirements:

Part 01 The Full Pipeline

From a blank terminal to a custom model running in Ollama, the pipeline has seven distinct phases. Each one builds on the last:

Part 02 Choosing Your Base Model

You never train a large model from scratch in 2026. You start from a pretrained base and customize it — think of it as tuning a precision engine rather than casting one from raw steel.

Downloading the Base Model

# Install Hugging Face CLI
pip install huggingface_hub

# Login with your HF token
huggingface-cli login

# Download Llama 3.1 8B Instruct
huggingface-cli download meta-llama/Llama-3.1-8B-Instruct \
  --local-dir ./models/llama-3.1-8b-instruct \
  --local-dir-use-symlinks Falsebash

Part 03 Training Data: What Makes a Model Smart

Data quality is the single most important variable in how capable your fine-tuned model becomes. Excellent training code on bad data still produces a bad model — there are no shortcuts here.

What Kind of Data You Need

For a model that sounds like a knowledgeable, natural human, you want: conversation pairs (user message + assistant response), Q&A datasets with high-quality answers, reasoning chains showing step-by-step thinking, domain-specific knowledge for your target use case, and diversity across topics, styles, and lengths.

Data Format — What MLX Expects

MLX fine-tuning expects JSONL format — one JSON object per line, using the standard conversation structure:

{"messages": [
  {"role": "system",    "content": "You are a helpful assistant."},
  {"role": "user",      "content": "What is photosynthesis?"},
  {"role": "assistant", "content": "Photosynthesis is the process..."}
]}jsonl

Downloading and Preparing Data

from datasets import load_dataset
import json

ds = load_dataset('teknium/OpenHermes-2.5', split='train')
ds = ds.shuffle(seed=42).select(range(100000))  # filter to 100k rows

def to_chat_format(row):
    messages = []
    if row.get('system_prompt'):
        messages.append({'role': 'system', 'content': row['system_prompt']})
    messages.append({'role': 'user',      'content': row['instruction']})
    messages.append({'role': 'assistant', 'content': row['output']})
    return {'messages': messages}

converted = [to_chat_format(r) for r in ds]
split = int(len(converted) * 0.95)  # 95/5 train/valid split

with open('data/train.jsonl', 'w') as f:
    for row in converted[:split]: f.write(json.dumps(row) + '\n')
with open('data/valid.jsonl', 'w') as f:
    for row in converted[split:]: f.write(json.dumps(row) + '\n')python

Part 04 Environment Setup

# 1. Install Homebrew
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

# 2. Install Python 3.11 via pyenv
brew install pyenv
pyenv install 3.11.9
pyenv global 3.11.9

# 3. Create virtual environment
python -m venv ~/llm-env
source ~/llm-env/bin/activate

# 4. Install MLX and fine-tuning tools
pip install mlx mlx-lm
pip install huggingface_hub datasets transformers
pip install numpy tqdm sentencepiece

# 5. Verify MLX sees Apple Silicon
python -c "import mlx.core as mx; print(mx.default_device())"
# Should print: Device(gpu, 0)bash

Part 05 Measuring Intelligence: Baseline First

Before touching any training, establish your baseline benchmarks. This is non-negotiable — you cannot claim a model improved if you did not measure where it started.

Quick Sanity Test

from mlx_lm import load, generate

model, tokenizer = load('./models/llama-3.1-8b-instruct')

test_prompts = [
    'Explain quantum entanglement simply.',
    'Write a Python function to reverse a linked list.',
    'What is the difference between empathy and sympathy?',
    'If you have 3 apples and give away 1.5, how many remain?',
]

for prompt in test_prompts:
    messages = [{'role': 'user', 'content': prompt}]
    text = tokenizer.apply_chat_template(messages, tokenize=False)
    response = generate(model, tokenizer, prompt=text, max_tokens=300)
    print(f'PROMPT: {prompt}')
    print(f'RESPONSE: {response}')
    print('---')python

Formal Benchmarks

pip install lm-eval

lm_eval --model hf \
  --model_args pretrained=./models/llama-3.1-8b-instruct \
  --tasks hellaswag,arc_easy,arc_challenge,winogrande,mmlu \
  --device mps \
  --output_path ./evals/baseline/bash

Part 06 LoRA Fine-Tuning with MLX

What is LoRA?

LoRA (Low-Rank Adaptation) makes fine-tuning an 8B model feasible on 16 GB of RAM. Rather than updating all 8 billion weights — which would require 80–160 GB VRAM — LoRA freezes the base model entirely and trains small adapter matrices layered on top of the attention layers. At the end, those adapters merge back in seamlessly.

Running the Training

mlx_lm.lora \
  --model ./models/llama-3.1-8b-instruct \
  --train \
  --data ./data \
  --batch-size 2 \
  --lora-layers 16 \
  --iters 2000 \
  --learning-rate 1e-5 \
  --grad-checkpoint \
  --adapter-path ./adapters \
  --save-every 200 \
  --val-every 100 \
  --max-seq-length 2048bash

What to Watch During Training

Part 07 Validating That the Model Is Learning

This is where most people skip ahead — don't. Validation is the mechanism that distinguishes a model that actually improved from one that just memorized your training set or quietly degraded its general capability.

Test the Fine-Tuned Model

from mlx_lm import load, generate

# Load base model with your LoRA adapter active
model, tokenizer = load(
    './models/llama-3.1-8b-instruct',
    adapter_path='./adapters'
)

# Use the SAME prompts as your baseline — this is the comparison
test_prompts = [
    'Explain quantum entanglement simply.',
    'Write a Python function to reverse a linked list.',
    'What is the difference between empathy and sympathy?',
]

for prompt in test_prompts:
    messages = [{'role': 'user', 'content': prompt}]
    text = tokenizer.apply_chat_template(messages, tokenize=False)
    response = generate(model, tokenizer, prompt=text, max_tokens=400)
    print(f'PROMPT: {prompt[:50]}...')
    print(f'RESPONSE: {response}')
    print('===')python

Human Quality Evaluation

Read 20 responses and score each dimension on a 1–5 scale. This is still the gold standard — benchmarks measure breadth, human review measures quality.

Re-Run Formal Benchmarks

# Merge adapters first — required for lm-eval
python -m mlx_lm.fuse \
  --model ./models/llama-3.1-8b-instruct \
  --adapter-path ./adapters \
  --save-path ./models/finetuned-merged

# Benchmark the merged model
lm_eval --model hf \
  --model_args pretrained=./models/finetuned-merged \
  --tasks hellaswag,arc_easy,arc_challenge,winogrande,mmlu \
  --device mps \
  --output_path ./evals/finetuned/

# A drop of >3 points on MMLU = catastrophic forgetting
# Ideal result: same or better on knowledge, improved on conversation stylebash

Part 08 Merging, Quantizing & GGUF

Step 1: Fuse LoRA into the Base Model

python -m mlx_lm.fuse \
  --model ./models/llama-3.1-8b-instruct \
  --adapter-path ./adapters \
  --save-path ./models/finetuned-merged \
  --de-quantize  # ensures full float16 output for best qualitybash

Step 2: Quantize to 4-bit

Quantization compresses from ~15 GB (float16) down to ~4.5 GB (4-bit int). Inference becomes significantly faster and the model fits comfortably alongside other running applications.

python -m mlx_lm.convert \
  --hf-path ./models/finetuned-merged \
  --mlx-path ./models/finetuned-4bit \
  -q --q-bits 4 --q-group-size 64bash

Step 3: Convert to GGUF

# Build llama.cpp with Metal support
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DLLAMA_METAL=ON -DLLAMA_CURL=ON
cmake --build build --config Release -j $(sysctl -n hw.ncpu)

# Convert to GGUF (float16 first)
python convert_hf_to_gguf.py \
  ./models/finetuned-merged \
  --outfile ./models/my-llm-f16.gguf \
  --outtype f16

# Quantize to Q4_K_M — best quality/size tradeoff
./build/bin/llama-quantize \
  ./models/my-llm-f16.gguf \
  ./models/my-llm-q4_k_m.gguf \
  Q4_K_Mbash

Part 09 Deploying to Ollama

# Create Modelfile
cat > Modelfile << 'EOF'
FROM ./models/my-llm-q4_k_m.gguf

SYSTEM """
You are a highly intelligent, helpful assistant. You are direct,
thoughtful, and give thorough answers when needed but concise answers
when brevity is appropriate. You admit when you don't know something
and never fabricate facts.
"""

PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER top_k 40
PARAMETER repeat_penalty 1.1
PARAMETER num_ctx 4096
EOF

# Register and run
ollama serve &
ollama create my-llm -f Modelfile
ollama run my-llmbash

Generation Parameter Reference

Part 10 Troubleshooting & Iteration

Part 11 Complete Checklist

Environment

• Homebrew installed
• pyenv + Python 3.11 installed and set as global
• MLX virtual environment created and activated
• mlx, mlx-lm, datasets, huggingface_hub installed
• Hugging Face account created + CLI logged in
• Ollama installed and service running
• llama.cpp cloned and compiled with Metal support

Data Pipeline

• OpenHermes-2.5 downloaded and filtered to 100k rows
• Converted to JSONL chat format
• Split into train.jsonl (95%) and valid.jsonl (5%)
• Format verified with json.loads test

Training

• Llama 3.1 8B Instruct downloaded to ./models/
• Baseline benchmarks run and scores saved
• LoRA training run — 2,000+ steps minimum
• Loss curve decreasing — no divergence or plateau
• Checkpoints saved every 200 steps

Post-Training & Deployment

• Test prompts run on fine-tuned model
• Formal benchmarks re-run on merged model
• Benchmark scores compared to baseline — no catastrophic forgetting
• LoRA adapters fused into base model
• Model quantized to Q4_K_M GGUF
• Modelfile created with system prompt and generation parameters
• Model imported into Ollama and tested interactively

Appendix Quick Reference

# Fine-tune
mlx_lm.lora --model ./models/llama-3.1-8b-instruct --train --data ./data ...

# Test with adapter active
mlx_lm.generate --model ./models/llama-3.1-8b-instruct --adapter-path ./adapters

# Merge adapter into base
python -m mlx_lm.fuse --model [base] --adapter-path ./adapters --save-path [out]

# Convert to GGUF
python llama.cpp/convert_hf_to_gguf.py [merged_model] --outfile model.gguf

# Quantize
./llama.cpp/build/bin/llama-quantize model-f16.gguf model-q4.gguf Q4_K_M

# Register in Ollama
ollama create my-llm -f Modelfile

# Run
ollama run my-llmbash

Useful Links

MLX Examples ↗  ·  llama.cpp ↗  ·  Hugging Face ↗  ·  Ollama Docs ↗  ·  lm-evaluation-harness ↗