George AI

Media George AI in Action

Chapter 01 System Architecture & AI Routing

George uses a two-tier intelligence model: a local on-device LLM running on Apple Silicon GPU via mlx_lm for fast, private answers, and a cascade of cloud APIs for questions requiring real-time data.

Smart 3-Step Routing

Step 1 — Fast Pattern Matching — Before invoking any model, server.py runs the query through compiled regex patterns. Zero tokens, zero API calls.

# server.py — fast_route()
def fast_route(user_input, has_cloud):
    if PERSONAL_LOCAL_PATTERNS.search(user_input): return 'LOCAL'
    if not has_cloud: return 'LOCAL'
    if FORCE_CLOUD_PATTERNS.search(user_input): return 'CLOUD'
    if CLOUD_PATTERNS.search(user_input): return 'CLOUD'
    if LOCAL_PATTERNS.search(user_input): return 'LOCAL'
    return None  # ambiguous — go to Step 2python

Step 2 — LLM Router — If pattern matching returns None, George uses its own local LLM to classify the query. A 4-token inference call at temperature=0 returns only the word LOCAL or CLOUD.

Step 3 — Cloud Cascade — When a query needs the internet, George tries providers in order:

Streaming Response Format

{"sentence": "Here is the first sentence.", "source": "local"}
{"sentence": "And here is the second.",      "source": "cloud"}
{"done": true, "full": "Complete text",      "source": "cloud"}json

Chapter 02 Screen Vision

George can look at your screen on demand. Say "Hey George, what's on my screen?" and George takes a screenshot, encodes it, and sends it to a cloud vision model.

Trigger phrases: "what's on my screen" · "describe my screen" · "what do you see" · "look at my screen" · "read the screen" · "summarize this page"

// GeorgeController.swift — handleScreenVision()
let path = "/tmp/george_screen.png"
_ = await shell("screencapture -x " + path)
{ "type": "image_url", "image_url": { "url": "data:image/png;base64,..." } }swift

Chapter 03 Hardware Integration

George integrates with macOS hardware through official Apple APIs and standard Unix tools — no kernel extensions, no custom drivers.

Microphone

GPU — Local AI Inference

The local LLM runs on Apple Silicon GPU (M1/M2/M3/M4) using Apple's MLX framework via mlx_lm. Loaded into GPU memory at startup.

Hardware Stats

Chapter 04 Memory Engine

George's memory system is what makes it feel like a real companion rather than a stateless chatbot. Everything lives in ~/.george/memory.json — never sent to the cloud.

{ "persona": { "name": "George", "traits": ["curious","warm","witty","direct","honest"] },
  "history": [ /* last 80 conversation turns */ ],
  "facts":   [ /* up to 300 plain-English facts about you */ ] }json

Proactive Fact Mining

Every user message is scanned by a regex-based fact extractor before it reaches the AI. Categories detected:

Name · Spouse/Partner · Children · Extended family · Job · Location · Age · Pets · Hobbies (21 types) · Music genres (27 types)

Correction System

If you say "correction", "remember I...", "actually I...", or "I don't listen to X", George removes the conflicting fact and stores the corrected version.

Chapter 05 Orb Visual System

The floating orb is a real-time status display built in pure SwiftUI with 11 distinct visual styles.

The Five Orb States

The 11 Orb Styles

Chapter 06 Chess Engine

George includes a complete, self-contained chess implementation in Swift with no external libraries — full rules, minimax search, alpha-beta pruning, and natural-language move commentary.

Board Representation

Pieces encoded as raw integers 0–12, stored as a flat 64-element array. ChessBoard is a Swift struct (value type) — every speculative move creates an independent copy.

struct ChessBoard {
    var squares: [ChessPiece] = Array(repeating: .empty, count: 64)
    var whiteKingsideCastle  = true
    var whiteQueensideCastle = true
    var blackKingsideCastle  = true
    var blackQueensideCastle = true
    var enPassantSquare: Int = -1
    var whiteToMove = true
}swift

Material Values

Minimax with Alpha-Beta Pruning

Searches 4 moves deep. Without pruning: ~810,000 positions (30⁴). Alpha-beta reduces the effective branching factor from ~30 to ~7 — responses under one second on Apple Silicon.

private static func alphabeta(_ board: ChessBoard, depth: Int,
                                alpha: Int, beta: Int, maximising: Bool) -> Int {
    if depth == 0 { return board.evaluate() }
    if b <= a { break }  // beta cutoff — prune remaining branches
}swift

Chapter 07 Reinforcement Learning Game Player

George can play Arkanoid autonomously using a Q-learning agent. It watches the game screen via screenshot, infers state, chooses actions, and sends real keyboard inputs to control the paddle.

Q-Learning Algorithm

# Bellman equation — GamePlayer.swift updateQ()
bestNext = max(qTable[nextState][action] for action in actions)
updated  = current + α × (reward + γ × bestNext − current)
qTable[state][action] = updatedpython

Reward Function

Chapter 08 App Entry Point

main.swift is 46 lines. AppDelegate.swift is 42 lines. Small files, enormous consequences.

Activation Policy

app.setActivationPolicy(.accessory)
// No Dock icon · Not in Cmd+Tab · No menu bar
// George is ambient — always present, never intrusiveswift

Boot Sequence

1. OS loads George binary → main.swift executes
2. installCrashHandlers() registers SIGBUS, SIGSEGV, SIGABRT, SIGILL, SIGFPE
3. NSApplication.shared creates the app singleton
4. AppDelegate() instantiated and wired to app
5. setActivationPolicy(.accessory) — hidden from Dock
6. app.run() — main run loop starts, never returns
7. applicationDidFinishLaunching fires
8. NSPanel created and configured (.borderless, .nonactivatingPanel)
9. Panel positioned bottom-right using NSScreen.main.visibleFrame
10. GeorgeController() created — Memory + Agent loaded from disk
11. OrbView SwiftUI view tree created
12. NSHostingView bridges SwiftUI into AppKit
13. panel.makeKeyAndOrderFront — orb appears on screen
14. Task { await george.boot() } scheduled async
15. Boot: requests mic + speech recognition permissions
16. Boot: launches server.py as child process
17. Boot: pings localhost:8765 until server responds
18. Boot: server.py loads local LLM into Apple Silicon GPU
19. Boot: state → .idle — breathing animation begins
20. George speaks a random greeting — fully operational

Chapter 09 Setup & Mobile UI

setup.sh — One-Command Installation

./setup.sh
~/Desktop/George.command   # double-click launcherbash

8 steps: Verify prerequisites → Create ~/.george/ → Copy server.py → Enter API keys → Install mlx-lm → Install yt-dlp → swift build -c release → Download Llama 3.2 3B (~1.8 GB)

Mobile Web UI

A self-contained single-file web app served by server.py at GET /. Open it on any phone, tablet, or computer — no install required. Dark space theme, CSS-only animated orb, streaming text, voice input via Web Speech API.