Urdu Fine-tuning Pipeline — Usman Chaudhary

The same customer fine-tuning workflow taken end-to-end on two stacks — fast local iteration on Apple Silicon, then a containerized cloud training pipeline on GCP — with an honest before/after on what fine-tuning a small model actually buys, and what it costs.

A frontier model already speaks better Urdu than anything I could fine-tune — so the goal was never to win at Urdu. The goal was the infrastructure: can I stand up a customer's fine-tuning workflow end-to-end, on two different stacks, and then evaluate the result honestly instead of declaring victory on a single number? The deliverable is the pipeline and the judgment around it — not the model.

📦

Data

26K Urdu instruction pairs (chat format), 90/10 split, staged to GCS

💻

Local

MLX LoRA on Llama 3.2 3B, Apple Silicon — fast, private iteration

🐳

Containerize

Docker → Cloud Build → Artifact Registry — a reproducible image

☁️

Cloud train

Mistral 7B, 16 LoRA layers, GCP GPU VM — 1,000 steps

📊

Evaluate

Base vs fine-tuned, local vs cloud — honest before/after

Finding 1 — Validation loss is a proxy, not the truth

The lower-loss checkpoint generated worse text

On the local run, the iteration-1000 checkpoint produced visibly better Urdu than the iteration-800 checkpoint — even though 800 had the lower validation loss. Validation loss measures average per-token probability on held-out text; it is not the same thing as generation quality, and the two can diverge. The discipline that follows: read the actual outputs before picking a checkpoint. Never ship the number you didn't sanity-check.

Finding 2 — Fine-tuning moves the whole distribution

It gained conversational fluency and lost some crisp recall

The adapter pulled the model toward the shape of the training data — explanatory, conversational Urdu — and that helped cross-lingual instruction-following noticeably. But it slightly regressed terse factual recall: the base model answered cleanly where the fine-tune got vaguer. The base knowledge is still in the frozen weights; the adapter just biased the output behavior toward verbose explanation over direct retrieval. A known LoRA tradeoff — worth measuring and stating, not hiding.

Finding 3 — Quota is not capacity

The real cloud-ML lesson came from the infrastructure, not the model

A new GCP project starts with every GPU quota at zero. Even after the quota approvals landed, in-region capacity for managed L4 training jobs simply wasn't available across multiple regions — request after request failed on "resources insufficient." Quota is authorization; capacity is availability, and they are not the same lever. The run only completed after pivoting off managed training jobs to a Compute Engine GPU VM, which draws from a deeper, less-contended pool. Navigating that is the actual work of cloud ML — and a stronger signal than a first-try success would have been.

Finding 4 — Match the tool to the bottleneck

A commodity cloud GPU is not a speed-up over Apple Silicon

The cloud run on a T4 was not faster than local iteration on a Mac — a T4 is an older, modest card, and gradient checkpointing plus full-set evaluation passes slowed it further. Cloud beats local on speed only with dedicated high-end GPUs (A100/H100), which are exactly the contended, hard-to-get ones. The cloud pipeline's real value is reproducibility and scale headroom for models too big to fit locally — not raw throughput on whatever card you can actually schedule.

Same task, same 1,000-step budget, two deliberately different stacks. The cloud run adapts a larger model across four times as many layers; the local run optimizes for speed and privacy.

LOCAL — APPLE SILICON (MLX)

Llama 3.2 3B Instruct (4-bit) · LoRA on 4 layers, ~1.7M trainable · 1,000 iterations · best validation loss 1.392 · role: fast, private, on-device iteration.

CLOUD — GCP GPU

Mistral 7B Instruct v0.3 (4-bit NF4) · LoRA on 16 layers, ~3.4M trainable · 1,000 steps · evaluation loss 1.022 · role: reproducible, scalable production pipeline.

At a matched budget, the deeper adapter on the larger model reached a lower evaluation loss. But per Finding 1, loss is a proxy — the honest claim is "lower loss," not automatically "better Urdu," until the generations are read side by side.

Real training telemetry from the cloud run's trainer_state.json — 1,000 steps on Mistral 7B. Training loss is logged every 50 steps; evaluation loss is measured on the full held-out set every 250. This curve is the evidence behind the 1.02 — a clean descent, no divergence, no spikes.

Training loss 1.29 → 1.03 over 1,000 steps; eval loss 1.11 → 1.02 across four checkpoints. Eval loss falling in step with training loss — rather than rising — is the signal that the adapter generalized rather than overfit at this budget.

LOCAL STACK MLX-LM LoRA on Apple Silicon (MPS). Llama 3.2 3B Instruct, 4-bit. 4 LoRA layers, ~1.7M trainable parameters (0.05%). 1,000 iterations. Chosen for fast iteration loops and fully on-device, private experimentation.

CLOUD STACK HuggingFace + PEFT + bitsandbytes. Mistral 7B Instruct v0.3, 4-bit NF4 quantization, bfloat16 compute. 16 LoRA layers, ~3.4M trainable parameters (0.047%). 1,000 steps, gradient checkpointing to fit in 15 GB. Eval loss 1.022.

PIPELINE Docker → Cloud Build → Artifact Registry for a reproducible training image. Data and trained adapters in Google Cloud Storage. Training executed on a Compute Engine GPU VM after managed-job capacity proved unavailable in-region — the honest path, documented as part of the artifact.

DATA 26,019 Urdu instruction pairs in chat/messages format (user + assistant turns), 2,891 held out for evaluation. Formatted into a consistent instruction/response template shared across both stacks so the comparison stays clean.

EVALUATION Base vs fine-tuned across Urdu generation quality, cross-lingual instruction-following, and factual recall — plus local-vs-cloud at a matched step budget. Findings reported honestly, including the regressions.

P6 EdgePatch — air-gapped vulnerability remediation. An offline agent that finds, proves, patches, and verifies memory-safety vulnerabilities in C with zero network egress. Deterministic tooling carries the proof burden; a small local model proposes patches.

P7 Open benchmark + publication. Open-source the work and benchmark autonomous patch quality against real CVEs with ground-truth human patches — a cited artifact, not just a demo.