UrduGPT — Usman Chaudhary

A character-level transformer built from scratch on 290 million Urdu characters — with attention visualization, interpretability analysis, and a direct comparison against Gemma 4 12B. The first interpretability study on a Urdu language model.

📚

Corpus

Urdu Wikipedia dump — 264K articles, 290M chars, 327 unique characters

🔤

Tokenizer

Character-level — every Urdu glyph maps to a unique integer ID

🧠

Model

6 layers × 6 heads × 384 dims — full transformer from scratch

⚡

Train

5,000 steps on Apple Silicon MPS — AdamW, gradient clipping

🔍

Interpret

Attention weight extraction across all 36 heads — 4 Urdu phrases

Finding 1 — Layer 0 specialization

All of Layer 0 spontaneously became "previous token" heads

Without any instruction, 5 of 6 heads in Layer 0 learned to attend strongly to the immediately preceding character. This makes sense for Urdu — the language has highly predictable character sequences due to its morphological structure. The model discovered that tracking "what came before me" is the most useful signal for predicting the next character. One head (L0H5) specialized differently — behaving as a positional anchor across all phrases tested.

Finding 2 — Hierarchical learning

Layer 0 learns character sequences. Layers 1–5 learn word patterns.

A dramatic behavioral shift occurs between Layer 0 and Layer 1. By Layer 1, LOCAL attention (attending to a window of nearby tokens) dominates — the model is now building word-level context rather than character-level sequences. This mirrors findings from interpretability research on much larger models like GPT-2, reproduced here at 11M parameters on a low-resource language nobody has studied this way before.

Finding 3 — Dataset bias

UrduGPT learned that "Pakistan" means cricket

Regardless of the seed word, UrduGPT frequently generates cricket-related content. Urdu Wikipedia has a disproportionate amount of cricket articles, so the model over-indexed on this pattern. This is a real problem in low-resource language modeling: when one topic dominates the available corpus, the model's world model becomes distorted. A genuine finding about dataset composition in Urdu NLP.

36 heads across 6 layers measured on 4 Urdu phrases. Layer 0: dominated by PREV heads (purple) — the model learned to track the previous character. Layers 1–5: LOCAL attention (teal) takes over, building word-level context. Same hierarchical shift seen in GPT-2 and larger models, reproduced here at 11M parameters on Urdu.

پاکستان — PREV heads in L0, LOCAL in deeper layers

بہت اچھا ہے — same pattern, different phrase type

The same seeds run through both models. The gap illustrates exactly what scale, training data, and RLHF buy you — and what 41 minutes on a MacBook cannot.

SEED: علم (knowledge)

علم ایوارڈ اسکول میں بین الاقوامی بین الاقوامی بین الاقوامی بین الاقوامی...

SEED: علم (knowledge)

علم نور ہے، جہالت اندھیرا۔

SEED: محبت (love)

محبت کرنے والے حوالے کے صحن کھالے میں پیدا ہوئے اور انھیں میں سے نایاب درگاہ ہو گیا۔

SEED: محبت (love)

محبت ایک شفقت ہے، ایک چاہت ہے۔ یہ دلوں کو جوڑتی ہے۔

SEED: آج کا موسم (today's weather)

آج کا موسم ہو گئے۔ کیفیت کے بعد انھوں نے اسٹیڈیم کے خلاف رنز بنائے...

SEED: آج کا موسم (today's weather)

آج کا موسم بہت ٹھنڈا اور خوشگوار ہے۔

UrduGPT learned form. Gemma 4 learned meaning. The difference is 1,000× parameters, hundreds of billions of training tokens, and RLHF fine-tuning.

MODEL Decoder-only transformer following Karpathy's nanoGPT architecture. 6 layers, 6 attention heads per layer, 384 embedding dimensions, 256-token context window. 10,990,407 total parameters.

TOKENIZER Character-level — 327 unique Urdu/Arabic script characters. Every possible Urdu text is representable. The model learns morphology from scratch rather than relying on subword segmentation.

TRAINING AdamW optimizer, lr=3e-4, batch size 32, block size 256. Gradient clipping at 1.0. Dropout 0.1. 5,000 steps on Apple Silicon MPS GPU. Val loss 0.49 at convergence.

CORPUS Urdu Wikipedia — 264,959 articles, 290M characters. Custom XML parser strips markup, keeps only Urdu Unicode range (U+0600–U+06FF). 90/10 train/val split.

INTERPRETABILITY Attention weight extraction across all 36 heads (6 layers × 6 heads). Heads classified as PREV, LOCAL, SELF, or MIXED. Behavior visualized across 4 Urdu phrases spanning proper nouns, verb phrases, and predicates.

P5 Urdu Fine-tuned Assistant — LoRA fine-tune Llama 3.2 on Urdu corpus. What UrduGPT cannot do (answer questions with meaning), a fine-tuned 3B parameter model can. Runs locally, private.

P6 Mechanistic Interpretability — deeper circuit analysis on the P5 model. What attention heads specialize in at 3B scale vs 11M? Apply TransformerLens to find specific Urdu morphology circuits.

P7 Multilingual extension — extend the Urdu model to Farsi and Arabic. All three share Arabic script. The OSS contribution: a multilingual Abrahamic language model covering ~500M speakers in an underserved script family.