Our AI model and intelligence initiative.
Omnis is AfriStack's long-term AI development programme, covering model training, adaptation, AI agents, inference, orchestration, evaluation and intelligent infrastructure.
We have already trained and tested our own models. This page is the honest record of what they achieved, what they cannot do, and where they go next.
A language model trained from scratch.
A compact, decoder-only language model built to validate our end-to-end model development pipeline. No pretrained weights. No external inference API. The tokenizer, the architecture, the training loop and the data pipeline are ours.
Pretraining
15,000 steps · 983M tokens · ~12 hours on one 6 GB consumer laptop GPU
| Metric | Step 500 | Final |
|---|---|---|
| Held-out perplexity | 237.5 | 41.2 |
| Validation loss | 5.4700 | 3.7179 |
| Next-token top-1 | 19.9% | 34.5% |
| Next-token top-5 | 36.9% | 55.9% |
| Factual continuation, top-10 | 0% | 75% |
| Median fact rank | 209 | 7 |
Instruction tuning
504 steps · 63,000 conversations · 8.4M tokens · ~15 minutes
After instruction tuning, the model answers rather than continuing text, and holds its own identity — final validation loss 1.6479, down from a 3.7179 base.
Identity holding is the point of a proprietary identity set. Without one, a model trained on web text claims to be somebody else's assistant.
What it cannot do — stated plainly
A model at this scale is not a product, and we do not present it as one. It is a learning vehicle that proves the pipeline, the tokenizer and the training stack work end to end. Anyone who tells you a model this size can power a consumer assistant is selling something.
Repetition loops
Some prompts fall into repetition. Mitigated at inference with a repetition penalty, and expected to improve with more data rather than more steps.
Train/validation gap
A +0.68 gap indicates we reached the useful ceiling for this parameter count on this corpus. More permissive data is already queued for the next dataset revision.
Thin world knowledge
The corpus is educational and mathematical, so literature and science land better than geography. Answer format is correct; answer content is often wrong at this size.
Architecture
A decoder-only transformer using modern-standard components, chosen for stability and efficiency rather than novelty.
- RMSNorm pre-normalisation — cheaper and more stable than LayerNorm
- RoPE rotary position embeddings — extrapolates better than learned positions
- SwiGLU feed-forward — stronger than a ReLU or GELU MLP at equal parameter count
- Tied embeddings — input embedding doubles as output projection
- Flash attention and no biases in linear layers
Presets
| Preset | Layers | Heads | d_model | Context | Params |
|---|---|---|---|---|---|
| Omnis Tiny 0.1 | 8 | 8 | 512 | 512 | ~42M |
| Omnis Tiny 0.2 | 12 | 12 | 768 | 1024 | ~110M |
| Omnis Small 1.0 | 24 | 16 | 1024 | 2048 | ~340M |
Every artefact is versioned and reproducible: the model, the tokenizer, the dataset and the benchmark each carry their own identifier, and nothing enters training without an entry in the source registry.
An adapted open-weight model for real product workloads.
Training a capable general model from scratch is a nine-figure problem. Adapting a strong, permissively-licensed open-weight model to our own tasks is not — and it ships today.
Omnis Base is our adaptation track. We take an open-weight base model with a commercially usable licence, then train low-rank adapters against benchmarks derived from our own products rather than generic leaderboards.
Five successive adapter generations have been trained and evaluated so far, each one a measured step rather than a guess. The adapters target every attention and feed-forward projection in the model, which is what lets a small number of trained parameters move behaviour meaningfully.
Self-hosting rather than renting an API matters for four reasons: we control the cost, the latency, the availability, and where customer data goes. The gateway interface stays identical when our own models take over.
Adaptation setup
| Method | LoRA (low-rank adaptation) |
|---|---|
| Rank | 16 |
| Alpha | 32 |
| Dropout | 0.05 |
| Targets | all 7 projections |
| Adapter generations | 5 |
| Base model | open-weight, permissive licence |
Deployed behind the Omnis Gateway, where routing, fallback and per-provider measurement are handled centrally.
Each stage exists to make the next one cheaper to get right.
We do not skip stages. Every failure at the smallest size costs hours; the same failure at the largest costs six figures.
Omnis Tiny 0.1
Good for: nothing user-facing. It proves the stack — data pipeline, tokenizer, architecture, training loop, evaluation harness — works end to end.
Omnis Tiny 1.0
Good for: language identification, text classification, and quality filtering for our own data pipeline. Useful internally, not externally.
Omnis Small 1.0
The first genuinely shippable model. Yorùbá, Igbo and Hausa translation, autocomplete, extraction, classification and narrow assistants. This is the real target, and it is within reach of a funded round rather than a nine-figure one.
Omnis Base 1.0 and beyond
Good for: a real assistant — instruction following, retrieval-augmented generation and tool use. Larger still only if the business demands it and revenue funds it.
Where a small model beats a large one
A one-billion-parameter model that is the best in the world at Yorùbá is a far better business than a thirty-billion-parameter model that is fortieth-best at English. Frontier labs are weak on African languages, local curricula and local business context — not because they cannot be strong, but because the data is not on the open web and the market is too small for them to prioritise. That weakness is structural, it is measurable, and it is defensible.