No🧢 test

🚧 Idea, not built yet. This is the approach. Implementation is future work. The full write-up, diagram, and code live on GitHub.

Approach

My idea for reaching the goal faster is to improve the training data, but the validation set is drawn from the same distribution as the training data. So “cleaning the corpus” is a trap. That rules out the obvious move of aggressively filtering or rewriting. What’s left is the safe lever: don’t change what the model sees, change when it sees it. So the idea is an easy -> hard curriculum. Early in training, sample mostly easy, well-formed documents, over time relax the preference, by the end, sample the full FineWeb mix as-is.

ell = (ell_raw + 0.5) / 10.5          # learnability score → (0, 1]
for s in range(S):
    lam = s / (S - 1)                 # training progress: 0 → 1
    w = ell ** (ALPHA * (1 - lam))    # easy-first early, flat by the end
    p = w / w.sum()
    write_shard(sample(docs, p, TOK)) # resample this shard from p

Because the schedule ends on the full data, every hard and rare document still gets seen the training mix can’t drift away from the val set. That’s the whole reason it’s safe: it changes when the model meets a document, never whether.

Creating the curriculum: scoring 5B tokens cheaply

For any of this to work I need a learnability score per document and it has to be cheap enough to run over 5 billion tokens. The trick is to distill an expensive judgment into a cheap classifier in three hops:

• Teacher — a strong model (gpt-5-mini class) gives high-quality reference labels plus reflective feedback.
• GEPA — DSPy GEPA evolves the prompt of a small Qwen3.5-0.8B labeler against the teacher.
• Distill to fastText — the optimized 0.8B labeler annotates a representative sample, then I train a fastText classifier on (text -> label), cheap enough to sweep the whole corpus.

Two signals come out: learnability (0–10) drives the curriculum ordering, and a junk boolean catches genuine garbage — encoding noise, boilerplate, link farms which I down-weight rather than delete, again so the mix never permanently shifts off the val distribution.

The framing borrows from the FineWeb line of work: FineWeb-Edu scores documents by educational value with a small classifier, my learnability signal is the same trick, just aimed at coherence and difficulty rather than “educational” and Ultra-FineWeb is where the distill-to-cheap-fastText move comes from.

Full write-up and the resampling code are on GitHub.