Trillions of tokens on someone else’s tab

Our look at the economics of open weights discussed a nearly billion-dollar estimate for reproducing the model training runs counted on Hugging Face. Data preparation was outside that calculation. Yet the same platform hosts another vast store of work already done: open training corpora. Someone has extracted the text, stripped out junk and duplicates, tested filtering choices by training experimental models, and released the results. The next team can start with a prepared corpus and skip those expenses.

A few Hugging Face projects make the scale clear. The English-language FineWeb originally contained 15 trillion tokens and occupied 44 TB; its current dataset card lists more than 18.5 trillion tokens. The multilingual FineWeb2 contains roughly 20 TB across 5 billion documents in more than a thousand languages. FinePDFs contributes around 3 trillion tokens extracted from 475 million PDF documents in 1,733 languages. Each offers processed data that another team can download and feed into its own training pipeline.

Other organizations have built comparable resources. The Allen Institute for AI’s Dolma contains about 3 trillion tokens drawn from the web, scientific publications, code, and other sources. DCLM provides a much larger starting pool—240 trillion tokens—for experiments in data selection. Turning such a pool into a filtered training corpus takes substantial work. Value accumulates at each stage: gathering source material, extracting usable text, selecting documents, and checking whether the resulting data actually trains better models.

That last step comes with a real compute bill. A secondary account of FineWeb’s development puts its ablation experiments at around 80,000 H100 GPU-hours across more than 70 experimental models. At an illustrative $4 per GPU-hour, that would be $320,000 for those experiments alone. The FineWeb report separately records another 6,000 H100 GPU-hours to apply an educational-quality classifier to 15 trillion tokens—$24,000 at the same assumed rate. CPU processing, storage, software development, and staff time would all come on top. Even these two components give some sense of what went into the preparation methods that subsequent users inherit along with the data.

“We applied the classifier to the 15T tokens of FineWeb, a process that required 6,000 H100 GPU hours.” — FineWeb report

Open corpora also let teams build new datasets from several groups’ work. Zyphra’s Zyda-2 contains roughly 5 trillion tokens assembled from DCLM, FineWeb-Edu, Zyda-1, and Dolma’s Common Crawl component. The team removed duplicates across the component corpora, then applied further filtering. According to its technical report, the cross-corpus deduplication stage removed about 11% of the input tokens. Any inventory of available data therefore has to account for overlap: the same Common Crawl pages can reach several repositories through different processing pipelines. Each successful processing method can still add value to that shared source material.

Zyda-2 also offers a rough cost benchmark for this next round of processing. In its account of using NVIDIA NeMo Curator, Zyphra estimates about two days on eight GPU nodes, compared with three weeks on eight CPU nodes. Using the eight-H100 node configuration described in the post gives 3,072 GPU-hours. At the same illustrative $4 per accelerator-hour, that works out to about $12,300 in GPU-equivalent cost; the full bill depends on node pricing and supporting infrastructure. This calculation prices an additional pass over corpora that other teams have already prepared. All the upstream work is inherited.

The cumulative economics become easier to see along that chain. One team pays to investigate filters. Another reuses its classifier. A third combines corpora and removes overlaps. A fourth prepares a dataset for a particular language or field. Processing code and experimental results travel with the files. A new lab can spend its budget on its own research question while adopting solutions others have already tested. That matters especially when a team can afford a handful of training runs, but would have to cut dozens of comparative experiments from its plans.

Replacement cost is a useful starting point for valuing this shared resource: which operations have already been performed, how much compute and labor they required, and which costs the next user can avoid. Rights to the underlying texts remain a separate accounting question; paying to process them does not buy those rights. The engineering work still has economic value. That billion dollars in open weights rests on a foundation of data with a price of its own. Every new experiment using a prepared corpus draws on that foundation without making another team pay to build it all over again.