The Finetuner's Fallacy: When to Pretrain with Your Finetuning Data

We applied this approach using math proofs as the domain specific knowledge, and found that a 1B model trained with SPT closed 133% of the performance gap between standard 1B and 3B models. This means that the gains of domain expertise can be delivered without a corresponding increase in size, leading to the advancement of domain knowledge with lower inference costs, memory requirements, and greater deployment. Ultimately, this makes domain data better learned.

This reduces the pretraining tokens needed to reach a given domain loss by up to 1.75x.

Mix your domain data into pretraining

Let the domain dataset consist of a fraction, δ, of the pretraining tokens. After SPT, we finetune on the same dataset exactly as we would without SPT. δ is the only new hyperparameter.

One potential concern was overfitting to domain data, since even small values of δ still mean a lot of repetition (a 300M-token domain dataset and a 200B-token run, δ = 2% is 13 epochs over the dataset). But this does not end up being the case.

SPT beats finetuning alone in all three domains

To investigate finetuning across domains, we pretrained OLMo-1B on Dolma (a general web corpus) for 200B tokens. We then tested on three structured domains, MusicPile (symbolic music), ChemPile (chemistry), and ProofPile (mathematical proofs), each dataset being roughly 300M tokens. We also sweep δ over 0%, 0.1%, 1%, 2%, and 5%.

To ensure SPT's gains were not simply due to repeated exposure to the domain data, we optimized the standard fine-tuning baseline across training epochs, learning rates, and warmup schedules.

We define the relative gain over naive pretraining as the percentage reduction in post-fine tuning test loss between the SPT model vs the general model. MusicPile, the domain least like web text, improved the most (2.0%), followed by ProofPile (1.5%) and then ChemPile (0.8%). ChemPile’s text is closest to the Dolma distribution, which reflects why it might have had the smallest gain.

Since the loss differences are 1% to 2%, we also decided to explore advantages in compute, parameters, and accuracy.

• Compute. The SPT model matches naive pretraining's best 200B-token result using 1.75x fewer tokens for MusicPile, 1.56x fewer for ProofPile, and 1.40x fewer for ChemPile.

• Parameters. We compared our SPT model with a 3B model pretrained on the same 200B Dolma tokens, and fine-tuned. On ProofPile, the 1B SPT model outright beats the 3B model by 33%. the SPT model also achieved 81% of the performance of the 3B model on MusicPile and 23% on ChemPile.

• Accuracy. At 200B pretraining tokens, 2% SPT improves MATH by up to 6 points and MusicTheoryBench by up to 4 points over the finetuning-only baseline.

SPT models forget less during finetuning

SPT also preserves more general capability. Allocating a small fraction of pretraining tokens to domain data has almost no effect on general loss, since after 200B tokens, both naive and SPT runs reach similar loss on Dolma.

The main difference appears during finetuning. The naive model starts with higher domain loss and needs larger updates to reduce it, which increases its general loss. Alternatively, the SPT model starts closer to the target loss, needs smaller updates, and thus forgets less. As a result, larger SPT mixtures reach both lower domain loss and lower general loss.

We also checked whether replay, where general data is mixed back in during finetuning, gives the same benefit. However, this did not seem to be the case. At replay rates of 0%, 10%, and 20%, SPT outperforms naive pretraining. And SPT without replay also outperforms naive pretraining with 10% replay. Thus, replay during finetuning does not reproduce the effect of SPT.

SPT pays off when the domain is rare and you have enough data

How far from web-text your domain is. To study this in a controlled setting, we use English-to-Japanese translation: we pretrain 160M LLaMA models on 20B tokens of FineWeb2 and finetune on JParaCrawl parallel data. By varying the share of Japanese monolingual text in pretraining, we can control how much the pretraining distribution already overlaps with the target, without giving away the translation task itself.

As the Japanese monolingual share drops from 10% to 0.1%, relative gain climbs from about 2% to about 5%, and then plateaus. The less the pretraining corpus already covers the domain, the more SPT helps.

We also investigated whether we could predict the benefits of SPT before training using standard measures of similarity between general and domain-specific data. While these metrics successfully tracked performance gains in our Japanese-language experiments, they produced inconsistent results across the three specialized domains. Instead, the strongest predictor was the domain-specific loss of a general-purpose model after fine-tuning. This suggests that the difficulty of adapting to a domain may be more informative than its similarity to general pretraining data.

Your mixture depends on your training horizon.

The mixture amounts depends on pretraining tokens. Below 30B tokens, δ = 10% performs best. By 50B, it begins to degrade from repetition while δ = 5% continues to improve, and past 240B tokens δ = 2% is best. However, at every budget, some value of δ beats naive pretraining. Generally, the best value decreases as training gets longer. To address this, we fit scaling laws on a set of smaller runs that help predict how each mixture behaves over longer training.

Choosing between SPT and finetuning

The conventional approach to domain-specific model development treats a model’s pretraining data as fixed, introducing specialized knowledge only through fine-tuning. Our findings challenge this assumption, demonstrating that incorporating domain-specific data during pretraining can produce more capable models than fine-tuning alone. Importantly, this approach can achieve stronger domain performance with smaller models, reducing the computational costs of deployment and making models more economical at scale.