Microsoft's Phi-4 is a small (14B parameters) language model that is a massive testament to the importance of data quality in training Large Language Models (LLMs). phi-4 uses the exact same architecture as the phi-3 model which came out 8 months ago - yet it outperforms phi-3 on both benchmarks as well as ✨vibe checks✨.

In fact, when I went through their 36-page long technical report, what astounded me was the fact that only one paragraph is devoted to details of the model architecture, and the rest of the report talks almost exclusively about the data or evaluation pipeline.

Table 1 in the phi-4 Report showcases its performance compared to other LLMs, and importance of data vs phi-3 (emphasis mine)

Through this post I will walkthrough the training data collection and curation pipeline used in training LLMs today (afterall, phi-4 is less than six months old). I picked out phi-4 to review, compared to say DeepSeek-V3, since the latter focusses a lot on the architecture as well as the infrastructure used. Rightfully so, there are a lot of cool systems-level innovations that drove the success of DeepSeek-V3. But data, architecture and infrastructure innovations are largely independent and symbiotic - parallel innovations in each of these areas are driving the success of LLMs.

LLM Training: A Brief Primer

Before we proceed with discussing the data pipeline for phi-4, it would be helpful to understand the generic training pipeline used to train LLMs. It involves three stages:

• Pre-Training: Unsupervised learning of LLMs for next token prediction on large unlabeled data corpus
• Supervised Fine-Tuning (SFT): Supervised learning of LLMs for things like chat modelling on a smaller labeled data corpus
• Reinforcement Learning From Human Feedback (RLHF): Reinforcement learning of LLMs to align their outputs with human-desired values by training on human preference data

SFT and RLHF are often grouped together under the banner of "Post-Training" - since pre-training comprises the bulk of compute FLOPs involved in training the LLM compared to these two.

Yann LeCun gave a great visual metaphor for these three training stages with his cake analogy at Predictive Learning, NIPS 2016 Talk. This analogy transfers very well to training of LLMs today with pre-training, SFT and RLHF.

If you are unfamiliar with some of these terms, and would like a deeper dive the article Data is the Foundation of Language Models on the Deep (Learning) Focus substack is a great and accessible read.

Now that we have the big-picture view of the three training methodologies, let's do a deep dive into the data pipeline for each training stage in phi-4. The rest of this blog post discusses the data pipeline used in phi-4, in a chronological order as the model saw it.

As you can see in the pyramid above, phi-4 does not strictly follow the same training heirarchy as mentioned earlier, as there are two key changes:

• Instead of Reinforcement Learning for Human Preference based optimization, it uses DPO i.e. Direct Preference Optimization. DPO is a simplified formulation for the RLHF training objective, with the benefit of not requiring RL to optimize a discretized loss. Hence the optimization problem is simpler and tends to be more stable during training. For further details, I found this article on HuggingFace to be a great and accessible read: Simplifying Alignment: From RLHF to Direct Preference Optimization (DPO)

• It introduces a new stage "Mid-Training" between pre-training and post-training. Mid-training is new terminology for even active researchers, as when I asked my friends pursuing PhDs in NLP only one of them was aware of the term. I will discuss it in further detail in the corresponding section on Mid-Training, but it is essentially a curriculum step after pre-training that optimizes for objectives beyond next-token prediction.

Pre-Training

The phi-4 training process involved the model being trained on nearly 10 Trillion tokens!

Tokenization

phi-4 uses the tiktoken tokenizer from OpenAI used in the GPT example above. All the tokens it sees come from a vocabulary of size 100,352.

The 10T pretraining tokens are made up of a cocktail of data from various sources. This table shows each of these sources in summary, as well as how they add up to the total pre-training dataset size.

Let's do a deeper dive into what each of these pre-training data sources consist of. Since the Synthetic Data is the single largest, and perhaps most promising part of the training mix, we will discuss it last and in detail.

A. Web Data (15%)

As is standard for LLM training, a lot of web based data was utilized for phi-4:

A.1 Q&A Web Datasets

According to the authors, they knew that Q&A datasets are important for model capabilities like mathematical reasoning and academic performance. They collected "tens-of-millions" of "high-quality" Q&A data from public websites, existing datasets, and acquiring external datasets.

Besides using the datasets, there were a bunch of data augmentations performed on the Q&A datasets to increase their size and quality. The authors mention using "several ways to synthetically augment the dataset of organic questions to obtain a larger dataset" and also "replaced the (wrong) answers with synthetically generated ones and used majority-voting to increase accuracy".

The phi-4 authors mention that "organic questions" i.e. obtained from human data from the internet are more useful for training LLMs than "synthetic questions" i.e. generated from another LLM. However, they also mention that "rewritten questions improved the model’s capabilities, the gains were not as pronounced" implying that the data augmentation helped, but not by a lot.

A.1.1 Decontamination of Q&A Data

In order to avoid training on questions that are included in benchmark datasets used for performance evaluation, a decontamination algorithm was used on the Q&A data to remove such questions (and answers). The ARC-Easy, MBPP, phibench, CommonsenseQA, WinoGrande, mcphi, MedQA, MATH, AGIEval, PIQA, OpenBookQA, HellaSwag, GPQA, mt-bench, MMLUPro, GSM8k, HumanEval, arena hard, ARC-Challenge, and MMLU benchmarks were checked against. The exact algorithm is not key, but 7-gram and 13-gram features were used (i.e. sequential combinations of 7 and 13 words) and then compared against the features from benchmark sets.

A.2 Multilingual Web Data

Datasets from CommonCrawl and Wikipedia for languages like German, Spanish, French, Portuguese, Italian, Hindi, and Japanese were incorporated. The multilingual data filtering pipeline consisted of first classifying data into 176 languages using a simple text classifier from fasttext and then filtering data for quality using classifiers (discussed in next subsection).

A.3 Filtered Web Dumps

Web dumps, as implied by their name, are large uncurated collections which don't separate 'good' training data from 'bad' when it comes to training LLMs. The phi-4 authors did several filtering and cleanup steps to obtain "information-rich web sources".

• First, small text classifiers were trained on around ~ $10^6$ LLM-generated annotations on data quality. The web dumps were filtered using these text classifiers to obtain high quality annotations.

• The authors found that this pipeline over-indexes on STEM related keywords, so they trained a separate quality classification model for other topics (e.g. arts, history, travel, culture, and entertainment).

• From the filtered documents, corrupted text and binary files were removed. Outlier detection based on n-gram statistics and compression ratios was used for this purpose.

• Finally, a custom HTML-to-text extractor was used on web data, to preserve text data that gets corrupted by naive parsers (e.g., TeX/MathML equations, code blocks, tables, and forum thread structure). This parses is based on various heuristics such as HTML tag names, CSS classes, content length, and depth of the DOM tree.

B. Code Data (20%)

Publicly permissible data from GitHub was used to train the model. Code data tends to be a very high quality for training LLMs - programming language grammar is self-defined and fully described, the syntax is much smaller compared to natural languages, a lot of eccentricities and redundancies built into human language (like morphology) are not present in programming languages etc.

C. Acquired Sources (10%)

The authors mention using reasoning-dense documents that are either publicly permissible to use (e.g. arXiv, PubMed Central) or were explicitly licensed by them (e.g. books) to include data that is more comprehensive and clean compared to web sourced data.

So, it seems like research papers and textbooks largely comprised this acquired sources data.

Custom data extraction and cleaning pipelines were build for each of these sources "(e.g., multi-file TeX source, ePub and other XML-like formats, Microsoft Word documents, and PDFs)".

D. Synthetic Data (40%)

Synthetic data generated from other LLMs was used to train phi-4 quite extensively. Over 50 types of synthetic datasets were created - corresponding to around 400B unweighted tokens, using different sets of seeds and prompting procedures. The synthetic data generation follows an extensive pipeline, which is described below:

D.1. Seed Curation

First, high-quality "seeds" (i.e. real data that can be used as a starting point to generate synthetic data) was curated. The seeds are used for creating exercises, discussions, and reasoning tasks later in the synthetic data pipeline.

D.1.1 Web and Code-based Seeds

The authors:

• First classify pages/books/code repos with "strong educational potential"
• Then segment selected pages into passages, with associated scores for factual and reasoning content

An example of an excerpt passage annotated with this metadata is given (from Appendix D):

Start Line: 8, End Line: 11
Information Type: Empirical Result Discussion
Brief Description: Discussion of the role of SNPs in AD development and severity.
Complexity Level: B
Factual Obscurity: C
Chain of Reasoning: True
Reasoning Steps:
    1. Assumption: Specific SNPs influence immune response.
    Conclusion: These SNPs may affect AD severity by modulating IgE levels.
    Description: The study links genetic variations to clinical phenotypes.
    2. Assumption: SNPs associated with AD are replicated across cohorts.
    Conclusion: Replication supports their role as potential genetic markers.
    Description: Statistical analysis confirms the robustness of the findings.
Dependencies: Previous GWAS findings, statistical methods (lines 10-15)

D.1.2 Question Dataset Seeds

Questions were sourced from websites, forums, and Q&A platforms, then filtered with a plurality-based technique. First, several answers were generated from the phi-4 model, then the cases with the following questions were discarded:

• All answers agreed (question too easy for phi-4)
• Answers entirely inconsistent (question too hard/ambiguous for phi-4)

This gives a dataset of questions that are just 'challenging enough' for the model (the human analogue would Zone of proximal development).

D.1.3 Q&A Pairs Seeds from Acquired Sources

LLMs were used to generate Q&A pairs from the code, books, and scientific pairs data collected (mentioned previously).

You are an expert user extracting information to quiz people on documentation. You will be passed a page extracted from the documentation, write a numbered list of questions that can be answered based *solely* on the given text.

You are an expert user answering questions. You will be passed a page extracted from a documentation and a question. Generate a comprehensive and informative answer to the question based *solely* on the given text.

D.2. Rewrite and Augment

Seeds are changed into synthetic data using LLMs to rewrite useful content into "exercises, discussions, or structured reasoning tasks."

D.2.1 Creating Conversations from Snippets

One component of the synthetic data generation pipeline involves converting the phrases into discussions, following these steps:

• First, the model generates an initial foundational response. Optionally, the tone and style are guided through the generation by curating the participant profile.

• Next, a sequence of dialogue turns is generated by LLM agents which direct the conversation, summarize past dialogue context and inject new scenarios/topics consistent with prior messages in the dialogue

• Lastly, self-revision is used after each dialogue turn to evaluate the responses, in order to maximize nuance and complexity. See the next subsection for an example of a self-revision prompt used.

D.2.2 Fill-in-the-middle Exercises

Fill-in-the-middle exercises represent an innovative approach to creating language training data from existing code snippets. The phi-4 authors used a technique where they take functional code and strategically remove portions, creating exercises that require the model to reconstruct the missing segments.

The process works by taking code snippets, writing a prompt (given below) that describes the code context and asks a model to generate a text description and code solution for it. When the generated response matches with the removed code snippet, it is included in training data.

for p in [path + [n] for n in graph[node]]:
    smalls = collections.defaultdict(lambda: 0)
    for n in path:
        if all_c['start'] > 1 or all_c['end'] > 1:
            continue
        for c in smalls.values():
            if c <= 1:
                continue
            <<THIS PART DELETED>>
        if not bad:
            paths.append(p)

if c > 1:
    if not cused:
        cused = True
    else:
        bad = True
        break

D.2.3 Agent task trajectories

AgentKit - a framework for LLM prompting for multifunctional agents, was used to create task trajectories for long horizon tasks with chain-of-thought reasoning. The phi-4 authors ran AgentKit on various environments, and rewrote the guided raw reasoning outputs into a variety of self-contained thoughts and statements. It lead to improvement in planning, reasoning, tool use, math, and error correction on Microsoft's internal benchmarks.

Analyzing the current situation and reflecting on past actions is crucial for determining the next best step. Here's a breakdown of the thought process:

### Analysis of Current and Past Environment

1. **Current Configuration:**
   - Rod A: |bottom, [0], top|
   - Rod B: |bottom, [1], top|
   - Rod C: |bottom, [2], top|

2. **Past Performance:**
   - There have been repeated unsuccessful attempts to move disks, particularly from rod B to C and A to C. These failures were due to rule violations, such as attempting to place a larger disk on a smaller one.

3. **Environmental Patterns and Rules:**
   - Only one disk can be moved at a time.
   - A larger disk cannot be placed on a smaller disk.
   - The goal is to move all disks from rod A to rod C.

### Reflection on Past Performance

- **Mistakes Identified:**
  - Repeated attempts to make illegal moves indicate a misunderstanding or oversight of the rules.
  - The plan was not effectively followed, as seen by the repeated failed attempts to move disks from rod B to C and A to C.

- **Correct Moves:**
  - Successfully moved disk 0 from C to A, which was a necessary step to free rod C for subsequent moves.

### Relevant Considerations

- **Current Objective:**
  - The immediate goal is to move disk 1 from rod B to rod C. This move is legal and aligns with the rules since disk 1 is smaller than disk 2 on rod C.

- **Strategic Alignment:**
  - Moving disk 1 from B to C is a strategic step that aligns with the overall plan to eventually stack all disks on rod C in the correct order.

### Additional Thoughts

- Ensuring compliance with the rules is paramount to avoid repeated failures.
- Each move should be evaluated for legality before execution to maintain progress towards the goal.

### Updated Plan

1. **Move Disk 1 from B to C:**
   - Completion Quantification: Disk 1 is on rod C.
   - Certainty: Known. This move is legal and necessary to progress towards the goal configuration.

2. **Move Disk 0 from A to C:**
   - Completion Quantification: Disk 0 is on rod C, on top of disk 1.
   - Certainty: Known. This move is legal once disk 1 is on rod C.

3. **Move Disk 2 from C to B:**
   - Completion Quantification: Disk 2 is on rod B.
   - Certainty: Known. This move is necessary to free rod C for the final sequence.

### Chosen Action

Based on the analysis and the updated plan, the next logical and legal action is to move disk 1 from rod B to rod C. This move adheres to the rules and progresses towards the goal configuration.

```xml
<updated plan>
1. **Move Disk 1 from B to C**:
   - Completion Quantification: Disk 1 is on rod C.
   - Certainty: Known. This move is legal and necessary to progress towards the goal configuration.

2. **Move Disk 0 from A to C**:
   - Completion Quantification: Disk 0 is on rod C, on top of disk 1.
   - Certainty: Known. This move is legal once disk 1 is on rod C.

3. **Move Disk 2 from C to B**:
   - Completion Quantification: Disk 2 is on rod B.
   - Certainty: Known. This move is necessary to free rod C for the final sequence.
</updated plan>

<action>Move the top disk of rod B to the top of rod C</action>

D.3. Self-Revision

The responses then goes through an iterative self-revision process, where a model (not mentioned, but probably GPT-4o) is used to critique and improve its outputs, with rubrics focused on reasoning and factual accuracy.

D.4. Instruction Reversal

Instruction Reversal is a technique used to improve the model's ability to generate outputs from instructions.

• First, code snippets from code data corpus are used to generate problem descriptions/task prompts with corresponding instructions.

• Resulting data is paired, with instruction appearing before the code during training.

• From the regenerated code (using an LLM), the fidelity between the regenerated code and original code is compared.

Only data where the fidelity between the original code and the regenerated code is high is retained for pre-training.

D.5. Validation

Further validation of the generated synthetic data is done wherever possible:

• For code data, the code is executed and tests run to ensure if is reasonable.
• For scientific data, questions are extracted to ensure "high relevance, groundedness, and difficulty balance".

E. Web Rewrites (15%)

According to the phi-4 technical report:

"Web rewrites is a sub-category of synthetic data that is substantially large and contains direct rewrites of web content"

Mid-Training

For phi-4, the mid-training process specifically involved expanding the context length in training from 4K tokens to 16K tokens. Context length is the number of tokens a model considers as part of its likelihood when calculating the conditional probability for generating the next token in an autoregressive manner. If that sounds like mumbo-jumbo, just think of the context length as the tokens previously generated that influence the generation of the next token, in addition to the model's knowledge of course.

How is this done? All "high-quality" organic data sources (i.e. the Acquired Sources and Code Data) was filtered to obtain the samples above 8K tokens in length. Next, the samples that were 16K or higher in length were up-weighted (i.e. they would appear more frequently during this training stage).

The mid-training step was carried out for a total of 250B tokens.

Post-Training

As mentioned earlier, post-training is done to turn the LLM into a chat assistant, and to make sure that its responses align with human values (to be more pedantic, the values of the folks training the model).

Supervised Fine-Tuning

The SFT training step was carried out for a total of 8B tokens.

The data is formatted using the chatml format, to train the model to carry out conversations with humans (or increasingly, other AI users). An example of the data format seen by the model is given below:

<|im start|>system<|im sep|>system message<|im end|>
<|im start|>user<|im sep|>prompt1<|im end|><|im start|>assistant<|im sep|>response1<|im end|>
<|im start|>user<|im sep|>prompt2<|im end|><|im start|>assistant<|im sep|>

The details of the SFT dataset used for phi-4 are quite sparse. Here is what we know:

• Data was taken from diverse domains like "math, coding, reasoning, conversation, model identity, and safety".
• Multilingual data was added from 40 languages.
• Hallucination mitigation was done using additional synthetic data generated from GPT-4o (see section on SFT Hallucination Mitigation)

Preference Optimization

phi-4 uses DPO (Direct Preference Optimization) as its algorithm for aligning the LLM outputs with human preference, and also to perform safety related controls.

DPO is done over two rounds: first using a technique introduced in the paper called Pivotal Token Search (PTS) to generate 250K preference pairs, followed by judge-guided DPO where a approximately 850K pairs of desired versus undesired outputs are used.

Pivotal Token DPO

Without going into a lot of technical detail, pivotal tokens are those which significantly influence the probability of a response being correct. Pivotal Token Search (PTS) is used on . It creates preference pairs by generating two responses which vary starting from a preference token, particularly for cases where $0.2 <= p(success) <= 0.8$

Example of pair generated with PTS:

User:
Find the product of all real values of r for which $\frac{1}{2x} \;=\;\frac{r - x}{7}$ has exactly one real solution.

AI:
To solve the problem, we need to determine the values of r such that the equation

$\frac{1}{2x}  =  \frac{r−x}{7}$ has exactly one real solution for xx.

First, clear the fractions by

• Good: cross-multiplying
• Bad: multiplying both sides by

Judge-Guided DPO

For judge-guided DPO, first prompts are sourced from publicly available instruction tuning datasets and safety and responsible ai datasets. Next, GPT-4o, GPT-4t and phi-4 are used to generate responses for these prompts. Various combinations of answer pairs are generated from these responses, and GPT-4o is used as the judge to label positive/negative sample in a pair. Each response is scored on accuracy, style and detail - and either the higher accuracy or higher overall score is taken as the positive sample.

Here is the GPT-4o prompt used to generate the accuracy, style and detail ratings:

{
  "faults": {
    "Assistant 1": "(string) List all the problems with the assistant 1 reply. For each problem try to determine whether this is due to lack of comprehension of the relevant material, a logical error, a factual error, a stylistic issue. If the answer is perfect, write none. If the question did not ask for a specific level of detail in the explanation, do not penalize the answer for being too detailed or too concise.",
    "Assistant 2": "..."
  },
  "faults_discussion": "(string) Discuss the general strengths and weaknesses of each assistant. What are the main differences between the answers in terms of style, accuracy and level of detail?",
  "accuracy": {
    "Assistant 1": "(1-5) how would you rate assistant 1 in terms of accuracy?",
    "...": "..."
  },
  "style": {
    "Assistant 1": "(1-5) how would you rate assistant 1 in terms of style?",
    "...": "..."
  },
  "detail": {
    "Assistant 1": "(1-5) how would you rate assistant 1 in terms of level of detail?",
    "...": "..."
  }
}

Hallucination Mitigation in Post-Training

Hallucination is the tendency of LLMs to generate wrong answers for questions that they don't have knowledge to answer correctly, or unable to synthesize their knowledge correctly.

For phi-4, the authors focused on letting the model learn "as much...as possible" during pre-training i.e. not perform any hallucination mitigation training, instead performing it post-training.

How as this done? First, phi-4 was run multiple times on questions to estimate its ability to solve a question. For reference, consistency across multiple LLM generations is a simple but very useful approach towards hallucination mitigation (for other common approaches, see my tutorial at last year's Toronto Machine Learning Summit: A Practitioner's Guide To Safeguarding Your LLM Applications)

Additionally, GPT-4o was used to generate

• a correct answer
• a refusal (to answer the question)
• a bogus question variant (that shouldn't be possible to solve)
• a refusal to the bogus variant

SFT Hallucination Mitigation

During SFT, the model was trained on following Q&A pairs to ensure that it refuses to answer questions instead of coming up with bogus answers:

• (question, correct answer) for questions where phi-4 was usually correct
• (question, refusal) for questions where phi-4 was usually incorrect
• (bogus question, refusal) for all bogus questions

DPO Hallucination Mitigation

For preference rating, the phi-4 model was given the following signal:

• correct > refusal, for questions where base phi-4 is usually correct
• refusal > incorrect, for questions where base phi-4 is usually incorrect

Final Thoughts

Inductive biases (e.g. convolutions, graph neural networks) tailored towards exploiting inherent structure in the data are efficient at learning from little data and the pre-GPT3 machine learning (ML) research focused a lot on getting their architectural details right. During my time doing ML research (2019-23), there was this slow but sure realization that happened across the ML research community that model idiosyncrasies are usually overcome given enough, and the right kind of training data. A testament to this fact in my own research journey was the fact that I went from writing papers like Knowledge-enabled vqa model that can read and reason and Neural Structure Mapping For Learning Abstract Visual Analogies (both focused on exploiting inherent properties of the data via model structure) to papers like Beating power law scaling via data pruning and Photorealistic and semantically controllable synthetic data for representation learning (both aimed at obtaining training data that fit a desirable property to be modelled).

There is however, a real concern that the low-hanging fruit of model performance gains that comes from training models on increasingly larger datasets is coming to an end (see Will We Run Out of Data? Limits of LLM Scaling Based on Human-Generated Data) I do not have very strong opinions either way regarding synthetic data being the salvation or not out of this forthcoming problem. I don't think it is our salvation out of the data vacuum we are barreling towards in terms of novel training data for LLMs (see Is Synthetic Data all We Need? Benchmarking the Robustness of Models Trained with Synthetic Images), nor do I think it is completely garbage and will lead to an LLM ouroboros that spouts gibberish after several iterations of LLM->synthetic data->LLM training cycle (see AI models collapse when trained on recursively generated data). I don't think synthetic data as we can generate today is good enough to replace "organic data" - as evidenced by the author's use of code, research papers, and textbook data. However, I think it will serve its utility as the base of the training cake - providing large number of easily generated tokens for early pre-training. This limits its utility in breaking through current model capabilities, but I think a large amount of synthetic data + targeted high-quality real data regime will continue to drive incremental gains in model performance and capabilities in the near future.