Knowledge Distillation for LLMs

The goal is not to build a smaller model, but to build a smaller model that remembers what the bigger one learned.

Distill, Memory-Retentive AI Agent

1. Classical Distillation Framework

1.1 The Teacher-Student Paradigm

Knowledge distillation (Hinton et al., 2015) trains a smaller "student" model to match the output probability distribution of a larger "teacher" model, rather than training the student solely on hard ground-truth labels. Recall from Chapter 06 that a language model produces logits (raw, unnormalized scores for each token in the vocabulary) which are then converted to probabilities via the softmax function. The key insight of distillation is that the teacher's probability distribution over all possible tokens contains far richer information than a single correct answer. When the teacher assigns 0.6 probability to "happy," 0.2 to "glad," and 0.1 to "joyful," these "soft" probabilities encode semantic relationships that hard labels cannot convey. Figure 16.1.1 shows this teacher-student training setup.

1.2 Temperature and Soft Targets

The temperature parameter T controls how "soft" the teacher's output distribution becomes. At T=1 (normal softmax), the teacher's distribution is peaked on the most likely token. As T increases, the distribution becomes smoother, revealing the relative probabilities of less likely tokens. This "dark knowledge" in the non-top predictions encodes the teacher's understanding of semantic similarity and uncertainty.

As Figure 16.1.5 illustrates, increasing the temperature gradually exposes the teacher's uncertainty across the full vocabulary, letting the student learn from the relationships between alternatives rather than just the top prediction.

The softmax with temperature is computed as:

where $z_{i}$ are the logits (pre-softmax values). Common temperature values range from 1.5 to 4.0. To see why temperature matters, consider logits [5.0, 2.0, 0.5] for three tokens. At T=1, softmax produces [0.92, 0.05, 0.01], a sharply peaked distribution that barely distinguishes the non-top tokens. At T=2, the distribution softens to [0.72, 0.18, 0.10], revealing that the second token is more plausible than the third. At T=4, we get [0.55, 0.26, 0.19], exposing even more of the teacher's uncertainty. Higher temperatures expose more of the teacher's knowledge but also introduce more noise. The distillation loss is scaled by T² to compensate for the gradient magnitude reduction caused by softening. Code Fragment 16.1.2 shows this approach in practice.

# Compute the KL-divergence distillation loss between teacher and student
# Soft targets from the teacher transfer more information than hard labels
import torch

import torch.nn as nn

import torch.nn.functional as F

class DistillationLoss(nn.Module):

 """Combined distillation and task loss for LLM training."""

 def __init__(self, temperature=2.0, alpha=0.5):

 super().__init__()

 self.temperature = temperature

 self.alpha = alpha # Weight for distillation vs hard label loss

 self.kl_loss = nn.KLDivLoss(reduction="batchmean")

 def forward(self, student_logits, teacher_logits, labels):

 # Soft targets: soften both distributions with temperature

 T = self.temperature

 student_soft = F.log_softmax(student_logits / T, dim=-1)

 teacher_soft = F.softmax(teacher_logits / T, dim=-1)

 # KL divergence loss (scaled by T^2)

 distill_loss = self.kl_loss(student_soft, teacher_soft) * (T ** 2)

 # Hard label cross-entropy loss

 hard_loss = F.cross_entropy(

 student_logits.view(-1, student_logits.size(-1)),

 labels.view(-1),

 ignore_index=-100,

 )

 # Combined loss

 return self.alpha * distill_loss + (1 - self.alpha) * hard_loss

{'train_loss': 1.3842, 'train_runtime': 284.7, 'train_samples_per_second': 7.02}
Distillation complete. Evaluate student against teacher.

Why does distillation work so well? The core intuition is that a teacher model's soft probability distribution contains far more information than hard labels. When a teacher assigns probabilities [0.6, 0.2, 0.1, 0.05, 0.05] to five candidate tokens, it is implicitly encoding semantic relationships: "happy" and "glad" are closely related (both get high probability), while "table" is semantically distant (near-zero probability). A student trained on these soft targets learns these relationships in every training step, effectively receiving a compressed version of the teacher's knowledge about language structure. Hard labels ("the answer is happy") convey none of this relational information. This is why distilled models can be surprisingly capable despite being 10x smaller: they inherit the teacher's understanding of token relationships, not just its final predictions.

2. White-Box vs. Black-Box Distillation

The distillation approach depends on what access you have to the teacher model. White-box distillation requires access to the teacher's internal logits; black-box distillation works only with the teacher's text outputs.

Figure 16.1.6 contrasts the two paradigms visually. The comparison table below summarizes their key differences in practice.

In practice, most LLM distillation today is black-box, simply because the most capable teachers (GPT-4, Claude) are only available through APIs. The remainder of this section examines both approaches in detail.

2.1 White-Box Distillation

When you have full access to the teacher's weights and logits, white-box distillation transfers the maximum amount of information. The following code fragment shows a complete training loop that loads both teacher and student, generates soft targets from the teacher, and trains the student on the combined loss.

# Compute the KL-divergence distillation loss between teacher and student
# Soft targets from the teacher transfer more information than hard labels
import torch

from transformers import AutoModelForCausalLM, AutoTokenizer

from torch.utils.data import DataLoader

def white_box_distillation(

 teacher_model_id: str,

 student_model_id: str,

 train_dataset,

 temperature: float = 2.0,

 alpha: float = 0.5,

 epochs: int = 3,

 lr: float = 2e-5,

):

 """Train student to match teacher logit distribution."""

 # Load teacher (frozen, in eval mode)

 teacher = AutoModelForCausalLM.from_pretrained(

 teacher_model_id, torch_dtype=torch.bfloat16, device_map="auto"

 )

 teacher.eval()

 for param in teacher.parameters():

 param.requires_grad = False

 # Load student (trainable)

 student = AutoModelForCausalLM.from_pretrained(

 student_model_id, torch_dtype=torch.bfloat16, device_map="auto"

 )

 loss_fn = DistillationLoss(temperature=temperature, alpha=alpha)

 optimizer = torch.optim.AdamW(student.parameters(), lr=lr)

 dataloader = DataLoader(train_dataset, batch_size=4, shuffle=True)

 for epoch in range(epochs):

 student.train()

 total_loss = 0

 for batch in dataloader:

 input_ids = batch["input_ids"].to(student.device)

 labels = batch["labels"].to(student.device)

 # Get teacher logits (no gradient)

 with torch.no_grad():

 teacher_out = teacher(input_ids=input_ids)

 teacher_logits = teacher_out.logits

 # Get student logits

 student_out = student(input_ids=input_ids)

 student_logits = student_out.logits

 # Compute combined loss

 loss = loss_fn(student_logits, teacher_logits, labels)

 loss.backward()

 optimizer.step()

 optimizer.zero_grad()

 total_loss += loss.item()

 print(f"Epoch {epoch+1}: avg_loss = {total_loss/len(dataloader):.4f}")

 return student

2.2 Black-Box Distillation

When the teacher is an API model (GPT-4, Claude, Gemini), you cannot access logits. Instead, you generate a large dataset of high-quality (input, output) pairs from the teacher, then fine-tune the student on these pairs using standard supervised training. The quality of the distilled student depends heavily on the diversity and quality of the generated training data. Code Fragment 16.1.4 shows this in practice.

# Generate teacher labels asynchronously for large-scale distillation
# Async batching maximizes throughput when labeling with an API-based teacher
import asyncio

from openai import AsyncOpenAI

import json

client = AsyncOpenAI()

async def generate_distillation_data(

 prompts: list[str],

 teacher_model: str = "gpt-4o",

 system_prompt: str = "You are a helpful assistant.",

 max_concurrent: int = 10,

) -> list[dict]:

 """Generate training data from API teacher for black-box distillation."""

 semaphore = asyncio.Semaphore(max_concurrent)

 results = []

 async def call_teacher(prompt):

 async with semaphore:

 response = await client.chat.completions.create(

 model=teacher_model,

 messages=[

 {"role": "system", "content": system_prompt},

 {"role": "user", "content": prompt},

 ],

 temperature=0.7,

 max_tokens=2048,

 )

 return {

 "instruction": prompt,

 "response": response.choices[0].message.content,

 "teacher": teacher_model,

 }

 tasks = [call_teacher(p) for p in prompts]

 results = await asyncio.gather(*tasks, return_exceptions=True)

 # Filter out errors

 valid = [r for r in results if isinstance(r, dict)]

 print(f"Generated {len(valid)}/{len(prompts)} training examples")

 return valid

# Generate dataset

# training_data = asyncio.run(generate_distillation_data(prompts))

# Then fine-tune student model on this data using standard SFT

3. Case Studies in LLM Distillation

3.1 Orca: Learning from Complex Explanations

Microsoft's Orca (2023) demonstrated that small models can dramatically improve by learning not just the teacher's answers but its reasoning process. Orca trained a 13B student on millions of examples from GPT-4 that included detailed chain-of-thought explanations, step-by-step reasoning, and self-correction. The key innovations were: using system prompts to elicit rich explanations from the teacher, curating diverse and challenging prompts, and training the student on the full reasoning trace rather than just the final answer.

3.2 Phi Series: Textbook-Quality Data

Microsoft's Phi models (Phi-1, Phi-1.5, Phi-2, Phi-3) showed that data quality matters more than model size. Rather than distilling on conversational data, the Phi team used GPT-4 to generate "textbook-quality" synthetic training data: carefully structured explanations, worked examples, and exercises across diverse topics. Phi-3 (3.8B parameters) achieves performance competitive with much larger models on reasoning benchmarks, demonstrating that a small model trained on exceptional data can outperform a larger model trained on mediocre data.

3.3 Distilled DeepSeek-R1

DeepSeek distilled their large R1 reasoning model (671B MoE) into a family of smaller dense models (1.5B, 7B, 8B, 14B, 32B, 70B). The distillation process used 800K samples of the R1 teacher's chain-of-thought reasoning traces. The distilled models retain strong mathematical and coding reasoning abilities, with the 32B distilled variant outperforming many larger models on math benchmarks. This demonstrates that reasoning capabilities, which were previously thought to require enormous scale, can be effectively compressed through distillation. Figure 16.1.3 summarizes the key principles shared by successful distillation projects.

4. Small-but-Capable Models

Distillation has enabled a new class of small models that achieve remarkable performance relative to their size. These models demonstrate that the right combination of architecture, training data, and distillation can produce efficient models for deployment on edge devices, mobile platforms, or high-throughput serving scenarios.

Beyond Distillation: Training Efficient Small Models Directly

Not all small, capable models are created through distillation. Several complementary approaches produce efficient models by rethinking how small models are trained from scratch or derived from larger ones.

• Sheared LLaMA (Xia et al., 2023) uses structured pruning to derive smaller models from larger pretrained ones. Rather than training a 1.3B model from scratch, the approach starts with LLaMA 2 7B and removes entire attention heads, layers, and hidden dimensions guided by importance scores. The pruned model is then briefly retrained on a small data budget, producing a 1.3B model that outperforms one trained from scratch with equivalent compute.
• MiniCPM (Hu et al., 2024) demonstrates that 2B-parameter models can achieve performance comparable to 7B models through careful training recipe optimization: WSD learning rate scheduling, high-quality data mixtures, and extended training well beyond the Chinchilla-optimal token count.
• The Phi family (Microsoft) makes the strongest case that data quality can substitute for model size. Phi-1 (1.3B) matched GPT-3.5 on code generation by training on "textbook-quality" synthetic data. Phi-2 (2.7B) and Phi-3 (3.8B) extended this approach, consistently outperforming models 5x their size on reasoning benchmarks.

The common thread is the Chinchilla lesson applied at small scale (see Section 06.3): an undertrained 7B model is often worse than a well-trained 2B model. For deployment on edge devices and cost-sensitive applications, these approaches complement the distillation and quantization techniques from Section 09.1.

5. Practical Distillation Pipeline

Here is a complete pipeline that combines data generation from an API teacher with student training, demonstrating the end-to-end black-box distillation workflow. Code Fragment 16.1.3 shows this approach in practice.

# Load and prepare the distillation dataset with teacher-generated labels
# Each example pairs an input with the teacher model's soft predictions
from datasets import Dataset

from transformers import AutoModelForCausalLM, AutoTokenizer

from trl import SFTTrainer, SFTConfig

from peft import LoraConfig

import json

# Step 1: Prepare distillation dataset

# (Assume we have generated data from teacher API)

distillation_data = [

 {

 "messages": [

 {"role": "user", "content": "Explain gradient descent."},

 {"role": "assistant", "content": "Gradient descent is an optimization..."},

 ]

 },

 # ... thousands more examples

]

dataset = Dataset.from_list(distillation_data)

# Step 2: Configure student with LoRA (parameter-efficient)

student_id = "meta-llama/Llama-3.2-3B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(student_id)

lora_config = LoraConfig(

 r=32, # Higher rank for distillation

 lora_alpha=64,

 target_modules="all-linear",

 task_type="CAUSAL_LM",

)

# Step 3: Train student on teacher-generated data

sft_config = SFTConfig(

 output_dir="./distilled-student",

 num_train_epochs=3,

 per_device_train_batch_size=4,

 gradient_accumulation_steps=4,

 learning_rate=2e-4,

 bf16=True,

 max_seq_length=4096,

 logging_steps=10,

 save_strategy="epoch",

)

trainer = SFTTrainer(

 model=student_id,

 args=sft_config,

 train_dataset=dataset,

 peft_config=lora_config,

)

trainer.train()

print("Distillation complete. Evaluate student against teacher.")

6. Distillation Licensing and Usage Policies

Before building a distillation pipeline, you must understand the legal constraints imposed by model providers. Most major LLM providers restrict or prohibit using their model outputs to train competing models. Violating these terms can result in service termination, legal liability, or both. The landscape varies significantly across providers.

6.1 Provider Policies (as of 2026)

The practical implication is clear: if you plan to distill knowledge from a proprietary API into your own model, verify that the provider's terms permit it. For many production use cases, starting with an open-weight teacher (Llama, Mistral, DeepSeek) avoids licensing uncertainty entirely. When using a proprietary API, the safest approach is to use the provider's own fine-tuning service rather than extracting outputs for external training.

7. Speculative Distillation

Speculative decoding uses a small "draft" model to propose multiple tokens at once, which a larger model then verifies in a single forward pass. Speculative distillation trains the draft model specifically to mimic the larger model's token distribution, improving the acceptance rate (how often the large model agrees with the draft) and thus the overall throughput. This technique turns distillation into a serving-time optimization rather than just a training-time technique. Figure 16.1.4 shows this propose-and-verify workflow.

8. Distillation for Reasoning: Chain-of-Thought Preservation

Standard distillation transfers a teacher's input-output mapping to a smaller student, but this approach discards the teacher's intermediate reasoning. When a 70B model solves a math problem, it internally constructs a chain of logical steps before arriving at the answer. A student trained only on final answers learns what to output but not how to reason, limiting its ability to generalize to novel problems. Chain-of-thought (CoT) distillation addresses this by training the student to reproduce the teacher's explicit reasoning traces alongside the final answers.

The procedure is straightforward. First, the teacher model generates solutions to a large set of training problems using chain-of-thought prompting, producing step-by-step reasoning followed by a final answer. Second, these complete reasoning traces become the training targets for the student. The student learns to generate the full chain of thought, not just the conclusion. This approach was central to the Distilled DeepSeek-R1 models, where 800K reasoning traces from the R1 teacher were used to fine-tune Qwen and Llama base models. The resulting students (1.5B to 70B parameters) achieved reasoning performance competitive with models several times their size.

CoT distillation introduces a design choice: should the student be trained on the teacher's complete reasoning trace, or should the traces be filtered and curated? In practice, filtering is essential. Teacher models sometimes produce redundant steps, circular reasoning, or incorrect intermediate conclusions that happen to reach the right final answer. A robust pipeline filters traces by (1) verifying the final answer against a ground truth, (2) checking that intermediate steps are logically consistent, and (3) removing excessively long or repetitive reasoning chains. The Orca series demonstrated that curating high-quality explanations (with explicit reasoning structure: "First, ..., Therefore, ..., Finally, ...") produces better students than training on raw, unfiltered teacher outputs.

A key finding from reasoning distillation research is that the student's reasoning format does not need to match the teacher's. A teacher that uses verbose, natural-language reasoning can be distilled into a student that produces compact, structured reasoning (numbered steps or symbolic notation) as long as the training data is reformatted accordingly. This flexibility allows practitioners to optimize the student's reasoning format for their deployment constraints: verbose reasoning for user-facing explanations, compact reasoning for latency-sensitive applications where the chain of thought is used internally but not shown to the user.

pip install transformers torch datasets tqdm

Teacher: 354,823,168 params
Student: 81,912,576 params
Compression: 4.3x

# Load teacher (gpt2-medium, frozen) and student (gpt2, trainable).
# The student will learn to mimic the teacher's output distribution.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

teacher = AutoModelForCausalLM.from_pretrained("gpt2-medium").to(device)
teacher.eval()
for p in teacher.parameters():
 p.requires_grad = False

student = AutoModelForCausalLM.from_pretrained("distilgpt2").to(device)
tokenizer = AutoTokenizer.from_pretrained("gpt2")
tokenizer.pad_token = tokenizer.eos_token

t_params = sum(p.numel() for p in teacher.parameters())
s_params = sum(p.numel() for p in student.parameters())
print(f"Teacher: {t_params:,} params")
print(f"Student: {s_params:,} params")
print(f"Compression: {t_params / s_params:.1f}x")

# Distillation loss: KL divergence on temperature-scaled soft targets
# blended with cross-entropy on hard (ground-truth) labels.
import torch.nn.functional as F

def distillation_loss(student_logits, teacher_logits, labels,
 temperature=3.0, alpha=0.5):
 """Combined distillation loss with soft and hard targets."""
 # TODO: Compute soft targets from teacher (temperature-scaled softmax)
 # TODO: Compute soft predictions from student (temperature-scaled log-softmax)
 # TODO: KL divergence between student and teacher distributions
 # TODO: Standard cross-entropy on hard labels
 # TODO: Combine: alpha * T^2 * KL + (1-alpha) * CE
 pass

# Quick test
b, s, v = 2, 10, 50257
loss = distillation_loss(torch.randn(b,s,v), torch.randn(b,s,v),
 torch.randint(0, v, (b,s)))
print(f"Test loss: {loss.item():.4f}")

Test loss: 11.2438

# Training loop: forward both teacher and student on each batch,
# compute the combined distillation loss, and update student only.
from datasets import load_dataset
from torch.utils.data import DataLoader
from tqdm import tqdm

dataset = load_dataset("wikitext", "wikitext-2-raw-v1", split="train")
dataset = dataset.filter(lambda x: len(x['text'].strip()) > 50)

def tokenize(examples):
 return tokenizer(examples['text'], truncation=True,
 max_length=128, padding='max_length')

tokenized = dataset.select(range(2000)).map(
 tokenize, batched=True, remove_columns=['text'])
tokenized.set_format('torch')
loader = DataLoader(tokenized, batch_size=8, shuffle=True)

optimizer = torch.optim.AdamW(student.parameters(), lr=5e-5)
student.train()

for epoch in range(2):
 total_loss = 0
 for batch in tqdm(loader, desc=f"Epoch {epoch+1}"):
 input_ids = batch['input_ids'].to(device)
 attn = batch['attention_mask'].to(device)

 student_out = student(input_ids, attention_mask=attn)
 with torch.no_grad():
 teacher_out = teacher(input_ids, attention_mask=attn)

 loss = distillation_loss(
 student_out.logits, teacher_out.logits, input_ids)
 loss.backward()
 optimizer.step()
 optimizer.zero_grad()
 total_loss += loss.item()

 print(f"Epoch {epoch+1} avg loss: {total_loss / len(loader):.4f}")

Epoch 1: 100%|██████████| 250/250 [01:42<00:00, 2.44it/s]
Epoch 1 avg loss: 4.7261
Epoch 2: 100%|██████████| 250/250 [01:38<00:00, 2.53it/s]
Epoch 2 avg loss: 3.8934

# Evaluate perplexity: compare original student, distilled student,
# and teacher to quantify how much knowledge transferred.
import math

def compute_perplexity(model, eval_loader):
 model.eval()
 total_loss, total_tokens = 0, 0
 with torch.no_grad():
 for batch in eval_loader:
 ids = batch['input_ids'].to(device)
 attn = batch['attention_mask'].to(device)
 out = model(ids, attention_mask=attn, labels=ids)
 total_loss += out.loss.item() * attn.sum().item()
 total_tokens += attn.sum().item()
 return math.exp(total_loss / total_tokens)

eval_data = load_dataset("wikitext", "wikitext-2-raw-v1", split="validation")
eval_data = eval_data.filter(lambda x: len(x['text'].strip()) > 50)
eval_tok = eval_data.select(range(500)).map(
 tokenize, batched=True, remove_columns=['text'])
eval_tok.set_format('torch')
eval_loader = DataLoader(eval_tok, batch_size=8)

original = AutoModelForCausalLM.from_pretrained("distilgpt2").to(device)
print(f"Teacher PPL: {compute_perplexity(teacher, eval_loader):.2f}")
print(f"Original student PPL: {compute_perplexity(original, eval_loader):.2f}")
print(f"Distilled student PPL: {compute_perplexity(student, eval_loader):.2f}")

Teacher PPL: 24.87
Original student PPL: 46.13
Distilled student PPL: 36.41

# Complete distillation lab: load teacher/student, train with KL
# divergence loss, evaluate perplexity before and after distillation.
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset
from torch.utils.data import DataLoader
import torch, torch.nn.functional as F, math
from tqdm import tqdm

device = "cuda" if torch.cuda.is_available() else "cpu"
teacher = AutoModelForCausalLM.from_pretrained("gpt2-medium").to(device)
teacher.eval()
for p in teacher.parameters(): p.requires_grad = False
student = AutoModelForCausalLM.from_pretrained("distilgpt2").to(device)
tokenizer = AutoTokenizer.from_pretrained("gpt2")
tokenizer.pad_token = tokenizer.eos_token

def distillation_loss(s_logits, t_logits, labels, T=3.0, alpha=0.5):
 soft_t = F.softmax(t_logits / T, dim=-1)
 soft_s = F.log_softmax(s_logits / T, dim=-1)
 kl = F.kl_div(soft_s, soft_t, reduction='batchmean')
 ce = F.cross_entropy(s_logits.view(-1, s_logits.size(-1)), labels.view(-1))
 return alpha * (T**2) * kl + (1 - alpha) * ce

ds = load_dataset("wikitext", "wikitext-2-raw-v1", split="train")
ds = ds.filter(lambda x: len(x['text'].strip()) > 50)
def tok(ex): return tokenizer(ex['text'], truncation=True, max_length=128, padding='max_length')
train_tok = ds.select(range(2000)).map(tok, batched=True, remove_columns=['text'])
train_tok.set_format('torch')
loader = DataLoader(train_tok, batch_size=8, shuffle=True)

opt = torch.optim.AdamW(student.parameters(), lr=5e-5)
student.train()
for epoch in range(2):
 total = 0
 for batch in tqdm(loader, desc=f"Epoch {epoch+1}"):
 ids, attn = batch['input_ids'].to(device), batch['attention_mask'].to(device)
 s_out = student(ids, attention_mask=attn)
 with torch.no_grad(): t_out = teacher(ids, attention_mask=attn)
 loss = distillation_loss(s_out.logits, t_out.logits, ids)
 loss.backward(); opt.step(); opt.zero_grad(); total += loss.item()
 print(f"Epoch {epoch+1}: {total/len(loader):.4f}")

def ppl(model, dl):
 model.eval(); tl, tt = 0, 0
 with torch.no_grad():
 for b in dl:
 i, a = b['input_ids'].to(device), b['attention_mask'].to(device)
 o = model(i, attention_mask=a, labels=i)
 tl += o.loss.item()*a.sum().item(); tt += a.sum().item()
 return math.exp(tl/tt)

ev = load_dataset("wikitext","wikitext-2-raw-v1",split="validation")
ev = ev.filter(lambda x: len(x['text'].strip())>50)
ev_t = ev.select(range(500)).map(tok, batched=True, remove_columns=['text'])
ev_t.set_format('torch')
el = DataLoader(ev_t, batch_size=8)
orig = AutoModelForCausalLM.from_pretrained("distilgpt2").to(device)
print(f"Teacher: {ppl(teacher,el):.2f}")
print(f"Original: {ppl(orig,el):.2f}")
print(f"Distilled: {ppl(student,el):.2f}")

Epoch 1: 100%|██████████| 250/250 [01:42<00:00, 2.44it/s]
Epoch 1: 4.7261
Epoch 2: 100%|██████████| 250/250 [01:38<00:00, 2.53it/s]
Epoch 2: 3.8934
Teacher: 24.87
Original: 46.13
Distilled: 36.41