Linking Experiment Runs to Git Commits

Linking Experiment Runs to Git Commits

An experiment-tracker entry that records "val_loss = 0.32" is useless three months later unless you can recover the exact code that produced it. The reliable way to make every run reproducible is to inject the Git commit hash (and, where useful, a dedicated Git tag) into the tracker's metadata at run start. This section gives you three patterns that compose: stamp every run with git_sha, fail loud on a dirty working tree, and (for milestone runs) drop a Git tag whose name is the run ID.

Pattern 1: Capture the Commit Hash at Run Start

The cheapest, most portable recipe: shell out to git rev-parse HEAD and write the result into the tracker's config dict. It works the same in W&B, MLflow, CometML, or a plain JSON log.

import subprocess

def git_sha(short=False) -> str:
    """Return the current HEAD commit hash, or 'unknown' outside a repo."""
    fmt = ["--short"] if short else []
    try:
        return subprocess.check_output(
            ["git", "rev-parse", *fmt, "HEAD"], stderr=subprocess.DEVNULL,
        ).strip().decode()
    except (subprocess.CalledProcessError, FileNotFoundError):
        return "unknown"

# W&B: inject into config so it appears as a filterable column.
import wandb
wandb.init(project="llm-finetuning", config={"git_sha": git_sha()})

# MLflow: log_param works the same way.
import mlflow
with mlflow.start_run():
    mlflow.log_param("git_sha", git_sha())
    mlflow.log_param("git_sha_short", git_sha(short=True))

Stored this way, the commit hash becomes a sortable, filterable column in the tracker's UI. To rebuild any past result you run git checkout <sha> in a clean clone, reinstall the pinned dependencies (Section I.4), and rerun the same command with the same config file.

Pattern 2: Detect a Dirty Working Tree

A commit hash only reproduces the run if the working tree matched the commit at run start. If you trained while uncommitted edits were sitting in your editor, the SHA is a lie. Use git describe --dirty (or test git status --porcelain for output) to make the lie visible.

import subprocess

def git_describe_dirty() -> str:
    """Return e.g. 'v1.0.2-3-gabc123' or 'abc123-dirty' if uncommitted edits exist."""
    return subprocess.check_output(
        ["git", "describe", "--always", "--dirty", "--tags"],
    ).strip().decode()

def ensure_clean_or_warn(strict=False):
    dirty = subprocess.check_output(["git", "status", "--porcelain"]).strip()
    if dirty:
        msg = f"Working tree has uncommitted changes:\n{dirty.decode()}"
        if strict:
            raise RuntimeError(msg)
        import warnings; warnings.warn(msg)

# Production training: fail fast if anything is uncommitted.
ensure_clean_or_warn(strict=True)
wandb.init(config={"git_describe": git_describe_dirty()})

Pattern 3: Git Tag per Milestone Run

For baseline runs, paper-result runs, or production-promoted checkpoints, lift the linkage from "buried in a config dict" to a first-class Git object: tag the commit with the run ID. This makes the inverse lookup trivial (git log --tags shows every run that ever shipped), and DVC's dvc exp save command provides a similar tag-as-experiment-ID workflow.

import subprocess
import wandb

run = wandb.init(project="llm-finetuning", name="lora-r16-baseline")

# ... training loop ...

# On completion: tag the commit with the run ID for permanent linkage.
tag = f"run/{run.id}"
msg = f"W&B run {run.name} ({run.url})"
subprocess.run(["git", "tag", "-a", tag, "-m", msg], check=True)
subprocess.run(["git", "push", "origin", tag], check=True)

wandb.log({"git_tag": tag})

Putting the Three Patterns Together

A complete experiment-launch helper combines all three: it refuses to start a strict run on a dirty tree, stamps every run with the SHA, and adds a Git tag once training completes successfully. A single ~30-line module then becomes the only entrypoint your training scripts call into.