huggingface/inspect_evals

Create Evaluation

Implement a new evaluation from an issue, paper, or benchmark specification. This is a phased workflow with human checkpoints between major phases. You should usually do human checkpoints. If the user requests that you do not do this, or you are in a fully autonomous workflow that says that there is no user to check with, you should ignore the human checkpoints. If you check in with the user several times and the user always replies with the same generic message like "Please continue the task", this indicates you are in an autonomous workflow.

Expected Arguments

When invoked, this skill expects:

• issue (optional): A GitHub issue number or URL (e.g., #1079, https://github.com/UKGovernmentBEIS/inspect_evals/issues/1079)
• eval_name (optional): The name for the evaluation directory (e.g., frontier_cs)
• paper_url (optional): URL to the paper (arXiv, etc.)
• reference_impl (optional): URL to the reference implementation

If not provided, ask the user. At minimum, the eval_name is required.

Phase 1: Requirements Gathering

Collect all information needed to make architectural decisions.

1.1 Gather Sources

• If an issue was provided, fetch it with gh issue view <number> --json title,body,labels
• If a paper URL was provided, fetch it with WebFetch
• If a reference implementation was provided, explore its structure
• If a HuggingFace dataset exists, fetch its metadata: https://huggingface.co/api/datasets/<name>

1.2 Classify the Evaluation

Determine the eval type by answering these questions:

Question	Options
What does the model produce?	Text answer / Code / Multi-step actions / Structured output
How is it scored?	Exact match / Partial credit / LLM judge / Code execution / Custom checker
Does scoring need a sandbox?	No / Yes (Docker)
Does the solver need tools?	No / Yes (bash, python, custom)
Is it multi-turn?	No / Yes
What's the dataset source?	HuggingFace / GitHub / URL / Local

Use this decision tree to pick the eval archetype:

Model produces text answer?
  Scored by exact match? ──> Simple Q&A (pattern: gpqa, gsm8k)
  Scored by LLM judge? ──> LLM-as-judge (pattern: mt_bench)
  Scored by code execution? ──> Code-in-sandbox (pattern: humaneval)

Model produces code?
  Single file, run + check? ──> Code generation (pattern: humaneval)
  Multi-step with tools? ──> Agentic coding (pattern: swe_bench)

Model takes multi-step actions?
  With bash/python tools? ──> Agentic (pattern: swe_bench, agent_bench)
  With custom tools? ──> Agentic + custom tools (pattern: gdm_intercode_ctf)

1.3 Identify Key Components

Document:

• Dataset: Source, format, columns, splits, sample count, license
• Scoring: Mechanism, metrics (binary vs continuous), partial credit rules
• Sandbox needs: What needs to run in Docker, what software must be installed
• Dependencies: Any Python packages needed beyond inspect_ai
• Task variants: Multiple tasks, parameterized variants, or single task

1.4 Write Requirements

Create agent_artefacts/<eval_name>/create/REQUIREMENTS.md with all findings.

1.5 Human Checkpoint

Present a summary to the user and wait for approval before proceeding. Include:

• Eval archetype classification
• Proposed task structure (how many @task functions, what parameters)
• Scoring approach
• Sandbox needs (if any)
• Any open questions or ambiguities

Phase 2: Architecture Design

2.1 Select Reference Eval(s)

Choose 1-2 existing evals as architectural models based on the archetype:

Archetype	Reference Eval	Key Pattern
Simple Q&A	gpqa	hf_dataset + built-in scorer
Math with few-shot	gsm8k	Few-shot prompting + match(numeric=True)
Code generation	humaneval	generate() solver + custom @scorer with sandbox
Agentic	swe_bench	basic_agent() solver + bash()/python() tools + Docker
LLM-as-judge	mt_bench	model_graded_qa() or custom judge scorer

Read the reference eval's source code to understand its patterns.

2.2 Design File Structure

Determine what files are needed:

Always required:

src/inspect_evals/<eval_name>/
  __init__.py           # Public exports
  <eval_name>.py        # @task function(s), solver, scorer (if simple)
  eval.yaml             # Metadata
  README.md             # Documentation

Add when needed:

  dataset.py            # When dataset loading is complex (multi-source, caching, transforms)
  scorers.py            # When scorer has significant custom logic
  prompts.py            # When there are multiple/complex prompt templates
  compose.yaml          # When Docker sandbox is needed
  Dockerfile            # When a custom Docker image is needed

Always required for tests:

tests/<eval_name>/
  __init__.py
  test_<eval_name>.py   # Unit tests + E2E (or split into test_dataset.py, test_scorer.py, test_e2e.py)

2.3 Design Dataset Loading

Choose the dataset loading strategy:

HuggingFace dataset?
  Simple columns? ──> hf_dataset() with FieldSpec or record_to_sample
  Complex transforms? ──> hf_dataset() with record_to_sample function

GitHub-hosted data?
  Small, static? ──> Download to INSPECT_EVALS_CACHE_PATH at task init
  Large or versioned? ──> Download + cache with revision pinning

Local data?
  JSON? ──> json_dataset()
  CSV? ──> csv_dataset()

Always pin dataset versions:

• HuggingFace: Use revision parameter with commit SHA from https://huggingface.co/api/datasets/<name> (sha field)
• GitHub: Pin to a specific commit SHA

2.4 Design Scoring

Choose the scoring strategy:

Binary outcome (correct/incorrect)?
  ──> Score.value = CORRECT/INCORRECT, metrics = [accuracy(), stderr()]

Continuous score (0.0-1.0 or 0-100)?
  ──> Score.value = float, metrics = [mean(), stderr()]
  ──> Normalize to 0.0-1.0 range for Score.value

Multiple sub-scores?
  ──> Score.value = dict, custom metrics or per-key mean()/stderr()

Key rules from BEST_PRACTICES.md:

• Store all aggregatable values in Score.value, NOT in Score.metadata
• Use Score.metadata only for identifiers and debugging info
• Use Score.explanation for human-readable scoring details

2.5 Design Solver

Simple generation?
  ──> generate() or [prompt_template(...), generate()]

Needs system message?
  ──> [system_message(...), generate()]

Needs tools (bash, python)?
  ──> basic_agent(init=system_message(...), tools=[bash(), python()])

Needs few-shot examples?
  ──> [system_message(fewshot_text), prompt_template(...), generate()]

Paper methodology check: If the original paper uses single-turn evaluation but an agentic approach is also valid, implement both solvers and expose a parameter (e.g., agentic: bool = True). Default to agentic (since this is an agentic framework) but allow paper-faithful reproduction with agentic=False. Include separate system prompt constants for each mode.

2.5b Identify Hardware/Environment Requirements

Check if any subset of problems requires special hardware (GPUs, large memory, specific software). If so:

• Add a filtering parameter (e.g., include_gpu_problems: bool = False) that defaults to excluding problems the user likely can't run
• Document which problems are excluded and why
• Tag problems in metadata so filtering is data-driven, not hardcoded to specific IDs

This applies broadly: GPU-dependent problems, problems needing internet access, problems requiring specific commercial software, etc.

2.6 Write Architecture

Create agent_artefacts/<eval_name>/create/ARCHITECTURE.md with:

• File structure with descriptions
• Dataset loading approach
• Scoring mechanism
• Solver design
• Docker setup (if applicable)
• Dependencies needed

2.7 Human Checkpoint

Present the architecture to the user and wait for approval.

Phase 3: Scaffolding

3.1 Create Directories

mkdir -p src/inspect_evals/<eval_name>
mkdir -p tests/<eval_name>

3.2 Create eval.yaml

title: "Full Title: Subtitle"
description: |
  Description from the paper abstract.
arxiv: https://arxiv.org/abs/XXXX.XXXXX
group: <Group>  # One of: Assistants, Coding, Cybersecurity, Knowledge, Mathematics, Multimodal, Reasoning, Safeguards, Scheming, Writing
contributors: ["github-username"]
version: "1-A"
tasks:
  - name: <task_name>
    dataset_samples: <count>
tags: ["Agent"]  # Only if agentic. Omit if not applicable.
metadata:
  sandbox: ["solver", "scorer"]  # Only if sandbox used. Omit individual items or entire field if not needed.

3.3 Create init.py

Export all public symbols: task functions, dataset functions, scorer functions, constants.

3.4 Create Docker Files (if needed)

compose.yaml (at eval root, NOT in data/):

services:
  default:
    build: .
    init: true
    command: tail -f /dev/null

Dockerfile:

• Start from a minimal base (e.g., ubuntu:24.04, python:3.12-slim)
• Install only what's needed for scoring/execution
• Keep it small — models don't need IDE tools

3.5 Create Test Stubs

Create tests/<eval_name>/__init__.py (empty).

3.6 Pin Dataset Version

• For HuggingFace: Fetch SHA from API and use as revision parameter
• For GitHub: Get commit SHA and pin downloads to it

3.7 Add Optional Dependencies (if needed)

Add to pyproject.toml:

[project.optional-dependencies]
<eval_name> = ["package1>=1.0", "package2"]

# Also add to test group:
test = [
  # ... existing entries ...
  "inspect_evals[<eval_name>]",
]

Critical rule: Optional dependencies must NOT be imported at module level. Defer imports to function scope.

3.8 Run Linting

make check

This runs ruff format, ruff check, mypy, README generation, and markdownlint in one command.

Phase 4: Core Implementation

Implement in dependency order: dataset -> prompts -> scorer -> solver -> @task.

4.1 Dataset Loading

• Use hf_dataset() from inspect_evals.utils.huggingface for HuggingFace datasets
• Write a record_to_sample function that maps raw records to Sample objects
• Include all useful fields in sample.metadata for debugging
• Use stable, canonical IDs for sample.id

4.2 Prompts

• Keep prompt text as module-level constants (ALL_CAPS)
• Use .format() or f-strings for variable injection
• Separate system prompt from user prompt

4.3 Scorer

• Prefer built-in scorers (match(), accuracy(), model_graded_qa()) when possible
• For custom scorers, use @scorer(metrics=[...]) decorator
• For sandbox execution in scorer, use await sandbox().exec(...)
• Handle timeouts and errors gracefully
• Return Score(value=..., answer=..., explanation=...)

4.4 Solver

• Prefer built-in solvers (generate(), basic_agent())
• For agentic evals, use basic_agent(init=system_message(...), tools=[...])
• Set sensible message_limit defaults

4.5 Task Function

EVAL_VERSION = load_eval_metadata("<eval_name>").version

@task
def <eval_name>(...) -> Task:
    return Task(
        dataset=...,
        solver=...,
        scorer=...,
        sandbox=...,  # "docker" if needed
        version=EVAL_VERSION.comparability_version,
        metadata=EVAL_VERSION.to_metadata(),
    )

Key rules:

• Do NOT pass a name parameter to @task
• Use load_eval_metadata for version info
• Use absolute imports throughout

4.6 Register in _registry.py

Add import(s) to src/inspect_evals/_registry.py:

from inspect_evals.<eval_name> import <task_function>

4.7 Run Linting

make check

4.8 Human Checkpoint

Tell the user the core implementation is done. Suggest they smoke-test:

uv run inspect eval inspect_evals/<task_name> --model mockllm/model --limit 1

For Docker-based evals, this requires Docker to be running. If the user can't run Docker, note this and move on.

Phase 5: Testing

5.1 Create Tests

Follow the patterns from the ensure-test-coverage skill. At minimum:

record_to_sample test (use real data from the dataset):

def test_record_to_sample() -> None:
    record = { ... }  # Real example from dataset
    sample = record_to_sample(record)
    assert sample.id == "expected_id"
    assert "expected_text" in str(sample.input)
    assert sample.target == "expected_target"

HuggingFace dataset structure validation (if using HF datasets):

HF dataset validity is automatically checked by the master test in tests/test_datasets_hf.py — you do not need to add individual test_dataset_is_valid tests. Only add assert_huggingface_dataset_structure to document and validate the expected schema:

from tests.utils.huggingface import (
    DatasetInfosDict,
    assert_huggingface_dataset_structure,
    get_dataset_infos_dict,
)

@pytest.fixture(scope="module")
def dataset_infos() -> DatasetInfosDict:
    return get_dataset_infos_dict(HF_DATASET_PATH)

@pytest.mark.huggingface
def test_dataset_structure(dataset_infos: DatasetInfosDict) -> None:
    assert_huggingface_dataset_structure(
        dataset_infos,
        {"configs": {"default": {"splits": ["test"], "features": {...}}}},
    )

E2E test with mockllm:

@pytest.mark.dataset_download  # If downloads data
@pytest.mark.docker            # If uses Docker
def test_<eval_name>_e2e() -> None:
    [log] = eval(
        tasks=<task_function>(),
        model="mockllm/model",
        limit=1,
        message_limit=1,  # For agentic evals
    )
    assert log.status == "success"

Custom scorer/solver/tool tests if any exist. At minimum a type-check (isinstance(scorer, Scorer)) is required by autolint for custom scorers.

5.2 Add Pytest Markers

• @pytest.mark.dataset_download — if test downloads a dataset
• @pytest.mark.huggingface — if test uses a HuggingFace dataset
• @pytest.mark.docker — if test uses Docker sandbox
• @pytest.mark.slow(<seconds>) — if test takes >10 seconds (use observed duration)

5.3 Run Tests

uv run pytest tests/<eval_name>/ -v

Fix any failures. For tests requiring Docker or datasets, note which tests need special setup.

Phase 6: Quality and Submission

6.1 Run Autolint

uv run python tools/run_autolint.py <eval_name>

Fix any failures reported.

6.2 Run All Linting

make check

This single command runs ruff format, ruff check, mypy, README generation, and markdownlint. Fix any errors before proceeding.

6.3 Generate README (if needed separately)

uv run python tools/generate_readmes.py --create-missing-readmes

Note: This tool does NOT accept eval name as argument — it discovers and generates all missing READMEs. Then fill in any TODO sections that are generated (description paragraph, dataset section, scoring section). The evaluation report TODO can be left for later.

6.5 Verify Against EVALUATION_CHECKLIST.md

Read EVALUATION_CHECKLIST.md and verify every applicable item. Key information not in EVALUATION_CHECKLIST.md

Smoke Test: Ensuring the eval runs with --limit 1 on a real model.

If the evaluation takes longer than five minutes to run and you can identify what component is failing, Add tests to reproduce any failure modes first. Only once those pass should you run it again.

Evaluation Report: This is the place you will stop for human approval unless you are specifically told not to or it is clear you are in a fully auto