huggingface/inspect_evals

Prepare Eval For Submission

Workflow Steps

To prepare an evaluation for submission as a pull request:

1. Add custom dependencies in the pyproject.toml file, if required. Don't forget to add your evaluation to the [test] dependencies if you've added any custom dependencies.

   [project.optional-dependencies]
   worldsense = ["pandas", "matplotlib"]  # existing groups
   your_evaluation = ["example-python-package"]  # add this line for your evaluation
   
   test = [
      "inspect_evals[dev]",  # existing groups
      "inspect_evals[your_evaluation]",  # add this line for your evaluation
   ]
   

   Pin the package version if your evaluation might be sensitive to version updates.

   Make sure your module does not import your custom dependencies when the module is imported. This breaks the way inspect_ai imports tasks from the registry.

   To check for this, pip uninstall your optional dependences and run another eval with --limit=0. If it fails due to one of your imports, you need to change how you are importing that dependency

2. Add pytest tests covering the custom functions included in your evaluation. See the Testing Standards section for more details.

3. Run pytest to ensure all new and existing tests pass.

   To install test dependencies:

   uv sync --extra test
   

   Then run the tests

   uv run pytest
   

4. Add an eval.yaml file to your evaluation directory (e.g., src/inspect_evals/your_eval/eval.yaml) to include it in the documentation (which is automatically generated and deployed by Github Actions after merge). See the eval.yaml Reference section in CONTRIBUTING.md for a complete example and field documentation.

   Don't forget to add any custom dependencies to pyproject.toml if applicable.

5. Run linting, formatting, and type checking and address any issues:

   make check EVAL=<eval_name>
   

   Note that we automatically run these commands on all incoming PRs, so be sure your contribution passes those before opening a PR. run_autolint.py handles automated checks

6. Fill in the missing sections of the generated README.md which are marked with 'TODO:'.

7. Create a changelog fragment to document your changes by running uv run scriv create.

<!-- Reference-style links -->