FLAME Framework

Paper · Dataset

We introduce Fine-grained, LArge-scale Model Evaluation (FLAME), a framework for automated, comprehensive benchmark generation grounded in external knowledge sources such as textbooks and technical references

Repository Layout

src/
  cfg/task_generation/
    agent_config.yaml       # Model/provider and deduplication settings
    pipeline_config.yaml    # Corpus, output, resume, and loop settings
  schemas/                  # Lightweight task-generation schemas and I/O
  task_generation/
    runner.py               # Main CLI entrypoint
    agentic_pipeline.py     # Generation, repair, verification loop
    designer_agent.py       # Designer model wrapper
    verifier_agent.py       # Verifier model wrapper
    prompts.py              # Prompt templates
    blueprints/
      blueprints.json       # Difficulty and Bloom's-level combinations
    sample_chapter_text_files/
      *.txt                 # Example chapter corpus
  utils/
    model_client_utils.py   # Model client construction and retry helpers

Requirements

• Python 3.11–3.12
• Runtime dependencies from pyproject.toml
• API credentials for the providers configured in src/cfg/task_generation/agent_config.yaml

Common environment variables are:

GOOGLE_API_KEY
ANTHROPIC_API_KEY
OPENAI_API_KEY

OPENAI_API_KEY is only required when deduplication is enabled.

Installation

Python 3.10 is no longer supported so the AutoGen image dependency can use Pillow 13 or later with its security fixes.

Install Poetry 2.2.1, then install the exact dependency versions from poetry.lock:

poetry install --only main
cp .env.example .env

Enter your own provider credentials in the local .env file. Keep it private; do not commit it or include it in release archives. Run the commands below inside the Poetry environment, for example:

poetry run python -m src.task_generation.runner

License

This release is licensed under Creative Commons Attribution-NonCommercial- NoDerivatives 4.0 International (CC-BY-NC-ND-4.0). See LICENSE for the full standard terms. Third-party dependencies and user-supplied source materials retain their own terms; this licence does not grant rights to them.

Configuration

Pipeline behavior is controlled by:

src/cfg/task_generation/pipeline_config.yaml

Important fields:

• experiment_id: output experiment folder name
• output_base_dir: root directory for generated artifacts
• book_chapter_dir: chapter corpus directory under src/task_generation/
• blueprints_file: blueprint JSON file under src/task_generation/blueprints/
• max_retries: retry budget for failed candidates
• num_tasks_per_combo: fallback task count when a blueprint omits num_tasks
• checkpoint: incremental save and resume settings

Model behavior is controlled by:

src/cfg/task_generation/agent_config.yaml

This file configures the designer model, verifier model, and optional embedding-based deduplication.

Task difficulty and Bloom's-level coverage are controlled by:

src/task_generation/blueprints/blueprints.json

Each blueprint combination can define:

• difficulty
• blooms_level
• num_tasks

Run

From the repository root:

python3 -m src.task_generation.runner

The runner creates a fresh task tag automatically.

Resume Or Reuse A Task Tag

Pass an existing or desired task generation task tag:

python3 -m src.task_generation.runner --tasks-tag _YYYYMMDD_HHMMSS

When resuming, the runner skips generation units whose tasks.json output already exists and can also resume from per-combination checkpoints when checkpointing is enabled.

Parallel Run

Use the helper script:

bash src/task_generation/run_parallel_task_gen.sh ''

The first argument is an optional task tag. Leave it empty to create a fresh tag.

You can control the number of workers with WORKER_COUNT:

WORKER_COUNT=4 bash src/task_generation/run_parallel_task_gen.sh ''

Equivalent direct worker commands:

python3 -m src.task_generation.runner --tasks-tag _YYYYMMDD_HHMMSS --worker-index 0 --worker-count 4
python3 -m src.task_generation.runner --tasks-tag _YYYYMMDD_HHMMSS --worker-index 1 --worker-count 4
python3 -m src.task_generation.runner --tasks-tag _YYYYMMDD_HHMMSS --worker-index 2 --worker-count 4
python3 -m src.task_generation.runner --tasks-tag _YYYYMMDD_HHMMSS --worker-index 3 --worker-count 4

Outputs

Generated tasks are written to:

<output_base_dir>/<experiment_id>/tasks/<tasks_tag>/<area_id>/<capability_id>/tasks.json

Each generation-unit directory may also contain:

chapter_summary.json
verification_stats.json
token_stats.json
dedup_report.json
discarded_tasks.json
embedding_cache.json
checkpoints/

Example tasks.json:

{
  "metadata": {
    "experiment_id": "test_exp",
    "output_base_dir": "base_output",
    "timestamp": "2026-05-04T20:00:00Z",
    "input_stage_tag": "chapter_placeholders",
    "output_stage_tag": "_20260504_200000",
    "resume": false
  },
  "tasks": [
    {
      "task_id": "task_chapter_001_000001",
      "task_statement": "A binary classifier is evaluated on an imbalanced dataset. Which metric best captures the fraction of true positive examples that were predicted positive?",
      "task_type": "multiple_choice",
      "solution_type": "multiple_choice",
      "difficulty": "medium",
      "bloom_level": "Apply",
      "choices": [
        { "label": "A", "solution": "Accuracy" },
        { "label": "B", "solution": "Precision" },
        { "label": "C", "solution": "Recall" },
        { "label": "D", "solution": "F1 score" },
        { "label": "E", "solution": "None of the above" }
      ],
      "capability_id": "cap_ch_chapter_001_00000000",
      "capability_name": "placeholder_capability_chapter_001",
      "area_id": "area_ch_chapter_001_00000000",
      "area_name": "placeholder_area_chapter_001",
      "domain_id": "domain_000",
      "domain_name": "evaluation",
      "generation_metadata": {
        "correct_answer": "C",
        "source_chapter_id": "chapter_001",
        "capability_source_mode": "chapter_placeholder",
        "solution_graph": {
          "nodes": [
            {
              "id": "V1",
              "content": "Recall is defined as the fraction of true positive examples that are predicted positive."
            },
            {
              "id": "V2",
              "content": "The task asks for the metric matching that definition."
            }
          ],
          "edges": [
            {
              "from": "V1",
              "to": "V2",
              "operation": "Match the metric definition to the requested quantity."
            }
          ]
        },
        "solution_steps": [
          "Identify the quantity in the question: true positive examples predicted positive.",
          "Compare that quantity with the metric definitions.",
          "Select recall because it uses true positives divided by all actual positives."
        ],
        "complete_solution": "Recall measures the fraction of actual positive examples that were predicted positive, so the correct answer is C."
      }
    }
  ]
}

Example verification_stats.json:

{
  "chapter_id": "chapter_001",
  "chapter_relpath": "chapter_001.txt",
  "book_name": "root_corpus",
  "capability_id": "cap_ch_chapter_001_00000000",
  "area_id": "area_ch_chapter_001_00000000",
  "num_verifier_calls": 1,
  "verification_logs": [
    {
      "task_batch_id": "chapter_001_medium_Apply",
      "attempt_index": 0,
      "attempt_human": "1/4",
      "chapter_id": "chapter_001",
      "chapter_relpath": "chapter_001.txt",
      "blueprint_key": "medium_Apply",
      "difficulty": "medium",
      "blooms_level": "Apply",
      "question_index_in_batch": 0,
      "seed_generation_index": 1,
      "seed_generation_target": 1,
      "candidate_origin": "seed",
      "is_seed_task": true,
      "candidate_index_within_seed": 0,
      "candidate_total_within_seed": 1,
      "summary": {
        "overall_verdict": "Pass",
        "json_format_valid": "Yes",
        "mcq_integrity": "Yes",
        "blooms_alignment": "Yes",
        "constraint_compliance": "Yes"
      }
    }
  ]
}

Placeholder Metadata

The runner creates schema-compatible placeholder metadata directly from each chapter file. For every chapter, it builds:

• a Domain from the blueprint domain,
• an Area with a deterministic chapter-derived area_id,
• a Capability with a deterministic chapter-derived capability_id.

These placeholder objects satisfy the same schema chain used by generated task artifacts:

Domain -> Area -> Capability -> Task

Later on, the capability and areas can be included manually.

Notes

• Keep source text files under the configured book_chapter_dir.
• Keep blueprint counts small for smoke tests.
• Disable dedup.enabled in agent_config.yaml if you want to test generation without embedding calls.

Citation

If you use FLAME, please cite our paper, Fine-Grained Benchmark Generation for Comprehensive Evaluation of Foundation Models:

@misc{islam2026finegrainedbenchmarkgenerationcomprehensive,
  title={Fine-Grained Benchmark Generation for Comprehensive Evaluation of Foundation Models},
  author={Mohammed Saidul Islam and Negin Baghbanzadeh and Farnaz Kohankhaki and Afshin Cheraghi and Ali Kore and Shayaan Mehdi and Elham Dolatabadi and Arash Afkanpour},
  year={2026},
  eprint={2605.18824},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2605.18824},
}