Repository navigation
feat(gooddata-eval): evaluate Analytics Catalog title/description #1961
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -132,8 +132,8 @@ gd-eval run \ | |
| | `--runs K` | `2` | Independent runs per item. | | ||
| | `--gate` | `any` | Which verdict decides an item: `any` = pass@K (a run passing is enough), `power` = pass^K (every run must pass, so the verdict measures stability). Identical at `--runs 1`. Kinds that repeat K runs only — `power` is refused when the dataset also has non-agentic items or `agentic_conversation`, which are always decided on pass@K. | | ||
| | `--concurrency K` | `1` | Number of items evaluated concurrently. `1` = sequential (default). Increase to load-test the agent under simultaneous requests — see *Concurrency and workspace safety* below. | | ||
| | `--judge-model MODEL` | `gpt-4o` | Model used for LLM-as-judge scoring — `agentic_general_question`, `agentic_guardrail`, `general_question`, `guardrail`, `dashboard_summary` and `document_takeaway`. Also settable via `GD_EVAL_JUDGE_MODEL`. Two things to weigh before changing it: the gpt-5 family rejects `temperature=0`, so verdicts stop being reproducible (the run warns when this happens); and choosing the same model the agent runs means the judge grades its own family's output. | | ||
| | `--reasoning-effort LEVEL` | server default | `LOW`, `MEDIUM` or `HIGH`, sent as `options.reasoningEffort` on every chat message. Requires the `enableGenAiReasoningEffort` feature flag on the target organization — without it the server ignores the value. Applies to chat items only; `dashboard_summary` and `document_takeaway` items go through their own endpoints, which have no such option. | | ||
| | `--judge-model MODEL` | `gpt-4o` | Model used for LLM-as-judge scoring — `agentic_general_question`, `agentic_guardrail`, `general_question`, `guardrail`, `dashboard_summary`, `document_takeaway` and `catalog_metadata`. Also settable via `GD_EVAL_JUDGE_MODEL`. Two things to weigh before changing it: the gpt-5 family rejects `temperature=0`, so verdicts stop being reproducible (the run warns when this happens); and choosing the same model the agent runs means the judge grades its own family's output. | | ||
| | `--reasoning-effort LEVEL` | server default | `LOW`, `MEDIUM` or `HIGH`, sent as `options.reasoningEffort` on every chat message. Requires the `enableGenAiReasoningEffort` feature flag on the target organization — without it the server ignores the value. Applies to chat items only; `dashboard_summary`, `document_takeaway` and `catalog_metadata` items go through their own endpoints, which have no such option. | | ||
|
|
||
| **Concurrency and workspace safety.** Agentic kinds that create workspace objects | ||
| (`agentic_metric_skill`, `agentic_alert_skill`, `agentic_conversation`, `agentic_kda_skill`) always run one at a | ||
|
|
@@ -708,7 +708,8 @@ A dataset is a folder of `.json` files, one per question: | |
| ``` | ||
|
|
||
| Supported `test_kind` values: `visualization`, `metric_skill`, `alert_skill`, | ||
| `search_tool`, `general_question`, `guardrail`, `dashboard_summary`, `document_takeaway`. | ||
| `search_tool`, `general_question`, `guardrail`, `dashboard_summary`, `document_takeaway`, | ||
| `catalog_metadata`. | ||
|
|
||
| ### `visualization` items | ||
|
|
||
|
|
@@ -866,6 +867,51 @@ and raises only when no run completed. A read timeout is not retried, and the CI | |
| point records it as an errored run too. Any other 4xx and an unsupported URL scheme are | ||
| never retried and fail the item in both. | ||
|
|
||
| ### `catalog_metadata` items | ||
|
|
||
| Catalog items call the Analytics Catalog generate endpoints | ||
| (`POST /api/v1/actions/workspaces/{ws}/ai/analyticsCatalog/generateTitle` or | ||
| `/generateDescription`) for one object, named by a `catalog_input` block. In a Langfuse | ||
| dataset the block goes in the item's `metadata` or its `input` object (`input` wins); a | ||
| value that is not an object fails the item. An item carrying | ||
| `catalog_input` is a `catalog_metadata` item unless it declares another `test_kind`. The | ||
|
Comment on lines
+876
to
+877
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Document when another inferred kind takes precedence. A Langfuse item with valid
|
||
| judge reads the item's `question` as the ground truth, so state the object's real metadata | ||
| there in plain words (what a metric computes, which filters a visualization has). | ||
|
|
||
| ```json | ||
| { | ||
| "id": "catalog-001", | ||
| "dataset_name": "analytics_catalog_generate", | ||
| "test_kind": "catalog_metadata", | ||
| "question": "Description for the metric '# of Single Orders'. Its MAQL counts customers who placed exactly one order.", | ||
| "catalog_input": {"object_type": "Metric", "object_id": "of_orders_1", "output": "description"}, | ||
| "expected_output": { | ||
| "must_include": ["Says that it counts customers who placed exactly one order."], | ||
| "must_not_include": ["Says the metric counts orders rather than customers."], | ||
| "rubric": [] | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| `object_type` is one of `Visualization`, `Dashboard`, `Metric`, `Fact`, `Attribute`; | ||
| `output` is `title` or `description`; unknown keys are rejected. Every `expected_output` | ||
| key is optional: | ||
|
|
||
| - `expect_note` — `true` for an object the endpoint must decline: the item passes on no | ||
| text plus a non-empty `note`, and nothing else is graded. | ||
| - `format` — overrides of the deterministic checks, which gate pass/fail: `max_words` | ||
| (title 10, description 80), `max_sentences` (title 1, description 4), `no_end_period` | ||
| (title), `no_markup`, `no_ids`. `false` or `null` switches a check off; unknown keys are | ||
| rejected. A generated text (`generated`) and an empty `note` (`no_note`) always apply. | ||
| - `must_include`, `must_not_include`, `rubric` — judge criteria, graded like | ||
| `dashboard_summary`. Criteria may name the object's metadata; the judge sees no data. | ||
|
|
||
| A 200 answer with no usable text (a null value without a note, or the placeholder | ||
| `"string"`) is asked again once; `unusable_retries` in the item detail records it, with | ||
| `word_count`. The judge steps are calibrated with `make calibrate-catalog-metadata` (needs | ||
|
Comment on lines
+910
to
+911
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Qualify when the report contains If both 200 responses contain no text and no note, the evaluator records
|
||
| `OPENAI_API_KEY`); re-run it after changing them. Request errors are handled as for | ||
| `document_takeaway`; the CI entry point is `evaluate_catalog_metadata`. | ||
|
Comment on lines
+912
to
+913
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. List the catalog client’s retryable HTTP statuses. The reference to
|
||
|
|
||
| ### Items asked in a dashboard context (`user_context`) | ||
|
|
||
| An item can be asked the way a user asks from an open dashboard or an attached widget. | ||
|
|
@@ -966,6 +1012,7 @@ from that turn on until another turn changes it, as the attached context does in | |
| | `guardrail` | Refusal/redirect (visualization response auto-fails) | `[llm-judge]` | | ||
| | `dashboard_summary` | Dashboard summary (via `/summary` endpoint) scored against a rubric by LLM | `[llm-judge]` | | ||
| | `document_takeaway` | One short takeaway for a page (via `/takeaway` endpoint): deterministic format checks plus a rubric scored by LLM | `[llm-judge]` | | ||
| | `catalog_metadata` | Analytics Catalog title or description for one object (via the `analyticsCatalog/generate*` endpoints): format checks plus a rubric scored by LLM | `[llm-judge]` | | ||
|
|
||
| ## Optional extras | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,67 @@ | ||
| # (C) 2026 GoodData Corporation | ||
| """Does the real judge grade catalog_metadata criteria on the right side? | ||
|
|
||
| tests/test_catalog_metadata_evaluator.py pins the wiring; it cannot pin the verdict, because | ||
| the verdict is an LLM's. This does, against tests/fixtures/catalog_metadata_calibration.json: | ||
| each case is one item question (the object's real metadata), one criterion and one text, and | ||
| the criterion must be satisfied ("pass") or not ("fail") on every run. | ||
|
|
||
| The fixture keeps texts that state a wrong fact the question contradicts: a violation judge | ||
| that grades correctness instead of presence scores those as absent and lets them pass. | ||
|
|
||
| Run it after changing _POSITIVE_STEPS / _VIOLATION_STEPS: | ||
|
|
||
| OPENAI_API_KEY=... make calibrate-catalog-metadata | ||
|
|
||
| It is a script rather than a test because CI has no OPENAI_API_KEY. Exits 1 on any mismatch. | ||
| """ | ||
|
|
||
| import json | ||
| import os | ||
| import sys | ||
| from pathlib import Path | ||
|
|
||
| from gooddata_eval.core.evaluators.catalog_metadata import CatalogMetadataEvaluator | ||
| from gooddata_eval.core.models import CatalogInput, ChatResult, DatasetItem | ||
|
|
||
| RUNS = 3 | ||
| FIXTURE = Path(__file__).resolve().parent.parent / "tests" / "fixtures" / "catalog_metadata_calibration.json" | ||
| _FIELDS = {"include": "must_include", "exclude": "must_not_include"} | ||
|
|
||
|
|
||
| def main() -> int: | ||
| if not os.environ.get("OPENAI_API_KEY"): | ||
| raise SystemExit("OPENAI_API_KEY required") | ||
| cases = json.loads(FIXTURE.read_text()) | ||
| evaluator = CatalogMetadataEvaluator() | ||
| print(f"judge model: {evaluator._positive_judge.model}, runs: {RUNS}") | ||
| mismatches = 0 | ||
| for case in cases: | ||
| item = DatasetItem( | ||
| id=case["label"], | ||
| dataset_name="catalog_metadata_calibration", | ||
| test_kind="catalog_metadata", | ||
| question=case["question"], | ||
| expected_output={_FIELDS[case["kind"]]: [case["criterion"]], "format": {}}, | ||
| catalog_input=CatalogInput.model_validate( | ||
| {"object_type": "Metric", "object_id": "calibration", "output": case["output"]} | ||
| ), | ||
| ) | ||
| chat = ChatResult(text_response=case["text"], raw_response={"note": None}) | ||
| key = f"{case['kind']}_0" | ||
| verdicts = [evaluator.evaluate(item, chat).detail.get(key) for _ in range(RUNS)] | ||
| ungraded = sum(not isinstance(v, bool) for v in verdicts) | ||
| satisfied = sum(v is True for v in verdicts) | ||
| want = RUNS if case["want"] == "pass" else 0 | ||
| # An ungraded run proves nothing, on either side. | ||
| ok = satisfied == want and not ungraded | ||
| mismatches += not ok | ||
| print( | ||
| f"{case['label']:48} want {case['want']:4} satisfied {satisfied}/{RUNS}{f' ungraded {ungraded}' if ungraded else ''} {'OK' if ok else 'MISMATCH'}" | ||
| ) | ||
| print(f"\n{'all calibration texts on the right side' if not mismatches else f'{mismatches} mismatch(es)'}") | ||
| return 1 if mismatches else 0 | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| sys.exit(main()) |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| # (C) 2026 GoodData Corporation |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The README describes item-level failure, but a malformed
catalog_inputfails the whole Langfuse dataset load.In
load_langfuse_dataset, one list comprehension maps every raw item. Acatalog_inputthat is not a JSON object, such as[], raisesValueErrorinside_item_from_raw. Nothing catches the error, so no item in the dataset runs. The README sentence "value that is not an object fails the item" describes behavior the loader does not implement.Choose one correction. Either document that a malformed
catalog_inputaborts the dataset load, or change the loader to record the error on that item and continue with the remaining items. Keep the documentation and the loader behavior consistent.