A Text-to-SQL evaluation toolkit for benchmarking a text-to-SQL agent against a dataset and scoring the results. It bundles two workflows in one project:

ingestion — extract a source database's schema into Auto Ontology's Postgres + pgvector stores, compile the semantic layer, and enrich the graph with metadata and custom analyses.

NeMo Gym benchmark — run an evaluation set through two arms, a schema-only LLM control and the ontology-grounded agent, scored by one deterministic verifier and reported as a single delta.

The typical lifecycle is ingest → semantic compile → benchmark : you ingest a database so the agent can retrieve its schema, then run the benchmark to see how much that grounding is worth against a schema-only control. No LLM judge is involved: scoring is deterministic execution match, with each benchmark's official rule where it has one (BIRD, BEAVER).

Important

Auto Ontology is installed automatically.

uv sync installs the pinned NVIDIA/auto-ontology Git revision.…