Skip to main content

Augur (de-augur)

점술관. 자연어를 SQL로 옮기되, 근거(스키마)를 대고, 틀리면 왜 틀렸는지 센다.

Parquet 레이크에 자연어로 질문한다. 스키마 RAG(테이블·컬럼·샘플값 검색) → LLM → DuckDB 실행. 모든 호출은 trace(JSONL)로 남고, augur eval은 골든 SQL과 실행 결과를 대조해 실패 모드를 자동 분류한다. 벡터 DB 없음, SDK 없음(httpx), 임베딩 없음(어휘 검색). 필요해지면 retrieve.Scorer 구현체 하나를 추가한다.

question ─▶ retrieve(catalog) ─▶ prompt ─▶ LLM ─▶ SQL ─▶ guard(SELECT only) ─▶ DuckDB
                │                                            │ error? → repair ×1
                └── SchemaDoc (table, path, columns, samples) └──────────────▶ Trace

사용

pip install de-augur
augur auth set anthropic                 # 숨김 입력 → ~/.config/augur/credentials.json (0600)
augur auth status                        # env | file | none — 값은 앞 4자리만

augur index eval/sources.yaml            # {tables: {orders: './data/orders/*.parquet'}} → catalog.json
augur ask "How many orders are paid?"    # SQL은 stderr, 결과는 stdout(jsonl)
augur eval eval/cases.yaml               # 실패 모드별 집계 + Parquet/JSONL 기록
augur report                             # run 간 추이
gladius query "SELECT failure_mode, count(*) FROM './data/augur/eval/*.parquet' GROUP BY 1"

키는 ANTHROPIC_API_KEY/OPENAI_API_KEY 환경변수 → augur auth set으로 저장한 파일 → (TTY라면) 실행 시 숨김 프롬프트 순으로 찾는다. 인자로는 받지 않는다 — 셸 히스토리에 남는다. 파일 위치는 AUGUR_HOME으로 바꿀 수 있다. 키는 trace·로그·repr에 절대 찍히지 않는다.

OpenAI 호환 API(Gemini, Groq, OpenRouter, 로컬 서버)는 --provider openai에 OPENAI_BASE_URL을 준다. 키는 그대로 OPENAI_API_KEY에 넣는다.

export OPENAI_BASE_URL=https://generativelanguage.googleapis.com/v1beta/openai
augur eval packages/augur/eval/cases.yaml --provider openai --model gemini-flash-latest

실패 모드 (자동 분류, 단계 순)

모드 판정 원인 층
RETRIEVAL_MISS 케이스의 tables가 검색 결과에 없음 retrieve
NON_SELECT 쓰기 문장 — 실행 전 차단 prompt/model
UNANSWERABLE 모델이 답 불가 선언(정답은 answerable) prompt/model
HALLUCINATED_TABLE 카탈로그에 없는 경로 참조 prompt/model
HALLUCINATED_COLUMN DuckDB "not found in FROM clause" schema render/model
EXEC_ERROR 그 외 실행 실패(문법 등) model
WRONG_SHAPE 실행됐지만 컬럼 수/행 수가 골든과 다름 semantic (집계 축·필터)
WRONG_RESULT 모양은 같은데 값이 다름 semantic (조건·리터럴)
CORRECT 집합 비교 일치 —

앞 단계 실패가 뒷 단계 실패를 만든다. 그래서 RETRIEVAL_MISS가 결과 일치보다 우선한다 — "우연히 맞은 것"은 개선 신호가 아니다.

케이스 파일

cases:
  - id: l3-001
    question: Total paid revenue per product name.
    golden_sql: SELECT ... JOIN ...
    tables: [orders, products]     # 검색이 반드시 맞혀야 하는 테이블
    answerable: true               # false면 UNANSWERABLE이 정답
    notes: 자유 텍스트 (수동 라벨, 함정 설명)

동봉 eval/cases.yaml 34건 = L1 단일 집계 · L2 필터/그룹 · L3 JOIN · L4 시간 · L5 함정 (답 불가, 없는 리터럴, DML 유도, 한국어). eval/make_fixture.py가 seed 고정 합성 데이터를 만든다.

하지 않는 것

  • ORDER BY 의미 검증(집합 비교만). 순서가 답인 케이스는 notes에 표시.
  • 스키마 수백 개 규모의 검색 품질 보장 — 어휘 검색의 한계이며 측정 대상.
  • 자동 재시도 2회 이상. 1회 repair로 못 고치면 실패로 센다.

Metadata

Release files for de-augur 0.6.5

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for de-augur 0.6.5
File Size Uploaded Provenance Actions
de_augur-0.6.5.tar.gz 29.1 kB With provenance Details for de_augur-0.6.5.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for de-augur 0.6.5
File Interpreter ABI Platform Actions
de_augur-0.6.5-py3-none-any.whl Python 3 none any Details for de_augur-0.6.5-py3-none-any.whl

Total release size: 56.2 kB

Release files / de_augur-0.6.5.tar.gz

Download URL de_augur-0.6.5.tar.gz
Size 29.1 kB
Tags Source
SHA-256 checksum
How to use checksums
718f368b344b7b77dd443ffd428e65a2d4b8e680ca5917eaaa73f852d694b7e6
BLAKE2b-256 checksum
How to use checksums
71d4eef3a5fc48cd1ba78c29eedccf1f9da76f6674d334c8a5bb16425d3647e4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

Release files / de_augur-0.6.5-py3-none-any.whl

Download URL de_augur-0.6.5-py3-none-any.whl
Size 27.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
e8b59592e8b1950128d1bafec7d1f659ad9137b99e2804dcd667e7e4a15143e7
BLAKE2b-256 checksum
How to use checksums
7f7c10109fbf88d900b49b66668f7a12dfcc0970975da80d2986293b1e1c921e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

Security for de-augur 0.6.5

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for de_augur-0.6.5.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page