Skip to main content

Augur (de-augur)

점술관. 자연어를 SQL로 옮기되, 근거(스키마)를 대고, 틀리면 왜 틀렸는지 센다.

Parquet 레이크에 자연어로 질문한다. 스키마 RAG(테이블·컬럼·샘플값 검색) → LLM → DuckDB 실행. 모든 호출은 trace(JSONL)로 남고, augur eval은 골든 SQL과 실행 결과를 대조해 실패 모드를 자동 분류한다. 벡터 DB 없음, SDK 없음(httpx), 임베딩 없음(어휘 검색). 필요해지면 retrieve.Scorer 구현체 하나를 추가한다.

question ─▶ retrieve(catalog) ─▶ prompt ─▶ LLM ─▶ SQL ─▶ guard(SELECT only) ─▶ DuckDB
                │                                            │ error? → repair ×1
                └── SchemaDoc (table, path, columns, samples) └──────────────▶ Trace

사용

pip install de-augur
augur auth set anthropic                 # 숨김 입력 → ~/.config/augur/credentials.json (0600)
augur auth status                        # env | file | none — 값은 앞 4자리만

augur index eval/sources.yaml            # {tables: {orders: './data/orders/*.parquet'}} → catalog.json
augur ask "How many orders are paid?"    # SQL은 stderr, 결과는 stdout(jsonl)
augur eval eval/cases.yaml               # 실패 모드별 집계 + Parquet/JSONL 기록
augur report                             # run 간 추이
gladius query "SELECT failure_mode, count(*) FROM './data/augur/eval/*.parquet' GROUP BY 1"

키는 ANTHROPIC_API_KEY/OPENAI_API_KEY 환경변수 → augur auth set으로 저장한 파일 → (TTY라면) 실행 시 숨김 프롬프트 순으로 찾는다. 인자로는 받지 않는다 — 셸 히스토리에 남는다. 파일 위치는 AUGUR_HOME으로 바꿀 수 있다. 키는 trace·로그·repr에 절대 찍히지 않는다.

OpenAI 호환 API(Gemini, Groq, OpenRouter, 로컬 서버)는 --provider openai에 OPENAI_BASE_URL을 준다. 키는 그대로 OPENAI_API_KEY에 넣는다.

export OPENAI_BASE_URL=https://generativelanguage.googleapis.com/v1beta/openai
augur eval packages/augur/eval/cases.yaml --provider openai --model gemini-flash-latest

실패 모드 (자동 분류, 단계 순)

모드 판정 원인 층
RETRIEVAL_MISS 케이스의 tables가 검색 결과에 없음 retrieve
NON_SELECT 쓰기 문장 — 실행 전 차단 prompt/model
UNANSWERABLE 모델이 답 불가 선언(정답은 answerable) prompt/model
HALLUCINATED_TABLE 카탈로그에 없는 경로 참조 prompt/model
HALLUCINATED_COLUMN DuckDB "not found in FROM clause" schema render/model
EXEC_ERROR 그 외 실행 실패(문법 등) model
WRONG_SHAPE 실행됐지만 컬럼 수/행 수가 골든과 다름 semantic (집계 축·필터)
WRONG_RESULT 모양은 같은데 값이 다름 semantic (조건·리터럴)
CORRECT 집합 비교 일치 —

앞 단계 실패가 뒷 단계 실패를 만든다. 그래서 RETRIEVAL_MISS가 결과 일치보다 우선한다 — "우연히 맞은 것"은 개선 신호가 아니다.

케이스 파일

cases:
  - id: l3-001
    question: Total paid revenue per product name.
    golden_sql: SELECT ... JOIN ...
    tables: [orders, products]     # 검색이 반드시 맞혀야 하는 테이블
    answerable: true               # false면 UNANSWERABLE이 정답
    notes: 자유 텍스트 (수동 라벨, 함정 설명)

동봉 eval/cases.yaml 34건 = L1 단일 집계 · L2 필터/그룹 · L3 JOIN · L4 시간 · L5 함정 (답 불가, 없는 리터럴, DML 유도, 한국어). eval/make_fixture.py가 seed 고정 합성 데이터를 만든다.

하지 않는 것

  • ORDER BY 의미 검증(집합 비교만). 순서가 답인 케이스는 notes에 표시.
  • 스키마 수백 개 규모의 검색 품질 보장 — 어휘 검색의 한계이며 측정 대상.
  • 자동 재시도 2회 이상. 1회 repair로 못 고치면 실패로 센다.

Metadata

Release files for de-augur 0.6.6

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for de-augur 0.6.6
File Size Uploaded Provenance Actions
de_augur-0.6.6.tar.gz 29.2 kB With provenance Details for de_augur-0.6.6.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for de-augur 0.6.6
File Interpreter ABI Platform Actions
de_augur-0.6.6-py3-none-any.whl Python 3 none any Details for de_augur-0.6.6-py3-none-any.whl

Total release size: 56.2 kB

Release files / de_augur-0.6.6.tar.gz

Download URL de_augur-0.6.6.tar.gz
Size 29.2 kB
Tags Source
SHA-256 checksum
How to use checksums
47d4eae037f70536ec95a51719eef8a90d6ff36030785fde6bc7625d1b7a0ec0
BLAKE2b-256 checksum
How to use checksums
9e810c83f96497c52d7c21f10bea10fbe642cc82f6b14b84e4daf7f9c26c7940
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 22, 2026.

Transparency log

Release files / de_augur-0.6.6-py3-none-any.whl

Download URL de_augur-0.6.6-py3-none-any.whl
Size 27.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
016e03a0a6b4fcca4842863388b82841a7ed89877d001d081b257a757f04dc98
BLAKE2b-256 checksum
How to use checksums
7ee419f22fe949d0fdf477d76e3f7911de1e527049678442fa5ce3f6d4e253fd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 22, 2026.

Transparency log

Security for de-augur 0.6.6

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for de_augur-0.6.6.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 22, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page