Chumoli

O‘zbekiston ma’lumot manbalari uchun engil EL vosita
dlt (Apache 2.0) ustida · UI + CLI

SQL, REST, fayl/S3, MoySklad va Bitrix24 manbalaridan — prodga tayyor — hamda O‘zbekiston manbalaridan (Click, Payme, Uzum, Didox) va Meta Ads’dan (beta) ma’lumotni olib DuckDB, PostgreSQL, lokal fayl yoki S3 ga yuklang. Brauzerda boshqaring yoki CLI orqali ishga tushiring.

Tez boshlash

Talab: Python 3.11+

pip install chumoli
chumoli ui

• Dashboard: http://127.0.0.1:8000/
• API docs: http://127.0.0.1:8000/api/docs
• To‘xtatish: Ctrl+C

Brauzersiz: chumoli ui --no-open yoki chumoli-api.

Ishlab chiqish (repo)

git clone https://github.com/farrux05-ai/chumoli.git
cd chumoli
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
chumoli ui

Birinchi pipeline

1. + Yangi yoki Ulagichlar
2. Manba tanlang → ulanishni tekshiring → saqlang
3. Run → Preview

UI dagi demo: SQL → DuckDB, REST (CBU valyuta kurslari) → DuckDB, Volume (sintetik).

Nima bor (v0.1)

Beta connectorlar

Click, Payme, Uzum Market, Didox va Meta Ads (Facebook Ads) kod bazasida bor va UI da beta deb belgilangan. Ular hali barcha merchant/sandbox muhitlarida to‘liq tasdiqlanmagan. Ishlatishdan oldin o‘z API kalitingiz bilan sinang; xato topsangiz Issues oching.

Prodga tayyor (stable): SQL, REST API, Filesystem/S3, sintetik volume, MoySklad va Bitrix24 — UI da ular badge’siz ko‘rinadi.

Destinationlar

Lokal fayl va S3 katalogda alohida. dlt _dlt_* metadata yashirin staging da qoladi; foydalanuvchi papkasiga faqat toza jadvallar chiqadi. Preview: papka yo‘li, fayllar, CSV namunasi.

Ma’lumot qayerda

~/.chumoli/                 # tizim (yashirin)
  chumoli_control.db
  master.key
  api.key
  pipelines/                # dlt state
  fs_staging/               # dlt filesystem + _dlt_* (yashirin)

~/chumoli-data/             # foydalanuvchi ko‘radigan
  <pipeline>.duckdb
  warehouse.duckdb          # path ko‘rsatilmagan DuckDB uchun fallback
  examples/
  exports/<pipeline>/       # toza CSV/Parquet

export CHUMOLI_HOME=/var/lib/chumoli
export CHUMOLI_DATA=/var/lib/chumoli-data
chumoli ui

master.key ni control DB bilan birga zaxiralang — kalitsiz secrets ochilmaydi.

CLI

chumoli --help
chumoli ui                 # dashboard + brauzer
chumoli ui --port 8080
chumoli list
chumoli run <pipeline>

Docker

docker compose up --build
# http://127.0.0.1:8000/

• Bind: 127.0.0.1:8000 (tashqi tarmoq uchun reverse proxy + CHUMOLI_API_KEY + TLS)
• Volume: chumoli_data → /data (CHUMOLI_HOME + CHUMOLI_DATA)
• Bitta uvicorn worker — scheduler va run lock shu processda

Muhit o‘zgaruvchilari

Telegram: dashboard Sozlamalar → bot token + chat id.

Ishlab chiqish

pip install -e ".[dev]"
PYTHONPATH=src python -m pytest tests/ -q

CI (GitHub Actions): har push/PR da Python 3.11 va 3.12 da testlar + Docker image build.

Litsenziya

Apache-2.0 · GitHub