Data Universe Miner Doctor (du-doctor)

A focused, read-only diagnostic CLI for Bittensor Data Universe miners (NETUID 13, repo macrocosm-os/data-universe).

It answers exactly one question:

“Why is my Data Universe miner not earning well?”

This is not a general Bittensor dashboard, not a TAO portfolio tracker, and not a generic alert bot. It is a subnet-specific doctor that inspects your miner's setup on the box where it runs and tells you, in priority order, what to fix.

1. What is Data Universe Miner Doctor?

du-doctor runs locally on your miner's Ubuntu/Linux server and inspects:

1. Bittensor hotkey registration
2. Metagraph miner metrics (UID, rank, trust, consensus, incentive, emission)
3. Data Universe repo status (git remote, branch, behind-upstream, expected files)
4. PM2 miner process status (online / errored / restart loops)
5. Python / server environment (OS, Python, disk, RAM, CPU, internet, GPU)
6. Data Universe scraping configuration (scraping_config.json)
7. API credential presence (Apify / Reddit / X) — never the values
8. Local database / data freshness (SQLite size, mtime, row counts)
9. Logs and common Data Universe errors (with redacted excerpts)
10. Basic network / axon / port reachability (local only)
11. Heuristic possible reasons for low incentive or low emission

It produces a colourful terminal report, machine-readable JSON, or a shareable markdown report.

2. Who it is for

• Data Universe (SN13) miners debugging low or dropping rewards.
• Operators who want a fast, repeatable health check before/after changes.
• Anyone helping a miner remotely who wants a safe, redacted report to read.

3. What it checks

See the list above and the example output. Every check returns one of four statuses:

The overall status is the worst non-skipped status across all checks.

4. What it does not do

• It does not trade, transfer, stake, unstake, or register anything.
• It does not modify your miner, repo, config, or .env.
• It does not auto-update anything.
• It does not upload logs, .env, or telemetry anywhere.
• It does not need (or ask for) your wallet password.

5. Safety / privacy policy

This tool is read-only.

• It never asks for seed phrases.
• It never asks for mnemonics or private keys.
• It never asks for coldkey/wallet passwords.
• It never transfers TAO.
• It never registers hotkeys.
• It never uploads logs or .env.
• It only diagnoses your local setup and public chain/metagraph data.

Redaction guarantees:

• API keys, tokens, secrets, and KEY=VALUE pairs are redacted everywhere (terminal, JSON, markdown).
• .env is read only to check which variable names exist — values are never read into output.
• Hotkeys are masked to first6...last6 (e.g. 5F3abc...XyZ921) unless you explicitly pass --unsafe-show-full-hotkey.
• SS58 addresses found in logs are masked too.

The only files du-doctor writes are under ~/.du-doctor/ (your config and a local metrics snapshot) and the markdown report you ask it to create.

6. Installation

Install from PyPI (Python 3.10+). Run it on the same server as your miner so it can see PM2, processes, logs, and the DB:

python3 -m venv ~/du-doctor && source ~/du-doctor/bin/activate
pip install --upgrade pip
pip install data-universe-miner-doctor

The bittensor SDK is an optional extra (it is large). Include it for live chain/metagraph checks — or install du-doctor into the venv your miner already uses (which already has the SDK):

pip install "data-universe-miner-doctor[bittensor]"

Without bittensor, every other check still works; the chain checks just report CRITICAL with an install hint.

Install the latest directly from GitHub (e.g. to get unreleased changes):

pip install "git+https://github.com/siteauditor/data-universe-miner-doctor.git"
# with the bittensor extra:
pip install "data-universe-miner-doctor[bittensor] @ git+https://github.com/siteauditor/data-universe-miner-doctor.git"

Install from source (for development):

git clone https://github.com/siteauditor/data-universe-miner-doctor.git
cd data-universe-miner-doctor
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

Docker is supported but optional — see Docker.

7. Quick start

du-doctor init

du-doctor check \
  --hotkey <your-hotkey-ss58> \
  --repo-path /path/to/data-universe \
  --scraping-config /path/to/data-universe/scraping_config.json \
  --env-path /path/to/data-universe/.env

Or put everything in the config once (du-doctor init then edit ~/.du-doctor/config.yaml) and just run du-doctor check.

8. Config examples

du-doctor init writes ~/.du-doctor/config.yaml. A fully-commented example is in examples/config.example.yaml. Key fields:

netuid: 13
network: finney

# Provide a hotkey ss58 directly (simplest)...
hotkey_ss58: "5F3sa2TJ...XyZ921"
# ...or a wallet name + hotkey name (only the PUBLIC ss58 is read; never unlocked)
wallet_name: ""
wallet_hotkey_name: ""

subnet_repo_path: "/home/ubuntu/data-universe"
pm2_process_name: ""          # optional exact PM2 app name
miner_port: null              # optional axon port, e.g. 8091

scraping_config_path: "/home/ubuntu/data-universe/scraping_config.json"
env_path: "/home/ubuntu/data-universe/.env"

Thresholds (drop %, disk/RAM/CPU %, stale-data hours, PM2 restarts, repo-behind commits) are all tunable in the config.

9. CLI examples

# Create the config
du-doctor init
du-doctor init --force --repo-path /home/ubuntu/data-universe

# Run all checks once (full terminal report)
du-doctor check
du-doctor doctor                      # alias for check

# Override config at run time
du-doctor check --hotkey 5F3sa2TJ... --repo-path ~/data-universe --miner-port 8091

# Machine-readable JSON (pipe into jq, a bot, a dashboard, ...)
du-doctor check --json | jq .overall_status

# Show OK/SKIPPED detail too, or disable colour
du-doctor check --verbose
du-doctor check --no-color

# Re-run on an interval and print only status changes
du-doctor watch --interval 300

# Write a shareable, redacted markdown report
du-doctor report --format markdown            # -> du-doctor-report.md
du-doctor report -o /tmp/miner-report.md

Exit codes (handy for cron/automation): 0 = OK, 1 = WARNING, 2 = CRITICAL.

du-doctor check options

--config, --netuid, --network, --hotkey, --wallet-name, --wallet-hotkey, --repo-path, --scraping-config, --env-path, --pm2-process-name, --miner-process-name, --miner-port, --json, --verbose, --no-color, --unsafe-show-full-hotkey.

10. Example output

Data Universe Miner Doctor
Overall status: CRITICAL

[OK] System: Ubuntu 22.04 detected
[OK] System: Python 3.10.12 (requires >=3.10)
[OK] Bittensor: bittensor SDK installed (v6.9.3).
[OK] Bittensor: Connected to 'finney' (block 4129877).
[OK] Bittensor: Registered on subnet 13, UID 142 (incentive 0.0123, emission 0.0041).
[WARNING] Bittensor: incentive dropped 31% (>= 25%) since the last snapshot.
[CRITICAL] Process: No Data Universe miner process found in PM2.
[WARNING] Repo: 6 commits behind origin/main.
[CRITICAL] Config: Apify scraper enabled but missing credentials: APIFY_API_TOKEN or APIFY_TOKEN.
[WARNING] Data: Local data size has not changed since last check. Scraper may be idle or blocked.
[WARNING] Logs: Found 12 warning log line(s) matching: rate limit, retrying.

Suggested fixes (priority order):
 1. Start the miner process:
    pm2 start python -- ./neurons/miner.py --wallet.name <wallet-name> --wallet.hotkey <hotkey-name>
 2. Add the missing scraper credentials to your .env (e.g. APIFY_API_TOKEN / REDDIT_*).
 3. Investigate local data — ensure the scraper is producing fresh, growing data.
 4. Update the Data Universe repo: cd <repo> && git pull
 5. Address scraper warnings in the logs (rate limits, upload/storage errors).

A full markdown example is in examples/sample-report.md.

11. How to interpret results

• Start at Suggested fixes (priority order) — it lists the highest-impact problems first, in roughly this order:
  1. Hotkey not registered
  2. Cannot connect to Subtensor
  3. Miner process not running
  4. PM2 restart loop
  5. Missing scraping credentials
  6. Invalid scraping_config.json
  7. No local data / stale data
  8. Repo very outdated
  9. Axon / port problem
  10. Rate limits / scraper errors
  11. Generic labels / low config quality
• CRITICAL items almost always block earnings — fix those first.
• The scoring/earning section is heuristic: it suggests possible reasons (“may reduce value”, “check subnet docs”), not the exact on-chain formula.

12. Common Data Universe miner problems (and what du-doctor shows)

13. Roadmap

• More precise SN13 scoring heuristics as the subnet's rules are confirmed.
• Optional richer SQLite introspection (per-source freshness).
• Config "lint" suggestions for label strategy.
• Plugin system so the same engine can doctor other subnets — du-doctor check --subnet <key> selects a SubnetProfile (du_doctor/profiles/); third parties register their own via the du_doctor.profiles entry-point group.

Docker (optional)

Docker is not required; local install is the recommended path. If you want it:

docker build -t du-doctor .                       # without bittensor SDK
docker build --build-arg WITH_BITTENSOR=1 -t du-doctor .   # with it
docker run --rm -v "$PWD":/work -w /work du-doctor check --repo-path /work/data-universe

Inside a container, PM2/process/log visibility depends on what you mount and share. Running locally on the miner box is more accurate.

Development

pip install -e ".[dev]"
pytest -q

Tests mock anything that would need a live chain (the bittensor SDK is mocked), so the suite runs fully offline.

License

MIT.