BotScope

Python-first observability for an Internet-wide census of automated traffic.

BotScope is an Internet-wide bot traffic census: the Global Observatory federates public crawler/IP panels, crawl catalogs, and optional CDN estimates into a worldwide automation picture — alongside a local analyzer and native Qt desktop Observatory for authorized logs, sessions, and live capture.

Install from PyPI: pip install botscope · GUI: pip install "botscope[gui]"

Network contribution is OFF by default. BotScope does not perform unauthorized scanning. Zero-config for your own logs: no account, no cloud profile, and no API key required to classify local traffic.

Screenshots

Observatory — KPI cards, composition ring, Traffic Pulse, and category breakdown on the bundled synthetic demo corpus (DEMO DATA banner visible).

Global — Internet-wide census from zero-auth public sources; Cloudflare Radar optional	Events — virtualized table, query language, Classification Inspector
Bot Library — known signatures vs observed-in-dataset markers	Sources — registry status for public feeds and optional providers

Settings — local theme, privacy, and optional Cloudflare Radar token for CDN estimates (field shown empty; no account required).

Video demo

Animated tour of the main Observatory pages (demo data):

Screenshot strip · Recording script / MP4 placeholder

To capture a short screen recording yourself (launch → Demo → KPIs → Global → Events), follow the steps in docs/assets/demo/README.md and drop botscope-demo.mp4 (or .webm) beside the GIF.

What BotScope is

BotScope is an Internet-wide census of automated traffic, with a local measurement workstation for traffic you are authorized to analyze:

• Build a worldwide automation census in Global Observatory from federated zero-auth public sources (crawler IP ranges, Common Crawl catalog, and similar) plus optional Cloudflare Radar CDN estimates
• Classify requests from combined/common access logs (and optional PCAP / live paths)
• Separate OBSERVED totals from CLASSIFIED shares, with provenance badges
• Keep an honest UNKNOWN outcome instead of forcing certainty
• Explore both global census views and local sessions in a native desktop Observatory (Qt / PySide6 — not a website)

What BotScope is not

• Not limited to a single site or sensor — Global Observatory is the Internet-wide census surface
• Not a claim that one local log alone equals the whole Internet (local shares stay labeled local; the census comes from federated global sources)
• Not a cloud SaaS — analysis and preferences stay on your machine by default
• Not a substitute for authorization: only analyze systems and traffic you own or have permission to measure

The product’s primary global story is the Internet-wide census. Local Observatory KPIs remain dataset-scoped so you can compare your sensors against that census without conflating the two.

Features

Full matrix: FEATURES.md. Machine-readable registry: botscope features.

Quickstart

Requires Python 3.10+. Install from PyPI:

pip install "botscope[gui]"

Core (CLI / library only):

pip install botscope

From source (contributors):

git clone https://github.com/theworker02/botscope.git
cd botscope
python -m venv .venv
# Windows: .venv\Scripts\activate
# macOS/Linux: source .venv/bin/activate
pip install -e ".[gui,dev]"

Environment check and synthetic demo:

botscope doctor
botscope demo --output demo_analysis.bscope
botscope open demo_analysis.bscope

Desktop Observatory:

botscope gui
# or simply: botscope

In the GUI: Demo loads the bundled synthetic corpus (shows a DEMO DATA banner). Open analyzes an authorized access log. Drag-and-drop of logs, PCAPs, or .bscope sessions is supported.

Analyze a log from the CLI:

botscope analyze path/to/access.log --output analysis.bscope
botscope report analysis.bscope --format markdown --output report.md

Python API:

from botscope import Analyzer

result = Analyzer().analyze("access.log")
print(result.automation_fraction)
print(result.stats.by_category)

More detail: docs/guides/QUICKSTART.md · docs/guides/INSTALLATION.md

CLI overview

botscope with no subcommand launches the Observatory. Use --no-gui to print help without opening a window.

Run botscope --help or botscope <command> --help for options. Tutorials live under docs/tutorials/.

Architecture (summary)

Access log / PCAP / live sensor
        │
        ▼
   Ingest + privacy transforms
        │
        ▼
   Classify (rules, identity, optional ML)
        │
        ├──► .bscope session store (events, aggregates, workspace)
        ├──► CLI reports / export / research packs
        └──► Observatory GUI (KPIs, Events, Global census, Live, …)
                 │
                 └──► Global federation → Internet-wide census
                      (zero-auth public sources; Cloudflare Radar if you supply a token)

• Internet-wide census: Global Observatory is the census product — federated public panels and optional CDN estimates.
• Local-first: sessions and preferences stay on disk unless you explicitly enable network contribution.
• Provenance-aware: OBSERVED counts are never relabeled as CLASSIFIED shares; local KPIs stay distinct from the global census.
• Methodology notes: docs/research/METHODOLOGY.md · docs/research/GLOBAL_ESTIMATION.md.

Deeper maps: docs/architecture/REPOSITORY_MAP.md · diagrams in docs/architecture/diagrams/ · GUI guide docs/GUI.md

Documentation

Asset inventory (logos, screenshots, demo): docs/assets/README.md

Optional extras

pip install -e ".[gui,network,dev]"

Cloudflare Radar is optional. Leave Settings blank for normal local-log workflows; add your Radar Read token only if you want CDN bot/human estimates in Global Observatory. See docs/sources/CLOUDFLARE_RADAR.md.

Principles

1. Treat Global Observatory as an Internet-wide census, built from federated sources with provenance
2. Prefer UNKNOWN over forced certainty
3. Separate OBSERVED counts from CLASSIFIED shares — and local KPIs from the global census
4. Privacy transforms and local-first storage by default
5. Network contribution remains OFF unless explicitly enabled

Contributing

Development setup, style (Ruff), and PR guidelines: CONTRIBUTING.md.
Code of conduct: CODE_OF_CONDUCT.md.
Security reports: SECURITY.md (do not file public issues for vulnerabilities).
Support pointers: SUPPORT.md.

pip install -e ".[dev]"
pytest
ruff check src tests scripts

License

Apache License 2.0 — see LICENSE.

Citation helpers: botscope citation and CITATION.cff.