nordcrawl 0.1.1

LLM-ready batch crawling for Nordic company data

NordCrawl

LLM-ready batch crawling for Nordic company data.

NordCrawl crawls URL lists from CSV/JSON, extracts structured Swedish company data, and can upsert the results directly into Supabase.

Live demo

The screenshot above is based on a real crawl run with Rich output.

• Demo input: examples/urls.json
• Verified output: examples/results.json
• More details: examples/README.md

Why people star projects like this

• Fast to try: one command from file or CLI
• Useful immediately: real data extraction for Swedish company pages
• Practical output: JSON, CSV, and Supabase sync
• Production-minded: retries, dedupe, rate limiting, and validation

Features

• Batch crawling with configurable concurrency
• Per-domain rate limiting
• Retry logic with exponential backoff
• Swedish company data extraction
  • company name
  • org number
  • email
  • phone
  • address
• Consistent JSON schema via Pydantic
• CSV/JSON import and export
• Supabase upserts with dedupe by URL
• CLI-first workflow with a clean Python API

PyPI

The package name is nordcrawl.

After publishing, install it with:

pip install nordcrawl

Quick start

1) Install dependencies

pip install -r requirements.txt

2) Configure environment

Copy the example file and fill in your Supabase keys:

cp .env.example .env

3) Run a crawl

python3 main.py crawl-urls \
  https://www.ikea.com/se/sv/ \
  https://www.hemkop.se/ \
  https://www.ica.se/ \
  --output results.json

CLI

Crawl from file

python3 main.py crawl data/test_urls.json --output results.json --format json

Save directly to Supabase

python3 main.py crawl data/test_urls.json --save-db

Show stats

python3 main.py stats --limit 50

Search stored companies

python3 main.py search "IKEA" --limit 20

Export from Supabase

python3 main.py export company_data.csv --format csv

Python API

import asyncio
from src import BatchCrawler, SupabaseManager

async def main() -> None:
    crawler = BatchCrawler(max_concurrent=5, rate_limit_delay=2.0)
    results = await crawler.crawl_urls([
        "https://www.ikea.com/se/sv/",
        "https://www.hemkop.se/",
        "https://www.ica.se/",
    ])

    print(results)

asyncio.run(main())

Input formats

CSV

url
https://www.example.se
https://www.example2.se

JSON

[
  "https://www.example.se",
  {"url": "https://www.example2.se"}
]

Supabase table

NordCrawl expects a table named company_data with these columns:

• url
• company_name
• org_number
• email
• phone
• address
• postal_code
• city
• website
• crawled_at
• status
• raw_html_hash
• extraction_confidence
• created_at
• updated_at

The migration is included here:

supabase/migrations/20240325000001_create_company_data_table.sql

Example output

{
  "url": "https://www.example.se",
  "company_name": "Exempel AB",
  "org_number": "556123-4567",
  "email": "[e-mail]",
  "phone": "[phone]",
  "address": "Storgatan 1",
  "postal_code": "123 45",
  "city": "Stockholm",
  "status": "completed",
  "extraction_confidence": 0.85
}

Docker / Supabase note

Local supabase start requires Docker access. If you do not want to run Supabase locally:

1. Apply the migration in your remote Supabase project.
2. Set SUPABASE_URL and SUPABASE_KEY in .env.
3. Run the crawler with --save-db.

Project structure

.
├── main.py
├── requirements.txt
├── src/
│   ├── api/
│   ├── crawlers/
│   ├── database/
│   ├── extractors/
│   └── models/
├── supabase/
│   └── migrations/
└── tests/

Roadmap

• Better selector coverage for Swedish business websites
• Richer address parsing
• Supabase auth-aware dashboard
• Web UI for crawl jobs
• More sample datasets and benchmarks

Contributing

PRs are welcome.

1. Fork the repo
2. Create a branch
3. Add tests
4. Open a pull request

License

Apache-2.0

---

If you like the project, star it on GitHub to help it grow.