Skip to main content

datamarket

datamarket is a Python library of reusable scraping, data ingestion, and integration utilities used across DataMarket projects.

This README explains what the library is, how to run it locally, how to use it, and where deeper documentation lives.

It solves a practical problem: different scrapers and ETL jobs often re-implement the same low-level pieces (HTTP retries, proxy rotation, SQLAlchemy batch writes, cloud storage clients, LLM wrappers). This repository centralizes those capabilities in a single package so projects can stay focused on business logic.

Project Overview

  • Primary value: standardized interfaces for data collection, transformation, and delivery.
  • Language/runtime: Python ^3.12 (from pyproject.toml).
  • Package manager/build: Poetry (pyproject.toml, poetry.lock).
  • Testing: pytest-based tests in tests/.
  • Lint/format: pre-commit hooks (Ruff + Ruff format) via pre-commit-config submodule.

High-Level Architecture

Core package code lives in src/datamarket/ and is organized by responsibility:

  • src/datamarket/interfaces/: service-facing interfaces (LLM, SQLAlchemy, proxy, AWS, Azure Blob, Drive, FTP, Tinybird, Nominatim, PeerDB).
  • src/datamarket/utils/: shared helpers (HTTP client wrapper, config loading, logging, Playwright/Selenium helpers, string normalization, data quality sampler).
  • src/datamarket/exceptions/: custom exception types used across request and proxy workflows.
  • src/datamarket/params/: static parameter dictionaries/constants (for example, Nominatim enrichment data).

For architecture diagrams and deeper design notes, see docs/2. Architecture Overview.md.

Prerequisites

  • Python 3.12.
  • pip (for install) and optionally poetry (for dependency/workflow management).
  • Optional: Conda if you want to use the bootstrap helper in init.sh.

Installation

To install this library in your Python environment:

pip install datamarket

Environment Setup

Option A: Poetry workflow

poetry install
poetry shell

Option B: Conda bootstrap script

init.sh creates a Conda environment named <package>_env, installs the package in editable mode, initializes submodules, and installs pre-commit hooks.

bash init.sh

Basic Usage

This section shows how to use the package from consumer projects.

  • Import interfaces directly from module paths, for example:
    • from datamarket.interfaces.llm import LLMInterface
    • from datamarket.interfaces.proxy import ProxyInterface
    • from datamarket.interfaces.alchemy import AlchemyInterface
  • Load INI-style config using datamarket.utils.main.get_config when needed.
  • Run end-to-end examples from examples/ for LLM and vision use cases.

Development Workflow

Run examples

python examples/llm_usage_examples.py
python examples/llm_vision_examples.py

Run tests

pytest -v

Lint and format

This repo uses pre-commit hooks defined in pre-commit-config/.pre-commit-config.yaml:

pre-commit run --all-files

Build artifacts

poetry build

Built distributions are output to dist/.

Configuration

This library is configuration-driven. Most interfaces expect either:

  • a dict-like object (config["section"]["key"]), or
  • a ConfigParser/RawConfigParser object for INI files.

Common sections used by interfaces include:

  • [llm] for LLMInterface (provider, api_key, model).
  • [db] for AlchemyInterface and Postgres peer operations.
  • [proxy] for ProxyInterface (hosts, optional tor_password).
  • [tinybird], [osm], [drive].
  • Profile-based sections such as [aws:<profile>], [azure:<profile>], [ftp:<profile>].
  • PeerDB-specific sections: [peerdb], [clickhouse], [peerdb-s3].

See the generated wiki pages in docs/ for concrete config and workflow details, especially docs/3. Workflows.md and docs/Deep Dive/Interfaces.md.

Deployment and Release Notes

  • This repository is a library package, not a deployable service.
  • Release packaging is supported through Poetry (poetry build) and Twine can be used for publishing.
  • CI/CD release automation is not configured in this repository (no .github/workflows/ present).
  • See docs/4. ADRs.md for architecture-level release and maintenance trade-offs.

Upgrading the camoufox extra (0.11.1+)

From 0.11.1 the camoufox extra installs the official camoufox[geoip] package instead of the cloverlabs-camoufox fork, with Playwright 1.62.0. 0.11.1 pins camoufox 0.5.8 and 0.11.2 pins 0.5.9b4, an upstream pre-release whose build hides Juggler from pages.

  • Use a fresh environment. cloverlabs-camoufox and camoufox write to the same camoufox/ module. pip install -U on top of the fork leaves its camoufox/webgl/ package behind, and that breaks import camoufox; datamarket then raises an error with the fix. The repos' init.sh (and so dix update env) recreates the environment, which avoids this. To repair an environment in place: pip uninstall -y cloverlabs-camoufox camoufox && pip install --force-reinstall --no-deps camoufox==0.5.9b4
  • The browser build comes from the package. camoufox 0.5.x pairs with the build named in its browser-pin.json (0.5.9b4 ↔ 156.0.1-beta.39) and datamarket fetches it on first launch. To change builds, change the camoufox pin, not camoufox_version.
  • Old and new environments can share a machine. The browser cache (~/.cache/camoufox, config.json included) is shared by every environment of the user. datamarket launches every Camoufox build by its path, so it never rewrites active_version, which is the key environments on the fork read to pick their build. camoufox fetch marks what it installs as active, and datamarket puts the previous value back right after, so environments on the fork, including their direct Camoufox(...) calls, keep launching the build they launched before. Code that calls Camoufox(...) directly with camoufox 0.5.x does not get this, because camoufox itself rewrites active_version.
  • If ~/.cache/camoufox has files but no .0.5_FLAG, the first 0.5.x fetch deletes the whole cache ("Cleaning old data...").

cloakbrowser extra (0.6.x)

  • The keyless free binary is still Chromium 146.0.7680.177.5, downloaded to ~/.cloakbrowser (CLOAKBROWSER_CACHE_DIR). The wrapper still checks for a newer free binary in the background; set CLOAKBROWSER_AUTO_UPDATE=false to freeze it.
  • On Linux it spoofs Windows and warns once per cache dir when the Windows font set is incomplete. That is a detectable mismatch: install the fonts on the worker (see the CloakBrowser README) rather than silencing it with CLOAKBROWSER_SUPPRESS_FONT_WARNING=1.
  • It prints a Pro banner to stderr on the first launch and every few days after; it is harmless.
  • CloakBrowser stopped publishing free binaries at 146 (May 2026); every newer build needs a licence key. See the clearcote extra for an open alternative.

clearcote extra (experimental)

PlaywrightCrawler(browser="clearcote") drives the Clearcote open build: a stealth Chromium whose patches are BSD-3 and reproducible from source.

  • The SDK pin selects the browser: clearcote 0.55.1 ↔ open build v0.1.0-pre.23 (Chromium 150.0.7871.114), downloaded to ~/.cache/clearcote (CLEARCOTE_CACHE) with its own cross-process install lock. Do not set CLEARCOTE_AUTO_UPDATE, or each worker may pick up a newer build on its own.
  • Each launch gets a fresh fingerprint seed, the timezone and language of the proxy's exit IP, and humanized input. Override any of them with clearcote_options.
  • The default persona matches the host OS. Spoofing Windows on Linux leaves host-native font and canvas hashes unless clearcote_options includes a fingerprint_profile captured on Windows.
  • page.expose_function / expose_binding do not work on this build.

Troubleshooting

  • ModuleNotFoundError for optional features: install required extras (for example .[llm], .[pytest], .[boto3]).
  • Configuration must contain 'llm' section: include [llm] with api_key before creating LLMInterface.
  • No working proxies available: verify [proxy] hosts format (host:port or user:pass@host:port) and network access.
  • SQLAlchemy connection errors: verify [db] credentials and engine string.
  • Pre-commit command not found: install pre-commit in your active environment.

Contributing (Summary)

  • Keep changes scoped and aligned with existing module boundaries in src/datamarket/.
  • Add or update tests under tests/ for behavioral changes.
  • Run pytest -v and pre-commit run --all-files before opening a PR.
  • Keep docs current when interfaces, config keys, or workflows change.

Documentation Map

  • Wiki home: docs/Home.md
  • Project overview: docs/1. Project Overview.md
  • Architecture overview (C4): docs/2. Architecture Overview.md
  • Workflows: docs/3. Workflows.md
  • Architecture decisions: docs/4. ADRs.md
  • Deep dives: docs/Deep Dive/Interfaces.md, docs/Deep Dive/LLM.md, docs/Deep Dive/SQLAlchemy.md, docs/Deep Dive/Utilities.md, docs/Deep Dive/Geo Enrichment.md
  • Digital twin artifacts: docs/_twin/inventory.json, docs/_twin/graph.json, docs/_twin/domain-map.md, docs/_twin/patterns.md

Documentation Status

Diataxis type: Reference.

  • This README is the entry point and is maintained incrementally from validated repository summaries.
  • Current generated references in this run:
    • docs/1. Project Overview.md
    • docs/2. Architecture Overview.md
    • docs/3. Workflows.md
    • docs/4. ADRs.md
  • Known unknowns:
    • UNKNOWN: linked pages under the external docs submodule may differ from this local snapshot.

License

GPL-3.0-or-later. See LICENSE.

Sources: README.md (summary_hash: 905c027d111146820a6ea5c807c7b4a0f7094f9b36ab6528f36500a3f5e07520)

Metadata

Release files for datamarket 0.11.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for datamarket 0.11.2
File Size Uploaded Provenance Actions
datamarket-0.11.2.tar.gz 114.3 kB Without provenance Details for datamarket-0.11.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for datamarket 0.11.2
File Interpreter ABI Platform Actions
datamarket-0.11.2-py3-none-any.whl Python 3 none any Details for datamarket-0.11.2-py3-none-any.whl

Total release size: 244.7 kB

Release files / datamarket-0.11.2.tar.gz

Download URL datamarket-0.11.2.tar.gz
Size 114.3 kB
Tags Source
SHA-256 checksum
How to use checksums
8e2fd4b7fca1c8e5ee9c0a78eaad9a30fc193caa73f351061eb766ff761ce248
BLAKE2b-256 checksum
How to use checksums
bcd0e60aeb8fbd6506cff07fcc275d3a3f92e8c7ec13d9bf0c8689aa5d989dc9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.15

Release files / datamarket-0.11.2-py3-none-any.whl

Download URL datamarket-0.11.2-py3-none-any.whl
Size 130.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
a213d27441e55840de61bb8d461a8975a5b3ff52061abe7e02a162e4a0b0c5e1
BLAKE2b-256 checksum
How to use checksums
8359e236c3fe8152ec823eb6ce6ce331c56b0ba13d7bbe38793c59a484bce262
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.15

Security for datamarket 0.11.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.11.2 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page