DAS Setup Helper

A Python CLI tool to set up crawlers, manage acquisitions, and extract structured data in WIHP (EUROSTAT). It provides four pipelines: creating crawlers, triggering acquisitions, stopping acquisitions, and extracting data from JSON acquisitions to processed CSV files.

---

Requirements

• Python >= 3.12
• uv — fast Python package and project manager

Install uv (once, globally):

curl -LsSf https://astral.sh/uv/install.sh | sh

---

Installation

From PyPI (recommended)

# Create a project directory
mkdir das-setup-helper && cd das-setup-helper

# Install the package in an isolated environment
uv add das-setup-helper

All CLI commands (das-setup-crawler, das-create-acq, das-stop-acq, das-extract-data) are then available via uv run.

From source (contributors)

# Clone the repository
git clone <repo-url>
cd das-setup-helper

# Install all dependencies including dev tools
make sync
# or directly:
uv sync --extra dev --no-install-project

The source workflow installs dependencies without installing the repository as an editable package. This prevents development commits after a numbered .devN tag from being interpreted as a new package version by setuptools_scm. The Make targets run the pipeline modules directly from the working tree.

---

Configuration

Copy the template and fill in your values:

cp env.sh.template env.sh
# edit env.sh with your settings

Key variables:

Variable	Description
STORAGE_BACKEND	local (default) or s3
USE_SECRET_MANAGER	false (default, uses env vars) or true (uses AWS Secrets Manager)
AWS_SECRET_MANAGER_NAME	Secret name in Secrets Manager — only needed if USE_SECRET_MANAGER=true
S3_AWS_WIHP_CONFIG_DIRECTORY	Path to the config directory (local path or S3 prefix)
S3_AWS_WIHP_INPUT_SOURCES_TO_CREATE	Path to the input CSV file with sources to create
S3_AWS_WHIP_READY_FOR_CRAWLING_DIRECTORY	Directory where crawler outputs are written
S3_AWS_STAGING_DIRECTORY	Root directory for staging JSON content files (default staging)
S3_AWS_PROCESSED_DIRECTORY	Root directory for processed CSV output files (default processed)
OVERWRITE_EXISTING	false (default, skip existing CSVs) or true (re-process and overwrite)
WIHP_API_URL	WIHP API base URL
WIHP_TOKEN_URL	WIHP OAuth token endpoint
WIHP_API_KEY	WIHP API key
WIHP_GROUP	WIHP group (e.g. /MNE)

Local storage

When STORAGE_BACKEND=local, paths are local filesystem paths. The local_storage/ directory mirrors the S3 structure:

local_storage/
├── config/                   # input files
├── ready_for_crawling/       # crawler outputs
├── ready_to_export/
├── done/
├── staging/                  # input JSON acquisition files
└── processed/                # extracted CSV output files and _metadata/

The only input file required to run das-setup-crawler:

local_storage/MNE/config/dashelper/sources_to_create.csv

Format (semicolon-separated):

mne_name;wikipedia_url
Airbus;https://en.wikipedia.org/wiki/Airbus
TotalEnergies;https://en.wikipedia.org/wiki/TotalEnergies

---

Running the pipelines

Load your environment variables then run a pipeline:

source env.sh

# 1. Create a crawler from the input CSV
make run-crawler

# 2. Trigger acquisitions for existing crawlers
make run-acq

# 3. Stop running acquisitions
make run-stop

# 4. Extract data from staging JSON files to processed CSV files
make run-extract
# or via CLI entry point:
uv run das-extract-data [--force]

When the published package is installed with uv add das-setup-helper, the equivalent commands are uv run das-setup-crawler, uv run das-create-acq, uv run das-stop-acq, and uv run das-extract-data.

Pipeline order: setup-crawler → create-acq → stop-acq → extract-data

---

Development

# Run tests
make test

# Format code
make lint

# Refresh dependency lock (without upgrading)
make lock

# Upgrade all dependencies
make lock-upgrade

Available make targets:

make info

---

Deployment (prod/staging)

In production (MWAA / Lambda), set:

USE_SECRET_MANAGER=true
AWS_SECRET_MANAGER_NAME=prod/DASHelper
STORAGE_BACKEND=s3

All other variables are loaded automatically from AWS Secrets Manager. The IAM role must have secretsmanager:GetSecretValue permission on the secret.

---

Release

The complete contributor and release-manager workflow is documented in Release process.

The package version is derived from Git tags and follows Python package versioning conventions (PEP 440):

• Development tags use vX.Y.Z.devN.
• Release candidate tags use vX.Y.Z.rcN.
• Final releases use vX.Y.Z.

GitLab CI release flow:

• On development, run tag-dev-patch, tag-dev-minor, tag-dev-major, or tag-dev-same. The resulting development tag is published to the internal GitLab Python registry.
• Merge development into test without squashing, then run tag-rc. The resulting release candidate is published to TestPyPI.
• Merge test into main without squashing, then run tag-release. The final tag is published to PyPI and creates a GitLab Release with notes generated from the commits since the previous final tag.

Versions remain entirely tag-driven through setuptools_scm. Commit messages do not determine the next package version.

Commit convention

Commit subjects and merge request titles follow Conventional Commits:

feat(api): add source filtering
fix: handle an expired token
refactor!: remove the legacy connector

Supported types are feat, fix, perf, refactor, docs, test, build, ci, chore, and revert. Install the local commit-msg check with:

make pre-commit-install

The CI validates merge request titles and commits independently of local hooks. Feature merge requests should be squashed using their Conventional Commit title. Promotion merge requests from development to test and from test to main must not be squashed, so the final release history retains the commits used to generate release notes.

SonarQube runs in the quality stage for merge requests and for the development and main branches.

To allow the release-tag job to push tags from CI, define this GitLab CI/CD variable:

• GIT_PUSH_TOKEN

Also define the repository tokens used for publishing:

• TEST_PYPI_TOKEN
• PYPI_TOKEN

For SonarQube, also define:

• SONAR_TOKEN

License

This project is distributed under the European Union Public Licence v1.2 (EUPL-1.2). The full licence text is available in LICENSE and on the European Commission website: https://commission.europa.eu/about/departments-and-executive-agencies/digital-services/open-source-strategy-history/european-union-public-licence_en