pyeurostat 1.0.0

Modern Python client for EuroStat data using SDMX API

pyeurostat

Modern Python client for EuroStat data - Clean SDMX API implementation that fixes the flags bug in the original library.

Features

• Standalone Implementation: Direct SDMX 2.1 API client (no buggy eurostat dependency)
• Fixes Critical Bug: Proper flag handling - no more "column mismatch" errors
• Simple API: Easy-to-use EuroStatExplorer class for data access
• Loop Downloads: Built-in support for downloading multiple parameter values
• Tested: Comprehensive test suite with 100% passing tests

Installation

From PyPI (recommended):

pip install pyeurostat

From source:

git clone https://github.com/Onerafaz/pyeurostat.git
cd pyeurostat
pip install -e .

Dependencies:

• pandas >= 1.3.0
• plotly >= 5.0.0
• requests >= 2.25.0

Quick Start

from pyeurostat import EuroStatExplorer

# Initialize the explorer
es = EuroStatExplorer()

# Download unemployment data for Portugal
df = es.client.get_data_df(
    "une_rt_a",  # Unemployment dataset
    filter_pars={
        'geo': 'PT',
        'age': 'TOTAL',
        'sex': 'T',
        'unit': 'PC_ACT',
        'startPeriod': '2020',
        'endPeriod': '2023'
    },
    flags=True,  # Now works correctly!
    verbose=True
)

print(f"Downloaded: {df.shape}")
print(df.head())

# Download multiple countries using loop
df = es.download_with_filters(
    "une_rt_a",
    filter_pars={
        'age': 'TOTAL',
        'sex': 'T',
        'unit': 'PC_ACT',
        'startPeriod': '2020',
        'endPeriod': '2023'
    },
    loop_param='geo',
    loop_values=['PT', 'ES', 'IT', 'FR', 'DE'],
    flags=True
)

The Flags Bug Fix

The original eurostat library has a critical bug where flags=True causes:

ValueError: 270 columns passed, passed data had 269 columns

This happens because the library creates headers with _value and _flag suffixes but the data rows don't always include flag values, causing a column mismatch.

pyeurostat fixes this by:

1. Direct SDMX API implementation (no dependency on buggy library)
2. Proper row padding to match header length
3. Correct TSV parsing with flag handling

Comparison

Original library (FAILS):

import eurostat
df = eurostat.get_data_df("une_rt_a", flags=True, filter_pars={...})
# Error: 270 columns passed, passed data had 269 columns

pyeurostat (WORKS):

from pyeurostat import EuroStatExplorer
es = EuroStatExplorer()
df = es.download_with_filters("une_rt_a", flags=True, ...)
# Success: Proper flag handling built-in

API Reference

EuroStatExplorer

Main class for interacting with EuroStat data.

from pyeurostat import EuroStatExplorer
es = EuroStatExplorer()

Methods

download_with_filters(dataset_code, filter_pars, loop_param=None, loop_values=None, flags=False, verbose=True)

Download dataset with custom filters and optional looping.

Parameters:

• dataset_code (str): EuroStat dataset code (e.g., 'une_rt_a')
• filter_pars (dict): Filter parameters (e.g., {'geo': 'PT', 'startPeriod': '2020'})
• loop_param (str, optional): Parameter to loop through (e.g., 'geo')
• loop_values (list, optional): Values to loop (e.g., ['PT', 'ES', 'IT'])
• flags (bool): Include data quality flags (default: False)
• verbose (bool): Print progress messages (default: True)

Returns: pandas DataFrame

get_available_filters()

Get all available filter options for the loaded dataset.

filters = es.get_available_filters()
print(filters)

unpivot_data(df, time_pattern=r'^\d{4}')

Convert wide-format data to long format (unpivot/melt).

SDMXClient (Low-level)

Direct SDMX API client for advanced usage.

from pyeurostat.sdmx_client import SDMXClient

client = SDMXClient()
data = client.get_data("une_rt_a", filter_pars={...}, flags=True)
df = client.get_data_df("une_rt_a", filter_pars={...})

Examples

Example 1: Simple Download

from pyeurostat import EuroStatExplorer

es = EuroStatExplorer()
df = es.client.get_data_df(
    "une_rt_a",
    filter_pars={
        'geo': 'PT',
        'age': 'TOTAL',
        'startPeriod': '2023'
    }
)

Example 2: Download with Flags

df = es.client.get_data_df(
    "une_rt_a",
    filter_pars={'geo': 'PT'},
    flags=True  # Get data quality flags
)

# Check flags
print(df[[col for col in df.columns if '_flag' in col]])

Example 3: Loop Through Multiple Values

df = es.download_with_filters(
    "une_rt_a",
    filter_pars={
        'age': 'TOTAL',
        'sex': 'T',
        'unit': 'PC_ACT'
    },
    loop_param='geo',
    loop_values=['PT', 'ES', 'IT', 'FR', 'DE']
)

Example 4: Time Period Formats

# Annual data (use year format)
df = es.client.get_data_df("une_rt_a", filter_pars={'startPeriod': '2020', 'endPeriod': '2023'})

# Monthly data (use YYYY-MM format)
df = es.client.get_data_df("monthly_dataset", filter_pars={'startPeriod': '2020-01', 'endPeriod': '2020-12'})

# Quarterly data (use YYYY-Q# format)
df = es.client.get_data_df("quarterly_dataset", filter_pars={'startPeriod': '2020-Q1', 'endPeriod': '2020-Q4'})

Migration from Original Library

Replace this:

import eurostat
df = eurostat.get_data_df("une_rt_a", flags=True)

With this:

from pyeurostat import EuroStatExplorer
es = EuroStatExplorer()
df = es.client.get_data_df("une_rt_a", flags=True)

Testing

The package includes a comprehensive test suite:

python test_comprehensive.py

Test coverage:

• Package initialization
• Simple dataset download
• Download with flags (the critical bug fix!)
• Loop downloads (multiple parameter values)
• Parameter retrieval
• Filter extraction

Contributing

Contributions are welcome! This package aims to provide a more robust and maintained alternative to the original eurostat library.

Development Setup

git clone https://github.com/Onerafaz/pyeurostat.git
cd pyeurostat
pip install -e ".[dev]"

Reporting Issues

Found a bug or have a feature request? Please open an issue on GitHub.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• Inspired by the original eurostat library
• Built on EuroStat's SDMX 2.1 API
• Thanks to the open-source community

Resources

• EuroStat Website
• EuroStat Data Browser
• SDMX Standards
• Original eurostat library (for reference)

Important Notes

Dataset Codes

Find dataset codes at the EuroStat Data Browser:

• une_rt_a - Unemployment rates (annual)
• nama_10_gdp - GDP and main components
• DS-059341 - International trade data

Time Period Formats

Use appropriate time formats based on data frequency:

• Annual: "2023" or "2020"
• Monthly: "2023-01" or "2020-12"
• Quarterly: "2023-Q1" or "2020-Q4"

Flag Values

Common flag values in EuroStat data:

• None or empty: Normal value
• p: Provisional
• e: Estimated
• b: Break in time series
• u: Low reliability
• c: Confidential

Roadmap

• Direct SDMX API implementation
• Fix flags bug
• Comprehensive testing
• PyPI publication
• Async download support
• Caching mechanism
• More visualization methods
• Documentation website