XER-Technologies-metadata-extractor 0.2.2

Internal data extraction utilities

XER Technologies Metadata Extractor

A Python package for extracting metadata from CSV and binary files with memory-efficient processing for large files (100MB+).

Installation

pip install XER_Technologies_metadata_extractor

Usage

Local Files

from XER_Technologies_metadata_extractor import LocalFileAdapter

adapter = LocalFileAdapter("data/dir")
results = adapter.process_metadata(csv_file="file.csv", bin_file="file.bin")

# For large files
results = adapter.process_large_csv_chunked("large_file.csv", chunk_size=10000)

S3 Storage

from XER_Technologies_metadata_extractor import S3Adapter

adapter = S3Adapter(s3_client, "bucket-name")
results = adapter.process_metadata(csv_object_key="file.csv")

Direct Data Processing

from XER_Technologies_metadata_extractor import MetadataExtractor

extractor = MetadataExtractor()
with open("file.csv", "rb") as f:
    results = extractor.process_metadata(csv_data=f, csv_filename="file.csv")

Development

git clone <repo-url>
cd XERMetaDataExtractor
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate
pip install -e ".[dev]"

Quality Checks

pytest
black .
mypy .
ruff check .

Build Configuration

This package uses pyproject.toml for modern Python packaging standards. The configuration includes:

Project Metadata

• Name: XER_Technologies_metadata_extractor
• Version: 0.1.0
• Python Requirements: >=3.8
• Dependencies: pandas>=2.0.0

Development Dependencies

The [project.optional-dependencies] section defines development tools:

• Testing: pytest, pytest-cov
• Code Formatting: black, isort
• Linting: ruff, mypy

Build System

Uses setuptools with automatic package discovery from the src/ directory.

Installing from Other Repositories

From Git Repository (HTTPS)

pip install git+https://github.com/your-org/XERMetaDataExtractor.git

From Git Repository (SSH)

pip install git+ssh://[e-mail]/your-org/XERMetaDataExtractor.git

From Specific Branch/Tag

pip install git+https://github.com/your-org/XERMetaDataExtractor.git@main
pip install git+https://github.com/your-org/XERMetaDataExtractor.git@v0.1.0

From Local Repository

pip install /path/to/XERMetaDataExtractor

Editable Install from Repository

For development across multiple projects:

git clone https://github.com/your-org/XERMetaDataExtractor.git
pip install -e ./XERMetaDataExtractor

Building Distribution Packages

# Install build tools
pip install build

# Build wheel and source distribution
python -m build

# Install from built wheel
pip install dist/XER_Technologies_metadata_extractor-0.1.0-py3-none-any.whl

Private Repository Access

For private repositories, ensure proper authentication:

# Using personal access token
pip install git+https://token:[e-mail]/your-org/XERMetaDataExtractor.git

# Using SSH key (recommended for private repos)
pip install git+ssh://[e-mail]/your-org/XERMetaDataExtractor.git

To tag a new version and publish to pypi

git tag -a vwhatsinyourtoml -m "Release 0.1.2"
git push origin vwhatsinyourtoml