pyspfileuploader 1.0.5

A versatile Python package for uploading files to multiple storage backends including local, S3, Azure ADLS, and Blob storage. Supports ZIP file extraction and multiple file uploads.

File Uploader

A Python package for seamlessly uploading files to various storage backends (Local, S3, Azure Blob, ADLS) with job tracking, error logging, and comprehensive file management capabilities.

Features

• Multiple storage backend support:
  • Local filesystem
  • AWS S3
  • Azure Blob Storage
  • Azure Data Lake Storage (ADLS)
• Robust job tracking with SQLite database
• Detailed error logging and exception handling
• Intelligent folder structure with date-based organization
• Advanced ZIP file support with safety checks:
  • File size validation (100MB per file limit)
  • Total uncompressed size limit (200MB)
  • Zip file integrity verification
• Custom date handling for file organization
• Comprehensive error handling and custom exceptions
• Support for file metadata tracking

Installation

pip install pyspfileuploader

Configuration

The package uses environment variables for configuration. Create a .env file with the required variables based on your chosen storage type:

Local Storage

STORAGE_TYPE=local
BASE_PATH=/path/to/storage

AWS S3

STORAGE_TYPE=s3
AWS_BUCKET_NAME=your-bucket
AWS_ACCESS_KEY_ID=your-access-key
AWS_SECRET_ACCESS_KEY=your-secret-key
REGION_NAME=us-east-1  # optional

Azure Blob Storage

STORAGE_TYPE=blob
BLOB_CONNECTION_STRING=your-connection-string
BLOB_CONTAINER_NAME=your-container

Azure Data Lake Storage

STORAGE_TYPE=adls
ADLS_ACCOUNT_NAME=your-account
ADLS_FILE_SYSTEM_NAME=your-filesystem
ADLS_CREDENTIAL=your-credential

Usage

Command Line

pyspfileuploader /path/to/file [job_id] [storage_type] [created_date]

Python API

from file_uploader import upload_file, display_job_details

# Simple upload with automatic job ID generation
job_id = upload_file("/path/to/file")

# Upload with custom job ID and storage type
job_id = upload_file("/path/to/file", job_id="CUSTOM_JOB_ID_001", storage_type="s3")

# Upload with specific created date
job_id = upload_file("/path/to/file", created_date="14-07-2025")

# Upload ZIP file (will be automatically extracted and validated)
job_id = upload_file("/path/to/archive.zip")

# Get status for a given file upload
# Default output format set to JSON mode
display_job_details(job_id=job_id, output_format='dataframe')

Storage Backend Selection

The storage backend is determined by the STORAGE_TYPE environment variable. If not specified, it defaults to "local" storage. You can also override the storage type per upload using the storage_type parameter.

File Size Limits

• Individual file size limit: 100MB
• Total uncompressed size limit for ZIP files: 200MB

Job Tracking

Each upload operation is assigned a unique job ID, which can be used to:

• Track the upload status
• Retrieve file metadata
• Access uploaded file locations

Development

1. Clone the repository
2. Install poetry: pip install poetry
3. Install dependencies: poetry install
4. Run tests: poetry run pytest

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Run tests
5. Submit a pull request

License

MIT License