theloop 0.1.0

Upload files from URLs to cloud storage

TheLoop

A Python CLI tool for uploading files from URLs directly to cloud storage. Currently supports Google Cloud Storage (GCP) with a modular architecture designed for easy extension to other cloud providers.

Features

• 🚀 Upload files directly from URLs to cloud storage
• ☁️ Google Cloud Storage support
• 📦 Streaming uploads with progress tracking
• ⚙️ Configurable settings and logging
• 🔧 Extensible architecture for additional cloud providers
• 🛡️ Async/await for efficient stream processing

Installation

Prerequisites

• Python 3.11 or higher
• Google Cloud Storage credentials (for GCP uploads)

Install from PyPI

pip install theloop

Install from Source

git clone https://github.com/matterai/theloop.git
cd theloop
pip install -e .

Development Installation

git clone https://github.com/matterai/theloop.git
cd theloop
uv pip install -e .
uv pip install pytest pytest-cov pytest-asyncio

Usage

Basic Upload

Upload a file from a URL to Google Cloud Storage:

theloop upload "https://example.com/file.pdf" "my-bucket" "uploads/file.pdf"

Advanced Usage

Using Service Account Credentials

theloop upload \
  "https://example.com/file.pdf" \
  "my-bucket" \
  "uploads/file.pdf" \
  --credentials ~/path/to/service-account.json \
  --project my-gcp-project-id

Specify Cloud Provider

theloop upload \
  "https://example.com/file.pdf" \
  "my-bucket" \
  "uploads/file.pdf" \
  --provider gcp

Configuration

View current settings:

theloop settings

Settings are stored in ~/.theloop/settings.json and include:

• Logging configuration
• Chunk size for uploads
• Default cloud provider settings

Authentication

Google Cloud Platform

1. Application Default Credentials (recommended for local development):

   gcloud auth application-default login
   

2. Service Account Key:

   theloop upload <url> <bucket> <path> --credentials /path/to/service-account.json
   

3. Environment Variable:

   export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account.json"
   theloop upload <url> <bucket> <path>
   

Development

Project Structure

src/theloop/
├── __init__.py
├── cli.py                 # CLI interface using Typer
├── interfaces.py          # Protocol definitions
└── services/
    ├── config_manager.py  # Settings management
    ├── gcp_uploader.py    # GCP-specific upload logic
    └── logging_configurator.py

Setting Up Development Environment

1. Clone the repository:

   git clone https://github.com/matterai/theloop.git
   cd theloop
   

2. Install dependencies:

   uv pip install -e .
   uv pip install pytest pytest-cov pytest-asyncio
   

3. Run tests:

   pytest
   

4. Run from source:

   uv run python -m src.main upload <url> <bucket> <path>
   

Testing

The project uses pytest for testing with support for async code:

# Run all tests
pytest

# Run with coverage
pytest --cov=theloop

# Run specific test file
pytest tests/test_uploader.py

Example test structure:

import pytest

@pytest.mark.asyncio
async def test_async_upload():
    # Test async upload functionality
    pass

Adding a New Cloud Provider

1. Create uploader class in services/:

   from theloop.interfaces import Uploader
   
   class NewProviderUploader(Uploader):
       async def upload_stream_async(self, url: str, bucket: str, path: str) -> None:
           # Implement upload logic
           pass
   

2. Update CLI in cli.py:

   def _get_uploader(provider: str, settings: Settings, ...) -> Uploader:
       if provider == "new-provider":
           return NewProviderUploader(...)
       # ... existing providers
   

Code Style and Standards

• Follow PEP 8 for Python code style
• Use type hints throughout the codebase
• Implement proper error handling with user-friendly messages
• Use async/await for I/O operations
• Follow the existing patterns for configuration and logging

Contributing

We welcome contributions! Please follow these steps:

1. Fork the repository
2. Create a feature branch:

   git checkout -b feature/your-feature-name
   

3. Make your changes following the development guidelines
4. Add tests for new functionality
5. Run tests to ensure everything works:

   pytest
   

6. Commit your changes:

   git commit -m "Add your feature description"
   

7. Push to your fork:

   git push origin feature/your-feature-name
   

8. Create a Pull Request

Contribution Guidelines

• Ensure all tests pass
• Add tests for new features
• Update documentation as needed
• Follow existing code style and patterns
• Write clear commit messages
• Keep pull requests focused and small

License

This project is licensed under the MIT License - see the LICENSE section below for details.

Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions

Roadmap

• AWS S3 support
• Azure Blob Storage support
• Configuration file support
• Batch upload functionality
• Resume interrupted uploads

---

License Text

MIT License

Copyright (c) 2024 Vladimir Vlasiuk

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.