repo-fetcher 1.0.0

Grab any repository, anywhere, anytime.

Repo-Fetcher

Grab any repository, anywhere, anytime.

Repo-Fetcher is a production-ready Python library for downloading repositories from various platforms with maximum flexibility and performance. Designed for developers, DevOps engineers, and automation systems that need reliable, configurable repository downloading.

Features

• Multi-platform support: GitHub, GitLab, Git, npm (placeholder), Bitbucket (placeholder), and generic URLs
• Dual execution modes: Synchronous and asynchronous APIs
• Batch operations: Download multiple repositories in parallel
• Flexible configuration: Extensive configuration options for every use case
• Progress tracking: Real-time progress bars for downloads
• Error handling: Comprehensive error handling with retry mechanisms
• Extensible architecture: Easy to add support for new platforms
• Type hints: Full type annotations for better IDE support
• Zero dependencies: Core functionality works without external dependencies

Table of Contents

• Installation
• Quick Start
• Core Concepts
• API Reference
  • RepoFetcher Class
  • DownloadConfig Model
  • DownloadResult Model
  • Platform Fetchers
  • Exceptions
• Usage Examples
  • Basic Usage
  • Advanced Configuration
  • Batch Operations
  • Async Operations
  • Custom Fetchers
  • CLI Usage
• Platform-Specific Features
  • GitHub
  • GitLab
  • Git
  • Generic
• Error Handling
• Performance Tuning
• Extending Repo-Fetcher
• Testing
• Contributing
• License

Installation

Prerequisites

• Python 3.8 or higher
• Git (for Git clone operations)

Install from PyPI

pip install repo-fetcher

Install from source

git clone https://github.com/neuxdotdev/repo-fetcher.git
cd repo-fetcher
pip install -e .

Optional Dependencies

For enhanced functionality, install optional dependencies:

# For async operations
pip install aiohttp

# For progress bars
pip install tqdm

# For additional archive formats
pip install python-lzma bz2file

Quick Start

from repo_fetcher import RepoFetcher, DownloadConfig

# Create a fetcher instance
fetcher = RepoFetcher()

# Download a GitHub repository
config = DownloadConfig(
    url="https://github.com/python/cpython",
    dest="./downloads/cpython",
    branch="3.12"
)

result = fetcher.download(config)

if result.success:
    print(f" Downloaded {result.url}")
    print(f" Location: {result.extracted_path}")
    print(f" Size: {result.size_bytes:,} bytes")
    print(f" Duration: {result.duration_seconds:.2f} seconds")
else:
    print(f" Failed: {result.error}")

Core Concepts

1. Platforms

Repo-Fetcher supports multiple repository platforms. Each platform has its own optimized fetcher implementation.

2. Configuration

All download behavior is controlled through DownloadConfig - a comprehensive dataclass with sensible defaults.

3. Execution Models

• Synchronous: Blocking operations, simple to use
• Asynchronous: Non-blocking, better for I/O-bound operations
• Batch: Parallel downloads for multiple repositories

4. Results

All operations return DownloadResult objects containing success status, metadata, and any errors.

API Reference

RepoFetcher Class

The main entry point for all download operations.

class RepoFetcher:
    """Main class for downloading repositories from various platforms."""

    def __init__(self, default_config: Optional[Dict[str, Any]] = None):
        """
        Initialize RepoFetcher.

        Args:
            default_config: Default configuration to use for all downloads
        """

    def download(self, config: Union[Dict[str, Any], DownloadConfig]) -> DownloadResult:
        """
        Download repository synchronously.

        Args:
            config: Download configuration

        Returns:
            DownloadResult object
        """

    async def download_async(self, config: Union[Dict[str, Any], DownloadConfig]) -> DownloadResult:
        """
        Download repository asynchronously.

        Args:
            config: Download configuration

        Returns:
            DownloadResult object
        """

    def download_batch(self, configs: list, max_workers: int = 5, show_progress: bool = True) -> list:
        """
        Download multiple repositories in parallel.

        Args:
            configs: List of download configurations
            max_workers: Maximum number of parallel downloads
            show_progress: Show progress bar for batch

        Returns:
            List of DownloadResult objects
        """

    async def download_batch_async(self, configs: list, max_concurrent: int = 5, show_progress: bool = True) -> list:
        """
        Download multiple repositories asynchronously.

        Args:
            configs: List of download configurations
            max_concurrent: Maximum number of concurrent downloads
            show_progress: Show progress bar for batch

        Returns:
            List of DownloadResult objects
        """

    def register_fetcher(self, platform: Platform, fetcher_class):
        """
        Register a custom fetcher for a platform.

        Args:
            platform: Platform identifier
            fetcher_class: Fetcher class (must inherit from BaseFetcher)
        """

DownloadConfig Model

Complete configuration for repository downloads.

@dataclass
class DownloadConfig:
    """Configuration for repository download."""

    # Platform & Source
    platform: Platform = Platform.GITHUB
    url: Optional[str] = None
    token: Optional[str] = None
    ssh_key_path: Optional[Union[str, Path]] = None

    # Branch/Tag/Commit
    branch: Optional[str] = None
    tag: Optional[str] = None
    commit: Optional[str] = None

    # Target/Destination
    dest: Union[str, Path] = "."
    keep_archive: bool = False
    overwrite: bool = False
    mkdirs: bool = True

    # File/Folder selection
    path: Optional[str] = None
    files: Optional[List[str]] = None
    exclude: Optional[List[str]] = None
    flatten: bool = False

    # Format/Compression
    archive_format: ArchiveFormat = ArchiveFormat.TAR_GZ
    compression_level: int = 6

    # Network
    timeout: int = 30
    retries: int = 3
    retry_delay: float = 1.0
    verify_ssl: bool = True
    proxy: Optional[Dict[str, str]] = None

    # Performance/Async
    async_mode: bool = False
    chunk_size: int = 8192
    max_connections: int = 10

    # Logging & Progress
    show_progress: bool = True
    logger: Optional[logging.Logger] = None
    log_level: LogLevel = LogLevel.INFO

    # Post-processing/Hooks
    pre_download_hook: Optional[Callable[[Dict[str, Any]], Any]] = None
    post_download_hook: Optional[Callable[[Dict[str, Any]], Any]] = None
    post_extract_hook: Optional[Callable[[Dict[str, Any]], Any]] = None
    checksum_verify: Optional[str] = None  # 'md5', 'sha1', 'sha256'

    # Error Handling
    ignore_errors: bool = False
    raise_on_error: bool = True
    retry_on_fail: bool = True

    # Extensibility
    custom_fetcher: Optional[Callable] = None
    custom_extractor: Optional[Callable] = None

DownloadResult Model

Result object returned from all download operations.

@dataclass
class DownloadResult:
    """Result of a download operation."""

    success: bool
    platform: Platform
    url: str
    destination: Path
    archive_path: Optional[Path] = None
    extracted_path: Optional[Path] = None
    size_bytes: int = 0
    duration_seconds: float = 0.0
    error: Optional[str] = None
    metadata: Dict[str, Any] = field(default_factory=dict)

Platform Fetchers

BaseFetcher (Abstract Class)

class BaseFetcher(ABC):
    """Abstract base class for all platform fetchers."""

    @abstractmethod
    def get_download_url(self) -> str:
        """Get the download URL for the repository."""

    @abstractmethod
    async def download_async(self) -> DownloadResult:
        """Download repository asynchronously."""

    @abstractmethod
    def download_sync(self) -> DownloadResult:
        """Download repository synchronously."""

    def get_headers(self) -> Dict[str, str]:
        """Get HTTP headers for the request."""

    def validate_config(self) -> None:
        """Validate configuration before download."""

GitHubFetcher

class GitHubFetcher(BaseFetcher):
    """GitHub repository fetcher."""

    def get_download_url(self) -> str:
        """Get GitHub archive download URL."""
        # Returns: https://api.github.com/repos/{owner}/{repo}/tarball/{ref}

GitLabFetcher

class GitLabFetcher(BaseFetcher):
    """GitLab repository fetcher."""

    def get_download_url(self) -> str:
        """Get GitLab archive download URL."""
        # Returns: https://gitlab.com/api/v4/projects/{encoded_project}/repository/archive.tar.gz?sha={ref}

GenericFetcher

class GenericFetcher(BaseFetcher):
    """Generic Git and HTTP fetcher."""

    def get_download_url(self) -> str:
        """Get download URL for generic fetcher."""

Exceptions

class RepoFetcherError(Exception):
    """Base exception for all Repo-Fetcher errors."""

class DownloadError(RepoFetcherError):
    """Raised when download fails."""

class AuthenticationError(RepoFetcherError):
    """Raised when authentication fails."""

class InvalidConfigError(RepoFetcherError):
    """Raised when configuration is invalid."""

class PlatformNotSupportedError(RepoFetcherError):
    """Raised when platform is not supported."""

class ArchiveError(RepoFetcherError):
    """Raised when archive operations fail."""

class NetworkError(RepoFetcherError):
    """Raised when network operations fail."""

class HookError(RepoFetcherError):
    """Raised when hook execution fails."""

Usage Examples

Basic Usage

Download from GitHub

from repo_fetcher import RepoFetcher, DownloadConfig

fetcher = RepoFetcher()
config = DownloadConfig(
    url="https://github.com/python/cpython",
    dest="./downloads/cpython",
    branch="3.12"
)
result = fetcher.download(config)

Download from GitLab

config = DownloadConfig(
    url="https://gitlab.com/gitlab-org/gitlab-docs",
    dest="./downloads/gitlab-docs",
    platform=Platform.GITLAB,
    tag="v16.10.0"
)
result = fetcher.download(config)

Clone Git Repository

config = DownloadConfig(
    url="https://github.com/python/cpython.git",
    dest="./downloads/cpython-clone",
    platform=Platform.GIT,
    branch="3.12"
)
result = fetcher.download(config)

Advanced Configuration

Authentication

# GitHub Personal Access Token
config = DownloadConfig(
    url="https://github.com/owner/private-repo",
    token="ghp_your_token_here"
)

# GitLab Private Token
config = DownloadConfig(
    url="https://gitlab.com/owner/private-repo",
    platform=Platform.GITLAB,
    token="glpat_your_token_here"
)

# SSH Key Authentication
config = DownloadConfig(
    url="[e-mail]:owner/repo.git",
    platform=Platform.GIT,
    ssh_key_path="/path/to/private_key"
)

Custom Archive Format

from repo_fetcher.constants import ArchiveFormat

config = DownloadConfig(
    url="https://github.com/owner/repo",
    archive_format=ArchiveFormat.ZIP,  # Options: TAR_GZ, TAR_BZ2, TAR_XZ, ZIP
    keep_archive=True  # Keep the archive file after extraction
)

Network Configuration

config = DownloadConfig(
    url="https://github.com/owner/repo",
    timeout=60,  # 60 second timeout
    retries=5,   # Retry 5 times on failure
    verify_ssl=False,  # Disable SSL verification (not recommended)
    proxy={
        "http": "http://proxy.example.com:8080",
        "https": "http://proxy.example.com:8080"
    }
)

File Selection

config = DownloadConfig(
    url="https://github.com/owner/repo",
    files=["src/*.py", "README.md"],  # Only download specific files
    exclude=["*.tmp", "test/*"],      # Exclude certain patterns
    flatten=True                      # Flatten directory structure
)

Hooks

def pre_download_hook(context):
    print(f"Starting download of {context['config'].url}")
    # Can modify config if needed

def post_extract_hook(context):
    result = context['result']
    print(f"Download completed: {result.size_bytes} bytes")
    # Can run post-processing scripts

config = DownloadConfig(
    url="https://github.com/owner/repo",
    pre_download_hook=pre_download_hook,
    post_extract_hook=post_extract_hook
)

Batch Operations

Synchronous Batch

configs = [
    DownloadConfig(
        url="https://github.com/pallets/flask",
        dest="./downloads/flask"
    ),
    DownloadConfig(
        url="https://github.com/django/django",
        dest="./downloads/django"
    ),
    DownloadConfig(
        url="https://gitlab.com/gitlab-org/gitlab-docs",
        dest="./downloads/gitlab-docs",
        platform=Platform.GITLAB
    )
]

results = fetcher.download_batch(
    configs,
    max_workers=3,      # Download 3 repositories in parallel
    show_progress=True  # Show progress bar
)

for i, result in enumerate(results):
    status = "" if result.success else ""
    print(f"{status} {result.url}")

Asynchronous Batch

import asyncio

async def download_multiple():
    fetcher = RepoFetcher()
    configs = [...]  # Same as above

    results = await fetcher.download_batch_async(
        configs,
        max_concurrent=3,
        show_progress=True
    )

    return results

# Run in async context
results = asyncio.run(download_multiple())

Batch from JSON File

// downloads.json
[
	{
		"url": "https://github.com/pallets/flask",
		"dest": "./downloads/flask",
		"tag": "3.0.0"
	},
	{
		"url": "https://github.com/django/django",
		"dest": "./downloads/django",
		"branch": "main",
		"platform": "github"
	},
	{
		"url": "https://gitlab.com/gitlab-org/gitlab-docs",
		"dest": "./downloads/gitlab-docs",
		"platform": "gitlab",
		"overwrite": true
	}
]

import json

with open('downloads.json', 'r') as f:
    configs_data = json.load(f)

configs = [DownloadConfig(**data) for data in configs_data]
results = fetcher.download_batch(configs)

Async Operations

Basic Async

import asyncio
from repo_fetcher import RepoFetcher, DownloadConfig

async def async_download():
    fetcher = RepoFetcher()
    config = DownloadConfig(
        url="https://github.com/python/cpython",
        dest="./downloads/cpython"
    )

    result = await fetcher.download_async(config)
    return result

# Run the async function
result = asyncio.run(async_download())

Async with Progress

async def download_with_progress():
    fetcher = RepoFetcher()
    config = DownloadConfig(
        url="https://github.com/python/cpython",
        dest="./downloads/cpython",
        show_progress=True
    )

    result = await fetcher.download_async(config)
    return result

Custom Fetchers

Creating a Custom Fetcher

from repo_fetcher.platforms.base import BaseFetcher
from repo_fetcher.models import DownloadResult, DownloadConfig
from repo_fetcher.constants import Platform
import requests

class CustomPlatformFetcher(BaseFetcher):
    """Custom fetcher for a specific platform."""

    def __init__(self, config: DownloadConfig):
        super().__init__(config)
        # Parse URL or setup custom logic
        self.api_url = self._parse_custom_url()

    def _parse_custom_url(self) -> str:
        """Parse custom platform URL."""
        # Implement your parsing logic
        return f"https://api.custom-platform.com/repo/{self.config.url}"

    def get_download_url(self) -> str:
        """Get download URL for custom platform."""
        return self.api_url

    def download_sync(self) -> DownloadResult:
        """Download repository synchronously."""
        # Implement custom download logic
        response = requests.get(self.get_download_url())
        # Process response
        result = DownloadResult(
            success=True,
            platform=Platform.GENERIC,
            url=self.config.url or "",
            destination=self.config.dest
        )
        return result

    async def download_async(self) -> DownloadResult:
        """Download repository asynchronously."""
        # Implement async logic
        return await asyncio.to_thread(self.download_sync)

# Register the custom fetcher
fetcher = RepoFetcher()
fetcher.register_fetcher(Platform("custom"), CustomPlatformFetcher)

# Use the custom fetcher
config = DownloadConfig(
    url="custom://owner/repo",
    platform="custom"
)
result = fetcher.download(config)

CLI Usage

Repo-Fetcher includes a comprehensive CLI interface.

Basic Commands

# Download GitHub repository
python -m repo_fetcher --url https://github.com/owner/repo --dest ./downloads

# Download specific branch
python -m repo_fetcher --url https://github.com/owner/repo --branch develop

# Download from GitLab
python -m repo_fetcher --url https://gitlab.com/owner/repo --platform gitlab

# Clone Git repository
python -m repo_fetcher --url https://github.com/owner/repo.git --platform git

# Batch download from config file
python -m repo_fetcher --batch downloads.json

# Output in JSON format
python -m repo_fetcher --url https://github.com/owner/repo --output json

# Show examples
python -m repo_fetcher --example

# Show version
python -m repo_fetcher --version

CLI Help

python -m repo_fetcher --help

Platform-Specific Features

GitHub

API Rate Limiting

GitHub API has rate limits. Authenticated requests have higher limits.

# Use token for higher rate limits
config = DownloadConfig(
    url="https://github.com/owner/repo",
    token="your_github_token_here"
)

Repository References

# By branch
config = DownloadConfig(url="...", branch="main")

# By tag
config = DownloadConfig(url="...", tag="v1.0.0")

# By commit hash
config = DownloadConfig(url="...", commit="a1b2c3d4e5f6")

GitLab

Project Path Encoding

GitLab API requires URL-encoded project paths.

# URL: https://gitlab.com/group/subgroup/project
# API: https://gitlab.com/api/v4/projects/group%2Fsubgroup%2Fproject/...

Private Repositories

config = DownloadConfig(
    url="https://gitlab.com/owner/private-repo",
    platform=Platform.GITLAB,
    token="glpat_your_token_here"  # GitLab personal access token
)

Git

SSH Authentication

config = DownloadConfig(
    url="[e-mail]:owner/repo.git",
    platform=Platform.GIT,
    ssh_key_path="/path/to/ssh/key"
)

Branch/Tag Selection

# Clone specific branch
config = DownloadConfig(
    url="https://github.com/owner/repo.git",
    platform=Platform.GIT,
    branch="feature-branch"
)

# Clone specific tag
config = DownloadConfig(
    url="https://github.com/owner/repo.git",
    platform=Platform.GIT,
    tag="v1.0.0"
)

Generic

Direct File Downloads

# Download any file via HTTP/HTTPS
config = DownloadConfig(
    url="https://example.com/file.tar.gz",
    platform=Platform.GENERIC,
    dest="./downloads"
)

Archive Detection

Generic fetcher automatically detects Git repositories vs direct files.

Error Handling

Try-Catch Pattern

from repo_fetcher import RepoFetcher, DownloadConfig
from repo_fetcher.exceptions import DownloadError, PlatformNotSupportedError

fetcher = RepoFetcher()
config = DownloadConfig(
    url="https://github.com/owner/repo",
    dest="./downloads"
)

try:
    result = fetcher.download(config)
except DownloadError as e:
    print(f"Download failed: {e}")
except PlatformNotSupportedError as e:
    print(f"Platform not supported: {e}")
except Exception as e:
    print(f"Unexpected error: {e}")

Configuring Error Behavior

config = DownloadConfig(
    url="https://github.com/owner/repo",
    raise_on_error=False,  # Don't raise exceptions
    ignore_errors=True,    # Continue even on errors
    retry_on_fail=True,    # Automatically retry on failure
    retries=3              # Retry up to 3 times
)

result = fetcher.download(config)
if not result.success:
    print(f"Download failed but exceptions suppressed: {result.error}")

Custom Error Handling

def error_handler(context):
    """Custom error handler."""
    error = context.get('error')
    config = context.get('config')
    print(f"Error downloading {config.url}: {error}")
    # Can log to file, send notification, etc.

# Use in batch operations
configs = [...]
results = fetcher.download_batch(configs)

for result in results:
    if not result.success:
        error_handler({'error': result.error, 'config': ...})

Performance Tuning

Connection Pooling

from repo_fetcher import RepoFetcher

# Configure default connection settings
fetcher = RepoFetcher(default_config={
    'timeout': 60,
    'max_connections': 10,
    'chunk_size': 16384  # 16KB chunks
})

Parallel Downloads

# Adjust based on your network and system
config = DownloadConfig(
    url="https://github.com/large/repo",
    max_connections=5,  # More connections for large files
    chunk_size=32768    # 32KB chunks for faster downloads
)

Memory Management

config = DownloadConfig(
    url="https://github.com/large/repo",
    chunk_size=8192,  # Smaller chunks for memory-constrained environments
    keep_archive=False  # Delete archive after extraction to save space
)

Extending Repo-Fetcher

Adding New Platforms

1. Create a new fetcher class:

from repo_fetcher.platforms.base import BaseFetcher
from repo_fetcher.models import DownloadResult

class NewPlatformFetcher(BaseFetcher):
    def get_download_url(self) -> str:
        # Implement URL generation
        pass

    def download_sync(self) -> DownloadResult:
        # Implement synchronous download
        pass

    async def download_async(self) -> DownloadResult:
        # Implement asynchronous download
        pass

2. Register the platform:

from repo_fetcher.constants import Platform

# Add to Platform enum
Platform.NEW_PLATFORM = "new_platform"

# Register the fetcher
fetcher = RepoFetcher()
fetcher.register_fetcher(Platform.NEW_PLATFORM, NewPlatformFetcher)

Custom Extractors

def custom_extractor(archive_path: Path, extract_dir: Path) -> Path:
    """Custom archive extraction logic."""
    # Implement custom extraction
    import custom_archive_lib
    custom_archive_lib.extract(archive_path, extract_dir)
    return extract_dir

config = DownloadConfig(
    url="https://github.com/owner/repo",
    custom_extractor=custom_extractor
)

Hooks System

def pre_download_hook(context: Dict[str, Any]) -> None:
    """Called before download starts."""
    config = context['config']
    print(f"Starting download of {config.url}")

    # Can modify config
    if "test" in config.url:
        context['config'].branch = "test-branch"

def post_extract_hook(context: Dict[str, Any]) -> None:
    """Called after extraction completes."""
    result = context['result']
    config = context['config']

    # Run post-processing
    if config.platform == Platform.GITHUB:
        print(f"GitHub repo extracted: {result.extracted_path}")

config = DownloadConfig(
    url="https://github.com/owner/repo",
    pre_download_hook=pre_download_hook,
    post_extract_hook=post_extract_hook
)

Testing

Unit Tests

import pytest
from repo_fetcher import RepoFetcher, DownloadConfig
from repo_fetcher.exceptions import DownloadError

def test_github_download():
    """Test GitHub repository download."""
    fetcher = RepoFetcher()
    config = DownloadConfig(
        url="https://github.com/python/cpython",
        dest="./test_downloads"
    )

    result = fetcher.download(config)
    assert result.success
    assert result.platform == Platform.GITHUB
    assert result.size_bytes > 0

Integration Tests

@pytest.mark.integration
def test_batch_download():
    """Test batch download functionality."""
    fetcher = RepoFetcher()
    configs = [
        DownloadConfig(url="https://github.com/pallets/flask"),
        DownloadConfig(url="https://github.com/django/django")
    ]

    results = fetcher.download_batch(configs, max_workers=2)
    assert len(results) == 2
    assert all(r.success for r in results)

Mock Testing

from unittest.mock import Mock, patch

def test_with_mocks():
    """Test with mocked HTTP responses."""
    with patch('requests.get') as mock_get:
        mock_response = Mock()
        mock_response.status_code = 200
        mock_response.headers = {'content-length': '1024'}
        mock_response.iter_content.return_value = [b'data']
        mock_get.return_value = mock_response

        fetcher = RepoFetcher()
        config = DownloadConfig(url="https://example.com/repo")
        result = fetcher.download(config)

        assert result.success

Contributing

We welcome contributions! Here's how to get started:

Development Setup

# Fork and clone the repository
git clone https://github.com/your-username/repo-fetcher.git
cd repo-fetcher

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install development dependencies
pip install -e ".[dev]"

Code Style

# Format code
black repo_fetcher/ tests/

# Sort imports
isort repo_fetcher/ tests/

# Check types
mypy repo_fetcher/

# Lint code
flake8 repo_fetcher/ tests/

Pull Request Process

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add/update tests
5. Update documentation
6. Run tests and linting
7. Submit pull request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions
• Email: [e-mail]

Acknowledgements

• Thanks to all contributors who have helped shape Repo-Fetcher
• Inspired by various package managers and download utilities
• Built with by the Neux team

---

Repo-Fetcher - Grab any repository, anywhere, anytime.

Benchmarks

Download Speed Comparison

# Test with different configurations
python benchmarks/speed_test.py

Memory Usage

# Monitor memory usage during downloads
python benchmarks/memory_test.py

Best Practices

1. Use Context Managers

from repo_fetcher import RepoFetcher

with RepoFetcher() as fetcher:
    results = fetcher.download_batch(configs)

2. Configure Logging

import logging
from repo_fetcher import RepoFetcher, DownloadConfig
from repo_fetcher.constants import LogLevel

logging.basicConfig(level=logging.INFO)
config = DownloadConfig(
    url="...",
    log_level=LogLevel.DEBUG,
    logger=logging.getLogger("my_app")
)

3. Handle Large Files

config = DownloadConfig(
    url="...",
    chunk_size=65536,  # 64KB chunks for large files
    show_progress=True,
    retry_on_fail=True,
    retries=5
)

4. Use Async for I/O Bound Operations

import asyncio

async def download_multiple_repos(urls):
    fetcher = RepoFetcher()
    tasks = []

    for url in urls:
        config = DownloadConfig(url=url)
        tasks.append(fetcher.download_async(config))

    return await asyncio.gather(*tasks, return_exceptions=True)

Roadmap

Planned Features

• npm registry support - Full npm package downloading
• Bitbucket support - Complete Bitbucket integration
• Docker registry - Download Docker images and manifests
• Resumable downloads - Resume interrupted downloads
• Checksum verification - Automatic hash verification
• Plugin system - Extensible plugin architecture
• Web UI - Graphical interface for downloads
• REST API - HTTP API for remote operations

In Progress

• GitHub support - Complete GitHub integration
• GitLab support - Complete GitLab integration
• Git support - Direct Git clone operations
• Generic HTTP - Download any HTTP/HTTPS resource
• Batch operations - Parallel downloads
• Async support - Asynchronous operations
• Comprehensive CLI - Command-line interface

---

**Happy fetching! **

If you find Repo-Fetcher useful, please consider:

• Giving it a on GitHub
• Reporting any issues you encounter
• Contributing to the codebase
• Sharing it with your colleagues