host-image-backup 0.1.3

A Python-based command-line tool for backing up images from multiple image hosting services to local storage

Host Image Backup

English | 简体中文

Host Image Backup is a modular Python CLI tool for backing up images from various image hosting services to your local machine with ease.

Features

• Modular architecture: Easily extend with new providers
• Multi-provider support: Aliyun OSS, Tencent COS, SM.MS, Imgur, GitHub
• Visual progress: Progress bars for backup operations
• Rich CLI interface: Intuitive command-line experience
• Flexible configuration: YAML-based config management
• Resume support: Continue interrupted transfers
• Comprehensive logging: Detailed operation logs
• Well tested: Reliable with test coverage
• Upload support: Upload images to cloud storage
• Metadata management: Track backup operations and files
• Batch operations: Upload multiple files at once
• Duplicate detection: Find and manage duplicate files
• Image compression: High fidelity compression with quality control

---

Supported Providers

Provider	Features	Notes
OSS	List, backup, upload, delete, file info	Requires Aliyun credentials
COS	List, backup, upload, delete, file info	Requires Tencent credentials
SM.MS	List, backup	Public API, rate limits apply
Imgur	List, backup	Requires Imgur client ID/secret
GitHub	List, backup	Requires GitHub token & access

---

Installation

Requirements:

• Python 3.10 or newer
• pip or uv package manager

Install from PyPI:

pip install host-image-backup
pip install --upgrade host-image-backup
host-image-backup --help
hib --help

Development install:

git clone https://github.com/WayneXuCN/HostImageBackup.git
cd HostImageBackup
uv lock
uv sync --all-extras
# Or use pip:
pip install -e ".[dev]"

---

Configuration

Quick start:

host-image-backup init
# Edit config file:
# Linux/macOS: ~/.config/host-image-backup/config.yaml
# Windows: %APPDATA%/host-image-backup/config.yaml

Example configuration:

default_output_dir: "./backup"
max_concurrent_downloads: 5
timeout: 30
retry_count: 3
log_level: "INFO"
providers:
  oss:
    enabled: true
    access_key_id: "your_access_key"
    access_key_secret: "your_secret_key"
    bucket: "your_bucket_name"
    endpoint: "oss-cn-hangzhou.aliyuncs.com"
    prefix: "images/"
  cos:
    enabled: false
    secret_id: "your_secret_id"
    secret_key: "your_secret_key"
    bucket: "your_bucket_name"
    region: "ap-guangzhou"
    prefix: "images/"
  sms:
    enabled: false
    api_token: "your_api_token"
  imgur:
    enabled: false
    client_id: "your_client_id"
    client_secret: "your_client_secret"
    access_token: "your_access_token"
    refresh_token: "your_refresh_token"
  github:
    enabled: false
    token: "ghp_your_personal_access_token"
    owner: "your_username"
    repo: "your_repository"
    path: "images"

Configuration fields:

Field	Description	Required	Default
default_output_dir	Backup directory	No	"./backup"
max_concurrent_downloads	Parallel downloads	No	5
timeout	Request timeout (seconds)	No	30
retry_count	Retry attempts	No	3
log_level	Logging level	No	"INFO"
access_key_id	Aliyun OSS access key ID	Yes	-
access_key_secret	Aliyun OSS access key secret	Yes	-
bucket	OSS/COS bucket name	Yes	-
endpoint	OSS endpoint URL	Yes	-
region	COS region	Yes	-
secret_id	Tencent COS secret ID	Yes	-
secret_key	Tencent COS secret key	Yes	-
api_token	SM.MS API token	Yes	-
client_id	Imgur client ID	Yes	-
client_secret	Imgur client secret	Yes	-
access_token	Imgur access token	Yes	-
refresh_token	Imgur refresh token	No	-
token	GitHub token	Yes	-
owner	GitHub repo owner	Yes	-
repo	GitHub repo name	Yes	-
path	Folder path in repo	No	""

---

CLI Usage

Quick start:

host-image-backup config init
hib config init
host-image-backup provider test oss
hib provider test oss
host-image-backup provider list
hib provider list
host-image-backup backup start oss --output ./my-backup
hib backup start oss --output ./my-backup
host-image-backup backup all --output ./full-backup
hib backup all --output ./full-backup

Command groups:

Group	Command	Description
Config	config init	Initialize configuration file
Provider	provider list	List providers
	provider test	Test provider connection
	provider info	Show provider info
Backup	backup start	Backup from specific provider
	backup all	Backup from all enabled providers
Upload	upload file	Upload single image
	upload directory	Upload multiple images
Data	data stats	Show backup statistics
	data history	Show backup history
	data duplicates	Find duplicate files
	data cleanup	Clean up backup files
	data verify	Verify backup file integrity
	data compress	Compress images

Examples:

host-image-backup backup start oss
hib backup start oss
host-image-backup backup start oss --output ~/Pictures/backup --limit 100
hib backup start oss --output ~/Pictures/backup --limit 100
host-image-backup backup start imgur --config ./my-config.yaml --verbose
hib backup start imgur --config ./my-config.yaml --verbose
host-image-backup backup start github --no-skip-existing
hib backup start github --no-skip-existing
host-image-backup backup all --output ~/backup --limit 50 --verbose
hib backup all --output ~/backup --limit 50 --verbose

Global options:

-c, --config PATH    Custom config file path
-v, --verbose        Enable verbose logging
--help               Show help message

---

Use Cases

• Backup and migrate images from cloud providers to local storage
• Aggregate images from multiple services
• Automate scheduled backups (cron jobs, CI/CD)
• Archive management for organized local image storage
• Disaster recovery: maintain offline copies

Examples:

host-image-backup backup all --output ~/PhotoBackup
hib backup all --output ~/PhotoBackup

# Scheduled backup (cron)
0 2 * * * /usr/local/bin/host-image-backup backup all --output /backup/images --limit 100
0 2 * * * /usr/local/bin/hib backup all --output /backup/images --limit 100

# Migration between providers
host-image-backup backup start old-provider --output ./migration-temp
hib backup start old-provider --output ./migration-temp

---

Troubleshooting

Authentication errors:

• Check credentials and config file format
• Verify token expiration and permissions
• Test providers: host-image-backup provider test <provider>

Network issues:

• Check internet connection
• Increase timeout in config
• Use --verbose for details
• Check provider service status

File system errors:

• Ensure output directory exists and is writable

mkdir -p ~/backup && chmod 755 ~/backup
chmod 600 ~/.config/host-image-backup/config.yaml

Rate limiting:

• Lower max_concurrent_downloads
• Add delays between requests
• Use --limit to control download volume

Debugging:

host-image-backup provider test oss --verbose
hib provider test oss --verbose
host-image-backup provider info github
hib provider info github
host-image-backup backup start imgur --verbose --limit 5
hib backup start imgur --verbose --limit 5

Log analysis:

tail -f logs/host_image_backup_*.log
grep -i error logs/host_image_backup_*.log
grep -E "(Successfully|Failed)" logs/host_image_backup_*.log

---

Security

• Never commit credentials to version control
• Use environment variables for sensitive data
• Set restrictive permissions on config files:

chmod 600 ~/.config/host-image-backup/config.yaml

Environment variable example:

export OSS_ACCESS_KEY_ID="your_key"
export OSS_ACCESS_KEY_SECRET="your_secret"
export GITHUB_TOKEN="ghp_your_token"

Reference in config:

providers:
  oss:
    access_key_id: "${OSS_ACCESS_KEY_ID}"
    access_key_secret: "${OSS_ACCESS_KEY_SECRET}"

• Use HTTPS endpoints
• Consider VPN/private networks for sensitive data

---

Development

Setup:

git clone https://github.com/WayneXuCN/HostImageBackup.git
cd HostImageBackup
python -m venv .venv
source .venv/bin/activate  # Linux/macOS
# .venv\Scripts\activate   # Windows
uv lock
uv sync --all-extras
pre-commit install

Running:

uv run -m host_image_backup.cli

Testing:

pytest
pytest --cov=src/host_image_backup
pytest tests/test_config.py
pytest -v

Code quality:

ruff format src tests
mypy src
ruff check src tests
make lint

Add new providers:

1. Create provider class in src/host_image_backup/providers/
2. Implement required methods from BaseProvider
3. Add config class in src/host_image_backup/config/config_models.py
4. Update provider registry in src/host_image_backup/providers/provider_manager.py
5. Add tests
6. Update documentation

---

Roadmap

Planned features:

• Enhanced error handling
• Configuration validation
• Progress persistence for interrupted backups
• Performance optimization

Future versions:

• Web UI for configuration and monitoring
• Advanced filtering (date, type, size)
• Cloud-to-cloud transfers
• Incremental backups

Additional providers:

• Cloudinary
• AWS S3
• Google Drive
• Dropbox
• OneDrive

---

Contributing

Contributions are welcome!

• Report bugs and request features
• Improve documentation and examples
• Add or enhance providers
• Write tests and improve coverage
• Improve CLI and user experience

---

License

MIT License. See LICENSE for details.