internet-archive-extractor 0.0.4

Tool for extracting archived web sites from the Internet Archive saving as WARC files.

The project currently only supports converting output from Bitdruids WaybackUp program to be converted to WARC files.

WaybackupToWarc

This project is designed to read a CSV file containing URLs, remove any instances of port 80 from those URLs, and generate WARC-GZ files based on the cleaned data. The CSV file can be constructed by using the following tool: Python Wayback Machine Downloader

Project Structure

WaybackupToWarc
├── src
│   ├── main.py        # Main script for processing CSV and generating WARC files
│   └── utils.py       # Utility functions for reading CSV and modifying URLs
├── requirements.txt    # List of dependencies for the project
└── README.md           # Documentation for the project

Requirements

To run this project, you need to install the following dependencies:

• warcio: For creating WARC files.
• pandas: For handling CSV data.

You can install the required packages using pip:

pip install -r requirements.txt

Usage

1. Place your CSV file in the appropriate directory.
2. Update the src/main.py file to specify the path to your CSV file.
3. Run the main script:

python src/main.py

This will process the CSV file, remove port 80 from the URLs, and generate the corresponding WARC-GZ files.

Contributing

Feel free to submit issues or pull requests if you have suggestions or improvements for the project.