Exoskeleton

pypi version Supported Python Versions Build Last commit

Machine Learning and other applications make it necessary to download thousands or sometimes hundreds of thousands of files.

Using a high-speed-connection carries the risk to run an involuntary denial-of-service attack on the servers that provide those files and webpages.

Exoskeleton is a Python framework that helps you build a crawler / scraper that avoids too high loads on the connection and instead runs permanently and fault tolerant to ultimately download all files.

Its main functionalities are:

Managing the download queue and document data within a MariaDB database.
Avoid processing the same URL more than once.
Working through the queue by either
- downloading files to disk,
- storing the page source code into a database table,
- storing the page text,
- or making PDF-copies of webpages.
Managing already downloaded files:
- Storing multiple versions of a specific file.
- Assigning labels to downloads, so they can be found and grouped easily.
Sending progress reports to the admin.

Documentation

How To Use Exoskeleton

Example Uses

Downloading an Archive : A quite complex use case requiring some custom SQL. This is the actual project that triggered the development of exoskeleton.

Technical Documentation

Example

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import logging

import exoskeleton

logging.basicConfig(level=logging.DEBUG)

# Create a bot
# exoskeleton makes reasonable assumptions about
# parameters left out, like:
# - host = localhost
# - port = 3306 (MariaDB standard)
# - ...
exo = exoskeleton.Exoskeleton(
    project_name='Bot',
    database_settings={'database': 'exoskeleton',
                       'username': 'exoskeleton',
                       'passphrase': ''},
    # True, to stop after the queue is empty, Otherwise it will
    # look consistently for new tasks in the queue:
    bot_behavior={'stop_if_queue_empty': True},
    filename_prefix='bot_',
    chrome_name='chromium-browser',
    target_directory='/home/myusername/myBot/'
)

exo.add_file_download('https://www.ruediger-voigt.eu/examplefile.txt')
# => Will be saved in the target directory. The filename will be the
#    chosen prefix followed by the database id and .txt.

exo.add_file_download(
    'https://www.ruediger-voigt.eu/examplefile.txt',
    {'example-label', 'foo'})
# => Duplicate will be recognized and not added to the queue,
#    but the labels will be associated with the file in the
#    database.


exo.add_file_download(
    'https://www.ruediger-voigt.eu/file_does_not_exist.pdf')
# => Nonexistent file: will be marked, but will not stop the bot.

# Save a page's code into the database:
exo.add_save_page_code('https://www.ruediger-voigt.eu/')

# Use chromium or Google chrome to generate a PDF of the website:
exo.add_page_to_pdf('https://github.com/RuedigerVoigt/exoskeleton')

# work through the queue:
exo.process_queue()

Metadata

Release files for exoskeleton 2.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for exoskeleton 2.1.1
File	Size	Uploaded	Provenance
exoskeleton-2.1.1.tar.gz	28.7 kB	Apr 27, 2022	Without provenance	Details

Built distribution (wheel)

Table of built distributions (wheels) for exoskeleton 2.1.1
File	Interpreter	ABI	Platform	Reset
exoskeleton-2.1.1-py3-none-any.whl 40.9 kB Apr 27, 2022 Without provenance	Python 3	none	any	Details

Total release size: 69.6 kB

Release files / exoskeleton-2.1.1.tar.gz

Download URL	exoskeleton-2.1.1.tar.gz
Size	28.7 kB
Tags	Source
SHA-256 checksum How to use checksums	`b9ebca371be6ed368a460d0ecb2b692f8bfc73a4434aac0276c57500259f6d21`
BLAKE2b-256 checksum How to use checksums	`f4b7824212638aba455b5a661d577adf2b3b901b9549bb7b93f3fa5d0d334199`
Upload date	Apr 27, 2022
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.0 CPython/3.8.10`

Release files / exoskeleton-2.1.1-py3-none-any.whl

Download URL	exoskeleton-2.1.1-py3-none-any.whl
Size	40.9 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`1c84298fa2ee2a0b2c63d3cfe384dcc4fa62e156bb42ce9288ffa7fe81dd9c44`
BLAKE2b-256 checksum How to use checksums	`9f5298e510b3fc9fdba27ee347419a047ea55b7d52755590abd956772857de5d`
Upload date	Apr 27, 2022
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.0 CPython/3.8.10`

Security for exoskeleton 2.1.1

Report project as malware

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.