Skip to main content

A library to create a bot / spider / crawler.

Project description

Exoskeleton

pypi version Supported Python Versions Build Last commit Downloads Coverage

Machine Learning and other applications make it necessary to download thousands or sometimes hundreds of thousands of files.

Using a high-speed-connection carries the risk to run an involuntary denial-of-service attack on the servers that provide those files and webpages.

Exoskeleton is a Python framework that helps you build a crawler / scraper that avoids too high loads on the connection and instead runs permanently and fault tolerant to ultimately download all files.

Its main functionalities are:

  • Managing the download queue and document data within a MariaDB database.
  • Avoid processing the same URL more than once.
  • Working through the queue by either
    • downloading files to disk,
    • storing the page source code into a database table,
    • storing the page text,
    • or making PDF-copies of webpages.
  • Managing already downloaded files:
    • Storing multiple versions of a specific file.
    • Assigning labels to downloads, so they can be found and grouped easily.
  • Sending progress reports to the admin.

Documentation

How To Use Exoskeleton

Example Uses

  • Downloading an Archive : A quite complex use case requiring some custom SQL. This is the actual project that triggered the development of exoskeleton.

Technical Documentation

Example

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import logging

import exoskeleton

logging.basicConfig(level=logging.DEBUG)

# Create a bot
# exoskeleton makes reasonable assumptions about
# parameters left out, like:
# - host = localhost
# - port = 3306 (MariaDB standard)
# - ...
exo = exoskeleton.Exoskeleton(
    project_name='Bot',
    database_settings={'database': 'exoskeleton',
                       'username': 'exoskeleton',
                       'passphrase': ''},
    # True, to stop after the queue is empty, Otherwise it will
    # look consistently for new tasks in the queue:
    bot_behavior={'stop_if_queue_empty': True},
    filename_prefix='bot_',
    chrome_name='chromium-browser',
    target_directory='/home/myusername/myBot/'
)

exo.add_file_download('https://www.ruediger-voigt.eu/examplefile.txt')
# => Will be saved in the target directory. The filename will be the
#    chosen prefix followed by the database id and .txt.

exo.add_file_download(
    'https://www.ruediger-voigt.eu/examplefile.txt',
    {'example-label', 'foo'})
# => Duplicate will be recognized and not added to the queue,
#    but the labels will be associated with the file in the
#    database.


exo.add_file_download(
    'https://www.ruediger-voigt.eu/file_does_not_exist.pdf')
# => Nonexistent file: will be marked, but will not stop the bot.

# Save a page's code into the database:
exo.add_save_page_code('https://www.ruediger-voigt.eu/')

# Use chromium or Google chrome to generate a PDF of the website:
exo.add_page_to_pdf('https://github.com/RuedigerVoigt/exoskeleton')

# work through the queue:
exo.process_queue()

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

exoskeleton-2.1.1.tar.gz (28.7 kB view details)

Uploaded Source

Built Distribution

exoskeleton-2.1.1-py3-none-any.whl (40.9 kB view details)

Uploaded Python 3

File details

Details for the file exoskeleton-2.1.1.tar.gz.

File metadata

  • Download URL: exoskeleton-2.1.1.tar.gz
  • Upload date:
  • Size: 28.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.0 CPython/3.8.10

File hashes

Hashes for exoskeleton-2.1.1.tar.gz
Algorithm Hash digest
SHA256 b9ebca371be6ed368a460d0ecb2b692f8bfc73a4434aac0276c57500259f6d21
MD5 f42fe31587875d8f80293f55df64f0e2
BLAKE2b-256 f4b7824212638aba455b5a661d577adf2b3b901b9549bb7b93f3fa5d0d334199

See more details on using hashes here.

File details

Details for the file exoskeleton-2.1.1-py3-none-any.whl.

File metadata

  • Download URL: exoskeleton-2.1.1-py3-none-any.whl
  • Upload date:
  • Size: 40.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.0 CPython/3.8.10

File hashes

Hashes for exoskeleton-2.1.1-py3-none-any.whl
Algorithm Hash digest
SHA256 1c84298fa2ee2a0b2c63d3cfe384dcc4fa62e156bb42ce9288ffa7fe81dd9c44
MD5 f25f23ce2bd2a9a78f874cf91ba2908d
BLAKE2b-256 9f5298e510b3fc9fdba27ee347419a047ea55b7d52755590abd956772857de5d

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page