Memorious

The solitary and lucid spectator of a multiform, instantaneous and almost intolerably precise world.

—Funes the Memorious, Jorge Luis Borges

https://github.com/alephdata/memorious/workflows/memorious/badge.svg

memorious is a light-weight web scraping toolkit. It supports scrapers that collect structured or un-structured data. This includes the following use cases:

  • Make crawlers modular and simple tasks re-usable

  • Provide utility functions to do common tasks such as data storage, HTTP session management

  • Integrate crawlers with the Aleph and FollowTheMoney ecosystem

  • Get out of your way as much as possible

Design

When writing a scraper, you often need to paginate through through an index page, then download an HTML page for each result and finally parse that page and insert or update a record in a database.

memorious handles this by managing a set of crawlers, each of which can be composed of multiple stages. Each stage is implemented using a Python function, which can be re-used across different crawlers.

The basic steps of writing a Memorious crawler:

  1. Make YAML crawler configuration file

  2. Add different stages

  3. Write code for stage operations (optional)

  4. Test, rinse, repeat

Documentation

The documentation for Memorious is available at alephdata.github.io/memorious. Feel free to edit the source files in the docs folder and send pull requests for improvements.

To build the documentation, inside the docs folder run make html

You’ll find the resulting HTML files in /docs/_build/html.

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

3 maintainers

PyPI data
Data sourced directly from PyPI's database.
Avatar for occrp from gravatar.com occrp Avatar for pudo from gravatar.com pudo Avatar for sunu from gravatar.com sunu

Credits

Author: Organized Crime and Corruption Reporting Project

License

MIT

Classifiers

Report project as malware

Release files for memorious 2.6.5

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for memorious 2.6.5
File Size Uploaded
memorious-2.6.5.tar.gz 41.0 kB Details

Built distribution (wheel)

Table of built distributions (wheels) for memorious 2.6.5
File Interpreter ABI Platform
memorious-2.6.5-py2.py3-none-any.whl 52.4 kB Python 3, Python 2 none any Details

Total release size: 93.5 kB

Release files / memorious-2.6.5.tar.gz

Download URL memorious-2.6.5.tar.gz
Size 41.0 kB
Tags Source
SHA-256 checksum
How to use checksums 		5690d32309cc7a269190bd157df7b6a4c9f9f9e896367ea1ba02d483c211e76d
BLAKE2b-256 checksum
How to use checksums 		508597ec7c1f8bdd90f73347b3972a5b6c663f5995e7a49e4cd3f73c46af8510
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/4.0.2 CPython/3.11.7

Release files / memorious-2.6.5-py2.py3-none-any.whl

Download URL memorious-2.6.5-py2.py3-none-any.whl
Size 52.4 kB
Tags Python 2 Python 3
SHA-256 checksum
How to use checksums 		5997259e0e5e3e92012bd87d506dfd947f8900c53e8c5717696169a523c48780
BLAKE2b-256 checksum
How to use checksums 		70dcf8543dbc42b92a041bfa59a5aa57e61e9a8906cd8bde1bcbe6fcca51dbe1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/4.0.2 CPython/3.11.7

Release history Release notifications | RSS feed

This release

2.6.5 This release

2 release files

2.6.4

2 release files

2.6.3

2 release files

2.6.2

2 release files

2.5.0

2 release files

2.4.5

2 release files

2.4.4

2 release files

2.4.3

2 release files

2.4.2

2 release files

2.4.1

2 release files

2.4.0

2 release files

2.3.4

2 release files

2.3.3

2 release files

2.3.2

2 release files

2.3.1

2 release files

2.3.0

2 release files

2.2.0

2 release files

2.1.1

2 release files

2.1.0

2 release files

2.0.0

2 release files

1.9.0

2 release files

1.8.4

2 release files

1.8.3

2 release files

1.8.2

2 release files

1.8.0

2 release files

1.7.4

2 release files

1.7.3

2 release files

1.7.2

2 release files

1.7.1

2 release files

1.7.0

2 release files

1.6.2

2 release files

1.6.1

2 release files

1.6.0

2 release files

1.5.6

2 release files

1.5.5

2 release files

1.5.4

2 release files

1.5.3

2 release files

1.5.2

2 release files

1.5.1

2 release files

1.5.0

2 release files

1.4.3

2 release files

1.4.2

2 release files

1.4.1

2 release files

1.4.0

2 release files

1.3.0

2 release files

1.2.10

2 release files

1.2.9

2 release files

1.2.8

2 release files

1.2.5

2 release files

1.2.4

2 release files

1.2.3

2 release files

1.2.1

2 release files

1.2.0

2 release files

1.1.3

2 release files

1.1.2

2 release files

1.1.1

2 release files

1.1.0

2 release files

1.0.0

2 release files

0.14.2

2 release files

0.14.1

2 release files

0.14.0

2 release files

0.13.0

2 release files

0.12.0

2 release files

0.11.1

2 release files

0.11.0

2 release files

0.10.1

2 release files

0.10.0

2 release files

0.9.2

2 release files

0.9.1

2 release files

0.9.0

2 release files

0.8.0

2 release files

0.7.20

2 release files

0.7.19

2 release files

0.7.18

2 release files

0.7.17

2 release files

0.7.16

2 release files

0.7.15

2 release files

0.7.14

2 release files

0.7.13

2 release files

0.7.12

2 release files

0.7.11

2 release files

0.7.10

2 release files

0.7.9

2 release files

0.7.8

2 release files

0.7.7

2 release files

0.7.6

2 release files

0.7.4

2 release files

0.7.3

2 release files

0.7.2

2 release files

0.7.1

2 release files

0.7.0

2 release files

0.6.1

2 release files

0.6.0

2 release files

0.5.5

2 release files

0.5.4

2 release files

0.5.3

2 release files

0.5.2

2 release files

0.5.1

2 release files

0.5.0

2 release files

0.4.12

2 release files

0.4.11

2 release files

0.4.10

2 release files

0.4.9

2 release files

0.4.8

2 release files

0.4.7

2 release files

0.4.6

2 release files

0.4.5

2 release files

0.4.4

1 release file

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page