Scalable and Compliant Web Crawler
Project description
DPK Connector
DPK Connector is a scalable and compliant web crawler developed for data acquisition towards LLM development. It is built on Scrapy. For more details read the documentation.
Virtual Environment
The project uses pyproject.toml and a Makefile for operations.
To do development you should establish the virtual environment
make venv
and then either activate
source venv/bin/activate
or set up your IDE to use the venv directory when developing in this project
Library Artifact Build and Publish
To test, build and publish the library
make test build publish
To up the version number, edit the Makefile to change VERSION and rerun the above. This will require committing both the Makefile and the autotmatically updated pyproject.toml file.
How to use
See the overview.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file data_prep_connector-0.2.3.dev0.tar.gz.
File metadata
- Download URL: data_prep_connector-0.2.3.dev0.tar.gz
- Upload date:
- Size: 16.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.12.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 50312c39b843c80db8f9fd2ec6eb2ba825583c5c983c7f1f7b0ced3b604a5421 |
|
| MD5 | 49572774d806e5785b12eec868b8c8cd |
|
| BLAKE2b-256 | 4b6dea842b50a335c3422d6dba24817606551bec1418bfa63bd942eda3736668 |
File details
Details for the file data_prep_connector-0.2.3.dev0-py3-none-any.whl.
File metadata
- Download URL: data_prep_connector-0.2.3.dev0-py3-none-any.whl
- Upload date:
- Size: 17.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.12.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | faf44f0328e891c2c6957c6978f316b620b5e9afa624a5465588b3e419fc911a |
|
| MD5 | a7e3f4259395e7c51e460914d1ae1827 |
|
| BLAKE2b-256 | e2613a6d9f56e92e09c9fed652dd54906f65e71b135fb5213452f9db755bb517 |
