schul-cloud-url-crawler
This crawler fetches ressources from urls and posts them to a server.
Purpose
The purpose of this crawler is:
We can provide test data to the API.
It can crawl ressources which are not active and cannot post.
Other crawl services can use this crawler to upload their conversions.
It has the full crawler logic but does not transform into other formats.
Maybe we can create recommendations or a library for crawlers from this case.
Requirements
The crawler should work as follows:
Provide urls
as command line arguments
as a link to a file with one url per line
Provide ressources
as one ressource in a file
as a list of ressources
The crawler must be invoked to crawl.
Example
This example gets a ressource from the url and post it to the api.
python3 -m ressource_url_crawler http://localhost:8080 \
https://raw.githubusercontent.com/schul-cloud/ressources-api-v1/master/schemas/ressource/examples/valid/example-website.jsonAuthentication
You can specify the authentication like this:
--basic=username:password for basic authentication
--apikey=apikey for api key authentication
Further Requirements
The crawler does not post ressources twice. This can be implemented by
caching the ressources locally, to see if they changed
compare ressource
compare timestamp
removing the ressources from the database if they are updated after posting new ressources.
This may require some form of state for the crawler. The state could be added to the ressources in a X-Ressources-Url-Crawler-Source field. This allows local caching and requires getting the objects from the database.
1 maintainer
Metadata
1 maintainer
Release files for schul-cloud-url-crawler 1.0.17
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| schul_cloud_url_crawler-1.0.17.tar.gz | 14.2 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| schul_cloud_url_crawler-1.0.17-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 34.4 kB
Release files / schul_cloud_url_crawler-1.0.17.tar.gz
| Download URL | schul_cloud_url_crawler-1.0.17.tar.gz |
|---|---|
| Size | 14.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
cfdc59458d10c3c38aaa1e79288ee935abc5a3ae1e9f05fdb1898514fd538131
|
|
BLAKE2b-256 checksum How to use checksums |
40ee5161e6791606a2fe6c459a9711dfd3a915f3ecdd72505eaeea8b7635f7a1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
Release files / schul_cloud_url_crawler-1.0.17-py3-none-any.whl
| Download URL | schul_cloud_url_crawler-1.0.17-py3-none-any.whl |
|---|---|
| Size | 20.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b67a81c05bc4b51eba0e5693a52dd0b7672f948d571769d51fda4df0ca6f6f58
|
|
BLAKE2b-256 checksum How to use checksums |
a15eb4c3c51a07c207146695d9f672a2d9d5caf99e4c0b578d73131b8c5e8d02
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
Security for schul-cloud-url-crawler 1.0.17
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
