ekrhizoc
ekrhizoc (E6c): A web crawler
Contents
Definition
εκρίζωση (Greek) ekrízosi / uprooting, eradication
Also known as E6c.
Use Case
Implementation of a simple python web crawler.
Input: URL (seed).
Output: Simple textual sitemap (to show links between pages).
Requirements
- The crawler is limited to one subdomain (exclude external links).
- No use of web crawling libraries/frameworks (e.g. scrapy).
- (Optional) Use of HTML handling Libraries/Frameworks.
- Production-ready code.
Assumptions
- The input URL (seed) is limited to only one at every run.
- The targeted URL(s) are static pages (no backend javascript parsing required).
- Links to be extracted from HTML anchor
<a>elements. - Valid links include
- Valid URL
- Non empty
- Matches a valid url pattern
- Does not exceed the
E6C_MAX_URL_LENGTHlength in characters - Possible to convert a relative urls to a full url
- Link is not visited before
- Link is not part of an ignored file type
- Link has the same domain as the seed url
- Link is not restricted by the robots.txt file
- Valid URL
Design
This project implements a Basic Universal Crawler based on breadth first search graph traversal.
Configuration
Behaviour of the application can be configured via Environment Variables.
| Environment Variable | Description | Type | Default Value |
|---|---|---|---|
E6C_LOG_LEVEL |
Level of logging - overrides verbose/quiet flag | string | - |
E6C_LOG_DIR |
Directory to save logs | string | - |
E6C_BIN_DIR |
Directory to save any output (bin) | string | bin |
E6C_IGNORE_FILETYPES |
File types of websites to ignore (e.g. ".filetype1,.filetype2") | string | ".png,.pdf,.txt,.doc,.jpg,.gif" |
E6C_URL_REQUEST_TIMER |
Time (in seconds) to wait per request (not to populate server with multiple requests) | float | 0.1 |
E6C_MAX_URLS |
The maximum number of urls to fetch/crawl | integer | 10000 |
E6C_MAX_URL_LENGTH |
The maximum length (character count) of a url to fetch/crawl | integer | 300 |
Development
Configure your local development
- Clone repo on your local machine
- Install
condaorminiconda - Create your local project environment (based on
conda,poetry,pre-commit):
$ make env - (Optional) Update existing local project environment:
$ make env-update
Run locally
On a terminal, run the following (execute on project's root directory):
- Activate project environment:
$ . ./scripts/helpers/environment.sh - Run the CLI using
poetry:
$ ekrhizoc
Contribute
[ Not Available ]
Testing
(part of CI/CD)
[ Work in progress... ]
To run the tests, open a terminal and run the following (execute on project's root directory):
- Activate project environment:
$ . ./scripts/helpers/environment.sh - To run pytest:
$ make test - To check test coverage:
$ make test-coverage
Versioning
Increment the version number:
$ poetry version {bump rule}
where valid bump rules are:
- patch
- minor
- major
- prepatch
- preminor
- premajor
- prerelease
Changelog
Use CHANGELOG.md to track the evolution of this package.
The [UNRELEASED] tag at the top of the file should always be there to log the work until a release occurs.
Work should be logged under one of the following subtitles:
- Added
- Changed
- Fixed
- Removed
On a release, a version of the following format should be added to all the current unreleased changes in the file.
## [major.minor.patch] - YYYY-MM-DD
Deployment
Pip package
On a terminal, run the following (execute on project's root directory):
- Activate project environment:
$ . ./scripts/helpers/environment.sh - To build pip package:
$ make build-package - To publish pip package (requires credentials to PyPi):
$ make publish-package
Docker image
On a terminal, run the following (execute on project's root directory):
- Activate project environment:
$ . ./scripts/helpers/environment.sh - To build docker image:
$ make build-docker
Production
For production, a Docker image is used. This image is published publicly on docker hub.
- First pull image from docker hub:
$ docker pull nichelia/ekrhizoc:{version} - Execute CLI via docker run:
$ docker run --rm -it -v ~/ekrhizoc_bin:/tmp/bin nichelia/ekrhizoc:{version} {command}
This command mounts the application's bin (outcome) to user's root directory under ekrhizoc_bin folder.
where version is the published application version
Metadata
Release files for ekrhizoc 0.1.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| ekrhizoc-0.1.2.tar.gz | 16.9 kB | Without provenance | Details for ekrhizoc-0.1.2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| ekrhizoc-0.1.2-py3-none-any.whl | Python 3 | none | any | Details for ekrhizoc-0.1.2-py3-none-any.whl |
Total release size: 34.8 kB
Release files / ekrhizoc-0.1.2.tar.gz
| Download URL | ekrhizoc-0.1.2.tar.gz |
|---|---|
| Size | 16.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
89eeecd50a60d3d6fde8bd0881368230a2cd44e38609049433880998e6ef0540
|
|
BLAKE2b-256 checksum How to use checksums |
c3124a6f87cb29bd0c32af7e8b801078ce200a623ec8bbcab325d479c867077d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.0.0 CPython/3.8.3 Linux/5.3.0-1028-azure
|
Release files / ekrhizoc-0.1.2-py3-none-any.whl
| Download URL | ekrhizoc-0.1.2-py3-none-any.whl |
|---|---|
| Size | 17.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
898ce87267f27798fb1cac4644f38d27b779bba868eae46349cb3270cfb35804
|
|
BLAKE2b-256 checksum How to use checksums |
7969cfbd1f5398df1ad6971c122172f2a111f632c68ba6e21fd2eb6af3a851d7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.0.0 CPython/3.8.3 Linux/5.3.0-1028-azure
|
Security for ekrhizoc 0.1.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.