#######################################################
SWAgent Crawler
#######################################################
This is a simple web crawler specifically designed to scrape data from swranking.com continuously in order to build a database with data useful enoughn to train a ML model to make RTA draft predictions in real time.
The package contains two helper classes:
- USERAGENT: creates randomized user_agents to
send through the REST request t obtain data from the websites API.
- SEEKER: this is the actual crawler that finds
the information for us and then sends it out as a json object.
The package main routine focuses on a basic ETL schema. afte obtaining the data from the seeker object it then transfoms the data to be in the format wanted by the Database. Then we send it to the local db of the VM to store for further processing by other jobs.
1 maintainer
Metadata
1 maintainer
Release files for swagenttools 0.3.20
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| swagenttools-0.3.20.tar.gz | 43.3 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| swagenttools-0.3.20-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 86.5 kB
Release files / swagenttools-0.3.20.tar.gz
| Download URL | swagenttools-0.3.20.tar.gz |
|---|---|
| Size | 43.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
b5eeacefda646112a21160499a52c64516b3ff8368ef05ec9166f65bf9520bf2
|
|
BLAKE2b-256 checksum How to use checksums |
09fe1d3d7a97c6957b6dad4f8cb4c79a0161bc7f497343101d33c8ee8f79c17e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.8.10
|
Release files / swagenttools-0.3.20-py3-none-any.whl
| Download URL | swagenttools-0.3.20-py3-none-any.whl |
|---|---|
| Size | 43.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
1fc349aa2362ca188597967223b4315f2c3c30938d49a1e5f38538eb1abb1f6e
|
|
BLAKE2b-256 checksum How to use checksums |
af35f73c8c48302b29fcae09ae84e44ff9ec1e9af042856b887c739c59e2b2a4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.8.10
|
