web2db
Fetches the full text of input URLs and persists them to sqlite3 DB file.
Fetching is resumable and comes with a progressbar.
Install:
pip install web2db
Quickstart:
import web2db
web2db.dump('data.db', urls=[
'https://www.google.com',
'https://www.yahoo.com',
'https://www.msn.com'
])
Query the DB file:
df = web2db.to_df(sqlite3_file_path)
print(df.shape)
print(df)
SQL Schema:
- Table:
-
WebPages
url fulltext status_code text text int
-
Features:
- Resumable webpage fetching
- Saves to local SQLITE3 DB
- tqdm progress bar
1 maintainer
Release files for web2db 0.1.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| web2db-0.1.5.tar.gz | 2.1 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| web2db-0.1.5-py2.py3-none-any.whl | Python 2, Python 3 | none | any | Details |
Total release size: 4.5 kB
Release files / web2db-0.1.5.tar.gz
| Download URL | web2db-0.1.5.tar.gz |
|---|---|
| Size | 2.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
87f3f2e109e446f913c116396022de63765f11ae33998bbefc4e5b25cc348ff2
|
|
BLAKE2b-256 checksum How to use checksums |
a5ccc266bd98567645f011b94b3b0b0bd40e30050e4438ff8250b62464d596b1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/50.3.0 requests-toolbelt/0.9.1 tqdm/4.49.0 CPython/3.7.8
|
Release files / web2db-0.1.5-py2.py3-none-any.whl
| Download URL | web2db-0.1.5-py2.py3-none-any.whl |
|---|---|
| Size | 2.4 kB |
| Tags | Python 2 Python 3 |
|
SHA-256 checksum How to use checksums |
4d25c612be301941d1d15c1028a9e824fa203bdd08c042bc6c17d28fd4306798
|
|
BLAKE2b-256 checksum How to use checksums |
a20f3751d7f307830ceff167bfbc44d7257b0876872f9cf21f5007c3a3960e11
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/50.3.0 requests-toolbelt/0.9.1 tqdm/4.49.0 CPython/3.7.8
|
