Get and storing the HTML data of a website using a cache system.
Project description
HTTPPOOL
Get and storing the HTML data of a website using a cache system
Explore the docs »
Report Bug
·
Request Feature
About The Project
This library is developed using pure Python code, and is responsible for storing locally, through a cache system using a Producer-Consumer algorithm, the HTML information obtained from any web page. It includes a daemon process that is responsible for automatically updating the cache through a set time, independent for each URL and also includes a tracing system using a logger to monitor the status of the service.
Getting Started
Prerequisites
You need to make sure you have installed the following modules.
pip install requests
If you do not have the library shown above installed, the httppool package will add it for you during its installation.
Installation
pip install Httppool
Usage
Example 1
It will show you and store all the information of the web's HTML
import Httppool as htp
url = 'https://www.prensa-latina.cu/deportes/'
content = htp.get_url_content(url)
print(content)
Example 2
If you use it in conjunction with a Web Scraping library like BeautifulSoup you can get the data you are interested in
import Httppool as htp
from bs4 import BeautifulSoup
url = 'https://www.entumovil.cu/'
content = htp.get_url_content(url)
soup = BeautifulSoup(content, 'html.parser')
response = soup.find('div', id='etm-desription')
print(response.prettify())
# the content of response is:
# <div id="etm-desription">
# <p>
# Somos un equipo de desarrollo de servicios y aplicaciones para móviles
# pertenecientes a la empresa Desoft. Desde sus
# inicios trabajamos orientados a la satisfacción de las necesidades que
# soliciten las empresas de informatizar sus procesos, así como brindar a
# la población la posibilidad de obtener información de su interés a
# través de consultas, suscripciones, notificaciones y compras on line mediante mensajería de texto (SMS)…
# <br/>
# </p>
# <p>
# <br/>
# </p>
# </div>
Example 3
You can run the daemon process for automatic cache update
from Httppool.deamon import HttppoolDeamon
deamon = HttppoolDeamon()
deamon.start()
# It will update the information of each URL
# that is in the cache folder (httppool-cache)
# every 15 min by default. This parameter can be changed
Cache Structure
When using the library for the first time, 3 files are created for each URL in the httppool-cache folder and httppool-logger.log file. The following structure is created in the root directory of the current project:
My_python_project
├── httppool-cache
│ ├── f47eda8cfe42cc7aa478686b321f22c6.lastaccess
│ ├── f47eda8cfe42cc7aa478686b321f22c6.url
│ └── f47eda8cfe42cc7aa478686b321f22c6.webpagedata
├── httppool-logger.log
├── my_python_script.py
.lastaccess
The file f47eda8cfe42cc7aa478686b321f22c6.lastaccess shows the following information:
read: 2025-01-06 11:05:04 # last date of reading
write: 2025-01-06 11:05:04 # last date of writing
retry: 15 # time interval taken to update the cache
.url
The file f47eda8cfe42cc7aa478686b321f22c6.url shows the following information:
# web page URL
https://www.entumovil.cu/
# headers to be sent using the requests library
headers: {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'}
.webpagedata
The file f47eda8cfe42cc7aa478686b321f22c6.webpagedata shows the following information:
<!-- web page HTML information for future Scraping -->
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Title</title>
</head>
<body>
<div class="item " data-ref="empresariales">
<a class="service-link" href="/web/servicios/empresariales/#service_item43s">
<div class="service-button-label">
<p>Micropagos</p>
</div>
<div class="service-button" style="background-image:url(/media/iconMICROPAGO.png) ">
</div>
</a>
</div>
</body>
</html>
httppool-logger.log
The library has a logger to track the traces of the cache system based on the producer-consumer algorithm:
2025-01-06 11:05:02,571 : INFO : HTTPPOOL : Client : Consumming URL --> https://www.entumovil.cu/
2025-01-06 11:05:02,572 : DEBUG : HTTPPOOL : Starting new HTTPS connection (1): www.entumovil.cu:443
2025-01-06 11:05:03,610 : DEBUG : HTTPPOOL : https://www.entumovil.cu:443 "GET / HTTP/1.1" 200 None
2025-01-06 11:05:04,055 : INFO : HTTPPOOL : Producer : Got URL content --> https://www.entumovil.cu/
License
Distributed under the MIT License. See LICENSE for more information.
Contact
Email: alexdev.workenv@gmail.com
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file httppool-1.0.6.tar.gz.
File metadata
- Download URL: httppool-1.0.6.tar.gz
- Upload date:
- Size: 10.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.0.1 CPython/3.12.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
9fec90c514a4fb265b8a74d95e19b883af7be9cc7c69b14496d204a32a753a8d
|
|
| MD5 |
6b0713533bd7f0b00721942c167a59d8
|
|
| BLAKE2b-256 |
a345a5d3cc57d9e6b5a6987781a2dec4a814f5fa78863299ac576d6f8bfa7e8c
|
File details
Details for the file Httppool-1.0.6-py3-none-any.whl.
File metadata
- Download URL: Httppool-1.0.6-py3-none-any.whl
- Upload date:
- Size: 8.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.0.1 CPython/3.12.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
3fb64c778477e2923828aba3dabe30fac8f28ad9d795e5c5a4655dd90ff1c64e
|
|
| MD5 |
f30cc71e7eaddeb82e74df6825effd65
|
|
| BLAKE2b-256 |
232a3209c89c2bb2f634def49d662d9422babe5d64e6478042317e224627dd79
|
