Get and storing the HTML data of a website using a cache system.

Navigation

Verified details

These details have been verified by PyPI
Project links
GitHub Statistics
Maintainers
Avatar for alexdevzz from gravatar.com alexdevzz

Unverified details

These details have not been verified by PyPI
Meta
  • License: MIT License
  • Author: alexdev
  • Requires: Python >=3.8
Classifiers
Report project as malware

Project description

Version Static Badge Static Badge Static Badge Static Badge Framework


Logo

HTTPPOOL

Get and storing the HTML data of a website using a cache system

Explore the docs »

Report Bug · Request Feature

About The Project

This library is developed using pure Python code, and is responsible for storing locally, through a cache system using a Producer-Consumer algorithm, the HTML information obtained from any web page. It includes a daemon process that is responsible for automatically updating the cache through a set time, independent for each URL and also includes a tracing system using a logger to monitor the status of the service.

Getting Started

Prerequisites

You need to make sure you have installed the following modules.

pip install requests

If you do not have the library shown above installed, the httppool package will add it for you during its installation.

Installation

pip install Httppool

Usage

Example 1

It will show you and store all the information of the web's HTML

import Httppool as htp

url = 'https://www.prensa-latina.cu/deportes/'
content = htp.get_url_content(url)

print(content)

Example 2

If you use it in conjunction with a Web Scraping library like BeautifulSoup you can get the data you are interested in

import Httppool as htp
from bs4 import BeautifulSoup

url = 'https://www.entumovil.cu/'

content = htp.get_url_content(url)
soup = BeautifulSoup(content, 'html.parser')
response = soup.find('div', id='etm-desription')

print(response.prettify())

# the content of response is:

# <div id="etm-desription">
#     <p>
#         Somos un equipo de desarrollo de servicios y aplicaciones para móviles
#         pertenecientes a la empresa Desoft. Desde sus
#         inicios trabajamos orientados a la satisfacción de las necesidades que
#         soliciten las empresas de informatizar sus procesos, así como brindar a
#         la población la posibilidad de obtener información de su interés a
#         través de consultas, suscripciones, notificaciones y compras on line  mediante mensajería de texto (SMS)…
#         <br/>
#     </p>
#     <p>
#         <br/>
#     </p>
# </div>

Example 3

You can run the daemon process for automatic cache update

from Httppool.deamon import HttppoolDeamon

deamon = HttppoolDeamon()
deamon.start()

# It will update the information of each URL 
# that is in the cache folder (httppool-cache) 
# every 15 min by default. This parameter can be changed

Cache Structure

When using the library for the first time, 3 files are created for each URL in the httppool-cache folder and httppool-logger.log file. The following structure is created in the root directory of the current project:

My_python_project
├── httppool-cache
│   ├── f47eda8cfe42cc7aa478686b321f22c6.lastaccess
│   ├── f47eda8cfe42cc7aa478686b321f22c6.url
│   └── f47eda8cfe42cc7aa478686b321f22c6.webpagedata
├── httppool-logger.log
├── my_python_script.py

.lastaccess

The file f47eda8cfe42cc7aa478686b321f22c6.lastaccess shows the following information:

read: 2025-01-06 11:05:04   # last date of reading

write: 2025-01-06 11:05:04  # last date of writing

retry: 15  # time interval taken to update the cache

.url

The file f47eda8cfe42cc7aa478686b321f22c6.url shows the following information:

# web page URL
https://www.entumovil.cu/  

# headers to be sent using the requests library
headers: {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'}

.webpagedata

The file f47eda8cfe42cc7aa478686b321f22c6.webpagedata shows the following information:

<!-- web page HTML information for future Scraping -->
<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="UTF-8">
        <title>Title</title>
    </head>
    
    <body>
        <div class="item " data-ref="empresariales">
            <a class="service-link" href="/web/servicios/empresariales/#service_item43s">
                <div class="service-button-label">
                    <p>Micropagos</p>
                </div>
                <div class="service-button" style="background-image:url(/media/iconMICROPAGO.png) ">
                </div>
            </a>
        </div>
    </body>
</html>

httppool-logger.log

The library has a logger to track the traces of the cache system based on the producer-consumer algorithm:

2025-01-06 11:05:02,571 : INFO : HTTPPOOL : Client : Consumming URL --> https://www.entumovil.cu/ 

2025-01-06 11:05:02,572 : DEBUG : HTTPPOOL : Starting new HTTPS connection (1): www.entumovil.cu:443 

2025-01-06 11:05:03,610 : DEBUG : HTTPPOOL : https://www.entumovil.cu:443 "GET / HTTP/1.1" 200 None 

2025-01-06 11:05:04,055 : INFO : HTTPPOOL : Producer : Got URL content --> https://www.entumovil.cu/

License

Distributed under the MIT License. See LICENSE for more information.

Contact

Email: alexdev.workenv@gmail.com

Project details

Verified details

These details have been verified by PyPI
Project links
GitHub Statistics
Maintainers
Avatar for alexdevzz from gravatar.com alexdevzz

Unverified details

These details have not been verified by PyPI
Meta
  • License: MIT License
  • Author: alexdev
  • Requires: Python >=3.8
Classifiers

Release history Release notifications | RSS feed

1.0.23

1.0.22

1.0.21

1.0.20

1.0.18

This version

1.0.17

1.0.16

1.0.14

1.0.11

1.0.10

1.0.7

1.0.6

1.0.5

1.0.4

1.0.3

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

httppool-1.0.17.tar.gz (10.0 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

httppool-1.0.17-py3-none-any.whl (7.9 kB view details)

Uploaded Python 3

File details

Details for the file httppool-1.0.17.tar.gz.

File metadata

  • Download URL: httppool-1.0.17.tar.gz
  • Upload date:
  • Size: 10.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.12.9

File hashes

Hashes for httppool-1.0.17.tar.gz
Algorithm Hash digest
SHA256 4867c93fe7d2417cd32ab4bb442e9a62b138484d54629d360a9244c7f6dee4c9
MD5 3c90539a4f65cbdd663614397c12e32e
BLAKE2b-256 d086ae896d5af63d2a18a480c4db21c9d9a14cfaafd626c2b4e6f65aa6b760a4

See more details on using hashes here.

Provenance

The following attestation bundles were made for httppool-1.0.17.tar.gz:

Publisher: publish-httppool.yml on alexdevzz/httppool-ce-services

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file httppool-1.0.17-py3-none-any.whl.

File metadata

  • Download URL: httppool-1.0.17-py3-none-any.whl
  • Upload date:
  • Size: 7.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.12.9

File hashes

Hashes for httppool-1.0.17-py3-none-any.whl
Algorithm Hash digest
SHA256 9fac8d673a81591f029263db80a70ea24efd5c7bffb8951c1d507994d2f33e94
MD5 2376b331f580390cda46859cb24c4575
BLAKE2b-256 57d1e9beb31c698baf8304df3e8c5ed9da3c893b532172886a765ffb0c925651

See more details on using hashes here.

Provenance

The following attestation bundles were made for httppool-1.0.17-py3-none-any.whl:

Publisher: publish-httppool.yml on alexdevzz/httppool-ce-services

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page