Get and storing the HTML data of a website using a cache system.

Navigation

Verified details

These details have been verified by PyPI
Project links
GitHub Statistics
Maintainers
Avatar for alexdevzz from gravatar.com alexdevzz

Unverified details

These details have not been verified by PyPI
Meta
  • License: MIT License
  • Author: alexdev
  • Requires: Python >=3.8
Classifiers
Report project as malware

Project description

Version Static Badge Static Badge Static Badge Static Badge Framework


Logo

HTTPPOOL

Get and storing the HTML data of a website using a cache system

Explore the docs »

Report Bug · Request Feature

About The Project

This library is developed using pure Python code, and is responsible for storing locally, through a cache system using a Producer-Consumer algorithm, the HTML information obtained from any web page. It includes a daemon process that is responsible for automatically updating the cache through a set time, independent for each URL and also includes a tracing system using a logger to monitor the status of the service.

Getting Started

Prerequisites

You need to make sure you have installed the following modules.

pip install requests

If you do not have the library shown above installed, the httppool package will add it for you during its installation.

Installation

pip install Httppool

Usage

Example 1

It will show you and store all the information of the web's HTML

import Httppool as htp

url = 'https://www.prensa-latina.cu/deportes/'
content = htp.get_url_content(url)

print(content)

Example 2

If you use it in conjunction with a Web Scraping library like BeautifulSoup you can get the data you are interested in

import Httppool as htp
from bs4 import BeautifulSoup

url = 'https://www.entumovil.cu/'

content = htp.get_url_content(url)
soup = BeautifulSoup(content, 'html.parser')
response = soup.find('div', id='etm-desription')

print(response.prettify())

# the content of response is:

# <div id="etm-desription">
#     <p>
#         Somos un equipo de desarrollo de servicios y aplicaciones para móviles
#         pertenecientes a la empresa Desoft. Desde sus
#         inicios trabajamos orientados a la satisfacción de las necesidades que
#         soliciten las empresas de informatizar sus procesos, así como brindar a
#         la población la posibilidad de obtener información de su interés a
#         través de consultas, suscripciones, notificaciones y compras on line  mediante mensajería de texto (SMS)…
#         <br/>
#     </p>
#     <p>
#         <br/>
#     </p>
# </div>

Example 3

You can run the daemon process for automatic cache update

from Httppool.deamon import HttppoolDeamon

deamon = HttppoolDeamon()
deamon.start()

# It will update the information of each URL 
# that is in the cache folder (httppool-cache) 
# every 15 min by default. This parameter can be changed

Cache Structure

When using the library for the first time, 3 files are created for each URL in the httppool-cache folder and httppool-logger.log file. The following structure is created in the root directory of the current project:

My_python_project
├── httppool-cache
│   ├── f47eda8cfe42cc7aa478686b321f22c6.lastaccess
│   ├── f47eda8cfe42cc7aa478686b321f22c6.url
│   └── f47eda8cfe42cc7aa478686b321f22c6.webpagedata
├── httppool-logger.log
├── my_python_script.py

.lastaccess

The file f47eda8cfe42cc7aa478686b321f22c6.lastaccess shows the following information:

read: 2025-01-06 11:05:04   # last date of reading

write: 2025-01-06 11:05:04  # last date of writing

retry: 15  # time interval taken to update the cache

.url

The file f47eda8cfe42cc7aa478686b321f22c6.url shows the following information:

# web page URL
https://www.entumovil.cu/  

# headers to be sent using the requests library
headers: {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'}

.webpagedata

The file f47eda8cfe42cc7aa478686b321f22c6.webpagedata shows the following information:

<!-- web page HTML information for future Scraping -->
<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="UTF-8">
        <title>Title</title>
    </head>
    
    <body>
        <div class="item " data-ref="empresariales">
            <a class="service-link" href="/web/servicios/empresariales/#service_item43s">
                <div class="service-button-label">
                    <p>Micropagos</p>
                </div>
                <div class="service-button" style="background-image:url(/media/iconMICROPAGO.png) ">
                </div>
            </a>
        </div>
    </body>
</html>

httppool-logger.log

The library has a logger to track the traces of the cache system based on the producer-consumer algorithm:

2025-01-06 11:05:02,571 : INFO : HTTPPOOL : Client : Consumming URL --> https://www.entumovil.cu/ 

2025-01-06 11:05:02,572 : DEBUG : HTTPPOOL : Starting new HTTPS connection (1): www.entumovil.cu:443 

2025-01-06 11:05:03,610 : DEBUG : HTTPPOOL : https://www.entumovil.cu:443 "GET / HTTP/1.1" 200 None 

2025-01-06 11:05:04,055 : INFO : HTTPPOOL : Producer : Got URL content --> https://www.entumovil.cu/

License

Distributed under the MIT License. See LICENSE for more information.

Contact

Email: alexdev.workenv@gmail.com

Project details

Verified details

These details have been verified by PyPI
Project links
GitHub Statistics
Maintainers
Avatar for alexdevzz from gravatar.com alexdevzz

Unverified details

These details have not been verified by PyPI
Meta
  • License: MIT License
  • Author: alexdev
  • Requires: Python >=3.8
Classifiers

Release history Release notifications | RSS feed

1.0.23

1.0.22

1.0.21

1.0.20

1.0.18

1.0.17

1.0.16

1.0.14

1.0.11

This version

1.0.10

1.0.7

1.0.6

1.0.5

1.0.4

1.0.3

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

httppool-1.0.10.tar.gz (10.0 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

httppool-1.0.10-py3-none-any.whl (7.9 kB view details)

Uploaded Python 3

File details

Details for the file httppool-1.0.10.tar.gz.

File metadata

  • Download URL: httppool-1.0.10.tar.gz
  • Upload date:
  • Size: 10.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.12.9

File hashes

Hashes for httppool-1.0.10.tar.gz
Algorithm Hash digest
SHA256 bf623ef8ba31ea15728bc2b9a31cd7b739138a7fe1c1e0194cb00401b1f3b886
MD5 6d23b130d274740ba902d3d08b7a2227
BLAKE2b-256 c40bc4ac3a24b1488ff45aa9119691259f24d9017fbb97330ca339bc1688bb92

See more details on using hashes here.

Provenance

The following attestation bundles were made for httppool-1.0.10.tar.gz:

Publisher: publish-httppool.yml on alexdevzz/httppool-ce-services

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file httppool-1.0.10-py3-none-any.whl.

File metadata

  • Download URL: httppool-1.0.10-py3-none-any.whl
  • Upload date:
  • Size: 7.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.12.9

File hashes

Hashes for httppool-1.0.10-py3-none-any.whl
Algorithm Hash digest
SHA256 bf51b24e3ec3e5fc20ff8cad79c88de577cb5cb87b51bb31d484d2d3069f4791
MD5 88a42051eb972a6686bcb51c21c0e903
BLAKE2b-256 3567402a82b9423690e20af9c43093cee19d6a12064a72ff78fbece6ffc9137c

See more details on using hashes here.

Provenance

The following attestation bundles were made for httppool-1.0.10-py3-none-any.whl:

Publisher: publish-httppool.yml on alexdevzz/httppool-ce-services

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page