Get and storing the HTML data of a website using a cache system.

Navigation

Verified details

These details have been verified by PyPI
Project links
GitHub Statistics
Maintainers
Avatar for alexdevzz from gravatar.com alexdevzz

Unverified details

These details have not been verified by PyPI
Meta
  • License: MIT License
  • Author: alexdev
  • Requires: Python >=3.8
Classifiers
Report project as malware

Project description

Version Static Badge Static Badge Static Badge Static Badge Framework


Logo

HTTPPOOL

Get and storing the HTML data of a website using a cache system

Explore the docs »

Report Bug · Request Feature

About The Project

This library is developed using pure Python code, and is responsible for storing locally, through a cache system using a Producer-Consumer algorithm, the HTML information obtained from any web page. It includes a daemon process that is responsible for automatically updating the cache through a set time, independent for each URL and also includes a tracing system using a logger to monitor the status of the service.

Getting Started

Prerequisites

You need to make sure you have installed the following modules.

pip install requests

If you do not have the library shown above installed, the httppool package will add it for you during its installation.

Installation

pip install Httppool

Usage

Example 1

It will show you and store all the information of the web's HTML

import Httppool as htp

url = 'https://www.prensa-latina.cu/deportes/'
content = htp.get_url_content(url)

print(content)

Example 2

If you use it in conjunction with a Web Scraping library like BeautifulSoup you can get the data you are interested in

import Httppool as htp
from bs4 import BeautifulSoup

url = 'https://www.entumovil.cu/'

content = htp.get_url_content(url)
soup = BeautifulSoup(content, 'html.parser')
response = soup.find('div', id='etm-desription')

print(response.prettify())

# the content of response is:

# <div id="etm-desription">
#     <p>
#         Somos un equipo de desarrollo de servicios y aplicaciones para móviles
#         pertenecientes a la empresa Desoft. Desde sus
#         inicios trabajamos orientados a la satisfacción de las necesidades que
#         soliciten las empresas de informatizar sus procesos, así como brindar a
#         la población la posibilidad de obtener información de su interés a
#         través de consultas, suscripciones, notificaciones y compras on line  mediante mensajería de texto (SMS)…
#         <br/>
#     </p>
#     <p>
#         <br/>
#     </p>
# </div>

Example 3

You can run the daemon process for automatic cache update

from Httppool.deamon import HttppoolDeamon

deamon = HttppoolDeamon()
deamon.start()

# It will update the information of each URL 
# that is in the cache folder (httppool-cache) 
# every 15 min by default. This parameter can be changed

Cache Structure

When using the library for the first time, 3 files are created for each URL in the httppool-cache folder and httppool-logger.log file. The following structure is created in the root directory of the current project:

My_python_project
├── httppool-cache
│   ├── f47eda8cfe42cc7aa478686b321f22c6.lastaccess
│   ├── f47eda8cfe42cc7aa478686b321f22c6.url
│   └── f47eda8cfe42cc7aa478686b321f22c6.webpagedata
├── httppool-logger.log
├── my_python_script.py

.lastaccess

The file f47eda8cfe42cc7aa478686b321f22c6.lastaccess shows the following information:

read: 2025-01-06 11:05:04   # last date of reading

write: 2025-01-06 11:05:04  # last date of writing

retry: 15  # time interval taken to update the cache

.url

The file f47eda8cfe42cc7aa478686b321f22c6.url shows the following information:

# web page URL
https://www.entumovil.cu/  

# headers to be sent using the requests library
headers: {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'}

.webpagedata

The file f47eda8cfe42cc7aa478686b321f22c6.webpagedata shows the following information:

<!-- web page HTML information for future Scraping -->
<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="UTF-8">
        <title>Title</title>
    </head>
    
    <body>
        <div class="item " data-ref="empresariales">
            <a class="service-link" href="/web/servicios/empresariales/#service_item43s">
                <div class="service-button-label">
                    <p>Micropagos</p>
                </div>
                <div class="service-button" style="background-image:url(/media/iconMICROPAGO.png) ">
                </div>
            </a>
        </div>
    </body>
</html>

httppool-logger.log

The library has a logger to track the traces of the cache system based on the producer-consumer algorithm:

2025-01-06 11:05:02,571 : INFO : HTTPPOOL : Client : Consumming URL --> https://www.entumovil.cu/ 

2025-01-06 11:05:02,572 : DEBUG : HTTPPOOL : Starting new HTTPS connection (1): www.entumovil.cu:443 

2025-01-06 11:05:03,610 : DEBUG : HTTPPOOL : https://www.entumovil.cu:443 "GET / HTTP/1.1" 200 None 

2025-01-06 11:05:04,055 : INFO : HTTPPOOL : Producer : Got URL content --> https://www.entumovil.cu/

License

Distributed under the MIT License. See LICENSE for more information.

Contact

Email: alexdev.workenv@gmail.com

Project details

Verified details

These details have been verified by PyPI
Project links
GitHub Statistics
Maintainers
Avatar for alexdevzz from gravatar.com alexdevzz

Unverified details

These details have not been verified by PyPI
Meta
  • License: MIT License
  • Author: alexdev
  • Requires: Python >=3.8
Classifiers

Release history Release notifications | RSS feed

This version

1.0.23

1.0.22

1.0.21

1.0.20

1.0.18

1.0.17

1.0.16

1.0.14

1.0.11

1.0.10

1.0.7

1.0.6

1.0.5

1.0.4

1.0.3

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

httppool-1.0.23.tar.gz (10.4 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

httppool-1.0.23-py3-none-any.whl (8.4 kB view details)

Uploaded Python 3

File details

Details for the file httppool-1.0.23.tar.gz.

File metadata

  • Download URL: httppool-1.0.23.tar.gz
  • Upload date:
  • Size: 10.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.12.9

File hashes

Hashes for httppool-1.0.23.tar.gz
Algorithm Hash digest
SHA256 6ffba4042788ce87cc1452c73a843bb75217e3aee2c48da7e6aaa1efdabe1f75
MD5 3d051d21085a1a9061f4230a81fb0305
BLAKE2b-256 f05a5f8a36238b6595f7ab8d526e0bb49a1942c121fa548f3ac4ca544edb4b6f

See more details on using hashes here.

Provenance

The following attestation bundles were made for httppool-1.0.23.tar.gz:

Publisher: publish-httppool.yml on alexdevzz/httppool-ce-services

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file httppool-1.0.23-py3-none-any.whl.

File metadata

  • Download URL: httppool-1.0.23-py3-none-any.whl
  • Upload date:
  • Size: 8.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.12.9

File hashes

Hashes for httppool-1.0.23-py3-none-any.whl
Algorithm Hash digest
SHA256 ec1a588f35788b908320604adc5c631ac2a03445d1f78bd56467706fec4e829b
MD5 3d65397bdf6e4caf955b3734bba16579
BLAKE2b-256 d699b8fc9edaafbcadc71e366b94d7bf5c69c692b109828424118f0b61cc928c

See more details on using hashes here.

Provenance

The following attestation bundles were made for httppool-1.0.23-py3-none-any.whl:

Publisher: publish-httppool.yml on alexdevzz/httppool-ce-services

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page