Skip to main content

from requests_toolbelt import user_agent

Amina Group Page Scraper

this package is help you to get seo information on a webpage. the types are info are:

JSON-LD schema info

json ld schema is a graph which is used to define what is the web page and what can you find in this webpage. we use detectors to find info on page and categorize them.

Article

the types on page which are defined as article are

  • "Article"
  • "NewsArticle",
  • "TechArticle",
  • "ScholarlyArticle",
  • "BlogPosting",
  • "Report",
  • "Blog"

these types on page are identified as article

Contact and About

for contact pages we have

  • "ContactPage"
  • "AboutPage"

Domain and Organization

for pages which are root of the domain we have domain info and organization which are specify the brand of the page

for example name of organization which have url is identified as brand name of the website

Other Types

  • navigation
  • person
  • product
  • service
  • category

are other types which we identify as ld json schema all these are gathered as ENTITY

Entity

entity is a data type we provide which have these fields

  • id: (string)
  • name: (string) can be null
  • type: (string) this field defined by type detectors which are explained above
  • alternate_name: (string) can be null
  • url: (string) can be null
  • description: (string) can be null
  • raw: (dictionary)
  • date_published: (date time) can be null
  • date_modified: (date time) can be null

usage

#import Entity pipeline
from page_scraper import EntityPipeline

#import detectors you need
from page_scraper import ArticleDetector,DomainDetector

#import http fetcher and parser to fetch the page correctly and parse the html
from page_scraper import HttpFetcher, PageParser

from page_scraper import build_page_context

#from builders import page builder

# define url
url = "https://amina-group.com/"

# get and parse the page to create a Page instance
fetcher = HttpFetcher()
parser = PageParser()
fetched_url = fetcher.get(url)
parsed_info = parser.parse(fetched_url)
# create page instance using build_page_context
page = build_page_context(parsed_info)

# create pipeline with detectors you want
pipeline = EntityPipeline(
    [
        ArticleDetector(),
        DomainDetector(),
    ]
)

# run the pipeline using page
pipeline.run(page)

we let you classify the links on page but first before using this you need to add Beautiful soup object as soup property of the page

BeautifulSoup instance

from bs4 import BeautifulSoup

page.soup = BeautifulSoup(fetched_url.content, "html.parser")

every url in page (except the links on ld json) are stored as UrlContext data type

for main urls we have:

  • canonical (stores as canonical property of page as well)
  • navigation
  • internal

you can use scraper pipeline to get each or all urls

usage

# import pipeline
from page_scraper import ScraperPipeline

# import scrapers
from page_scraper import GetNavLinks(),InternalScraper(),CanonicalScraper()

# create pipeline
scrape_pipeline = ScraperPipeline([
    GetNavLinks(),
    InternalScraper(),
    CanonicalScraper(),
])

# now use page and remember page soup property must be set
scrape_pipeline.run(page)

Content Scrape

what need of content is headers from h1,h2,h3,h4,h5,h6 we want links and the type of links images videos audios

headings

headings are also have a data type which is PageHeading and after scraping each heading stored as PageHeading instance

the page have a property named headings page.headings and heading scraper stores heading of the page to this property

media tags

media tags are also url and they will be treated as UrlContext as well the media types are:

  • image
  • video
  • audio

each media tag in url context define as {meida_name}_TAG for example IMAGE_TAG for image

you can get the video of a page with 3 different ways

  1. video tag
  2. iframe
  3. embed or object tag

NOTE: you cannot get iframe by a normal url fetcher because iframe will only loads if javascript available.

usage

# import pipeline
from page_scraper import ContentPipeline

# import scrapers
from page_scraper import HeadingScraper(),\
    ImageScraper(),VideoTagScraper(),AudioTagScraper(),\
    VideoIframeScraper(),VideoEmbedScraper()

# create pipeline
content_pipeline = ContentPipeline([
    HeadingScraper(),
    ImageScraper(),
    VideoTagScraper(),
    AudioTagScraper(),
    VideoIframeScraper(),
    VideoEmbedScraper()
])

# now use page and remember page soup property must be set
content_pipeline.run(page)

Meta Scrape

in meta scrape we get page required meta such as robots (index and follow) status of page. meta title and meta description social tags (og:) page type and site name.

some scrapers of this pipeline have direct effect to properties of PageContext like is_title which means the title tag found in title tag

first lets explain each scraper

Title Scraper

this scraper tries to get page title from title tag.

if title tag found in page is_title property of page will be sets to true (it's false by default)

Description scraper

this scraper tries to find meta title with name "description".

if meta description found in page is_description property of page will be sets to true (it's false by default)

Robots scraper

sets page is_index and is_follow properties

Social scraper (OG)

social scrapers defines social title, description, site name, modified time and page type

usage

# import pipeline
from page_scraper import ContentPipeline

# import scrapers
from page_scraper import TitleScraper(),RobotScraper(),DescriptionScraper(),\
            OgTitleScraper(),OgDescriptionScraper(),OgTypeScraper(),\
            OgSiteNameScraper(),ModifiedTimeScraper()

# create pipeline
content_pipeline = ContentPipeline([
    TitleScraper(),
    RobotScraper(),
    DescriptionScraper(),
    OgTitleScraper(),
    OgDescriptionScraper(),
    OgTypeScraper(),
    OgSiteNameScraper(),
    ModifiedTimeScraper()
])

# now use page and remember page soup property must be set
content_pipeline.run(page)

new request method

from page_scraper import HttpFetcher

user_agent = "user agent"
timeout=20
fetcher = HttpFetcher(user_agent, timeout)

you can use default user agent or add yours. and timeout is 20 by default but you can add yours as well

then you can send request using http fetcher

from page_scraper import HttpFetcher
fetcher = HttpFetcher()

url="https://mydomain.ext/the-page-i-look-for/"
method="POST"
headers={"Authorization": "Bearer myjwttok3n"}
payload={"message":"hello"}
query_params={"page":1}

result = fetcher.request(url=url, method=method, headers=headers, payload=payload, query_params=query_params, allow_redirects=True)

Metadata

Release files for amina-page-scraper 1.4.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for amina-page-scraper 1.4.2
File Size Uploaded Provenance Actions
amina_page_scraper-1.4.2.tar.gz 15.4 kB Without provenance Details for amina_page_scraper-1.4.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for amina-page-scraper 1.4.2
File Interpreter ABI Platform Actions
amina_page_scraper-1.4.2-py3-none-any.whl Python 3 none any Details for amina_page_scraper-1.4.2-py3-none-any.whl

Total release size: 41.4 kB

Release files / amina_page_scraper-1.4.2.tar.gz

Download URL amina_page_scraper-1.4.2.tar.gz
Size 15.4 kB
Tags Source
SHA-256 checksum
How to use checksums
5f7376fb1ee446d19e7bbad63d8db36d8fe1972d1594798681c6720d4c6eebfc
BLAKE2b-256 checksum
How to use checksums
bbe8aabc3a287149034490a1424c515f7ff059d2eae49f203ef2fb1c78605f3e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.4

Release files / amina_page_scraper-1.4.2-py3-none-any.whl

Download URL amina_page_scraper-1.4.2-py3-none-any.whl
Size 26.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
b753321003b388abe18f07b1ac2930d0b65dcdceb446415ab2b26e28faf16b27
BLAKE2b-256 checksum
How to use checksums
1869162c8d52cee1cf6b9ffe57b1348cca945486b9b54707d83c6f9851dc8611
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.4

Security for amina-page-scraper 1.4.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page