from requests_toolbelt import user_agent
Amina Group Page Scraper
this package is help you to get seo information on a webpage. the types are info are:
JSON-LD schema info
json ld schema is a graph which is used to define what is the web page and what can you find in this webpage. we use detectors to find info on page and categorize them.
Article
the types on page which are defined as article are
- "Article"
- "NewsArticle",
- "TechArticle",
- "ScholarlyArticle",
- "BlogPosting",
- "Report",
- "Blog"
these types on page are identified as article
Contact and About
for contact pages we have
- "ContactPage"
- "AboutPage"
Domain and Organization
for pages which are root of the domain we have domain info and organization which are specify the brand of the page
for example name of organization which have url is identified as brand name of the website
Other Types
- navigation
- person
- product
- service
- category
are other types which we identify as ld json schema all these are gathered as ENTITY
Entity
entity is a data type we provide which have these fields
- id: (string)
- name: (string) can be null
- type: (string) this field defined by type detectors which are explained above
- alternate_name: (string) can be null
- url: (string) can be null
- description: (string) can be null
- raw: (dictionary)
- date_published: (date time) can be null
- date_modified: (date time) can be null
usage
#import Entity pipeline
from page_scraper import EntityPipeline
#import detectors you need
from page_scraper import ArticleDetector,DomainDetector
#import http fetcher and parser to fetch the page correctly and parse the html
from page_scraper import HttpFetcher, PageParser
from page_scraper import build_page_context
#from builders import page builder
# define url
url = "https://amina-group.com/"
# get and parse the page to create a Page instance
fetcher = HttpFetcher()
parser = PageParser()
fetched_url = fetcher.get(url)
parsed_info = parser.parse(fetched_url)
# create page instance using build_page_context
page = build_page_context(parsed_info)
# create pipeline with detectors you want
pipeline = EntityPipeline(
[
ArticleDetector(),
DomainDetector(),
]
)
# run the pipeline using page
pipeline.run(page)
Link scrape
we let you classify the links on page but first before using this you need to add Beautiful soup object as soup property of the page
BeautifulSoup instance
from bs4 import BeautifulSoup
page.soup = BeautifulSoup(fetched_url.content, "html.parser")
links of the page
every url in page (except the links on ld json) are stored as UrlContext data type
for main urls we have:
- canonical (stores as canonical property of page as well)
- navigation
- internal
you can use scraper pipeline to get each or all urls
usage
# import pipeline
from page_scraper import ScraperPipeline
# import scrapers
from page_scraper import GetNavLinks(),InternalScraper(),CanonicalScraper()
# create pipeline
scrape_pipeline = ScraperPipeline([
GetNavLinks(),
InternalScraper(),
CanonicalScraper(),
])
# now use page and remember page soup property must be set
scrape_pipeline.run(page)
Content Scrape
what need of content is headers from h1,h2,h3,h4,h5,h6 we want links and the type of links images videos audios
headings
headings are also have a data type which is PageHeading and after scraping each heading stored as PageHeading instance
the page have a property named headings page.headings and heading scraper stores heading of the page to this property
media tags
media tags are also url and they will be treated as UrlContext as well the media types are:
- image
- video
- audio
each media tag in url context define as {meida_name}_TAG for example IMAGE_TAG for image
you can get the video of a page with 3 different ways
- video tag
- iframe
- embed or object tag
NOTE: you cannot get iframe by a normal url fetcher because iframe will only loads if javascript available.
usage
# import pipeline
from page_scraper import ContentPipeline
# import scrapers
from page_scraper import HeadingScraper(),\
ImageScraper(),VideoTagScraper(),AudioTagScraper(),\
VideoIframeScraper(),VideoEmbedScraper()
# create pipeline
content_pipeline = ContentPipeline([
HeadingScraper(),
ImageScraper(),
VideoTagScraper(),
AudioTagScraper(),
VideoIframeScraper(),
VideoEmbedScraper()
])
# now use page and remember page soup property must be set
content_pipeline.run(page)
Meta Scrape
in meta scrape we get page required meta such as robots (index and follow) status of page. meta title and meta description social tags (og:) page type and site name.
some scrapers of this pipeline have direct effect to properties of PageContext like is_title which means the title tag found in title tag
first lets explain each scraper
Title Scraper
this scraper tries to get page title from title tag.
if title tag found in page is_title property of page will be sets to true (it's false by default)
Description scraper
this scraper tries to find meta title with name "description".
if meta description found in page is_description property of page will be sets to true (it's false by default)
Robots scraper
sets page is_index and is_follow properties
Social scraper (OG)
social scrapers defines social title, description, site name, modified time and page type
usage
# import pipeline
from page_scraper import ContentPipeline
# import scrapers
from page_scraper import TitleScraper(),RobotScraper(),DescriptionScraper(),\
OgTitleScraper(),OgDescriptionScraper(),OgTypeScraper(),\
OgSiteNameScraper(),ModifiedTimeScraper()
# create pipeline
content_pipeline = ContentPipeline([
TitleScraper(),
RobotScraper(),
DescriptionScraper(),
OgTitleScraper(),
OgDescriptionScraper(),
OgTypeScraper(),
OgSiteNameScraper(),
ModifiedTimeScraper()
])
# now use page and remember page soup property must be set
content_pipeline.run(page)
new request method
from page_scraper import HttpFetcher
user_agent = "user agent"
timeout=20
fetcher = HttpFetcher(user_agent, timeout)
you can use default user agent or add yours. and timeout is 20 by default but you can add yours as well
then you can send request using http fetcher
from page_scraper import HttpFetcher
fetcher = HttpFetcher()
url="https://mydomain.ext/the-page-i-look-for/"
method="POST"
headers={"Authorization": "Bearer myjwttok3n"}
payload={"message":"hello"}
query_params={"page":1}
result = fetcher.request(url=url, method=method, headers=headers, payload=payload, query_params=query_params, allow_redirects=True)
Metadata
Release files for amina-page-scraper 1.4.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| amina_page_scraper-1.4.3.tar.gz | 15.4 kB | Without provenance | Details for amina_page_scraper-1.4.3.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| amina_page_scraper-1.4.3-py3-none-any.whl | Python 3 | none | any | Details for amina_page_scraper-1.4.3-py3-none-any.whl |
Total release size: 41.4 kB
Release files / amina_page_scraper-1.4.3.tar.gz
| Download URL | amina_page_scraper-1.4.3.tar.gz |
|---|---|
| Size | 15.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
0e2d76ce3d34eb8bf41faf39945e56b50ecaf234827fe8266a20001a43662172
|
|
BLAKE2b-256 checksum How to use checksums |
ccf80a4ec40c2b634614f27ad2777186619678bd8bd4f45771c3387e6df736fd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.4
|
Release files / amina_page_scraper-1.4.3-py3-none-any.whl
| Download URL | amina_page_scraper-1.4.3-py3-none-any.whl |
|---|---|
| Size | 26.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
26276b32f450102e91668209f62a7b0c7978949747e78888d7907dbedd70fbc5
|
|
BLAKE2b-256 checksum How to use checksums |
feb90a7e7130c253acc48137f1cc07353b829f9a1eb7c739e312f40e86989955
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.4
|
Security for amina-page-scraper 1.4.3
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.