Skip to main content

pyvigate

Pyvigate: A Python framework that combines headless browsing with LLMs that assists you in your data solutions, product tours, building RAG applications, web automation, functional testing, and many more!

DownloadsDownloads Documentation

Installation

Pyvigate can be installed using pip or directly from the source for the latest version.

Using pip

pip install pyvigate

Installing from source

git clone https://github.com/kindsmiles/pyvigate.git
cd pyvigate
pip install .

Components

Pyvigate consists of several key components designed to work together seamlessly for web automation tasks.

PlayWrightEngine:

PlayWright is one library we use for headless browsing and other browser automation tasks.

from pyvigate.core.engine import PlaywrightEngine

engine = PlaywrightEngine(headless=True)
await engine.start_browser()

QueryEngine (with Azure OpenAI)

QueryEngine incorporates AI to dynamically detect web page elements, significantly improving the efficiency and reliability of automated interactions. It also can help the user navigate and also create their own applications, which involve curating data, creating RAG applications, product tour, functional testing, etc.

from pyvigate.ai.query_engine import QueryEngine

query_engine = QueryEngine(
    api_key=os.getenv("OPENAI_API_KEY"),
    azure_api_version=os.getenv("AZURE_API_VERSION"),
    azure_endpoint=os.getenv("AZURE_ENDPOINT"),
    azure_llm_deployment_name=os.getenv("LLM_DEPLOYMENT_NAME"),
    azure_embedding_deployment_name=os.getenv("EMBEDDING_DEPLOYMENT_NAME")
)

Login

Some products can be accessed by the browser only after the login. We can do this either manually identifying the login selectors or letting the AI detect the UI elements where the credentials can be passed.The Login component utilizes QueryEngine to intelligently identify login forms and fields, streamlining the login process.

from pyvigate.core.login import Login

login = Login(query_engine)
await login.perform_login(engine.page, "https://example.com/login", "username", "password")

Scraping

With Scraping, Pyvigate offers powerful data extraction capabilities, enabling the collection of content from web pages post-login or navigation.

from pyvigate.services.scraping import Scraping

scraping = Scraping(data_dir="data")
content = await scraping.extract_data_from_page(engine.page)
print("Scraped content:", content)

Caching

The Caching component allows for the local storage of web page content, facilitating offline analysis and reducing bandwidth usage.

from pyvigate.services.caching import Caching

caching = Caching(cache_dir="html_cache")
await caching.cache_page_content(engine.page, "https://example.com/page")

Full Example

Bringing it all together, here's how you can use Pyvigate to login, scrape content, and cache it:

import asyncio
from dotenv import load_dotenv
from pyvigate.core.engine import PlaywrightEngine
from pyvigate.core.login import Login
from pyvigate.services.scraping import Scraping
from pyvigate.services.caching import Caching
from pyvigate.ai.query_engine import QueryEngine
import os

load_dotenv()

async def login_and_scrape():
    engine = PlaywrightEngine(headless=True)
    await engine.start_browser()

    query_engine = QueryEngine(api_key=os.getenv("OPENAI_API_KEY"))
    login = Login(query_engine)
    await login.perform_login(engine.page, "https://example.com/login", os.getenv("USERNAME"), os.getenv("PASSWORD"))

    scraping = Scraping(data_dir="data")
    content = await scraping.extract_data_from_page(engine.page)
    print("Scraped content:", content)

    caching = Caching(cache_dir="html_cache")
    await caching.cache_page_content(engine.page, "https://example.com/dashboard")

    await engine.stop_browser()

if __name__ == "__main__":
    asyncio.run(login_and_scrape())

Donations

We are looking to cover some of our API costs, so we accept any donations that you can make:

Sponsor me on Buy Me a Coffee

Metadata

Release files for pyvigate 0.0.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pyvigate 0.0.3
File Size Uploaded Provenance Actions
pyvigate-0.0.3.tar.gz 11.8 kB Without provenance Details for pyvigate-0.0.3.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pyvigate 0.0.3
File Interpreter ABI Platform Actions
pyvigate-0.0.3-py3-none-any.whl Python 3 none any Details for pyvigate-0.0.3-py3-none-any.whl

Total release size: 25.0 kB

Release files / pyvigate-0.0.3.tar.gz

Download URL pyvigate-0.0.3.tar.gz
Size 11.8 kB
Tags Source
SHA-256 checksum
How to use checksums
db0d38a68374a0e07487fd976354d7e17f6a0e76564a9084ab9ee14953244cb2
BLAKE2b-256 checksum
How to use checksums
c85d70ac60f058223b0c31bd06f819e410c5c7814f2b9bc2623f46147ec62290
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/5.0.0 CPython/3.9.16

Release files / pyvigate-0.0.3-py3-none-any.whl

Download URL pyvigate-0.0.3-py3-none-any.whl
Size 13.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
5f6166af282274b1a0ca859f89c1761bf7291d6cbf3bf0d781a6d92f83da21ad
BLAKE2b-256 checksum
How to use checksums
b6d8e4e2609d8371aa376730ac815c114b6cbad346cebb5224cf41c3e519ffeb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/5.0.0 CPython/3.9.16

Security for pyvigate 0.0.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page