pyvigate
Pyvigate: A Python framework that combines headless browsing with LLMs that assists you in your data solutions, product tours, building RAG applications, web automation, functional testing, and many more!
Installation
Pyvigate can be installed using pip or directly from the source for the latest version.
Using pip
pip install pyvigate
Installing from source
git clone https://github.com/kindsmiles/pyvigate.git
cd pyvigate
pip install .
Components
Pyvigate consists of several key components designed to work together seamlessly for web automation tasks.
PlayWrightEngine:
PlayWright is one library we use for headless browsing and other browser automation tasks.
from pyvigate.core.engine import PlaywrightEngine
engine = PlaywrightEngine(headless=True)
await engine.start_browser()
QueryEngine (with Azure OpenAI)
QueryEngine incorporates AI to dynamically detect web page elements, significantly improving the efficiency and reliability of automated interactions. It also can help the user navigate and also create their own applications, which involve curating data, creating RAG applications, product tour, functional testing, etc.
from pyvigate.ai.query_engine import QueryEngine
query_engine = QueryEngine(
api_key=os.getenv("OPENAI_API_KEY"),
azure_api_version=os.getenv("AZURE_API_VERSION"),
azure_endpoint=os.getenv("AZURE_ENDPOINT"),
azure_llm_deployment_name=os.getenv("LLM_DEPLOYMENT_NAME"),
azure_embedding_deployment_name=os.getenv("EMBEDDING_DEPLOYMENT_NAME")
)
Login
Some products can be accessed by the browser only after the login. We can do this either manually identifying the login selectors or letting the AI detect the UI elements where the credentials can be passed.The Login component utilizes QueryEngine to intelligently identify login forms and fields, streamlining the login process.
from pyvigate.core.login import Login
login = Login(query_engine)
await login.perform_login(engine.page, "https://example.com/login", "username", "password")
Scraping
With Scraping, Pyvigate offers powerful data extraction capabilities, enabling the collection of content from web pages post-login or navigation.
from pyvigate.services.scraping import Scraping
scraping = Scraping(data_dir="data")
content = await scraping.extract_data_from_page(engine.page)
print("Scraped content:", content)
Caching
The Caching component allows for the local storage of web page content, facilitating offline analysis and reducing bandwidth usage.
from pyvigate.services.caching import Caching
caching = Caching(cache_dir="html_cache")
await caching.cache_page_content(engine.page, "https://example.com/page")
Full Example
Bringing it all together, here's how you can use Pyvigate to login, scrape content, and cache it:
import asyncio
from dotenv import load_dotenv
from pyvigate.core.engine import PlaywrightEngine
from pyvigate.core.login import Login
from pyvigate.services.scraping import Scraping
from pyvigate.services.caching import Caching
from pyvigate.ai.query_engine import QueryEngine
import os
load_dotenv()
async def login_and_scrape():
engine = PlaywrightEngine(headless=True)
await engine.start_browser()
query_engine = QueryEngine(api_key=os.getenv("OPENAI_API_KEY"))
login = Login(query_engine)
await login.perform_login(engine.page, "https://example.com/login", os.getenv("USERNAME"), os.getenv("PASSWORD"))
scraping = Scraping(data_dir="data")
content = await scraping.extract_data_from_page(engine.page)
print("Scraped content:", content)
caching = Caching(cache_dir="html_cache")
await caching.cache_page_content(engine.page, "https://example.com/dashboard")
await engine.stop_browser()
if __name__ == "__main__":
asyncio.run(login_and_scrape())
Donations
We are looking to cover some of our API costs, so we accept any donations that you can make:
Metadata
Release files for pyvigate 0.0.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| pyvigate-0.0.3.tar.gz | 11.8 kB | Without provenance | Details for pyvigate-0.0.3.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| pyvigate-0.0.3-py3-none-any.whl | Python 3 | none | any | Details for pyvigate-0.0.3-py3-none-any.whl |
Total release size: 25.0 kB
Release files / pyvigate-0.0.3.tar.gz
| Download URL | pyvigate-0.0.3.tar.gz |
|---|---|
| Size | 11.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
db0d38a68374a0e07487fd976354d7e17f6a0e76564a9084ab9ee14953244cb2
|
|
BLAKE2b-256 checksum How to use checksums |
c85d70ac60f058223b0c31bd06f819e410c5c7814f2b9bc2623f46147ec62290
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.9.16
|
Release files / pyvigate-0.0.3-py3-none-any.whl
| Download URL | pyvigate-0.0.3-py3-none-any.whl |
|---|---|
| Size | 13.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
5f6166af282274b1a0ca859f89c1761bf7291d6cbf3bf0d781a6d92f83da21ad
|
|
BLAKE2b-256 checksum How to use checksums |
b6d8e4e2609d8371aa376730ac815c114b6cbad346cebb5224cf41c3e519ffeb
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.9.16
|
Security for pyvigate 0.0.3
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.