Crawler and search tools used by Sirji.
Project description
Sirji is an Open Source AI Software Development Agent.
Built with ❤️ by True Sparrow
Sirji Tools
sirji-tools
is a PyPI package that provides tools for:
- Crawling (downloading web pages to markdown files)
- Searching on Google
- Custom Logging
Installation
Setup Virtual Environment
We recommend setting up a virtual environment to isolate Python dependencies, ensuring project-specific packages without conflicting with system-wide installations.
python3 -m venv venv
source venv/bin/activate
Install Package
Install the package from PyPi:
pip install sirji-tools
Usage
Environment Variables
Ensure that following environment variables are set:
export SIRJI_WORKSPACE="Absolute folder path for Sirji to use as it's workspace. Note that a .sirji folder will be created inside it."
export SIRJI_RUN_ID='Unique alphanumeric ID for each run. Note that a sub folder named by this ID will be created inside of .sirji folder to store logs, etc.'
Crawl URLs
Crawl URLs tool will be used to crawl the web pages and extract the information from the web pages. And store the information for the further processing by researcher.
from sirji_tools import crawl_urls
urls = ['https://www.google.com', 'https://www.yahoo.com']
crawl_urls(urls, 'workspace/researcher')
Search
Search tool will be used to search the information from the web pages based on the search terms provided. It returns the list of URLs related to the search terms.
from sirji_tools import search_for
search_term = 'python programming'
urls = search_for(search_term)
Logger
Logger tool will be used to log the information in the log file. It will be used to log the information to show the progress of the execution.
from sirji_tools.logger import p_logger
p_logger.info("Log line here")
For Contributors
- Fork and clone the repository.
- Create and activate the virtual environment as described above.
- Set the environment variables as described above.
- Install the package in editable mode by running the following command from repository root:
pip install -e .
Running Tests and Coverage Analysis
Tests can be run, and coverage can be analyzed in a few simple steps:
# Install testing dependencies
pip install pytest coverage
# Execute tests
pytest
# Measure coverage, excluding test files
coverage run --omit="tests/*" -m pytest
coverage report
License
Distributed under the MIT License. See LICENSE
for more information.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for sirji_tools-0.0.7-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7536bdba533a5a9d13c5d8cd9663240594bc189e8aa3d571bf83e9679d0db5e8 |
|
MD5 | 558b067ad4ecb68915872eff9c287d7e |
|
BLAKE2b-256 | aacd1aae5ced911a14bd01f33134393f05fd852b4c33fd06b1582432b389ac29 |