Crawler and search tools used by Sirji.
Project description
sirji-tools
sirji-tools is a PyPI package that provides tools for:
- Crawling (downloading web pages to markdown files)
- Searching (using the Google Search API for search results)
- Custom Logging
Installation
pip install sirji-tools
Usage
Crawl URLs
Crawl URLs tool will be used to crawl the web pages and extract the information from the web pages. And store the information for the further processing by researcher.
from sirji_tools import crawl_urls
urls = ['https://www.google.com', 'https://www.yahoo.com']
crawl_urls(urls, 'workspace/researcher')
Search
Search tool will be used to search the information from the web pages based on the search terms provided. It returns the list of URLs related to the search terms.
from sirji_tools import search_for
search_term = 'python programming'
urls = search_for(search_term)
Logger
Logger tool will be used to log the information in the log file. It will be used to log the information to show the progress of the execution.
from sirji_tools.logger import p_logger
p_logger.info("Log line here")
License
Distributed under the MIT License. See LICENSE for more information.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
sirji-tools-0.0.4.tar.gz
(9.3 kB
view hashes)
Built Distribution
Close
Hashes for sirji_tools-0.0.4-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | cd1c11ff2c5015f2894c274bd8899eaf194209a35d36f31690a5a029a49a241e |
|
| MD5 | ae1cdda61ab807a418cfd769ee1122b9 |
|
| BLAKE2b-256 | 6f3b09d1a61d6384618b3ad573224a637bd12112e32125235795966f3080c6d1 |
