Language Crawler
Project description
langcrawler
langcrawler is a language crawler written in Python.
Requirement
- PostgreSQL >= 12.4
- Python >= 3.8
- Redis >= 6.0
Installation
Ubuntu
apt update
apt install -y python3-dev python3-pip python3-setuptools
pip install langcrawler
Windows
# Install Python, Microsoft Visual C++ and Windows SDK
pip install langcrawler
Updating
pip install langcrawler --upgrade
Running
langcrawler \
--lang-type go,javascript,php,python,rust,typescript \
--pg-address 127.0.0.1:5432 \
--pg-login postgres/postgres \
--redis-address 127.0.0.1:6379 \
--redis-pass redis \
--repo-count 10 \
--repo-host gerrit,github,gitlab
Usage
usage: langcrawler [-h] [--lang-type LANG_TYPE] [--pg-address PG_ADDRESS]
[--pg-login PG_LOGIN] [--redis-address REDIS_ADDRESS]
[--redis-pass REDIS_PASS] [--repo-count REPO_COUNT]
[--repo-host REPO_HOST] [-v]
Language Crawler
optional arguments:
-h, --help show this help message and exit
--lang-type LANG_TYPE
language type
--pg-address PG_ADDRESS
postgres address (host:port)
--pg-login PG_LOGIN postgres login (user/pass)
--redis-address REDIS_ADDRESS
redis address (host:port)
--redis-pass REDIS_PASS
redis pass
--repo-count REPO_COUNT
repository count
--repo-host REPO_HOST
repository host
-v, --version show program's version number and exit
License
Project License can be found here.
Reference
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
langcrawler-0.0.1.tar.gz
(10.5 kB
view hashes)
Built Distribution
Close
Hashes for langcrawler-0.0.1-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dd7eefc5b293ea065eeccc47049042af99b35564e4d5a7c37936f809ef4fc24f |
|
MD5 | b2871a9293a9b3192214220f96f72f29 |
|
BLAKE2b-256 | 238a478f3443d0cfeb619c7081ebf0a53fdbd8f048ff0486261612be9113f080 |