langcrawler
langcrawler is a language crawler written in Python.
See Lang Fetcher as a language fetcher from langcrawler.
Requirement
- PostgreSQL >= 12.4
- Python >= 3.8
- Redis >= 6.0
Installation
Ubuntu
apt update
apt install -y python3-dev python3-pip python3-setuptools
pip install langcrawler
Windows
# Install Python, Microsoft Visual C++ and Windows SDK
pip install langcrawler
Updating
pip install langcrawler --upgrade
Running
langcrawler \
--pg-address 127.0.0.1:5432 \
--pg-login postgres/postgres \
--redis-address 127.0.0.1:6379 \
--redis-pass redis \
--repo-count 1 \
--repo-host gerrit,github,gitlab \
--repo-lang go,javascript,php,python,rust,typescript
Usage
usage: crawler.py [-h] [--pg-address PG_ADDRESS] [--pg-login PG_LOGIN]
[--redis-address REDIS_ADDRESS] [--redis-pass REDIS_PASS]
[--repo-count REPO_COUNT] [--repo-host REPO_HOST]
[--repo-lang REPO_LANG] [-v]
Language Crawler
optional arguments:
-h, --help show this help message and exit
--pg-address PG_ADDRESS
postgres address (host:port)
--pg-login PG_LOGIN postgres login (user/pass)
--redis-address REDIS_ADDRESS
redis address (host:port)
--redis-pass REDIS_PASS
redis pass
--repo-count REPO_COUNT
repository count
--repo-host REPO_HOST
repository host
--repo-lang REPO_LANG
repository language
-v, --version show program's version number and exit
Design
License
Project License can be found here.
Reference
Metadata
Release files for langcrawler 0.0.4
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| langcrawler-0.0.4.tar.gz | 11.6 kB | Without provenance | Details for langcrawler-0.0.4.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| langcrawler-0.0.4-py2.py3-none-any.whl | Python 2, Python 3 | none | any | Details for langcrawler-0.0.4-py2.py3-none-any.whl |
Total release size: 28.5 kB
Release files / langcrawler-0.0.4.tar.gz
| Download URL | langcrawler-0.0.4.tar.gz |
|---|---|
| Size | 11.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c70476d723f84739b84fd8bec2c3fadad0077c2889dbbeb870961e657409bdab
|
|
BLAKE2b-256 checksum How to use checksums |
cc4c34cd72717d5a066ffcfbdc8f05970d861d1596ecc3279608dd447746c608
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/50.3.0 requests-toolbelt/0.9.1 tqdm/4.50.0 CPython/3.6.9
|
Release files / langcrawler-0.0.4-py2.py3-none-any.whl
| Download URL | langcrawler-0.0.4-py2.py3-none-any.whl |
|---|---|
| Size | 16.9 kB |
| Tags | Python 2 Python 3 |
|
SHA-256 checksum How to use checksums |
0fb4547acb835f42b3c6fe5eb0573e1e03901b02ba715750616c030aec8e2fca
|
|
BLAKE2b-256 checksum How to use checksums |
f87eb3dd26d76d51dc70b8abf31741b6b6af9bc79e62da6a5d8d1a7d3882ddce
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/50.3.0 requests-toolbelt/0.9.1 tqdm/4.50.0 CPython/3.6.9
|
Security for langcrawler 0.0.4
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.