Declarative web parsers
Project description
Soup Stars
Soup Stars is a framework for building web parsers with Python. It is designed to make building, deploying, and scheduling web parsers easier by simplifying what you need to get started.
Quickstart
pip install soupstars
The client is also available as a docker image.
docker pull soupstars/client
Building a parser
Create a new parser using the soupstars
command. The create
command will use a template parser.
soupstars create -m myparser.py
Parsers are simple python modules.
cat myparser.py
Notice that the only set up required is the special parse
decorator and a variable named url
for the web page you want to parse.
from soupstars import parse
url = "https://corbettanalytics.com/"
@parse
def h1(soup):
return soup.h1.text
You can test that the parser functions correctly.
soupstars run -m myparser.py
Use soupstars --help
to see a full list of available commands.
More documentation is available here.
Development
Start the docker services.
docker-compose up -d
Run the tests.
docker-compose run --rm client pytest -vs
Releasing
New tags that pass on CI will automatically be pushed to docker hub.
To deploy to PyPI requires manually running the following commands.
pip3 install twine
python3 setup.py sdist bdist_wheel
twine upload dist/*
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for soupstars-2.11.17-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2c10bf9bb8006206ee0c311f8c6c537bae8dcf9fcaa72b9b9b474b0f5951ccdb |
|
MD5 | c2060a9895c20a1234012064e3c32480 |
|
BLAKE2b-256 | 24a0e3e6f2ac5efc728d29d962d64c2256fc2ac7a54d958197ea699f2e5f3385 |