Skip to main content

An intelligent web service to automatically detect web content and extract information from it.

Project description

Webspot

Webspot is an intelligent web service to automatically detect web content and extract information from it.

Demo

中文

Screenshots

Detected Results

Extracted Fields

Extracted Data

Get Started

Docker

Make sure you have installed Docker and Docker Compose.

# clone git repo
git clone https://github.com/crawlab-team/webspot

# start docker containers
docker-compose up -d

Then you can access the web UI at http://localhost:9999.

API Reference

Once you started Webspot, you can go to http://localhost:9999/redoc to view the API reference.

Architecture

The overall process of how Webspot detects meaningful elements from HTML or web pages is shown in the following figure.

graph LR
    hr[HtmlRequester]
    gl[GraphLoader]
    d[Detector]
    r[Results]

    hr --"html + json"--> gl --"graph"--> d --"output"--> r

Development

You can follow the following guidance to get started.

Pre-requisites

  • Python >=3.8 and <=3.10
  • Go 1.16 or higher
  • MongoDB 4.2 or higher

Install dependencies

# dependencies
pip install -r requirements.txt

Configure Environment Variables

Database configuration is located in .env file. You can copy the example file and modify it.

cp .env.example .env

Start web server

# start development server
python main.py web

Code Structure

The core code is located in webspot directory. The main.py file is the entry point of the web server.

webspot
├── cmd     # command line tools
├── crawler # web crawler
├── data    # data files (html, json, etc.)
├── db      # database
├── detect  # web content detection
├── graph   # graph module
├── models  # models
├── request # request helper
├── test    # test cases
├── utils   # utilities
└── web     # web server

TODOs

Webspot is aimed at automating the process of web content detection and extraction. It is far from ready for production use. The following features are planned to be implemented in the future.

  • Table detection
  • Nested list detection
  • Export to spiders
  • Advanced browser request

Disclaimer

Please follow the local laws and regulations when using Webspot. The author is not responsible for any legal issues caused by. Please read the Disclaimer for details.

Community

If you are interested in Webspot, please add the author's WeChat account "tikazyq1" noting "Webspot" to enter the discussion group.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

webspot-0.1.4.tar.gz (34.3 kB view hashes)

Uploaded Source

Built Distribution

webspot-0.1.4-py3-none-any.whl (47.9 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page