Skip to main content

Analyze Scrapy Cloud data

Project description


PyPI PyPI - Python Version GitHub Build Status Codecov Code style: black GitHub commit activity

pip install arche

Arche (pronounced Arkey) helps to verify scraped data using set of defined rules, for example:

  • Validation with JSON schema
  • Coverage (items, fields, categorical data, including booleans and enums)
  • Duplicates
  • Garbage symbols
  • Comparison of two jobs

We use it in Scrapinghub, among the other tools, to ensure quality of scraped data


Arche requires Jupyter environment, supporting both JupyterLab and Notebook UI

For JupyterLab, you will need to properly install plotly extensions

Then just pip install arche


To check the quality of scraped data continuously. For example, if you scraped a website, a typical approach would be to validate the data with Arche. You can also create a schema and then set up Spidermon

Developer Setup

pipenv install --dev
pipenv shell


Any contributions are welcome! See if you want to take on something or suggest an improvement/report a bug.


Most recent releases are shown at the top. Each release shows:

  • Added: New classes, methods, functions, etc
  • Changed: Additional parameters, changes to inputs or outputs, etc
  • Fixed: Bug fixes that don't change documented behaviour

Note that the top-most release is changes in the unreleased master branch on Github. Parentheses after an item show the name or github id of the contributor of that change.

Keep a Changelog, Semantic Versioning.

[0.3.6] (2019-07-12)



  • Arche.report_all() does not shorten report by default, added short parameter.
  • Data is consistent with Dash and Spidermon: _type, _key fields are dropped from dataframe, raw data, basic schema, #104, #106
  • df.index now stores _key instead
  • basic_json_schema() works with deleted jobs
  • start is supported for Collections, #112
  • enum is counted as a category tag, #18
  • Garbage Symbols searches in str representation of nested fields instead of expanded df, #130
  • Show real coverage difference (negative\positive) instead of absolute, #114


  • Arche.glance(), #88
  • Item links in Schema validation errors, #89
  • Empty NAN bars on category graphs, #93
  • data_quality_report(), #95
  • Wrong number of Collection Items if it contains item 0, #112


  • Responses Per Item Ratio rule
  • Deprecated expand parameter and removed flat_df, since Garbage Rule deal with nested data itself, #133

[0.3.5] (2019-05-14)


  • Arche() supports any iterables with item dicts, fixing jsonschema consistency, #83
  • Items.from_array to read raw data from iterables, #83




[0.3.4] (2019-05-06)


  • basic_json_schema() fails with long 1.0 types, #80

[0.3.3] (2019-05-03)


  • Accept dataframes as source or target, #69


  • data_quality_report plots the same "Fields Coverage" instead of green "Scraped Fields Coverage"
  • Plot theme changed from ggplot2 to seaborn, #62
  • Same target and source raise an error, was a warning before
  • Passed rules marked with green PASSED.



  • Deprecated Arche.basic_json_schema(), use basic_json_schema()
  • Removed as redundant - documentation lives in notebooks

[0.3.2] (2019-04-18)


  • Allow reading private raw schemas directly from bitbucket, #58


  • Progress widgets are removed before printing graphs
  • New plotly v4 API


  • Failing Compare Prices For Same Urls when url is nan, #67
  • Empty graphs in Jupyter Notebook, #63


  • Scraped Items History graphs

[0.3.1] (2019-04-12)


  • Empty graphs due to lack of plotlyjs, #61

[0.3.0] (2019-04-12)


  • Big notebook size, replaced cufflinks with plotly and ipython, #39


  • Fields Coverage now is printed as a bar plot, #9
  • Fields Counts renamed to Coverage Difference and results in 2 bar plots, #9, #51:
    • Coverage from job stats fields counts which reflects coverage for each field for both jobs
    • Coverage difference more than 5% which prints >5% difference between the coverages (was ratio difference before)
  • Compare Scraped Categories renamed to Category Coverage Difference and results in 2 bar plots for each category, #52:
    • Coverage for field which reflects value counts (categories) coverage for the field for both jobs
    • Coverage difference more than 10% for field which shows >10% differences between the category coverages
  • Boolean Fields plots Coverage for boolean fields graph which reflects normalized value counts for boolean fields for both jobs, #53


  • cufflinks dependency
  • Deprecated category_field tag



  • new arche.rules.duplicates.find_by() to find duplicates by chosen columns
import arche
from arche.readers.items import JobItems
df = JobItems(0, "235801/1/15").df
arche.rules.duplicates.find_by(df, ["title", "category"]).show()
  • basic_json_schema().json() prints a schema in JSON format
  • to print a rule result, e.g.
from arche.rules.garbage_symbols import garbage_symbols
from arche.readers.items import JobItems
items = JobItems(0, "235801/1/15")
  • notebooks to documentation


  • Tags rule returns unused tags, #2
  • basic_json_schema() prints a schema as a python dict


  • Arche().basic_json_schema() deprecated in favor of arche.basic_json_schema()



  • Arche().basic_json_schema() not using items_numbers argument


  • Last release without CHANGES updates

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

arche-0.3.6.tar.gz (2.7 MB view hashes)

Uploaded source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page