arche 0.3.6

Analyze Scrapy Cloud data

Arche

pip install arche

Arche (pronounced Arkey) helps to verify scraped data using set of defined rules, for example:

• Validation with JSON schema
• Coverage (items, fields, categorical data, including booleans and enums)
• Duplicates
• Garbage symbols
• Comparison of two jobs

We use it in Scrapinghub, among the other tools, to ensure quality of scraped data

Installation

Arche requires Jupyter environment, supporting both JupyterLab and Notebook UI

For JupyterLab, you will need to properly install plotly extensions

Then just pip install arche

Why

To check the quality of scraped data continuously. For example, if you scraped a website, a typical approach would be to validate the data with Arche. You can also create a schema and then set up Spidermon

Developer Setup

pipenv install --dev
pipenv shell
tox

Contribution

Any contributions are welcome! See https://github.com/scrapinghub/arche/issues if you want to take on something or suggest an improvement/report a bug.

Changes

Most recent releases are shown at the top. Each release shows:

• Added: New classes, methods, functions, etc
• Changed: Additional parameters, changes to inputs or outputs, etc
• Fixed: Bug fixes that don't change documented behaviour

Note that the top-most release is changes in the unreleased master branch on Github. Parentheses after an item show the name or github id of the contributor of that change.

Keep a Changelog, Semantic Versioning.

[0.3.6] (2019-07-12)

Added

• Categories rule with a plot showing unique values and count per field. By default, report_all() only includes fields which have less or equal to 10 unique values. See https://arche.readthedocs.io/en/latest/nbs/Rules.html#Category-fields, #100
• Category documentation

Changed

• Arche.report_all() does not shorten report by default, added short parameter.
• Data is consistent with Dash and Spidermon: _type, _key fields are dropped from dataframe, raw data, basic schema, #104, #106
• df.index now stores _key instead
• basic_json_schema() works with deleted jobs
• start is supported for Collections, #112
• enum is counted as a category tag, #18
• Garbage Symbols searches in str representation of nested fields instead of expanded df, #130
• Show real coverage difference (negative\positive) instead of absolute, #114

Fixed

• Arche.glance(), #88
• Item links in Schema validation errors, #89
• Empty NAN bars on category graphs, #93
• data_quality_report(), #95
• Wrong number of Collection Items if it contains item 0, #112

Removed

• Responses Per Item Ratio rule
• Deprecated expand parameter and removed flat_df, since Garbage Rule deal with nested data itself, #133

[0.3.5] (2019-05-14)

Added

• Arche() supports any iterables with item dicts, fixing jsonschema consistency, #83
• Items.from_array to read raw data from iterables, #83

Changed

• If reading from pandas df directly, store raw data in numpy array. See gotchas http://pandas.pydata.org/pandas-docs/stable/user_guide/gotchas.html#support-for-integer-na

Fixed

Removed

[0.3.4] (2019-05-06)

Fixed

• basic_json_schema() fails with long 1.0 types, #80

[0.3.3] (2019-05-03)

Added

• Accept dataframes as source or target, #69

Changed

• data_quality_report plots the same "Fields Coverage" instead of green "Scraped Fields Coverage"
• Plot theme changed from ggplot2 to seaborn, #62
• Same target and source raise an error, was a warning before
• Passed rules marked with green PASSED.

Fixed

• Online documentation now renders graphs https://arche.readthedocs.io/en/latest/, #41
• Error colours are back in report_all().

Removed

• Deprecated Arche.basic_json_schema(), use basic_json_schema()
• Removed Quickstart.md as redundant - documentation lives in notebooks

[0.3.2] (2019-04-18)

Added

• Allow reading private raw schemas directly from bitbucket, #58

Changed

• Progress widgets are removed before printing graphs
• New plotly v4 API

Fixed

• Failing Compare Prices For Same Urls when url is nan, #67
• Empty graphs in Jupyter Notebook, #63

Removed

• Scraped Items History graphs

[0.3.1] (2019-04-12)

Fixed

• Empty graphs due to lack of plotlyjs, #61

[0.3.0] (2019-04-12)

Fixed

• Big notebook size, replaced cufflinks with plotly and ipython, #39

Changed

• Fields Coverage now is printed as a bar plot, #9
• Fields Counts renamed to Coverage Difference and results in 2 bar plots, #9, #51:
  • Coverage from job stats fields counts which reflects coverage for each field for both jobs
  • Coverage difference more than 5% which prints >5% difference between the coverages (was ratio difference before)
• Compare Scraped Categories renamed to Category Coverage Difference and results in 2 bar plots for each category, #52:
  • Coverage for field which reflects value counts (categories) coverage for the field for both jobs
  • Coverage difference more than 10% for field which shows >10% differences between the category coverages
• Boolean Fields plots Coverage for boolean fields graph which reflects normalized value counts for boolean fields for both jobs, #53

Removed

• cufflinks dependency
• Deprecated category_field tag

[2019.03.25]

Added

• CHANGES.md
• new arche.rules.duplicates.find_by() to find duplicates by chosen columns

import arche
from arche.readers.items import JobItems
df = JobItems(0, "235801/1/15").df
arche.rules.duplicates.find_by(df, ["title", "category"]).show()

• basic_json_schema().json() prints a schema in JSON format
• Result.show() to print a rule result, e.g.

from arche.rules.garbage_symbols import garbage_symbols
from arche.readers.items import JobItems
items = JobItems(0, "235801/1/15")
garbage_symbols(items).show()

• notebooks to documentation

Changed

• Tags rule returns unused tags, #2
• basic_json_schema() prints a schema as a python dict

Deprecated

• Arche().basic_json_schema() deprecated in favor of arche.basic_json_schema()

Removed

Fixed

• Arche().basic_json_schema() not using items_numbers argument

2019.03.18

• Last release without CHANGES updates