Skip to main content

Python library to work with CSVW described tabular data

Project description

csvw

Build Status PyPI Documentation Status

This package provides

  • a Python API to read and write relational, tabular data according to the CSV on the Web specification and
  • commandline tools for reading and validating CSVW data.

Links

Installation

This package runs under Python >=3.8, use pip to install:

$ pip install csvw

CLI

csvw2json

Converting CSVW data to JSON

$ csvw2json tests/fixtures/zipped-metadata.json 
{
    "tables": [
        {
            "url": "tests/fixtures/zipped.csv",
            "row": [
                {
                    "url": "tests/fixtures/zipped.csv#row=2",
                    "rownum": 1,
                    "describes": [
                        {
                            "ID": "abc",
                            "Value": "the value"
                        }
                    ]
                },
                {
                    "url": "tests/fixtures/zipped.csv#row=3",
                    "rownum": 2,
                    "describes": [
                        {
                            "ID": "cde",
                            "Value": "another one"
                        }
                    ]
                }
            ]
        }
    ]
}

csvwvalidate

Validating CSVW data

$ csvwvalidate tests/fixtures/zipped-metadata.json 
OK

csvwdescribe

Describing tabular-data files with CSVW metadata

$ csvwdescribe --delimiter "|" tests/fixtures/frictionless-data.csv
{
    "@context": "http://www.w3.org/ns/csvw",
    "dc:conformsTo": "data-package",
    "tables": [
        {
            "dialect": {
                "delimiter": "|"
            },
            "tableSchema": {
                "columns": [
                    {
                        "datatype": "string",
                        "name": "FK"
                    },
                    {
                        "datatype": "integer",
                        "name": "Year"
                    },
                    {
                        "datatype": "string",
                        "name": "Location name"
                    },
                    {
                        "datatype": "string",
                        "name": "Value"
                    },
                    {
                        "datatype": "string",
                        "name": "binary"
                    },
                    {
                        "datatype": "string",
                        "name": "anyURI"
                    },
                    {
                        "datatype": "string",
                        "name": "email"
                    },
                    {
                        "datatype": "string",
                        "name": "boolean"
                    },
                    {
                        "datatype": {
                            "dc:format": "application/json",
                            "base": "json"
                        },
                        "name": "array"
                    },
                    {
                        "datatype": {
                            "dc:format": "application/json",
                            "base": "json"
                        },
                        "name": "geojson"
                    }
                ]
            },
            "url": "tests/fixtures/frictionless-data.csv"
        }
    ]
}

Python API

Find the Python API documentation at csvw.readthedocs.io.

A quick example for using csvw from Python code:

import json
from csvw import CSVW
data = CSVW('https://raw.githubusercontent.com/cldf/csvw/master/tests/fixtures/test.tsv')
print(json.dumps(data.to_json(minimal=True), indent=4))
[
    {
        "province": "Hello",
        "territory": "world",
        "precinct": "1"
    }
]

Known limitations

  • We read all data which is specified as UTF-8 encoded using the utf-8-sig codecs. Thus, if such data starts with U+FEFF this will be interpreted as BOM and skipped.
  • Low level CSV parsing is delegated to the csv module in Python's standard library. Thus, if a commentPrefix is specified in a Dialect instance, this will lead to skipping rows where the first value starts with commentPrefix, even if the value was quoted.
  • Also, cell content containing escapechar may not be round-tripped as expected (when specifying escapechar or a csvw.Dialect with quoteChar but doubleQuote==False), when minimal quoting is specified. This is due to inconsistent csv behaviour across Python versions (see https://bugs.python.org/issue44861).

CSVW conformance

While we use the CSVW specification as guideline, this package does not (and probably never will) implement the full extent of this spec.

  • When CSV files with a header are read, columns are not matched in order with column descriptions in the tableSchema, but instead are matched based on the CSV column header and the column descriptions' name and titles atributes. This allows for more flexibility, because columns in the CSV file may be re-ordered without invalidating the metadata. A stricter matching can be forced by specifying "header": false and "skipRows": 1 in the table's dialect description.

However, csvw.CSVW works correctly for

from the CSVW Test suites.

Compatibility with Frictionless Data Specs

A CSVW-described dataset is basically equivalent to a Frictionless DataPackage where all Data Resources are Tabular Data. Thus, the csvw package provides some conversion functionality. To "read CSVW data from a Data Package", there's the csvw.TableGroup.from_frictionless_datapackage method:

from csvw import TableGroup
tg = TableGroup.from_frictionless_datapackage('PATH/TO/datapackage.json')

To convert the metadata, the TableGroup can then be serialzed:

tg.to_file('csvw-metadata.json')

Note that the CSVW metadata file must be written to the Data Package's directory to make sure relative paths to data resources work.

This functionality - together with the schema inference capabilities of frictionless describe - provides a convenient way to bootstrap CSVW metadata for a set of "raw" CSV files, implemented in the csvwdescribe command described above.

See also

License

This package is distributed under the Apache 2.0 license.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

csvw-3.4.0.tar.gz (71.8 kB view details)

Uploaded Source

Built Distribution

csvw-3.4.0-py2.py3-none-any.whl (58.2 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file csvw-3.4.0.tar.gz.

File metadata

  • Download URL: csvw-3.4.0.tar.gz
  • Upload date:
  • Size: 71.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.10.12

File hashes

Hashes for csvw-3.4.0.tar.gz
Algorithm Hash digest
SHA256 65fe1b10c24049c097bffe01e03fee765cb7942533fe0a17b4d7ae42748155d2
MD5 b9b91d32b7f1f1254e0fb1d063633af2
BLAKE2b-256 4603762a4a14f914ffd0125e92697b5cdf27a1402163756e51dc5d483f193d0b

See more details on using hashes here.

File details

Details for the file csvw-3.4.0-py2.py3-none-any.whl.

File metadata

  • Download URL: csvw-3.4.0-py2.py3-none-any.whl
  • Upload date:
  • Size: 58.2 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.10.12

File hashes

Hashes for csvw-3.4.0-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 60bd6e414bc3ebfa98c3952d45b12a29186a2687b2a3e37cec44ea3ae8dac794
MD5 ba9743053911d41298c244eaaa21ade3
BLAKE2b-256 87857f7557895009c66d7421d8d141c2dc9ea15c63ab59cf7204c3a065c3a250

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page