dwcahandler 1.1.2

Python package to handle Darwin Core Archive (DwCA) operations. This includes creating a DwCA zip file from one or more csvs, reading a DwCA, merge two DwCAs, validate DwCA and delete records from DwCA based on one or more key columns

dwcahandler

About

Python package to handle Darwin Core Archive (DwCA) operations. This includes creating a DwCA zip file from csv, reading a DwCA, merge two DwCAs, validate DwCA and delete records from DwCA based on one or more key columns

Motivation

This package was developed from a module in ALA's data preingestion to produce a DwCA for pipelines ingestion. ALA receive different forms of data from various data providers in the form of CSV and text files, API harvest and DwCA, this is needed to package up the data into DwCA.

The operations provided by dwcahandler includes creating a dwca from csv/text file, merge 2 dwcas, delete records in dwca and perform core key validations like testing duplicates of one or more keys, empty and duplicate keys.

Technologies

This package is developed in Python. Tested with Python 3.12, 3.11, 3.10 and 3.9

 

Setup

• Clone the repository.
• If using pyenv, install the required python version and activate it locally

pyenv local <python version>

• Install the dependency in local virtual environment

poetry shell
poetry install

• To import the darwin core and all the gbif extensions class row types and terms into dwcahandler

poetry run update-terms

 

Build

To build dwcahandler package

poetry build

 

Installation

Install published package

pip install dwcahandler

To use locally built package in a virtual environment:

pip install <folder>/dwcahandler/dist/dwcahandler-<version>.tar.gz

To install published package from testpypi

pip install -i https://test.pypi.org/simple/ dwcahandler

 

DwcaHandler is currently supporting the latest gbif extensions.

DwCA with the following extensions that have been ingested and tested in ALA are:

• Darwin Core Terms and Class RowTypes
• Simple Multimedia https://rs.gbif.org/extension/gbif/1.0/multimedia.xml
• Extended Measurement Or Fact http://rs.iobis.org/obis/terms/ExtendedMeasurementOrFact

Terms

• Terms are listed in terms.csv

from dwcahandler import DwcaHandler

df_terms, df_class = DwcaHandler.list_terms()
print(df_terms, df_class)

Class

• Listed in class-rowtype.csv
• Used in MetaElementTypes class enum name:

from dwcahandler import MetaElementTypes

print(MetaElementTypes.OCCURRENCE)
print(MetaElementTypes.MULTIMEDIA)

To list all the Class Rowtypes

from dwcahandler import DwcaHandler

DwcaHandler.list_class_rowtypes()

 

Examples of dwcahandler usages:

• Create Darwin Core Archive from csv file.
• Keys in core content are used as id/core id for Dwca with extensions and must be supplied in the data for core and extensions
• If core data have more than 1 key (for eg: institutionCode, collectionCode and catalogNumber), resulting dwca would generate id/core id for extension
• Validation is performed to make sure that the keys are unique in the core of the Dwca by default
• If keys are supplied for the content extension, the validation will be run to check the uniqueness of the keys in the content
• If keys are not provided, the default keys is eventID for event content and occurrenceID for occurrence content
• In creating a dwca with multimedia extension, provide format and type values in the Simple Multimedia extension, otherwise, dwcahandler will attempt to fill these info by guessing the mimetype from url.
• For convenience, if occurrence text file contain dwc term associatedMedia and no multimedia extension is supplied, dwcahandler attempts to extract out the multimedia url from associatedMedia into simple multimedia extemsion.

from dwcahandler import ContentData
from dwcahandler import DwcaHandler
from dwcahandler import MetaElementTypes
from dwcahandler import Dataset, Eml, Description, Contact, Name

core_csv = ContentData(data=["/tmp/occurrence.csv"], type=MetaElementTypes.OCCURRENCE, keys=["occurrenceID"])
ext_csvs = [ContentData(data=["/tmp/multimedia.csv"], type=MetaElementTypes.MULTIMEDIA)]

eml = Eml(
    dataset=Dataset(
        dataset_name="A Dataset",
        abstract=Description("Description of dataset"),
        creator=Contact(individual_name=Name(first_name="Jane", last_name="Doe"), email="[e-mail]"),
    )
)

DwcaHandler.create_dwca(core_csv=core_csv, ext_csv_list=ext_csvs, eml_content=eml, output_dwca="/tmp/dwca.zip")

 

• Create Darwin Core Archive from pandas dataframe
• In creating a dwca with multimedia extension, provide format and type values in the Simple Multimedia extension, otherwise, dwcahandler will attempt to fill these info by guessing the mimetype from url.

from dwcahandler import DwcaHandler
from dwcahandler.dwca import ContentData
from dwcahandler import MetaElementTypes
from dwcahandler import Dataset, Eml, Description, Contact, Name
import pandas as pd

core_df = pd.read_csv("/tmp/occurrence.csv")
core_frame = ContentData(data=core_df, type=MetaElementTypes.OCCURRENCE, keys=["occurrenceID"])

ext_df = pd.read_csv("/tmp/multimedia.csv")
ext_frame = [ContentData(data=ext_df, type=MetaElementTypes.MULTIMEDIA)]

eml = Eml(
    dataset=Dataset(
        dataset_name="A Dataset",
        abstract=Description("Description of dataset"),
        creator=Contact(individual_name=Name(first_name="Jane", last_name="Doe"), email="[e-mail]"),
    )
)

DwcaHandler.create_dwca(core_csv=core_frame, ext_csv_list=ext_frame, eml_content=eml, output_dwca="/tmp/dwca.zip")

 

• Convenient helper function to build Darwin Core Archive from a list of csv files.
• Build event core DwCA if event.txt file is supplied, otherwise, occurrence core DwCA if occurrence.txt is supplied.
• Raises error if neither event.txt nor occurrence.txt is in the list
• Class row types are determined by file names of the text files.
• If no content keys provided, the default keys are eventID for event content and occurrenceID for occurrence content
• Delimiter for txt files are comma delimiter by default. For tab delimiter, supply CsvEncoding

from dwcahandler import DwcaHandler
from dwcahandler import Dataset, Eml, Description, Contact, Name

eml = Eml(
    dataset=Dataset(
        dataset_name="A Dataset",
        abstract=Description("Description of dataset"),
        creator=Contact(individual_name=Name(first_name="Jane", last_name="Doe"), email="[e-mail]"),
    )
)

DwcaHandler.create_dwca_from_file_list(
    files=["/tmp/event.csv", "/tmp/occurrence.csv"], eml_content=eml, output_dwca="/tmp/dwca.zip"
)

 

• Convenient helper function to create Darwin Core Archive from csv files in a zip files.
• Build event core DwCA if event.txt file is supplied, otherwise, occurrence core DwCA if occurrence.txt is supplied in the zip file
• Raises error if neither event.txt nor occurrence.txt is in the zip file
• Class row types are determined by file names of the text files.
• If no content keys provided, the default keys are eventID for event content and occurrenceID for occurrence content.
• Delimiter for txt files are comma delimiter by default. For tab delimiter, supply CsvEncoding

from dwcahandler import DwcaHandler
from dwcahandler import Dataset, Eml, Description, Contact, Name

eml = Eml(
    dataset=Dataset(
        dataset_name="A Dataset",
        abstract=Description("Description of dataset"),
        creator=Contact(individual_name=Name(first_name="Jane", last_name="Doe"), email="[e-mail]"),
    )
)

DwcaHandler.create_dwca_from_zip_content(zip_file="/tmp/txt_files.zip", eml_content=eml, output_dwca="/tmp/dwca.zip")

 

• Merge two Darwin Core Archives into a single file
• Set extension sync to True to remove existing extension records before merging. Default for extension sync is False

from dwcahandler import DwcaHandler, MetaElementTypes

DwcaHandler.merge_dwca(dwca_file="/tmp/dwca.zip", delta_dwca_file="/tmp/delta-dwca.zip",
                       output_dwca="/tmp/new-dwca.zip", 
                       keys_lookup={MetaElementTypes.OCCURRENCE:["occurrenceID"]})

 

• Delete Rows from core file in Darwin Core Archive

from dwcahandler import ContentData
from dwcahandler import DwcaHandler, MetaElementTypes

delete_csv = ContentData(data=["/tmp/old-records.csv"], type=MetaElementTypes.OCCURRENCE, keys=["occurrenceID"])

DwcaHandler.delete_records(dwca_file="/tmp/dwca.zip",
                           records_to_delete=delete_csv,
                           output_dwca="/tmp/new-dwca.zip")

 

Support for building Ecological Markup Language (EML) via DwcaHandler Eml class

• DwcaHandler supports generating EML for dataset metadata.
• Eml object can be passed into DwcaHandler to create DwCA.
• Eml class requires a mandatory dataset and an optional additional metadata to generate the eml.
• Failure to provide dataset or an empty additional metadata is supplied, Eml class will still generate the eml string but an error will be displayed.
• For more info on the Eml Class, see eml.py
• See sample EML eml-sample.xml

from dwcahandler import (
    Name,
    Address,
    Contact,
    Description,
    BoundingCoordinates,
    GeographicCoverage,
    DateRange,
    CalendarDate,
    TemporalCoverage,
    TaxonomicClassification,
    TaxonomicCoverage,
    Coverage,
    KeywordSet,
    Dataset,
    AdditionalMetadata,
    GBIFMetadata,
    Eml,
)

contact_person: Contact = Contact(
    individual_name=Name(first_name="John", last_name="Doe"),
    address=Address(city="City", postal_code="ABC-123", country="Country"),
    organization_name="An Organization",
    email="[e-mail]",
    userid="https://orcid.org/0000-0000-0000-0000",
)

dataset = Dataset(
    dataset_name="Test Dataset",
    alternate_identifier=[
        "https://ipt/eml.do?r=a-resource",
        "https://another-website/dataset-info",
    ],
    keyword_set=[
        KeywordSet(
            keyword="Occurrence",
            keyword_thesaurus="http://rs.gbif.org/vocabulary/gbif/dataset_type_2015-07-10.xml",
        )
    ],
    abstract=Description(
        description="Lorem Ipsum is simply dummy text of the printing and typesetting industry. \n"
                    "Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when \n"
                    "an unknown printer took a galley of type and scrambled it to make a type specimen book"
    ),
    creator=contact_person,
    published_date="2020-05-01",
    coverage=Coverage(
        geographic_coverage=GeographicCoverage(
            description="The data set contains records of herbarium specimens",
            bounding_coordinates=BoundingCoordinates(west="1.0", east="2.0", north="3.0", south="4.0"),
        ),
        temporal_coverage=TemporalCoverage(
            DateRange(
                begin_date=CalendarDate(calendar_date="2020-01-01"),
                end_date=CalendarDate(calendar_date="2020-01-31"),
            )
        ),
        taxonomic_coverage=TaxonomicCoverage(
            general_taxonomic_coverage="All vascular plants are identified as species or genus",
            taxonomic_classification=[
                TaxonomicClassification(taxon_rank_name="Genus", taxon_rank_value="Acacia"),
                TaxonomicClassification(taxon_rank_name="Genus", taxon_rank_value="Acacia"),
            ],
        ),
    ),
    intellectual_rights=[
        Description(
            description=
    """"
    This work is licensed under a 
    <ulink url='http://creativecommons.org/licenses/by/4.0/legalcode'><citetitle>Creative Commons Attribution (CC-BY) 4.0 License</citetitle></ulink>
    """
        )
    ],
    contact=contact_person,
)

# GBIF Additional Metadata require at least a citation. 
# If gbif field is supplied, it must be a list of dictionaries
additional_metadata = AdditionalMetadata(
    metadata=GBIFMetadata(
        citation="Researchers should cite this work as follows: xxxxx",
        gbif=[{"resourceLogoUrl": "http://logo_url"}, {"hierarchyLevel":"dataset"}]
    )
)

eml = Eml(dataset=dataset,
          additional_metadata=additional_metadata)

eml_xml_str = eml.build_eml_xml()
print(eml_xml_str)