Skip to main content

Python library for uploading (bulk) data to Dataverse

Project description

Dataverse Uploader
PyPI version Build Badge Build Badge

Python equivalent to the DVUploader written in Java. Complements other libraries written in Python and facilitates the upload of files to a Dataverse instance via Direct Upload.

Features

  • Parallel direct upload to a Dataverse backend storage
  • Files are streamed directly instead of being buffered in memory
  • Supports multipart uploads and chunks data accordingly

https://github.com/gdcc/python-dvuploader/assets/30547301/671131b1-d188-4433-9f77-9ec0ed2af36e


Getting started

To get started with DVUploader, you can install it via PyPI

python3 -m pip install dvuploader

or by source

git clone https://github.com/gdcc/python-dvuploader.git
cd python-dvuploader
python3 -m pip install .

Quickstart

Programmatic usage

In order to perform a direct upload, you need to have a Dataverse instance running and a cloud storage provider. The following example shows how to upload files to a Dataverse instance. Simply provide the files of interest and utilize the upload method of a DVUploader instance.

import dvuploader as dv


# Add file individually
files = [
    dv.File(filepath="./small.txt"),
    dv.File(directory_label="some/dir", filepath="./medium.txt"),
    dv.File(directory_label="some/dir", filepath="./big.txt"),
    *dv.add_directory("./data"), # Add an entire directory
]

DV_URL = "https://demo.dataverse.org/"
API_TOKEN = "XXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"
PID = "doi:10.70122/XXX/XXXXX"

dvuploader = dv.DVUploader(files=files)
dvuploader.upload(
    api_token=API_TOKEN,
    dataverse_url=DV_URL,
    persistent_id=PID,
    n_parallel_uploads=2, # Whatever your instance can handle
)

Command Line Interface

DVUploader ships with a CLI ready to use outside scripts. In order to upload files to a Dataverse instance, simply provide the files of interest, persistent identifier and credentials.

Using arguments

dvuploader my_file.txt my_other_file.txt \
           --pid doi:10.70122/XXX/XXXXX \
           --api-token XXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX \
           --dataverse-url https://demo.dataverse.org/ \

Using a config file

Alternatively, you can also supply a config file that contains all necessary informations for the uploader. The config file is a JSON/YAML file that contains the following keys:

  • persistent_id: Persistent identifier of the dataset to upload to.
  • dataverse_url: URL of the Dataverse instance.
  • api_token: API token of the Dataverse instance.
  • files: List of files to upload. Each file is a dictionary with the following keys:
    • filepath: Path to the file to upload.
    • directory_label: Optional directory label to upload the file to.
    • description: Optional description of the file.
    • mimetype: Mimetype of the file.
    • categories: Optional list of categories to assign to the file.
    • restrict: Boolean to indicate that this is a restricted file. Defaults to False.

In the following example, we upload three files to a Dataverse instance. The first file is uploaded to the root directory of the dataset, while the other two files are uploaded to the directory some/dir.

# config.yml
persistent_id: doi:10.70122/XXX/XXXXX
dataverse_url: https://demo.dataverse.org/
api_token: XXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
files:
    - filepath: ./small.txt
    - filepath: ./medium.txt
      directory_label: some/dir
    - filepath: ./big.txt
      directory_label: some/dir

The config file can then be used as follows:

dvuploader --config-path config.yml

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dvuploader-0.2.3.tar.gz (17.6 kB view details)

Uploaded Source

Built Distribution

dvuploader-0.2.3-py3-none-any.whl (20.9 kB view details)

Uploaded Python 3

File details

Details for the file dvuploader-0.2.3.tar.gz.

File metadata

  • Download URL: dvuploader-0.2.3.tar.gz
  • Upload date:
  • Size: 17.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.8.3 CPython/3.12.1 Linux/6.5.0-1018-azure

File hashes

Hashes for dvuploader-0.2.3.tar.gz
Algorithm Hash digest
SHA256 83f43da7864ef6d01b4b2c8b062b846b1ebd1c96e3269004120c7556b7aa03b5
MD5 2b6aeb372b5cb2d4493b2efce8dee57b
BLAKE2b-256 4bef06bad030e7350c306d5e0c05b8438fd4d5275cc34c3950368191a962e7ae

See more details on using hashes here.

File details

Details for the file dvuploader-0.2.3-py3-none-any.whl.

File metadata

  • Download URL: dvuploader-0.2.3-py3-none-any.whl
  • Upload date:
  • Size: 20.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.8.3 CPython/3.12.1 Linux/6.5.0-1018-azure

File hashes

Hashes for dvuploader-0.2.3-py3-none-any.whl
Algorithm Hash digest
SHA256 65282cd54c764c64e1242d4d71811e3a20b386c2529d8f96c339e0a38a1dce16
MD5 ff36d05cbb7d1c98b645c4752327c34d
BLAKE2b-256 89138d042aedfd1f8cbdd01c3c295d4baf4b473d5da371347d6a2dd81524dd52

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page