gitlab2zenodo 0.0b8

Sends gitlab snapshots to zenodo Automatically.

Make your code and data citable with Zenodo and Gitlab!

This script is still in beta ! Please signal any issues you encounter.

Sends snapshots of your gitlab repository to zenodo automatically, for each new version tag. Zenodo provides a DOI for your code or data, which makes it citable. By publishing automatically when creating new versions, you can make sure the zenodo record stays up to date.

• Sending snapshots is triggered by the creation of tags with version names (such as "v1.0.0" or "0.2")
• You can provide the metadata as a json file, otherwise the script simply updates the latest available record.
• If not given any deposit identifier, the program adds a new deposit. If given a deposit identifier, it creates a new version. See below on how to provide a deposit identifier.
• By default, the script sends an archive and metadata, but does not go through with publication, in order to let you review the changes. You then need to go to zenodo and publish manually. A parameter is available to publish automatically, but use at your own risks.

This uses gitlab pipelines, but should be accessible even if you have never used them.

Installation and setup

NOTE: At first, I recommend you do this setup using zenodo's sandbox. You will need to create an account there, even if you already have a normal zenodo account. You should only generate a real zenodo token, and setup changes to be pushed to zenodo, once you are certain everything is setup correctly.

1. Connect to your Zenodo account (or create one if you do not have one yet)

2. Create an API token on zenodo, checking deposit write and deposit publish as rights. Copy the string right away, it is shown only once. If you failed to do this, simply remove the token and create a new one.

3. Create a custom variable in the Gitlab UI with the exact name and the token as a value.

   • To do this, go to your repository, then in the left bar in Settings > CI/CD.
   • Expand the Variable tab.
   • Click "Add Variable"
   • Enter zenodo_token as the variable name and the token as a value.
   • Because this token would allow anyone to push versions and deposits to zenodo, you should check both "Protect" and "Mask".
   • This means that you need to define protected tags, that is, tags which only some users can create, and for which the pipelines have access to the token.
     • This happens in Settings > Repository > Protected Tags.
     • I use "v*" (and call version tags "v1.0.0", etc).
     • Enter the wildcard you want, then click "create wildcard",
     • select the users which can create these tags, then click "Protect".

4. If you already have a zenodo record for this repository, add a second variable, with the name zenodo_record and the digits of the initial zenodo record as a number. This number is the last part of your DOI (10.5072/zenodo.123456). This is used by gitlab2zenodo in order to know which record to update with new versions.

   • If you do not have a zenodo record yet, the first time gitlab2zenodo is triggered, it will create one. You should then add the identifier of this new record as a gitlab variable.

5. Create a .zenodo.json file, and fill-in any metadata you want to pass to zenodo. You can check the InvenioRDM api documentation (Zenodo's underlying framework).

• If you don't feel comfortable with writing a full JSON metadata, or if your file is refused by the script, you might want to fill Zenodo's user-friendly form for the first release, and then use the script for further updates. Once you filled in Zenodo's form, you can easily download it with:

   pip install gtlab2zenodo
   g2z-get-meta -i [your repository ID] -t [your zenodo token] -d > .zenodo.json
   # -d Tells zenodo to look for a draft record.

• gitlab2zenodo will always update the version when sending to zenodo so that it matches the tag name (removing initial "v")
• unless you define these values in .zenodo.json, gitlab2zenodo will add two relations pointing to the repository:
  • compiled by: the url of your overall repository
  • identical to: the url of the specific tag which is being uploaded
• If you already have a zenodo deposit with full metadata for the repository, you can use gitlab2zenodo to retrieve the existing metadata:

    pip install gtlab2zenodo
    g2z-get-meta -i [your repository ID] -t [your zenodo token] > .zenodo.json

• Here is a basic example of .zenodo.json file, adapted from the InvenioRDM api documentation :

{
    'metadata': {
        'title': 'My first upload',
        "resource_type": {
            "id": "dataset"
        },
        'description': '<div>This is my first upload</div>',
        'creators': [{
            "person_or_org": {
                "name": "Nielsen, Lars Holm",
                "type": "personal",
                "given_name": "Lars Holm",
                "family_name": "Nielsen",
                "identifiers": [{
                    "scheme": "orcid",
                    "identifier": "0000-0001-8135-3489"
                }],
            },
            "affiliations": [{
              "id": "01ggx4157",
              "name": "CERN",
            }]
        }]
    }
}

6. Create a .gitlab-ci.yml file and write the following code. If you already have a pipeline, simply add a job. Make sure python3.6 is available.

image: python:3.6

send-snapshot:
  rules:
    - if: $CI_COMMIT_TAG =~ /^v?[0-9]+\.[0-9]+/
  script:
    - pip install gitlab2zenodo
    - git archive --format zip --output ${CI_COMMIT_TAG#v}.zip ${CI_COMMIT_TAG}
    - g2z-send -i $zenodo_record -t $zenodo_token -s -m .zenodo.json ${CI_COMMIT_TAG#v}.zip

If your repository already exists, you can also leave out the metadata file. In that case, the script will reuse the metadata from the previous releases and update the version and the publication date.

7. Add these lines to your .gitattributes, or create one if there is none. This is so that we do not upload these files to zenodo:

.zenodo.json       export-ignore
.gitlab-ci.yml     export-ignore

Usage

Any tag with a version name, such as "v1.0.0" should send a snapshot of your repo to zenodo.

The metadata of the new deposit or version will be updated using the current version of your .zenodo.json file.

Unless you used the -p tag, you should now go to zenodo, review the submitted snapshot, then publish it when it is ready.

If there was no prior zenodo deposit for this repository, one will be created at the first upload. Go back to add it as a variable, so that the next tags create new versions on zenodo.

Here is a quick summary of gitlab2zenodo's behavior and requirements depending on the status of your repository:

Status	Behaviour	Metadata	ID
No repository	Create a record	Required	No
An unpublished repository	Update the record	Optional	Required
A published repository	Create a new version	Optional	Required

A walk-through the pipeline file

Here is a line-by-line walk through the suggested pipeline file:

• image: python:3.6 requires a docker image intended for python3.6.
• rules: specify when to run the pipeline.
• - if: $CI_COMMIT_TAG =~ /^v?[0-9]+\.[0-9]+/: run only when CI_COMMIT_TAG is given (new tag triggered pipeline), and the tag looks like a version number. The regex assumes that a version number starts with a sequence of a number and a dot, any times, maybe preceded by "v". You can change it to your liking if your version numbers follow a different syntax. Note that gitlab2zenodo sets the new version number in the metadata following a similar regex.
• send-snapshot: is the name of the job we define. The name can be changed, it is not important.
• script: defines the list of command which are executed as this job.
• pip install gitlab2zenodo installs this package,
• git archive --format zip --output ${CI_COMMIT_TAG#v}.zip ${CI_COMMIT_TAG} creates the archive to be uploaded. The name is that of the tag, minus any initial "v".
• - g2z-send -i $zenodo_record -t $zenodo_token -s -m .zenodo.json ${CI_COMMIT_TAG#v}.zip sends the archive to zenodo.
  • -i the zenodo record to send the snapshot to. The record ID can also be given directly.
  • -t the zenodo token - this should always be passed as an environment variable.
  • -s ensures that you send only to zenodo's sandbox.
  • -m .zenodo.json provides the input file for the metadata. It could be any other json file.
  • by default, this will send a snapshot and edit metadata, but NOT publish. You can ensure publication by adding -p, but beware: this can not be undone and files can not be edited after publication.