gitlab2prov 2.2.0

Extract provenance information (W3C PROV) from GitLab projects.

Welcome to gitlab2prov! 👋

gitlab2prov is a Python library and command line tool that extracts provenance information from GitLab projects.

---

The gitlab2prov data model has been designed according to W3C PROV specification. The model documentation can be found here.

️🏗️ ️Installation

Please note that this tool requires Git to be installed on your machine.

Clone the project and install using pip:

pip install .

Or install the latest release from PyPi:

pip install gitlab2prov

To install gitlab2prov with all extra dependencies require the [dev] extras:

pip install .[dev]            # clone repo, install with extras
pip install gitlab2prov[dev]  # PyPi, install with extras

⚡ Getting started

gitlab2prov needs a personal access token to clone git repositories and to authenticate with the GitLab API. Follow this guide to create an access token with the required scopes.

🚀‍ Usage

gitlab2prov can be configured using the command line interface or by providing a configuration file in .yaml format.

Command Line Usage

The command line interface consists of commands that can be chained together like a unix pipeline.

Usage: gitlab2prov [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...

  Extract provenance information from GitLab projects.

Options:
  --version        Show the version and exit.
  --verbose        Enable logging to 'gitlab2prov.log'.
  --config FILE    Read config from file.
  --validate FILE  Validate config file and exit.
  --help           Show this message and exit.

Commands:
  combine                  Combine multiple graphs into one.
  extract                  Extract provenance information for one or more...
  load                     Load provenance files.
  merge-duplicated-agents  Merge duplicated agents based on a name to...
  pseudonymize             Pseudonymize a provenance graph.
  save                     Save provenance information to a file.
  stats                    Print statistics such as node counts and...

Configuration Files

gitlab2prov supports configuration files in .yaml format that are functionally equivalent to command line invocations.

To read configuration details from a file instead of specifying on the command line, use the --config option:

# initiate a run using a config file
gitlab2prov --config config/example.yaml

You can validate your config file using the provided JSON-Schema gitlab2prov/config/schema.json that comes packaged with every installation:

# check config file for syntactical errors
gitlab2prov --validate config/example.yaml

Config file example:

- extract:
        url: ["https://gitlab.com/example/foo"]
        token: tokenA
- extract:
        url: ["https://gitlab.com/example/bar"]
        token: tokenB
- load:
        input: [example.rdf]
- pseudonymize:
- combine:
- save:
        output: combined
        format: [json, rdf, xml, dot]
- stats:
        fine: true
        explain: true
        formatter: table

The config file example is functionally equivalent to this command line invocation:

gitlab2prov extract -u https://gitlab.com/example/foo -t tokenFoo \
            extract -u https://gitlab.com/example/bar -t tokenBar \
            load -i example.rdf                                   \
            pseudonymize                                          \
            combine                                               \
            save -o combined -f json -f rdf -f xml -f dot         \
            stats --fine --explain --formatter table

🎨 Provenance Output Formats

gitlab2prov supports output formats that the prov library provides:

• PROV-N
• PROV-O (RDF)
• PROV-XML
• PROV-JSON
• Graphviz (DOT)

🤝 Contributing

Contributions and pull requests are welcome!
For major changes, please open an issue first to discuss what you would like to change.

✨ How to cite

If you use GitLab2PROV in a scientific publication, we would appreciate citations to the following paper:

• Schreiber, A., de Boer, C. and von Kurnatowski, L. (2021). GitLab2PROV—Provenance of Software Projects hosted on GitLab. 13th International Workshop on Theory and Practice of Provenance (TaPP 2021), USENIX Association

Bibtex entry:

@InProceedings{SchreiberBoerKurnatowski2021,
  author    = {Andreas Schreiber and Claas de~Boer and Lynn von~Kurnatowski},
  booktitle = {13th International Workshop on Theory and Practice of Provenance (TaPP 2021)},
  title     = {{GitLab2PROV}{\textemdash}Provenance of Software Projects hosted on GitLab},
  year      = {2021},
  month     = jul,
  publisher = {{USENIX} Association},
  url       = {https://www.usenix.org/conference/tapp2021/presentation/schreiber},
}

You can also cite specific releases published on Zenodo:

✏️ References

Influencial Software for gitlab2prov

• Martin Stoffers: "Gitlab2Graph", v1.0.0, October 13. 2019, GitHub Link, DOI 10.5281/zenodo.3469385

• Quentin Pradet: "How do you rate limit calls with aiohttp?", GitHub Gist, MIT LICENSE

Influencial Papers for gitlab2prov:

• De Nies, T., Magliacane, S., Verborgh, R., Coppens, S., Groth, P., Mannens, E., and Van de Walle, R. (2013). Git2PROV: Exposing Version Control System Content as W3C PROV. In Poster and Demo Proceedings of the 12th International Semantic Web Conference (Vol. 1035, pp. 125–128).

• Packer, H. S., Chapman, A., and Carr, L. (2019). GitHub2PROV: provenance for supporting software project management. In 11th International Workshop on Theory and Practice of Provenance (TaPP 2019).

Papers that refer to gitlab2prov:

• Andreas Schreiber, Claas de Boer (2020). Modelling Knowledge about Software Processes using Provenance Graphs and its Application to Git-based VersionControl Systems. In ICSEW'20: Proceedings of the IEEE/ACM 42nd Conference on Software Engineering Workshops (pp. 358–359).

• Tim Sonnekalb, Thomas S. Heinze, Lynn von Kurnatowski, Andreas Schreiber, Jesus M. Gonzalez-Barahona, and Heather Packer (2020). Towards automated, provenance-driven security audit for git-based repositories: applied to germany's corona-warn-app: vision paper. In Proceedings of the 3rd ACM SIGSOFT International Workshop on Software Security from Design to Deployment (pp. 15–18).

• Andreas Schreiber (2020). Visualization of contributions to open-source projects. In Proceedings of the 13th International Symposium on Visual Information Communication and Interaction. ACM, USA.

📜 Dependencies

gitlab2prov depends on several open source packages that are made freely available under their respective licenses.

Package	License
GitPython
click
python-gitlab
prov
jsonschema
ruamel.yaml
pydot

📝 License

This project is MIT licensed.
Copyright © 2019 German Aerospace Center (DLR) and individual contributors.