ms-graph-exporter 0.1.0rc4.dev201910092

A distributed Celery application to export time-domain data periodically from Microsoft Graph API into a buffer key-value store.

MsGraphExporter

MsGraphExporter is an application that performs periodic export of time-domain data like Azure AD user signins from Microsoft Graph API into a buffer key-value store (currently supports Redis) for subsequent processing. It uses Celery task queue for parallel processing, gevent greenlets for concurrent uploads, relies on the Graph API pagination to control memory footprint and respects Graph API throttling. The application could be deployed as a single-container worker or as a set of multiple queue-specific workers for high reliability and throughput.

Getting Started

Follow these instructions to use the application.

Installing

MsGraphExporter is distributed through the Python Package Index as ms-graph-exporter. Run the following command to:

• install a specific version

  pip install "ms-graph-exporter==0.1"
  

• install the latest version

  pip install "ms-graph-exporter"
  

• upgrade to the latest version

  pip install --upgrade "ms-graph-exporter"
  

• install optional DEV dependencies like pytest framework and plugins necessary for performance and functional testing

  pip install "ms-graph-exporter[test]"
  

MsGraphExporter is also distributed as the Alpine-based docker container and published on Docker Hub as undp/ms-graph-exporter.

Service Principal credentials with correct access rights to Graph API must be defined for docker/docker-compose.yamlexample deployment to work. Following .env file should be placed at the root of the project folder, or following ENV variables defined in other way:

GRAPH_CLIENT_ID=<clientId>
GRAPH_CLIENT_SECRET=<clientSecret>
GRAPH_TENANT=<tenant>

Then, docker deployment could be initiated with:

docker-compose --file docker/docker-compose.yaml up

Requirements

• Python >= 3.6

Built using

• Celery - Distributed task queue
• gevent - concurrent data upload to Redis
• redis-py - Python interface to Redis

Versioning

We use Semantic Versioning Specification as a version numbering convention.

Release History

For the available versions, see the tags on this repository. Specific changes for each version are documented in CHANGELOG.md.

Also, conventions for git commit messages are documented in CONTRIBUTING.md.

Authors

• Oleksiy Kuzmenko - OK-UNDP@GitHub - Initial design and implementation

Acknowledgments

• Hat tip to all individuals shaping design of this project by sharing their knowledge in articles, blogs and forums.

License

Unless otherwise stated, all authors (see commit logs) release their work under the MIT License. See LICENSE.md for details.

Contributing

There are plenty of ways you could contribute to this project. Feel free to:

• submit bug reports and feature requests
• outline, fix and expand documentation
• peer-review bug reports and pull requests
• implement new features or fix bugs

See CONTRIBUTING.md for details on code formatting, linting and testing frameworks used by this project.