Skip to main content
Join the official 2019 Python Developers SurveyStart the survey!

A distributed Celery application to export time-domain data periodically from Microsoft Graph API into a buffer key-value store.

Project description

MsGraphExporter

Python 3.6+ Code style: black License: MIT ReadTheDocs Build Status

MsGraphExporter is an application that performs periodic export of time-domain data like Azure AD user signins from Microsoft Graph API into a buffer key-value store (currently supports Redis) for subsequent processing. It uses Celery task queue for parallel processing, gevent greenlets for concurrent uploads, relies on the Graph API pagination to control memory footprint and respects Graph API throttling. The application could be deployed as a single-container worker or as a set of multiple queue-specific workers for high reliability and throughput.

Getting Started

Follow these instructions to use the application.

Installing

MsGraphExporter is distributed through the Python Package Index as ms-graph-exporter. Run the following command to:

  • install a specific version

    pip install "ms-graph-exporter==0.1"
    
  • install the latest version

    pip install "ms-graph-exporter"
    
  • upgrade to the latest version

    pip install --upgrade "ms-graph-exporter"
    
  • install optional DEV dependencies like pytest framework and plugins necessary for performance and functional testing

    pip install "ms-graph-exporter[test]"
    

MsGraphExporter is also distributed as the Alpine-based docker container and published on Docker Hub as undp/ms-graph-exporter.

Service Principal credentials with correct access rights to Graph API must be defined for docker/docker-compose.yamlexample deployment to work. Following .env file should be placed at the root of the project folder, or following ENV variables defined in other way:

GRAPH_CLIENT_ID=<clientId>
GRAPH_CLIENT_SECRET=<clientSecret>
GRAPH_TENANT=<tenant>

Then, docker deployment could be initiated with:

docker-compose --file docker/docker-compose.yaml up

Requirements

  • Python >= 3.6

Built using

Versioning

We use Semantic Versioning Specification as a version numbering convention.

Release History

For the available versions, see the tags on this repository. Specific changes for each version are documented in CHANGELOG.md.

Also, conventions for git commit messages are documented in CONTRIBUTING.md.

Authors

  • Oleksiy Kuzmenko - OK-UNDP@GitHub - Initial design and implementation

Acknowledgments

  • Hat tip to all individuals shaping design of this project by sharing their knowledge in articles, blogs and forums.

License

Unless otherwise stated, all authors (see commit logs) release their work under the MIT License. See LICENSE.md for details.

Contributing

There are plenty of ways you could contribute to this project. Feel free to:

  • submit bug reports and feature requests
  • outline, fix and expand documentation
  • peer-review bug reports and pull requests
  • implement new features or fix bugs

See CONTRIBUTING.md for details on code formatting, linting and testing frameworks used by this project.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for ms-graph-exporter, version 0.1.0rc4
Filename, size File type Python version Upload date Hashes
Filename, size ms_graph_exporter-0.1.0rc4-py3-none-any.whl (49.9 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size ms-graph-exporter-0.1.0rc4.tar.gz (33.8 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page