A distributed Celery application to export time-domain data periodically from Microsoft Graph API into a buffer key-value store.
MsGraphExporter is an application that performs periodic export of time-domain data like Azure AD user signins from Microsoft Graph API into a buffer key-value store (currently supports Redis) for subsequent processing. It uses Celery task queue for parallel processing, gevent greenlets for concurrent uploads, relies on the Graph API pagination to control memory footprint and respects Graph API throttling. The application could be deployed as a single-container worker or as a set of multiple queue-specific workers for high reliability and throughput.
Follow these instructions to use the application.
install a specific version
pip install "ms-graph-exporter==0.1"
install the latest version
pip install "ms-graph-exporter"
upgrade to the latest version
pip install --upgrade "ms-graph-exporter"
install optional DEV dependencies like
pytestframework and plugins necessary for performance and functional testing
pip install "ms-graph-exporter[test]"
Service Principal credentials with correct access rights to
Graph API must be defined for
docker/docker-compose.yamlexample deployment to work. Following
.env file should be placed at the root of the project folder, or following ENV variables defined in other way:
GRAPH_CLIENT_ID=<clientId> GRAPH_CLIENT_SECRET=<clientSecret> GRAPH_TENANT=<tenant>
Then, docker deployment could be initiated with:
docker-compose --file docker/docker-compose.yaml up
- Python >= 3.6
- Celery - Distributed task queue
- gevent - concurrent data upload to Redis
- redis-py - Python interface to Redis
We use Semantic Versioning Specification as a version numbering convention.
Also, conventions for
git commit messages are documented in CONTRIBUTING.md.
- Oleksiy Kuzmenko - OK-UNDP@GitHub - Initial design and implementation
- Hat tip to all individuals shaping design of this project by sharing their knowledge in articles, blogs and forums.
There are plenty of ways you could contribute to this project. Feel free to:
- submit bug reports and feature requests
- outline, fix and expand documentation
- peer-review bug reports and pull requests
- implement new features or fix bugs
See CONTRIBUTING.md for details on code formatting, linting and testing frameworks used by this project.
Release history Release notifications
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size ms_graph_exporter-0.1.0rc4-py3-none-any.whl (49.9 kB)||File type Wheel||Python version py3||Upload date||Hashes View hashes|
|Filename, size ms-graph-exporter-0.1.0rc4.tar.gz (33.8 kB)||File type Source||Python version None||Upload date||Hashes View hashes|
Hashes for ms_graph_exporter-0.1.0rc4-py3-none-any.whl
Hashes for ms-graph-exporter-0.1.0rc4.tar.gz