SimpleScheduler A Cron Library for Openmetadata
Project description
OpenMetadata Simple Scheduler
We are going to build a library based on simple scheduler for only cron-related ingestion purpose.
Scheduler
openmetadata-simplescheduler
is a flexible python library for building your own cron-like system to schedule jobs, which is to run a tornado process to serve REST APIs and a web ui.
Try it NOW
From source code:
git clone https://github.com/open-metadata/simplescheduler.git
cd simplescheduler
make simple
Open your browser and go to localhost:7777.
How to build Your own cron-replacement
Install simplescheduler
Using pip (from GitHub repo)
#
# Put this in requirements.txt, then run
# pip install -r requirements.txt
#
# If you want the latest build
git+https://github.com/open-metadata/simplescheduler.git#egg=simplescheduler
pip install -e git+https://github.com/open-metadata/simplescheduler.git#egg=simplescheduler
Three things
You have to implement three things for your scheduler, i.e., Settings
, Server
, and Jobs
.
Settings
In your implementation, you need to provide a settings file to override default settings (e.g., settings in simple_scheduler). You need to specify the python import path in the environment variable SIMPLESCHEDULER_SETTINGS_MODULE
before running the server.
All available settings can be found in default_settings.py file.
Server
You need to have a server file to import and run sdscheduler.server.server.SchedulerServer
.
Jobs
Each job should be a standalone class that is a subclass of sdscheduler.job.JobBase
and put the main logic of the job in run()
function.
After you set up Settings
, Server
and Jobs
, you can run the whole thing like this:
SIMPLESCHEDULER_SETTINGS_MODULE=simple_scheduler.settings \
PYTHONPATH=.:$(PYTHONPATH) \
python simple_scheduler/scheduler.py
Install dependencies
# Each time we introduce a new dependency in setup.py, you have to run this
make install
Run unit tests
make test
Clean everything and start from scratch
make clean
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for openmetadata-simplescheduler-0.2.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | ac7880137128457c46c3d141e4c51445812dd84e3eb8a981a4b2714293eac149 |
|
MD5 | d7ef3c4851b1722839d91bcd814c5a30 |
|
BLAKE2b-256 | 976875e74f465bfe7c8c236e2ec09e59ed24134c2aaf37c9052f106d9d54bccc |
Hashes for openmetadata_simplescheduler-0.2.1-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2fa873f9eba6ef3b54703ffdd533ed70107a113d7a77672bbd2771fe9b84da28 |
|
MD5 | 6b3c25fc3cc03a85f4139b8e10728493 |
|
BLAKE2b-256 | 13e0cf95f714597494e5d94175f0d82114ad98dbd9295cc638e9aab3c73b4a6e |