A pure Python remote job scheduler system
- This pscheduler service is critical dependent upon the usage of shared filesystem across the remote machines. Setup NFS for all the remote machines.
- Have same login credentials for all the remote machines.
- Basic Linux utilities like ssh and nohup should be installed on all the remote machines.
pscheduler is a Python 3 only module and can be easily installed using pip:
pip install pscheduler
- Start the service using pscheduler service start. This will create a directory .pscheduler under your home directory.
- Edit file ~/.pscheduler/hosts.cfg and add all the IP addresses/domain aliases of the remote machines in each line. Save and exit. Alternately if you are in an HPC environment and LSF is already installed, then you can use the provided script bhostsWrapper.py to directly populate the remote machine. You should then edit it to remove the head node machine from the list and any other required host.
- If you have not already setup password-less login into remote machines, then use the provided script batchSetupLoginKeys.py to set it up.
There are four basic utilities:
service start/stop the background service hosts get information on remote machines sub submit jobs jobs monitor running/pending job
See helpfiles of individual subcommands for further details
It is critical that the background service is started before the submitting any job
- Start service: pscheduler service start
- Submit job: pscheduler sub "sleep 10"
Pending and running job configuration files are stored under ~/.pscheduler/jobs in PEND and RUN directory respectively. By default the finished job configuration file containing standard output(JSON format) are stored under ~/.pscheduler/jobs/FINISH, but users can choose a custom location for this file by using -o flag while submitting job and providing the output location path.
~/.pscheduler/DAEMON.log contains the log of the background service. You should have a look at this file to see if any error messages have been thrown.
Note that pscheduler will not will ignore the scheduling by other schedulers and simply launch jobs based on availability of resources (currently only number of CPU cores).
Roadmap for future versions
- Implement jobs subcommand
- Job restart in case of resource spike
- Improve code for catching fatal exceptions
- FIRST PRE ALPHA RELEASE
- CLI CREATED using wrapper.py
- BHOST wrapper script separated form phosts module
- Introduced hosts.cfg: A user editable list of hosts
- Batch script made for creating login keys
- PSUB now saves in JSON format
- DEV: Submit to PyPi using python script
- DEV: Automatic update of version in setup.py
- Added pscheduler in scripts for command line invocation
- Deployment fix
- Import fix
- Json fix
- jobs subcommand implemented
- phosts doublehost check issue rectified
- submission process improved in daemon
- class naming convention changed
- default host fixed to localhost. passwordless loging into localhost created
- DEV: Travis CI now being used for testing
- pdaemon fix
- Rolledback SSH keygen
Parashar Dhapola (email@example.com)
Release history Release notifications
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size & hash SHA256 hash help||File type||Python version||Upload date|
|pscheduler-0.1.0-py3-none-any.whl (14.2 kB) Copy SHA256 hash SHA256||Wheel||py3|
|pscheduler-0.1.0.tar.gz (11.3 kB) Copy SHA256 hash SHA256||Source||None|