Skip to main content

A Python library and simple command-line frontend for computational job submission to multiple resources.

Project description

GC3Pie is a suite of Python classes (and command-line tools built upon them) to aid in submitting and controlling batch jobs to clusters and grid resources seamlessly. GC3Pie aims at providing the building blocks by which Python scripts that combine several applications in a dynamic workflow can be quickly developed.

The GC3Pie suite is comprised of three main components:

  • GC3Libs: A python package for controlling the life-cycle of a Grid or batch computational job

  • GC3Utils: Command-line tools exposing the main functionality provided by GC3Libs

  • GC3Apps: Driver scripts to run large job campaigns

GC3Libs

GC3Libs provides services for submitting computational jobs to Grids and batch systems and controlling their execution, persisting job information, and retrieving the final output.

GC3Libs takes an _application-oriented approach to batch computing. A generic Application class provides the basic operations for controlling remote computations, but different Application subclasses can expose adapted interfaces, focusing on the most relevant aspects of the application being represented. Specific interfaces are already provided for the GAMESS and Rosetta suites; new ones can be easily created by subclassing the generic Application class.

GC3Utils

Most of the time users have lots of different accounts on several diverse resources. The idea underlying GC3Utils is that a user can submit and control a computational job from one single place with a few simple commands.

Commands are provided to submit a job (gsub), check its running status (gstat), get a snapshot of the output files (gget, gtail), or cancel it (gkill).

GC3Apps

There is a need in some scientific communities, to run large job campaigns to analyze a vast number of data files with the same application. The single-job level of control implemented by GC3Utils in this case is not enough: you would have to implement “glue scripts” to control hundreds or thousand scripts at once. GC3Pie has provisons for this, in the form of re-usable Python classes that implement a single point of control for job families.

The GC3Apps scripts are driver scripts that run job campaigns using the supported applications on a large set of input data. They can be used in production as-is, or adapted to suit your data processing needs.

Installation instructions and further reading

For up-to-date information, please visit the GC3Pie home page at: http://gc3pie.googlecode.com/

Installation instructions are in the INSTALL.txt file (in this same directory), or can be read online at: http://code.google.com/p/gc3pie/wiki/InstallGC3Pie

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

gc3pie-2.1.tar.gz (945.4 kB view details)

Uploaded Source

File details

Details for the file gc3pie-2.1.tar.gz.

File metadata

  • Download URL: gc3pie-2.1.tar.gz
  • Upload date:
  • Size: 945.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for gc3pie-2.1.tar.gz
Algorithm Hash digest
SHA256 868332757bd015225d4cf23a12bf8f4fd15f076f5dedc9afa72f0b837a248645
MD5 3b924b5dba84987a09e87fd845816d54
BLAKE2b-256 b6159d56715924c6c88fddb0587f798e5382b9f08efe6478685a4b5d8d4a5783

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page