jinlava 8.2.0

AWS based distributed scheduler and job runner

Lava

Lava is an AWS based distributed scheduler and job runner. It is scalable, robust and built using native AWS components, wherever possible.

Lava has been used extensively in a large, business critical data warehouse / data lake environment to run tens of thousands of jobs a day on a 24x7 basis, with jobs lasting anywhere between a few seconds and 12 hours.

Typical jobs perform tasks such as:

• AWS environment command, control and maintenance.

• Data ETL jobs, large and small.

• Inter-company data exchanges.

Lava was built as a result of terminal frustration with some commercial and open source options that, too often, proved to be either fragile, or hard to install, maintain or use in a complex multi-user, multi-environment context.

Genesis

Lava was developed at Origin Energy as part of the Jindabyne initiative. While not part of our core IP, it proved valuable internally, and we're sharing it in the hope it's useful to others.

Kudos to Origin for fostering a culture that empowers its people to build complex technology solutions in-house.

Features

Lava features include:

• Provision of an integrated orchestration and execution environment that is readily scalable from a desktop installation to a large auto scaling fleet.

• Jobs can be scheduled, triggered by AWS and external events or initiated by other jobs.

• Ability to connect to a variety of database types, including Postgres, MySQL, Oracle, Microsoft SQL Server and Redshift.

• Enhanced support for some AWS RDS and RDS Aurora databases.

• Built-in connectors to a handful of useful services, including databases, file shares, Microsoft SharePoint, email servers and Slack.

• Ability to run SQL jobs as well as executable payloads, including docker based payloads and native code bundles.

• Secure management of connection credentials to avoid the need to embed them into job payloads.

Installation and Usage

See the user guide for details.