Skip to main content
Join the official Python Developers Survey 2018 and win valuable prizes: Start the survey!

A utility which manages cloud instances and aids in setting up a fuzzing cluster.

Project description

Logo

Laniakea is a utility for managing instances at various cloud providers and aids in setting up a fuzzing cluster.

Build Status IRC

Table of Contents

Setup

python3 -m pip install laniakea

or

pipenv install laniakea
pipenv run laniakea -h

Laniakea Help Menu

usage: laniakea [-verbosity {1,2,3,4,5}] [-settings path] [-userdata path] [-list-userdata-macros] [-print-userdata]
                [-userdata-macros k=v [k=v ...]]
                ...

Laniakea Runtime v0.9

Laniakea Cloud Providers:
  Use -h to see the help menu of each provider.


    azure                         Microsoft Azure
    packet                        Packet Bare Metal
    ec2                           Amazon Elastic Cloud Computing

Laniakea Base Parameters:
  -verbosity {1,2,3,4,5}          Log sensitivity. (default: 2)
  -settings path                  Laniakea core settings. (default: /Users/posidron/Library/Application
                                  Support/laniakea/laniakea.json)

UserData Parameters:
  -userdata path                  UserData script for the provisioning process. (default: None)
  -list-userdata-macros           List available macros. (default: False)
  -print-userdata                 Print the UserData script to stdout. (default: False)
  -userdata-macros k=v [k=v ...]  Custom macros for the UserData. (default: None)

The exit status is 0 for non-failures and 1 for failures.

Packet Bare Metal

Add your Packet auth token and a project name with the associated project id to the packet.json configuration file.

cat laniakea/examples/packet.json
{
    "auth_token": "YOUR_AUTH_TOKEN",
    "projects": {
        "fuzzing": "YOUR_PROJECT_ID"
    }
}

Basic Usage Examples

Creating either on-demand (-create-demand) or spot (-create-spot) devices:

laniakea packet -project fuzzing -create-demand -tags fuzzers -count 3

Show created devices by applying a tag based filter:

laniakea packet -project fuzzing -list-devices -only tags=fuzzers

Terminate all devices, matching the filter criteria:

laniakea packet -project fuzzing -terminate -only tags=fuzzers

Packet Help Menu

usage: laniakea packet [-h] [-create-demand | -create-spot | -reboot [n] | -stop [n] | -terminate [n]]
                       [-create-volume s [s ...]] [-conf path] [-list-projects] [-list-plans] [-list-operating-systems]
                       [-list-spot-prices] [-list-facilities] [-list-devices] [-project project] [-tags seq [seq ...]]
                       [-region region] [-os name] [-plan name] [-max-spot-price #] [-count #] [-only k=v [k=v ...]]

optional arguments:
  -h, --help                show this help message and exit

Mandatory Packet Parameters:
  -create-demand            Create an on demand based bare metal device instance. (default: False)
  -create-spot              Create a spot price based bare metal device instance. (default: False)
  -reboot [n]               Reboot active instances. (default: None)
  -stop [n]                 Stop active instances. (default: None)
  -terminate [n]            Terminate active instances. (default: None)

Optional Parameters:
  -create-volume s [s ...]  Create storage: <plan> <size> <region> <description> (default: None)
  -conf path                Packet configuration (default: /Users/posidron/Library/Application
                            Support/laniakea/examples/packet/packet.json)
  -list-projects            List available projects. (default: False)
  -list-plans               List available plans. (default: False)
  -list-operating-systems   List available operating systems. (default: False)
  -list-spot-prices         List spot prices. (default: False)
  -list-facilities          List available facilities. (default: False)
  -list-devices             List devices under given project name. (default: False)
  -project project          The project to perform operations on. (default: fuzzing)
  -tags seq [seq ...]       Tags associated with the instance. (default: None)
  -region region            The facility in which the instance is going to run. (default: nrt1)
  -os name                  The operating system for the created instance. (default: ubuntu_18_04)
  -plan name                The instance type to run. (default: baremetal_0)
  -max-spot-price #         Max price for spot instances. (default: 0.05)
  -count #                  The amount of devices to be spawned. (default: 1)
  -only k=v [k=v ...]       Filter instances by criterias. (default: None)

Amazon EC2

Add your AWS credentials to a custom profile inside your ~/.boto configuration file.

[profile laniakea]
aws_access_key_id = <your_access_key_id>
aws_secret_access_key = <your_secret_key>

Complement the provided amazon.json file with your AWS AMI information (see laniakea -h for location).

# Example: an on-demand instance
"default": {
  "image_id":"ami-<AMI_ID>",
  "instance_type": "<INSTANCE_TYPE>",
  "security_groups": ["laniakea"],
  "key_name": "<AWS_KEY_NAME>",
  "instance_profile_name": "<name-of-role>",
  "min_count": 3,
  "max_count": 3
}

# Example: a spot instance
"peach": {
  "image_id":"ami-<AMI_ID>",
  "instance_type": "<INSTANCE_TYPE>",
  "security_groups": ["laniakea"],
  "key_name": "<AWS_KEY_NAME>",
  "instance_profile_name": "<name-of-role>",
  "count": 3
}

Add your UserData script - which is going to be used for provisioning your EC2 instances - to the userdata/ folder.

In the likely case that you want to use a custom UserData script rather than modifying the default.sh file, then you need to point the -userdata parameter to that file.

Please refer to https://help.ubuntu.com/community/CloudInit to learn more about UserData scripts.

Basic Usage Examples

Run N on-demand instances with a custom -userdata script

laniakea ec2 -create-on-demand -tags Name=peach -userdata userdata/peach.private.sh

Run N spot instances with a custom -userdata script and a -max-spot-price of $0.05

laniakea ec2 -create-spot -tags Name=peach -image-name peach -userdata userdata/peach.private.sh -image-args count=10

Show which instances are running and are tagged with the name 'peach'

laniakea ec2 -status -only tag:Name=peach instance-state-code=16

Filters support wildcards. Example: "tag:Name=peach-*" would be suitable to list all instances having the word "peach" as prefix of a tag name. For a list of available filters refer to http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-DescribeInstances.html

Terminate all running instances which are tagged with the name 'peach'

laniakea ec2 -terminate -only tag:Name=peach

Scale down and terminate the oldest N running instances

laniakea ec2 -terminate N -only tag:Name=peach

Terminate a specific instance by id

laniakea ec2 -status -only tag:Name=peach instance-id=i-9110fa9e

List available macros in a UserData script

laniakea ec2 -list-userdata-macros -userdata userdata/peach.pit.sh

EC2 Help Menu

python3 -m laniakea ec2 -h
usage: laniakea ec2 [-h] [-create-on-demand | -create-spot | -stop [n] | -terminate [n] | -status | -run cmd |
                    -list-userdata-macros | -print-userdata] [-userdata path] [-userdata-macros k=v [k=v ...]]
                    [-tags k=v [k=v ...]] [-only k=v [k=v ...]] [-images path] [-image-name str]
                    [-image-args k=v [k=v ...]] [-profile str] [-max-spot-price #] [-region REGION] [-zone ZONE]
                    [-root-device-type {ebs,instance_store}] [-ebs-size EBS_SIZE] [-ebs-volume-type {gp2,io1,standard}]
                    [-ebs-volume-delete-on-termination]

optional arguments:
  -h, --help                            show this help message and exit

Mandatory EC2 Parameters:
  -create-on-demand                     Create on-demand instances. (default: False)
  -create-spot                          Create spot instances. (default: False)
  -stop [n]                             Stop active instances. (default: None)
  -terminate [n]                        Terminate active instances. (default: None)
  -status                               List current state of instances. (default: False)
  -run cmd                              Execute commands via SSH (default: )
  -list-userdata-macros                 List available macros. (default: False)
  -print-userdata                       Print the UserData script to stdout. (default: False)

UserData Parameters:
  -userdata path                        UserData script for cloud-init process. (default:
                                        /Users/posidron/Library/Application Support/laniakea/userdata/ec2/default.sh)
  -userdata-macros k=v [k=v ...]        Custom macros for the UserData. (default: None)

Optional Parameters:
  -tags k=v [k=v ...]                   Assign tags to instances. (default: None)
  -only k=v [k=v ...]                   Filter instances by criterias. (default: None)
  -images path                          EC2 image definitions. (default: /Users/posidron/Library/Application
                                        Support/laniakea/amazon.json)
  -image-name str                       Name of image definition. (default: default)
  -image-args k=v [k=v ...]             Custom image arguments. (default: None)
  -profile str                          AWS profile name in the .boto configuration. (default: laniakea)
  -max-spot-price #                     Max price for spot instances. (default: 0.05)
  -region REGION                        EC2 region name. (default: us-west-2)
  -zone ZONE                            EC2 placement zone. (default: None)
  -root-device-type {ebs,instance_store}
                                        The root device type. (default: ebs)
  -ebs-size EBS_SIZE                    The root disk space size. (default: None)
  -ebs-volume-type {gp2,io1,standard}   The root disk volume type. (default: gp2)
  -ebs-volume-delete-on-termination     Delete the root EBS volume on termination. (default: False)

UserData Reference

Laniakea supports various macros to construct and maintain user-data files.

@import(path_to_other_userdata_file)@
@macro_name@

You can use the -list-userdata-macros option to print out available macros inside a user-data file. Each of these macros can then be substituted with the -userdata-macros option.

Azure

Laniakea supports supports Azure by creating Virtual Machine instances using Azure Resource Management (ARM) Templates. These are JSON files that describe how a Virtual Machine should be set up and deployed. This includes parameters such as: machine size, OS parameters, configuration scripts, etc. An example template can be found in the laniaka/examples/azure/template.json. An example configuration script can be found at http://www.github.com/rforbes/azure-configs/deploy-domino.ps1

When we create resources in Azure we start by creating a Resource Group. Azure uses the Resource Group to store all the resources that are created. This includes, the Virtual machine, any storage for the VM, network interfaces, and IP addresses. We use the -fuzzer flag to set the name of the Resource Group. The name cannot be longer than 12 characters. In order to delete a pool, we delete the Resource Group.

We keep keys and other secrets in AWS using credstash.

Add your AWS credentials to a custom profile inside your ~/.boto configuration file.

[profile laniakea]
aws_access_key_id = <your_access_key_id>
aws_secret_access_key = <your_secret_key>

Create a azure.json file. This file contains the secrets required for accessing and launching in Azure, the username and password of the VMs being created, and the AWS credentials for accessing credstash. Below is example:

Complement the provided amazon.json file with your AWS AMI information (see laniakea -h for location).

{
    "keys": {
        "subscription_id":  "",
        "client_id": "",
        "client_secret": "",
        "tenant_id": ""
    },
    "credentials": {
        "username": "",
        "password": ""
    },
    "aws-credentials": {
        "aws_key_id":"",
        "aws_secret":""
    }
}

The subscription ID, client ID, client secret, and tenant ID are all found in the Azure portal.

Virtual Machine configuration happens using a powershell script that is called in the ARM template.

THe following section of the ARM template is where the script is set.

"properties": {
    "publisher": "Microsoft.Compute",
    "type": "CustomScriptExtension",
    "typeHandlerVersion": "1.9",
    "autoUpgradeMinorVersion": true,
    "settings": {
        "fileUris": [
            "https://raw.githubusercontent.com/rforbes/azure-configs/master/deploy-domino.ps1"
        ]
    },

Basic Usage Examples

Run 3 instances

laniakea azure -create -fuzzer domino -region eastus count 3

Terminate all running instances

laniakea azure -terminate -group-name domino

Azure Help Menu

python3 -m laniakea azure -h
usage: laniakea azure [-h] [-region name] [-count n] [-create] [-delete] [-group-name name]
                      [-azure path] [-template path]

optional arguments:
  -h, --help        show this help message and exit

Mandatory Azure Parameters:
  -region name      Azure region. (default: None)
  -count n          Number of instances to launch. (default: 1)
  -create           Create an instance pool. (default: False)
  -delete           Delete an instance pool. (default: False)
  -group-name name  Group name to be deleted. (default: None)
  -azure path       Deployment template for Windows Azure (default:
                    C:\Users\rforbes\AppData\Local\Mozilla Security\laniakea\azure.json)

UserData Parameters:
  -template path    Deployment template for Windows Azure (default:
                    C:\Users\rforbes\AppData\Local\Mozilla
                    Security\laniakea\userdata\azure\template.json)

Extending Laniakea

To extend Laniakea with new cloud providers you need to ...

  • Add a new folder in laniakea/core/providers/<cloud_provider>
  • Write a command-line interface and put it into the __init__.py
  • Write an API manager class and name it manager.py
  • Add additional files (i.e userdata scripts) to laniakea/userdata/
  • Add additional configuration files to laniakea/examples/

API Documentation

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
laniakea-1.16.0-py3-none-any.whl (33.7 kB) Copy SHA256 hash SHA256 Wheel py3 Aug 27, 2018
laniakea-1.16.0.tar.gz (27.6 kB) Copy SHA256 hash SHA256 Source None Aug 27, 2018

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page