No project description provided
Project description
Python command line app for managing groups of secrets (passwords, API keys, etc) and other project variables. Reduces security risks from things like weak default passwords, secrets stored in files in the source code repository directory.
Version: 23.4.2
Free software: Apache 2.0 License
Documentation: https://python_secrets.readthedocs.org.
Features
Uses the openstack/cliff command line framework for a robust and full-featured CLI. It is easy to add new commands and features!
Supports a “drop-in” model for defining variables in a modular manner (something like the python-update-dotdee program), supporting simplified bulk setting or generating values of variables as needed.
Like python-update-dotdee, psec produces a single master .json file to hold variables defined by the drop-in group description files. That means you can use that file directly to set variables to be used from within other programs like Ansible (e.g. ansible-playbook playbook.yml -e @"$(psec secrets path)")
Support multiple simultaneous sets of secrets (environments) for flexibility and scalability in multi-environment deployments and to support different use cases or different combinations of secrets.
Supports changing the storage location of secrets and variables to allow them to be stored on secure mobile media (such as self-encrypting external SSD or Flash drives) or encrypted disk images mounted at run-time to ensure the confidentiality of data at rest.
List the groups of variables (and how many secrets in each group).
Describe secrets by their variable name and type (e.g., password, uuid4, random_base64). You can also include a descriptive string to prompt the user for a value, a list of options to choose from (or * for “any value the user enters”), and a list of environment variables to export for other programs to use at run time.
Allows manual entry of values, setting non-secret variables from a default value, or automatic generation of secrets according to their type.
Manually set string variables based on the output of simple commands. This allows interfacing with external programs for obtaining secrets, such as Vault by Hashicorp.
Generate unique values for variables, or use a single value per type to simplify use of secrets in access control of services while supporting a “break-glass” process to quickly regenerate secrets when needed.
Show the variables and their unredacted values (or redacted them to maintain secrecy during demonstrations or in documentation).
Export the variables (optionally with a specific prefix string) to the environment and run a command that inherits them (e.g., to pass variables to terraform for provisioning cloud instances).
Output the variables and values in multiple different formats (CSV, JSON, YAML) for use in shell scripts, etc. using cliff features.
Send secrets to other users on demand using GPG encrypted email to protect the secrets while in transit and while at rest in users’ email inboxes.
Makes it easy to store temporary files (e.g., the output from Jinja template rendering) that may contain secrets outside of the source repo directory in an environment-specific tmp/ directory.
Limitations
Secrets are stored in unencrypted form in the environments directories. Permissions are set to limit access, but this is not an “encrypt data at rest” solution like Vault by Hashicorp.
Does not handle secure distributed access for users on remote systems. You must use something like Vault by Hashicorp or libfuse/sshfs for secure (realtime) distributed access.
Does not handle secure distribution of newly generated secrets out to distributed systems that need them. You will need to use a program like Ansible and related playbooks for pushing out and changing secrets (or for retrieving backups). Look at the D2 Ansible playbooks (https://github.com/davedittrich/ansible-dims-playbooks) for example playbooks for doing these tasks.
Does not clean up the environment-specific tmp/ directories. (You need to handle that in code, but at least they are less likely to end up in a Git commit.)
Usage Concepts
There is a separate Usage chapter with individual command documentation. The remainder of this section covers higher level usage concepts necessary to best use the python_secrets package in your open source software project.
Directories and files
There are three file system concepts that are important to understand regarding secrets storage:
The root secrets base directory for secrets storage;
The environment for organizing a set of secrets and secret group descriptions;
The secrets file and group descriptions.
Secrets Base Directory
psec expects to store all of files in a directory tree known as a secrets base directory. Originally, this was intended to be located in the current user’s home directory. Unless you over-ride the name of this directory, it defaults to .secrets on Linux and secrets on Windows.
The ability to locate this directory in a different file system path is supported by command line options and an environment variable so you can store files on an exported file share, in a common location for use by a group on a workstation, or to move the contents to an encrypted disk or a different partition with more disk space.
The first time you use ever use psec, there will likely be no directory:
$ tree ~/.secrets
/Users/dittrich/.secrets [error opening dir]
0 directories, 0 files
Environments
Environments are sub-directories within the root secrets directory. You can just create the directory structure without any files. You create one environment per set of unique secrets that you need to manage. This could be one for open source Program A, one for Program B, etc., or it could be one for development, one for testing, one for production, etc. (or any combination).
The command environments create creates an environment. Since this program is designed to support multiple environments, a name for the new environment is required. The name of the environment can be provided explicitly, or it can be inferred from the base name of the current working directory:
$ pwd
/Users/dittrich/git/python_secrets
$ psec environments create
environment directory /Users/dittrich/.secrets/python_secrets created
$ tree ~/.secrets
/Users/dittrich/.secrets
└── python_secrets
└── secrets.d
2 directories, 0 files
Let’s say we want to create empty environments for the three deployments (development, testing, and production). The names can be assigned explicitly by (a) giving an argument on the command line, (b) using the -e or --environment command line flag, or (c) by setting the environment variable D2_ENVIRONMENT:
$ psec environments create development
environment directory /Users/dittrich/.secrets/development created
$ psec --environment testing environments create
environment directory /Users/dittrich/.secrets/testing created
$ D2_ENVIRONMENT=production psec environments create
environment directory /Users/dittrich/.secrets/production created
$ tree ~/.secrets
/Users/dittrich/.secrets
├── development
│ └── secrets.d
├── production
│ └── secrets.d
├── python_secrets
│ └── secrets.d
└── testing
└── secrets.d
8 directories, 0 files
If you want to create more than one environment at once, you will have to specify all of the names on the command line as arguments:
$ psec environments create development testing production
environment directory /Users/dittrich/.secrets/development created
environment directory /Users/dittrich/.secrets/testing created
environment directory /Users/dittrich/.secrets/production created
If you are using one source repository for building multiple deployments, of course you can’t rely on the basename of the directory for all deployments. The default environment can be set, shown, or unset, using the environments default command.
$ psec environments default --help
usage: psec environments default [-h] [--unset-default] [environment]
Manage default environment via file in cwd
positional arguments:
environment
optional arguments:
-h, --help show this help message and exit
--unset-default Unset localized environment default
If no default is explicitly set, the default that would be applied is returned:
$ cd ~/git/python_secrets
$ psec environments default
default environment is "python_secrets"
You can get a list of all available environments at any time, including which one would be the default used by sub-commands:
$ psec environments list
+-------------+---------+
| Environment | Default |
+-------------+---------+
| development | No |
| testing | No |
| production | No |
+-------------+---------+
The following shows setting and unsetting the default:
$ psec environments default testing
default environment set to "testing"
$ psec environments default
testing
$ psec environments list
+-------------+---------+
| Environment | Default |
+-------------+---------+
| development | No |
| testing | Yes |
| production | No |
+-------------+---------+
$ psec environments default --unset-default
default environment unset
The environment directories are useable for storing all secrets and sensitive files (e.g., backups of certificates, databases, etc.) associated with an environment.
For convenience, there is a command environments tree that produces output similar to the Unix tree command:
$ psec -e d2 environments tree
/Users/dittrich/.secrets/d2
├── backups
│ ├── black.secretsmgmt.tk
│ │ ├── letsencrypt_2018-04-06T23:36:58PDT.tgz
│ │ └── letsencrypt_2018-04-25T16:32:20PDT.tgz
│ ├── green.secretsmgmt.tk
│ │ ├── letsencrypt_2018-04-06T23:45:49PDT.tgz
│ │ └── letsencrypt_2018-04-25T16:32:20PDT.tgz
│ ├── purple.secretsmgmt.tk
│ │ ├── letsencrypt_2018-04-25T16:32:20PDT.tgz
│ │ ├── trident_2018-01-31T23:38:48PST.tar.bz2
│ │ └── trident_2018-02-04T20:05:33PST.tar.bz2
│ └── red.secretsmgmt.tk
│ ├── letsencrypt_2018-04-06T23:45:49PDT.tgz
│ └── letsencrypt_2018-04-25T16:32:20PDT.tgz
├── dittrich.asc
├── keys
│ └── opendkim
│ └── secretsmgmt.tk
│ ├── 201801.private
│ ├── 201801.txt
│ ├── 201802.private
│ └── 201802.txt
├── secrets.d
│ ├── ca.json
│ ├── consul.json
│ ├── jenkins.json
│ ├── rabbitmq.json
│ ├── trident.json
│ ├── vncserver.json
│ └── zookeper.json
├── secrets.json
└── vault_password.txt
To just see the directory structure and not files, add the --no-files option:
$ psec -e d2 environments tree --no-files
/Users/dittrich/.secrets/d2
├── backups
│ ├── black.secretsmgmt.tk
│ ├── green.secretsmgmt.tk
│ ├── purple.secretsmgmt.tk
│ └── red.secretsmgmt.tk
├── keys
│ └── opendkim
│ └── secretsmgmt.tk
└── secrets.d
Secrets and group descriptions
The environment directories just created are all empty. Secrets are stored in a JSON file (.json) within the environment’s directory, and group descriptions are stored in a drop-in directory with the same base name, but with an extention of .d instead of .json (following the Linux drop-in configuration style directories used by programs like rsyslog, dnsmasq, etc.)
The default secrets file name is secrets.json, which means the default descriptions directory would be named secrets.d.
You can define environment variables to point to the secrets base directory in which a set of different environments can be configured at one time, to define the current environment, and to change the name of the secrets file to something else.
$ env | grep ^D2_
D2_SECRETS_BASEDIR=/Users/dittrich/.secrets
D2_ENVIRONMENT=do
Each environment is in turn rooted in a directory with the environment’s symbolic name (e.g., do for DigitalOcean in this example, and goSecure for the GitHub davedittrich/goSecure VPN project.)
$ tree -L 1 ~/.secrets
/Users/dittrich/.secrets
├── do
└── goSecure
3 directories, 0 files
Each set of secrets for a given service or purpose is described in its own file.
.
├── secrets.d
│ ├── ca.json
│ ├── consul.json
│ ├── jenkins.json
│ ├── rabbitmq.json
│ ├── trident.json
│ ├── vncserver.json
│ └── zookeper.json
└── secrets.json
You can see one of the descriptions files from the template in this repository using cat tests/secrets.d/myapp.json:
[
{
"Variable": "myapp_pi_password",
"Type": "password",
"Prompt": "Password for myapp 'pi' user account",
"Export": "DEMO_pi_password"
},
{
"Variable": "myapp_app_password",
"Type": "password",
"Prompt": "Password for myapp web app",
"Export": "DEMO_app_password"
},
{
"Variable": "myapp_client_psk",
"Type": "string",
"Prompt": "Pre-shared key for myapp client WiFi AP",
"Options": "*",
"Export": "DEMO_client_psk"
},
{
"Variable": "myapp_client_ssid",
"Type": "string",
"Prompt": "SSID for myapp client WiFi AP",
"Options": "myapp_ssid,*",
"Export": "DEMO_client_ssid"
},
{
"Variable": "myapp_ondemand_wifi",
"Type": "boolean",
"Prompt": "'Connect on demand' when connected to wifi",
"Options": "true,false",
"Export": "DEMO_ondemand_wifi"
},
{
"Variable": "myapp_optional_setting",
"Type": "boolean",
"Prompt": "Optionally do something",
"Options": "false,true",
"Export": "DEMO_options_setting"
}
]
The psec program uses the openstack/cliff command line interface framework, which supports multiple output formats. The default format the table format, which makes for nice clean output. (Other formats will be described later.)
The groups can be listed using the groups list command:
$ psec groups list
+---------+-------+
| Group | Items |
+---------+-------+
| jenkins | 1 |
| myapp | 4 |
| trident | 2 |
+---------+-------+
The variables in one or more groups can be shown with the groups show command:
$ psec groups show trident myapp
+---------+-----------------------+
| Group | Variable |
+---------+-----------------------+
| trident | trident_sysadmin_pass |
| trident | trident_db_pass |
| myapp | myapp_app_password |
| myapp | myapp_client_psk |
| myapp | myapp_client_ssid |
| myapp | myapp_ondemand_wifi |
| myapp | myapp_pi_password |
+---------+-----------------------+
When integrating a new open source tool or project, you can create a new group and clone its secrets descriptions. This does not copy any values, just the descriptions, allowing the current environment to manage its own values.
$ psec groups create newgroup --clone-from ~/git/goSecure/secrets/secrets.d/gosecure.json
created new group "newgroup"
$ psec groups list 2>/dev/null
+----------+-------+
| Group | Items |
+----------+-------+
| jenkins | 1 |
| myapp | 5 |
| newgroup | 12 |
| trident | 2 |
+----------+-------+
Showing Secrets
To examine the secrets, use the secrets show command:
$ psec secrets show
+------------------------+----------+----------+------------------------+
| Variable | Type | Value | Export |
+------------------------+----------+----------+------------------------+
| jenkins_admin_password | password | REDACTED | jenkins_admin_password |
| myapp_app_password | password | REDACTED | DEMO_app_password |
| myapp_client_psk | string | REDACTED | DEMO_client_ssid |
| myapp_client_ssid | string | REDACTED | DEMO_client_ssid |
| myapp_ondemand_wifi | boolean | REDACTED | DEMO_ondemand_wifi |
| myapp_pi_password | password | REDACTED | DEMO_pi_password |
| trident_db_pass | password | REDACTED | trident_db_pass |
| trident_sysadmin_pass | password | REDACTED | trident_sysadmin_pass |
+------------------------+----------+----------+------------------------+
By default, the values of secrets are redacted when output. To show the values in clear text in the terminal output, add the --no-redact flag:
$ psec secrets show --no-redact
+------------------------+----------+------------------------------+------------------------+
| Variable | Type | Value | Export |
+------------------------+----------+------------------------------+------------------------+
| jenkins_admin_password | password | fetch.outsider.awning.maroon | jenkins_admin_password |
| myapp_app_password | password | fetch.outsider.awning.maroon | DEMO_app_password |
| myapp_client_psk | string | PSK | DEMO_client_psk |
| myapp_client_ssid | string | SSID | DEMO_client_ssid |
| myapp_ondemand_wifi | boolean | true | DEMO_ondemand_wifi |
| myapp_pi_password | password | fetch.outsider.awning.maroon | DEMO_pi_password |
| trident_db_pass | password | fetch.outsider.awning.maroon | trident_db_pass |
| trident_sysadmin_pass | password | fetch.outsider.awning.maroon | trident_sysadmin_pass |
+------------------------+----------+------------------------------+------------------------+
If you don’t care about redaction and want to turn it off and save the dozen keystrokes it takes to type `` –no-redact``, you can export the environment variable D2_NO_REDACT set to (case-insensitive) “true”, “1”, or “yes”. Anything else leaves the default the same. We’ll do this now for later examples.
$ export D2_NO_REDACT=true
The default is also to show all secrets. If you only want to process a subset of secrets, you have two ways to do this.
Specify the variables you want to show on the command line as arguments:
$ psec secrets show rabbitmq_default_user_pass rabbitmq_admin_user_pass +----------------------------+----------+--------------------------------------+ | Variable | Type | Value | +----------------------------+----------+--------------------------------------+ | rabbitmq_default_user_pass | password | handheld.angrily.letdown.frisk | | rabbitmq_admin_user_pass | password | handheld.angrily.letdown.frisk | +----------------------------+----------+--------------------------------------+
Use the --group flag and specify the group(s) you want to show as command line arguments:
$ psec secrets show --group jenkins trident +----------------------------+----------+--------------------------------------+ | Variable | Type | Value | +----------------------------+----------+--------------------------------------+ | jenkins_admin_password | password | handheld.angrily.letdown.frisk | | trident_db_pass | password | handheld.angrily.letdown.frisk | | trident_sysadmin_pass | password | handheld.angrily.letdown.frisk | +----------------------------+----------+--------------------------------------+
Describing Secrets and Secret Types
To describe the secrets in the select environment, use the secrets describe command:
$ psec secrets describe
+----------------------------+----------+--------------------------------------------+
| Variable | Type | Prompt |
+----------------------------+----------+--------------------------------------------+
| google_oauth_client_id | string | Google OAuth2 client id |
| google_oauth_client_secret | string | Google OAuth2 client secret |
| google_oauth_refresh_token | string | Google OAuth2 refresh token |
| google_oauth_username | None | google_oauth_username |
| jenkins_admin_password | password | Password for Jenkins "admin" account |
| myapp_app_password | password | Password for myapp web app |
| myapp_client_psk | string | Pre-shared key for myapp client WiFi AP |
| myapp_client_ssid | string | SSID for myapp client WiFi AP |
| myapp_ondemand_wifi | boolean | "Connect on demand" when connected to wifi |
| myapp_pi_password | password | Password for myapp "pi" user account |
| trident_db_pass | password | Password for Trident postgres database |
| trident_sysadmin_pass | password | Password for Trident sysadmin account |
+----------------------------+----------+--------------------------------------------+
$ psec secrets describe --group trident
+-----------------------+----------+----------------------------------------+
| Variable | Type | Prompt |
+-----------------------+----------+----------------------------------------+
| trident_db_pass | password | Password for Trident postgres database |
| trident_sysadmin_pass | password | Password for Trident sysadmin account |
+-----------------------+----------+----------------------------------------+
To get a description of the available secret types, add the --types flag.
$ psec secrets describe --types
+------------------+----------------------------------+
| Type | Description |
+------------------+----------------------------------+
| password | Simple (xkcd) password string |
| string | Simple string |
| boolean | Boolean ("true"/"false") |
| crypt_6 | crypt() SHA512 ("$6$") |
| token_hex | Hexadecimal token |
| token_urlsafe | URL-safe token |
| sha256_digest | DIGEST-SHA256 (user:pass) digest |
| uuid4 | UUID4 token |
| random_base64 | Random BASE64 token |
+------------------+----------------------------------+
Generating and Setting variables
Secrets are generated using the secrets generate command and are set manually using the secrets set command.
$ psec help secrets generate
usage: psec secrets generate [-h] [-U] [args [args ...]]
Generate values for secrets
positional arguments:
args
optional arguments:
-h, --help show this help message and exit
-U, --unique Generate unique values for each type of secret (default:
False)
..
$ psec secrets set --help
usage: psec secrets set [-h] [--undefined] [args [args ...]]
Set values manually for secrets
positional arguments:
args
optional arguments:
-h, --help show this help message and exit
--undefined Set values for undefined variables (default: False)
To regenerate all of the non-string secrets at once, using the same value for each type of secret to simplify things, use the secrets generate command:
$ psec secrets generate
$ psec secrets show --column Variable --column Value
+----------------------------+----------------------------------------------+
| Variable | Value |
+----------------------------+----------------------------------------------+
| trident_db_pass | gargle.earlobe.eggplant.kissable |
| ca_rootca_password | gargle.earlobe.eggplant.kissable |
| consul_key | HEvUAItLFZ0+GjxfwTxLDKq5Fbt86UtXrInzpf71GGY= |
| jenkins_admin_password | gargle.earlobe.eggplant.kissable |
| rabbitmq_default_user_pass | gargle.earlobe.eggplant.kissable |
| rabbitmq_admin_user_pass | gargle.earlobe.eggplant.kissable |
| trident_sysadmin_pass | gargle.earlobe.eggplant.kissable |
| vncserver_password | gargle.earlobe.eggplant.kissable |
| zookeeper_uuid4 | 769a77ad-b06f-4018-857e-23f970c777c2 |
+----------------------------+----------------------------------------------+
You can set one or more variables manually using secrets set and specifying the variable and value in the form variable=value:
$ psec secrets set trident_db_pass="rural coffee purple sedan"
$ psec secrets show --column Variable --column Value
+----------------------------+----------------------------------------------+
| Variable | Value |
+----------------------------+----------------------------------------------+
| trident_db_pass | rural coffee purple sedan |
| ca_rootca_password | gargle.earlobe.eggplant.kissable |
| consul_key | HEvUAItLFZ0+GjxfwTxLDKq5Fbt86UtXrInzpf71GGY= |
| jenkins_admin_password | gargle.earlobe.eggplant.kissable |
| rabbitmq_default_user_pass | gargle.earlobe.eggplant.kissable |
| rabbitmq_admin_user_pass | gargle.earlobe.eggplant.kissable |
| trident_sysadmin_pass | gargle.earlobe.eggplant.kissable |
| vncserver_password | gargle.earlobe.eggplant.kissable |
| zookeeper_uuid4 | 769a77ad-b06f-4018-857e-23f970c777c2 |
+----------------------------+----------------------------------------------+
Or you can generate one or more variables in a similar manner by adding them to the command line as arguments to secrets generate:
$ psec secrets generate rabbitmq_default_user_pass rabbitmq_admin_user_pass
$ psec secrets show --column Variable --column Value
+----------------------------+----------------------------------------------+
| Variable | Value |
+----------------------------+----------------------------------------------+
| trident_db_pass | rural.coffee.purple.sedan |
| ca_rootca_password | gargle.earlobe.eggplant.kissable |
| consul_key | HEvUAItLFZ0+GjxfwTxLDKq5Fbt86UtXrInzpf71GGY= |
| jenkins_admin_password | gargle.earlobe.eggplant.kissable |
| rabbitmq_default_user_pass | embezzle.xerox.excess.skydiver |
| rabbitmq_admin_user_pass | embezzle.xerox.excess.skydiver |
| trident_sysadmin_pass | gargle.earlobe.eggplant.kissable |
| vncserver_password | gargle.earlobe.eggplant.kissable |
| zookeeper_uuid4 | 769a77ad-b06f-4018-857e-23f970c777c2 |
+----------------------------+----------------------------------------------+
A set of secrets for an open source project can be bootstrapped using the following steps:
Create a template secrets environment directory that contains just the secrets definitions. This example uses the template found in the davedittrich/goSecure repository (directory https://github.com/davedittrich/goSecure/tree/master/secrets).
Use this template to clone a secrets environment, which will initially be empty:
$ psec environments create test --clone-from ~/git/goSecure/secrets new password variable "gosecure_app_password" is unset new string variable "gosecure_client_ssid" is unset new string variable "gosecure_client_ssid" is unset new string variable "gosecure_client_psk" is unset new password variable "gosecure_pi_password" is unset new string variable "gosecure_pi_pubkey" is unset environment directory /Users/dittrich/.secrets/test created
$ psec -e test secrets show --no-redact --fit-width +-----------------------+----------+-------+ | Variable | Type | Value | +-----------------------+----------+-------+ | gosecure_app_password | password | None | | gosecure_client_ssid | string | None | | gosecure_client_psk | string | None | | gosecure_pi_password | password | None | | gosecure_pi_pubkey | string | None | +-----------------------+----------+-------+
First, generate all secrets whose type is not string:
$ psec -e test secrets generate new password variable "gosecure_app_password" is unset new string variable "gosecure_client_ssid" is unset new string variable "gosecure_client_ssid" is unset new string variable "gosecure_client_psk" is unset new password variable "gosecure_pi_password" is unset new string variable "gosecure_pi_pubkey" is unset $ psec -e test secrets show --no-redact --fit-width +-----------------------+----------+------------------------------+ | Variable | Type | Value | +-----------------------+----------+------------------------------+ | gosecure_app_password | password | brunt.outclass.alike.turbine | | gosecure_client_psk | string | None | | gosecure_client_ssid | string | None | | gosecure_pi_password | password | brunt.outclass.alike.turbine | | gosecure_pi_pubkey | string | None | +-----------------------+----------+------------------------------+
Finally, manually set the remaining string type variables:
$ psec -e test secrets set --undefined new string variable "gosecure_client_psk" is unset new string variable "gosecure_client_ssid" is unset new string variable "gosecure_pi_pubkey" is unset Pre-shared key for goSecure client WiFi AP? [None]: atjhK5AlsQMw3Zh SSID for goSecure client WiFi AP? [None]: YourWiFiSSID SSH public key for accessing "pi" account? [None]: @~/.ssh/new_rsa.pub $ psec -e test secrets show --no-redact --fit-width +-----------------------+----------+------------------------------------------------------------------------------------------+ | Variable | Type | Value | +-----------------------+----------+------------------------------------------------------------------------------------------+ | gosecure_app_password | password | brunt.outclass.alike.turbine | | gosecure_client_psk | string | atjhK5AlsQMw3Zh | gosecure_client_ssid | string | YourWiFiSSID | | gosecure_pi_password | password | brunt.outclass.alike.turbine | | gosecure_pi_pubkey | string | ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+qUIucrPvRkTmY0tgxr9ac/VtBUHhYHfOdDVpU99AcryLMWiU | | | | uQ2/NVikfOfPo5mt9YTQyqRbeBzKlNgbHnsxh0AZatjhK5AlsQMw3ZhZUcLYZbt7szuQy8ineN0potlCJoVaMSOb | | | | 9htf9gAPvzwxUnHxg35jPCzAXYAi3Erc6y338+CL0XxQvCogXOA+MwH7wZGgdT3WpupLG/7HAr/3KJEQQk1FlS2m | | | | Rd+WuewnLbKkqBP21N+48ccq6XhEhAmlzzr9SENw5DMmrvMAYIYkoTwUeD3Qx4YebjFkCxZw+w7AafEFn0Kz6vCX | | | | 4mp/6ZF/Ko+o04HM2sVr6wtCu2dB dittrich@localhost | +-----------------------+----------+------------------------------------------------------------------------------------------+
You are now ready to compile your software, or build your project!
There is also a mechanism to run simple commands (i.e., basic arguments with no special inline command substitution or variable expansion features of shells like bash) and use the resulting output as the value.
For this example, let’s assume an environment that requires a CIDR notation address for ingres access control (e.g., when using Amazon Web Services to allow control of instances from your remote laptop).
$ psec -e xgt secrets set aws_cidr_allowed=""
$ psec -e secrets show --no-redact aws_cidr_allowed
+------------------+--------+-------+
| Variable | Type | Value |
+------------------+--------+-------+
| aws_cidr_allowed | string | |
+------------------+--------+-------+
The psec program has a utility feature that will return the current routable IP source address as an IP address, or using CIDR notation. The variable can be set in one of two ways:
Via (non-interactive) inline command subtitution from the terminal shell:
$ psec -e xgt secrets set aws_cidr_allowed="$(psec utils myip --cidr)"
Interactively when prompted using simple command line form:
$ psec -e xgt secrets set aws_cidr_allowed aws_cidr_allowed? []: !psec utils myip --cidr
The variable now contains the output of the specified program:
$ psec secrets show --no-redact aws_cidr_allowed
+------------------+--------+------------------+
| Variable | Type | Value |
+------------------+--------+------------------+
| aws_cidr_allowed | string | 93.184.216.34/32 |
+------------------+--------+------------------+
Processing templates
Outputting structured information for use in other scripts
Once secrets are created and stored, they will eventually need to be accessed in order to use them in program execution. This can be done by passing the .json secrets file itself to a program, or by outputting the variables in other formats like CSV, JSON, or as environment type variables.
Passing the secrets file by path
One way to do this is to take advantage of command line options like Ansible’s --extra-vars and passing it a path to the .json secrets file. (See Passing Variables On The Command Line). Here is how to do it.
Let’s assume we want to use consul_key variable to configure Consul using Ansible. Here is the variable as stored:
$ psec secrets show consul_key
+------------+-----------+----------------------------------------------+
| Variable | Type | Value |
+------------+-----------+----------------------------------------------+
| consul_key | token_hex | HEvUAItLFZ0+GjxfwTxLDKq5Fbt86UtXrInzpf71GGY= |
+------------+-----------+----------------------------------------------+
Using Ansible’s debug module, we can verify that this variable is not set by any previously loaded Ansible inventory:
$ ansible -i localhost, -m debug -a 'var=consul_key' localhost
localhost | SUCCESS => {
"consul_key": "VARIABLE IS NOT DEFINED!"
}
In order for Ansible to set the consul_key variable outside of any pre-defined inventory files, we need to pass a file path to the --extra-vars option. The path can be obtained using the psec secrets path command:
$ psec secrets path
/Users/dittrich/.secrets/python_secrets/secrets.json
It is possible to run this command in an in-line command expansion operation in Bash. Ansible expects the file path passed to -extra-vars to start with an @ character, so the command line to use would look like this:
$ ansible -i localhost, -e @"$(psec secrets path)" -m debug -a 'var=consul_key' localhost
localhost | SUCCESS => {
"consul_key": "HEvUAItLFZ0+GjxfwTxLDKq5Fbt86UtXrInzpf71GGY="
}
Ansible now has the value and can use it in templating configuration files, or so forth.
Other programs like Hashicorp terraform look for environment variables that begin with TF_VAR_ and use them to set terraform variables for use in modules. To prove we are running in a sub-shell, we will first change the shell prompt.
$ PS1="test> "
test> psec -e test --export-env-vars --env-var-prefix="TEST_" run bash
$ env | grep '^TEST_'
TEST_gosecure_pi_pubkey=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+qUIucrPvRkTmY0tgxr9ac/VtBUHhYHfOdDVpU99AcryLMWiU [...]
TEST_gosecure_client_psk=atjhK5AlsQMw3Zh
TEST_gosecure_client_ssid=YourWiFiSSID
TEST_gosecure_pi_password=brunt.outclass.alike.turbine
TEST_gosecure_app_password=brunt.outclass.alike.turbine
$ exit
test>
Operational Security
As noted in the Limitations section above, secrets are stored in plaintext plaintext form (i.e., they are not encrypted) in files. Those files are in turn stored in a directory in the file system, subject to Linux file ownership and permission access controls.
The default location for storing these files is in an environment directory in a subdirectory of the user’s home directory whose name starts with a period character (a.k.a., a dot). Files (or directories) whose name starts with a period are known as dot files, or hidden files because the ls command does not show it unless you use the -a flag.
The secrets environment directories can also be used to store other files besides secrets. One such use case is storing JSON Web Tokens (JWTs) used as bearer tokens by protocols like Google’s OAuth 2.0 Mechanism for securing access to web services and APIs. While this improves security in terms of remote access, is not not without its own risks (including the JWT file being stored in the file system for an indefinite period of time).
JSON Web Tokens (JWT) are Dangerous for User Sessions—Here’s a Solution, by Raja Rao, June 24, 2021
Stop Using JSON Web Tokens For Authentication. Use Stateful Sessions Instead, by Francisco Sainz, April 4, 2022
What’s the Secure Way to Store JWT?, by Yang Liu, July 23, 2020
Besides JWTs, other use cases for storing sensitive files within psec environments include backups of database contents, Let’s Encrypt certificates, SSH keys, or other secrets necessary for ensuring cloud instances can be destroyed and recreated without losing state or requiring regeneration (and redistribution or revalidation) of secrets.
The output of init –help mentions this risk and offers a way to mitigate some of the risk by locating the secrets storage base directory within a directory that is stored on an encrypted USB-connected disk device or encrypted disk image, or a removable device or remote file system, that is only mounted when needed and unmounted as soon as possible. This ensures sensitive data that are not being actively used are left encrypted in storage. The D2_SECRETS_BASEDIR environment variable or -d option allow you to specify the directory to use.
The psec CLI has a secure deletion mechanism that over-writes file contents prior to deletion, helping to reduce leaving remnants of secrets in unallocated file system storage, similar to the way the Linux shred command works.
Python Script Security
Last, but certainly not least, take the time to read up on Python Security and understand the types and sources of security vulnerabilities related to Python programs. Keep these ideas in mind when using and/or modifying this program.
As part of testing, the Bandit security validation program is used. (See Getting started with Bandit).
In situations where Bandit warnings can safely be ignored, the # nosec comment appears on source code lines. Comments as to why these can be safely ignored are included in the code. (Please feel free to issue pull requests if you disagree.)
One runtime security mechanism employed by psec is control of the process’ umask. This is important when running programs that create files, which will inherit their permissions per the process umask. The umask will be inherited by every new child process and can be set in the user’s .bashrc (or other shell initialization) file.
The psec run command can be used to run programs as child processes, optionally exporting environment variables as well, so controlling the umask results in improved file permission security regardless of whether the user knows to set their process umask.
You can see the effect in these two examples.
First, by setting the umask to 0 you see the very permissive file permissions (as well as getting a warning from psec about finding a file with lax permissions):
$ psec --umask 0o000 run -- dd if=/dev/random count=1 of=$(psec environments path --tmpdir)/foo
1+0 records in
1+0 records out
512 bytes copied, 0.000019 s, 2.7 MB/s
$ ls -l $(psec environments path --tmpdir)/foo
[!] file /Users/dittrich/.secrets/python_secrets/tmp/foo is mode 0o100666
-rw-rw-rw- 1 dittrich staff 512 Sep 8 13:05 /Users/dittrich/.secrets/python_secrets/tmp/foo
$ rm $(psec environments path --tmpdir)/foo
Now when using the default --umask value, the file permissions are restricted (and thus no more warning):
$ psec run -- dd if=/dev/random count=1 of=$(psec environments path --tmpdir)/foo
1+0 records in
1+0 records out
512 bytes copied, 0.000243 s, 2.1 MB/s
$ ls -l $(psec environments path --tmpdir)/foo
-rw------- 1 dittrich staff 512 Sep 8 13:04 /Users/dittrich/.secrets/python_secrets/tmp/foo
$ rm $(psec environments path --tmpdir)/foo
Bugs, Enhancements, and Future Work
Feature requests (and of course bug reports) are highly encouraged. You can do that by opening an issue on GitHub. Better yet, make a pull request with your own fix or feature. (Check there to see if one may already exist.)
If you want to help, there are some things that are on the “to do” list. These are tracked on this repository’s GitHub Projects page.
General or more elaborate potential enhancements are listed here:
Increase test coverage (test driven development is a Good Thing(TM))
The Mantl project (GitHub mantl/mantl) employs a security-setup script that takes care of setting secrets (and non-secret related variables) in a monolithic manner. It has specific command line options, specific secret generation functions, and specific data structures for each of the component subsystems used by mantl/mantl. This method is not modular or extensible, and the security-setup script is not generalized such that it can be used by any other project. These limitations are primary motivators for writing python_secrets, which could eventually replace security-setup.
At this point, the Mantl security.yml file can be read in and values can be manually set, as seen here:
$ psec -d ~/git/mantl --secrets-file security.yml secrets show -f yaml secrets descriptions directory not found - Value: admin:password Variable: chronos_http_credentials - Value: chronos Variable: chronos_principal - Value: S0JMz5z8oxQGQXMyZjwE0ZCmu4zeJV4oWDUrdc25MBLx Variable: chronos_secret - Value: 88821cbe-c004-4cff-9f91-2bc36cd347dc Variable: consul_acl_agent_token - Value: f9acbe14-28d3-4d06-a1c9-c617da5ebb4e Variable: consul_acl_mantl_api_token - Value: de54ae85-8226-4146-959f-8926b0b8ee55 Variable: consul_acl_marathon_token - Value: dfc9b244-5140-41ad-b93a-ac5c2451fb95 Variable: consul_acl_master_token - Value: e149b50f-cb5c-4efe-be96-26a52efdc715 Variable: consul_acl_secure_token - Value: 719f2328-6446-4647-adf6-310013bac636 Variable: consul_acl_vault_token - Value: Z0niD1jeiTkx7xaoewJm2A== Variable: consul_gossip_key - Value: true Variable: do_chronos_auth - Value: true Variable: do_chronos_iptables - Value: true Variable: do_chronos_ssl - Value: true Variable: do_consul_auth - Value: true Variable: do_consul_ssl - Value: true Variable: do_mantl_api_auth - Value: true Variable: do_mantlui_auth - Value: true Variable: do_mantlui_ssl - Value: true Variable: do_marathon_auth - Value: true Variable: do_marathon_iptables - Value: true Variable: do_marathon_ssl - Value: true Variable: do_mesos_auth - Value: true Variable: do_mesos_follower_auth - Value: true Variable: do_mesos_framework_auth - Value: true Variable: do_mesos_iptables - Value: true Variable: do_mesos_ssl - Value: false Variable: do_private_docker_registry - Value: mantl-api Variable: mantl_api_principal - Value: Se4R9nRy8WTAgmU9diJyIPwLYsBU+V1yBxTQumiOriK+ Variable: mantl_api_secret - Value: admin:password Variable: marathon_http_credentials - Value: marathon Variable: marathon_principal - Value: +Y5bvIsWliFvcWgbXGWa8kwT6Qf3etogQJe+cK+IV2hX Variable: marathon_secret - Value: - principal: marathon secret: +Y5bvIsWliFvcWgbXGWa8kwT6Qf3etogQJe+cK+IV2hX - principal: chronos secret: S0JMz5z8oxQGQXMyZjwE0ZCmu4zeJV4oWDUrdc25MBLx - principal: mantl-api secret: Se4R9nRy8WTAgmU9diJyIPwLYsBU+V1yBxTQumiOriK+ Variable: mesos_credentials - Value: follower Variable: mesos_follower_principal - Value: Q53uAa2mNM0UNe2RUjrX6k7QvK6ojjH1gHXYLcm3Lmfr Variable: mesos_follower_secret - Value: password Variable: nginx_admin_password - Value: true Variable: security_enabled - Value: chronos Variable: zk_chronos_user - Value: JWPO11z4lU5qeilZ Variable: zk_chronos_user_secret - Value: hsr+R6YQBAOXoY84a8ne8bU0opg= Variable: zk_chronos_user_secret_digest - Value: marathon Variable: zk_marathon_user - Value: UBh77ok2svQAqWox Variable: zk_marathon_user_secret - Value: mo2mQGXcsc21zB4wYD18jn+Csks= Variable: zk_marathon_user_secret_digest - Value: mesos Variable: zk_mesos_user - Value: L3t9FEMsXehqeBvl Variable: zk_mesos_user_secret - Value: bHYvGteRBxou4jqJ8XWAYmOmzxs= Variable: zk_mesos_user_secret_digest - Value: super Variable: zk_super_user - Value: 2DyL/n/GLi3Q0pa75z9OjODGZKC1RCaEiKNV1ZXo1Wpk Variable: zk_super_user_secret $ psec -d ~/git/mantl --secrets-file security.yml secrets show -f csv | grep nginx_admin_password secrets descriptions directory not found "nginx_admin_password","password" $ psec -d ~/git/mantl --secrets-file security.yml secrets set nginx_admin_password=newpassword secrets descriptions directory not found $ psec -d ~/git/mantl --secrets-file security.yml secrets show -f csv | grep nginx_admin_password secrets descriptions directory not found "nginx_admin_password","newpassword"
There are a few things that can be done to use psec as a replacement for the security-setup script. These include:
Produce secrets descriptions in a security.d directory.
Remove the variables that are not secrets requiring regeneration for rotation or “break-glass” procedures (e.g., like chronos_principal, which is a userID value, and do_mesos_auth, which is a boolean flag).
Break down more complex data structures (specifically, the mesos_credentials list of dictionaries with keys principal and secret). These could instead be discrete variables like marathon_secret (which appears to be the secret associated with the invariant “variable” marathon_principal).
Credits
Tools used in rendering this package:
Development of this program was supported in part under an Open Source Development Grant from the Comcast Innovation Fund.
History
23.4.2 (2023-04-20)
Added
Added bats test for secrets find command.
Changed
Switched to using f-strings for formatting.
Fixed broken bats tests.
Update GitHub Actions workflows.
23.4.1 (2023-04-19)
Added
Added secrets find command.
Added support for new variable type boolean.
Changed
Updated GitHub Actions workflows (default to Python 3.9.16).
Drop Python 3.7, 3.8, add Python 3.11 (default to 3.10) for tox.
Fixed downstream dependency and pip installation problems.
Resolved new pep8 and bandit findings.
22.6.1 (2022-06-21)
Added
Added –ignore-missing option to continue when settings variables.
Added ‘Operational Security’ section to README.
22.6.0 (2022-06-10)
Added
Add about command to expose selected settings for situational awareness.
Add pytest code coverage reporting.
Add BATS runtime tests related to changes.
Changed
Fix caching bug with non-unique secret generation.
Fix bugs with setting/deleting secrets.
Improve secrets basedir initialization logic.
Expand use of pathlib.Path.
Improvements to source code, test, and vscode launch configuration quality.
22.5.1 (2022-05-25)
Changed
Switch to using factory pattern for secrets generation.
General code quality and test improvements.
Improve secrets get command logic and help.
Fix utils yaml-to-json subcommand and tests.
Resolve setuptools warnings.
Separate utility functions from utils subcommands.
Removed
Retire consul_key secret type in favor of token_base64.
Retire insecure secrets types (e.g., use of SHA1).
22.5.0 (2022-05-11)
Added
Test support for Python 3.10.
Add better logging controls.
Changed
Generalize Google OAuth2 email functionality.
Improve use and testing of exceptions.
22.1.0 (2022-01-22)
Added
Add init command and –init flag to initialize secrets base directory.
Ensure overridden values via flags are exported to process environment for subprocesses to use.
Add missing tests for features added in a previous release.
Add and start using application-specific exception classes.
Changed
Move functions and variables to utils to improve reuse ability.
Use get_ prefix more consistently for getter method/function names.
Over-ride cliff formatter class globally in app parser setup.
Use pathlib.Path for paths for cleaner code.
Fix bugs in environments delete command.
Fix bugs in –from-options feature of secrets get and secrets set.
Improvements to source code, test, and vscode launch configuration quality.
21.11.0 (2021-11-22)
Added
Add Help attribute to descriptions for URL to more information.
Changed
General code quality, documentation, and testing enhancements
Move tmpdir path creation to secrets_environment.SecretsEnvironment().
Move umask() function and variables to utils.
Removed
Drop Python 3.6 support due to it being EOL.
21.9.1 (2021-09-15)
Added
Added secrets tree subcommand.
Changed
Fixed bugs with environments path –tmpdir subcommand and run subcommand with –elapsed option when no environment exists.
Changed license file name.
Improved documentation.
21.9.0 (2021-09-07)
Added
Increased test coverage to address bugs (below) being fixed.
Changed
Fixed bugs in Makefile and tox.ini file.
Fixed bug setting undefined variables.
Switched from numpy to Python secrets module for random bytes.
Increased key size from 16 to 32 bits for consul_key, token_hex and token_urlsafe.
21.8.0 (2021-08-12)
Changed
Fixed bug in setup.py+setup.cfg
21.7.0 (2021-07-30)
Added
Secrets descriptions for demoing HypriotOS Flash mods Medium article
Changed
Improve secrets set –from-options
General code quality, documentation, and testing enhancements
21.6.0 (2021-06-23)
Added
Ability to set and generate secrets from defaults options
Ability to create an alias for an existing environment
Allow retroactive mirroring of new secrets
Changed
Switched from pbr to setuptools_scm for version numbering
Switched to more secure random number generation
21.2.0 (20201-02-23)
Added
Improve GitHub Actions workflows
Overall documentation and code enhancements
Improve handling of wildcards in options list
Changed
Fix bugs with handling empty lists, cloning environments, BATS tests
Increase password complexity a bit more
Fix ReadTheDocs
20.11.0 (2020-11-17)
Added
Add secrets create and secrets delete commands
Changed
Normalize all logger and exception output text
Refactoring code for better modulatiry
Normalize group create and group delete code
Normalize secrets show and secrets describe code
Fix bug that left variables missing after cloning
Add Python 3.9 to testing matrix
Switch from .yml to .json format for secrets
Expand IP address support in utils subcommand
20.8.1 (2020-08-11)
Changed
Fixes to v20.8.0
20.8.0 (2020-08-11)
Added
Add GitHub workflow to publish to test.pypi.org
Add secrets backup and secrets restore logic
Open web browser to documentation for help
Changed
Go back to date-based version numbering
General CI/CD workflow updates
Improve directory handling in environments path
20.2.15 (2012-02-15)
Added
Added Python 3.8 support to test matrix
Changed
Fix bug in environments default
Put elapsed time (and BELL) on stdout
Fix bug in environments tree
Allow setting vars using diff names+environment
19.12.0 (2019-12-16)
Added
Add and document new boolean data type
Add groups delete command
Changed
Improve default environment handling
Improve tox+BATS testing
Address security issue per “Your xkcd passwords are pwned” article
General code quality and test improvements
Add protection from over-writing existing env vars
Add Options attribute
19.11.1 (2019-11-29)
Changed
Enhancements to better support Windows 10
Allow cloning group descriptions from environment
Fix tty/no-tty handling with environments delete
Expose terraform command on -v
Validate variable exists in environment
Fix broken environments tree code
19.10.1 (2019-10-20)
Changed
Move BATS unit tests into tox testing
Avoid attempting interactive things when no tty
Improve file and directory permissions logic
19.10.0 (2019-10-14)
Added
Working SSH key and configuration management
Use bullet for interactive list selection
Elapsed timer feature
Parsing of terraform output to extract SSH public keys
umask control for better new file permission settings
Support configuring terraform tfstate backend
Allow setting secrets by copying from another environment
Changed
Numerous bug fixes
Refine testing
Option to only show undefined variables
Sort environments when listing
19.9.0 (2019-09-05)
Added
Add environments delete subcommand
Allow cloning environment from an existing one
Changed
19.8.3 (2019-08-28)
Changed
Dynamically get version number
General testing enhancements
General code quality enhancements
Ensure more secure file permissions
19.8.2 (2019-08-23)
Changed
General code quality enhancements
19.8.0 (2019-08-22)
Added
IP address determination
Allow cloning new group in an empty environment
Make python -m psec work
JSON output method
Environment aliasing feature
Changed
General code quality and testing enhancements
Be more explicit about default environment
Tighten permissions on cloned environments/groups
Add insecure permissions checking
19.5.1 (2019-05-08)
Changed
Add HISTORY.rst file
19.4.5 (2019-05-08)
Added
Add command ssh config to manage SSH configuration snippet for use by update-dotdee to generate ~/.ssh/config file
Add command ssh known-hosts add and ssh known-hosts remove to manage system known_hosts file(s)
Changed
Generalized exception to fix –version bug
Clean up temporary docs/psec_help.txt file
19.4.4 (2019-04-21)
Changed
Fix Bats dependencies/tests
Fix broken documentation (wt?)
Fix messed up release tagging
19.4.0 (2019-04-19)
Added
Python 3.7 coverage for Travis CI
Changed
Complete –help output (epilog text) in all commands
Install a script ‘psec’ to complement console_script entry point
Clarify arguments in –help output
Deprecated
The ‘python_secrets’ command is now just ‘psec’
19.3.1 (2019-04-06)
Added
Add environments rename command
Add utils set-aws-credentials command to mirror AWS CLI credentials
Use autoprogram_cliff for self-documentation
Add cliff.sphinxext for documentation
Changed
Refactored SecretsEnvironment() so autoprogram_cliff works
18.11.0 (2018-11-09)
Added
Add “–type” option to “secrets describe”
Improve visibility into default environment
Add screencasts to documenation
Add RST checks to ensure PyPi documentation works
Add feedback about minimum Python version
Add --json output to environments path
Add reference to proof-of-concept using goSecure fork
Changed
The “secrets describe” command now describes variables and types
Allow secrets set to set any type (not just string)
18.9.0 (2018-09-27)
Added
Switched to calendar version numbering
Finish GPG encrypted email delivery of secrets
groups create command
Improve error handling consistency when no environment exists
0.16.0 (2018-09-12)
Added
Use attribute maps instead of lookup loops
Add Prompt attribute in descriptions for better UX when setting variables
Note new undefined variables when adding groups or environments create --clone-from
When exporting vars, also export PYTHON_SECRETS_ENVIRONMENT w/environment name
Add reference to Python Security coding information
environments tree command
environments path command with features supporting Ansible Lookup Plugin
secrets get command
groups path command
environments default command
0.14.0 (2018-08-30)
Added
Option to export secrets as environment variables (with optional prefix)
Can now set secrets (any specified or all undefined) via command line
utils myip command returns routable IP address (with CIDR option)
run command allows running commands with exported environment variables
Changed
Renamed template comamnd to utils tfoutput
Removed
Dropped support for Python 3.4, 3.5, since secrets module only in Python >= 3.6
0.10.0 (2018-08-23)
Added
New string type for manually set secrets
secrets path command provides path to secrets .yml file
template command (Jinja templating)
Default environment to basename of cwd
Clone environment from skeleton directory in repo
0.9.1 (2018-08-19)
Added
secrets describe command
environments create command
environments list command
Expand secrets types and generation methods
Add initial feature for sending secrets via email using Google OAuth2 SMTP
Removed
Drop Python 2.7 support (at least for now…)
Security
Add six for securing input call
0.8.0 (2018-05-11)
(TBD)
0.4.0 (2018-05-01)
(TBD)
0.3.6 (2018-04-29)
(TBD)
0.3.0 (2018-04-27)
First release on PyPI.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for python_secrets-23.4.2-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 211d932dc16f89dfc51d90913373603f6b414523da865afe8bce8908b1287400 |
|
MD5 | 07584802bd77184c6babc96608157053 |
|
BLAKE2b-256 | ec1dfb9f314d75407aba641f010149827a879e314fdf4e02f3ed941f8c14f87d |