pynonymizer 1.4.0

An anonymization tool for production databases

pynonymizer

pynonymizer is a universal tool for translating sensitive production database dumps into anonymized copies.

This can help you support GDPR/Data Protection in your organization without compromizing on quality testing data.

Why are anonymized databases important?

The primary source of information on how your database is used is in your production database. In most situations, the production dataset is usually significantly larger than any development copy, and would contain a wider range of data.

From time to time, it is prudent to run a new feature or stage a test against this dataset, rather than one that is artificially created by developers or by testing frameworks. Anonymized databases allow us to use the structures present in production, while stripping them of any personally identifiable data that would consitute a breach of privacy for end-users and subsequently a breach of GDPR.

With Anonymized databases, copies can be processed regularly, and distributed easily, leaving your developers and testers with a rich source of information on the volume and general makeup of the system in production. It can be used to run better staging environments, integration tests, and even simulate database migrations.

below is an excerpt from an anonymized database:

id	salutation	firstname	surname	email	dob
1	Dr.	Bernard	Gough	[e-mail]	2000-07-03
2	Mr.	Molly	Bennett	[e-mail]	2014-05-19
3	Mrs.	Chelsea	Reid	[e-mail]	1974-09-08
4	Dr.	Grace	Armstrong	[e-mail]	1963-12-15
5	Dr.	Stanley	James	[e-mail]	1976-09-16
6	Dr.	Mark	Walsh	[e-mail]	2004-08-28
7	Mrs.	Josephine	Chambers	[e-mail]	1916-04-04
8	Dr.	Stephen	Thomas	[e-mail]	1995-04-17
9	Ms.	Damian	Thompson	[e-mail]	2016-10-02
10	Miss	Geraldine	Harris	[e-mail]	1910-09-28
11	Ms.	Gemma	Jones	[e-mail]	1990-06-03
12	Dr.	Glenn	Carr	[e-mail]	1998-04-19

How does it work?

pynonymizer replaces personally identifiable data in your database with realistic pseudorandom data, from the Faker library or from other functions. There are a wide variety of data types available which should suit the column in question, for example:

• unique_email
• company
• file_path
• [...]

For a full list of data generation strategies, see the docs on strategyfiles

Process outline

1. Restore from dumpfile to temporary database.
2. Anonymize temporary database with strategy.
3. Dump resulting data to file.
4. Drop temporary database.

If this workflow doesnt work for you, see process control to see if it can be adjusted to suit your needs.

Supported Databases

• mysql
• More coming soon!

Requirements

• Python >= 3.6

mysql

• mysql/mysqldump: You will need these utilities in your path.
• An active database connection, (mysql >= 5.5) either local or remote (to restore, anonymize, and dump from)
• A backup in Single-file mysqldump output (schema and data)

Getting Started

Usage

1. Write a strategyfile for your database
2. See below:

usage: pynonymizer [-h] [--input INPUT] [--strategy STRATEGYFILE]
                   [--output OUTPUT] [--db-type DB_TYPE] [--db-host DB_HOST]
                   [--db-name DB_NAME] [--db-user DB_USER]
                   [--db-password DB_PASSWORD] [--fake-locale FAKE_LOCALE]
                   [--start-at STEP] [--skip-steps STEP [STEP ...]]
                   [--stop-at STEP] [-v]

A tool for writing better anonymization strategies for your production
databases.

optional arguments:
  -h, --help            show this help message and exit
  --input INPUT, -i INPUT
                        The source dumpfile to read from. [file.sql,
                        file.sql.gz] [$PYNONYMIZER_INPUT]
  --strategy STRATEGYFILE, -s STRATEGYFILE
                        A strategyfile to use during anonymization.
                        [$PYNONYMIZER_STRATEGY]
  --output OUTPUT, -o OUTPUT
                        The destination to write the dumped output to.
                        [file.sql, file.sql.gz] [$PYNONYMIZER_OUTPUT]
  --db-type DB_TYPE, -t DB_TYPE
                        Type of database to interact with. More databases will
                        be supported in future versions. default: mysql
                        [$PYNONYMIZER_DB_TYPE]
  --db-host DB_HOST, -d DB_HOST
                        Database hostname or IP address.
                        [$PYNONYMIZER_DB_HOST]
  --db-name DB_NAME, -n DB_NAME
                        Name of database to restore and anonymize in. If not
                        provided, a unique name will be generated from the
                        strategy name. This will be dropped at the end of the
                        run. [$PYNONYMIZER_DB_NAME]
  --db-user DB_USER, -u DB_USER
                        Database credentials: username. [$PYNONYMIZER_DB_USER]
  --db-password DB_PASSWORD, -p DB_PASSWORD
                        Database credentials: password. Recommended: use
                        environment variables to avoid exposing secrets in
                        production environments. [$PYNONYMIZER_DB_PASSWORD]
  --fake-locale FAKE_LOCALE, -l FAKE_LOCALE
                        Locale setting to initialize fake data generation.
                        Affects Names, addresses, formats, etc.
                        [$PYNONYMIZER_FAKE_LOCALE]
  --start-at STEP       Choose a step to begin the process (inclusive).
                        [$PYNONYMIZER_START_AT]
  --skip-steps STEP [STEP ...]
                        Choose one or more steps to skip.
                        [$PYNONYMIZER_SKIP_STEPS]
  --stop-at STEP        Choose a step to stop at (inclusive).
                        [$PYNONYMIZER_STOP_AT]
  -v, --version         show program's version number and exit

License

MIT