yourbase 5.0.6

Skip tests based on tracing data

YourBase Python Acceleration

Tests are important. For large monoliths, they're also a major source of drag on velocity.

YourBase is a tool that traces your tests to determine which functions each test depends on. It can later use this information to determine which tests don't need to run because their code paths have not changed. These tests are skipped automatically.

No configuration, setup, or babysitting required. All you need is

pip install yourbase

YourBase works with Python 2.7+ and Python 3.5+; using pytest or unittest.

Access

YourBase will work for free locally for 14 days. Please consider purchasing a license if you like it -- those who do generally get more back than they spend.

A license also lets YourBase work in your CI, and synchronize its tracing data among an arbitrary number of machines (without leaving your network).

First run

After installing yourbase, if you are using unittest you must put

import yourbase.plugins.unittest as yb; yb.attach()

in a file that runs before your tests. If you are using pytest this step is not necessary.

Run your tests with the same command you typically use. You should see some output from YourBase similar to

[YB] Starting Python acceleration

The first run will be cold, so if you just want to see YourBase in action and your tests are going to take a while, you can run a subset of tests. Tracing data for the subset will be used correctly even if you later run all tests.

After the run finishes, running again will skip all tests. Modifying a dependency will run only tests whose code paths touched the changed code. You're YourBased! 🚀

Installing for yourself vs. your team

If you add YourBase to your requirements file, it will work with no configuration for your whole team. However if you want to just try it out yourself, you can choose not to add it. Either way, YourBase will behave the same.

Synchronizing tracing data

By default, YourBase's tracing data will not leave the machine it was gathered on without further configuration. This means CI or containerized test runs will not be able to share tracing data forward, kneecapping YourBase.

To solve this, YourBase can be made to share tracing data among any number of machines using an S3 bucket as a collaboration point. To do so, all machines should set

YOURBASE_BUCKET=s3://BUCKETNAME

where BUCKETNAME is an S3 bucket that each machine has Get/Put/List access to. Any successful tracing data generated for a clean working tree will be synchronized to this location after tests have finished.

The tracing data includes test names, file names, function names, and a small amount of metadata like commit hash and build time. Your code will never be uploaded to the bucket. Neither your code nor any of your tracing data will touch YourBase servers.

It is safe to share one bucket between multiple repositories, even if across multiple YourBase-supported languages.

You can optionally specify a key prefix using the path component:

# A trailing slash will be added if absent
YOURBASE_BUCKET=s3://BUCKETNAME/path/to/anywhere

AWS credentials

We'll use the system AWS credentials by default. If you're setting those to bogus values for the purposes of your tests, you can set these environment variables instead:

YOURBASE_AWS_ACCESS_KEY_ID=changeme
YOURBASE_AWS_SECRET_ACCESS_KEY=changeme

If these are set, we'll use them instead of the system credentials.

Cohorting / sharding

YourBase supports sharding your tests without negatively affecting tracing or acceleration. It uses consistent hashing to split tests into cohorts that stay the same between runs, even as the test pool grows or shrinks.

1. Set YOURBASE_COHORT_COUNT to the number of cohorts your tests should be split into. This should be the same among all shards.
2. Set YOURBASE_TEST_COHORT to the cohort ID this run should identify as, starting with 1. This should be different among all shards.

Without these set, YourBase assumes a value of "1" for both, meaning one shard will run one cohort; that cohort will contain all tests.

Note that tests are only guaranteed to remain in the same cohort as long as YOURBASE_COHORT_COUNT doesn't change.

Known issues

Incorrectly skipped tests

If you are using unittest and define your own setUp/tearDown functions, be sure they call super before performing other actions:

class MyTestClass:
   def setUp(self):
      super(self.__class__, self).setUp()
      # ...

   def tearDown(self):
      super(self.__class__, self).tearDown()
      # ...

If you are not defining your own setUp and tearDown functions, you do not need to do this.

Errors

If you run into errors about the _sqlite3 module not being found, first run

# Debian-based
sudo apt-get install libsqlite3-dev
# macOS
brew install sqlite3

then rebuild and reinstall the Python version you are using. If you use pyenv, this will look something like

pyenv install --force <PYTHON_VERSION>

# If that doesn't work, try
PYTHON_CONFIGURE_OPTS="--enable-loadable-sqlite-extensions" pyenv install --force <PYTHON_VERSION>

Conflicts

moto

YourBase has a logical conflict with the moto package. If you use YourBase with S3 sync (see "Synchronizing tracing data" above) and use moto mocks in your tests, moto will globally mock the boto3 library YourBase uses to communicate with S3. This will cause errors with stack traces pointing either to your tests or to YourBase internals, depending on initialization order. See spulec/moto#3140 for details. As a workaround you can manually start and stop moto mocks as needed, turn off YourBase's S3 sync feature by unsetting YOURBASE_BUCKET, or remove uses of moto in tests.

Coverage.py's dynamic_context option

YourBase has a tracing conflict with one particular config setting in your .coveragerc:

[run]
dynamic_context = test_function

If this specific key is set to this specific value, YourBase will "miss" some functions when tracing tests, causing its subsequent runs to skip tests that should not be skipped.

You can work around this by either unsetting the option, or by enabling YourBase's experimental line-granularity tracing, which does not conflict with this setting:

export YOURBASE_TRACING_TYPE=line

Although this circumvents the issue, this tracing type is in early stages and is less stable than function-granularity tracing (the default).

If you do not have this specific setting set in your .coveragerc or in Coverage.py's command line arguments, you do not have to worry about this issue.

Contributing

This open source package is a lightweight wrapper for YourBase proprietary code. We welcome contributions to this wrapper, but at this time we have not built shims or mocks to allow it to be tested end to end without both pieces.

Code style

We use Black for code formatting, which is similar in personality to gofmt -- ruthless consistency, no configuration. Your build will not pass CI if the Black run doesn't come back clean, so we recommend you have your editor automatically run it on save. You can run it manually with

black .