yourbase 5.1.5

Skip tests based on tracing data

YourBase Python Acceleration

Tests are important. For large monoliths, they're also a major source of drag on velocity.

YourBase is a tool that traces your tests to determine which functions each test depends on. It can later use this information to determine which tests don't need to run because their code paths have not changed. These tests are skipped automatically.

No configuration, setup, or babysitting required. All you need is

pip install yourbase

YourBase works with Python 2.7+ and Python 3.5+; using pytest or unittest.

Access

YourBase will work for free locally for 14 days. Please consider purchasing a license if you like it -- those who do generally get more back than they spend.

A license also lets YourBase work in your CI, and synchronize its tracing data among an arbitrary number of machines (without leaving your network).

First run

After installing yourbase, if you are using unittest you must put

import unittest
import yourbase

yourbase.attach(unittest)

in a file that runs before your tests (e.g. tests/__init__.py). Do not do this step if you are using pytest.

Run your tests with the same command you typically use. You should see some output from YourBase similar to

[YB] Starting Python acceleration

The first run will be cold, so if you just want to see YourBase in action and your tests are going to take a while, you can run a subset of tests. Tracing data for the subset will be used correctly even if you later run all tests.

After the run finishes, running again will skip all tests. Modifying a dependency will run only tests whose code paths touched the changed code. You're YourBased! 🚀

Installing for yourself vs. your team

If you add YourBase to your requirements file, it will work with no configuration for your whole team. However if you want to just try it out yourself, you can choose not to add it. Either way, YourBase will behave the same.

Forcing specific tests to run

You can tell YourBase to never skip specific tests with decorators:

# pytest
import pytest

@pytest.mark.do_not_accelerate
def test_function():
   # ...

# unittest
import yourbase.plugins.unittest as yourbase

@yourbase.do_not_accelerate
class TestClass(unittest.TestCase):
   def test_function():
      # ...

The test or group will never be skipped due to unchanged dependencies. If you are using test cohorting (below), it may still be skipped if it is not assigned to the running shard.

We do not yet support this feature for unittest.

Synchronizing tracing data

By default, YourBase's tracing data will not leave the machine it was gathered on without further configuration. This means CI or containerized test runs will not be able to share tracing data forward, kneecapping YourBase.

To solve this, YourBase can be made to share tracing data among any number of machines using an S3 bucket as a collaboration point. To do so, all machines should set

YOURBASE_REMOTE_CACHE=s3://<bucketname>[/key/prefix]

where <bucketname> is an S3 bucket that each machine has Get/Put/List access to. Tracing data generated for passing builds from clean working trees will be synchronized to this location, then used for future runs if a local cache is not present.

This tracing data includes:

• test names
• file names
• class and function names
• relationships between of the above
• a small amount of metadata like commit hash, build time, and Python version

Your code base will never be uploaded to the bucket. Neither your code nor your tracing data will touch YourBase servers.

It is safe to share one bucket between multiple repositories, even if across multiple languages.

AWS credentials

We'll use the system AWS credentials by default. This can be configured via environment variables or AWS CLI configuration files.

# To use system AWS credentials via environment variables, ensure your
# credentials are exported in your environment.
export AWS_ACCESS_KEY_ID=changeme
export AWS_SECRET_ACCESS_KEY=changeme

If you're setting the system AWS credentials to bogus values for your tests or CI, you can set these YourBase-specific environment variables instead:

# To set YourBase specific AWS credentials, export the following:
export YOURBASE_AWS_ACCESS_KEY_ID=changeme
export YOURBASE_AWS_SECRET_ACCESS_KEY=changeme

If these are set, we'll use them instead of the system credentials.

Test parallelization

YourBase has first-party support for acceleration-friendly parallelization:

1. Set YOURBASE_COHORT_COUNT to the number of shards
2. Set YOURBASE_ACTIVE_COHORT to the shard ID (in the range [1, YOURBASE_COHORT_COUNT])

YourBase uses consistent hashing to split your tests across shards so they do not get "reshuffled" when other tests are added or removed. This will give parallelized tests the full benefits of acceleration.

CircleCI users do not have to set the above environment variables as YourBase will inherit them automatically. However you must remove any CircleCI test splitting or globbing, as YourBase will automatically choose which tests to run "just in time".

Compatibility

Poetry

YourBase is compatible with the packaging and dependency management tool Poetry. Once the YourBase package is installed, simply use Poetry as normally.

For example:

# Install the YourBase package
poetry add yourbase

# Execute tests
poetry run pytest

Known issues

Incorrectly skipped tests

If you are using unittest and define your own setUp/tearDown functions, be sure they call super before performing other actions:

class MyTestClass:
   def setUp(self):
      super(self.__class__, self).setUp()
      # ...

   def tearDown(self):
      super(self.__class__, self).tearDown()
      # ...

If you are not defining your own setUp and tearDown functions, you do not need to do this.

Errors

If you run into errors about the _sqlite3 module not being found, first run

# Debian-based
sudo apt-get install libsqlite3-dev
# macOS
brew install sqlite3

then rebuild and reinstall the Python version you are using. If you use pyenv, this will look something like

pyenv install --force <PYTHON_VERSION>

# If that doesn't work, try
PYTHON_CONFIGURE_OPTS="--enable-loadable-sqlite-extensions" pyenv install --force <PYTHON_VERSION>

Conflicts

Coverage.py's dynamic_context option

YourBase has a tracing conflict with one particular config setting in your .coveragerc:

[run]
dynamic_context = test_function

If this specific key is set to this specific value, YourBase will "miss" some functions when tracing tests, causing its subsequent runs to skip tests that should not be skipped.

You can work around this by either unsetting the option, or by enabling YourBase's experimental line-granularity tracing, which does not conflict with this setting:

export YOURBASE_TRACING_TYPE=line

Although this circumvents the issue, this tracing type is in early stages and is less stable than function-granularity tracing (the default).

If you do not have this specific setting set in your .coveragerc or in Coverage.py's command line arguments, you do not have to worry about this issue.

Proxy objects

Python objects that opaquely wrap other objects by overriding Python builtins like __name__ and __class__ can cause tracing issues in YourBase that may manifest as errors from within those proxy objects. If you experience these issues, you can set

export YOURBASE_TIMID=true

to use a slower tracing algorithm that will avoid these errors. Tracing overhead is dramatically increased using this flag, so we don't recommend setting this if you are not experiencing issues.

Contributing

This open source package is a lightweight wrapper for YourBase proprietary code. We welcome contributions to this wrapper, but at this time we have not built shims or mocks to allow it to be tested end to end without both pieces.

Code style

We use Black for code formatting, which is similar in personality to gofmt -- ruthless consistency, no configuration. Your build will not pass CI if the Black run doesn't come back clean, so we recommend you have your editor automatically run it on save. You can run it manually with

black .

Changelog

• 5.1.4 (2021-05-18)
  • Fixes a bug with unittest support that could cause blank tracing data
  • Fixes a bug with unittest support that could cause tracing data to be written even for failed tests
• 5.1.3 (2021-05-18)
  • Fixes an error condition if pytest isn't installed
• 5.1.2 (2021-05-17)
  • Fixes CircleCI sharded environments from not being inherited by YourBase correctly
• 5.1.1 (2021-05-17)
  • Adds first-party support for test parallelization
• 5.0.10 (2021-05-13)
  • Fixes a compatibility issue with Celery
  • Fixes a compatibility issue with "proxy objects" causing tests to error
  • Improves error messages related to Git availability
• 5.0.9 (2021-05-11)
  • Fixes an issue processing certain styles of PEP 263 tags (e.g. # -*- coding: utf-8 -*-)
  • Adds some friendlier error messages and/or fallbacks caused by Git issues
• 5.0.7 (2021-05-06)
  • Fixes a JSON decoding issue present in Python 3.5
  • Fixes a breaking bug when YourBase is enabled alongside tests that use moto mocks