femtotemplate 0.7.19

Femtosense template project

femtotemplate

A template for creating a new python-based project repo

Also a good walkthrough to using the Femtosense development environment

Walkthrough

This walkthrough covers the development lifecycle for someone working on the project. We'll install our package locally, make a code change, run tests locally, then push the project to Github. This will trigger a number of things which happen automatically: the tests will be rerun in a fixed environment, and the built package will be uploaded to our private package index.

Concepts

At Femtosense, Python-based projects each have their own source repository (repo) tracked by Github. Projects are distributed amongst the team as packages which are stored in a package index. We maintain a private package index server to share packages amongst the team. This makes installing other team members' work as easy as running pip install. Using packages also enforces in-source inter-project dependency encoding in the setup.py files. Packages must be versioned according to Semantic Versioning. Packages are indexed as project_name-version. Packages are either indexed as volatile or nonvolatile (mutable/immutable might have been better names). Nonvolatile packages cannot be modified once they are indexed. This makes sense for packages based on master branch--those are supposed to be clean and stable. Volatile packages can be modified over and over, which makes sense for development branches. How packages are stored is handled automatically based on the branch name. Master is the only branch that is nonvolatile. Keep this in mind when making dependencies on development versions of packages.

A Continuous Integration/Continuous Deployment server (CI/CD server) is used to automatically run tests when code changes are made, and publish new package versions to the index automatically. Automatic testing ensures that problems can't fall through the cracks because users didn't always run all the tests. It also ensures that the build environment is consistent and repeatable. The CI/CD server watches for notifications from Github that code changes have been made, then pulls the most recent version of the code and runs its tests. Test results propagate back to Github, and run progress is viewable in a dashboard. Once the build is complete, the package is created and published to the package index server. Automatic publishing of packages means that code developers don't have to worry about manually uploading packages to the server.

Docker is used to provide the environment used by the CI/CD server. Docker Containers are conceptually like virtual machine instances, and are used to run the tests. We use a centralized repo with a Dockerfile and set of scripts to set up a consistent environment for each project's CI/CD run.

Cython is used to obfuscate the .py files that we don't want to be easy to read. This is configured by the obfuscation_cfg.yaml file. Cython converts the selected Python files to C, then compiles them to binary libraries, which are installed as part of the Python wheel package.

In typical use, a developer might have to edit a few build-related things in a repo, other than their own code:

1. The setup.py file: to describe dependencies on other projects and external python packages
2. The femtotemplate/VERSION file : to declare a new code version (this is the only place you should ever enter a version number)
3. The femtotemplate/ENV_REQUIREMENTS.sh file : to describe non-python environmental dependencies (e.g. a system package like librosa)
4. The obfuscation_conf.yaml file : to specify which .py files to exclude from obfuscation.

In typical use, the only "new" commands the user might invoke are:

1. pip install -e . : to install the package they're working on and its dependencies
2. Docker commands (only under some circumstances) : in the event that the developer wishes to reproduce the testing environment locally, as a sanity check

Important Files

File Name	Purpose	Edit When?
NAME	project/repo name	once, if starting a new project based on femtotemplate
LICENSE	legal boilerplate	never
MANIFEST.in	include for setup.py	once, if starting a new project based on femtotemplate (have to change femtotemplate -> new package name)
setup.py	instructions for pip to install	never, probably, unless you have unusual installation requirements
------------------------------------	---------------	------------
femtotemplate/	where the actual code goes	OFTEN, this is where your code goes
femtotemplate/VERSION	project version number	OFTEN, as you make code changes
femtotemplate/PY_REQUIREMENTS	python packages that this project depends on	rarely, to add new python dependencies
femtotemplate/ENV_REQUIREMENTS.sh	non-python environment setup commands	rarely to add new non-python dependencies
----------------	---------	------------
buildspec.yml	instructions for AWS codebuild (CI server)	never, probably
obfuscation_cfg.yaml	configuration for which parts and how the package gets obfuscated	used in core compiler packages, not in open source packages
obfuscation.py	python script to obfuscate (and deobfuscate) the package	never, probably

Getting Started

Development for python-based packages should be done in Ubuntu. Use the most recent Ubuntu AMI if working on AWS.

You need to have git and python tools installed. This walkthrough assumes a conda install for python/pip/pytest (see conda via Anaconda).

Clone this repo. cd into it. Check out the dev branch:

git clone https://github.com/femtosense/femtotemplate.git
cd femtotemplate
git checkout -b dev origin/dev

To download internal dependencies, you need to set up your local pip to find Femtosense's private package server. Copy the contents from the buildscripts to_append.conf to you ~/.pip/pip.conf.

If you are not developing on a machine inside AWS, you will have to VPN into our private virtual public cloud before installing internal dependencies.

Installing the Package

Is easy! In the directory with setup.py, run

./femtotemplate/ENV_REQUIREMENTS.sh

You'll see the script install some non-Python dependencies (you may have to enter your password for sudo priviledges). Then run

pip install -e .

You'll see it download a couple packages. We've now successfully installed the package on our system's python. If you launch python, you can now do:

>>> import femtotemplate
>>> femtotemplate.__version__
'x.y.z'    <--- the values for x, y, and z change with the template version

You'll see that __version__ has the same value as what's in the femtotemplate/VERSION file. The code blob in femtotemplate/__init__.py creates this attribute.

The femtotemplate/VERSION file is the single source of version-truth! You should never have to enter a version number anywhere else.

It's useful to understand what is happening when pip install is invoked. pip looks at the setup.py file, and, importantly, it uses the install_requires keyword to figure out which other packages are needed. We've configured this to look at femtotemplate/PY_REQUIREMENTS. If you bring in a new Python package dependency, add it as a new line to this file. This is how we encode interproject (repo-to-repo) dependencies.

Packages internal to Femtosense are stored on a PyPI-like server running inside of AWS. In order to download from this server, we told pip about the server's IP when we modified ~/.pip/pip.conf.

When a project relies on a dependency that is NOT a python package (e.g. cmake, librosa, etc.), we must put the install instructions for these dependencies in femtotemplate/ENV_REQUIREMENTS.sh. This should just contain commands like sudo apt install some_system_package. Ultimately, it is up to the maintainer of each package to keep this script up-to-date, so that all environmental dependencies (including the environmental dependencies of other python packages that this package depends on) are installed by this script. It is important to document your packages external dependencies in the README. There are some tools to help determine parent packages' system dependencies (see below).

Packages are published to this server automatically by the CI server, which runs when we make commits. Which we'll do next...

Making a Change and Observing the CI Build

We're currently on the dev branch. It's good practice to not work directly on either of the mainline branches that all repos have (master and dev).

For the tutorial, we can stay in dev.

Now we'll make a change and push the result. This will trigger the CI build to start running on another server in AWS. In this case, the build runs the tests in the 'canonical' environment, and publishes the built package to the package index.

Making a Code Change

The only code in this repo is a single python file. There's a list of names, you can add yours to that. The test parses the list of names and makes sure they only have letters. (I hope your name doesn't have any numbers in it).

Running the Tests Locally

You can run the test locally before you push. This is typically a good idea. On more complicated projects, you might run some subset of the complete test battery (maybe just the stuff you thought should be affected, because running ALL the tests takes a long time). The CI server will run everything to make sure the commit is clean.

cd femtotemplate/test
pytest -v

Will produce something like:

======================= test session starts =======================
platform linux -- Python 3.6.9, pytest-3.3.2, py-1.9.0, pluggy-0.13.1 -- /usr/bin/python3
cachedir: ../../.cache
rootdir: /home/ubuntu/femtotemplate, inifile:
plugins: devpi-server-5.5.0
collected 1 item                                                                                                                                                                                                  

test_names.py::test_no_numbers_in_names PASSED               [100%]

==================== 1 passed in 0.11 seconds =====================

You can try putting a "bad" name in to see what happens. Pytest will only show printouts for tests that failed. Fix it before moving on.

Actually, when you push, the tests will run inside a Docker container on the build server. See the "All About Docker" section below to learn how to run the tests this way locally. In most cases, this shouldn't be necessary, but could serve as a sanity check that your environment hasn't diverged meaningfully from the testing environment.. There's not too much more to learn to use Docker.

Pushing the Change to Github and Triggering the CI Build

Before we commit and push our change, we need to update femtotemplate/VERSION. Increment either the last or second-to-last field (this isn't a real project, don't worry about what incrementing one or the other means).

Now we can push our change to Github. This will trigger a CI build (assuming the repo's CI is set up--which it is for femtotemplate, see the "Setting up CI" section below if it isn't).

git commit -a -m "your message here"
git push origin dev

Now go to the Github page for the repo. The last commit made will show near the top of the page, with your commit message. After a minute or two, you should see a little green check mark next to the commit. (hopefully not a little red x because they failed; an orange circle means they're still running) If you click this check, it will take you the AWS codebuild page, assuming you're logged in. Here you can see the build log.

After running the tests, the CI server will publish the built package to private package index. This makes it accessible to other team members. Your new version will be available on other machines via pip install femtotemplate==VERSION (assuming the other machine's pip also knows where our private package index lives).

Obfuscation

I. Locally test obfuscation

There are two ways to test obfuscation on your local machine: 1. Run codebuild locally (tutorial here). 2. Run the obfuscation commands used in buildscripts.

The exact commands used in option 2 could change over time, so please refer to the current buildscripts for the latest usage. However, the following commands are likely to work:

python buildscripts/obfuscation/obfuscate.py -x . # Convert selected .py files to .pyx
pip install . # Build and install the package, with .pyx files compiled to binaries

The installed package can then be tested as usual.

Note on testing: It is strongly recommended that you separate tests from the package's source code. That way, your tests will import the installed version of the package, which is obfuscated. See here for more info.

When you are done testing, you can revert your .pyx files back to .py with the following command:

python obfuscate.py -rd .

If you wish to inspect the generated .c files, you can use the command:

python obfuscate.py -xb .

II. Excluding a python file from obfuscation

obfuscation_cfg.yaml is meant to be very easy to edit to exclude certain files from obfuscation. Add to py_exclusions to exclude files.

To exclude the file femtotemplate/names.py, we should change py_exclusions to:

py_exclusions:
  - 'setup.py'
  - 'obfuscate.py'
  - 'test/*.py'
  - 'femtotemplate/names.py'

These patterns are passed to Python's glob.glob to expand special characters like * and **. For example, to exclude all python files in the femtotemplate subdirectory, use

py_exclusions:
  - 'femtotemplate/*.py'

If you edit this list, be careful not to remove setup.py, obfuscate.py, or test/*.py.

III. Deploying an obfuscated package

Simply set the Docker build argument FS_OBFUSCATE=1 in Codebuild's buildspec.yml file. This ensures that the Python package is obfuscated before it is tested and uploaded to devpi.

Recap/Command Reference

# clone repo, switch to dev
git clone https://github.com/femtosense/femtotemplate.git
cd femtotemplate
git checkout -b dev origin/dev

# tell pip where to find the femtosense private package index
# you need to VPN into AWS if your machine isn't in AWS

# set up the non-Python environment
./femtotemplate/ENV_REQUIREMENTS.sh
# install the package locally
pip install -e .

# run tests
pytest -v test

# (edit code, update the VERSION)

# commit and push
git commit -a -m "your message here"
git push origin dev

# build runs automatically, you can see results on Github
# new package version is published automatically, 
# retrievable simply by pip install

Determining Parent Package Environment Dependencies

There is a little bit of scripting to help with the process of writing your ENV_REQUIREMENTS.sh file. Specifically, packages configured like femtotemplate will have a __env_requirements__ attribute which has the contents of their ENV_REQUIREMENTS.sh. Packages configured like femtotemplate also have a __parent_env_requirements__ attribute that concatenates the __env_requirements__ of all first-degree parent packages.

You can use this to generate a starting point for your own setup_env.sh as follows:

import femtotemplate
f = open('ENV_REQUIREMENTS.sh', 'w') # any dummy file name would work
f.writelines([line + '\n' for line in femtotemplate.__parent_env_requirements__]) # writelines doesn't add newlines, so we do it before writing
f.close()

Should produce a file with:

# requirements from dummy_femtopackage
sudo apt-get install vim

Which is the contents of femtotemplate's one dependency's (dummy_femtopackage) ENV_REQUIREMENTS.sh.

Starting Your Own Project Based on femtotemplate

It is a good idea to base your project on this femtotemplate repository. Do the following:

• Copy this repo and rename it and the femtotemplate/femtotemplate directory to your new project name (Note: don't clone this repo because you will pull in all of its history into your project's history.)
• Change the contents of NAME to your project's name
• Change the contents of this README.md for your project
• Change the contents of <your project>/VERSION to 0.1.0
• Delete <your project>/names.py and <your project>/test/test_names.py
• Change the contents of <your project>/PY_REQUIREMENTS for your project's Python package dependencies
• Change the contents of <your project>/ENV_REQUIREMENTS.sh for your project's non-Python dependencies
• Set up the CI build

All About buildspecs and Docker

As long as you're working exclusively with python, and your build consists of only running pytests in reponame/test, you won't need to modify the buildspec.

The buildspec.yml is AWS Codebuild's (CI service provider) instructions. This is effectively equivalent to Jenkins' jenkinsfile. Notice that codebuild's instructions are just to do a docker build.

Look at the commands it uses: docker build . will create a docker image using the Dockerfile in the directory that the command is invoked from. Our Dockerfile is sourced from the buildscripts central repo. The -t flag adds a tag. In this case, we call the image NAME:VERSION.

docker run NAME:VERSION will run the CMD specified in the Dockerfile. In our case, it runs pytest.

If you install docker, you can run the same commands that the build server does to replicate its exact environment locally.

# install docker
sudo apt-get update
sudo apt-get install -y docker.io

# start the docker service
sudo service docker start

# build the docker image
export NAME=`cat NAME`
export VERSION=`cat ${NAME}/VERSION`

sudo docker build -t ${NAME}:${VERSION} --build-arg NAME=${NAME} --build-arg VERSION=${VERSION} .
sudo docker tag ${NAME}:${VERSION} ${NAME}:latest

# now you can look at your available images
sudo docker images

# now run the tests in docker, this will run the CMD in the Dockerfile
sudo docker run ${NAME}:latest

To show running containers (you can have more than one)

sudo docker container ls

To kill one if it gets stuck for some reason

sudo docker kill CONTAINER_ID