treecrawl 0.1.23

libraries to make it easier to maniuplate files in a directory tree

treecrawl

libraries to make it easier to maniuplate files in a directory tree

• Free software: MIT license

• Documentation: https://treecrawl.readthedocs.io.

Usage

This project makes it a little easier to edit directory trees and to test those edits.

This example uses the Transformer class to rewrite the contents of all the files in a directory to upper case text. is_target() and transform() should always be overridden. You should almost always create and use an alternative to Transformer.write_string_to_output(). Treating everything like a string will cause problems with editing and testing with any unicode at all. It’s really just meant for a simple example.

class MakeUpper(Transformer):
"""Convert non-ASCII files to ASCII"""

def __init__(self, input, output, dry_run=False):
    super().__init__(input=input, output=output, dry_run=dry_run)

def is_target(self, i_file):
    """
    I generally use opt-in targeting to avoid corrupting files i don't want
    to target when I override Transformer.is_target().  I use extensions
    where it's adequate, but if I need something more robust, I might use
    python-magic.

    """
    included_extensions = [".txt"]

    # if it's not a file, right?
    if not os.path.isfile(i_file):
        return False

    # Regardless of extension if the file is in a .git directory
    # exclude it
    if ".git" in i_file.split(os.path.sep):
        return False

    # now target only files ending in ".txt
    # i could use
    _, ext = os.path.splitext(i_file)
    if ext in included_extensions:
        return True

    return False

def transform(self, source_file, destination_file):
    from treecrawl.utility import file_to_string

    contents = file_to_string(source_file)
    contents = contents.upper()
    self.write_string_to_output(contents, destination_file)

** CAUTION!! ** treecrawl doesn’t protect you from mistreating your files by, for example, corrupting a binary file because you transformed it like a text file. In fact, utility.file_to_string() encodes binary to utf-8 ignoring errors, so it will help you wreck your files.

This project also helps me test transformations using golden files. The following example shows how to enable pytest –update_golden to update the golden files automatically

First I need to setup conftest.py for the pytest flag:

import pytest
from treecrawl.utility import locate_subdir


def pytest_addoption(parser):
    parser.addoption(
        "--update_golden",
        action="store_true",
        help="Update golden files before running tests",
    )


@pytest.fixture
def update_golden(request):
    return request.config.getoption("--update_golden")


@pytest.fixture(scope="session", autouse=True)
def testdata():
    return locate_subdir("testdata")

Next I create a parameterized test case for make upper. I have to manually create the input test data. Refer to tests/testdata/test_make_upper for an example.

@pytest.mark.parametrize(
    "test_case",
    ["pets", "cities"],
)
def test_make_upper(test_case, tmp_path, request, testdata, update_golden):
    c = CaseHelper(
        testdata,
        request.node.originalname,
        test_case,
        str(tmp_path),
        update_golden=update_golden,
    )

    """when update golden is set by running pytest --update_golden,
    the project golden files are deleted. This step generates new ones from
    the the function under test """
    if update_golden:
        _ = MakeUpper(c.input, c.golden)

    m = MakeUpper(c.input, c.actual)
    m.run()
    for r in c.compare():
        succeeded, compared = r
        assert succeeded
        if not succeeded:
            print("input: {}\nactual: {}\nexpected: {}".format(*compared))

It may also be important to override the CaseHelper.compare()

Credits

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.

Build Notes

Setup dev venv

python -m venv .treecrawl.venv
source .treecrawl.venv/bin/activate
pip install -r requirements-dev.txt

Tests

I use pyenv to provide multiple versions for nox python testing. in my case:

pyenv install 3.6.8
pyenv install 3.7.8
# in the project directory
pyenv local 3.6.8 3.7.8
make test

If other versions are flagged as missing or are skipped you can just pyenv instal them and add them to the project directory

run ‘make test’ to run all the tests. I use pyenv to install all of the supported python versions so nox can run the full matrix of tests for me

always run ‘ make lint’

History

0.1.3 (2020-07-17)

• First release on PyPI.

0.1.4 (2020-07-17)

• Reorganized modules and updateed documentation