libraries to make it easier to maniuplate files in a directory tree
Project description
treecrawl
libraries to make it easier to maniuplate files in a directory tree
Free software: MIT license
Documentation: https://treecrawl.readthedocs.io.
Usage
This project makes it a little easier to edit directory trees and to test those edits.
This example uses the Transformer class to rewrite the contents of all the files in a directory to upper case text. is_target() and transform() should always be overridden. You should almost always create and use an alternative to Transformer.write_string_to_output(). Treating everything like a string will cause problems with editing and testing with any unicode at all. It’s really just meant for a simple example.
class MakeUpper(Transformer):
"""Convert non-ASCII files to ASCII"""
def __init__(self, input, output, dry_run=False):
super().__init__(input=input, output=output, dry_run=dry_run)
def is_target(self, i_file):
"""
I generally use opt-in targeting to avoid corrupting files i don't want
to target when I override Transformer.is_target(). I use extensions
where it's adequate, but if I need something more robust, I might use
python-magic.
"""
included_extensions = [".txt"]
# if it's not a file, right?
if not os.path.isfile(i_file):
return False
# Regardless of extension if the file is in a .git directory
# exclude it
if ".git" in i_file.split(os.path.sep):
return False
# now target only files ending in ".txt
# i could use
_, ext = os.path.splitext(i_file)
if ext in included_extensions:
return True
return False
def transform(self, source_file, destination_file):
from treecrawl.utility import file_to_string
contents = file_to_string(source_file)
contents = contents.upper()
self.write_string_to_output(contents, destination_file)
** CAUTION!! ** treecrawl doesn’t protect you from mistreating your files by, for example, corrupting a binary file because you transformed it like a text file. In fact, utility.file_to_string() encodes binary to utf-8 ignoring errors, so it will help you wreck your files.
This project also helps me test transformations using golden files. The following example shows how to enable pytest –update_golden to update the golden files automatically
First I need to setup conftest.py for the pytest flag:
import pytest
from treecrawl.utility import locate_subdir
def pytest_addoption(parser):
parser.addoption(
"--update_golden",
action="store_true",
help="Update golden files before running tests",
)
@pytest.fixture
def update_golden(request):
return request.config.getoption("--update_golden")
@pytest.fixture(scope="session", autouse=True)
def testdata():
return locate_subdir("testdata")
Next I create a parameterized test case for make upper. I have to manually create the input test data. Refer to tests/testdata/test_make_upper for an example.
@pytest.mark.parametrize(
"test_case",
["pets", "cities"],
)
def test_make_upper(test_case, tmp_path, request, testdata, update_golden):
c = CaseHelper(
testdata,
request.node.originalname,
test_case,
str(tmp_path),
update_golden=update_golden,
)
"""when update golden is set by running pytest --update_golden,
the project golden files are deleted. This step generates new ones from
the the function under test """
if update_golden:
_ = MakeUpper(c.input, c.golden)
m = MakeUpper(c.input, c.actual)
m.run()
for r in c.compare():
succeeded, compared = r
assert succeeded
if not succeeded:
print("input: {}\nactual: {}\nexpected: {}".format(*compared))
It may also be important to override the CaseHelper.compare()
Credits
This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.
Build Notes
Setup dev venv
python -m venv .treecrawl.venv source .treecrawl.venv/bin/activate pip install -r requirements-dev.txt
Tests
I use pyenv to provide multiple versions for nox python testing. in my case:
pyenv install 3.6.8
pyenv install 3.7.8
# in the project directory
pyenv local 3.6.8 3.7.8
make test
If other versions are flagged as missing or are skipped you can just pyenv instal them and add them to the project directory
run ‘make test’ to run all the tests. I use pyenv to install all of the supported python versions so nox can run the full matrix of tests for me
always run ‘ make lint’
History
0.1.3 (2020-07-17)
First release on PyPI.
0.1.4 (2020-07-17)
Reorganized modules and updateed documentation
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for treecrawl-0.1.24-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e50c2d6ac0acd7c369afcffbc8c463cfae88734369db5a288c588c2af5728357 |
|
MD5 | 1c50f91f18bac3a6eb86160fa6a34de2 |
|
BLAKE2b-256 | ac1979d0667b2885825e2ecc3f24af07ddf32e105f202724ea0f9708c29f54ed |