Separate two file into lines observed in both-/first_only-/second_only. Programmed using Cython.
Project description
The filediffs package
filediffs
takes two files and separates them into
- lines found in both files
- lines found only in file 1
- lines found only in file 2
Code inspired by https://www.splinter.com.au/reconcilingcomparing-huge-data-sets-with-c/
Installation
For package installation, Poetry is used.
Inside pyproject.toml the python version, requirements, build instructions, package descriptions etc. are defined.
You can create a virtual environment for the package with poetry by
- installing poetry
pip install poetry
- calling
poetry install
to install from poetry.lock
To create a .tar.gz file and wheeles for publishing the package one can use poetry build
To publish the package, poetry publish
can be used. Though
the pipy credentials have to be set (see https://python-poetry.org/docs/repositories/#configuring-credentials).
Implementation:
Implemented in Cython.
Lines found in both files are not kept in memory but written to disk every 5.000.000 lines to preserve memory.
This way, even very large files can be separated. Only the diff has to fit in memory.
The file build_cython_setup.py
defines the cython build process.
The cpp files can be build using python build_cython_setup.py build_ext --inplace
.
Usage:
To use the method in python in interaction with cython, the file paths have to passed to the function as bytestrings.
from filediffs.filediffs import file_diffs
lines_only_in_file_1, lines_only_in_file_2 = file_diffs(
filename_1=b'path/to/file1.txt',
filename_2=b'path/to/file2.txt',
outpath_lines_present_in_both_files=b'output_path/to/lines_in_both.txt',
outpath_lines_present_only_in_file1=b'output_path/to/lines_only_in_file1.txt',
outpath_lines_present_only_in_file2=b'output_path/to/lines_only_in_file2.txt',
)
Inside the package directory, an example script filediffs_script.py
is provided.
It can be used to separate files from the terminal:
# To separate two files, simply pass the filepath to `filediffs/filediffs_script.py`
python filediffs/filediffs_script.py filediffs/tests/data/file_1.txt filediffs/tests/data/file_2.txt
# If you want to define the filenames of the separated files, optional arguments are provided for the script.
python filediffs/filediffs_script.py filediffs/tests/data/file_1.txt filediffs/tests/data/file_2.txt --out_filename_both out_both.txt --out_filename_only_in_file1 out_file1_only.txt --out_filename_only_in_file2 out_file2_only.txt
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for filediffs-0.0.3-cp39-cp39-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8d317b19f7d4cb8d3e6b5f8d1c6b7b6bbd0539f3c2c0e365b54d5dec8e2e20e2 |
|
MD5 | e2787ea0e1bb3c7868112b056e5bcd7f |
|
BLAKE2b-256 | 9f57792e30d2b758354b5c053c6b9110c4b7bdb16f9920fbc4e976833d9e9752 |
Hashes for filediffs-0.0.3-cp38-cp38-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5b833e6abfc13fd72bdc76f480f2486765814f516bbfb14528a38bf40f1cdb37 |
|
MD5 | 8214071301d24fecfee247caacefcf24 |
|
BLAKE2b-256 | 29312404820da52c33f7768e896a763dc83b6f4ec159bf214bc3fcfa0e589eff |
Hashes for filediffs-0.0.3-cp37-cp37m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 74c64edd1831b513e1cf0c2a2442888262075f450772b7603906c8cf16ccf01a |
|
MD5 | 06e5ebc532a2999669e766fbc1c08e85 |
|
BLAKE2b-256 | c259f204ed540bf95b1f44a0ea466a3f303a37c78eec4303f08e11a12fa67557 |
Hashes for filediffs-0.0.3-cp36-cp36m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6cbdcadf2fdc77959f24ddea7404e74b3600dbc6a6b3e5520021221f1c6ee979 |
|
MD5 | 3b5c0c740ea6c49c53c344167eef1d6e |
|
BLAKE2b-256 | f2c8e9040bf4b4613c429d77e5ce2e2c36f936988663657b6c42b8f9939c92f1 |
Hashes for filediffs-0.0.3-cp35-cp35m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ed14d85d62499a27a6f6f8430c73fe595555144f5c02012c6680b1a28b2ac614 |
|
MD5 | e51a3836a0b9c1b2319abbe9585700ee |
|
BLAKE2b-256 | 52b52fa67f4ad624b016ae36f60a191a6c0c492005597d6b43fc4fd0db46e3bb |