Skip to main content
Author:

Noah Gift

Version:
0.1.5

Summary

A deduplication command line tool and library. A relatively efficient algorithm based on searching like sized files, and then performing a full md5 checksum, is used to determine duplicate files/file objects. Files can be deleted upon discovery, and pattern matching can be used to limit search results. Finally, configuration file use is supported, and there is a developing API that lends itself to customization via an ActionsMixin class.

Example CLI Usage:

Size:

Search by size using –size or -s option:

liten.py -s 1 /mnt/raid         is equal to liten.py -s 1MB /mnt/raid
liten.py -s 1bytes /mnt/raid
liten.py -s 1KB /mnt/raid
liten.py -s 1MB /mnt/raid
liten.py -s 1GB /mnt/raid
liten.py c:\in d:\              is equal to liten.py -s 1MB c:\in d:\

Report Location:

Generate custom report path using -r or –report=/tmp/report.txt:

./liten.py --report=/tmp/test.txt /Users/ngift/Documents

By default a report will be created in CWD, called LitenDuplicateReport.csv

Config File:

You can use a config file in the following format:

[Options]
path=/tmp
size=1MB
pattern=*.m4v
delete=True

You can call the config file anything and place it anywhere.

Here is an example usage:

./liten.py --config=myconfig.ini

Verbosity:

All stdout can be suppressed by using –quiet or -q.

Delete:

By using –delete the duplicate files will be automatically deleted. The API has support for an interactive mode and a dry-run mode, they have not been implemented in the CLI as of yet.

Example Library/API Usage:

>>> Liten = Liten(spath='testData')
>>> dupeFileOne = 'testData/testDocOne.txt'
>>> checksumOne = Liten.createChecksum(dupeFileOne)
>>> dupeFileTwo = 'testData/testDocTwo.txt'
>>> checksumTwo = Liten.createChecksum(dupeFileTwo)
>>> nonDupeFile = 'testData/testDocThree_wrong_match.txt'
>>> checksumThree = Liten.createChecksum(nonDupeFile)
>>> checksumOne == checksumTwo
True
>>> checksumOne == checksumThree
False

There is also the concept of an Action, which can be implemented later, that will allow customizable actions to occur upon an a condition that gets defined as you walk down a tree of files.

Tests:

  • Run Doctests: ./liten -t or –test

  • Run test_liten.py

  • Run test_create_file.py then delete those test files using liten::

    python2.5 liten.py –delete /tmp

Display Options:

Stdout:

stdout will show you duplicate file paths and sizes such as:

Printing dups over 1 MB using md5 checksum: [SIZE] [ORIG] [DUP]
7 MB  Orig:  /Users/ngift/Downloads/bzr-0-2.17.tar
Dupe:  /Users/ngift/Downloads/bzr-0-4.17.tar

Report:

A report named LitenDuplicateReport.csv will be created in your current working directory:

Duplicate Version,     Path,       Size,       ModDate
Original, /Users/ngift/Downloads/bzr-0-2.17.tar, 7 MB, 07/10/2007 01:43:12 AM
Duplicate, /Users/ngift/Downloads/bzr-0-3.17.tar, 7 MB, 07/10/2007 01:43:27 AM

Debug Mode Environmental Variables:

  • To enable print statement debugging set LITEN_DEBUG to 1

  • To enable pdb break point debugging set LITEN_DEBUG to 2

  • LITEN_DEBUG_MODE = int(os.environ.get(‘LITEN_DEBUG’, 0))

  • Note: When DEBUG MODE is enabled, a message will appear to standard out

QUESTIONS: noah dot gift at gmail.com

Metadata

Release files for liten 0.1.6

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for liten 0.1.6
File Size Uploaded Provenance Actions
liten-0.1.6.tar.gz 9.5 kB Without provenance Details for liten-0.1.6.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for liten 0.1.6
File Interpreter ABI Platform Actions
liten-0.1.6-py2-none-any.whl Python 2 none any Details for liten-0.1.6-py2-none-any.whl

Total release size: 22.0 kB

Release files / liten-0.1.6.tar.gz

Download URL liten-0.1.6.tar.gz
Size 9.5 kB
Tags Source
SHA-256 checksum
How to use checksums
cfa32ef3faaa9ee50cedb4ababf2d04a697916b8df54d8caa86f1e05ea417377
BLAKE2b-256 checksum
How to use checksums
f537b609912784be69876beb1f386faf7bd5bcaa3d387f226c69267513dd0b49
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Release files / liten-0.1.6-py2-none-any.whl

Download URL liten-0.1.6-py2-none-any.whl
Size 12.5 kB
Tags Python 2
SHA-256 checksum
How to use checksums
26432a0624d9042916091e250e96e01c996806a1364eb84f9b7fe18c5913a348
BLAKE2b-256 checksum
How to use checksums
e30695ed41fde2b38500e8758d46b1de68ce01a3d6bbc401a178beab1bd96c1b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Security for liten 0.1.6

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.6 This release

2 release files No provenance

0.1.5.0

0.1.4.2

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page