Skip to main content
This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (pypi.python.org).
Help us improve Python packaging - Donate today!

scan and find double PDF files, by creating a hash database

Project Description

this package provides the pdfdbl command:

pdfdbl scan dir1 dir2

This will walk down the directories provided as argument and for the PDF files found, create a hash based on (in order):

- metadata if unique
- images, if the number of images matches the number of pages
- text

This assumes that pdfinfo, pdfimages and pdftotext` from the poppler-utils package are avaialable.

A “database” is build up in ~/.config/pdfdbl/pdf.lst against which further scans are tested.

Removing markings

In ruamel/pdfdouble/pdfdouble.py there are two methods that can be enhanced to filter out markings in the PDF that make them less unique and make virtually the same files to have different hashes.

For text the method PdfData.filter_for_marking should be extended to remove and markings from the string that is its arguments and return the result.

For scanned images the method PdfData.process_image_and_update needs to be enhanced, e.g. by cutting off the images bottom and top X lines, and by removing any gray background text by setting all black pixels to white. This function needs to update the hash passed in using the .update() method passing in the filtered data.

Restrictions

The current “database” cannot handle paths that contain newlines

This utility is currently Python 2.7 only.

Release History

Release History

This version
History Node

0.1.2

History Node

0.1.1

History Node

0.1

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
ruamel.pdfdouble-0.1.2-py2.py3-none-any.whl (9.7 kB) Copy SHA256 Checksum SHA256 py2.py3 Wheel Aug 29, 2015
ruamel.pdfdouble-0.1.2.tar.gz (11.3 kB) Copy SHA256 Checksum SHA256 Source Aug 29, 2015

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting