Skip to main content

Convert binary files to plain text for indexing.

Project description

Travis CI Status

A SmartFile Open Source project. Read more about how SmartFile uses and contributes to Open Source software.



Fulltext is meant to be used for full-text indexing of file contents for search applications.

Fulltext is a library that makes converting various file formats to plain text simple. Mostly it is a wrapper around shell tools. It will execute the shell program, scrape it’s results and then post-process the results to pack as much text into as little space as possible.

Supported formats

The following formats are supported using the command line apps listed.

  • application/pdf: pdftotext

  • application/msword: antiword

  • application/vnd.openxmlformats-officedocument.wordprocessingml.document: docx2txt

  • application/ convertxls2csv

  • application/rtf: unrtf

  • application/vnd.oasis.opendocument.text: odt2txt

  • application/vnd.oasis.opendocument.spreadsheet: odt2txt

  • application/zip: funzip

  • application/x-tar, gzip: tar & gunzip

  • application/x-tar, bzip2: tar & bunzip2

  • application/rar: unrar

  • text/html: html2text

  • text/xml: html2text

  • image/jpeg: exiftool

  • video/mpeg: exiftool

  • audio/mpeg: exiftool

  • application/octet-stream: strings

Installing tools

Fulltext uses the above command line programs to function. Therefore, it is not useful unless you have installed them. Many of them can be installed via your system’s package manager. I use Fedora, thus the following command installed most of the required packages.

$ sudo yum install xls2csv odt2txt antiword poppler-utils unrtf \
perl-Image-ExifTool html2text binutils unrar gzip bzip2 unzip

The docx2txt utility is not avaialable in a package.

The package names may differ on other systems, but for the most part will be similar.


To use the library, simply pass a filename to the .get() module function. A second optional argument default can provide a string to be returned in case of error. This way, if you are not concerned with exceptions, you can simply ignore them by providing a default. This is like how the dict.get() method works.

> import fulltext
> fulltext.get('does-not-exist.pdf', '< no content >')
'< no content >'
> fulltext.get('exists.pdf', '< no content >'')
'Lorem ipsum...'

There is also a quick way to check for the existence of all of the required tools.

> import fulltext
> fulltext.check()
Cannot execute command docx2txt, please install it.


Some formats require additional care, this is done in the post-processing step. For example, unrtf is the tool used to convert .rtf files to text. It prints a banner including the program version and some document metadata. This header is removed in post-processing.

A simple regular expression is used to convert adjacent whitespace characters to a single space.

This results in the highest word-per-byte ratio possible, allowing your full-text engine to quickly index the file contents.


Sometimes multiple tools can be used. For example, catdoc provides xls2csv, while xls2csv provides convertxls2csv. We should use whichever is present.

I would like to do away with commands as tuples, and simply use strings. This is something easyprocess can do.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fulltext-0.4-1.tar.gz (6.6 kB view hashes)

Uploaded source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page