ninvoice2data 0.4.16

Data extractor for PDF invoices - ninvoice2data
==============================================

|Circle CI|

This project has been selected for `GSoC
2018 <https://developers.google.com/open-source/gsoc/>`__. Read more
`here <https://wiki.debian.org/SummerOfCode2018/Projects/ExtractingDataFromPDFInvoicesAndBillsDetails>`__.

A modular Python library to support your accounting process. Tested on
Python 2.7 and 3.4+. Main steps:

1. extracts text from PDF files using different techniques, like
``pdftotext``, ``pdfminer`` or OCR – ``tesseract``, ``tesseract4`` or
``gvision`` (Google Cloud Vision).
2. searches for regex in the result using a YAML-based template system
3. saves results as CSV, JSON or XML or renames PDF files to match the
content.

With the flexible template system you can:

- precisely match content PDF files
- plugins available to match line items and tables
- define static fields that are the same for every invoice
- define custom fields needed in your organisation or process
- have multiple regex per field (if layout or wording changes)
- define currency
- extract invoice-items using the ``lines``-plugin developed by `Holger
Brunn <https://github.com/hbrunn>`__

Go from PDF files to this:

::

{'date': (2014, 5, 7), 'invoice_number': '30064443', 'amount': 34.73, 'desc': 'Invoice 30064443 from QualityHosting', 'lines': [{'price': 42.0, 'desc': u'Small Business StandardExchange 2010\nGrundgeb\xfchr pro Einheit\nDienst: OUDJQ_office\n01.05.14-31.05.14\n', 'pos': u'7', 'qty': 1.0}]}
{'date': (2014, 6, 4), 'invoice_number': 'EUVINS1-OF5-DE-120725895', 'amount': 35.24, 'desc': 'Invoice EUVINS1-OF5-DE-120725895 from Amazon EU'}
{'date': (2014, 8, 3), 'invoice_number': '42183017', 'amount': 4.11, 'desc': 'Invoice 42183017 from Amazon Web Services'}
{'date': (2015, 1, 28), 'invoice_number': '12429647', 'amount': 101.0, 'desc': 'Invoice 12429647 from Envato'}

Installation
------------

1. Install pdftotext

If possible get the latest
`xpdf/poppler-utils <https://poppler.freedesktop.org/>`__ version. It’s
included with macOS Homebrew, Debian and Ubuntu. Without it,
``pdftotext`` won’t parse tables in PDF correctly.

2. Install ``ninvoice2data`` using pip

::

pip install ninvoice2data

Usage
-----

Basic usage. Process PDF files and write result to CSV.

- ``ninvoice2data invoice.pdf``
- ``ninvoice2data *.pdf``

Choose any of the following input readers:

- pdftotext ``ninvoice2data --input-reader pdftotext invoice.pdf``
- tesseract ``ninvoice2data --input-reader tesseract invoice.pdf``
- pdf miner ``ninvoice2data --input-reader pdfminer invoice.pdf``
- tesseract4 ``ninvoice2data --input-reader tesseract4 invoice.pdf``
- gvision ``ninvoice2data --input-reader gvision invoice.pdf`` (needs ``GOOGLE_APPLICATION_CREDENTIALS`` env var)

Choose any of the following output formats:

- csv ``ninvoice2data --output-format csv invoice.pdf``
- json ``ninvoice2data --output-format json invoice.pdf``
- xml ``ninvoice2data --output-format xml invoice.pdf``

Save output file with custom name or a specific folder
``ninvoice2data --output-format csv --output-name myinvoices/invoices.csv invoice.pdf``

**Note:** You must specify the ``output-format`` in order to create
``output-name``

Specify folder with yml templates. (e.g. your suppliers)
``ninvoice2data --template-folder ACME-templates invoice.pdf``

Only use your own templates and exclude built-ins
``ninvoice2data --exclude-built-in-templates --template-folder ACME-templates invoice.pdf``

Processes a folder of invoices and copies renamed invoices to new
folder. ``ninvoice2data --copy new_folder folder_with_invoices/*.pdf``

Processes a single file and dumps whole file for debugging (useful when
adding new templates in templates.py)
``ninvoice2data --debug my_invoice.pdf``

Recognize test invoices:
``ninvoice2data ninvoice2data/test/pdfs/* --debug``

If you want to use it as a lib just do

::

from ninvoice2data import extract_data

result = extract_data('path/to/my/file.pdf')

Template system
---------------

See ``ninvoice2data/extract/templates`` for existing templates. Just extend the
list to add your own. If deployed by a bigger organisation, there should
be an interface to edit templates for new suppliers. 80-20 rule. For a
short tutorial on how to add new templates, see
`TUTORIAL.rst <TUTORIAL.rst>`__.

Templates are based on Yaml. They define one or more keywords to find
the right template and regexp for fields to be extracted. They could
also be a static value, like the full company name.

Template files are tried in alphabetical order.

We may extend them to feature options to be used during invoice
processing.

Example:

::

issuer: Amazon Web Services, Inc.
keywords:
- Amazon Web Services
fields:
amount: TOTAL AMOUNT DUE ON.*\$(\d+\.\d+)
amount_untaxed: TOTAL AMOUNT DUE ON.*\$(\d+\.\d+)
date: Invoice Date:\s+([a-zA-Z]+ \d+ , \d+)
invoice_number: Invoice Number:\s+(\d+)
partner_name: (Amazon Web Services, Inc\.)
options:
remove_whitespace: false
currency: HKD
date_formats:
- '%d/%m/%Y'
lines:
start: Detail
end: \* May include estimated US sales tax
first_line: ^ (?P<description>\w+.*)\$(?P<price_unit>\d+\.\d+)
line: (.*)\$(\d+\.\d+)
last_line: VAT \*\*

Development
-----------

If you are interested in improving this project, have a look at our
`developer guide <DEVELOP.rst>`__ to get you started quickly.

Roadmap and open tasks
----------------------

- integrate with online OCR?
- try to ‘guess’ parameters for new invoice formats.
- can apply machine learning to guess new parameters?

Maintainers
-----------

- `Manuel Riel <https://github.com/m3nu>`__
- `Alexis de Lattre <https://github.com/alexis-via>`__

Contributors
------------

- `Harshit Joshi <https://github.com/duskybomb>`__: As Google Summer of
Code student.
- `Holger Brunn <https://github.com/hbrunn>`__: Add support for parsing
invoice items.

Related Projects
----------------

- `OCR-Invoice <https://github.com/robela/OCR-Invoice>`__ (FOSS \| C#)
- `Docparser <https://docparser.com/>`__ (Commercial \| Web Service)
- `A-PDF <http://www.a-pdf.com/data-extractor/index.htm>`__
(Commercial)
- `PDFdeconstruct <http://www.glyphandcog.com/PDFdeconstruct.html?g6>`__
(Commercial)
- `CVision <http://www.cvisiontech.com/library/document-automation/forms-processing/extract-data-from-invoice.html>`__
(Commercial)

.. |Circle CI| image:: https://circleci.com/gh/invoice-x/ninvoice2data.svg?style=svg
:target: https://circleci.com/gh/invoice-x/ninvoice2data