# Data extractor for PDF invoices - invoice2data

[![Circle CI](https://circleci.com/gh/invoice-x/invoice2data.svg?style=svg)](https://circleci.com/gh/invoice-x/invoice2data)

This project has been selected for [GSoC 2018](https://developers.google.com/open-source/gsoc/). Read more [here](https://wiki.debian.org/SummerOfCode2018/Projects/ExtractingDataFromPDFInvoicesAndBillsDetails).

A modular Python library to support your accounting process. Tested on Python 2.7 and 3.4+. Main steps:

1. extracts text from PDF files using different techniques, like `pdftotext`, `pdfminer` or `tesseract` OCR.
2. searches for regex in the result using a YAML-based template system
3. saves results as CSV, JSON or XML or renames PDF files to match the content.

With the flexible template system you can:

- precisely match content PDF files
- define static fields that are the same for every invoice
- define custom fields needed in your organisation or process
- have multiple regex per field (if layout or wording changes)
- define currency
- extract invoice-items using the `lines`-plugin developed by [Holger Brunn](https://github.com/hbrunn)

Go from PDF files to this:

```
{'date': (2014, 5, 7), 'invoice_number': '30064443', 'amount': 34.73, 'desc': 'Invoice 30064443 from QualityHosting', 'lines': [{'price': 42.0, 'desc': u'Small Business StandardExchange 2010\nGrundgeb\xfchr pro Einheit\nDienst: OUDJQ_office\n01.05.14-31.05.14\n', 'pos': u'7', 'qty': 1.0}]}
{'date': (2014, 6, 4), 'invoice_number': 'EUVINS1-OF5-DE-120725895', 'amount': 35.24, 'desc': 'Invoice EUVINS1-OF5-DE-120725895 from Amazon EU'}
{'date': (2014, 8, 3), 'invoice_number': '42183017', 'amount': 4.11, 'desc': 'Invoice 42183017 from Amazon Web Services'}
{'date': (2015, 1, 28), 'invoice_number': '12429647', 'amount': 101.0, 'desc': 'Invoice 12429647 from Envato'}
```

## Installation

1. Install pdftotext

If possible get the latest [xpdf/poppler-utils](https://poppler.freedesktop.org/) version. It's included with macOS Homebrew, Debian and Ubuntu. Without it, `pdftotext` won't parse tables in PDF correctly.

2. Install `invoice2data` using pip

```
pip install invoice2data
```

## Usage

Basic usage. Process PDF files and write result to CSV.
- `invoice2data invoice.pdf`
- `invoice2data *.pdf`

Choose any of the following three input readers:
* pdftotext `invoice2data --input-reader pdftotext invoice.pdf`
* tesseract `invoice2data --input-reader tesseract invoice.pdf`
* pdf miner `invoice2data --input-reader pdfminer invoice.pdf`

Choose any of the following output formats:
* csv `invoice2data --output-format csv invoice.pdf`
* json `invoice2data --output-format json invoice.pdf`
* xml `invoice2data --output-format xml invoice.pdf`

Save output file with custom name or a specific folder
`invoice2data --output-format csv --output-name myinvoices/invoices.csv invoice.pdf`

**Note:** You must specify the `output-format` in order to create `output-name`

Specify folder with yml templates. (e.g. your suppliers)
`invoice2data --template-folder ACME-templates invoice.pdf`

Only use your own templates and exclude built-ins
`invoice2data --exclude-built-in-templates --template-folder ACME-templates invoice.pdf`

Processes a folder of invoices and copies renamed invoices to new folder.
`invoice2data --copy new_folder folder_with_invoices/*.pdf`

Processes a single file and dumps whole file for debugging (useful when adding new templates in templates.py)
`invoice2data --debug my_invoice.pdf`

Recognize test invoices:
`invoice2data invoice2data/test/pdfs/* --debug`

If you want to use it as a lib just do

```
from invoice2data import extract_data

result = extract_data('path/to/my/file.pdf')
```

## Template system

See `invoice2data/templates` for existing templates. Just extend the list to add your own. If deployed by a bigger organisation, there should be an interface to edit templates for new suppliers. 80-20 rule. For a short tutorial on how to add new templates, see [TUTORIAL.md](TUTORIAL.md).

Templates are based on Yaml. They define one or more keywords to find the right template and regexp for fields to be extracted. They could also be a static value, like the full company name.

Template files are tried in alphabetical order.

We may extend them to feature options to be used during invoice processing.

Example:

```
issuer: Amazon Web Services, Inc.
keywords:
- Amazon Web Services
fields:
amount: TOTAL AMOUNT DUE ON.*\$(\d+\.\d+)
amount_untaxed: TOTAL AMOUNT DUE ON.*\$(\d+\.\d+)
date: Invoice Date:\s+([a-zA-Z]+ \d+ , \d+)
invoice_number: Invoice Number:\s+(\d+)
partner_name: (Amazon Web Services, Inc\.)
options:
remove_whitespace: false
currency: HKD
date_formats:
- '%d/%m/%Y'
lines:
start: Detail
end: \* May include estimated US sales tax
first_line: ^ (?P<description>\w+.*)\$(?P<price_unit>\d+\.\d+)
line: (.*)\$(\d+\.\d+)
last_line: VAT \*\*
```

## Development
If you are interested in improving this project, have a look at our [developer guide](DEVELOP.md) to get you started quickly.

## Roadmap and open tasks

- integrate with online OCR?
- try to 'guess' parameters for new invoice formats.
- can apply machine learning to guess new parameters?

## Maintainers
- [Manuel Riel](https://github.com/m3nu)
- [Alexis de Lattre](https://github.com/alexis-via)

## Contributors
- [Harshit Joshi](https://github.com/duskybomb): As Google Summer of Code student.
- [Holger Brunn](https://github.com/hbrunn): Add support for parsing invoice items.

## Related Projects
* [OCR-Invoice](https://github.com/robela/OCR-Invoice) (FOSS | C#)
* [Docparser](https://docparser.com/) (Commercial | Web Service)
* [A-PDF](http://www.a-pdf.com/data-extractor/index.htm) (Commercial)
* [PDFdeconstruct](http://www.glyphandcog.com/PDFdeconstruct.html?g6) (Commercial)
* [CVision](http://www.cvisiontech.com/library/document-automation/forms-processing/extract-data-from-invoice.html) (Commercial)

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (1.0.1)

2 maintainers

PyPI data
Data sourced directly from PyPI's database.
Avatar for bosd from gravatar.com bosd Avatar for manuelriel from gravatar.com manuelriel

Credits

Author: Manuel Riel

License

MIT

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

invoice2data-0.2.93.tar.gz (28.9 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

invoice2data-0.2.93-py2.7.egg (89.4 kB view details)

Uploaded Egg

File details

Details for the file invoice2data-0.2.93.tar.gz.

File metadata

  • Download URL: invoice2data-0.2.93.tar.gz
  • Upload date:
  • Size: 28.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for invoice2data-0.2.93.tar.gz
Algorithm Hash digest
SHA256 841af162144a0f8c1ce39f2817ec3afab6ed5ace509f1ac12407bc2e91021fde
MD5 7255dfe7dd8d26e9d9e265588f3556c6
BLAKE2b-256 57fe79fea5c88de10744fc8d81f4cef40bf416f9fd9d7d9dc9add7bcd21810c0

See more details on using hashes here.

File details

Details for the file invoice2data-0.2.93-py2.7.egg.

File metadata

File hashes

Hashes for invoice2data-0.2.93-py2.7.egg
Algorithm Hash digest
SHA256 47a12dd4ed985a72e4999347ddd5a9b500a6631d92b6c281580db0707b8bbad1
MD5 e9f6304f5ec410497902095af2bc7d2c
BLAKE2b-256 2501ae1fad348547c00d18c79a4591f8564c59118fa1c47d17e493e914cfb641

See more details on using hashes here.

Release history Release notifications | RSS feed

1.0.1

21 files

1.0.0

21 files

0.5.0

2 files

0.4.7

2 files

0.4.6

2 files

0.4.5

2 files

0.4.4

2 files

0.4.3

2 files

0.4.2

2 files

0.4.1

2 files

0.4.0

2 files

0.3.6

2 files

0.3.5

2 files

0.3.4

2 files

0.3.3

2 files

0.3.2

2 files

0.3.1

2 files

0.2.103

2 files

0.2.101

2 files

0.2.100

2 files

0.2.99

2 files

0.2.98

2 files

0.2.97

2 files

0.2.96

2 files

0.2.95

2 files

0.2.94

2 files

This release

0.2.93 This release

2 files

0.2.92

2 files

0.2.91

2 files

0.2.90

2 files

0.2.89

2 files

0.2.88

2 files

0.2.87

2 files

0.2.86

2 files

0.2.85

2 files

0.2.84

2 files

0.2.83

2 files

0.2.82

2 files

0.2.81

2 files

0.2.80

2 files

0.2.79

2 files

0.2.78

2 files

0.2.77

2 files

0.2.76

2 files

0.2.75

2 files

0.2.74

2 files

0.2.73

2 files

0.2.72

2 files

0.2.71

2 files

0.2.70

2 files

0.2.69

2 files

0.2.67

2 files

0.2.66

2 files

0.2.65

2 files

0.2.64

2 files

0.2.63

2 files

0.2.62

2 files

0.2.61

2 files

0.2.59

2 files

0.2.58

2 files

0.2.56

2 files

0.2.55

2 files

0.2.54

2 files

0.2.53

2 files

0.2.51

2 files

0.2.49

2 files

0.2.47

2 files

0.2.45

2 files

0.2.44

2 files

0.2.43

2 files

0.2.42

2 files

0.2.41

2 files

0.2.40

2 files

0.2.39

2 files

0.2.38

2 files

0.2.36

2 files

0.2.34

2 files

0.2.33

2 files

0.2.31

2 files

0.2.30

2 files

0.2.29

2 files

0.2.28

2 files

0.2.27

2 files

0.2.26

2 files

0.2.25

2 files

0.2.24

2 files

0.2.22

2 files

0.2.21

2 files

0.2.20

2 files

0.2.19

2 files

0.2.18

2 files

0.2.17

2 files

0.2.16

2 files

0.2.15

2 files

0.2.14

2 files

0.2.13

2 files

0.2.10

1 file

0.2.9

2 files

0.2.8

2 files

0.2.5

1 file

0.2.4

1 file

0.2.3

1 file

0.2.2

1 file

0.2.1

1 file

0.2.0

1 file

0.1.2

1 file

0.0.1

1 file

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page