ipypdf 0.0.9

Jupyter widget for applying nlp to pdf documents

PDF Digitizer (It has a back button!!)

A Jupyter-based tool to help parse out structured text from a PDF document and explore the contents.

Requirements

• Tesseract
• jupyterlab (notebook also works)

Installation

It is highly recommended that you install this into a clean environment.

Note: layoutparser puts an upper bound on numpy (1.19.3), so if you want to use the Parse Layout button, it's best to install this in an empty environment.

Conda (recommended)

conda create -n ipypdf python pip jupyterlab tesseract -c conda-forge
conda activate ipypdf
python -m pip install ipypdf

No Conda (not tested)

Install Tesseract

python -m venv envs/ipypdf
cd envs/ipypdf/Scripts
activate.bat
python -m pip install jupyterlab 
python -m pip install ipypdf

Development

see DEVELOPMENT.md

Common Issues

• AutoTools widget keeps saying layoutparser is not installed
  • This is usually a problem with pywin32.
  • Try conda install pywin32
  • Also make sure that numpy is <1.19.3

Usage

ipypdf is build for jupyter lab but should also work in jupyter notebooks.

1. Launch jupyter lab with jupyter lab

from ipypdf import App
app = App("path/to/your/pdfs", bulk_render=False)
app

Features

Auto-Parser

layoutparser is used to determine the location of textblocks, images, and section headers. There is not currently a way to automatically determine the hierarchical position of these items.

Note: this is 4x speed

Also: This video is out-dated now. The AutoParse button will now attempt to sort all of the nodes. As well as attempting to deduce the 1st level of hierarchical structure.

Table Parsing

Cytoscape

Folders, PDF Documents, and Sections have a tab labeled Cytoscape. This runs a tfidf similarity calculation over all nodes beneath the selected item. I.e. if you select the root node, then all defined nodes will be included in the calculation. However, only those with a link to another node will be drawn (this is for speed, may change this in the future).

The color of each node denotes the pdf document it originated from.

Selecting a node in the graph will highlight the node in the DocTree. Clicking the node in the DocTree will render the first page of the node.

Digitizing Utilities

I recommend turning off Show Boxes as this changes pages every time you add a node (working on a better solution)

Each node has a specific set of tools available to use. Here are the tools provided when a Section node is selected. Starting from the left:

• Add Section Node adds a sub-node of type Section and selects it
• Add Text Node adds a sub-node of type Text and selects it
• Add Image Node ...
• Delete Node Delete the selected node and all of its children

Content Selector

Content is extracted from the rendered image. Text is extracted using Optical Character Recognition (OCR). Images don't do any image analysis, they just denote coordinates and page number so that they can be retreived later if need be.

When a Section node is selected, the selection tool will attempt to parse text from the portion of the page selected by the user. This text will overwrite the label assigned to the node.

When a Text node is selected, the selection tool will attempt to parse text from the selected area and append it to the node's content. This is because text blocks are not always perfectly rectangular, and often span multiple pages.

When an Image node is selected, the coordinates of the box are appended to the node's content.

Save Button

This will generate json files for each document in the directory with instructions for regenerating any nodes you have created when you open the tool again. Alternatively, you can just load the json into another script to extract the document structure if all you want is the text and the hierarchy.