First proposed here: https://github.com/jsvine/pdfplumber/issues/201
Adding this feature involved refactoring and re-engineering a good chunk
of the text-layout-extraction code. As part of that, this commit
introduces two new classes, in utils.py: LayoutEngine and TextLayout.
They should be considered provisional, and may change name/approach in
the future.
A fairly large commit, adding type annotations/hints to the entire
library, and refactoring the library accordingly.
Most of the refactoring changes should have no practical effect on
usage, but several others are notable:
- Added `TableSettings` class, a behind-the-scenes handler for managing
and validating table-extraction settings.
- Renamed the positional argument to `.to_csv(...)` and `.to_json(...)`
from `types` to `object_types`.
- Tweaked the output of `.to_json(...)` so that, if an object type is
not present for a given page, it has no key in the page's object
representation.
- Removed `utils.filter_objects(...)` and move the functionality to
within the `FilteredPage.objects` property calculation, the only part
of the library that used it.
- Removed code that sets `pdfminer.pdftypes.STRICT = True` and
`pdfminer.pdfinterp.STRICT = True`, since that [has now been the
default for a
while](https://github.com/pdfminer/pdfminer.six/commit/9439a3a31a347836aad1c1226168156125d9505f).
Per discussion at https://github.com/jsvine/pdfplumber/discussions/346
and input from @ramcdona, this commit changes pdfplumber's behavior
regarding floating point numbers. Specifically, it removes all
conversion of floats to Decimal objects. This brings several advantages:
- Increased precision (where applicable)
- Decreased code complexity
- Increased performance (~10% speedup on test suite)
- Increased fidelity to `pdfminer.six` output
These seem to outweigh the disadvantages:
- Some tests break (but have been easily fixed) due to increased
precision and/or floating point arithmetic artifacts
- Some users' scripts may also break, if they depend on strict equality
testing, though these *should* also be easily fixable
Because some form of automatic rounding may still be desirable for the
pdfplumber CLI utility, the conversion methods (.to_csv, .to_json) have
been adjusted to accept a `precision` argument.
"direction" == 1 when upright words go LTR and non-upright words go TTB,
and -1 when the opposite is true. This attribute should help in reusing
the results of `.extract_words` in other functions, so that the user
knows how the extracted word corresponds to the word's layout on the
page.
Taking @samkit-jain's smart question about sort-order into account, this
commit reworks `.extract_words`, making the sorting more explicit, and
adding an option to *not* sort, via `use_text_flow=True`, which follows
the original PDF's text flow, rather than presorting all characters top-down,
left-right.
To do so required rethinking the word-delineation logic,
which in turn motivated a shift toward a more object-oriented approach
(especially to keep track of settings without passing them from function
to function). Now `.extract_text`'s logic is encapsulated in a new class,
`TextExtractor`.
This commit refactors and hopefully makes clearer the logic in
utils.extract_words. It also adds a new parameter, `extra_attrs`, which
allows the user to pass a list of attributes on which to group all
characters.
For instance, passing `extra_attrs=["fontname", "size"]` will not allow
characters with different font names or sizes to become part of the same
word. As a benefit, those resulting word dicts will contain `"fontname"`
and `"size"` attributes — providing a long-requested feature (cf. issue
Moves most of the logic previously in cli.py to convert.py, for usage by
other submodules. Adds Container.to_json and Container.to_csv. Makes
adjustments/fixes to other parts of the library, based on edge-cases
encountered (such as infinite recursion in anntations).
- Fixes `.extract_words`, which had been returning incorrect results when `horizontal_ltr = False`
- Fixes `.resize_object`, which had been failing in various permutations
- Brings utils.py test coverage to 100%
Jeremy Singer-Vine