First proposed here: https://github.com/jsvine/pdfplumber/issues/201
Adding this feature involved refactoring and re-engineering a good chunk
of the text-layout-extraction code. As part of that, this commit
introduces two new classes, in utils.py: LayoutEngine and TextLayout.
They should be considered provisional, and may change name/approach in
the future.
Added weak tests for a couple of hard-to-test visual debugging methods
in display.py.
Also marked an exception-handling block in pdf.py as `pragma: nocover`
because we don't yet have a PDF to test it with, but the logic there is
simple and straightforward.
A fairly large commit, adding type annotations/hints to the entire
library, and refactoring the library accordingly.
Most of the refactoring changes should have no practical effect on
usage, but several others are notable:
- Added `TableSettings` class, a behind-the-scenes handler for managing
and validating table-extraction settings.
- Renamed the positional argument to `.to_csv(...)` and `.to_json(...)`
from `types` to `object_types`.
- Tweaked the output of `.to_json(...)` so that, if an object type is
not present for a given page, it has no key in the page's object
representation.
- Removed `utils.filter_objects(...)` and move the functionality to
within the `FilteredPage.objects` property calculation, the only part
of the library that used it.
- Removed code that sets `pdfminer.pdftypes.STRICT = True` and
`pdfminer.pdfinterp.STRICT = True`, since that [has now been the
default for a
while](https://github.com/pdfminer/pdfminer.six/commit/9439a3a31a347836aad1c1226168156125d9505f).
Now all tolerance settings have x/y versions as well.
This commit also changes `table.merge_edges(...)` behavior when
`join_tolerance` (and `x`/`y` variants) `<= 0`, so that joining is
attempted regardless, to handle cases of overlapping lines.
Per discussion at https://github.com/jsvine/pdfplumber/discussions/346
and input from @ramcdona, this commit changes pdfplumber's behavior
regarding floating point numbers. Specifically, it removes all
conversion of floats to Decimal objects. This brings several advantages:
- Increased precision (where applicable)
- Decreased code complexity
- Increased performance (~10% speedup on test suite)
- Increased fidelity to `pdfminer.six` output
These seem to outweigh the disadvantages:
- Some tests break (but have been easily fixed) due to increased
precision and/or floating point arithmetic artifacts
- Some users' scripts may also break, if they depend on strict equality
testing, though these *should* also be easily fixable
Because some form of automatic rounding may still be desirable for the
pdfplumber CLI utility, the conversion methods (.to_csv, .to_json) have
been adjusted to accept a `precision` argument.
See pdfminer.six's changelog for details:
https://github.com/pdfminer/pdfminer.six/blob/develop/CHANGELOG.md
... but a key difference is an improvement in how it assigns `line`,
`rect`, and `curve` objects. (Diagonal two-point lines, for instance,
are now `line` objects instead of `curve` objects.)
As a result, this commit also adjusts some of the tests, where the
pre-20211012 versions had been incorrectly assigning lines as `LTCurve`
objects.
This commit adds an `--laparams` flag to the pdfplumber CLI, giving it
more feature parity with the core library. To do so, it makes some
internal changes to `convert.py`, including changing the list of objects
to convert from *a predefined default list* to *all types extracted*.
pdfminer.six's `LTAnno` objects are not PDF annotations (which we
already provide access to via `.annots`, regardless of whether
`laparams` is set), but rather layout annotations. Per pdfminer.six
codebase:
> Note that, while a LTChar object has actual boundaries, LTAnno objects
> does not, as these are "virtual" characters, inserted by a layout
> analyzer according to the relationship between two characters (e.g. a
> space).
Because they have no boundaries, they cause problems for pdfplumber,
which expects bounding-box coordinates for all objects. See, e.g.,
issue #383, which this commit should fix.
This commit reinstates access to higher-level layout objects (such as
`textboxhorizontal`) when `laparams` is passed to
`pdfplumber.open(...)`. Had been removed in `0.5.24` via 1f87898.
Also adds a test for this behavior.
h/t @xv44586 for the initial inspiration 👍
These new methods return a version of the chars/page with duplicate
chars — those sharing the same text, fontname, size, and positioning
(within `tolerance` x/y) as other characters — removed.
"direction" == 1 when upright words go LTR and non-upright words go TTB,
and -1 when the opposite is true. This attribute should help in reusing
the results of `.extract_words` in other functions, so that the user
knows how the extracted word corresponds to the word's layout on the
page.
Taking @samkit-jain's smart question about sort-order into account, this
commit reworks `.extract_words`, making the sorting more explicit, and
adding an option to *not* sort, via `use_text_flow=True`, which follows
the original PDF's text flow, rather than presorting all characters top-down,
left-right.
To do so required rethinking the word-delineation logic,
which in turn motivated a shift toward a more object-oriented approach
(especially to keep track of settings without passing them from function
to function). Now `.extract_text`'s logic is encapsulated in a new class,
`TextExtractor`.
This commit refactors and hopefully makes clearer the logic in
utils.extract_words. It also adds a new parameter, `extra_attrs`, which
allows the user to pass a list of attributes on which to group all
characters.
For instance, passing `extra_attrs=["fontname", "size"]` will not allow
characters with different font names or sizes to become part of the same
word. As a benefit, those resulting word dicts will contain `"fontname"`
and `"size"` attributes — providing a long-requested feature (cf. issue
The same goals can be acheived by just testing the first page, instead
of the entire PDF, trimming approximately 20% off the total test suite's
execution time, although YMMV.
The test, responding to PR #77, was particularly slow and the
accompanying PDF somewhat large. Now that the fixes to the issues
identified in the PR are fairly tightly integrated into pdfplumber, we
can remove the tests with reasonable confidence, speeding up development
time.
Jeremy Singer-Vine