Inspired by https://github.com/jsvine/pdfplumber/discussions/852,
it turns out that .search(...) gets us most of the way, already.
Added a few params (`main_group`, `return_groups`, `return_chars`) to
.search(...) to enable this, which also make that method more generally
flexible.
Calculating the bounding boxes of the words is, upon reflection and
testing, not necessary. Instead, all we need is the latest character in
the current word.
With regular expressions, patterns made up entirely of optional groups
(e.g., r"(dfsdgfwerw)?") can return match objects that are, effectively,
empty. These were being treated as real search results, and throwing
errors in the process. Now they're being treated as non-results.
Separately but relatedly, whitespace-only searches were throwing errors.
This was due to (a) how PDFs generally represent whitespace (implicitly,
rather than explicit space characters), and (b) how pdfplumber
internally represents those spaces while performing layout analysis.
This caused search results to have no explicit bounding box, throwing
errors. Now, similarly to handling empty search results, we handle
all-whitespace search results by considering them to be non-results.
First proposed here: https://github.com/jsvine/pdfplumber/issues/201
Adding this feature involved refactoring and re-engineering a good chunk
of the text-layout-extraction code. As part of that, this commit
introduces two new classes, in utils.py: LayoutEngine and TextLayout.
They should be considered provisional, and may change name/approach in
the future.
A fairly large commit, adding type annotations/hints to the entire
library, and refactoring the library accordingly.
Most of the refactoring changes should have no practical effect on
usage, but several others are notable:
- Added `TableSettings` class, a behind-the-scenes handler for managing
and validating table-extraction settings.
- Renamed the positional argument to `.to_csv(...)` and `.to_json(...)`
from `types` to `object_types`.
- Tweaked the output of `.to_json(...)` so that, if an object type is
not present for a given page, it has no key in the page's object
representation.
- Removed `utils.filter_objects(...)` and move the functionality to
within the `FilteredPage.objects` property calculation, the only part
of the library that used it.
- Removed code that sets `pdfminer.pdftypes.STRICT = True` and
`pdfminer.pdfinterp.STRICT = True`, since that [has now been the
default for a
while](https://github.com/pdfminer/pdfminer.six/commit/9439a3a31a347836aad1c1226168156125d9505f).
Per discussion at https://github.com/jsvine/pdfplumber/discussions/346
and input from @ramcdona, this commit changes pdfplumber's behavior
regarding floating point numbers. Specifically, it removes all
conversion of floats to Decimal objects. This brings several advantages:
- Increased precision (where applicable)
- Decreased code complexity
- Increased performance (~10% speedup on test suite)
- Increased fidelity to `pdfminer.six` output
These seem to outweigh the disadvantages:
- Some tests break (but have been easily fixed) due to increased
precision and/or floating point arithmetic artifacts
- Some users' scripts may also break, if they depend on strict equality
testing, though these *should* also be easily fixable
Because some form of automatic rounding may still be desirable for the
pdfplumber CLI utility, the conversion methods (.to_csv, .to_json) have
been adjusted to accept a `precision` argument.
"direction" == 1 when upright words go LTR and non-upright words go TTB,
and -1 when the opposite is true. This attribute should help in reusing
the results of `.extract_words` in other functions, so that the user
knows how the extracted word corresponds to the word's layout on the
page.
Taking @samkit-jain's smart question about sort-order into account, this
commit reworks `.extract_words`, making the sorting more explicit, and
adding an option to *not* sort, via `use_text_flow=True`, which follows
the original PDF's text flow, rather than presorting all characters top-down,
left-right.
To do so required rethinking the word-delineation logic,
which in turn motivated a shift toward a more object-oriented approach
(especially to keep track of settings without passing them from function
to function). Now `.extract_text`'s logic is encapsulated in a new class,
`TextExtractor`.
This commit refactors and hopefully makes clearer the logic in
utils.extract_words. It also adds a new parameter, `extra_attrs`, which
allows the user to pass a list of attributes on which to group all
characters.
For instance, passing `extra_attrs=["fontname", "size"]` will not allow
characters with different font names or sizes to become part of the same
word. As a benefit, those resulting word dicts will contain `"fontname"`
and `"size"` attributes — providing a long-requested feature (cf. issue
Moves most of the logic previously in cli.py to convert.py, for usage by
other submodules. Adds Container.to_json and Container.to_csv. Makes
adjustments/fixes to other parts of the library, based on edge-cases
encountered (such as infinite recursion in anntations).
- Fixes `.extract_words`, which had been returning incorrect results when `horizontal_ltr = False`
- Fixes `.resize_object`, which had been failing in various permutations
- Brings utils.py test coverage to 100%
Jeremy Singer-Vine