... preferring to composite with a white background instead of removing
the alpha channel. This seems to more reliably produce high-quality
conversions (easier to read, fewer conversion artifacts) than either the
prior or other previous approaches.
The problem had been that `LayoutEngine.calculate(...)` was assuming
that len(char["text"]) would always equal 1, which is not true for
ligatures.
Thanks to @samkit-jain for finding a PDF that raised the error, and to
@bpugnaire for raising issue #683.
First proposed here: https://github.com/jsvine/pdfplumber/issues/201
Adding this feature involved refactoring and re-engineering a good chunk
of the text-layout-extraction code. As part of that, this commit
introduces two new classes, in utils.py: LayoutEngine and TextLayout.
They should be considered provisional, and may change name/approach in
the future.
Added weak tests for a couple of hard-to-test visual debugging methods
in display.py.
Also marked an exception-handling block in pdf.py as `pragma: nocover`
because we don't yet have a PDF to test it with, but the logic there is
simple and straightforward.
A fairly large commit, adding type annotations/hints to the entire
library, and refactoring the library accordingly.
Most of the refactoring changes should have no practical effect on
usage, but several others are notable:
- Added `TableSettings` class, a behind-the-scenes handler for managing
and validating table-extraction settings.
- Renamed the positional argument to `.to_csv(...)` and `.to_json(...)`
from `types` to `object_types`.
- Tweaked the output of `.to_json(...)` so that, if an object type is
not present for a given page, it has no key in the page's object
representation.
- Removed `utils.filter_objects(...)` and move the functionality to
within the `FilteredPage.objects` property calculation, the only part
of the library that used it.
- Removed code that sets `pdfminer.pdftypes.STRICT = True` and
`pdfminer.pdfinterp.STRICT = True`, since that [has now been the
default for a
while](https://github.com/pdfminer/pdfminer.six/commit/9439a3a31a347836aad1c1226168156125d9505f).
Now all tolerance settings have x/y versions as well.
This commit also changes `table.merge_edges(...)` behavior when
`join_tolerance` (and `x`/`y` variants) `<= 0`, so that joining is
attempted regardless, to handle cases of overlapping lines.
Per discussion at https://github.com/jsvine/pdfplumber/discussions/346
and input from @ramcdona, this commit changes pdfplumber's behavior
regarding floating point numbers. Specifically, it removes all
conversion of floats to Decimal objects. This brings several advantages:
- Increased precision (where applicable)
- Decreased code complexity
- Increased performance (~10% speedup on test suite)
- Increased fidelity to `pdfminer.six` output
These seem to outweigh the disadvantages:
- Some tests break (but have been easily fixed) due to increased
precision and/or floating point arithmetic artifacts
- Some users' scripts may also break, if they depend on strict equality
testing, though these *should* also be easily fixable
Because some form of automatic rounding may still be desirable for the
pdfplumber CLI utility, the conversion methods (.to_csv, .to_json) have
been adjusted to accept a `precision` argument.
See pdfminer.six's changelog for details:
https://github.com/pdfminer/pdfminer.six/blob/develop/CHANGELOG.md
... but a key difference is an improvement in how it assigns `line`,
`rect`, and `curve` objects. (Diagonal two-point lines, for instance,
are now `line` objects instead of `curve` objects.)
As a result, this commit also adjusts some of the tests, where the
pre-20211012 versions had been incorrectly assigning lines as `LTCurve`
objects.
This commit adds an `--laparams` flag to the pdfplumber CLI, giving it
more feature parity with the core library. To do so, it makes some
internal changes to `convert.py`, including changing the list of objects
to convert from *a predefined default list* to *all types extracted*.
pdfminer.six's `LTAnno` objects are not PDF annotations (which we
already provide access to via `.annots`, regardless of whether
`laparams` is set), but rather layout annotations. Per pdfminer.six
codebase:
> Note that, while a LTChar object has actual boundaries, LTAnno objects
> does not, as these are "virtual" characters, inserted by a layout
> analyzer according to the relationship between two characters (e.g. a
> space).
Because they have no boundaries, they cause problems for pdfplumber,
which expects bounding-box coordinates for all objects. See, e.g.,
issue #383, which this commit should fix.
This commit reinstates access to higher-level layout objects (such as
`textboxhorizontal`) when `laparams` is passed to
`pdfplumber.open(...)`. Had been removed in `0.5.24` via 1f87898.
Also adds a test for this behavior.
Jeremy Singer-Vine