... and in the process, deeply rewrite the text extraction code to be
more flexible with regards to reading/writing non top-to-bottom,
left-to-right text.
Major kudos and thanks to @afriedman412 for the initial idea, initial
prototype, and inspiration for these new features.
Breaking change: word["direction"] is no longer {1,-1} but instead
{"ltr","rtl","ttb","btt"}.
Deprecated: Text-extraction parameters `horizontal_ltr` and
`vertical_ttb`.
Thanks to PR from @jeremybmerrill, who writes:
> This form-parsing example handles form fields recursively contained within other form fields, removes the incorrect-assumption that field-names are unique and includes the alternate field name in output (which is often a very useful guide to what's in a field).
> the prior form-parsing example used the field-name ("T" entry) as the key in the form_data dict, implicitly assuming that the field name is globally unique within a document. That's not a correct assumption; nested field names are often simply a numeric index like 1 or 0. The prior example also entirely ignored the TU entry alternate field name.
This commit swaps out Wand (and its non-Python dependencies ImageMagick
and Ghostscript) for pypdfium2 for PageImage rendering. This has some
advantages:
- Less finicky: Wand often caused users problems, due to "MagickWand
shared library not found" and "PolicyError: not authorized `PDF'"
issues. By contrast, pypdfium2 seems (at least at first) to more
self-contained and not require any system-tweaking.
- Faster: pypdfium2 appears to render images more quickly than Wand (see
@cmdlineuser's tests in #899)
- More flexible: pypdfium2 appears to generate images with greater color
depth; by default, pdfplumber quantizes those images so that they
save/display compactly (in fact, with smaller file sizes than the
previous code), this commit also adds parameters to retain all/more of
the original, more detailed colors.
Thanks to @cmdlineuser in #899 for the suggestion.
Previously, `pdfplumber.Page` had these table-getting methods:
- `.find_tables(...)`
- `.extract_tables(...)`
- `.extract_table(...)`
For consistency/completeness's sake, this commit adds:
- `.find_table(...)`
... which, analogous to `.extract_table(...)`, returns the largest table
on the page.
Indeed, `.extract_table(...)` now uses `.find_table(...)` beneath the
hood.
Thanks to @pdille for the suggestion, here:
https://github.com/jsvine/pdfplumber/discussions/864#discussioncomment-5668209
This commit normalizes the type representation of `stroking_color` and
`non_stroking_color` values. Thanks to @dhdaines for pointing out this
inconsistency.
Previously, `pdfplumber` passed along `pdfminer.six`'s colors without
normalization. Due to quirks in `pdfminer.six`'s color handling, this
meant that those values could be floats, ints, lists, or tuples. This
commit normalizes all color values (when non-None) into n-tuples, where
(val,) represents grayscale colors, (val, val, val) represents RBG, and
(val, val, val, val) represents CMYK colors.
This should solve the consistency issue, although might cause breaking
changes to code that filters for non-tuple values — e.g., `[c for c in
page.chars if c == [1, 0 0]]`. Although breaking changes are unpleasant,
I think the tradeoff for longer-term consistency is worth it.
Most of the groundwork was already there to add a PDF/Page.curve_edges
property. And, inspired, by
https://github.com/jsvine/pdfplumber/discussions/858 and related issues,
we now include 0/90/180/270-degree oriented curve segments into the
default table-detection strategy. As before, you can still switch to the
"lines_strict" to use only lines defined as such (rather than also using
rect and curve edges).
Inspired by https://github.com/jsvine/pdfplumber/discussions/852,
it turns out that .search(...) gets us most of the way, already.
Added a few params (`main_group`, `return_groups`, `return_chars`) to
.search(...) to enable this, which also make that method more generally
flexible.
With regular expressions, patterns made up entirely of optional groups
(e.g., r"(dfsdgfwerw)?") can return match objects that are, effectively,
empty. These were being treated as real search results, and throwing
errors in the process. Now they're being treated as non-results.
Separately but relatedly, whitespace-only searches were throwing errors.
This was due to (a) how PDFs generally represent whitespace (implicitly,
rather than explicit space characters), and (b) how pdfplumber
internally represents those spaces while performing layout analysis.
This caused search results to have no explicit bounding box, throwing
errors. Now, similarly to handling empty search results, we handle
all-whitespace search results by considering them to be non-results.
Update ag-energy-roundup-curves.ipynb
Update the key name 'point' to 'pts' in the example to the latest version
* 💄
* Update ag-energy-roundup-curves.ipynb
💄
Jeremy Singer-Vine