This commit adds convenience methods to repair PDFs on the fly and/or to
write repaired PDFs to disk.
Currently, this does so via Ghostscript using the method we've asked
many users to try by following the instructions at
https://superuser.com/questions/278562/how-can-i-fix-repair-a-corrupted-pdf-file
Now, hopefully, this saves folks a few steps.
This commit swaps out Wand (and its non-Python dependencies ImageMagick
and Ghostscript) for pypdfium2 for PageImage rendering. This has some
advantages:
- Less finicky: Wand often caused users problems, due to "MagickWand
shared library not found" and "PolicyError: not authorized `PDF'"
issues. By contrast, pypdfium2 seems (at least at first) to more
self-contained and not require any system-tweaking.
- Faster: pypdfium2 appears to render images more quickly than Wand (see
@cmdlineuser's tests in #899)
- More flexible: pypdfium2 appears to generate images with greater color
depth; by default, pdfplumber quantizes those images so that they
save/display compactly (in fact, with smaller file sizes than the
previous code), this commit also adds parameters to retain all/more of
the original, more detailed colors.
Thanks to @cmdlineuser in #899 for the suggestion.
Inspired by https://github.com/jsvine/pdfplumber/issues/828
The PDF reference allows for "colors" to be defined as a series of
numbers and/or (much less commonly) patterns.
(See p. 288 and section 4.6 here:
https://ghostscript.com/~robin/pdf_reference17.pdf)
This commit separates out the pattern component of colors into their own
attributes, `stroking_pattern` and `non_stroking_pattern` so that they
don't muddle the interpretation of standard colors' tuple-of-numbers
representation.
This commit also adds code that attempts to fetch the `ncs`/`scs` color
space of each object. Due to current limitations of pdfminer.six,
however, the only such color space immediately available is the `ncs`
(non-stroking color space) property of char objects.
Previously, `pdfplumber.Page` had these table-getting methods:
- `.find_tables(...)`
- `.extract_tables(...)`
- `.extract_table(...)`
For consistency/completeness's sake, this commit adds:
- `.find_table(...)`
... which, analogous to `.extract_table(...)`, returns the largest table
on the page.
Indeed, `.extract_table(...)` now uses `.find_table(...)` beneath the
hood.
Thanks to @pdille for the suggestion, here:
https://github.com/jsvine/pdfplumber/discussions/864#discussioncomment-5668209
This commit normalizes the type representation of `stroking_color` and
`non_stroking_color` values. Thanks to @dhdaines for pointing out this
inconsistency.
Previously, `pdfplumber` passed along `pdfminer.six`'s colors without
normalization. Due to quirks in `pdfminer.six`'s color handling, this
meant that those values could be floats, ints, lists, or tuples. This
commit normalizes all color values (when non-None) into n-tuples, where
(val,) represents grayscale colors, (val, val, val) represents RBG, and
(val, val, val, val) represents CMYK colors.
This should solve the consistency issue, although might cause breaking
changes to code that filters for non-tuple values — e.g., `[c for c in
page.chars if c == [1, 0 0]]`. Although breaking changes are unpleasant,
I think the tradeoff for longer-term consistency is worth it.
When using relative=True for a re-crop, pdfplumber was passing the wrong
bounding box to the cropping function. This commit fixes that bug and
also refactors CroppedPage.__init__(...) for clarity and consistency's
sake.
Main edge-case was with `use_text_flow` on text-lines that then
backtracked. But this rewrite also aims to make the logic more explicit
and easier to follow.
As noted in #912, `use_text_flow` was not being handled consistently, as
characters and words were being re-sorted without checking first if this
parameter was set to `True`.
Most of the groundwork was already there to add a PDF/Page.curve_edges
property. And, inspired, by
https://github.com/jsvine/pdfplumber/discussions/858 and related issues,
we now include 0/90/180/270-degree oriented curve segments into the
default table-detection strategy. As before, you can still switch to the
"lines_strict" to use only lines defined as such (rather than also using
rect and curve edges).
Inspired by https://github.com/jsvine/pdfplumber/discussions/852,
it turns out that .search(...) gets us most of the way, already.
Added a few params (`main_group`, `return_groups`, `return_chars`) to
.search(...) to enable this, which also make that method more generally
flexible.
This commit aims to fix two things:
- The semi-crypticness of the previous version of char_begins_new_word
- The inconsistency (vs. the rest of the approach) in how the method
was comparing "top" to "bottom" for interline comparisons, instead of
"top" to "top", as rightly and helpfully pointed out by @bellma-lilly
in https://github.com/jsvine/pdfplumber/discussions/840
Based on the unit tests, this shouldn't change the output of
`pdfplumber` in the vast majority of use cases. It might affect some
output in edge-cases, for which I apologize for any inconvenience and
which I hope is balanced out by this more consistent approach's benefits
in the long run.
Calculating the bounding boxes of the words is, upon reflection and
testing, not necessary. Instead, all we need is the latest character in
the current word.
With regular expressions, patterns made up entirely of optional groups
(e.g., r"(dfsdgfwerw)?") can return match objects that are, effectively,
empty. These were being treated as real search results, and throwing
errors in the process. Now they're being treated as non-results.
Separately but relatedly, whitespace-only searches were throwing errors.
This was due to (a) how PDFs generally represent whitespace (implicitly,
rather than explicit space characters), and (b) how pdfplumber
internally represents those spaces while performing layout analysis.
This caused search results to have no explicit bounding box, throwing
errors. Now, similarly to handling empty search results, we handle
all-whitespace search results by considering them to be non-results.
Pillow 9.5 throws an error when you try to draw a rectangle where x0>x1
or top>bottom. For rects with small width or height but thick strokes,
we ended up hitting that error because we were trying to subtract the
stroke width from the area to be filled.
Update ag-energy-roundup-curves.ipynb
Update the key name 'point' to 'pts' in the example to the latest version
* 💄
* Update ag-energy-roundup-curves.ipynb
💄
Jeremy Singer-Vine