... and in the process, deeply rewrite the text extraction code to be
more flexible with regards to reading/writing non top-to-bottom,
left-to-right text.
Major kudos and thanks to @afriedman412 for the initial idea, initial
prototype, and inspiration for these new features.
Breaking change: word["direction"] is no longer {1,-1} but instead
{"ltr","rtl","ttb","btt"}.
Deprecated: Text-extraction parameters `horizontal_ltr` and
`vertical_ttb`.
These are related changes, since the previous approach of using
`PDF.stream.name` to distinguish between on-path and filelike-object PDFs
doesn't work for zipped files (and possibly other filelike-objects that
have a `name` property).
This commit adds convenience methods to repair PDFs on the fly and/or to
write repaired PDFs to disk.
Currently, this does so via Ghostscript using the method we've asked
many users to try by following the instructions at
https://superuser.com/questions/278562/how-can-i-fix-repair-a-corrupted-pdf-file
Now, hopefully, this saves folks a few steps.
Most of the groundwork was already there to add a PDF/Page.curve_edges
property. And, inspired, by
https://github.com/jsvine/pdfplumber/discussions/858 and related issues,
we now include 0/90/180/270-degree oriented curve segments into the
default table-detection strategy. As before, you can still switch to the
"lines_strict" to use only lines defined as such (rather than also using
rect and curve edges).
h/t @xv44586 for the initial inspiration 👍
These new methods return a version of the chars/page with duplicate
chars — those sharing the same text, fontname, size, and positioning
(within `tolerance` x/y) as other characters — removed.
Taking @samkit-jain's smart question about sort-order into account, this
commit reworks `.extract_words`, making the sorting more explicit, and
adding an option to *not* sort, via `use_text_flow=True`, which follows
the original PDF's text flow, rather than presorting all characters top-down,
left-right.
To do so required rethinking the word-delineation logic,
which in turn motivated a shift toward a more object-oriented approach
(especially to keep track of settings without passing them from function
to function). Now `.extract_text`'s logic is encapsulated in a new class,
`TextExtractor`.
The test, responding to PR #77, was particularly slow and the
accompanying PDF somewhat large. Now that the fixes to the issues
identified in the PR are fairly tightly integrated into pdfplumber, we
can remove the tests with reasonable confidence, speeding up development
time.
Moves most of the logic previously in cli.py to convert.py, for usage by
other submodules. Adds Container.to_json and Container.to_csv. Makes
adjustments/fixes to other parts of the library, based on edge-cases
encountered (such as infinite recursion in anntations).
- Fixes `.extract_words`, which had been returning incorrect results when `horizontal_ltr = False`
- Fixes `.resize_object`, which had been failing in various permutations
- Brings utils.py test coverage to 100%
.annos was non-functional, based on a misunderstanding of how
Annotation objects were represented in the PDF object. Also shifts
language from "annos" to "annots" to mirror pdfminer's nomenclature.
See https://github.com/jsvine/pdfplumber/issues/203
Root cause of decimalization bug flagged in that issue was
pdfminer.six's .resolve_all method not recusively resolving items within
tuples. Now we define our own method, with some slight optimization
tweaks.
Jeremy Singer-Vine