h/t @xv44586 for the initial inspiration 👍
These new methods return a version of the chars/page with duplicate
chars — those sharing the same text, fontname, size, and positioning
(within `tolerance` x/y) as other characters — removed.
"direction" == 1 when upright words go LTR and non-upright words go TTB,
and -1 when the opposite is true. This attribute should help in reusing
the results of `.extract_words` in other functions, so that the user
knows how the extracted word corresponds to the word's layout on the
page.
Taking @samkit-jain's smart question about sort-order into account, this
commit reworks `.extract_words`, making the sorting more explicit, and
adding an option to *not* sort, via `use_text_flow=True`, which follows
the original PDF's text flow, rather than presorting all characters top-down,
left-right.
To do so required rethinking the word-delineation logic,
which in turn motivated a shift toward a more object-oriented approach
(especially to keep track of settings without passing them from function
to function). Now `.extract_text`'s logic is encapsulated in a new class,
`TextExtractor`.
This commit refactors and hopefully makes clearer the logic in
utils.extract_words. It also adds a new parameter, `extra_attrs`, which
allows the user to pass a list of attributes on which to group all
characters.
For instance, passing `extra_attrs=["fontname", "size"]` will not allow
characters with different font names or sizes to become part of the same
word. As a benefit, those resulting word dicts will contain `"fontname"`
and `"size"` attributes — providing a long-requested feature (cf. issue
The same goals can be acheived by just testing the first page, instead
of the entire PDF, trimming approximately 20% off the total test suite's
execution time, although YMMV.
The test, responding to PR #77, was particularly slow and the
accompanying PDF somewhat large. Now that the fixes to the issues
identified in the PR are fairly tightly integrated into pdfplumber, we
can remove the tests with reasonable confidence, speeding up development
time.
This commit reworks the Page.process_objects method to be less complex
and more readable. Many of the changes stem from a realization that
pdfminer.six already stores most object attributes in the desired type;
we only need to convert those requiring decimalization.
This results in one notable change to the output, which is that
`upright` is no longer converted from a bool to an int. The necessary
downstream changes are reflected in utils.extract_words and convert.py.
Overall the changes result in no substantial change to performance; it
is possible that the code runs ever-so-slightly faster, but to a barely
detectable degree.
Samkit Jain