mirror of
https://github.com/jsvine/pdfplumber.git
synced 2026-08-29 16:40:24 +08:00
Jeremy Singer-Vine
397 lines
11 KiB
Plaintext
397 lines
11 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Demonstration of `pdfplumber.utils`\n",
|
|
"\n",
|
|
"This notebook uses our [example PDF](../pdfs/background-checks.pdf) from the FBI's National Instant Criminal Background Check System to demonstrate `pdfplumber.utils`."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Import `pdfplumber` and the relevant `utils`"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import pdfplumber\n",
|
|
"from pdfplumber.utils import within_bbox, extract_columns, collate_chars"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Load the PDF"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"pdf = pdfplumber.from_path(\"../pdfs/background-checks.pdf\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### For use in constructing the bounding boxes later, store the the page width"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"PDF_WIDTH = pdf.pages[0].width"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Here are the first three characters stored in the PDF:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"[{'size': 11.414, 'width': 4.642, 'pageid': 1, 'x1': 51.682, 'object_type': 'char', 'top': 67.486, 'text': 'S', 'x0': 47.04, 'upright': True, 'y1': 544.514, 'y0': 533.099, 'height': 11.414, 'doctop': 67.486, 'fontname': 'DCLTEC+Helvetica-Bold', 'adv': 0.667}, {'size': 11.414, 'width': 2.318, 'pageid': 1, 'x1': 53.916, 'object_type': 'char', 'top': 67.486, 'text': 't', 'x0': 51.599, 'upright': True, 'y1': 544.514, 'y0': 533.099, 'height': 11.414, 'doctop': 67.486, 'fontname': 'DCLTEC+Helvetica-Bold', 'adv': 0.333}, {'size': 11.414, 'width': 3.87, 'pageid': 1, 'x1': 57.863, 'object_type': 'char', 'top': 67.486, 'text': 'a', 'x0': 53.993, 'upright': True, 'y1': 544.514, 'y0': 533.099, 'height': 11.414, 'doctop': 67.486, 'fontname': 'DCLTEC+Helvetica-Bold', 'adv': 0.556}]\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"print(pdf.chars[:3])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Use `within_bbox` to focus on the main data table\n",
|
|
"\n",
|
|
"It starts around 77px from the top, and is about 410px tall. To select only these characters, we use `within_bbox`, and pass a bounding box of `(0, 77, PDF_WIDTH, 485)` as the `(x0, top0, x1, top1)` values."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"[{'size': 7.667, 'width': 3.842, 'pageid': 1, 'x1': 47.042, 'object_type': 'char', 'y0': 525.799, 'text': 'A', 'x0': 43.2, 'upright': True, 'y1': 533.465, 'top': 78.535, 'height': 7.667, 'doctop': 78.535, 'fontname': 'WEVZII+ArialMT', 'adv': 0.667}, {'size': 7.667, 'width': 1.279, 'pageid': 1, 'x1': 48.079, 'object_type': 'char', 'y0': 525.799, 'text': 'l', 'x0': 46.8, 'upright': True, 'y1': 533.465, 'top': 78.535, 'height': 7.667, 'doctop': 78.535, 'fontname': 'WEVZII+ArialMT', 'adv': 0.222}, {'size': 7.667, 'width': 3.203, 'pageid': 1, 'x1': 51.437, 'object_type': 'char', 'y0': 525.799, 'text': 'a', 'x0': 48.234, 'upright': True, 'y1': 533.465, 'top': 78.535, 'height': 7.667, 'doctop': 78.535, 'fontname': 'WEVZII+ArialMT', 'adv': 0.556}]\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"table_chars = within_bbox(pdf.chars, (0, 77, PDF_WIDTH, 485))\n",
|
|
"print(table_chars[:3])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Use `extract_columns` to divide the characters into rows and columns\n",
|
|
"\n",
|
|
"Because side-by-side characters don't abut one another exactly, we pass `x_tolerance=2`."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"[{0: 'Alabama', 1: '18,870', 2: '23,022', 3: '22,650', 4: '859', 5: '1,178', 6: '0', 7: '14', 8: '15', 9: '0', 10: '2,179', 11: '2,307', 12: '11', 13: '0', 14: '0', 15: '0', 16: '', 17: '', 18: '13', 19: '14', 20: '0', 21: '3', 22: '2', 23: '0', 24: '71,137'}, {0: 'Alaska', 1: '209', 2: '3,062', 3: '3,209', 4: '191', 5: '184', 6: '0', 7: '9', 8: '3', 9: '0', 10: '100', 11: '100', 12: '0', 13: '18', 14: '9', 15: '1', 16: '', 17: '', 18: '0', 19: '0', 20: '0', 21: '0', 22: '0', 23: '0', 24: '7,095'}]\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"table = extract_columns(table_chars, x_tolerance=2)\n",
|
|
"print(table[:2])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Convert keys and values to something more useful\n",
|
|
"\n",
|
|
"The dictionary keys returned by `extract_columns` are simply numbered in order, `0, 1, 2, ...`. Let's add the actual column names, and also convert strings-representing-numbers to the numbers themselves, e.g., `\"18,870\" -> 18870`:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 7,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"COLUMNS = [\n",
|
|
" \"state\",\n",
|
|
" \"permit\",\n",
|
|
" \"handgun\",\n",
|
|
" \"long_gun\",\n",
|
|
" \"other\",\n",
|
|
" \"multiple\",\n",
|
|
" \"admin\",\n",
|
|
" \"prepawn_handgun\",\n",
|
|
" \"prepawn_long_gun\",\n",
|
|
" \"prepawn_other\",\n",
|
|
" \"redemption_handgun\",\n",
|
|
" \"redemption_long_gun\",\n",
|
|
" \"redemption_other\",\n",
|
|
" \"returned_handgun\",\n",
|
|
" \"returned_long_gun\",\n",
|
|
" \"returned_other\",\n",
|
|
" \"rentals_handgun\",\n",
|
|
" \"rentals_long_gun\",\n",
|
|
" \"private_sale_handgun\",\n",
|
|
" \"private_sale_long_gun\",\n",
|
|
" \"private_sale_other\",\n",
|
|
" \"return_to_seller_handgun\",\n",
|
|
" \"return_to_seller_long_gun\",\n",
|
|
" \"return_to_seller_other\",\n",
|
|
" \"totals\"\n",
|
|
"]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 8,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def parse_value(k, x):\n",
|
|
" if k == 0: return x\n",
|
|
" if x == \"\": return None\n",
|
|
" return int(x.replace(\",\", \"\"))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 9,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def parse_row(row):\n",
|
|
" return dict((COLUMNS[k], parse_value(k, v)) for k, v in row.items())"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 10,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"parsed_table = [ parse_row(row) for row in table ]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Here's a sample row:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 11,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"{'admin': 1,\n",
|
|
" 'handgun': 1745,\n",
|
|
" 'long_gun': 2372,\n",
|
|
" 'multiple': 104,\n",
|
|
" 'other': 87,\n",
|
|
" 'permit': 383,\n",
|
|
" 'prepawn_handgun': 0,\n",
|
|
" 'prepawn_long_gun': 4,\n",
|
|
" 'prepawn_other': 0,\n",
|
|
" 'private_sale_handgun': 1,\n",
|
|
" 'private_sale_long_gun': 2,\n",
|
|
" 'private_sale_other': 0,\n",
|
|
" 'redemption_handgun': 132,\n",
|
|
" 'redemption_long_gun': 184,\n",
|
|
" 'redemption_other': 0,\n",
|
|
" 'rentals_handgun': None,\n",
|
|
" 'rentals_long_gun': None,\n",
|
|
" 'return_to_seller_handgun': 0,\n",
|
|
" 'return_to_seller_long_gun': 2,\n",
|
|
" 'return_to_seller_other': 0,\n",
|
|
" 'returned_handgun': 0,\n",
|
|
" 'returned_long_gun': 0,\n",
|
|
" 'returned_other': 0,\n",
|
|
" 'state': 'Wyoming',\n",
|
|
" 'totals': 5017}"
|
|
]
|
|
},
|
|
"execution_count": 11,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"parsed_table[-2]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Sort the data\n",
|
|
"\n",
|
|
"For demonstration purposes, let's list the rows with the highest number of handgun-only background checks:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 12,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Totals: 671,330 handgun-only checks\n",
|
|
"Pennsylvania: 62,752 handgun-only checks\n",
|
|
"Texas: 56,941 handgun-only checks\n",
|
|
"Florida: 50,796 handgun-only checks\n",
|
|
"California: 41,181 handgun-only checks\n",
|
|
"Ohio: 34,878 handgun-only checks\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"for row in list(reversed(sorted(parsed_table, key=lambda x: x[\"handgun\"])))[:6]:\n",
|
|
" print(\"{state}: {handgun:,d} handgun-only checks\".format(**row))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Use `within_bbox` and `collate_chars` to extract the report month\n",
|
|
"\n",
|
|
"The month of the report is listed in an area 35px to 60px from the top of the page. The code below isolates characters in that space, and then collates their text."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 13,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"month_chars = within_bbox(pdf.chars, (0, 35, PDF_WIDTH, 60))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 14,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"'November - 2015'"
|
|
]
|
|
},
|
|
"execution_count": 14,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"collate_chars(month_chars, x_tolerance=2)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"---\n",
|
|
"\n",
|
|
"---\n",
|
|
"\n",
|
|
"---"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.4.3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0
|
|
}
|