mirror of
https://github.com/jsvine/pdfplumber.git
synced 2026-08-29 16:40:24 +08:00
Jeremy Singer-Vine
34197f7d6d
Enables idiomatic-ish treatment, e.g.:
with pdfplumber.open(path) as pdf:
[do things]
370 lines
9.2 KiB
Plaintext
370 lines
9.2 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Demonstration of `pdfplumber`'s `extract_table` method.\n",
|
|
"\n",
|
|
"This notebook uses our [example PDF](../pdfs/background-checks.pdf) from the FBI's National Instant Criminal Background Check System."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Import `pdfplumber`"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import pdfplumber"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Load the PDF"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"pdf = pdfplumber.open(\"../pdfs/background-checks.pdf\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Get the first page"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"page_1 = pdf.pages[0]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Use `.crop` to focus on the main data table\n",
|
|
"\n",
|
|
"It starts around 80px from the top, and is about 405px tall. To select only these characters, we use `within_bbox`, and pass a bounding box of `(0, 80, PDF_WIDTH, 485)` as the `(x0, top, x1, bottom)` values."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"table_crop = page_1.crop((0, 80, page_1.width, 485))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Use `extract_table` to pull the data\n",
|
|
"\n",
|
|
"- Because the columns are separated by lines, we use `v=\"lines\"`\n",
|
|
"- Because the rows are, primarily, separated by gutters, we use `h=\"gutters\"`\n",
|
|
"- And because side-by-side characters don't abut one another exactly, we use `x_tolerance=2`."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"['Alabama', '18,870', '23,022', '22,650', '859', '1,178', '0', '14', '15', '0', '2,179', '2,307', '11', '0', '0', '0', None, None, '13', '14', '0', '3', '2', '0', '71,137']\n",
|
|
"['Alaska', '209', '3,062', '3,209', '191', '184', '0', '9', '3', '0', '100', '100', '0', '18', '9', '1', None, None, '0', '0', '0', '0', '0', '0', '7,095']\n",
|
|
"['Arizona', '2,303', '12,382', '9,041', '707', '618', '0', '5', '3', '0', '1,273', '648', '4', '76', '8', '1', None, None, '9', '6', '1', '1', '1', '0', '27,087']\n",
|
|
"['Arkansas', '3,298', '6,359', '11,611', '168', '376', '0', '12', '6', '1', '922', '2,275', '1', '0', '0', '0', None, None, '6', '12', '1', '0', '0', '0', '25,048']\n",
|
|
"['California', '98452', '41181', '35007', '4559', '0', '0', '0', '0', '0', '480', '433', '4', '0', '0', '0', None, None, '0', '0', '0', '0', '0', '0', '180116']\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"table = table_crop.extract_table(v=\"lines\", h=\"gutters\", x_tolerance=2)\n",
|
|
"for row in table[:5]:\n",
|
|
" print(row)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Convert keys and values to something more useful\n",
|
|
"\n",
|
|
"The dictionary keys returned by `extract_columns` are simply numbered in order, `0, 1, 2, ...`. Let's add the actual column names, and also convert strings-representing-numbers to the numbers themselves, e.g., `\"18,870\" -> 18870`:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"COLUMNS = [\n",
|
|
" \"state\",\n",
|
|
" \"permit\",\n",
|
|
" \"handgun\",\n",
|
|
" \"long_gun\",\n",
|
|
" \"other\",\n",
|
|
" \"multiple\",\n",
|
|
" \"admin\",\n",
|
|
" \"prepawn_handgun\",\n",
|
|
" \"prepawn_long_gun\",\n",
|
|
" \"prepawn_other\",\n",
|
|
" \"redemption_handgun\",\n",
|
|
" \"redemption_long_gun\",\n",
|
|
" \"redemption_other\",\n",
|
|
" \"returned_handgun\",\n",
|
|
" \"returned_long_gun\",\n",
|
|
" \"returned_other\",\n",
|
|
" \"rentals_handgun\",\n",
|
|
" \"rentals_long_gun\",\n",
|
|
" \"private_sale_handgun\",\n",
|
|
" \"private_sale_long_gun\",\n",
|
|
" \"private_sale_other\",\n",
|
|
" \"return_to_seller_handgun\",\n",
|
|
" \"return_to_seller_long_gun\",\n",
|
|
" \"return_to_seller_other\",\n",
|
|
" \"totals\"\n",
|
|
"]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 7,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def parse_value(i, x):\n",
|
|
" if i == 0: return x\n",
|
|
" if x == None: return None\n",
|
|
" return int(x.replace(\",\", \"\"))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 8,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def parse_row(row):\n",
|
|
" return dict((COLUMNS[i], parse_value(i, cell))\n",
|
|
" for i, cell in enumerate(row))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 9,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"parsed_table = [ parse_row(row) for row in table ]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Here's the first row, parsed:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 10,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"{'admin': 0,\n",
|
|
" 'handgun': 23022,\n",
|
|
" 'long_gun': 22650,\n",
|
|
" 'multiple': 1178,\n",
|
|
" 'other': 859,\n",
|
|
" 'permit': 18870,\n",
|
|
" 'prepawn_handgun': 14,\n",
|
|
" 'prepawn_long_gun': 15,\n",
|
|
" 'prepawn_other': 0,\n",
|
|
" 'private_sale_handgun': 13,\n",
|
|
" 'private_sale_long_gun': 14,\n",
|
|
" 'private_sale_other': 0,\n",
|
|
" 'redemption_handgun': 2179,\n",
|
|
" 'redemption_long_gun': 2307,\n",
|
|
" 'redemption_other': 11,\n",
|
|
" 'rentals_handgun': None,\n",
|
|
" 'rentals_long_gun': None,\n",
|
|
" 'return_to_seller_handgun': 3,\n",
|
|
" 'return_to_seller_long_gun': 2,\n",
|
|
" 'return_to_seller_other': 0,\n",
|
|
" 'returned_handgun': 0,\n",
|
|
" 'returned_long_gun': 0,\n",
|
|
" 'returned_other': 0,\n",
|
|
" 'state': 'Alabama',\n",
|
|
" 'totals': 71137}"
|
|
]
|
|
},
|
|
"execution_count": 10,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"parsed_table[0]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Sort the data\n",
|
|
"\n",
|
|
"For demonstration purposes, let's list the rows with the highest number of handgun-only background checks:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 11,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Totals: 671,330 handgun-only checks\n",
|
|
"Pennsylvania: 62,752 handgun-only checks\n",
|
|
"Texas: 56,941 handgun-only checks\n",
|
|
"Florida: 50,796 handgun-only checks\n",
|
|
"California: 41,181 handgun-only checks\n",
|
|
"Ohio: 34,878 handgun-only checks\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"for row in list(reversed(sorted(parsed_table, key=lambda x: x[\"handgun\"])))[:6]:\n",
|
|
" print(\"{state}: {handgun:,d} handgun-only checks\".format(**row))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Use `extract_text` to extract the report month\n",
|
|
"\n",
|
|
"It looks like the month of the report is listed in an area 35px to 65px from the top of the page. But there's also some other text directly above and below it. So when we crop for that area, we'll use `strict=True` to select only characters (and other objects) that are fully within the crop-box."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 12,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"month_crop = page_1.crop((0, 35, page_1.width, 65), strict=True)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 13,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"'November - 2015'"
|
|
]
|
|
},
|
|
"execution_count": 13,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"month_chars = month_crop.extract_text(x_tolerance=2, y_tolerance=2)\n",
|
|
"month_chars"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"---\n",
|
|
"\n",
|
|
"---\n",
|
|
"\n",
|
|
"---"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.4.3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0
|
|
}
|