diff --git a/examples/notebooks/utils-nics.ipynb b/examples/notebooks/extract-table-nics.ipynb similarity index 56% rename from examples/notebooks/utils-nics.ipynb rename to examples/notebooks/extract-table-nics.ipynb index 0875e31..e8ad31c 100644 --- a/examples/notebooks/utils-nics.ipynb +++ b/examples/notebooks/extract-table-nics.ipynb @@ -4,16 +4,16 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Demonstration of `pdfplumber.utils`\n", + "# Demonstration of `pdfplumber`'s `extract_table` method.\n", "\n", - "This notebook uses our [example PDF](../pdfs/background-checks.pdf) from the FBI's National Instant Criminal Background Check System to demonstrate `pdfplumber.utils`." + "This notebook uses our [example PDF](../pdfs/background-checks.pdf) from the FBI's National Instant Criminal Background Check System." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "### Import `pdfplumber` and the relevant `utils`" + "### Import `pdfplumber`" ] }, { @@ -24,8 +24,7 @@ }, "outputs": [], "source": [ - "import pdfplumber\n", - "from pdfplumber.utils import within_bbox, extract_columns, collate_chars" + "import pdfplumber" ] }, { @@ -50,7 +49,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### For use in constructing the bounding boxes later, store the the page width" + "### Get the first page" ] }, { @@ -61,14 +60,16 @@ }, "outputs": [], "source": [ - "PDF_WIDTH = pdf.pages[0].width" + "page_1 = pdf.pages[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "### Here are the first three characters stored in the PDF:" + "### Use `.crop` to focus on the main data table\n", + "\n", + "It starts around 80px from the top, and is about 405px tall. To select only these characters, we use `within_bbox`, and pass a bounding box of `(0, 80, PDF_WIDTH, 485)` as the `(x0, top, x1, bottom)` values." ] }, { @@ -77,26 +78,20 @@ "metadata": { "collapsed": false }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "[{'size': 11.414, 'width': 4.642, 'pageid': 1, 'x1': 51.682, 'object_type': 'char', 'top': 67.486, 'text': 'S', 'x0': 47.04, 'upright': True, 'y1': 544.514, 'y0': 533.099, 'height': 11.414, 'doctop': 67.486, 'fontname': 'DCLTEC+Helvetica-Bold', 'adv': 0.667}, {'size': 11.414, 'width': 2.318, 'pageid': 1, 'x1': 53.916, 'object_type': 'char', 'top': 67.486, 'text': 't', 'x0': 51.599, 'upright': True, 'y1': 544.514, 'y0': 533.099, 'height': 11.414, 'doctop': 67.486, 'fontname': 'DCLTEC+Helvetica-Bold', 'adv': 0.333}, {'size': 11.414, 'width': 3.87, 'pageid': 1, 'x1': 57.863, 'object_type': 'char', 'top': 67.486, 'text': 'a', 'x0': 53.993, 'upright': True, 'y1': 544.514, 'y0': 533.099, 'height': 11.414, 'doctop': 67.486, 'fontname': 'DCLTEC+Helvetica-Bold', 'adv': 0.556}]\n" - ] - } - ], + "outputs": [], "source": [ - "print(pdf.chars[:3])" + "table_crop = page_1.crop((0, 80, page_1.width, 485))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "### Use `within_bbox` to focus on the main data table\n", + "### Use `extract_table` to pull the data\n", "\n", - "It starts around 77px from the top, and is about 410px tall. To select only these characters, we use `within_bbox`, and pass a bounding box of `(0, 77, PDF_WIDTH, 485)` as the `(x0, top0, x1, top1)` values." + "- Because the columns are separated by lines, we use `v=\"lines\"`\n", + "- Because the rows are, primarily, separated by gutters, we use `h=\"gutters\"`\n", + "- And because side-by-side characters don't abut one another exactly, we use `x_tolerance=2`." ] }, { @@ -110,42 +105,18 @@ "name": "stdout", "output_type": "stream", "text": [ - "[{'size': 7.667, 'width': 3.842, 'pageid': 1, 'x1': 47.042, 'object_type': 'char', 'y0': 525.799, 'text': 'A', 'x0': 43.2, 'upright': True, 'y1': 533.465, 'top': 78.535, 'height': 7.667, 'doctop': 78.535, 'fontname': 'WEVZII+ArialMT', 'adv': 0.667}, {'size': 7.667, 'width': 1.279, 'pageid': 1, 'x1': 48.079, 'object_type': 'char', 'y0': 525.799, 'text': 'l', 'x0': 46.8, 'upright': True, 'y1': 533.465, 'top': 78.535, 'height': 7.667, 'doctop': 78.535, 'fontname': 'WEVZII+ArialMT', 'adv': 0.222}, {'size': 7.667, 'width': 3.203, 'pageid': 1, 'x1': 51.437, 'object_type': 'char', 'y0': 525.799, 'text': 'a', 'x0': 48.234, 'upright': True, 'y1': 533.465, 'top': 78.535, 'height': 7.667, 'doctop': 78.535, 'fontname': 'WEVZII+ArialMT', 'adv': 0.556}]\n" + "['Alabama', '18,870', '23,022', '22,650', '859', '1,178', '0', '14', '15', '0', '2,179', '2,307', '11', '0', '0', '0', None, None, '13', '14', '0', '3', '2', '0', '71,137']\n", + "['Alaska', '209', '3,062', '3,209', '191', '184', '0', '9', '3', '0', '100', '100', '0', '18', '9', '1', None, None, '0', '0', '0', '0', '0', '0', '7,095']\n", + "['Arizona', '2,303', '12,382', '9,041', '707', '618', '0', '5', '3', '0', '1,273', '648', '4', '76', '8', '1', None, None, '9', '6', '1', '1', '1', '0', '27,087']\n", + "['Arkansas', '3,298', '6,359', '11,611', '168', '376', '0', '12', '6', '1', '922', '2,275', '1', '0', '0', '0', None, None, '6', '12', '1', '0', '0', '0', '25,048']\n", + "['California', '98452', '41181', '35007', '4559', '0', '0', '0', '0', '0', '480', '433', '4', '0', '0', '0', None, None, '0', '0', '0', '0', '0', '0', '180116']\n" ] } ], "source": [ - "table_chars = within_bbox(pdf.chars, (0, 77, PDF_WIDTH, 485))\n", - "print(table_chars[:3])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Use `extract_columns` to divide the characters into rows and columns\n", - "\n", - "Because side-by-side characters don't abut one another exactly, we pass `x_tolerance=2`." - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "metadata": { - "collapsed": false - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "[{0: 'Alabama', 1: '18,870', 2: '23,022', 3: '22,650', 4: '859', 5: '1,178', 6: '0', 7: '14', 8: '15', 9: '0', 10: '2,179', 11: '2,307', 12: '11', 13: '0', 14: '0', 15: '0', 16: '', 17: '', 18: '13', 19: '14', 20: '0', 21: '3', 22: '2', 23: '0', 24: '71,137'}, {0: 'Alaska', 1: '209', 2: '3,062', 3: '3,209', 4: '191', 5: '184', 6: '0', 7: '9', 8: '3', 9: '0', 10: '100', 11: '100', 12: '0', 13: '18', 14: '9', 15: '1', 16: '', 17: '', 18: '0', 19: '0', 20: '0', 21: '0', 22: '0', 23: '0', 24: '7,095'}]\n" - ] - } - ], - "source": [ - "table = extract_columns(table_chars, x_tolerance=2)\n", - "print(table[:2])" + "table = table_crop.extract_table(v=\"lines\", h=\"gutters\", x_tolerance=2)\n", + "for row in table[:5]:\n", + " print(row)" ] }, { @@ -159,7 +130,7 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 6, "metadata": { "collapsed": true }, @@ -196,33 +167,34 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 7, "metadata": { "collapsed": true }, "outputs": [], "source": [ - "def parse_value(k, x):\n", - " if k == 0: return x\n", - " if x == \"\": return None\n", + "def parse_value(i, x):\n", + " if i == 0: return x\n", + " if x == None: return None\n", " return int(x.replace(\",\", \"\"))" ] }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 8, "metadata": { "collapsed": false }, "outputs": [], "source": [ "def parse_row(row):\n", - " return dict((COLUMNS[k], parse_value(k, v)) for k, v in row.items())" + " return dict((COLUMNS[i], parse_value(i, cell))\n", + " for i, cell in enumerate(row))" ] }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 9, "metadata": { "collapsed": false }, @@ -235,12 +207,12 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Here's a sample row:" + "Here's the first row, parsed:" ] }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 10, "metadata": { "collapsed": false }, @@ -248,40 +220,40 @@ { "data": { "text/plain": [ - "{'admin': 1,\n", - " 'handgun': 1745,\n", - " 'long_gun': 2372,\n", - " 'multiple': 104,\n", - " 'other': 87,\n", - " 'permit': 383,\n", - " 'prepawn_handgun': 0,\n", - " 'prepawn_long_gun': 4,\n", + "{'admin': 0,\n", + " 'handgun': 23022,\n", + " 'long_gun': 22650,\n", + " 'multiple': 1178,\n", + " 'other': 859,\n", + " 'permit': 18870,\n", + " 'prepawn_handgun': 14,\n", + " 'prepawn_long_gun': 15,\n", " 'prepawn_other': 0,\n", - " 'private_sale_handgun': 1,\n", - " 'private_sale_long_gun': 2,\n", + " 'private_sale_handgun': 13,\n", + " 'private_sale_long_gun': 14,\n", " 'private_sale_other': 0,\n", - " 'redemption_handgun': 132,\n", - " 'redemption_long_gun': 184,\n", - " 'redemption_other': 0,\n", + " 'redemption_handgun': 2179,\n", + " 'redemption_long_gun': 2307,\n", + " 'redemption_other': 11,\n", " 'rentals_handgun': None,\n", " 'rentals_long_gun': None,\n", - " 'return_to_seller_handgun': 0,\n", + " 'return_to_seller_handgun': 3,\n", " 'return_to_seller_long_gun': 2,\n", " 'return_to_seller_other': 0,\n", " 'returned_handgun': 0,\n", " 'returned_long_gun': 0,\n", " 'returned_other': 0,\n", - " 'state': 'Wyoming',\n", - " 'totals': 5017}" + " 'state': 'Alabama',\n", + " 'totals': 71137}" ] }, - "execution_count": 11, + "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ - "parsed_table[-2]" + "parsed_table[0]" ] }, { @@ -295,7 +267,7 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 11, "metadata": { "collapsed": false }, @@ -322,9 +294,20 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### Use `within_bbox` and `collate_chars` to extract the report month\n", + "### Use `collate_chars` to extract the report month\n", "\n", - "The month of the report is listed in an area 35px to 60px from the top of the page. The code below isolates characters in that space, and then collates their text." + "It looks like the month of the report is listed in an area 35px to 65px from the top of the page. But there's also some other text directly above and below it. So when we crop for that area, we'll use `strict=True` to select only characters (and other objects) that are fully within the crop-box." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "month_crop = page_1.crop((0, 35, page_1.width, 65), strict=True)" ] }, { @@ -333,17 +316,6 @@ "metadata": { "collapsed": false }, - "outputs": [], - "source": [ - "month_chars = within_bbox(pdf.chars, (0, 35, PDF_WIDTH, 60))" - ] - }, - { - "cell_type": "code", - "execution_count": 14, - "metadata": { - "collapsed": false - }, "outputs": [ { "data": { @@ -351,13 +323,14 @@ "'November - 2015'" ] }, - "execution_count": 14, + "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ - "collate_chars(month_chars, x_tolerance=2)" + "month_chars = month_crop.collate_chars(x_tolerance=2, y_tolerance=2)\n", + "month_chars" ] }, {