Skip to main content

ScrAPD

https://badge.fury.io/py/scrapd.svg https://circleci.com/gh/scrapd/scrapd.svg?style=svg https://coveralls.io/repos/github/scrapd/scrapd/badge.svg?branch=master

Extract data from APD news site.

ScrAPD is a small utility designed to help organizations retrieving traffic fatality data in a friendly manner.

Installation

ScrAPD requires Python 3.7+ to work.

pip install scrapd

Quickstart

Collect all the data as CSV:

scrapd --format csv

By default, scrapd does not display anything until it is done collecting the data. If you want to get some feedback about the process, you can enable logging, by adding the -v to the command you want to use. Multiple -v options increase the verbosity. The maximum is 3 (-vvv):

scrapd -v --format csv

To save the results to a file, use the shell redirection:

scrapd -v --format csv > results.csv

Examples

Retrieve the traffic fatalities that happened between January 15th 2019 and January 18th 2019, and output the results in json:

scrapd --from "Jan 15 2019" --to "Jan 18 2019" --format json

[
  {
    "case": "19-0161105",
    "crash": 2,
    "date": "2019-01-16",
    "fatalities": [
      {
        "age": 58,
        "dob": "1960-02-15",
        "ethnicity": "White",
        "first": "Ann",
        "gender": "Female",
        "generation": "",
        "last": "Bottenfield-Seago",
        "middle": ""
      }
    ],
    "latitude": 0.0,
    "link": "http://austintexas.gov/news/traffic-fatality-2-3",
    "location": "West William Cannon Drive and Ridge Oak Road",
    "longitude": 0.0,
    "notes": "The preliminary investigation shows that the grey, 2003 Volkwagen Jetta being driven by Ann Bottenfield-Seago failed to yield at a stop sign while attempting to turn westbound on to West William Cannon Drive from Ridge Oak Road. The Jetta collided with a black, 2017 Chevrolet truck that was eastbound in the inside lane of West William Cannon Drive. Bottenfield-Seago was pronounced deceased at the scene. The passenger in the Jetta and the driver of the truck were both transported to a local hospital with non-life threatening injuries. No charges are expected to be filed.",
    "time": "15:42:00"
  },
  {
    "case": "19-0150158",
    "crash": 1,
    "date": "2019-01-15",
    "fatalities": [
      {
        "age": 31,
        "dob": "1987-07-09",
        "ethnicity": "White",
        "first": "David",
        "gender": "Male",
        "generation": "",
        "last": "Sell",
        "middle": "Hilburn"
      }
    ],
    "latitude": 0.0,
    "link": "http://austintexas.gov/news/traffic-fatality-1-4",
    "location": "10500 block of N IH 35 SB",
    "longitude": 0.0,
    "notes": "The preliminary investigation shows that a 2000 Peterbilt semi truck was travelling southbound in the center lane on IH 35 when it struck pedestrian David Sell. The driver stopped as soon as it was possible to do so and remained on scene. He reported not seeing the pedestrian prior to impact given that it was still dark at the time of the crash. Sell was pronounced deceased at the scene at 6:24 a.m. No charges are expected to be filed.",
    "time": "06:20:00"
  }
]

Do the same research but output as CSV:

scrapd --from "Jan 15 2019" --to "Jan 18 2019" --format csv


crash,case,date,time,location,first,middle,last,generation,ethnicity,gender,dob,age,link,notes
2,19-0161105,2019-01-16,15:42:00,West William Cannon Drive and Ridge Oak Road,Ann,,Bottenfield-Seago,,White,Female,1960-02-15,58,http://austintexas.gov/news/traffic-fatality-2-3,"The preliminary investigation shows that the grey, 2003 Volkwagen Jetta being driven by Ann Bottenfield-Seago failed to yield at a stop sign while attempting to turn westbound on to West William Cannon Drive from Ridge Oak Road. The Jetta collided with a black, 2017 Chevrolet truck that was eastbound in the inside lane of West William Cannon Drive. Bottenfield-Seago was pronounced deceased at the scene. The passenger in the Jetta and the driver of the truck were both transported to a local hospital with non-life threatening injuries. No charges are expected to be filed."
1,19-0150158,2019-01-15,06:20:00,10500 block of N IH 35 SB,David,Hilburn,Sell,,White,Male,1987-07-09,31,http://austintexas.gov/news/traffic-fatality-1-4,The preliminary investigation shows that a 2000 Peterbilt semi truck was travelling southbound in the center lane on IH 35 when it struck pedestrian David Sell. The driver stopped as soon as it was possible to do so and remained on scene. He reported not seeing the pedestrian prior to impact given that it was still dark at the time of the crash. Sell was pronounced deceased at the scene at 6:24 a.m. No charges are expected to be filed.

Retrieve all the traffic fatalities from 2019 (as of Jan 20th 2019) in json, and enabling the logging to follow the progress of the process:

scrapd -v --from "1 1 2019" --to "Jan 20 2019" --format json

Fetching page 1...
Fetching page 2...
Fetching page 3...
Fetching page 4...
Fetching page 5...
Fetching page 6...
Fetching page 7...
Fetching page 8...
Fetching page 9...
Fetching page 10...
Fetching page 11...
Total: 3
[
  {
    "case": "19-0081623",
    "crash": 3,
    "date": "2019-01-08",
    "fatalities": [
      {
        "age": 15,
        "dob": "2003-02-18",
        "ethnicity": "Hispanic",
        "first": "Jesus",
        "gender": "Male",
        "generation": "",
        "last": "Servantez",
        "middle": ""
      }
    ],
    "latitude": 0.0,
    "link": "http://austintexas.gov/news/traffic-fatality-3-4",
    "location": "3600 block of South Capital of Texas Highway SB",
    "longitude": 0.0,
    "notes": "The preliminary investigation shows that the driver of a silver, 2018 KIA was traveling in the center lane of the 3600 block of South Capital of Texas Highway SB when the car collided with Jesus Servantez, a pedestrian in the roadway. The driver remained at the scene and told investigators that he did not see Servantez prior to the crash.\n\n\tJesus Servantez was transported to Saint David\u2019s South Austin Medical Center where he succumbed to his injuries on January 21, 2019. No charges are expected to be filed.",
    "time": "21:37:00"
  },
  {
    "case": "19-0161105",
    "crash": 2,
    "date": "2019-01-16",
    "fatalities": [
      {
        "age": 58,
        "dob": "1960-02-15",
        "ethnicity": "White",
        "first": "Ann",
        "gender": "Female",
        "generation": "",
        "last": "Bottenfield-Seago",
        "middle": ""
      }
    ],
    "latitude": 0.0,
    "link": "http://austintexas.gov/news/traffic-fatality-2-3",
    "location": "West William Cannon Drive and Ridge Oak Road",
    "longitude": 0.0,
    "notes": "The preliminary investigation shows that the grey, 2003 Volkwagen Jetta being driven by Ann Bottenfield-Seago failed to yield at a stop sign while attempting to turn westbound on to West William Cannon Drive from Ridge Oak Road. The Jetta collided with a black, 2017 Chevrolet truck that was eastbound in the inside lane of West William Cannon Drive. Bottenfield-Seago was pronounced deceased at the scene. The passenger in the Jetta and the driver of the truck were both transported to a local hospital with non-life threatening injuries. No charges are expected to be filed.",
    "time": "15:42:00"
  },
  {
    "case": "19-0150158",
    "crash": 1,
    "date": "2019-01-15",
    "fatalities": [
      {
        "age": 31,
        "dob": "1987-07-09",
        "ethnicity": "White",
        "first": "David",
        "gender": "Male",
        "generation": "",
        "last": "Sell",
        "middle": "Hilburn"
      }
    ],
    "latitude": 0.0,
    "link": "http://austintexas.gov/news/traffic-fatality-1-4",
    "location": "10500 block of N IH 35 SB",
    "longitude": 0.0,
    "notes": "The preliminary investigation shows that a 2000 Peterbilt semi truck was travelling southbound in the center lane on IH 35 when it struck pedestrian David Sell. The driver stopped as soon as it was possible to do so and remained on scene. He reported not seeing the pedestrian prior to impact given that it was still dark at the time of the crash. Sell was pronounced deceased at the scene at 6:24 a.m. No charges are expected to be filed.",
    "time": "06:20:00"
  }
]

The “Case” field is assigned by the Austin Police Department, while the “ID” field is a key assigned by ScrAPD to label a separate record for each deceased person.

Speed and accuracy

ScrAPD executes all the requests in an asynchronous manner. As a result it goes very fast.

It parses the information using both the text of the report itself and the Twitter tweet stored in the page metadata. Combining these two methods provides a high degree of confidence in the parsing and allows us to reach more than 95% of success rate.

Some statistics:

  • 99% of entries detected

  • 95% of entries correctly parsed

    • 5% of entries failed the parsing only partially

    • 0 complete failure

  • processing time: ~2m00s for ~140 entries

Who uses ScrAPD?

The Austin Pedestrian Advisory Council used ScrAPD to compile a detailed presentation of the status of the traffic deaths in Austin, TX:

The following organizations also use ScrAPD:

See more at http://docs.scrapd.org/who-is-using-us/

Metadata

Release files for scrapd 3.1.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distribution (wheel)

Table of built distributions (wheels) for scrapd 3.1.2
File Interpreter ABI Platform Actions
scrapd-3.1.2-py3-none-any.whl Python 3 none any Details for scrapd-3.1.2-py3-none-any.whl

Release files / scrapd-3.1.2-py3-none-any.whl

Download URL scrapd-3.1.2-py3-none-any.whl
Size 24.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
286a40b4ca860cd2eca26642195f24025af2012d6a17081bb7b86a1744f57ba0
BLAKE2b-256 checksum
How to use checksums
9b1f1c5711b5ce7fed3248c459095bf790363a8bcbd1645b16145cce3128d2dd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/2.0.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.1.0 requests-toolbelt/0.9.1 tqdm/4.47.0 CPython/3.7.3

Security for scrapd 3.1.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page