Data scraper infrastructure for OpenBlock (hyperlocal news for Django)
Code to help write scripts that import/crawl/parse data from the web into ebpub, as well as extract (US) street addresses from (English) text.
This package is part of OpenBlock. Originally developed for EveryBlock.com.
Problems can be reported to the issue tracker.
Discussion is on the ebcode google group or the #openblock channel on freenode.
Do not just try to easy_install or pip install ebdata. It has a lot of specific dependencies which can’t/shouldn’t be captured by setup.py.
Instead, see the full documentation at http://openblockproject.org/docs/install/index.html which includes links to pip requirements files and instructions on preparing your system.
OpenBlock is a web application and RESTful service that allows users to browse and search their local area for “hyper-local news” - to see what’s going on recently in the immediate geographic area.
For installation instructions and other documentation, see http://openblockproject.org/docs/ (or the .rst files in the docs/ directory).
For help, you can try the ebcode group: http://groups.google.com/group/ebcode or look for us in the #openblock IRC channel on irc.freenode.net.
OpenBlock began life as the open-source code released by Everyblock.com in June 2009. Originally created by Adrian Holovaty and the Everyblock team, it is now developed as an open-source (GPL) project by OpenPlans, at http://openblockproject.org.
Funding for the initial creation of Everyblock and the ongoing development of OpenBlock has been provided by the Knight Foundation (http://www.knightfoundation.org/).
As usual, install all dependencies, eg if you are upgrading a source checkout:
pip install -r ebpub/requirements.txt pip install -e ebpub pip install -r ebdata/requirements.txt pip install -e ebdata pip install -r obadmin/requirements.txt pip install -e obadmin pip install -r obdemo/requirements.txt pip install -e obdemo
As usual, sync and migrate the database:
django-admin.py syncdb django-admin.py migrate
In this release, there are some changes to the Block model and the import script, which combine to give you better search results and geocoding if you reload your blocks data. To do so:
The website should remain operational during this process, though some pages may briefly give errors during the process, and URLs for highways will change to include the prefix.
Too many to mention, but here are some:
places. Ticket #253
user_content. Ticket #211