.. fulmar documentation master file, created by
sphinx-quickstart on Tue Aug 2 14:19:45 2016.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Fulmar is a distributed crawler system. By using non-blocking network I/O,
Fulmar can handle hundreds of open connections at the same time. You can
extractthe data you need from websites. In a fast, simple, yet extensible way.
* `Source (github) <https: github.com="" tylderen="" fulmar="">`_
* `Wiki <https: github.com="" tylderen="" fulmar="" wiki="" links="">`_
Here is a simple example::
from fulmar.base_spider import BaseSpider
logger = logging.getLogger(__name__)
def detail_page(self, response):
page_lxml = response.page_lxml
except Exception as e:
You can save above code in a new file called `baidu_spider.py` and run in console::
fulmar start_project baidu_spider.py
If you have installed `redis`, you will get::
Successfully start the project, project name: "baidu_spider".
Finally, start Fulmar::
pip install fulmar
Fulmar is listed in `PyPI <http: pypi.python.org="" pypi="" fulmar="">`_ and
can be installed with ``pip`` or ``easy_install``. Note that the
source distribution includes demo applications that are not present
when Tornado is installed in this way, so you may wish to download a
copy of the source tarball as well.
**Manual installation**: Download tarball, then:
tar xvzf fulmar-|version|.tar.gz
python setup.py build
sudo python setup.py install
The Fulmar source code is `hosted on GitHub
<https: github.com="" tylderen="" fulmar="">`_.
**Prerequisites**: Fulmar runs on Python 2.7, and 3.3+
For Python 2, version 2.7.9 or newer is *strongly*
recommended for the improved SSL support.
This documentation is also available in `PDF and Epub formats
<https: readthedocs.org="" projects="" fulmar="" downloads=""/>`_.
TODO: Brief introduction on what you do with files - including link to relevant help section.