Skip to main content
Join the official 2019 Python Developers SurveyStart the survey!

A tipical crawler, returns a list of urls

Project description

# Copyright (C) 2011 Diego Pardilla Mata
#
# This file is part of SpiderBOY.
#
# SpiderBOY is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.

This is the SpiderBoy crawler.

It has (among others) the following major features

Search URL's from another URL and with a depth indicated

Installation:

See INSTALL file for information on how to install SpiderBOY

Starting SpiderBOY:

run from the command line: `spiderboy.py [-h] [-n NUMBER_OF_LEVELS] [url]'
run `spiderboy.py -h' for more startup options

More information:

http://code.sidelab.es/projects/diegopardilla/wiki

Written by:

see AUTHORS for more information

Reporting Bugs:

Please report bugs to our mailing list or our bugtracker:
munikes@members.fsf.org
pardilla@members.fsf.org
http://code.sidelab.es/projects/diegopardilla/issues/new

Contacting us:

http://packages.python.org/SpiderBOY (homepage)
munikes@members.fsf.org (mUniKeS)
pardilla@members.fsf.org (Diego Pardilla)

Project details


Release history Release notifications

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for SpiderBOY, version 1.0.0
Filename, size File type Python version Upload date Hashes
Filename, size SpiderBOY-1.0.0.linux-x86_64.exe (70.3 kB) File type Windows Installer Python version any Upload date Hashes View hashes
Filename, size SpiderBOY-1.0.0-py2.7.egg (8.5 kB) File type Egg Python version 2.7 Upload date Hashes View hashes
Filename, size SpiderBOY-1.0.0.tar.gz (4.6 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page