Lightweight python package for scraping and saving webpages and websites to local storage.
PyWebCopy © 5
Created By : Raja Tomar
License : MIT
Web Scraping and Saving Complete webpages and websites with python.
Web scraping and archiving tool written in Python
Archive any online website and its assets, css, js and
images for offilne reading, storage or whatever reasons.
It's easy with
Why it's great? because it -
- have a single-function basic usages
- lots of configuration for many custom needs
- provides several scraping packages in one Objects (thanks to their original owners)
Email me at
email@example.com of any query :)
pywebcopy is available on PyPi and is easily installable using
pip install pywebcopy
You are ready to go. Read the tutorials below to get started.
You should always check if the pywebcopy is installed successfully.
>>> import pywebcopy >>> pywebcopy.__version___ 5.x
Your version may be different, now you can continue the tutorial.
1.2 Basic Usages
To save any single page, just type in python console
from pywebcopy import save_webpage save_webpage( url='http://example-site.com/index.html', project_folder='path/to/downloads' )
To save full website (This could overload the target server, So, be careful)
from pywebcopy import save_website save_website( url='http://example-site.com/index.html', project_folder='path/to/downloads', )
1.2.1 Running Tests
Running tests is simple and doesn't require any external library. Just run this command from root directory of pywebcopy package
$ python -m unittest pywebcopy.tests
1.2.2 Webpage() object
from pywebcopy import WebPage url = 'http://example-site.com/index.html' or None project_loc = 'path/to/downloads/folder' wp = WebPage(url, project_folder default_encoding=None, HTML=None, **configKwargs ) # You can choose to load the page explicitly using # `requests` module wp.get(url, **requestsKwargs) # if you want assets only wp.save_assets() # if you want html only wp.save_html() # if you want complete webpage wp.save_complete()
BeautifulSoup methods are supported
you can also use any beautiful_soup methods on it
>>> links = wp.bs4.find_all('a') ['//docs.python.org/3/tutorial/', '/about/apps/', 'https://github.com/python/pythondotorg/issues', '/accounts/login/', '/download/other/']
LXML is completely supported
You can use any lxml methods on it. Read more about lxml at
>>> wp.lxml.xpath('//a', ..) [<Element 'a'>,<Element 'a'>]
PyQuery is Fully supported
You can use PyQuery methods on it .Read more about pyquery at
>>> wp.pq.select(selector, ..) ...
XPath is also supported
xpath is also natively supported which retures a :class:
requests_html.Element See more at
>>> wp.xpath('a') [<Element 'a' class='btn' href='https://help.github.com/articles/supported-browsers'>]
You can also select only elements containing certain text
>>> wp.find('a', containing='kenneth') [<Element 'a' href='http://kennethreitz.com/pages/open-projects.html'>, <Element 'a'
Tutorials: sample use-cases with pywebcopy
Common Settings and Errors
If you are getting
then check if website allows scraping of its content.
>>> import pywebcopy >>> pywebcopy.config['bypass_robots'] = True # rest of your code follows..
Overwrite existing files when copying
If you want to overwrite existing files in the directory then use the over_write config key.
>>> import pywebcopy >>> pywebcopy.config['over_write'] = True # rest of your code follows..
Changing your project name
By default the pywebcopy creates a directory inside project_folder with the url you have provided but you can change this using the code below
>>> import pywebcopy >>> pywebcopy.config['project_name'] = 'my_project' # rest of your code follows..
How to - Save Single Webpage
Particular webpage can be saved easily using the following methods.
Note: if you get
pywebcopy.exceptions.AccessError when running any of these code then use the code provided on later sections.
Webpage can easily be saved using an inbuilt funtion called
.save_webpage() which takes several
>>> import pywebcopy >>> pywebcopy.save_webpage(project_url='http://google.com', project_folder='c://Saved_Webpages/',) # rest of your code follows..
This use case is slightly more powerful as it can provide every functionallity of the WebPage data class.
>>> from pywebcopy import Webpage >>> wp = WebPage('http://google.com', 'e://tests/', project_name='Google') >>> wp.save_complete() # This Webpage object contains every methods of the Webpage() class and thus # can be reused for later usages.
Method 2 using Plain HTML
:New in version 4.x:
I told you earlier that Webpage object is powerful and can be manipulated in any ways.
One feature is that the raw html is now also accepted.
>>> from pywebcopy import Webpage >>> HTML = open('test.html').read() >>> base_url = 'http://example.com' # used as a base for downloading imgs, css, js files. >>> project_folder = '/saved_pages/' >>> wp = WebPage(base_url, project_folder, HTML=HTML) >>> wp.save_webpage()
How to - Whole Websites
Use caution when copying websites as this can overload or damage the servers of the site and rarely could be illegal, so check everything before you proceed.
Method 1 -
Using the inbuilt api
.save_website() which takes several arguments.
>>> import pywebcopy >>> pywebcopy.save_website(project_url='http://localhost:8000', project_folder='e://tests/')
Method 2 -
By creating a Crawler() object which provides several other functions as well.
>>> import pywebcopy >>> pywebcopy.config.setup_config(project_url='http://localhost:5000/', project_folder='e://tests/', project_name='LocalHost') >>> crawler = pywebcopy.Crawler('http://localhost:5000/') >>> crawler.crawl()
You can contribute in many ways
- reporting bugs on github repo: https://github.com/rajatomar788/pywebcopy/ or my email.
- creating pull requests on github repo: https://github.com/rajatomar788/pywebcopy/
- sending a thanks mail
If you have any suggestions or fixes or reports feel free to mail me :)
pywebcopy is highly configurable.
1.3.1 Direct Call Method
To change any configuration, just pass it to the
from pywebcopy.core import save_webpage save_webpage( url='http://some-site.com/', # required download_loc='path/to/downloads/', # required # config keys are case-insensitive any_config_key='new_value', another_config_key='another_new_value', ... # add many as you want :) )
This function is changed from
You can manually configure every configuration by using a
import pywebcopy url = 'http://example-site.com/index.html' download_loc = 'path/to/downloads/' pywebcopy.config.setup_config(url, download_loc) # done! >>> pywebcopy.config.config['url'] 'http://example-site.com/index.html' >>> pywebcopy.config.config['mirrors_dir'] 'path/to/downloads' >>> pywebcopy.config.config['project_name'] 'example-site.com' ## You can also change any of these by just adding param to ## `setup_config` call >>> pywebcopy.config.setup_config(url, download_loc,project_name='Your-Project', ...) ## You can also change any config even after ## the `setup_config` call pywebcopy.config.config['url'] = 'http://url-changed.com' # rest of config remains unchanged
1.3.3 List of available
below is the list of
config keys with their
default values :
told you there were plenty of
config vars available!
1.5 Undocumented Features
I built many utils and classes in this project to ease the tasks I was trying to do.
But, these task are also suitable for general purpose use.
if you want, you can help in generating suitable
documentation for these undocumented ones, then you can always email me.
- Optimization of existing code, upto 5x speed ups in certain cases
- Removed cluttering, improved readability
- A complete rewrite and restructing of core functionality.
core.setup_configfunction is changed to
utils.tracedecorator, which will print function_name, args, kwargs and return value when debug config key is True.
- new html-parsers ('html5lib', 'lxml') are supported for better webpages.
- html-parser is now defaulted to 'lxml'. You can use any through new
- fixed issue while changing
user-agentkey cracked webpages. You can now use any browser's user-agent id and it will get exact same page downloaded.
- fixed issue in
generators.extract_css_urlswhich was caused by
bytesdifference in python3.
- fixed issues in modules importing. (Thanks "Илья Игоревич").
errorhandlingto required functions
initfunction is replaced with
- three new
configautomation functions are added -
core.setup_config(creates every ideal config just from url and download location)
config.reset_config(resets the configuration to default state)
config.update_config(manual-mode version of
generators.generate_relative_pathsto a single function
- rewrite of majority of functions
- new module
urlis checked and resolved of any redirection before starting any work functions.
clean_upwere fixed which cleaned the dir before the log was completely written.
initcall now takes
urlarg by default and could raise a error when not supplied
- professional looking log entries
- rewritten archiving system now uses
exceptionshandling to prevent errors and eventual archive corruption
- more redundant code
- modules are now separated based on type e.g. Core, Generators, Utils etc.
- new helper functions and class
- Compatible with Python 2.6, 2.7, 3.6, 3.7
Release history Release notifications | RSS feed
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size pywebcopy-5.0.1.tar.gz (34.6 kB)||File type Source||Python version None||Upload date||Hashes View|
|Filename, size pywebcopy-5.0.1-py2.py3-none-any.whl (33.4 kB)||File type Wheel||Python version py2.py3||Upload date||Hashes View|
Hashes for pywebcopy-5.0.1-py2.py3-none-any.whl