SuperMiner 22.3.1.1

Web miner built based on selenium but more simple operations

Introduction

This a project built for the SuperWebMiner, which is also a homework of my class. We can use this basic web miner frame to do some web miner works, such as downloading a large quantity of pictures etc. The goal of this project is to enable everyone to start his/her own super mine engine, and at the same time this project pushes me to comprised AI system closer. It would be great for you to give me suggestions on this project, all of us make it better and stronger!

• Download Source Codes File Folder
• Command Support
• Document
• Release Notes

Copyright

• Author: Airscker
• Last Released Time: 2022-4
• Latest Edition: R22.2.0.0
• Open source project. Copyright (C) Airscker, [e-mail], Mozilla Public License Version 2.0

Basic steps

Here we give you all the steps and references for build your first engine

Preparations

• For Python Before you import code into our project, you need to download the whole zip file and then unfold it, enter the filefolder 'Codes' and open cmd here, then type in the command below and enter:

pip install -r requirements.txt

• For Browser Now you need to install Chrome browser(this project only support chrome currently). Secondly get your chrome's edtion number in Settings. Then download chrome driver according to your edition number here. Move the webdriver.exe into the Scripts root path of python, such as: C:\Python\Python39\Scripts

Import

wait until all download threads executed,copy the file 'SuperMiner.py' and put it in the root of your project, then open your project, type in:

import SuperMiner as SP

Start your first engine

Here we show the basic steps to download Hello world images

• Initialize your engine

Hello_engine=SP.SuperMiner(url='https://cn.bing.com/images/search?q=Hello+world')

• Start miner engine

Hello_engine=SP.MineEngine()

• Scroll the page to get more images

SP.Basic_Actions(engine=Hello_enigine,Obj_index=-2,Send_keys=False,rollpage=True)

• Get the attributes of the images

Attr=Hello_engine.Attributes('src',Hello_engine.Objects(Class='mimg'))

• Download Images

Hello_engine.Download(Attr,data_type='img')

• Close engine

Hello_engine.engine.quit()

Now you are able to see the images downloaded in 'downloads' file folder, because the network may not be good enough, some images may be crashed, it's just no problem.

To get more details, please see Document

2022-3-14

We go until we go wrong, then we keep on until we are right

For dream