preparser 2.0.5

a slight preparser to help parse webpage content or get request from urls,which supports win, mac and unix.

Description

this is a sight Parser to help you pre_parser the datas from specified website url or api, it help you get ride of the duplicate coding to get the request from the specified url and speed up the process with the threading pool and you just need focused on the bussiness proceess coding after you get the specified request response from the specified webpage or api urls

Attention

as this slight pre_parser for the old version 1.0.0, which only can help preparser the static html or api inform, but now from the 2.0.0 , I have added an new html_dynamic mode, which will help get all inform even generated by the JS code.

python version >= 3.9 

How to use

install

$ pip install preparser

Github Resouce ➡️ Github Repos

and also just feel free to fork and modify this codes. if you like current project, star ⭐ it please, uwu.

PyPI: ➡️ PyPI Publish

parameters

here below are some of the parameters you can use for initai the Object PreParser from the package preparser:

Parameters	Type	Description
url_list	list	The list of URLs to parse from. Default is an empty list.
request_call_back_func	Callable or None	A callback function according to the parser_mode to handle the BeautifulSoup object or request json Object. and if you want to show your business process failed, you can return None, otherwise please return a not None Object.
parser_mode	'html', 'api' or 'html_dynamic'	The pre-parsing datas mode,default is 'html'. html: parse the content from static html, and return an BeautifulSoup Object. api: parse the datas from an api, and return the json Object. html_dynamic: parse from the whole webpage html content and return an BeautifulSoup Object, even the content that generated by the dynamic js code. **and all of Object you can get when you defined the request_call_back_func, otherwise get it via the object of PreParer(....).cached_request_datas
cached_data	bool	weather cache the parsed datas, defalt is False.
start_threading	bool	Whether to use threading pool for parsing the data. Default is False.
threading_mode	'map' or 'single'	to run the task mode, default is single. map: use the map func of the theading pool to distribute tasks. single: use the submit func to distribute the task one by one into the theading pool.
stop_when_task_failed	bool	wheather need stop when you failed to get request from a Url,default is True
threading_numbers	int	The maximum number of threads in the threading pool. Default is 3.
checked_same_site	bool	wheather need add more headers info to pretend requesting in a same site to parse datas, default is True,to resolve the CORS Block.

example

#  test.py
from preparser import PreParser,BeautifulSoup,Json_Data,Filer


def handle_preparser_result(url:str,preparser_object:BeautifulSoup | Json_Data) -> bool:
    # here you can just write the bussiness logical you want
    
    # attention：
    # preparser_object type depaned on the `parser_mode` in the `PreParser`:
    #               'api' : preparser_object is the type of a Json_Data
    #               'html' : preparser_object is the type of a BeautifulSoup 
    
    ........
    
    # for the finally return:
    # if you want to show current result is failed just Return a None, else just return any object which is not None.
    return preparser_object


if __name__ == "__main__":
    
    #  start the parser
    url_list = [
        'https://example.com/api/1',
        'https://example.com/api/2',
        .....
    ]
  
    parser = PreParser(
        url_list=url_list,
        request_call_back_func=handle_preparser_result,
        parser_mode='api',    # this mode depands on you set, you can use the "api" or "html"
        start_threading=True,
        threading_mode='single',
        cached_data=True,
        stop_when_task_failed=False,
        threading_numbers=3,
        checked_same_site=True
    )
    
    #  start parse
    parser.start_parse()

    # when all task finished, you can get the all task result result like below:
    all_result = parser.cached_request_datas
    
    # if you want to terminal, just execute the function here below
    # parser.stop_parse()

    # also you can use the Filer to save the final result above
    # and also find the datas in the `result/test.json` 
    filer = Filer('json')
    filer.write_data_into_file('result/test',[all_result])

Get Help

Get help ➡️ Github issue

Update logs

• version 2.0.5 : remove the dynamic mode browser core install from setup into package call.

• version 2.0.4 : test the installing process command.

• version 2.0.3 : optimise the error alert for html_dynamic.

• version 2.0.2 : correct the README Doc of parser_mode.

• version 2.0.1 : update the README Doc.

• version 2.0.0 : add the new parser_mode of the html_dynamic, which help preparser all of the content from html , event it generated by the JS code.

• version 1.0.0 : basical version, only perparser the static html and api content.