Description

this is a sight Parser to help you pre_parser the datas from specified website url or api, it help you get ride of the duplicate coding to get the request from the specified url and speed up the process with the threading pool and you just need focused on the bussiness proceess coding after you get the specified request response from the specified webpage or api urls

Attention

as this slight pre_parser for the old version 1.0.0, which only can help preparser the static html or api inform, but now from the 2.0.0 , I have added an new html_dynamic mode, which will help get all inform even generated by the JS code.

python version >= 3.9 

How to use

install

$ pip install preparser

Github Resouce ➡️ Github Repos

and also just feel free to fork and modify this codes. if you like current project, star ⭐ it please, uwu.

PyPI: ➡️ PyPI Publish

parameters

here below are some of the parameters you can use for initial the Object PreParser from the package preparser:

example

#  test.py
from preparser import PreParser,BeautifulSoup,Json_Data,Filer


def handle_preparser_result(url:str,preparser_object:BeautifulSoup | Json_Data) -> bool:
    # here you can just write the bussiness logical you want
    
    # attention：
    # preparser_object type depaned on the `parser_mode` in the `PreParser`:
    #               'api' : preparser_object is the type of a Json_Data
    #               'html' : preparser_object is the type of a BeautifulSoup 
    
    ........
    
    # for the finally return:
    # if you want to show current result is failed just Return a None, else just return any object which is not None.
    return preparser_object


if __name__ == "__main__":
    
    #  start the parser
    url_list = [
        'https://example.com/api/1',
        'https://example.com/api/2',
        .....
    ]
  
    parser = PreParser(
        url_list=url_list,
        request_call_back_func=handle_preparser_result,
        parser_mode='api',    # this mode depands on you set, you can use the "api", "html",or 'html_dynamic'
        start_threading=True,
        threading_mode='single',
        cached_data=True,
        stop_when_task_failed=False,
        threading_numbers=3,
        checked_same_site=True
    )
    
    #  start parse
    parser.start_parse()

    # when all task finished, you can get the all task result result like below:
    all_result = parser.cached_request_datas
    
    # if you want to terminal, just execute the function here below
    # parser.stop_parse()

    # also you can use the Filer to save the final result above
    # and also find the datas in the `result/test.json` 
    filer = Filer('json')
    filer.write_data_into_file('result/test',[all_result])

Get Help

Get help ➡️ Github issue

Update logs

• version 2.0.8 : add the func of read_datas_from_file into Filer, to help read the datas from the specified type files.

• version 2.0.7 : add the ssl_certi_verified parameter to control weather ignored the error that caused by ssl certi verification when do the requesting.

• version 2.0.6 : add the html_dynamic_scope parameters to let user can specified the whole dynamic parse scope, which can help faster the preparser speed when the parser_mode is html_dynamic . and resort the additional tools into the ToolsHelper package.

• version 2.0.5 : remove the dynamic mode browser core install from setup into package call.

• version 2.0.4 : test the installing process command.

• version 2.0.3 : optimise the error alert for html_dynamic.

• version 2.0.2 : correct the README Doc of parser_mode.

• version 2.0.1 : update the README Doc.

• version 2.0.0 : add the new parser_mode of the html_dynamic, which help preparser all of the content from html , event it generated by the JS code.

• version 1.0.0 : basical version, only perparser the static html and api content.