Skip to main content
Join the official 2019 Python Developers SurveyStart the survey!

Wraps Pip packages (beautifulsoup and requests) to a more concise content extractor

Project description

Gencon Miner

A general content miner that leverages on Beautiful Soup and Requests to handle extraction. The main goal is to always imagine in terms of targetting parent elements in an HTML form then getting group of tags given that parent.

from gencon_miner import GenconMiner

From URL

url_miner = GenconMiner(url="http://google.com")
txt = url_miner.extract('title')
print(txt[0].text) # Google

From text

text_miner = GenconMiner(text="<p class='myclass'>Hello</p>")
txt = text_miner.extract('.myclass')
print(txt[0].text) # Hello

Convert all tag content to string

Note that contents in a tag will be delimited using newline.

meaning_of_life = """
    <p class='myclass'>
        Hello
        <span>darkness my old friend</span>
    </p>
    <b>And another one</b>
"""
bulk_miner = GenconMiner(text=meaning_of_life)
print(bulk_miner.to_text()) # Hello\ndarkness my old friend\nAnd another one

Parent to target

Use-case on walking document and extracting the targets.

song_of_the_day = """
    <table id="mother">
        <tr>
            <td class="target-1">Mamma Mia</td>
            <td class="target-2">Here I go again</td>
            <td class="target-3">My my</td>
            <td class="target-4">How can I resist you</td>
        </tr>
    </table>
"""
walk_miner = GenconMiner(text=song_of_the_day)
print(walk_miner.extract('#mother', '.target-1')[0].text) # Mamma Mia
print(walk_miner.extract('#mother', '.target-3')[0].text) # My my
print(walk_miner.extract('#mother', 'td'))
# [
#   <td class="target-1">Mamma Mia</td>,
#   <td class="target-2">Here I go again</td>,
#   <td class="target-3">My my</td>,
#   <td class="target-4">How can I resist you</td>
# ]

Author

Almer Mendoza

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for genconminer, version 0.1.7
Filename, size File type Python version Upload date Hashes
Filename, size GenconMiner-0.1.7-py2.py3-none-any.whl (4.2 kB) File type Wheel Python version py2.py3 Upload date Hashes View hashes
Filename, size GenconMiner-0.1.7.tar.gz (3.7 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page