Skip to main content

Wraps Pip packages (beautifulsoup and requests) to a more concise content extractor

Project description

Gencon Miner

A general content miner that leverages on Beautiful Soup and Requests to handle extraction. The main goal is to always imagine in terms of targetting parent elements in an HTML form then getting group of tags given that parent.

from gencon_miner import GenconMiner

From URL

url_miner = GenconMiner(url="http://google.com")
txt = url_miner.extract('title')
print(txt[0].text) # Google

From text

text_miner = GenconMiner(text="<p class='myclass'>Hello</p>")
txt = text_miner.extract('.myclass')
print(txt[0].text) # Hello

Convert all tag content to string

Note that contents in a tag will be delimited using newline.

meaning_of_life = """
    <p class='myclass'>
        Hello
        <span>darkness my old friend</span>
    </p>
    <b>And another one</b>
"""
bulk_miner = GenconMiner(text=meaning_of_life)
print(bulk_miner.to_text()) # Hello\ndarkness my old friend\nAnd another one

Parent to target

Use-case on walking document and extracting the targets.

song_of_the_day = """
    <table id="mother">
        <tr>
            <td class="target-1">Mamma Mia</td>
            <td class="target-2">Here I go again</td>
            <td class="target-3">My my</td>
            <td class="target-4">How can I resist you</td>
        </tr>
    </table>
"""
walk_miner = GenconMiner(text=song_of_the_day)
print(walk_miner.extract('#mother', '.target-1')[0].text) # Mamma Mia
print(walk_miner.extract('#mother', '.target-3')[0].text) # My my
print(walk_miner.extract('#mother', 'td'))
# [
#   <td class="target-1">Mamma Mia</td>,
#   <td class="target-2">Here I go again</td>,
#   <td class="target-3">My my</td>,
#   <td class="target-4">How can I resist you</td>
# ]

Author

Almer Mendoza

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

GenconMiner-0.1.3.tar.gz (3.6 kB view hashes)

Uploaded Source

Built Distribution

GenconMiner-0.1.3-py3-none-any.whl (6.5 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page