Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

== Install:

VCForPython27.msi (https://www.microsoft.com/en-us/download/details.aspx?id=44266)
lxml-3.5.0.win32-py2.7.exe (https://pypi.python.org/pypi/lxml/3.5.0)

$pip install crawlermaster

== write csslist.json:
example: techcrunch_news_csslist.json

{
"rawdata-converter-name":"techcrunch_news",
"local-html-file-pattern":{
"strBasedir":"C:\\Users\\muchu\\Downloads",
"strSuffixes":"news.html"
},
"csslist":[
{
"name": "techcrunch-news-title",
"sampleUrl": "https://techcrunch.com/2016/06/02/apple-app-store-goes-down/",
"cssRule": "header.page-title h1.tweet-title::text",
"sampleAns": [
"Apple App Store goes down"
],
"ansType": "exact"
},
{
"name":"techcrunch-news-tags",
"sampleUrl":"https://techcrunch.com/2016/06/02/apple-app-store-goes-down/",
"cssRule":"div.tags a.tag::text",
"sampleAns":[
"Apps",
"Apple",
"iCloud",
"iTunes",
"app-store"
],
"ansType":"exact"
},
{
"name":"techcrunch-news-pubtime",
"sampleUrl":"https://techcrunch.com/2016/06/02/apple-app-store-goes-down/",
"cssRule":"header.article-header div.byline time.timestamp::text",
"sampleAns":[
"4 hours ago"
],
"ansType":"exist"
}
]
}

== run test:

$crawlermaster csslist.json

== Uninstall:

$pip uninstall crawlermaster

Release files for crawlermaster 0.1.2a6

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for crawlermaster 0.1.2a6
File Size Uploaded
crawlermaster-0.1.2a6.zip 2.6 MB Details

Release files / crawlermaster-0.1.2a6.zip

Download URL crawlermaster-0.1.2a6.zip
Size 2.6 MB
Tags Source
SHA-256 checksum
How to use checksums
c158d1828340f37644ec4e827803869dd3281dd1a47d9f8a11347a8ec7ff6d76
BLAKE2b-256 checksum
How to use checksums
c39d8167b5feec10ed5e6ea777543d1312d1651855d01785f6a3b2b7eb101d79
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page