Skip to main content

Helpers to fetch & parse text on pages with requests, lxml, & beautifulsoup4

Project description

Install

Install system requirements for lxml

% sudo apt-get install -y libxml2 libxslt1.1 libxml2-dev libxslt1-dev zlib1g-dev

Try ``brew install libxml2`` if using a Mac.

Install with pip

% pip install parse-helper

Usage

In [1]: import parse_helper as ph

In [2]: ph.USER_AGENT
Out[2]: 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/55.0.2883.87 Chrome/55.0.2883.87 Safari/537.36'

In [3]: ph.youtube_serp('scaling redis')
Out[3]:
[{'duration': '1:14:51',
  'link': 'https://www.youtube.com/watch?v=rP9EKvWt0zo',
  'title': 'Scaling Redis at Twitter',
  'uploaded': '2 years ago',
  'user': 'Rackspace Developers',
  'views': '52,375 views'},
 {'duration': '19:47',
  'link': 'https://www.youtube.com/watch?v=oLjryfUZPXU',
  'title': 'Scaling Redis and Memcached at Wayfair',
  'uploaded': '2 years ago',
  'user': '@Scale',
  'views': '4,222 views'},
 {'duration': '18:29',
  'link': 'https://www.youtube.com/watch?v=3zxYaI3RQyM',
  'title': 'Horizontally scaled data processing architecture using Redis  - Ophir Hordan',
  'uploaded': '9 months ago',
  'user': 'Redis Labs',
  'views': '321 views'},
 {'duration': '20:12',
  'link': 'https://www.youtube.com/watch?v=0g9Jag4az0g',
  'title': 'Node.js & Redis at Scale',
  'uploaded': '1 year ago',
  'user': 'ViennaJS',
  'views': '1,146 views'},
 {'duration': '46:33',
  'link': 'https://www.youtube.com/watch?v=fyMXt2wI47E',
  'title': 'Fast Data at Internet Scale with Amazon ElastiCache for Redis',
  'uploaded': '4 months ago',
  'user': 'Amazon Web Services - Webinar Channel',
  'views': '1,028 views'},
 {'duration': '42:54',
  'link': 'https://www.youtube.com/watch?v=p3ytSdUQZzA',
  'title': 'FOWA 2013. Sharding and Scaling Your Database. Neha Narula',
  'uploaded': '2 years ago',
  'user': 'Future Insights',
  'views': '10,431 views'},
 {'duration': '22:56',
  'link': 'https://www.youtube.com/watch?v=a19OfVfqsa8',
  'title': 'Scaling Real-time Apps on Cloud Foundry Using Node.js and Redis',
  'uploaded': '4 years ago',
  'user': 'Cloud Foundry',
  'views': '3,063 views'},
 {'duration': '37:45',
  'link': 'https://www.youtube.com/watch?v=7tL2Eoiab7U',
  'title': 'Monitoring and  Scaling Redis at DataDog--Ilan Rabinovitch, DataDog',
  'uploaded': '8 months ago',
  'user': 'Redis Labs',
  'views': '150 views'},
 {'duration': '34:57',
  'link': 'https://www.youtube.com/watch?v=g69-3He_IYs',
  'title': 'Scaling Redis Cluster  Deployments for Genome Analysis--Terry Weatherland, IBM',
  'uploaded': '8 months ago',
  'user': 'Redis Labs',
  'views': '213 views'},
 {'duration': '22:22',
  'link': 'https://www.youtube.com/watch?v=pdrzRx0QHcU',
  'title': 'Sourav Sachin - Scaling with Unconventional Tech Stack nodejs + redis + mongodb',
  'uploaded': '4 years ago',
  'user': 'HasGeek TV',
  'views': '877 views'},
 {'duration': '25:58',
  'link': 'https://www.youtube.com/watch?v=6OvrFkLSoZ0',
  'title': 'Flight Lightning - Scaling Twitter core infrastructure',
  'uploaded': '2 years ago',
  'user': 'Twitter Dev',
  'views': '2,505 views'},
 {'duration': 'playlist',
  'link': 'https://www.youtube.com/watch?v=CoQcNgfPYPc&list=PL6QyeBG3jUEawDdZU-4DwBRYY0zyP_Rfn',
  'title': 'Redis',
  'uploaded': 'View full playlist (58 videos)',
  'user': 'Евгений Даниленко',
  'views': 'View full playlist (58 videos)'},
 {'duration': '49:35',
  'link': 'https://www.youtube.com/watch?v=UXnVYb-mqoo',
  'title': 'Josiah Carlson: Scaling Postgres With Some Help from Redis',
  'uploaded': '3 years ago',
  'user': 'PostgresOpen',
  'views': '449 views'},
 {'duration': '1:14:50',
  'link': 'https://www.youtube.com/watch?v=ahT006O7S9k',
  'title': 'AustinPHP - Scaling PHP Applications with Redis',
  'uploaded': '4 years ago',
  'user': 'AustinTechVideos',
  'views': '1,536 views'},
 {'duration': '34:39',
  'link': 'https://www.youtube.com/watch?v=p-XNGlUoPQg',
  'title': 'Scaling Rails Using Redis  with Limited Dev Resources-- Dmitry Polyakovsky, Snap Raise',
  'uploaded': '8 months ago',
  'user': 'Redis Labs',
  'views': '184 views'},
 {'duration': '27:04',
  'link': 'https://www.youtube.com/watch?v=nFJ7LD-2yXc',
  'title': 'Scaling with Redis Enterprise',
  'uploaded': '2 weeks ago',
  'user': 'Redis Labs',
  'views': '19 views'},
 {'duration': '39:22',
  'link': 'https://www.youtube.com/watch?v=A4xmIV0viv4',
  'title': 'How Hulu Scales Services to Support 400 Million Plays: C* , Redis, and SSD-Based Hardware',
  'uploaded': '2 years ago',
  'user': 'Hakka Labs',
  'views': '3,258 views'},
 {'duration': '27:04',
  'link': 'https://www.youtube.com/watch?v=3S-SXAxhgEQ',
  'title': 'Scaling with Redis Enterprise',
  'uploaded': '3 weeks ago',
  'user': 'Redis Labs',
  'views': '41 views'},
 {'duration': '10:37',
  'link': 'https://www.youtube.com/watch?v=S_jA39Uayak',
  'title': 'Predis in Laravel - Redis Series Episode 1',
  'uploaded': '1 year ago',
  'user': 'Christophe Limpalair',
  'views': '14,958 views'},
 {'duration': '45:53',
  'link': 'https://www.youtube.com/watch?v=jQNCuD_hxdQ',
  'title': 'GOTO 2014 • Scaling Pinterest • Marty Weiner',
  'uploaded': '2 years ago',
  'user': 'GOTO Conferences',
  'views': '5,904 views'}]

In [4]: ph.google_serp('scaling redis')
Out[4]:
[{'link': 'https://redis.io/topics/partitioning',
  'title': 'Partitioning: how to split data among multiple Redis instances. – Redis'},
 {'link': 'http://highscalability.com/blog/2014/9/8/how-twitter-uses-redis-to-scale-105tb-ram-39mm-qps-10000-ins.html',
  'title': 'How Twitter Uses Redis to Scale - 105TB RAM ... - High Scalability'},
 {'link': 'http://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/Scaling.RedisReplGrps.html',
  'title': 'Scaling Redis Clusters with Replica Nodes - Amazon ElastiCache'},
 {'link': 'http://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/Scaling.RedisStandalone.ScaleUp.html',
  'title': 'Scaling Up Single-Node Redis Clusters - Amazon ElastiCache'},
 {'link': 'https://redislabs.com/ebook/part-3-next-steps/chapter-10-scaling-redis/',
  'title': 'Chapter 10: Scaling Redis - Redis Labs'},
 {'link': 'https://redislabs.com/blog/scaling-out-redis-read-only-slaves-or-cluster/',
  'title': 'Scaling Out Redis: Read-Only Slaves or Cluster? - Redis Labs'},
 {'link': 'http://petrohi.me/post/6323289515/scaling-redis',
  'title': 'ten thousand hours • Scaling Redis'},
 {'link': 'https://www.quora.com/How-scalable-is-Redis',
  'title': 'How scalable is Redis? - Quora'},
 {'link': 'https://www.linkedin.com/pulse/how-twitter-uses-redis-scale-105tb-ram-39mm-qps-10000-iravani',
  'title': 'How Twitter Uses Redis To Scale - 105TB RAM, 39MM QPS ... - LinkedIn'},
 {'link': 'https://docs.microsoft.com/en-us/azure/redis-cache/cache-how-to-scale',
  'title': 'How to Scale Azure Redis Cache | Microsoft Docs'}]

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distribution

parse_helper-0.1.3-py3-none-any.whl (11.3 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page