A web-scraping-based python package that enables you to scrape data from various platforms.

These details have not been verified by PyPI

Project links

Homepage

GitHub Statistics

View statistics for this project via Libraries.io, or by using our public dataset on Google BigQuery

Project description

Documentation

Package Installation

pip install scrape-up

Examples

Scrape the numbers of followers on GitHub

# Import the required module
from scrape_up import github

# Instantiate an object with the username provided.
user = github.Users(username="nikhil25803")

# Call the followers function - it will return the number of followers
per user.followers()

The platforms we cover.

GitHub

from scrape_up import github

Scrape User details

Create an instance of the class Users.

user = github.Users(username="nikhil25803")

Methods	Details
`.followers()`	Returns the number of followers of a user.
`.following()`	Returns the number of following of a user.
`.get_avatar()`	Returns the avatar URL of a user.
`.get_bio()`	Returns the bio of a user.
`.get_repo()`	Returns the list of pinned repositories for a user.
`.repo_count()`	Returns the number of Repositories of a user.
`.star_count()`	Returns the number of stars of a user.
`.get_yearly_contributions()`	Returns the number of contributions made in 365 days frame.
`.get_repositories()`	Returns the list of repositories of a user.
`.get_starred_repos()`	Return the list of starred repositories of a user.
`.pul_requests()`	Return the number of pull requests opened in a repository.
`.get_followers()`	Returns the list of followers of a user.
`.get_following_users()`	Returns the list of users followed by a user.
`.get_achievements()`	Returns the list of achievements of a user.
`.get_status()`	Returns the status of a user.
`.get_contribution_streak()`	Returns the maximum contribution streak of a user in the past year starting from the current date.
`.get_repository_details()`	Returns the list of repositories with their details.
`.get_branch()`	Returns the list of branches in a repository.
`.get_merged_pull_requests()`	Returns the list of merged pull requests
`.get_open_issues()`	Returns the list of open issues
`.get_years_active()`	Returns the number of years that user have been active on github.

Scrape Repository details

Create an instance of the class Repository.

repository = github.Repository(username="nikhil25803", repository_name="scrape-up")

Methods	Details
`.fork_count()`	Returns the number of forks of a repository.
`.get_contributors()`	Returns the number of contributors of a repository.
`.topics()`	Returns the topics of a repository.
`.pull_requests()`	Returns the number of pull requests opened in a repository.
`.last_updated_at()`	Returns the last updated date of a repository.
`.tags()`	Returns the last ten tags of a repository.
`.releases()`	Returns the last ten releases of a repository.
`.issues_count()`	Returns number of issues in a repository
`.readme`	Saves the readme.md file of the given user to the current working directory. To view the readme.md with a live server, change ".md" to ".html" in "readme.md".
`.get_pull_requests_ids()`	Returns all ids of opened pull requests in a repository.
`.get_issues()`	Returns the list of all open issues in a repository.
`.commits()`	Returns the number of commits in a repository.
`.get_readme()`	Returns & saves README.md file of the special repository (if exists)
`.get_environment()`	Returns the latest deployed link of a repository (if exists).
`.watch_count()`	Returns the number of watchers of a repository
`.all_watchers()`	Returns the username of all watches of a repository

Scrape details of an issue

Create an instance of the class Issue

repository = github.Issue(username="nikhil25803", repository_name="scrape-up", issue_number=59)

Methods	Details
`.assignees()`	Returns the assignees of an issue.
`.labels()`	Returns the labels of an issue.
`.opened_by()`	Returns the name of the user, who opened the issue.
`.title()`	Returns the title of an issue.
`.is_milestone()`	Returns the milestone, if the issue is part of one or 'No milestone', if it's not.
`.opened_at()`	Returns a string containing the time when the issue was opened in ISO format.

Scrape a pull request details

Create an instance of the class PullRequest

pull_request = github.PullRequest(username="nikhil25803", repository_name="scrape-up", pull_request_number=30)

Methods	Details
`.commits()`	Returns the number of commits made in a pull request.
`.title()`	Returns the title of a pull request.
`.labels()`	Returns all the labels of a pull request, empty list in case of no labels.
`.files_changed()`	Returns the number of files changed in a pull request.
`.reviewers()`	Return the list of reviewers assigned in a pull request.

Scrape the details of an organization

Create an instance of class Organization

organization = github.Organization(organization_name="Clueless-Community")

Methods	Details
`.top_topics()`	Returns a list of the most used topics in an organization.
`.followers()`	Returns the number of followers of an organization.
`.top_languages()`	Returns the top languages used in an organization.
`.followers()`	Returns the number of followers of an organization.
`.avatar()`	Returns the avatar URL of an organization.
`.repositories()`	Returns the list of repositories of an organization.
`.people()`	Returns the list of people in an organization.
`.peoples()`	Returns the number of people in an organization.
`.get_location()`	Returns the location of an organization.
`.repository_details()`	Returns the list of repositories with their details.
`.pinned_repository()`	Returns the list of pinned repositories with their details.
`.get_organization_links()`	Returns a dictionary of important website links of a community.

GitHub Education

from scrape_up import github_education

Scrape user details

Create an instance of the Events class.

events = github_education.Events()

Methods	Details
`.get_events()`	Returns the latest events along with their title, image_url, description, date, location, language, tags, and link.

CodeChef

from scrape_up import codechef

Scrape user details

Create an object of class Codechef

user1 = codechef.User(id="username")

Methods	Details
`get_profile()`	Returns name, username, profile_image_link, rating, details etc.

Hacker News

from scrape_up import hackernews

Scrape article details

Create an instance of the Articles class.

articles = hackernews.Articles()

Methods	Details
`.articles_list()`	Returns the latest articles along with their score, author, author URL, time, comment count, and link.
`.new_articles()`	Returns the latest new articles along with their score, author, author URL, time, comment count, and link.
`.past_articles()`	Returns the past articles along with their score, author, author URL, time, comment count, and link.
`.ask_articles()`	Returns the asked articles along with their score, author, author URL, time, comment count, and link.
`.show_articles()`	Returns the show articles along with their score, author, author URL, time, comment count, and link.
`.jobs()`	Returns the jobs along with their time and link.

HackerEarth

from scrape_up import hackerearth

Create an object of class Challanges

hackerearth = hackerearth.Challanges()

Methods	Details
`get_upcoming()`	Get the details of upcoming challenges on Hackerearth.
`get_ongoing()`	Get the details of ongoing challenges on Hackerearth.
`get_hiring()`	Get the details of hiring challenges on Hackerearth.

HackerRank

from scrape_up import hackerrank

Scrape user details

Create an object of class User.

hackerank = hackerrank.User()

Methods	Details
`get_profile(id="username")`	Returns name, username, country, user_type, details, badges, verified_skills, social etc.
`get_skills()`	Returns a list of verified skills and their links

Scrape contest details

Create an object of class Contest.

hackerank = hackerrank.Contest()

Methods	Details
`active_contests()`	Returns information on active contests like title, status, and link
`archived_contests()`	Returns information regarding archived contests

Hashnode

from scrape_up import hashnode

Create an instance of Hashnode class.

blogs = hashnode.Hashnode()

Methods	Details
`.get_feed()`	Returns the blogs with title, descriptions, author, read time, like and comment count, date and link
`.get_featured()`	Returns the featured blogs with title, descriptions, author, like and comment count, date and link
`.get_recent()`	Returns the recent blogs with title, descriptions, author, like and comment count, date and link
`.search(topic)`	Returns the blogs with title, descriptions, author, like and comment count, date and link for a topic

ICC Rankings

from scrape_up import icc

Create an instance of ICC class.

scraper = icc.ICC()

Method	Details
`.team_rankings(format)`	Returns the list of rankings of teams of the desired format
`.player_ranking(type,format)`	Returns the list of player ranking of desired type and format
`.team_rankings_women(format)`	Returns the list of rankings of teams of the desired format
`.player_ranking_women(type,format)`	Returns the list of women player ranking of desired type and format

Academia

from scrape_up import academia

Create an instance of Academia class

academia = academia.Academia()

Method	Details
`get_research_topics(letter)`	Fetches and returns research topics starting with the given letter.
`get_research_papers(search)`	Fetches and returns research papers related to the given search term.

AskUbuntu

from scrape_up import askubuntu

Create an instance of Questions class.

questions = askubuntu.Questions("topic")

Methods	Details
`.getNewQuestions()`	Returns the new questions, views, votes, answer counts, and descriptions in JSON format
`.getActiveQuestions()`	Returns the active questions, views, votes, answer counts, and descriptions in JSON format
`.getUnansweredQuestions()`	Returns the unanswered questions, views, votes, answer counts, and descriptions in JSON format
`.getBountiedQuestions()`	Returns the bountied questions, views, votes, answer counts, and descriptions in JSON format
`.getFrequentQuestions()`	Returns the frequently asked questions, views, votes, answer counts, and descriptions in JSON format
`.getHighScoredQuestions()`	Returns the most voted questions, views, votes, answer counts, and descriptions in JSON format

BBC News

from scrape_up import bbcnews

First create an object of class BBCNews

user = bbcnews.BBCNews()

Methods	Details
`.get_headlines()`	Returns the list of object containig the headlines
`get_article()`	Returns an object with proper details about the articles

CoinMarketCap

from scrape_up import coinmarketcap

Create an instance of Crypto class

crypto = coinmarketcap.Crypto()

Method	Details
`get_top_cryptocurrencies()`	Fetches and returns data about the top cryptocurrencies.

CovidInfo

from scrape_up import covidinfo

Create an instance of the CovidInfo class.

response = covidinfo.CovidInfo()

Methods	Details
`.covid_data()`	Returns the list of all the covid data scraped from the website
`.total_cases()`	Returns the count of total covid cases all over the world
`.total_deaths()`	Returns the count of deaths covid cases all over the world
`.total_recovered()`	Returns the count of recovered covid cases all over the world
`.latest_news()`	Return the lastest news of the day

Cricbuzz

from scrape_up import cricbuzz

Create an instance of Cricubzz class.

	cricbuzz = cricbuzz.Cricubzz()

Methods	Details
`.get_live_matches()`	Returns a list of live matches from Cricbuzz.
`.get_recent_matches()`	Returns a list of recent matches from Cricbuzz.
`.get_upcoming_matches()`	Returns a list of upcoming matches from Cricbuzz.
`.get_series()`	Returns a dictionary of series in month and year format from Cricbuzz.
`.get_series_from_archive()`	Returns a list of series from archive from Cricbuzz.
`.get_matches_by_day()`	Returns a dictionary of matches by day from Cricbuzz.
`.get_series_matches()`	Returns a list of matches in a series from Cricbuzz.
`.get_series_stats()`	Returns a list of stats of players in a series from Cricbuzz.
`.get_teams_list()`	Returns a list of teams from Cricbuzz.
`.get_team_schedule()`	Returns a list of matches of a team from Cricbuzz.
`.get_team_players()`	Returns a list of players of a team from Cricbuzz.
`.get_team_results()`	Returns a list of past results of a team from Cricbuzz.
`.get_team_stats()`	Returns a list of player stats of a team from Cricbuzz.

Dribbble

from scrape_up import dribbble

Create an instance of Dribbble class.

shots = dribbble.Dribbble()

Methods	Details
`.get_shots()`	Returns the latest shots along with their title, designer and designer url like and view count and link.
`.search(topic)`	Returns the latest shots along with their title, designer and designer url like and view count and link for the searched topic.
`.get_animation()`	Returns the latest animation along with their title, designer and designer url like and view count and link.
`.get_branding()`	Returns the latest branding along with their title, designer and designer url like and view count and link.
`.get_illustration()`	Returns the latest illustration along with their title, designer and designer url like and view count and link.
`.get_mobile()`	Returns the latest mobile shots along with their title, designer and designer url like and view count and link.
`.get_webdesign()`	Returns the latest web-design shots along with their title, designer and designer url like and view count and link.

EazyDiner

from scrape_up import eazydiner

Create an instance of EazyDiner class.

restaurants = eazydiner.EazyDiner(location="city-name")

Methods	Details
`.get_restaurants()`	Returns the restaurants name, location, rating, cuisine and prices.
`.get_breakfast()`	Returns the restaurants name, location, rating, cuisine and prices for Breakfast.
`.get_lunch()`	Returns the restaurants name, location, rating, cuisine and prices for Lunch.
`.get_dinner()`	Returns the restaurants name, location, rating, cuisine and prices for Dinner.
`.dinner_with_discount()`	Returns a list of restaurants from the entered location with a 50% offer.
`.get_top10()`	Returns a list of the top 10 restaurants from a given city.

ESPN

from scrape_up import espn

Create an instance of ESPN class

espn = espn.ESPN()

Method	Details
`get_scoreboard()`	Fetches and returns the football scoreboards for a given date.
`get_tournaments()`	Fetches and returns information about football tournaments.
`get_teams()`	Fetches and returns information about football teams.

eBay

from scrape_up import ebay

Create an instance of EBAY class

quora = ebay.eBay()

Methods	Details
`.spotlights()`	Returns spotlight deals on eBay.
`.featured()`	Returns the featured deals on eBay.
`.specific_deals()`	Returns the specific deals on eBay.

Flipkart

from scrape_up import flipkart

Create an instance of Flipkart class.

item = flipkart.Flipkart()

Methods	Details
`.TVs()`	Returns the list of TV sets on flipkart
`.bestseller_books()`	Returns the list of bestselling books data listed on Flipkart.
`.mobiles()`	Returns the list of mobile phones under 50K along with their data.
`.sport_shoes()`	Returns the list of trendong sport shoes data.
`.laptops()`	Returns the list of laptop from flipkart.
`.camera()`	Returns the list of camera from flipkart.
`.computer()`	Returns the list of computer from flipkart.
`.tablets()`	Returns the list of tablets from flipkart.
`.cycle()`	Returns the list of bicycles from flipkart.
`.printers()`	Returns the list of printers from flipkart.
`.monitor()`	Returns the list of monitors from flipkart.
`.ac()`	Returns the list of acs from flipkart.
`.refrigerator()`	Returns the list of refrigerators from flipkart.
`.VRbox()`	Returns the list of VRbox from flipkart.
`.Speakers()`	Returns the list of Speakers from flipkart.

FlipkartClothing

Create an instance of FlipkartClothing class.

cloth = flipkart.FlipkartClothing()

Methods	Details
`.scrape()`	Returns the list of t-shirts with other relevant info
`.range()`	Returns the list of t-shirts between a particular price range.
`.minrating()`	Returns the list of t-shirts havinga minimum given rating.
`.gendermale()`	Returns the list of t-shirts which are for males.
`.genderfemale()`	Returns the list of t-shirts which are there for females.
`.size()`	Returns the list of tshirts havning a particular size.
`formal_shirts_for_male()`	It returns those t-shirts which are of a particular size

Flipkart laptops

Create an instance of FlipkartLaptops class.

item = flipkart.FlipkartLaptops()

Methods	Details
`.laptops()`	Returns the list of laptops with details

Flyrobu

from scrape_up import flyrobu

Create an instance of Flyrobu class.

flyrobu = flyrobu.Flyrobu()

Methods	Details
`.search(keyword)`	Returns the json data of all the details related to search with informing about the total amount of items found
`.get_product_details(product_name)`	Returns the json data of the product details based on the given `product_name`

HealthGrades

from scrape_up import healthgrades

Create an instance of HealthGrades class

hc = healthgrades.HealthGrades()

Method	Details
`get_best_hospitals(state)`	Fetches and returns information about the best hospitals in a state.

IMDB

from scrape_up import imdb

Create an instance of the IMDB class.

scraper = imdb.IMDB()

Methods	Details
`.top_rated()`	Returns the top-rated movies listed on IMDB.
`.scrape_genre_movies(genre)`	Returns the list of movies related to the genre you mentioned.
`.top_rated_shows()`	Returns the top-rated shows listed on IMDB.

IMDB - Movies

Create an instance of Movie class.

movie = imdb.Movie(movie_name)

Methods	Details
`.rating()`	Returns the IMDB rating of the movie
`.description()`	Returns the description, cast and director of the movie
`.more_movies()`	Returns similar movies recommended by IMDB
`.box_office()`	Returns budget, gross worldwide collections of the movie

IMDB - Actor

Create an instance of Actor class.

actor = imdb.Actor(actor_name)

Methods	Details
`.popular_movies()`	Returns the popular movies in which the actor has acted
`.all_movies()`	Returns all movies acted in and upcoming movies
`.awards()`	Returns the number of awards and nominations

IMDB - Celeb

Create an instance of Celeb class.

celeb = imdb.Celeb()

Methods	Details
`.top_celebs()`	Returns the name, roles, famous movie of the celeb

IMDB - Indian Movies

Create an instance of IndianMovies class.

indianmovies = imdb.IndianMovies()

Methods	Details
`.top_indian_movies()`	Returns the current list of top Indian movies

IMDB - Box Office

Create an instance of BoxOffice class.

boxoffice = imdb.BoxOffice()

Methods	Details
`.top_movies()`	Returns the top box office movies, weekend and total gross and weeks released

Project details

These details have not been verified by PyPI

Project links

Homepage

GitHub Statistics

View statistics for this project via Libraries.io, or by using our public dataset on Google BigQuery

Release history Release notifications | RSS feed

This version

1.1.7

Oct 13, 2023

1.1.6

Oct 10, 2023

1.1.5

Aug 25, 2023

1.0.0

Feb 24, 2023

0.1.1

Feb 11, 2023

0.1.0

Feb 11, 2023

0.0.9

Jan 28, 2023

0.0.8

Jan 28, 2023

0.0.6

Jan 28, 2023

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

scrape_up-1.1.7.tar.gz (72.4 kB view hashes)

Uploaded Oct 13, 2023 Source

Built Distribution

scrape_up-1.1.7-py3-none-any.whl (80.3 kB view hashes)

Uploaded Oct 13, 2023 Python 3

Hashes for scrape_up-1.1.7.tar.gz

Hashes for scrape_up-1.1.7.tar.gz
Algorithm	Hash digest
SHA256	`ab4c827eefd49b6b8a4f2e83490bbd0770d42f748c35acd2d8da74649d254ca3`
MD5	`9bb48abe35c9df947d1668627faa7770`
BLAKE2b-256	`a5ae864807dddb5bc62a54e48335311c079c3dd67a28fc537a85fda5bd2234ad`

Hashes for scrape_up-1.1.7-py3-none-any.whl

Hashes for scrape_up-1.1.7-py3-none-any.whl
Algorithm	Hash digest
SHA256	`074ccb0b5b56e1cc456a3a0c68084c88c071bb4cde0045238106af53ed916771`
MD5	`76f2af453ac340f294695f14865a228f`
BLAKE2b-256	`902a4f6113da87bb9890ab53aee5c6e2338f2911923a45ff2bdca645b7f28410`

scrape-up 1.1.7

Navigation

Verified details

Maintainers

Unverified details

Project links

GitHub Statistics

Meta

Classifiers

Project description

Documentation

Package Installation

Examples

Scrape the numbers of followers on GitHub

The platforms we cover.

GitHub

Scrape User details

Scrape Repository details

Scrape details of an issue

Scrape a pull request details

Scrape the details of an organization

GitHub Education

Scrape user details

CodeChef

Scrape user details

Hacker News

Scrape article details

HackerEarth

HackerRank

Scrape user details

Scrape contest details

Hashnode

ICC Rankings

Academia

AskUbuntu

BBC News

CoinMarketCap

CovidInfo

Cricbuzz

Dribbble

EazyDiner

ESPN

eBay

Flipkart

FlipkartClothing

Flipkart laptops

Flyrobu

HealthGrades

IMDB

IMDB - Movies

IMDB - Actor

IMDB - Celeb

IMDB - Indian Movies

IMDB - Box Office

Project details

Verified details

Maintainers

Unverified details

Project links

GitHub Statistics

Meta

Classifiers

Release history Release notifications | RSS feed

Download files

Source Distribution

Built Distribution