Skip to main content

https://github.com/souravs17031999/AUTOMATE-DATA-PIPELINE

Project description

Automate-Images + Videos-download

Introduction :

Data collection is one of the important steps in Data analysis , Machine learning and Deep learning. It is the process of gathering information of our interest, in our case it's the images of any type we want to gather. Without data, we can't model our problem and do analysis of it.

Objective :

👉 Automate the collection and building of images using Google API's for search and videos using Youtube API's.
👉 Option to send mails to anyone containing collected datasets as attachment.

Getting credentials for Google API and Youtube API:

For Google image API For Youtube API

Getting Started Locally :

  • After getting the credentials , we can now run the script for automation.
  • Firstly download all the files on your system using cmd (terminal) or git bash :
    git clone https://github.com/souravs17031999/AUTOMATE-DATA-PIPELINE

With Tkinter GUI :

For IMAGES :

  • Move to the directory where script is downloaded.
  • Now run the script in following way :
    python data_image_automation.py
    and arguments can be passed from interactive GUI.

For VIDEOS :

  • Move to the directory where script is downloaded.
  • Now run the script in following way :
    python data_video_automation.py
    and arguments can be passed from interactive GUI.
    Note : Two important args :

Client secrets file path: Give absolute path.
Chrome driver path : Give absolute path with .exe extension.

Output :

Some samples of images : output
output
output
output
output

Limitations and Scope for future work:

There are few limitations :

  • Currently Google and Youtube allows 100 requests in one API call and so the script is able to download in total 100API's in one go but this behaviour can be changed from the script by calling it multiple times after tweaking some parameters which controls the after and before id's of returned images and videos and can be checked in their API documentation.
  • [For Images] Considering big file sizes for sending as mails, i have resized , gray-scaled and then zipped the files and upload as attachment.
    This behaviour can be changed from the script by disabling some of the functions passing a optional parameter.
  • [For videos] Considering big file sizes for sending as mails, i have compressed the file sizes and reduced the duration of videos.
    This behaviour can be easily manipulated in the script itself.
  • Also, mail can be only sent for GMAIL users as of now, this can also be changed by manipulating the Ports of all email servers and controlling their behaviours in the script.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

images-automate-python-souravdlboy-0.2.tar.gz (2.6 kB view details)

Uploaded Source

Built Distribution

File details

Details for the file images-automate-python-souravdlboy-0.2.tar.gz.

File metadata

  • Download URL: images-automate-python-souravdlboy-0.2.tar.gz
  • Upload date:
  • Size: 2.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.41.1 CPython/3.7.0b5

File hashes

Hashes for images-automate-python-souravdlboy-0.2.tar.gz
Algorithm Hash digest
SHA256 5cf99ebadbf6094bac5305a6cbaaeba8fa3f81be0b604c69ed9cc4e8930b9837
MD5 0f544f33d69822ec4ec5ec729a2eb383
BLAKE2b-256 64ca69d65c135ae57c70ce0ebd919aec6a9052099a887ca6d511c8a1c525e6f6

See more details on using hashes here.

File details

Details for the file images_automate_python_souravdlboy-0.2-py3-none-any.whl.

File metadata

  • Download URL: images_automate_python_souravdlboy-0.2-py3-none-any.whl
  • Upload date:
  • Size: 3.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.41.1 CPython/3.7.0b5

File hashes

Hashes for images_automate_python_souravdlboy-0.2-py3-none-any.whl
Algorithm Hash digest
SHA256 a9869b5a6c70fb761d19bbb1a306dc27619bb527b1200098afb7e542d98b1bfb
MD5 bfc23971a67d5818fb7d968b90e1c61f
BLAKE2b-256 1038e062fd8ae56640511cab34620710148ae322c23c83062b5b708f6dc4b153

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page