CLI for Steam Store Data Ingestion ETL Pipeline
Project description
Steam Sales Analysis
Overview
Welcome to Steam Sales Analysis – an innovative project designed to harness the power of data for insights into the gaming world. We have meticulously crafted an ETL (Extract, Transform, Load) pipeline that covers every essential step: data retrieval, processing, validation, and ingestion. By leveraging the robust Steamspy and Steam APIs, we collect comprehensive game-related metadata, details, and sales figures.
But we don’t stop there. The culmination of this data journey sees the information elegantly loaded into a MySQL database hosted on Aiven Cloud. From this solid foundation, we take it a step further: the data is analyzed and visualized through dynamic and interactive Tableau dashboards. This transforms raw numbers into actionable insights, offering a clear window into gaming trends and sales performance. Join us as we dive deep into the data and bring the world of gaming to life!
Setup Instructions
General Use Case
For general use, setting up the environment and dependencies is straightforward:
-
Clone the repository:
git clone https://github.com/DataForgeOpenAIHub/Steam-Sales-Analysis.git cd steam-sales-analysis
-
Install the package:
pip install .
-
Configuration:
- Create an
.env
file in the root directory of the repository. - Add the following variables to the
.env
file:# Database configuration MYSQL_USERNAME=<your_mysql_username> MYSQL_PASSWORD=<your_mysql_password> MYSQL_HOST=<your_mysql_host> MYSQL_PORT=<your_mysql_port> MYSQL_DB_NAME=<your_mysql_db_name>
- Create an
Development Setup
For development purposes, you might need to have additional dependencies and tools:
-
Clone the repository:
git clone https://github.com/DataForgeOpenAIHub/Steam-Sales-Analysis.git cd steam-sales-analysis
-
Create a virtual environment:
- Using
venv
:python -m venv game source game/bin/activate # On Windows use `game\Scripts\activate`
- Using
conda
:conda env create -f environment.yml conda activate game
- Using
-
Install dependencies:
- Install general dependencies:
pip install -r requirements.txt
- Install development dependencies:
pip install -r dev-requirements.txt
- Install general dependencies:
-
Configuration:
- Create an
.env
file in the root directory of the repository. - Add the following variables to the
.env
file:# Database configuration MYSQL_USERNAME=<your_mysql_username> MYSQL_PASSWORD=<your_mysql_password> MYSQL_HOST=<your_mysql_host> MYSQL_PORT=<your_mysql_port> MYSQL_DB_NAME=<your_mysql_db_name>
- Create an
steamstore
CLI
CLI for Steam Store Data Ingestion ETL Pipeline
Usage:
$ steamstore [OPTIONS] COMMAND [ARGS]...
Options:
--install-completion
: Install completion for the current shell.--show-completion
: Show completion for the current shell, to copy it or customize the installation.--help
: Show this message and exit.
Commands:
clean_steam_data
: Clean the Steam Data and ingest into the Custom Databasefetch_steamspy_data
: Fetch from SteamSpy Database and ingest data into Custom Databasefetch_steamspy_metadata
: Fetch metadata from SteamSpy Database and ingest metadata into Custom Databasefetch_steamstore_data
: Fetch from Steam Store Database and ingest data into Custom Database
Detailed Command Usage
steamstore clean_steam_data
Clean the Steam Data and ingest into the Custom Database
Usage:
$ steamstore clean_steam_data [OPTIONS]
Options:
--batch-size INTEGER
: Number of records to process in each batch. [default: 1000]--help
: Show this message and exit.
steamstore fetch_steamspy_data
Fetch from SteamSpy Database and ingest data into Custom Database
Usage:
$ steamstore fetch_steamspy_data [OPTIONS]
Options:
--batch-size INTEGER
: Number of records to process in each batch. [default: 1000]--help
: Show this message and exit.
steamstore fetch_steamspy_metadata
Fetch metadata from SteamSpy Database and ingest metadata into Custom Database
Usage:
$ steamstore fetch_steamspy_metadata [OPTIONS]
Options:
--max-pages INTEGER
: Number of pages to fetch from. [default: 100]--help
: Show this message and exit.
steamstore fetch_steamstore_data
Fetch from Steam Store Database and ingest data into Custom Database
Usage:
$ steamstore fetch_steamstore_data [OPTIONS]
Options:
--batch-size INTEGER
: Number of app IDs to process in each batch. [default: 5]--bulk-factor INTEGER
: Factor to determine when to perform a bulk insert (batch_size * bulk_factor). [default: 10]--reverse / --no-reverse
: Process app IDs in reverse order. [default: no-reverse]--help
: Show this message and exit.
Database Integration
The project connects to a MySQL database hosted on Aiven Cloud
using the credentials provided in the .env
file. Ensure that the database is properly set up and accessible with the provided credentials.
Running Individual Parts of the ETL Pipeline
To execute the ETL pipeline, use the following commands:
-
To collect metadata:
steamstore fetch_steamspy_metadata
-
To collect SteamSpy data:
steamstore fetch_steamspy_data --batch-size 1000
-
To collect Steam data:
steamstore fetch_steamstore_data --batch-size 5 --bulk-factor 10
-
To clean Steam data:
steamstore clean_steam_data --batch-size 1000
This will start the process of retrieving data from the Steamspy and Steam APIs, processing and validating it, and then loading it into the MySQL database.
References:
API Used:
- Steamspy API
- Steam Store API - InternalSteamWebAPI
- Steam Web API Documentation
- RJackson/StorefrontAPI Documentation
- Steamworks Web API Reference
Repository
LICENSE
This repository is licensed under the MIT License
. See the LICENSE file for details.
Disclaimer
The content and code provided in this repository are for educational and demonstrative purposes only. The project may contain experimental features, and the code might not be optimized for production environments. The authors and contributors are not liable for any misuse, damages, or risks associated with the direct or indirect use of this code. Users are strictly advised to review, test, and completely modify the code to suit their specific use cases and requirements. By using any part of this project, you agree to these terms.Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for steamstore_etl-0.0.8-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0efd5f7f5b99bba96ae2e3a8bca53ef0424d600893956204261ca63096dad081 |
|
MD5 | 552e0ad474b7d07fae532d0d31666966 |
|
BLAKE2b-256 | 897fc3ffef4438ad2842f0db0357e5c4ec18ffa3114499307a61df2ba2148da5 |