ChallengerTeco
challengerteco is a Python library designed to streamline the process of training, evaluating, and comparing machine learning models in a distributed environment. It provides a flexible and configurable pipeline for data scientists and ML engineers to run model "challenger" competitions.
Description
This project encapsulates the end-to-end machine learning workflow, from data preparation to model evaluation. It is built to work with large datasets and can be executed on different platforms like Google Cloud Platform (GCP) or on-premise clusters.
The main goal of this library is to provide a standardized way to test different models and hyperparameters, making it easier to select the best-performing model for a given task.
Features
- Multi-platform support: Runs on Google Cloud Platform and on-premise environments.
- Multiple ML Models: Supports popular classification algorithms like:
- Random Forest
- Gradient Boosting
- Logistic Regression
- LightGBM
- XGBoost
- Data Preprocessing: Includes steps for:
- Target balancing (undersampling)
- Feature scaling
- Correlation analysis and removal
- Data type casting
- Hyperparameter Tuning: Uses
RandomizedSearchCVandGridSearchCVto find the best model parameters. - Model Evaluation: Evaluates models using AUC and other metrics.
- Persistence: Saves trained models and evaluation results.
Classes
query_manager
This class is a wrapper for database interactions. It abstracts the boilerplate code needed to connect to and query different data sources.
execute_query(query): Executes a SQL query.to_table(df, mode, nombre): Saves a DataFrame to a table.from_query(query): Creates a DataFrame from a SQL query.
Challenger
The main class of the library. It orchestrates the entire ML pipeline. It takes a large number of configuration parameters to control every aspect of the process.
The main method is EjecutarChallenger(), which runs the complete challenger process.
Installation
pip install challengerteco
Usage
The intended use is to create a JSON configuration file and instantiate the Challenger class from it.
config.json
{
"BALANCEAR_TARGET": true,
"ELIMINAR_CORRELACIONES": true,
"CASTEAR_BIGINT": true,
"REDONDEAR_DECIMALES": true,
"CON_SCALER": true,
"TIENE_TESTING": true,
"CORRER_RF": true,
"CORRER_GB": true,
"CORRER_LR": true,
"CORRER_LGBM": true,
"CORRER_XGB": true,
"CORRER_PRODUCTIVO": false,
"CAMPO_CLAVE": "client_id",
"TARGET": "purchase",
"modelo": "propensity_to_buy",
"ABT_VARIABLES": "var1, var2, var3",
"ABT_TABLA": "my_project.my_dataset.abt_table",
"TGT_TABLA": "my_project.my_dataset.target_table",
"TGT_VARIABLES": "target_variable",
"TGT_BALENCEO": 20,
"DECIMALES_VARIABLES_NUMERICAS": 4,
"COTA_CORRELACIONES": 0.9,
"REGISTROS_X_PARTICION": 100000,
"PORCENTAJE_TRAINING": 0.8,
"GB_param_test": {},
"LGBM_param_test": {},
"XGB_param_test": {},
"RF_param_test": {},
"LR_param_test": {},
"PERIODO": "202301",
"PERIODO_TRAIN1": "202207",
"PERIODO_TRAIN2": "202208",
"PERIODO_TRAIN3": "202209",
"PERIODO_TRAIN4": "202210",
"PERIODO_TRAIN5": "202211",
"PERIODO_TRAIN6": "202212",
"PERIODO_TEST1": "202301",
"PERIODO_TEST2": "202302",
"PERIODO_TEST3": "202303",
"MODELO_PRODUCTIVO": "RF",
"MODELO_PRODUCTIVO_param_test": {},
"GRABAR_BINARIOS": true,
"Tabla_Performance_Modelos": "my_project.my_dataset.performance_table"
}
main.py
from challengerteco import Challenger
from pyspark.sql import SparkSession
# You will need to provide your own bq and spark objects
# bq = bigquery.Client()
# spark = SparkSession.builder.getOrCreate()
challenger = Challenger.instanciar_desde_json('config.json')
challenger.bq = bq
challenger.spark = spark
challenger.reset_query_manager()
challenger.EjecutarChallenger()
TODOs
The following TODOs were found in the code and could be addressed in future versions:
- Clean up library imports.
- Add assertions to validate input parameters.
- Improve error handling and logging.
- Refactor duplicated code in
RandomForest,GradientBoosting, andLogisticRegressionmethods. - Review memory pressure in pandas-based methods.
- Improve output formatting for better readability.
- Remove hardcoded values and unused variables.
- Update deprecated commands (
hdfs dfs -copyFromLocal). - Add docstrings to methods.
This README.md provides a good overview of the project. It explains what the project does, its main features, how to install and use it, and what can be improved.
Metadata
Release files for challengerteco 0.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| challengerteco-0.2.0.tar.gz | 19.5 kB | Without provenance | Details for challengerteco-0.2.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| challengerteco-0.2.0-py3-none-any.whl | Python 3 | none | any | Details for challengerteco-0.2.0-py3-none-any.whl |
Total release size: 38.1 kB
Release files / challengerteco-0.2.0.tar.gz
| Download URL | challengerteco-0.2.0.tar.gz |
|---|---|
| Size | 19.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
564ace8a0d980a5cfe9812921004f13a9c28690cf79da841969c938e2a29318f
|
|
BLAKE2b-256 checksum How to use checksums |
b7c40d3895bbe1a6787037fd939a9578e23fe0fdc6ff6613a220c90e9fc985a8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.4.1 CPython/3.14.6 Linux/7.1.5-1-cachyos
|
Release files / challengerteco-0.2.0-py3-none-any.whl
| Download URL | challengerteco-0.2.0-py3-none-any.whl |
|---|---|
| Size | 18.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
a330e722a41af4093b06408bb00ed01849f6d46adcabcab2665ecf393509c997
|
|
BLAKE2b-256 checksum How to use checksums |
5e614ca4a0878eb54a00d193e5c5a9076f053a7fb4b692496ef84a0437b2b108
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.4.1 CPython/3.14.6 Linux/7.1.5-1-cachyos
|
Security for challengerteco 0.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.