Skip to main content

A small package for evaluating numer.ai model locally

Project description

A small library to reproduce the scores on numer.ai diagnistics dashboard.

Installation

pip install numereval

Structure

Structure

Numerai main tournament evaluation metrics

numereval.numereval.evaluate

A generic function to calculate basic per-era correlation stats with optional feature exposure and plotting.

Useful for evaluating custom validation split from training data without MMC metrics and correlation with example predictions.

from numereval.numereval import evaluate

evaluate(training_data, plot=True, feature_exposure=False)
Correlations plot Returned metrics
Training Correlations Metrics

numereval.numereval.diagnostics

To reproduce the scores on diagnostics dashboard locally with optional plotting of per-era correlations.

from numereval.numereval import diagnostics

validation_data = tournament_data[tournament_data.data_type == "validation"]

diagnostics(
    validation_data,
    plot=True,
    example_preds_loc="numerai_dataset_244\example_predictions.csv",
)
Validation plot Returned metrics
all eras validation plot all eras validation metrics

Specific validation eras

specify a list of eras in the format eras = ["era121", "era122", "era209"]

validation_data = tournament_data[tournament_data.data_type == "validation"]

eras = validation_data.era.unique()[11:-2]

numereval.numereval.diagnostics(
    validation_data,
    plot=True,
    example_preds_loc="numerai_dataset_244\example_predictions.csv",
    eras=eras,
)
Validation plot Returned metrics
all eras validation plot all eras validation metrics

Numerai Signals evaluation metrics

Note: Since predictions are neutralized against Numerai's internal features before scoring, results from numereval.signalseval.run_analytics() do not represent exact diagnostics and live scores.

import numereval
from numereval.signalseval import run_analytics, score_signals

#after assigning predictions
train_era_scores = train_data.groupby(train_data.index).apply(score_signals)
test_era_scores = test_data.groupby(test_data.index).apply(score_signals)

train_scores = run_analytics(train_era_scores, plot=False)
test_scores = run_analytics(test_era_scores, plot=True)

Test correlation plot

Test cumulative correlation plot

train_scores test_scores
train_Scores test_Scores

Thanks to Jason Rosenfeld for allowing the run_analytics() to be integrated into the library.

Docs will be updated soon!

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

numereval-0.2.5.tar.gz (6.9 kB view hashes)

Uploaded Source

Built Distribution

numereval-0.2.5-py3-none-any.whl (7.5 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page