trueloss 0.2.0

A Keras utility for measuring training loss and metrics in evaluation mode for clearer comparison with validation performance.

trueloss

trueloss is a Python library designed to compute and plot training loss and metrics on the training data under evaluation conditions, allowing them to be compared more directly with validation performance in Keras models.

The library was created to address a recurring observation in machine learning experiments: in some training runs, the model reports a higher loss on the training data than on the validation data.

This can occur because training and validation losses are often computed under different conditions. During training, the reported loss may include the data loss, regularization penalties, and the effects of training-only model behaviour. Validation, by contrast, is performed in evaluation mode.

As a simplified example:

1. Training objective

$$ \mathcal{L}_{\text{training}}

\mathcal{L}{\text{data}} + \mathcal{L}{\text{regularization}} $$

For L2 regularization:

$$ \mathcal{L}_{\text{training}}

\mathcal{L}_{\text{data}} + \lambda \lVert \theta \rVert^2 $$

2. Comparable training-data evaluation

$$ \mathcal{L}_{\text{trueloss}}

\mathcal{L}_{\text{data}} \quad \text{evaluated on the training data in evaluation mode} $$

As a result, the conventional training and validation curves may not always represent directly comparable quantities.

trueloss makes this distinction visible by measuring the model's performance on the training data using the same evaluation conditions applied to validation data. It does not replace the training loss reported by Keras. Instead, it adds a comparable loss and corresponding metrics to the existing Keras training history.

Version 0.2.0 adds compatibility with Keras 3 while preserving the existing trueloss workflow. It also extends support beyond classification to regression problems, including regression components used in applications such as bounding-box prediction in object detection.

Features

• Compatible with Keras 3.
• Supports both classification and regression workflows.
• Computes training loss under evaluation conditions for clearer comparison with validation loss.
• Computes corresponding evaluation-mode values for a wider range of Keras metrics.
• Supports regression components used in applications such as object detection.
• Plots training, validation, and comparable training-data performance.
• Accepts the same fit() parameters and defaults as Keras model.fit().
• Returns the same Keras History instance produced by model.fit().
• Adds base_loss and corresponding base_<metric_name> values to the training history.
• Preserves the behaviour of the original Keras model and existing training workflow.

Installation

Install trueloss from PyPI:

pip install trueloss

To install version 0.2.0 explicitly:

pip install trueloss==0.2.0

Usage

Basic Usage

Import TensorFlow and the trueloss class:

import tensorflow as tf
from trueloss import trueloss

The examples below use Keras with the TensorFlow backend.

Creating and Compiling a Model

Create and compile the Keras model as usual:

model = tf.keras.models.Sequential([
    tf.keras.layers.Input(shape=(784,)),
    tf.keras.layers.Dense(64, activation="relu"),
    tf.keras.layers.Dense(10, activation="softmax"),
])

model.compile(
    optimizer="adam",
    loss="sparse_categorical_crossentropy",
    metrics=["accuracy"],
)

Training the Model with trueloss

Create a trueloss instance and call its fit() method using the same parameters that you would pass to model.fit():

true_loss_instance = trueloss(
    model=model,
    plot=True,
)

history = true_loss_instance.fit(
    x_train,
    y_train,
    validation_data=(x_val, y_val),
    epochs=10,
    batch_size=32,
    verbose=1,
)

The public training workflow remains unchanged:

history = true_loss_instance.fit(...)

can be used in place of:

history = model.fit(...)

All positional and keyword arguments are passed through using the Keras fit() interface.

Training History

The object returned by true_loss_instance.fit() is the same Keras History instance returned by model.fit().

For a model compiled with accuracy as a metric, the history includes:

history.history["loss"]
history.history["accuracy"]

history.history["val_loss"]
history.history["val_accuracy"]

history.history["base_loss"]
history.history["base_accuracy"]

The additional entries represent performance on the training data measured under evaluation conditions.

For models using other metrics, trueloss adds the corresponding values using the base_ prefix. For example:

history.history["base_mae"]
history.history["base_mse"]

The exact metric names depend on the metrics supplied when compiling the Keras model.

Important Notes

Model Behaviour

The original model instance continues to behave normally after training through trueloss.

You can continue to use it for prediction:

predictions = model.predict(x_test)

evaluation:

results = model.evaluate(x_test, y_test)

saving:

model.save("model.keras")

and any other standard Keras operation.

Keras Integration

trueloss.fit() accepts the same training inputs and configuration parameters as model.fit(), including positional and keyword arguments.

Existing callbacks can still be passed normally:

history = true_loss_instance.fit(
    x_train,
    y_train,
    validation_data=(x_val, y_val),
    epochs=10,
    callbacks=[early_stopping, model_checkpoint],
)

Verbose Output

The normal Keras training output is preserved:

history = true_loss_instance.fit(
    x_train,
    y_train,
    epochs=10,
    verbose=1,
)

trueloss Is an Additional Signal

base_loss does not replace the loss reported by Keras.

The two values answer different questions:

• loss represents the loss reported during the optimization process.
• base_loss represents performance on the training data under evaluation conditions.

Keeping both values makes it easier to distinguish optimization behaviour from measured predictive performance.

Regression Usage

Version 0.2.0 extends support beyond classification models.

A regression model can be compiled and trained in the same way:

regression_model = tf.keras.models.Sequential([
    tf.keras.layers.Input(shape=(20,)),
    tf.keras.layers.Dense(64, activation="relu"),
    tf.keras.layers.Dense(32, activation="relu"),
    tf.keras.layers.Dense(1),
])

regression_model.compile(
    optimizer="adam",
    loss="mean_squared_error",
    metrics=["mean_absolute_error"],
)

true_loss_instance = trueloss(
    model=regression_model,
    plot=True,
)

history = true_loss_instance.fit(
    x_train,
    y_train,
    validation_data=(x_val, y_val),
    epochs=20,
    batch_size=32,
)

The resulting history includes values such as:

history.history["loss"]
history.history["val_loss"]
history.history["base_loss"]

history.history["mean_absolute_error"]
history.history["val_mean_absolute_error"]
history.history["base_mean_absolute_error"]

The same approach can be used with regression components in more complex models, including bounding-box coordinate regression in object-detection workflows.

Using a Dataset

Keras datasets and data loaders can be passed directly to fit():

train_dataset = tf.data.Dataset.from_tensor_slices(
    (x_train, y_train)
).batch(32)

validation_dataset = tf.data.Dataset.from_tensor_slices(
    (x_val, y_val)
).batch(32)

history = true_loss_instance.fit(
    train_dataset,
    validation_data=validation_dataset,
    epochs=10,
)

Plotting Only

To plot an existing Keras history without fitting a model, initialize trueloss without a model and call plot_fn() directly:

true_loss_instance = trueloss(plot=True)

true_loss_instance.plot_fn(history)

Complete Classification Example

import tensorflow as tf
from trueloss import trueloss


# Load and preprocess the data
(x_train, y_train), (x_val, y_val) = (
    tf.keras.datasets.mnist.load_data()
)

x_train = x_train.astype("float32") / 255.0
x_val = x_val.astype("float32") / 255.0


# Create and compile the model
model = tf.keras.models.Sequential([
    tf.keras.layers.Input(shape=(28, 28)),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(128, activation="relu"),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(10, activation="softmax"),
])

model.compile(
    optimizer="adam",
    loss="sparse_categorical_crossentropy",
    metrics=["accuracy"],
)


# Initialize trueloss
true_loss_instance = trueloss(
    model=model,
    plot=True,
)


# Train the model
history = true_loss_instance.fit(
    x_train,
    y_train,
    validation_data=(x_val, y_val),
    epochs=10,
    batch_size=32,
    verbose=1,
)


# Access the additional values
print(history.history["base_loss"])
print(history.history["base_accuracy"])

You can also view the example notebook.

Project Links

• PyPI
• Source code
• Issue tracker
• Example notebook

Contributing

Contributions are welcome.

To suggest an improvement, report unexpected behaviour, or contribute a fix, open an issue or submit a pull request through the GitHub repository.

Technical reviews, edge cases, and compatibility reports from research and production workflows are particularly valuable.

License

This project is licensed under the MIT License. See the LICENSE file for details.

Contact

For questions, feedback, or support, use the GitHub issue tracker or contact Siddique Abusaleh at [e-mail].

Acknowledgements

trueloss is built for Keras and TensorFlow workflows. Thanks to the contributors and maintainers of these projects for their work on the machine-learning ecosystem.

Citation

If you use trueloss in research, please consider citing it:

@misc{trueloss2026,
  author       = {Siddique Abusaleh},
  title        = {trueloss: Comparable Training Loss and Metrics for Keras Models},
  year         = {2026},
  publisher    = {GitHub},
  journal      = {GitHub repository},
  howpublished = {\url{https://github.com/trueloss/trueloss}}
}

By using trueloss, you can preserve the standard Keras training workflow while gaining a clearer view of model performance on the training data.