trueloss 0.2.1

A Keras utility for measuring training loss and metrics in evaluation mode for clearer comparison with validation performance.

trueloss

trueloss is a Python library for measuring and plotting a Keras model's loss and metrics on the training data under evaluation conditions.

It adds these values to the normal Keras training history, making training-data performance more directly comparable with validation performance.

Why trueloss?

Training loss and validation loss are not always measured under identical conditions.

During training:

• the model operates in training mode;
• dropout and other training-only behaviour may be active;
• model weights change between batches;
• the epoch loss is accumulated throughout optimization.

Validation is measured in evaluation mode using the model weights available after the epoch.

trueloss evaluates the model on the training data after every epoch using:

model.evaluate(...)

The results are added to the Keras history with the base_ prefix:

loss
=
loss reported during training

base_loss
=
loss evaluated on the training data after the epoch

val_loss
=
loss evaluated on the validation data

Because base_loss and val_loss are both measured under evaluation conditions, they are generally more directly comparable.

However, base_loss is not guaranteed to contain only the data loss. Keras evaluation may still include regularizers or losses added through add_loss(), depending on the model.

trueloss does not replace the normal Keras training loss. It provides an additional view of training-data performance.

Features

• Supports classification and regression models.
• Adds base_loss and base_<metric_name> values to the Keras history.
• Supports multiple compiled metrics.
• Automatically creates separate plots for loss and each detected metric.
• Supports arrays, TensorFlow datasets, and other Keras-compatible inputs.
• Returns the same Keras History object produced by model.fit().
• Supports TensorFlow Keras and Keras 3-based TensorFlow installations.

Installation

pip install trueloss

Basic Usage

import tensorflow as tf
from trueloss import trueloss

Create and compile the model normally:

model = tf.keras.models.Sequential([
    tf.keras.layers.Input(shape=(784,)),
    tf.keras.layers.Dense(64, activation="relu"),
    tf.keras.layers.Dense(10, activation="softmax"),
])

model.compile(
    optimizer="adam",
    loss="sparse_categorical_crossentropy",
    metrics=["accuracy"],
)

Create a trueloss instance and use its fit() method:

true_loss_instance = trueloss(
    model=model,
    plot=True,
)

history = true_loss_instance.fit(
    x=x_train,
    y=y_train,
    validation_data=(x_val, y_val),
    epochs=10,
    batch_size=32,
    verbose=1,
)

Instead of:

history = model.fit(...)

use:

history = true_loss_instance.fit(...)

The returned object is the normal Keras History object.

Using Multiple Metrics

You can compile the model with multiple compatible metrics:

model.compile(
    optimizer="adam",
    loss="sparse_categorical_crossentropy",
    metrics=[
        "accuracy",
        tf.keras.metrics.SparseTopKCategoricalAccuracy(
            k=3,
            name="top_3_accuracy",
        ),
    ],
)

trueloss detects the available metric names automatically. No metric name is hard-coded into the plotting function.

Recommended Parameter Style

Positional arguments are supported:

history = true_loss_instance.fit(
    x_train,
    y_train,
    epochs=10,
)

However, keyword arguments are recommended, especially when using several fit() parameters:

history = true_loss_instance.fit(
    x=x_train,
    y=y_train,
    validation_data=(x_val, y_val),
    epochs=10,
    batch_size=32,
    verbose=1,
)

If parameters do not behave as expected through the wrapper, use explicit keyword arguments instead of passing many optional parameters positionally.

Callbacks

When callbacks are required, pass them as a list:

history = true_loss_instance.fit(
    x=x_train,
    y=y_train,
    epochs=10,
    callbacks=[
        early_stopping,
        model_checkpoint,
    ],
)

When callbacks are not required, omit the parameter.

Do not pass:

callbacks=None

with the current version of trueloss.

Training History

For a model compiled with accuracy, the history may contain:

history.history["loss"]
history.history["accuracy"]

history.history["val_loss"]
history.history["val_accuracy"]

history.history["base_loss"]
history.history["base_accuracy"]

For multiple metrics, corresponding base_ values are added:

history.history["base_loss"]
history.history["base_accuracy"]
history.history["base_top_3_accuracy"]

For regression models, the history may contain:

history.history["base_mean_absolute_error"]
history.history["base_mean_squared_error"]

View all available values with:

print(history.history.keys())

Automatic Plotting

Set plot=True to display the plots automatically:

true_loss_instance = trueloss(
    model=model,
    plot=True,
)

The plotting function:

• creates one plot for loss;
• detects every additional metric;
• creates a separate subplot for each metric;
• plots training, validation, and base_ values when available.

For example, a model using accuracy and top-3 accuracy can produce:

Loss
Accuracy
Top 3 Accuracy

Manual Plotting

For greater control, disable automatic plotting:

true_loss_instance = trueloss(
    model=model,
    plot=False,
)

Train the model and inspect the history:

history = true_loss_instance.fit(
    x=x_train,
    y=y_train,
    validation_data=(x_val, y_val),
    epochs=10,
)

print(history.history.keys())

You can then create a custom plot:

import matplotlib.pyplot as plt

epochs = range(1, len(history.history["loss"]) + 1)

plt.plot(
    epochs,
    history.history["loss"],
    label="Training Loss",
)

plt.plot(
    epochs,
    history.history["base_loss"],
    label="Training-Data Evaluation Loss",
)

if "val_loss" in history.history:
    plt.plot(
        epochs,
        history.history["val_loss"],
        label="Validation Loss",
    )

plt.xlabel("Epoch")
plt.ylabel("Loss")
plt.legend()
plt.show()

This gives full control over the selected values, labels, colours, and layout.

Important Notes

Prefer Explicit Validation Data

Using explicit validation data is recommended:

history = true_loss_instance.fit(
    x=x_train,
    y=y_train,
    validation_data=(x_val, y_val),
    epochs=10,
)

Be careful when using:

validation_split=0.2

Keras creates this split internally, while the current trueloss callback receives the original input arrays. The base_ values may therefore be evaluated on the complete original data rather than only the portion used for training.

Dataset Inputs

A TensorFlow dataset can be passed directly:

train_dataset = tf.data.Dataset.from_tensor_slices(
    (x_train, y_train)
).batch(32)

validation_dataset = tf.data.Dataset.from_tensor_slices(
    (x_val, y_val)
).batch(32)

history = true_loss_instance.fit(
    x=train_dataset,
    validation_data=validation_dataset,
    epochs=10,
)

The training dataset is evaluated again after every epoch, so it should be finite and reusable.

Regression Usage

Regression models use the same workflow:

regression_model = tf.keras.models.Sequential([
    tf.keras.layers.Input(shape=(20,)),
    tf.keras.layers.Dense(64, activation="relu"),
    tf.keras.layers.Dense(32, activation="relu"),
    tf.keras.layers.Dense(1),
])

regression_model.compile(
    optimizer="adam",
    loss="mean_squared_error",
    metrics=[
        "mean_absolute_error",
        "mean_squared_error",
    ],
)

true_loss_instance = trueloss(
    model=regression_model,
    plot=True,
)

history = true_loss_instance.fit(
    x=x_train,
    y=y_train,
    validation_data=(x_val, y_val),
    epochs=20,
    batch_size=32,
)

The history may include:

history.history["loss"]
history.history["val_loss"]
history.history["base_loss"]

history.history["mean_absolute_error"]
history.history["val_mean_absolute_error"]
history.history["base_mean_absolute_error"]

history.history["mean_squared_error"]
history.history["val_mean_squared_error"]
history.history["base_mean_squared_error"]

The plotting function automatically creates separate plots for loss, mean absolute error, and mean squared error.

This can also be used with regression outputs in more complex models, including bounding-box coordinate regression, provided the model works normally with model.evaluate().

Plotting an Existing History

An existing compatible Keras history can be plotted directly:

true_loss_instance = trueloss()

true_loss_instance.plot_fn(history)

The function reads the available values from:

history.history

Complete Classification Example

import tensorflow as tf
from trueloss import trueloss


# Load and preprocess the data
(x_train, y_train), (x_val, y_val) = (
    tf.keras.datasets.mnist.load_data()
)

x_train = x_train.astype("float32") / 255.0
x_val = x_val.astype("float32") / 255.0


# Create the model
model = tf.keras.models.Sequential([
    tf.keras.layers.Input(shape=(28, 28)),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(128, activation="relu"),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(10, activation="softmax"),
])


# Compile the model
model.compile(
    optimizer="adam",
    loss="sparse_categorical_crossentropy",
    metrics=[
        "accuracy",
        tf.keras.metrics.SparseTopKCategoricalAccuracy(
            k=3,
            name="top_3_accuracy",
        ),
    ],
)


# Initialize trueloss
true_loss_instance = trueloss(
    model=model,
    plot=True,
)


# Train the model
history = true_loss_instance.fit(
    x=x_train,
    y=y_train,
    validation_data=(x_val, y_val),
    epochs=10,
    batch_size=32,
    verbose=1,
)


# View available values
print(history.history.keys())


# Access the additional values
print(history.history["base_loss"])
print(history.history["base_accuracy"])
print(history.history["base_top_3_accuracy"])

View the example notebook.

Project Links

• PyPI
• Source code
• Issue tracker
• Example notebook

Contributing

Contributions, compatibility reports, bug reports, and pull requests are welcome through the GitHub repository.

License

This project is licensed under the MIT License. See the LICENSE file for details.

Contact

For questions or support, use the GitHub issue tracker or contact Siddique Abusaleh at [e-mail].

Citation

@misc{trueloss,
  author       = {Siddique Abusaleh},
  title        = {trueloss: Comparable Training Loss and Metrics for Keras Models},
  year         = {2026},
  publisher    = {GitHub},
  journal      = {GitHub repository},
  howpublished = {\url{https://github.com/trueloss/trueloss}}
}