Skip to main content

gokart

Test Python Versions

Gokart solves reproducibility, task dependencies, constraints of good code, and ease of use for Machine Learning Pipeline.

Documentation for the latest release is hosted on readthedocs.

About gokart

Here are some good things about gokart.

  • The following meta data for each Task is stored separately in a pkl file with hash value
    • task output data
    • imported all module versions
    • task processing time
    • random seed in task
    • displayed log
    • all parameters set as class variables in the task
  • Automatically rerun the pipeline if parameters of Tasks are changed.
  • Support GCS and S3 as a data store for intermediate results of Tasks in the pipeline.
  • The above output is exchanged between tasks as an intermediate file, which is memory-friendly
  • pandas.DataFrame type and column checking during I/O
  • Directory structure of saved files is automatically determined from structure of script
  • Seeds for numpy and random are automatically fixed
  • Can code while adhering to SOLID principles as much as possible
  • Tasks are locked via redis even if they run in parallel

All the functions above are created for constructing Machine Learning batches. Provides an excellent environment for reproducibility and team development.

Here are some non-goal / downside of the gokart.

  • Batch execution in parallel is supported, but parallel and concurrent execution of task in memory.
  • Gokart is focused on reproducibility. So, I/O and capacity of data storage can become a bottleneck.
  • No support for task visualize.
  • Gokart is not an experiment management tool. The management of the execution result is cut out as Thunderbolt.
  • Gokart does not recommend writing pipelines in toml, yaml, json, and more. Gokart is preferring to write them in Python.

Getting Started

Within the activated Python environment, use the following command to install gokart.

pip install gokart

If you use S3 or GCS as a data store, install the corresponding extras:

pip install gokart[s3]   # S3 support
pip install gokart[gcs]  # GCS support
pip install gokart[all]  # both S3 and GCS

Quickstart

Minimal Example

A minimal gokart tasks looks something like this:

import gokart

class Example(gokart.TaskOnKart):
    def run(self):
        self.dump('Hello, world!')

task = Example()
output = gokart.build(task)
print(output)

gokart.build return the result of dump by gokart.TaskOnKart. The example will output the following.

Hello, world!

Type-Safe Pipeline Example

We introduce type-annotations to make a gokart pipeline robust. Please check the following example to see how to use type-annotations on gokart. Type checkers that support dataclass_transform, such as mypy and pyright, check this example without any plugin. (The mypy plugin is deprecated and does not support this style.)

import gokart

# `gokart.TaskOnKart[str]` means that the task dumps `str`
class StrDumpTask(gokart.TaskOnKart[str]):
    def run(self):
        self.dump('Hello, world!')

# `gokart.TaskOnKart[int]` means that the task dumps `int`
class OneDumpTask(gokart.TaskOnKart[int]):
    def run(self):
        self.dump(1)

# `gokart.TaskOnKart[int]` means that the task dumps `int`
class TwoDumpTask(gokart.TaskOnKart[int]):
    def run(self):
        self.dump(2)

class AddTask(gokart.TaskOnKart[int]):
    # `a` requires a task to dump `int`
    a: gokart.TaskInstanceParameter[gokart.TaskOnKart[int]] = gokart.TaskInstanceParameter()
    # `b` requires a task to dump `int`
    b: gokart.TaskInstanceParameter[gokart.TaskOnKart[int]] = gokart.TaskInstanceParameter()

    def requires(self):
        return dict(a=self.a, b=self.b)

    def run(self):
        # loading by instance parameter,
        # `a` and `b` are treated as `int`
        # because they are declared as `gokart.TaskOnKart[int]`
        a = self.load(self.a)
        b = self.load(self.b)
        self.dump(a + b)


valid_task = AddTask(a=OneDumpTask(), b=TwoDumpTask())
# the next line will show type error by type checkers
# because `StrDumpTask` dumps `str` and `AddTask` requires `int`
invalid_task = AddTask(a=OneDumpTask(), b=StrDumpTask())

This is an introduction to some of the gokart. There are still more useful features.

Please See Documentation .

Have a good gokart life.

Achievements

Gokart is a proven product.

Thanks

gokart is a wrapper for luigi. Thanks to luigi and dependent projects!

Metadata

Release files for gokart 1.13.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for gokart 1.13.0
File Size Uploaded Provenance Actions
gokart-1.13.0.tar.gz 90.1 kB Without provenance Details for gokart-1.13.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for gokart 1.13.0
File Interpreter ABI Platform Actions
gokart-1.13.0-py3-none-any.whl Python 3 none any Details for gokart-1.13.0-py3-none-any.whl

Total release size: 165.6 kB

Release files / gokart-1.13.0.tar.gz

Download URL gokart-1.13.0.tar.gz
Size 90.1 kB
Tags Source
SHA-256 checksum
How to use checksums
db04885c7d8aa49a2f40530e29beb97e19f0f59f2d0b9893e3a29d7745b5f338
BLAKE2b-256 checksum
How to use checksums
cccc3aa01e1307297d9a4a7233c7596736fa37739ccdb2b21fbee21be8c34af7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / gokart-1.13.0-py3-none-any.whl

Download URL gokart-1.13.0-py3-none-any.whl
Size 75.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
f6cfd1ce6299e2a7f0efbcd38f37c91ddbb6b144233e29546bb1492e1b02ba9f
BLAKE2b-256 checksum
How to use checksums
9c809c426e8c131a60523ca6668be42443258cd0308bfd49ee940ca3dee45ac5
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for gokart 1.13.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.13.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page