Wilt 0.0.2

REST HTTP API resource synchronisation

This feature allows synchronising collections of CI HTTP API resources in a SQLite database for analysis in SQL. The main use case is GitLab CI pipelines. Create a file called resmapfile.py like the following:

from wilt.resdb import resmap_api as rm


api = rm.Api(
    base_url='https://gitlab.com/api/v4/projects/4961127',
    default_headers={'PRIVATE-TOKEN': 'S3Cr3t'},
)

resources = [
    rm.Resmap(
        rm.SourceResourceCollection(
            '/jobs',
            request_params={'per_page': 100, 'sort': 'desc'},
            resource_id_key='id',
            resource_timestamp_key='created_at',
        ),
        rm.TargetTable(
            'job',
            extract_columns=[
                rm.ExtractColumn(
                    'id', rm.ColumnType.integer, name='job_id', pk=True
                ),
                rm.ExtractColumn('name', rm.ColumnType.text),
                rm.ExtractColumn('created_at', rm.ColumnType.datetime),
                rm.ExtractColumn('duration', rm.ColumnType.numeric),
                rm.ExtractColumn('status', rm.ColumnType.text),
                rm.ExtractColumn('pipeline.id', rm.ColumnType.integer),
            ],
        ),
        load_interval=rm.timedelta(days=2 * 365),
        sync_lookbehind=rm.timedelta(hours=6),
    ),
]

Then run wilt -v ci.rest sync-db to create and then synchronise a SQLite database with the HTTP API resources according to the mapping.

GitLab CI visualisation

Assuming the following pipeline synchronisation:

resources = [
    rm.Resmap(
        rm.SourceResourceCollection(
            '/pipelines',
            request_params={'per_page': 10, 'sort': 'desc'},
            page_size=10,
        ),
        rm.TargetTable(
            'pipeline_short',
            extract_columns=[
                rm.ExtractColumn(
                    'id', rm.ColumnType.integer, name='pipeline_short_id', pk=True
                ),
                rm.ExtractColumn('created_at', rm.ColumnType.datetime),
            ],
        ),
        subresources=[
            rm.SubResmap(
                rm.SourceSubresource('/pipelines/{id}', 'pipeline_id'),
                rm.TargetTable(
                    'pipeline',
                    extract_columns=[
                        rm.ExtractColumn(
                            'id', rm.ColumnType.integer, name='pipeline_id', pk=True
                        ),
                        rm.ExtractColumn('created_at', rm.ColumnType.datetime),
                        rm.ExtractColumn('duration', rm.ColumnType.numeric),
                        rm.ExtractColumn('ref', rm.ColumnType.text),
                    ],
                ),
            ),
        ],
    ),
]

the pipeline runtime can be visualised with a command like:

wilt ci.gitlab pipeline-runtime --clamp-max 3600 --after 2023-01-01T00:00:00

This produces plot.html file with interactive Plotly visualisation. See wilt ci.gitlab pipeline-runtime --help for more details.