pureskillgg-csgo-dsdk 3.1.0

What it does

The package (pureskillgg_csgo_dsdk) exposes exactly five public symbols from its top-level __init__:

• scrub_csds_pii - anonymize a CSDS match (manifest + per-channel DataFrames) in place.

• pop_overtime - remove rows for rounds past regular time from a channel DataFrame.

• SCRUB_CSDS_PII_CHANNEL_INSTRUCTIONS - a constant list of the CSDS channels a caller should load before scrubbing.

• csds_pii_channel_instructions - that list filtered to the channels a given manifest actually has. Load through this, not the raw constant.

• MissingColumns / UnsupportedChannelStructure - exceptions raised when a DataFrame lacks a required column.

The scrubber (scrubber/scrub_pii.py) rewrites a CSDS match in place. In the manifest it replaces jobId with the anonymous id everywhere (via a rapidjson dump / string-replace / load round-trip), redacts sharecode, demoId and metadata.bucket, and lowers matchDate precision to minutes. In the channel DataFrames it redacts identifying columns (sharecode and demo_id in header; name_new / name_old in player_name; name / clan_tag in player_personal), maps each unique steam_id to a letter (A, B, C …), zeroes ping in player_status, redacts text in player_chat, and caps inflated stats (player_info wins and rank_update win_count — the same quantity — over 2500 to 2501; each of commends_friendly / commends_leader / commends_teacher over 100 to 101). Every mutation also tags the corresponding manifest column origin with -redacted or -capped and flips that channel’s redacted flag.

The overtime filter (overtime/pop_overtime.py) drops rows whose round exceeds max_rounds_csgo (default 30) from any channel DataFrame that has a round column, mutating the input in place and returning the removed overtime rows. It raises MissingColumns if there is no round column. There is no automatic short-match detection; a caller may pass max_rounds_csgo=16 manually for short matches.

For a field-by-field breakdown of the scrubber’s redact-vs-cap rules and its manifest bookkeeping, see the deep dive linked under Documentation.

Pipeline role

This is a downstream-consumed helper library, not a stage in the match pipeline. It operates on CSDS channel data - the per-match DataFrames plus manifest produced upstream by the demo replay / CSDS parsing stage (csgo-rushb / csds).

Data-science and analytics code imports it to anonymize match data before it leaves the platform and to strip overtime rounds before modeling. Consumers include csgo-datascience, the coach / PPP / match-conversion pipeline (csgo-ppp, csgo-coach, csgo-progression), and dataset-export / Data-Exchange tooling. Any actual AWS I/O (reading the parsed match DataFrames from S3) is performed by those consuming repos through pureskillgg-dsdk readers/loaders, not here.

Public API

These are the confirmed exported symbols. They are the library’s only “jobs”; this package owns no AWS resources, queues, tables, or log groups.

• scrub_csds_pii(manifest, data) - anonymize a CSDS match in place. Replaces jobId with the anonymous id throughout the manifest, redacts sharecode / demoId / metadata.bucket, lowers matchDate to minute precision, redacts the identifying name/clan/sharecode/demo columns, maps steam_id values to letters, zeroes pings, caps inflated wins/commends, and flags every affected manifest channel/column as redacted or capped. Returns the (rewritten) manifest; the data dict is mutated in place.

• pop_overtime(df, *, max_rounds_csgo=30) - remove rows for rounds past regular time from a channel DataFrame (in place) and return the removed overtime rows. Raises MissingColumns if there is no round column.

• SCRUB_CSDS_PII_CHANNEL_INSTRUCTIONS - list of channel descriptors (player_name, header, player_personal, player_info, player_status, player_chat, rank_update) telling callers which CSDS channels to load before running scrub_csds_pii.

• csds_pii_channel_instructions(manifest) - the same list, filtered to the channels present in manifest. get_channels raises on a channel missing from the manifest, so a CSDS written before a channel existed fails if the raw constant is passed through; filter with this instead:

  data = loader.get_channels(csds_pii_channel_instructions(loader.manifest))
  manifest = scrub_csds_pii(loader.manifest, data)

• MissingColumns / UnsupportedChannelStructure - error types signaling a DataFrame lacks a required column. MissingColumns subclasses UnsupportedChannelStructure and carries the offending column names on its columns attribute.

Logs and observability

This is a pure Python library and owns no cloud resources, so there is nothing to find in CloudWatch. It has no AWS-SDK calls, no DynamoDB / SQS / SNS / S3 / Lambda / Step Functions usage, no DLQs, no Sentry integration, and no LOG_LEVEL handling. (The only bucket token in the code overwrites the manifest’s metadata.bucket field with the literal string redacted during scrubbing - it is not a real S3 bucket reference.)

• Runtime behavior: failures surface only as raised Python exceptions (MissingColumns / UnsupportedChannelStructure from this library, plus a RuntimeError if a manifest channel or column referenced during scrubbing is missing) propagated to whatever process imports the library. To find the logs for a real match run, look at the consuming service (for example csgo-datascience or the PPP/coach pipeline), not at this repo.

• CI/CD: the only observable surface owned by this repo is GitHub Actions (main test, format, publish to PyPI, version git-tag trigger). Failures show up as red GitHub Actions runs, not CloudWatch.

Documentation

• docs/scrub-csds-pii.md - deep dive on scrub_csds_pii: the exact fields redacted vs. capped, the steam_id -> letter mapping, the jobId -> id manifest round-trip, and the manifest channel/column bookkeeping. This is the privacy-critical path for any data leaving the platform (for example Data Exchange / academic datasets), so the precise rules are worth knowing before relying on it.

Quickstart

$ git clone https://github.com/pureskillgg/csgo-dsdk.git
$ git lfs install
$ git lfs pull
$ cd csgo-dsdk
$ uv sync

Run each command below in a separate terminal window:

$ make watch

Primary development tasks are defined in the Makefile.

Source Code

The source code is hosted on GitHub. Clone the project with

$ git clone https://github.com/pureskillgg/csgo-dsdk.git
$ git lfs install
$ git lfs pull

Requirements

You will need Python 3 and uv.

Install the development dependencies with

$ uv sync

Tests

Lint code with

$ make lint

Run tests with

$ make test

Run tests on changes with

$ make watch

Publishing

Use the uv version command to release a new version. Then run make version to commit and push a new git tag which will trigger a GitHub action.

Publishing may be triggered using on the web using a workflow_dispatch on GitHub Actions.

Secrets for Optional GitHub Actions

The version and format GitHub actions require a user with write access to the repository including access to read and write packages. Set these additional secrets to enable the action:

• GH_USER: The GitHub user’s username.

• GH_TOKEN: A personal access token for the user.

• GIT_USER_NAME: The name to set for Git commits.

• GIT_USER_EMAIL: The email to set for Git commits.

• GPG_PRIVATE_KEY: The GPG private key.

• GPG_PASSPHRASE: The GPG key passphrase.