sysstra-logging

Shared structured (JSON-line) logging for Sysstra services. Stdlib-only — zero install dependencies — so lean scripts (data-collection streamers, cron jobs) can pick it up without pulling in the full sysstra SDK (pandas/numpy/numba/redis/pymongo).

from sysstra_logging import get_logger, bind, clear_context

logger = get_logger("syss_sha_eios", log_file="logs/syss_sha_eios.log",
                    service="sysstra-trading-strategies")

bind(mode="lt", request_id=request_id, strategy="syss_sha_eios")
logger.info("order placed", extra={"order_id": order_id})

The identity block

Every line carries these 15 fields, in this order, blank when unknown — whether or not the writing service knows anything about them:

ts  level  schema  service  component  type  env  region  host  tenant  market  event  status  logger  message

That is what lets one Alloy config and one dashboard panel work across every repo: a consumer can group by repo, host, market or environment without knowing which service wrote the line.

• service names the repo (sysstra-data-collection-in) and comes from each repo's own SERVICE constant in its *_common.py. SERVICE in the environment overrides it; the logger name is the last-resort fallback.
• logger is the per-script or per-request logger name (syss_sha_eod-vt-<request_id>). It used to be passed into service's slot, so every line reported a request id as its service and nothing could group by repo.
• component says which part of a repo a line came from (streamer, common_runner, instance-orchestrator) — bind it where a repo has meaningful sub-parts, or set SERVICE_COMPONENT for a process-wide default.
• type is the process kind (api, worker, collector, controller) — falls back to SERVICE_TYPE.
• env defaults to prod; region is the deployment region (ap-south-1) — falls back to REGION.
• host falls back to socket.gethostname() because HOSTNAME is a shell variable and is often not exported.

Precedence for each field: extra= → bind() → environment → default. Env var mapping for every identity field lives in _ENV_FIELDS (logging_utils.py) — service→SERVICE, component→SERVICE_COMPONENT, type→SERVICE_TYPE, env→ENVIRONMENT, region→REGION, host→HOSTNAME, tenant→TENANT, market→MARKET.

Repo-wrapper helpers

Every repo's *_common.py has a create_logger(file_name, logger_name, stream=...) that used to hand-roll LOG_DIR resolution, level resolution, and the service-subdirectory path join. That's centralized here:

from sysstra_logging import create_service_logger

def create_logger(file_name, logger_name=__name__, stream=False):
    return create_service_logger(file_name, logger_name, SERVICE, DEFAULT_LOG_DIR, stream=stream)

create_service_logger() resolves LOG_DIR/LOG_LEVEL from the environment, joins SERVICE as a subdirectory, creates it if needed, and warns once (via check_service_name_drift) if a SERVICE_NAME env var is set and disagrees with the service argument — SERVICE_NAME is documentation/external-tooling surface only, never the source of truth. Data-collection repos, which prefix log filenames with a segment name, use resolve_segmented_log_file() instead — see logging_utils.py.

Domain fields

Everything else appears only where it applies — request_id, mode, strategy on the trading repos; segment, session on data collection; job, run_id on batch work. They are deliberately not blank-padded: a request_id: "" on a collection line would be indistinguishable from a trading line that failed to populate one, and collection has no such concept.

request_id is the Mongo {mode}_requests._id and is never generated. Batch jobs get run_id instead, so a query for one can never pick up the other.

Batch jobs

from sysstra_logging import get_logger, job_run

logger = get_logger("s3_sync", log_file=..., service="sysstra-data-services")

with job_run(logger, "s3_sync"):
    sync_bucket()

Emits job_start / job_complete / job_failed with duration_s, and binds job and run_id for the duration so every line the job logs in between is correlated too. Failures re-raise, so a cron job still exits non-zero.

Notes

Set LOG_FORMAT=text for a human-readable formatter during local dev / tail -f; the default is single-line JSON.

clear_context() must be called at the start of every unit of work (Celery task, HTTP request) that reuses a process/thread — otherwise bound fields from a previous task (most dangerously mode, paper vs. real-money) leak into the next one's log lines.

See sysstra_logging/logging_utils.py for the full API (bound(), get_context(), redirect_stdout_to(), ScrubFilter).