Skip to main content

Tools for working with Crash Stats (https://crash-stats.mozilla.org/)

Project description

Command line tools and library for interacting with Crash Stats (https://crash-stats.mozilla.org/).

Code:https://github.com/willkg/crashstats-tools
Documentation:Check the README.rst file
Changelog:Check the HISTORY.rst file
Issue tracker:https://github.com/willkg/crashstats-tools/issues
License:MPLv2
Status:Alpha
Community Participation Guidelines:
 Guidelines

Installing

crashstats-tools is available on PyPI. You can install it with:

$ pip install crashstats-tools

However, since it has commands that you want to use, you probably want to install it with pipx:

$ pipx install crashstats-tools

Support

This is an alpha-quality project that I spent some time on because it seemed helpful. If you have any problems, please write up an issue in the issue tracker and I’ll get to it when I’m able.

If you use this, like it, appreciate it, or have any positive feeling, please give it a GitHub star. That helps me:

  1. have a proxy for knowing whether it’s being used
  2. prioritize my time working on this project
  3. have a proxy for a list of users in case I have questions and need to ask advise

Tools

supersearch

Perform Super Search queries on Crash Stats.

Fetch 10 crash ids for Firefox:

$ supersearch --num=10 --product=Firefox

Fetch 57 crash ids that match a Super Search query:

$ supersearch --num=57 --supersearch-url="https://crash-stats.mozilla.org/search/?release_channel=nightly&version=70.0a1&product=Firefox&_sort=-date"

Fetch uuid, product, version, and build_id for crash reports that have “OOM” in the signature:

$ supersearch --_columns=uuid --_columns=product --_columns=version \
    --_columns=build_id --signature="~OOM"

Results are formatted as tab-delimited by default. JSON output is also available.

Note that this doesn’t support Super Search aggregations.

See Super Search API details:

fetch-data

Fetch data for specified crash reports.

This lets you download raw crash, dumps, and processed crash from Crash Stats.

Fetch processed crash data for specific crash id:

$ fetch-data --no-raw --no-dumps --processed 723cacd6-1684-420e-a1c7-f04240190731

Fetch raw crash data using supersearch command to generate crash ids:

$ supersearch --product=Firefox --num=10 | \
    fetch-data --raw --no-dumps --no-processed crashdir

reprocess

Let’s you specify crash reports for reprocessing.

Reprocess an individual crash report:

$ reprocess 723cacd6-1684-420e-a1c7-f04240190731

Reprocess crash reports based on a supersearch:

$ supersearch --num=5 | reprocess

Note

The reprocess command requires that you set CRASHSTATS_API_TOKEN in your environment with an API token that has the “Reprocess Crashes” permission.

API token

For supersearch and fetch-data, you need to use a API token to:

  • download data containing personally identifiable information
  • download security sensitive data
  • get out from the shadow of extreme API use rate limiting

You need an API token for reprocess–it doesn’t work without one.

If you have access, you can generate an API token here:

https://crash-stats.mozilla.org/api/tokens/

Once you have acquired one, set the CRASHSTATS_API_TOKEN environment variable when using crashstats-tools commands.

Remember to abide by the data access policy when using data from Crash Stats! The policy is specified here:

https://crash-stats.mozilla.org/documentation/memory_dump_access/

Use cases

These tools are helpful when downloading data for analysis as well as downloading data to test other tools with.

Example 1

I want to collect a bunch of crash report data to look at possible values of an annotation in Firefox crash reports that’s not available in Super Search, yet.

Since I’m looking just at annotations, all I need is the raw crash.

I would do something like this:

$ mkdir crashdata
$ supersearch --product=Firefox --num=1000 | \
    fetch-data --raw --no-dumps --no-processed crashdata

Then I can use jq or whatever to look at the crash report data in crashdata/raw_crash/.

Example 2

I want to test out a new JIT analysis tool that works on minidump files.

I would write a script like this:

#!/bin/bash

CRASHSTATS_API_TOKEN=foo
DATADIR=./crashdata
CRASHIDS=$(supersearch --product=Firefox --num=1000)

mkdir -p "${DATADIR}"

for crashid in ${CRASHIDS}
do
    echo "crashid ${crashid}"
    fetch-data --raw --dumps --no-processed "${DATADIR}" "${crashid}"

    # Not all crash reports have dumps--we only want to run analysis
    # on the ones that do.
    if [[ -e "crashdata/dump/${crashid}" ]]
    then
        echo "analyze dump ${crashid}..."
        # run my tool on the dump
    fi
done

Example 3

I want to get a list of crash ids for today (2019-07-30) where DOMFissionEnabled exists in the crash report.

I would do this:

$ supersearch --date=">=2019-07-30" --date="<2019-07-31" --dom_fission_enabled="!__null__"

Release process

  1. Create branch

  2. Update version and release date in crashstats_tools/__init__.py

  3. Update HISTORY.rst

  4. Push the branch, create a PR, review it, merge it

  5. Create a signed tag, push to github:

    git tag -s v0.1.0
    git push --tags REMOTE TAGNAME
    
  6. Build:

    python setup.py sdist bdist_wheel
    

    Make sure to use Python 3 with an updates requirements-dev.txt.

  7. Upload to PyPI:

    twine upload dist/*
    

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for crashstats-tools, version 1.0.2
Filename, size File type Python version Upload date Hashes
Filename, size crashstats_tools-1.0.2-py3-none-any.whl (20.8 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size crashstats-tools-1.0.2.tar.gz (21.1 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page