jp-idwr-db

jp-idwr-db makes Japan's IDWR infectious disease surveillance data easier to use.

It collects the official NIID/JIHS files, cleans and combines data published across different formats and archives, and releases analysis-ready weekly data as Parquet. You can use it through the Python package or read the release files directly with DuckDB, R, Arrow, Spark, or any other tool that supports Parquet.

IDWR reports are published weekly. jp-idwr-db checks for new data every week and publishes an updated, versioned snapshot via GitHub Actions and releases.

Install

pip install jp-idwr-db

Quick start

Load the combined dataset:

import jp_idwr_db as jp

df = (
    jp.load("unified", version="latest")
    .select(["date", "prefecture", "category", "disease", "count", "source"])
)

print(df)

shape: (6_201_180, 6)
┌────────────┬────────────┬──────────┬────────────────────────────────┬───────┬───────────────────────┐
│ date       ┆ prefecture ┆ category ┆ disease                        ┆ count ┆ source                │
│ ---        ┆ ---        ┆ ---      ┆ ---                            ┆ ---   ┆ ---                   │
│ date       ┆ str        ┆ str      ┆ str                            ┆ f64   ┆ str                   │
╞════════════╪════════════╪══════════╪════════════════════════════════╪═══════╪═══════════════════════╡
│ 1999-04-05 ┆ Aichi      ┆ total    ┆ AIDS                           ┆ 0.0   ┆ Confirmed cases       │
│ 1999-04-05 ┆ Aichi      ┆ total    ┆ Acute encephalitis (excluding… ┆ 0.0   ┆ Sentinel surveillance │
│ 1999-04-05 ┆ Aichi      ┆ total    ┆ Acute hemorrhagic conjunctivi… ┆ 3.0   ┆ Sentinel surveillance │
│ 1999-04-05 ┆ Aichi      ┆ total    ┆ Acute poliomyelitis            ┆ 0.0   ┆ Confirmed cases       │
│ 1999-04-05 ┆ Aichi      ┆ total    ┆ Acute viral hepatitis          ┆ 4.0   ┆ Confirmed cases       │
│ …          ┆ …          ┆ …        ┆ …                              ┆ …     ┆ …                     │
│ 2026-09-21 ┆ Yamanashi  ┆ total    ┆ Viral hepatitis(excluding hep… ┆ 0.0   ┆ All-case reporting    │
│ 2026-09-21 ┆ Yamanashi  ┆ total    ┆ West Nile fever                ┆ 0.0   ┆ All-case reporting    │
│ 2026-09-21 ┆ Yamanashi  ┆ total    ┆ Western equine encephalitis    ┆ 0.0   ┆ All-case reporting    │
│ 2026-09-21 ┆ Yamanashi  ┆ total    ┆ Yellow fever                   ┆ 0.0   ┆ All-case reporting    │
│ 2026-09-21 ┆ Yamanashi  ┆ total    ┆ Zika virus infection           ┆ 0.0   ┆ All-case reporting    │
└────────────┴────────────┴──────────┴────────────────────────────────┴───────┴───────────────────────┘

For most analyses, unified is the dataset you want.

You can also filter while loading:

tb = (
    jp.get_data(
        disease="Tuberculosis",
        year=2024,
        prefecture=["Tokyo", "Osaka", "Hokkaido"],
        version="latest",
    )
    .select(["date", "prefecture", "disease", "count", "source"])
)

print(tb)

The exact values depend on the selected release; the example intentionally omits a static output snapshot so it cannot drift from version="latest".

Release data are downloaded on first use and cached locally. Use version="latest" when you want the newest published snapshot.

Cache and release resolution can be configured with JP_IDWR_DB_CACHE_DIR, JP_IDWR_DB_DATA_VERSION, and JP_IDWR_DB_DATA_BASE_URL. The former JPINFECT_* names remain accepted as compatibility aliases.

Datasets

IDWR publishes each year twice: preliminary weekly reports during the year, and final annual tables about 15 months later that include late reports. The unified dataset uses the final tables for every year they cover and the preliminary reports only after that, without overlapping records. Preliminary counts can be substantially lower for diseases with reporting delays.

For preliminary sentinel years, the source teitenrui files report year-to-date cumulative counts, so jp-idwr-db converts them to weekly incidence within each year, prefecture, and disease:

weekly_count = cumulative_count[t] - cumulative_count[t - 1]

Weeks the totals do not determine are left empty and explained in count_status; nothing is imputed.

The date column represents the Monday at the start of the corresponding ISO surveillance week.

See DATASETS.md for detailed schema and coverage, and DISEASES.md for disease-by-disease temporal coverage.

Use the data without Python

The data releases are regular Parquet files, so the Python package is optional.

Each GitHub release contains:

• unified.parquet and the other dataset tables
• manifest.json with file metadata, schemas, sizes, and checksums
• an optional jp_idwr_db.duckdb database with views over the Parquet files

For example, with DuckDB:

SELECT date, prefecture, disease, count
FROM read_parquet('unified.parquet')
WHERE disease = 'Tuberculosis'
  AND year = 2024
ORDER BY date, prefecture;

Or query the packaged DuckDB database directly:

duckdb jp_idwr_db.duckdb

The same Parquet files can be read from R, Arrow, Spark, Polars, pandas, or any other compatible tool.

Data source and updates

The package builds on official infectious disease surveillance data published by the National Institute of Infectious Diseases (NIID) and the Japan Institute for Health Security (JIHS).

The source archive has changed over time:

• final annual surveillance tables (confirmed and sentinel) are published as .xls / .xlsx
• preliminary all-case reports (zensu) are published weekly as .csv
• preliminary sentinel reports (teitenrui) are published weekly as .csv

Source: JIHS IDWR Surveillance Data Tables, accessed 2026-10-07 for release v2026.10.7.

The underlying IDWR surveillance data are weekly. jp-idwr-db refreshes its published release every week, incorporating newly available reports into a new versioned snapshot.

Created by editing the JIHS IDWR surveillance data: source files are parsed, cleaned, normalized, and combined by this project, and preliminary sentinel cumulative counts are converted to weekly incidence as described above.

jp-idwr-db is an independent project and is not an official NIID or JIHS publication.

It is inspired by the R package by Tomonori Hoshi, jpinfect and adapted for Python and more generalized usage.

See the official JIHS terms of use.

More documentation

• Dataset schemas and coverage
• Disease coverage
• Data wrangling examples
• Contributing
• Changelog

License

GPL-3.0-or-later. See LICENSE.