jp-idwr-db

jp-idwr-db makes Japan's IDWR infectious disease surveillance data easier to use.

It collects the official NIID/JIHS files, cleans and combines data published across different formats and archives, and releases analysis-ready weekly data as Parquet. You can use it through the Python package or read the release files directly with DuckDB, R, Arrow, Spark, or any other tool that supports Parquet.

IDWR reports are published weekly. jp-idwr-db checks for new data and publishes an updated, versioned snapshot every two weeks via GitHub Actions and releases.

Install

pip install jp-idwr-db

Quick start

Load the combined dataset:

import jp_idwr_db as jp

df = (
    jp.load("unified", version="latest")
    .select(["date", "prefecture", "category", "disease", "count", "source"])
)

print(df)

shape: (5_513_187, 6)
┌────────────┬────────────┬──────────┬────────────────────────────────┬───────┬────────────────────┐
│ date       ┆ prefecture ┆ category ┆ disease                        ┆ count ┆ source             │
│ ---        ┆ ---        ┆ ---      ┆ ---                            ┆ ---   ┆ ---                │
│ date       ┆ str        ┆ str      ┆ str                            ┆ f64   ┆ str                │
╞════════════╪════════════╪══════════╪════════════════════════════════╪═══════╪════════════════════╡
│ 1999-04-05 ┆ Aichi      ┆ total    ┆ AIDS                           ┆ 0.0   ┆ Confirmed cases    │
│ 1999-04-05 ┆ Aichi      ┆ total    ┆ Acute poliomyelitis            ┆ 0.0   ┆ Confirmed cases    │
│ 1999-04-05 ┆ Aichi      ┆ total    ┆ Acute viral hepatitis          ┆ 4.0   ┆ Confirmed cases    │
│ 1999-04-05 ┆ Aichi      ┆ total    ┆ Amebiasis                      ┆ 0.0   ┆ Confirmed cases    │
│ 1999-04-05 ┆ Aichi      ┆ total    ┆ Anthrax                        ┆ 0.0   ┆ Confirmed cases    │
│ …          ┆ …          ┆ …        ┆ …                              ┆ …     ┆ …                  │
│ 2026-08-31 ┆ Yamanashi  ┆ total    ┆ Viral hepatitis(excluding hep… ┆ 0.0   ┆ All-case reporting │
│ 2026-08-31 ┆ Yamanashi  ┆ total    ┆ West Nile fever                ┆ 0.0   ┆ All-case reporting │
│ 2026-08-31 ┆ Yamanashi  ┆ total    ┆ Western equine encephalitis    ┆ 0.0   ┆ All-case reporting │
│ 2026-08-31 ┆ Yamanashi  ┆ total    ┆ Yellow fever                   ┆ 0.0   ┆ All-case reporting │
│ 2026-08-31 ┆ Yamanashi  ┆ total    ┆ Zika virus infection           ┆ 0.0   ┆ All-case reporting │
└────────────┴────────────┴──────────┴────────────────────────────────┴───────┴────────────────────┘

For most analyses, unified is the dataset you want.

You can also filter while loading:

tb = (
    jp.get_data(
        disease="Tuberculosis",
        year=2024,
        prefecture=["Tokyo", "Osaka", "Hokkaido"],
        version="latest",
    )
    .select(["date", "prefecture", "disease", "count", "source"])
)

print(tb)

The exact values depend on the selected release; the example intentionally omits a static output snapshot so it cannot drift from version="latest".

Release data are downloaded on first use and cached locally. Use version="latest" when you want the newest published snapshot.

Cache and release resolution can be configured with JP_IDWR_DB_CACHE_DIR, JP_IDWR_DB_DATA_VERSION, and JP_IDWR_DB_DATA_BASE_URL. The former JPINFECT_* names remain accepted as compatibility aliases.

Datasets

The unified dataset combines historical confirmed-case data with modern all-case and sentinel reporting while avoiding overlapping records.

One important transformation is applied to the sentinel data: the source teitenrui files report year-to-date cumulative counts, so jp-idwr-db converts them to weekly incidence within each year, prefecture, and disease.

weekly_count = cumulative_count[t] - cumulative_count[t - 1]

The date column represents the Monday at the start of the corresponding ISO surveillance week.

See DATASETS.md for detailed schema and coverage, and DISEASES.md for disease-by-disease temporal coverage.

Use the data without Python

The data releases are regular Parquet files, so the Python package is optional.

Each GitHub release contains:

• unified.parquet and the other dataset tables
• manifest.json with file metadata, schemas, sizes, and checksums
• an optional jp_idwr_db.duckdb database with views over the Parquet files

For example, with DuckDB:

SELECT date, prefecture, disease, count
FROM read_parquet('unified.parquet')
WHERE disease = 'Tuberculosis'
  AND year = 2024
ORDER BY date, prefecture;

Or query the packaged DuckDB database directly:

duckdb jp_idwr_db.duckdb

The same Parquet files can be read from R, Arrow, Spark, Polars, pandas, or any other compatible tool.

Data source and updates

The package builds on official infectious disease surveillance data published by the National Institute of Infectious Diseases (NIID) and the Japan Institute for Health Security (JIHS).

The source archive has changed over time:

• historical annual surveillance tables are published as .xls / .xlsx
• modern all-case reports (zensu) are published weekly as .csv
• modern sentinel reports (teitenrui) are published weekly as .csv

Source: JIHS IDWR Surveillance Data Tables, accessed 2026-09-16 for release v2026.9.16.

The underlying IDWR surveillance data are weekly. jp-idwr-db refreshes its published release approximately every two weeks, incorporating newly available reports into a new versioned snapshot.

Created by editing the JIHS IDWR surveillance data: source files are parsed, cleaned, normalized, and combined by this project, and sentinel cumulative counts are converted to weekly incidence as described above.

jp-idwr-db is an independent project and is not an official NIID or JIHS publication.

It is inspired by the R package by Tomonori Hoshi, jpinfect and adapted for Python and more generalized usage.

See the official JIHS terms of use.

More documentation

• Dataset schemas and coverage
• Disease coverage
• Data wrangling examples
• Contributing
• Changelog

License

GPL-3.0-or-later. See LICENSE.