Skip to main content

pysubgroup

Welcome to pysubgroup v2!

This repository and the pysubgroup package on PyPI now hold pysubgroup v2, a complete rewrite that replaces the previous version. The old version remains available in the git branch legacy. What is new:

  • Faster. In our benchmarks more than ten times faster than the old pysubgroup, thanks to batched evaluation of candidates on packed bitsets and optional numba kernels. It also finishes tasks where the old version times out.
  • More reliable. More than 1,900 tests, parity checks against the old pysubgroup (identical top-10 qualities in every benchmark task both versions finished), deterministic tie-breaking, and clearly defined handling of missing values.
  • Still easy to use. An intuitive API in which a search takes a few lines, as before: a target, a search space, a quality function, and an algorithm.
  • Modern API. Targets fitted in the scikit-learn style, immutable Condition and Rule objects, results as plain (quality, rule) pairs, NumPy 2 and Python 3.11 to 3.14.
  • Comprehensive. Search algorithms (DFS, best-first, OPUS, DpSubgroup, Apriori, SD-Map, SD-Map*, GP-growth, NumBSD, VLSD, beam search, DSSD, and generalization-aware search) for binary, numeric, multiclass, nominal, and vector-valued targets, plus exceptional model mining, significance filtering, and visualizations. Each algorithm is implemented from its publication.

⚠️ pysubgroup v2 was essentially rewritten from scratch, with most of the code actually generated by AI tools (at the time of writing, primarily Claude Opus and GPT-5.6 Sol), while design decisions and high-level code reviews were handled by humans.

While this might not sit well with purists, it has enabled us to implement more features, produce more efficient, robust, and thoroughly tested code, and focus our attention on the most scientifically interesting decisions—all while aiming for production-ready (we hope!) reliability. In fact, I personally feel much more confident in the current version of pysubgroup than in the previous version, which I wrote manually over the years. This is also reflected in the version jump from 0.9 to 2.0.

Note that the old, human-written version of pysubgroup is still available in the legacy branch.

A first example

Which groups of people are most likely to earn more than 50K a year?

from sklearn.datasets import fetch_openml
import pysubgroup as sd

data = fetch_openml("adult", version=2, as_frame=True).frame.drop(columns="fnlwgt")

target = sd.BinaryTarget().fit(data, "class", ">50K")
conditions = sd.generate_conditions(data.drop(columns="class"))
task = sd.SubgroupDiscoveryTask(
    data, target, conditions, sd.WRAccQF(), result_set_size=5, depth=2
)
result = sd.DFS().execute(task)
print(sd.to_dataframe(result, data, target))
   quality                                                          subgroup   size  relative_size  target_share   lift
0    0.095                                 marital-status=Married-civ-spouse  22379          0.458         0.446  1.864
1    0.089  marital-status=Married-civ-spouse ∧ native-country=United-States  20003          0.410         0.457  1.910
2    0.088                    marital-status=Married-civ-spouse ∧ race=White  20054          0.411         0.454  1.897
3    0.085                                   relationship=Husband ∧ sex=Male  19715          0.404         0.449  1.875
4    0.085                                              relationship=Husband  19716          0.404         0.449  1.875

Installation

pysubgroup v2 needs Python 3.11 to 3.14 with NumPy (>= 2), pandas, scikit-learn, and SciPy.

pip install pysubgroup

Optional extras:

pip install "pysubgroup[plot]"    # matplotlib visualizations
pip install "pysubgroup[numba]"   # numba kernels for numeric targets and NumBSD

The old version (v0.x) can still be installed from the legacy branch:

pip install "git+https://github.com/flemmerich/pysubgroup.git@legacy"

Subgroup Discovery

Subgroup Discovery is a well-established data mining technique that allows you to identify patterns in your data. More precisely, the goal of subgroup discovery is to identify descriptions of data subsets that show an interesting distribution with respect to a pre-specified target concept. For example, given a dataset of patients in a hospital, we could be interested in subgroups of patients for which a certain treatment X was successful. One example result could then be stated as:

"While in general the operation is successful in only 60% of the cases, for the subgroup of female patients under 50 that also have been treated with drug D, the success rate was 82%."

Here, a variable operation success is the target concept, and the identified subgroup has the interpretable description female=True AND age<50 AND drug_D=True. We call these single selection expressions (such as female=True) conditions. The interesting behavior for this subgroup is that the distribution of the target concept differs significantly from the distribution in the overall dataset. A discovered subgroup could also be seen as a rule:

female=True AND age<50 AND drug_D=True ==> Operation_outcome=SUCCESS

Computationally, subgroup discovery is challenging since a large number of such conjunctive subgroup descriptions have to be considered. Of course, finding computable criteria for which subgroups are likely to interest a user is also an eternal struggle. Therefore, a lot of literature has been devoted to the topic of subgroup discovery (including some of my own work). Overviews of the topic are, for example:

How to use

A search has four ingredients: a target (the property of interest), a search space (a list of conditions), a quality function, and an algorithm. They are combined in a SubgroupDiscoveryTask:

import pysubgroup as sd

target = sd.BinaryTarget().fit(data, "class", ">50K")
conditions = sd.generate_conditions(data.drop(columns="class"))
task = sd.SubgroupDiscoveryTask(
    data,
    target,
    conditions,
    sd.WRAccQF(),
    result_set_size=10,   # number of subgroups returned
    depth=2,              # maximum number of conditions per subgroup
)
result = sd.DFS().execute(task)

The result is a list of (quality, rule) pairs, best first:

for quality, rule in result:
    print(f"{quality:.4f}  {rule}")
# 0.0948  [marital-status=Married-civ-spouse]
# 0.0892  [marital-status=Married-civ-spouse ∧ native-country=United-States]
# ...

sd.to_dataframe(result, data, target) turns it into a table with sizes and target statistics, and sd.subgroup_data(result[0], data) returns the rows that a subgroup covers.

Numeric targets

For a numeric target, use NumericTarget with a numeric quality function. Filters such as a minimum subgroup size restrict the results:

target = sd.NumericTarget().fit(data, "hours-per-week")
conditions = sd.generate_conditions(data.drop(columns=["class", "hours-per-week"]))
task = sd.SubgroupDiscoveryTask(
    data, target, conditions, sd.StandardQFNumeric(a=0.5),
    result_set_size=5, depth=2, filters=[sd.MinSizeFilter(100)],
)
result = sd.DFS().execute(task)
print(sd.to_dataframe(result, data, target))
   quality                                                  subgroup   size  relative_size    mean  mean_difference
0  525.687                           relationship=Husband ∧ sex=Male  19715          0.404  44.166            3.744
1  525.670                                      relationship=Husband  19716          0.404  44.166            3.744
2  525.060              marital-status=Married-civ-spouse ∧ sex=Male  19899          0.407  44.145            3.722
...

Other algorithms

All exhaustive algorithms return the same subgroups; heuristic ones trade completeness for speed or diversity. Swap the algorithm without changing the task:

sd.BestFirstSearch().execute(task)
sd.BeamSearch(beam_width=10).execute(task)   # heuristic
sd.NumBSD().execute(task)                    # exhaustive, for numeric targets
sd.DSSD().execute(task)                      # diverse subgroup sets

Your own search space

generate_conditions returns an ordinary list of conditions; extend or replace it with your own:

from pysubgroup import Condition

conditions = sd.generate_conditions(data.drop(columns="class"), n_bins=5)
conditions.append(Condition("age", "gt", 35))
conditions.append(Condition("age", "interval", lower=20, upper=30))  # (20, 30]

Key classes

  • Condition: an atomic condition such as age > 35 or sex = Female.
  • Rule: a conjunction of conditions, the description of a subgroup.
  • Targets (BinaryTarget, NumericTarget, MulticlassTarget, ...): fitted on the data; they hold the target statistics.
  • Quality functions (WRAccQF, StandardQF, StandardQFNumeric, ...): score a subgroup and, for exhaustive searches, bound the quality of its refinements (optimistic estimates).
  • SubgroupDiscoveryTask: what to search for; an algorithm's execute runs it.

Learn more

  • The pysubgroup book: the full documentation, from a first discovery run to custom quality functions, with a reference part (API, choosing methods, glossary, troubleshooting).
  • examples/: notebooks on binary and numeric targets, visualizations, auditing a classifier (SubROC), exceptional model mining, and writing your own quality functions.

Cite

If you are using pysubgroup for your research, please consider citing our demo paper:

Lemmerich, F., & Becker, M. (2018, September). pysubgroup: Easy-to-use subgroup discovery in python. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECMLPKDD). pp. 658-662.

bibtex:

@inproceedings{lemmerich2018pysubgroup,
  title={pysubgroup: Easy-to-use subgroup discovery in python},
  author={Lemmerich, Florian and Becker, Martin},
  booktitle={Joint European Conference on Machine Learning and Knowledge Discovery in Databases},
  pages={658--662},
  year={2018}
}

License

We are happy about anyone using this software. Thus, this work is put under an Apache license. However, if this constitutes any hindrance to your application, please feel free to contact us; we are sure that we can work something out.

Apache License 2.0; see LICENSE.

Metadata

Release files for pysubgroup 2.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pysubgroup 2.0.0
File Size Uploaded Provenance Actions
pysubgroup-2.0.0.tar.gz 138.5 kB Without provenance Details for pysubgroup-2.0.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pysubgroup 2.0.0
File Interpreter ABI Platform Actions
pysubgroup-2.0.0-py3-none-any.whl Python 3 none any Details for pysubgroup-2.0.0-py3-none-any.whl

Total release size: 295.2 kB

Release files / pysubgroup-2.0.0.tar.gz

Download URL pysubgroup-2.0.0.tar.gz
Size 138.5 kB
Tags Source
SHA-256 checksum
How to use checksums
4af6bb97cf2bd795a02b81d7b691e8afc8f09f7b1c7067c6c61f14130b6681bb
BLAKE2b-256 checksum
How to use checksums
e4b387739a0586456522958edbcf54416fb90cea94c11f15516e479c55a5a332
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.16

Release files / pysubgroup-2.0.0-py3-none-any.whl

Download URL pysubgroup-2.0.0-py3-none-any.whl
Size 156.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
5afa58dc1561037f6a61a51f246d44a7d6572678712b2bdd93effd2919078976
BLAKE2b-256 checksum
How to use checksums
6246a0b352d1d6b119a81a0b09ab37d75e5b08b14b735e911cb6fdd1941282cd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.16

Security for pysubgroup 2.0.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page