ipaapi

A Python package and command-line tool for QIAGEN Ingenuity Pathway Analysis (IPA). Upload datasets into an IPA project using an explicit column mapping, submit them for analysis, and track the results — one file or several hundred.

Free software (MIT). Built on QIAGEN's python-api-demo example code — not an official QIAGEN product, and not endorsed by QIAGEN.

ipaapi submit ~/data --ID 1:hugo --FC 4:logratio --skip-rows 1 \
    --reference-set ipkb --project MyStudy --pattern _DEG

---

Contents

• Why this exists
• Installation
• Quick start
• How the mapping works
• Command-line reference
• Recipes
• Working with IPA — the undocumented parts
• Authentication
• Python API
• Troubleshooting
• How a submission is encoded
• Development
• Contributing
• Licence

---

Why this exists

QIAGEN's demo script works, but assumes a rigid file layout: the gene ID in column 0, then n_observations × n_measurements value columns in strict repeating order, every observation carrying the same measurement types in the same positions. Real analysis output rarely looks like that.

This package replaces that assumption with a declaration. You name the identifier column and describe each observation as a set of (column, measurement type) pairs. Columns may be in any order, named anything, and interleaved with columns the analysis should ignore.

It also fixes a number of things the demo got wrong or left out — see Differences from the demo.

---

Installation

git clone <this-repo> ipaapi && cd ipaapi
pip install -e .

Or build and install a wheel:

python3 -m pip wheel . --no-deps -w dist
python3 -m pip install dist/ipaapi-*.whl

Requires Python 3.9+, requests, requests-oauthlib, pandas.

Confirm what you're running — this reports the version, the install location, and whether it's an editable checkout rather than a built wheel:

$ ipaapi --version
ipaapi 1.0.0
installed at /usr/lib/python3.11/site-packages/ipaapi
python 3.11.5 (/usr/bin/python3)

---

Quick start

Say your file looks like this — a comment line, then a header, then data:

# generated by pipeline v3
Gene,Common_name,Control_mean,Treatment_mean,Fold_change,P-value,Q-value
ENSG00000229807,XIST,4.21,2.88,-1.33,0.001,0.02

Column positions are 0-based and counted from the header row:

0 Gene   1 Common_name   2 Control_mean   3 Treatment_mean   4 Fold_change   5 P-value   6 Q-value

Check the mapping without contacting IPA:

ipaapi validate results.csv --ID 1:hugo --FC 4:logratio --skip-rows 1

results: 2,338 rows
gene id: 'Common_name' (hugo)
observations: 1
  results:
    'Fold_change' -> Log Ratio

     Common_name  Fold_change
0           XIST        -1.33
...
1 file valid. Nothing was uploaded.

When that looks right, submit:

ipaapi submit results.csv --ID 1:hugo --FC 4:logratio --skip-rows 1 \
    --reference-set ipkb --project MyStudy

submitted results: 43595871

Submitted 1 analysis.
Analyses are running in IPA. Check on them with:
  ipaapi status 43595871
  ipaapi report 43595871
Recorded in ~/.local/state/ipaapi/submissions.tsv -- see 'ipaapi history'.

---

How the mapping works

Three ideas, and they mirror how IPA thinks about a dataset.

Measurement — one value column: which column, what kind of number it holds, and an optional cutoff.

Observation — a named sample or contrast, and the measurement columns belonging to it. One analysis is created per observation.

ColumnMapping — the identifier column, its type, and the observations.

ColumnMapping(
    gene_id_column="Common_name",
    gene_id_type="hugo",
    observations=[
        Observation("drug A vs ctrl", [
            Measurement("A_log2fc", MeasurementType.LOG_RATIO),
            Measurement("A_padj",   MeasurementType.FALSE_DISCOVERY, cutoff=0.05),
        ]),
        Observation("drug B vs ctrl", [
            # declared in a different order on purpose -- this is fine
            Measurement("B_padj",   MeasurementType.FALSE_DISCOVERY, cutoff=0.05),
            Measurement("B_log2fc", MeasurementType.LOG_RATIO),
        ]),
    ],
)

One constraint is imposed by IPA, not by this package. The wire format declares expvaltype, expvaltype2, … and cutoff, cutoff2, … once for the whole submission, then supplies per-observation column names against those slots. So every observation must contribute exactly one column per measurement type, and a given type carries one cutoff throughout. Both are checked before anything is uploaded, with an error that explains why.

Within those limits, order and naming are free — observations declared in different column orders are normalised automatically.

Everything is validated against the actual data before upload: columns exist, none is claimed twice, and values fall in the range IPA expects for their type. That last check matters more than it looks — see measurement types.

---

Command-line reference

ipaapi validate   check a mapping against file(s) without uploading
ipaapi submit     upload into a project and start analyses
ipaapi status     check the state of existing analyses
ipaapi report     print IPA Interpret links
ipaapi history    list analyses submitted through this tool

Mapping arguments

Used by validate and submit.

Flag	Form	Meaning
PATH	positional	a data file, or a directory to search
--ID	COLUMN:TYPE	0-based identifier column and its IPA gene ID type. May be given twice — see two identifier columns
--FC	COLUMN:TYPE[:CUTOFF]	0-based value column, measurement type, optional cutoff
--skip-rows	N	discard N lines above the header row
--sep	CHAR	field delimiter (sniffed from the header line by default)
--pattern	TEXT	when PATH is a directory: substring or glob selecting files
--recursive	flag	search subdirectories too
--observation	NAME	observation name in IPA (default: the filename). Single file only
--no-range-check	flag	skip the value-range validation
--list-id-types	flag	print all 33 gene ID types and exit

submit

Flag	Default	Meaning
--project	required	destination IPA project. Created if it doesn't exist, so a typo silently makes a new one
--reference-set	omit	ipkb, dataset, or omit. See the reference set
--wait	off	poll until analyses finish and print report links
--interval / --timeout	30s / 3600s	polling, only with --wait
--dry-run	off	validate and stop before login
--analysis-name / --dataset-name	filename	single file only
--log-file	~/.local/state/ipaapi/submissions.tsv	submission log

Authentication arguments

Used by every command that contacts IPA.

Flag	Meaning
--no-cache	ignore any cached token
--token-file	token cache path (default ~/.cache/ipaapi/token.json)
--application-name	applicationname IPA scopes the session to (default PythonAPI)
--browser	browser to launch for login, e.g. firefox

history

Flag	Meaning
--project / --since / --limit	filters
--status	look up each analysis's current state (requires login)
--log-file	read a different log

Environment variables

Variable	Purpose
IPAAPI_TOKEN_FILE	token cache location — set this if $HOME isn't writable
IPAAPI_LOG_FILE	submission log location

---

Recipes

Many files, one analysis each

ipaapi submit ~/data --pattern _DEG --ID 1:hugo --FC 4:logratio \
    --skip-rows 1 --reference-set ipkb --project Study1

--pattern takes plain text or a glob. Text with no *, ? or [ matches as a substring, so --pattern SampleA finds SampleA_DEG.txt and SampleA_raw.tsv. With no --pattern, *.txt/*.tsv/*.csv are searched. Hidden files are skipped and results sorted, so run order is predictable.

Every matched file must fit the same --ID/--FC positions.

Files are filed as they're processed

When PATH is a directory, each file moves as its outcome becomes known:

Outcome	Destination
IPA accepted it	submitted/
The file is at fault	failed/, with a .error.txt note beside it
Allowance exhausted, or IPA declined	left in place for the next run

submitted SampleA_DEG: 43595001
submitted SampleB_DEG: 43595002

Allowance exhausted while submitting SampleC_DEG:
REJECTED: the analysis allowance appears to be exhausted.
IPA said: 'Unable to run analysis: Analysis limit exceeded'

2 file(s) moved to submitted/
2 file(s) left in place for the next run
Re-run the same command later; the files left in place are exactly the ones
still to do.

The source directory shrinks to exactly the work outstanding, and re-running the identical command resumes. submitted/ and failed/ are excluded from discovery, so a run can't re-ingest its own output.

Nothing is moved when the command is at fault — a bad --ID type or a mapping that fails every file leaves the directory untouched, because that's a mistake to fix rather than data to quarantine. Single-file submits are never moved.

Draining a backlog against a daily allowance

Because a stopped run resumes cleanly, this is safe to leave unattended:

0 6 * * * cd ~/data && ipaapi submit ./ --pattern _DEG --ID 1:hugo \
    --FC 4:logratio --skip-rows 1 --reference-set ipkb --project Study1 \
    >> ~/ipaapi-cron.log 2>&1

It submits until the allowance runs out, files what succeeded, leaves the rest. Check the log after the first few runs — a cron job whose refresh token has expired fails into that file rather than prompting anyone.

Finding analysis IDs later

IPA's API cannot list the analyses on an account, so the package keeps its own log — every submission appends a timestamped row.

ipaapi history
ipaapi history --project Study1 --since 2026-08-01
ipaapi history --status

2026-08-05T08:35:53-06:00  43595039  Study1  SampleA_DEG
2026-08-05T08:35:53-06:00  43595041  Study1  SampleB_DEG

2 submission(s). Report links: ipaapi report 43595039 43595041

Plain TSV — grep it, open it in a spreadsheet. It only covers submissions made through this tool; anything submitted from the IPA client won't appear.

Comment lines above the header

# generated by pipeline v3, 2026-08-05
EnsemblID	log2FC	pval

--skip-rows 1 discards the preamble. Column numbers count from the header, so they don't change when you add it.

Skipping also fixes delimiter detection: the delimiter is sniffed from the header line, and a comment line is a bad thing to sniff — the one above has commas but no tabs, so without --skip-rows the file would be read as CSV and collapse into nonsense. Rather than let that through, a header that looks like a comment is rejected with a message pointing at this flag.

Two identifier columns

--ID may be given twice. The first is the primary; the second fills rows where the primary is blank (., NA, empty, and similar are all treated as missing).

ipaapi submit data.csv --ID 0:ensembl --ID 1:hugo --FC 4:logratio --project S1

Read this before relying on it. IPA accepts one geneidtype per submission. Rows filled from the second column are still uploaded under the primary's type, so they may fail to map. The fill count is always reported:

Warning: 344 of 2,338 rows took their identifier from the fallback column
'Common_name' (hugo). IPA is told a single gene ID type for the submission --
'ensembl' -- so those rows are uploaded under that declaration and may not map.

If a large fraction is being filled, using the fallback column as the only identifier is usually better than mixing.

---

Working with IPA

Most of this is either undocumented or documented somewhere hard to find. It's recorded here because getting it wrong is expensive — analyses consume a metered allowance.

Gene ID types

--ID COLUMN:TYPE takes any value from IPA's geneidtype list (Integration Module §3.1). ipaapi submit --list-id-types prints all 33.

Common ones: ensembl, hugo, entrezgene, refseq, swissprot, affymetrix, illumina, agilent.

Two things are not guessable:

• Human gene symbols are hugo. Not genesymbol, not hgnc, and not the desktop client's own label Gene Symbol — all three are rejected outright.
• Species rides on the identifier type. There is no species parameter: hugo human, mousesymeg mouse, ratsymeg rat.

A type outside the documented list produces a warning with a near-match suggestion but is still sent, since IPA is the authority and the list will age. An unrecognised value fails before anything is uploaded, and IPA names it.

The reference set

The background enrichment is scored against — the denominator of the Fisher's exact test behind every p-value.

Value	Background
ipkb	Ingenuity Knowledge Base (Genes Only, or + Endogenous Chemicals if chemicals are present)
dataset	the genes you uploaded
omit (default)	IPA chooses

Which to use depends on what you uploaded:

• Uploading a complete measured transcriptome with a cutoff? dataset is the better science — the background is what your assay could actually detect, which controls for detection bias.
• Uploading a pre-filtered hit list? dataset makes the background nearly identical to the foreground. Use ipkb.

§4.1.3.1 states that with the parameter omitted IPA picks by size — ipkb below 2000 identifiers, dataset at 2000 or more. In practice this has not been observed to hold: files of 1,804–6,245 rows all came back as Ingenuity Knowledge Base (Genes Only). Since the behaviour is unpredictable, set it explicitly for anything you intend to compare against itself.

Verify after the fact — the setting is recorded in every IPA export:

grep -h "^Reference set" *_IPA_output.txt | sort | uniq -c

Array platforms can also be named as reference sets, paired with a referencesettype. Not exposed here; see §4.1.3.

Measurement types

Value	Meaning	Valid range
ratio	Ratio	[0, +∞)
foldchange	Fold Change	(-∞, -1] and [1, +∞)
logratio	Log Ratio	(-∞, +∞)
pvalue	p-value	[0, 1]
falsediscovery	FDR / q-value	[0, 100]
intensity	Intensity	[0, +∞)
other	Other (normalised around zero)	(-∞, +∞)
gain_loss	Variant Gain/Loss	-2, -1, 0, 1, 2
classification	Variant ACMG Classification	-2, -1, 0, 1, 2

Out-of-range values are silently discarded by IPA. §3.1: "analysis will still proceed without errors or warning diagnostics" — offending entries are simply dropped. This is why the range check exists and why it refuses rather than warns. Declaring log2 fold changes as foldchange, for instance, would quietly discard every gene between −1 and 1, which in a typical scRNA-seq table is most of them.

The package helps in both directions:

• Values declared foldchange that cluster inside (−1, 1) → suggests logratio.
• A column declared logratio with no values in (−1, 1) → warns that it looks like signed fold change, since a real log ratio is centred on zero.

A column called Fold_change may hold either. Check the data, not the name.

What the API cannot do

• List your projects. --project creates one if the name doesn't exist, so a typo silently makes a new project rather than erroring.
• List your analyses. Every endpoint needs an ID you already hold — hence the local submission log.
• Tell you your remaining allowance. You discover the limit by hitting it.

Errors IPA actually returns

IPA answers a rejected submission with an HTML error page, not plain text. The reason is at the end, after support boilerplate. This package strips the boilerplate and the page footer, and classifies what's left:

IPA's message	Class	What the tool does
Unknown GeneId Type (X)	MalformedRequestError	stops; names the flag; moves nothing
Unable to run analysis: Analysis limit exceeded	QuotaExceededError	stops; leaves remaining files for the next run
Unable to run analysis: … (other)	AnalysisRefusedError	as above — reached the analysis logic, so not a parameter fault
anything else	SubmissionError	files that one under failed/

Quota matching is deliberately broad (ipaapi.client.QUOTA_PATTERNS plus HTTP 429): a false positive only leaves a file for the next run, while a false negative would quarantine a retryable submission. The raw response is always printed, so a misclassification is visible.

Interpret links

ipaapi report <id> fetches the IPA Interpret URL for a finished analysis. It checks status first, so an unfinished analysis says so rather than surfacing a bare HTTP 500.

These have been observed to return HTTP 500 even for succeeded analyses. The cause is unconfirmed — possibly the commercial add-on licence, possibly a stale endpoint path inherited from the demo. examples/probe_interpret.py prints the raw response for diagnosis. Analyses open fine in IPA itself.

---

Authentication

Browser-based OAuth 2.0 with PKCE. Your password never reaches this package.

1. A short-lived HTTP server binds 127.0.0.1:8000.
2. Your browser opens QIAGEN's authorization page; you log in there.
3. QIAGEN redirects back to localhost:8000 with a one-time code. The state parameter is verified, then the code plus the PKCE verifier is exchanged for a token.
4. The token is used as Authorization: Bearer … and the server shuts down.

Whichever account you log in as owns the datasets and projects.

The client ID is the public one any IPA user may use — it is not a secret.

Token caching and refresh

Tokens are cached at ~/.cache/ipaapi/token.json, owner-only (0600). Access tokens are short-lived, but a refresh token comes with them and is spent automatically: an expired cache is renewed over HTTP with no browser and no prompt. A browser login is only needed when the refresh token itself is rejected.

Deleting the cache is effectively logging out. --no-cache forces a fresh login. Be aware the token is plaintext on disk — anyone who can read your home directory can use it until it expires.

Headless servers

Because refresh is automatic, a token copied from a machine with a browser keeps renewing itself indefinitely:

# once, on a machine with a browser
ipaapi submit ... # or any command that logs in

scp ~/.cache/ipaapi/token.json server:~/.cache/ipaapi/token.json
ssh server chmod 600 ~/.cache/ipaapi/token.json

If $HOME isn't writable, the cache can't be saved and every run needs a fresh login — crippling on a headless box. Point it somewhere writable:

export IPAAPI_TOKEN_FILE=$HOME/ipaapi-token.json
export IPAAPI_LOG_FILE=$HOME/ipaapi-submissions.tsv

Both failures are reported loudly rather than swallowed, because a cache that never writes looks exactly like a token that expires instantly.

When an interactive login is genuinely needed, X forwarding is the cleanest route — the server-side browser renders locally and localhost:8000 resolves server-side where the callback listens, so no port forwarding is required:

ssh -X you@server        # ssh -Y from macOS, with XQuartz running

Failing that, forward the callback port and use your own browser:

ssh -L 8000:localhost:8000 you@server

The error message distinguishes DISPLAY unset from no browser found.

The redirect URI is pinned to http://localhost:8000 by the OAuth client registration, so the port is not configurable in practice.

Using a token obtained elsewhere

import os
from ipaapi import Credentials, IPAClient

client = IPAClient(Credentials.from_token(os.environ["IPA_TOKEN"]))

---

Python API

from ipaapi import (
    ColumnMapping, Dataset, IPAClient, Measurement, MeasurementType,
    Observation, ReferenceSet, TokenCache,
)

mapping = ColumnMapping(
    gene_id_column="Common_name",
    gene_id_type="hugo",
    observations=[
        Observation("HIV vs NEG", [
            Measurement("Fold_change", MeasurementType.LOG_RATIO),
        ]),
    ],
)

dataset = Dataset.from_file("results.csv", mapping, skip_rows=1)
print(dataset.describe())          # confirm before uploading

client = IPAClient.login(cache=TokenCache())
ids = client.submit(dataset, project="MyStudy", reference_set=ReferenceSet.IPKB)

for analysis_id, status in client.wait_for(ids).items():
    if status.succeeded:
        print(client.report_url(analysis_id))

Key objects:

Object	Purpose
ColumnMapping, Observation, Measurement	describe the file
Dataset.from_file / .from_frame	load and validate
IPAClient.login()	OAuth, with caching and refresh
.submit() .status() .wait_for() .results() .report_url()	the API
GENE_ID_TYPES	all 33 identifier types and what they mean
ipaapi.history	the submission log
ipaapi.errors	everything derives from IPAError

Results

results = client.results(analysis_id)
print(results.canonical_pathways.head())
cp, ur, df = results                  # unpacks like the demo's ipa_results()

Programmatic result retrieval is a commercial IPA add-on. Without it these calls raise ResultsUnavailableError. Submission, status polling and report links are unaffected.

---

Troubleshooting

Symptom	Cause	Fix
REJECTED: IPA does not recognise the gene ID type 'X'	not in IPA's vocabulary	--list-id-types; human symbols are hugo
declared 'foldchange' but holds N out-of-range value(s)	log2 values declared as linear fold change	--FC N:logratio
Could not find a header row … looks like a comment	preamble above the header	--skip-rows N
--FC refers to column N, but the file has only M column(s)	1-based counting, or wrong --skip-rows	positions are 0-based, from the header
Every row is missing an identifier	wrong column, or no header	check with head -1 file | tr '\t' '\n' | nl -v0
the analysis allowance appears to be exhausted	daily/period limit	re-run later; files left in place resume
Login prompt on every run	token cache not writable	export IPAAPI_TOKEN_FILE=...; check for a root-owned cache
Could not open a browser automatically	headless	ssh -X, or copy a token across
report returns HTTP 500 on a succeeded analysis	unconfirmed; possibly add-on licence	open the analysis in IPA; see examples/probe_interpret.py
Analyses have z-scores but no p-values	reference set equals the gene list	--reference-set ipkb
Half of all pathways significant	list too large for the background	apply a cutoff, or upload unfiltered data with a cutoff

Useful first move for any column problem:

head -1 yourfile.csv | tr ',\t' '\n' | nl -v0

---

How a submission is encoded

Worth knowing when debugging. --ID 1:hugo becomes three separate things:

From --ID	Wire parameter	Sent
the type	geneidtype=hugo	once
the column, resolved from position to header name	genecolname=Common_name	once
that column's values	geneid=XIST, geneid=UTY, …	once per row

The column number never leaves your machine.

The whole dataset travels in one application/x-www-form-urlencoded POST to /pa/api/v2/multiobsanalysis, which both creates the dataset in the project and starts one analysis per observation. Parameter naming is positional and irregular — for measurement slot k and observation i, both zero-based:

Parameter	Meaning
expvaltype, expvaltypeK+1	measurement type for slot k (global)
cutoff, cutoffK+1	cutoff for slot k (global, optional)
obsI+1name	observation name
expvalname, expvalK+1name	column label, first observation
obsI+1expvalname, obsI+1expvalK+1name	column label, later observations
geneid	one per data row
expvalue, expvalK+1	one per slot per observation, per row

Per-row value parameters carry no observation prefix — they cycle through the slots of observation 1, then observation 2, and so on. Order is load-bearing.

The body is properly percent-encoded. The demo concatenated it by hand, so any value containing a space, &, =, + or % corrupted the request — including the Group Max Intensity column in the demo's own sample dataset.

---

Development

src/ipaapi/
  __init__.py    public API and the version (single source of truth)
  models.py      MeasurementType, AnalysisStatus, ReferenceSet, GENE_ID_TYPES
  mapping.py     Measurement, Observation, ColumnMapping
  dataset.py     Dataset, load_table
  _payload.py    multiobsanalysis body construction
  auth.py        OAuth 2.0 + PKCE, Credentials, TokenCache, refresh
  client.py      IPAClient, error classification
  history.py     the submission log
  triage.py      submitted/ and failed/ filing
  cli.py         the ipaapi console script
  errors.py      exception hierarchy
tests/           offline; no network required
examples/        runnable scripts and diagnostics

pip install -e ".[dev]"
pytest

The suite is fully offline — mapping validation, the exact parameter layout of the submission body, encoding of hostile characters, error classification, triage behaviour, token cache and refresh logic.

Versioning. The version lives only in src/ipaapi/__init__.py; pyproject.toml reads it at build time. Bump it there and nowhere else, and add a CHANGELOG.md entry. ipaapi --version reports the install path too, which is what actually answers "am I running the wheel I think I am".

Differences from the demo

• Column mapping by name in any order, validated before upload.
• Request bodies are percent-encoded.
• OAuth: no CPU-spinning wait loop, state is verified, logins time out, the callback server is shut down, error redirects are handled, tokens are cached and refreshed.
• Submissions are never retried automatically — a retried POST could create a duplicate analysis. GETs retry with backoff.
• Typed exceptions; access tokens excluded from repr().
• No install_dependencies() shelling out to pip3.

---

Contributing

Issues and pull requests are welcome. The most useful contributions are corrections to the Working with IPA section — much of it was established by trial against a live account, and a few points have already had to be corrected more than once. If IPA behaves differently for you, that is worth reporting even without a code change.

pip install -e ".[dev]"
pytest

Tests are fully offline; none of them contact IPA.

---

Status

1.0 — stable and in production use against live IPA. The command line and the Python API are settled; breaking changes from here mean a major version bump. See CHANGELOG.md.

Known open questions, none of which affect submission:

• Interpret links (ipaapi report) have returned HTTP 500 for analyses that succeeded. Cause unconfirmed; possibly the commercial add-on licence.
• Programmatic result retrieval (client.results()) requires that same add-on and is largely untested here.
• The documented reference-set size rule does not match observed behaviour; set --reference-set explicitly.

---

Licence

MIT — see LICENSE. Free to use, modify and redistribute.

Not affiliated with, endorsed by, or supported by QIAGEN. IPA is QIAGEN's product; this is an independent client for its public API, built on the python-api-demo example code QIAGEN publishes. For questions about the API itself, QIAGEN's contact is AdvancedGenomicsSupport@qiagen.com — please don't send them bug reports about this package.