amex-israel-scraper

Scrape your own American Express Israel transactions, returning the same shape israeli-bank-scrapers returns.

Why this exists

Amex Israel moved to a platform called DigitalV3 (web.americanexpress.co.il) behind Cloudflare. israeli-bank-scrapers drives Puppeteer, and headless Chrome is refused by that WAF before it reaches a login form — so its amex scraper cannot reach the new platform at all.

Two things turned out to matter:

1. The browser fingerprint, not the behaviour. Cloudflare rejects headless Chrome on sight. Camoufox — a patched Firefox — gets through. Slowing down, moving the mouse, and adding waits do not help a browser that is refused before it acts.
2. Underneath the WAF, the site is a clean JSON API. Nothing here parses HTML. The browser exists only to hold a session Cloudflare trusts; every call is a fetch() evaluated inside it.

The result is a scraper that is browser-based but not page-scraping: fast, and stable across the site's visual redesigns.

Install

pip install amex-israel-scraper
python -m camoufox fetch        # one-off: downloads the patched Firefox

Python 3.11+. The Camoufox fetch is a few hundred MB and only needs doing once.

From a clone instead:

pip install -e .

Use it

from amex_israel import scrape

result = scrape(
    {
        "id": "...",  # ת"ז
        "card6Digits": "...",  # first 6 digits of the card
        "password": "...",  # the permanent password (no OTP: Amex IL does not send one)
    }
)

if not result.success:
    raise SystemExit(f"{result.errorType}: {result.errorMessage}")

for account in result.accounts:
    print(account.accountNumber, len(account.txns))
    for txn in account.txns:
        print(txn.date, txn.description, txn.chargedAmount)

Or from the shell — credentials come from the environment, never from arguments, because an argument lands in your shell history and in the process list:

export AMEX_ID=... AMEX_CARD6=... AMEX_PASSWORD=...
python -m amex_israel --session ~/.amex-session.json > result.json

History

By default you get the current open billing cycle. To walk closed statements back:

result = scrape(credentials, start_date="2024-01-01")

One request per card per month, so it is slow. DigitalV3 stops serving somewhere around two years back regardless of what you ask for.

Session reuse — do this if you run on a schedule

Logging in is the WAF-prone step; the API calls afterwards are not. result.deviceTrustData is the browser session, and handing it back skips the login entirely:

first = scrape(credentials)
session = first.deviceTrustData  # persist this

later = scrape(credentials, device_trust_data=session)  # no login at all

A dead session falls back to a normal login, so this never becomes a failure mode. The session file holds live authenticated cookies — treat it exactly like the password. The CLI writes it 0600.

Where it runs

Amex's WAF is materially more permissive from an Israeli residential IP. From a datacenter — any mainstream cloud — expect to be challenged or blocked, with correct credentials and a correct fingerprint. This was the single biggest factor in getting a reliable scrape, above every browser setting.

If you are seeing BLOCKED_BY_WAF, the IP is the first thing to change, not the code. Run with headless=False to see what the WAF is actually showing you.

Output

The same structures israeli-bank-scrapers produces, under the same names:

Including the convention that trips people up: a purchase is negative, a refund positive, exactly as originalAmount: -txn.dealSum does there. Amex's API reports a purchase as positive; the mapping negates.

result.to_dict() gives a plain JSON-able dict with enums flattened and empty fields dropped, so the payload matches what the JS library emits.

Beyond israeli-bank-scrapers

Everything above is drop-in compatible. These are additions — extra fields on otherwise-identical structures, so a consumer that ignores them sees exactly the standard shape.

Transaction.stableId — an id you can key a database on

identifier is the issuer's voucher number, and Amex does not assign one until a charge posts. A fresh charge has no identifier at all. Key on it directly and every un-posted charge in a scrape collides on the same empty key — then changes key once the voucher lands, which reads as a delete plus an insert.

stableId is the voucher when there is one, otherwise a hash of (card, date, amount, merchant), prefixed so the two kinds can never be confused. Expect exactly one transition per charge, from content key to voucher key; de-dup on (date, amount, description) across that boundary if it matters to you.

Transaction.billingMonth — which statement it landed on

Not derivable from the purchase date. Amex bills around the 9th, so a purchase on the 8th and one on the 10th belong to different statements despite being two days apart. Derive the cycle from the purchase date and boundary charges land in the wrong month, and a monthly total stops matching the bill you were actually sent. This field is what the issuer says, not what we inferred — and processedDate is anchored to it.

Transaction.isPosted — whether the issuer has given it an identity

Distinct from status. Pending/Completed describes settlement; this describes whether a voucher number exists yet. An un-posted charge is real and will be billed — it just cannot be keyed on its voucher, and the merchant may still revise it.

result.statementTotals — check the scrape against the issuer's own number

Amex publishes the official upcoming charge (futureDebits) separately from the transactions that make it up. statementTotals sums what was actually fetched, per card and month, so the two can be compared:

for total in result.statementTotals:
    print(total.accountNumber, total.billingMonth, total.total, total.transactionCount)

Equal means nothing was missed. A gap means the statement fetch dropped rows — which is otherwise a silent failure, because a short list of real transactions looks exactly like a correct one.

ScraperErrorTypes.BLOCKED_BY_WAF

Its own type because the remedy is unlike every other failure: nothing about the credentials is wrong, and retrying from the same IP will keep failing. See Where it runs.

The full statement, not the recent window

Not a field, but the reason the numbers reconcile. DigitalV3 offers a simpler-looking GetLatestTransactions, and it is a trap: it returns a narrow recent window that omits the monthly portions of installment plans and anything charged early in the cycle. Totals built from it look reasonable and quietly fail to match the bill. This library reads GetTransactionsList — the full statement — including the outOfStatementChargeDateVouchers bucket, which holds charges billed on a statement but dated outside it.

Errors

scrape() does not raise; a failure comes back as success=False with an errorType, matching the reference library.

Tests

pytest

The mapping — signs, dates, installments, the stable id, statement anchoring — is tested against recorded API rows with no browser and no network. That is where a mistake is silent, so that is where the tests are.

Scope and limits

• Cards only. Amex Israel is a card issuer here; there is no bank account to read.
• balance is always None. A card has a statement, not a balance. The field exists for shape parity.
• No OTP path. Amex Israel's flow uses a permanent password. There is no two-factor step to implement, and TWO_FACTOR_RETRIEVER_MISSING has no analogue.
• Hebrew descriptions. description is whatever the merchant registered, usually Hebrew.
• Not affiliated with, endorsed by, or supported by American Express.

Using this responsibly

This reads your own account, with your own credentials, the same data the website shows you when you log in. That is the only use it is built for. Keep the request volume near what a person would generate — polling in a loop is both rude and the fastest way to get an account flagged. Your credentials and the session file stay on your machine; nothing here transmits them anywhere.

Contributing

Issues and pull requests are welcome. The mapping (amex_israel/mapping.py) is deliberately free of any browser import, so its tests run anywhere with no network, no browser and no Amex account — pytest is enough, and CI runs it on 3.11 and 3.13.

The live scrape cannot be tested in CI, for the reason in Where it runs: it needs an Israeli residential IP and a real account. If you change the browser or login path, say in the PR how you verified it.

Licence

MIT — see LICENSE.