reconnectchina

The Python client for the ReConnect China API. It logs a researcher in, keeps their access token fresh, calls every operation of the API, and waits and retries when the API asks it to.

It stores nothing by itself. The tokens it obtains go to a token store you choose: a file only your script can read, your system's keyring, or nowhere at all.

This page is about the client. The API reference at api.reconnectchina.org/docs covers what the API does: the search parameters, dates and search terms, what a document holds, and the limits your organization's agreement sets.

Install

With pip:

pip install reconnectchina

In a project managed with uv:

uv add reconnectchina

The client needs Python 3.11 or later. It has no dependencies.

A first script

This logs in on the first run, keeps the login in a file next to the script, and searches the last thirty days:

from reconnectchina import Client, ConsolePrompt, SearchParameters, TokenFile

client = Client(
    tokens=TokenFile("reconnectchina-tokens.json"),
    prompt=ConsolePrompt(),
    offline=True,
)

first = client.search(
    SearchParameters(term="半导体 || semiconductor", strategy="body", from_="now-30d")
)
print(f"{first.hits} hits")

for summary in first.summaries:
    print(summary.id, summary.en.title)

The first run prints a link. Open it, log in with your reconnectchina.org account, and confirm. The script then carries on by itself. offline=True makes that login last beyond your session on reconnectchina.org. Later runs therefore reuse the saved file without asking, until the login has gone unused for 30 days or you revoke it under Applications in the reconnectchina.org account console.

Configuration

Client takes keyword arguments only, and every one is optional. The defaults are the public API over the whole corpus.

argument	default
server	https://api.reconnectchina.org	The API, and its login. https://sandbox.api.reconnectchina.org is the sandbox.
tokens	a TokenMemory of its own	Where the login is kept (see Tokens).
prompt	none	Shows the device login's link (see Logging in).
offline	False	Whether a login yields an offline token (see Logging in).
concurrency	1	How many requests this client has in progress at once.
retry	True	Whether to wait and retry when the API asks to.
max_retry_wait	600.0	How many seconds one call may spend waiting to retry.
on_retry	none	Called before each wait (see Waiting and retrying).
timeout	60.0	How many seconds one attempt may take.
transport	UrllibTransport()	What sends the requests (see Certificates and proxies).

server is an origin, such as https://api.reconnectchina.org, and must be https, because every request to it carries a token. The login is on the same origin, under /oauth. An origin rather than a host name is what the API itself hands out, so problem.server from a tier-served-elsewhere refusal can be passed as server as it is.

An argument given as None takes its default. The client reads no environment variables, so pass what you keep in the environment explicitly:

import os

from reconnectchina import Client

client = Client(server=os.environ.get("RECONNECTCHINA_SERVER"))

An argument the client cannot use is refused as soon as the client is made: a ValueError for a server that is not https, a TypeError for a value of the wrong type.

Logging in

The API accepts short-lived access tokens, which the client obtains from a longer-lived refresh token. You get a refresh token by logging in once with the device flow: the client shows a link, and the researcher opens it in any browser, logs in and confirms. No password passes through your code.

With a prompt, the client logs in by itself whenever it needs to: when it has no refresh token, or when the login server no longer accepts the one it has. Without one, it raises a LoginRequiredError instead, and you call login when it suits you, with a prompt of that call's own.

A prompt is any object with a show method. show receives a DeviceLogin, with the link to open and the time it expires_at. ConsolePrompt prints both to standard error. In a notebook, a prompt can show the link as something to click instead:

from IPython.display import Markdown, display

from reconnectchina import DeviceLogin


class NotebookPrompt:
    def show(self, login: DeviceLogin) -> None:
        display(
            Markdown(
                f"[Log in to ReConnect China]({login.link})"
                f" before {login.expires_at:%H:%M} UTC."
            )
        )


client.login(NotebookPrompt())

login returns once the researcher has confirmed. It raises a LoginFailedError when they decline or the link expires first.

By default a login yields an ordinary refresh token, which ends with the researcher's session on reconnectchina.org, after hours rather than months. That is enough for a notebook or a script that only needs to work while the researcher is at it.

A script or a service that runs without the researcher needs an offline token. That is a refresh token that keeps working until it has gone unused for 30 days, and for at most a year. It is a credential worth keeping safe, so the client asks for one only when you say so:

• offline=True on the client makes every login it starts ask for one, whether it starts the login by itself or through login();
• login(offline=True) asks for one for that login alone;
• login(offline=False) does the reverse on a client that has offline=True.

When a login ends

refresh_token_expires_at() returns when the refresh token stops working, as the login server last reported it. It returns a datetime in UTC, or None when the server reported no end.

For an offline token, every refresh moves the end out again by 30 days of disuse. For a script that runs regularly, the end therefore stays about 30 days away until the year is nearly up. When it comes closer than that, log in again ahead of time rather than finding out from a LoginRequiredError on the day it ends:

import sys
from datetime import datetime, timedelta, timezone

end = client.refresh_token_expires_at()
if end is not None and end - datetime.now(timezone.utc) < timedelta(days=14):
    print(f"The login ends on {end:%Y-%m-%d}; log in again before then.", file=sys.stderr)

login() on a client that already has a refresh token replaces it, and the store receives the new one. The old one keeps working until it ends, so every copy of a deployment's login can be replaced before any of them stops. Once none of them uses the old one, revoke it under Applications in the reconnectchina.org account console.

logout() revokes the refresh token at the login server and empties the store. Do that when a login is no longer needed, rather than only deleting the file: a copy of a revoked token is worthless.

Tokens

The client hands its login to a token store and gets it back from there. A store is any object with two methods: load(), which returns the Tokens last saved or None, and save(tokens), which keeps them, or forgets them when given None.

load is called once, before the first request. save is called each time any of the tokens changes:

• after a login;
• after each refresh (the login server may hand out a new refresh token with a new access token);
• with None after logout.

Three stores come with the client:

• TokenMemory, the default, keeps the tokens for as long as the object lives.
• TokenFile(path) keeps them in a JSON file that only your user can read. It replaces the file whole on every save, so an interrupted save leaves the old login rather than a broken file. logout() deletes the file. A file it cannot read is an error, and it is left as it is. On Windows, the file gets the permissions of the folder it is in.
• OfflineTokenFile(path) keeps an offline token alone, as plain text, in a file only your user can read (see An offline token in a file). It saves and deletes as TokenFile does.

Tokens holds:

attribute
refresh_token	The refresh token.
offline	Whether it is an offline token.
refresh_token_expires_at	When it stops working, as a datetime in UTC, or None.
access_token	The current access token, or None.
access_token_expires_at	When the access token stops working, or None.

to_dict() turns Tokens into something json.dumps takes, and Tokens.from_dict() turns that back. Keeping the access token is optional. A service that restarts often can keep it, so that a restart within its five minutes needs no refresh. A script can keep only the refresh token and what goes with it.

Both tokens are credentials. The client never logs them, never puts them in a URL, and sends them only to server. Tokens leaves them out of its repr(), so they do not appear in a traceback or a notebook's output either.

A store of your own can keep the login wherever you like, such as your system's keyring:

import json

import keyring

from reconnectchina import Client, Tokens


class KeyringTokens:
    def load(self) -> Tokens | None:
        stored = keyring.get_password("reconnectchina", "tokens")
        return Tokens.from_dict(json.loads(stored)) if stored else None

    def save(self, tokens: Tokens | None) -> None:
        if tokens is None:
            keyring.delete_password("reconnectchina", "tokens")
        else:
            keyring.set_password("reconnectchina", "tokens", json.dumps(tokens.to_dict()))


client = Client(tokens=KeyringTokens())

An offline token in a file

The API's documentation pages create an offline token for a script, and ask you to keep it in a file only you can read, such as reconnectchina-token, with nothing else in it. curl reads it from there, and so does OfflineTokenFile:

from reconnectchina import Client, ConsolePrompt, OfflineTokenFile

client = Client(
    tokens=OfflineTokenFile("reconnectchina-token"),
    prompt=ConsolePrompt(),
    offline=True,
)

The client writes the token back whenever the login server replaces it. Without the file, the first run logs in as in A first script and creates it, and a login the client starts once the token has ended yields another offline token. offline=True is what makes both of these offline tokens: an OfflineTokenFile refuses to keep any other kind.

An access token for a request of your own

access_token() returns the access token the client's own calls would send, for a request the client does not make itself: one sent with another HTTP library, or a curl command to paste. The client first refreshes the token if it has less than 30 seconds left. The token then lasts until five minutes after it was issued, and no longer, so ask again for each request rather than keeping it.

import json
import urllib.request

request = urllib.request.Request(
    "https://api.reconnectchina.org/v1/search",
    data=json.dumps({"from": "now-1d/d", "pageSize": 0}).encode(),
    headers={
        "Authorization": f"Bearer {client.access_token()}",
        "Content-Type": "application/json",
    },
)
with urllib.request.urlopen(request) as response:
    print(response.status)

A request like this is on its own: it gets none of the client's waiting and retrying (Waiting and retrying), and its body uses the API's own field names.

Calls

There is one method per operation of the API, named as in the API reference but in snake_case:

method	operation
search(parameters)	POST /v1/search
continue_search(next)	POST /v1/continue_search
get_details(ids)	POST /v1/detail, up to 32 ids
save_search(name, search)	PUT /v1/search/{name}
get_search(name)	GET /v1/search/{name}
run_search(name, run)	POST /v1/search/{name}
delete_search(name)	DELETE /v1/search/{name}
list_searches(page_size=…)	GET /v1/search
continue_search_list(next)	POST /v1/continue_search_list

search's parameters and run_search's run may be left out. Every call blocks until it has an answer. save_search and delete_search return None.

Parameters and results

Requests and responses are classes named as the schemas in the API reference, and are importable from reconnectchina. Their fields are the reference's, in snake_case: typeIn is type_in and pageSize is page_size. from is a Python keyword, so it is from_. Your editor shows each field's description from the reference.

A request class checks what it is given as it is made. A field it does not have and a value of the wrong type are both errors, before anything is sent:

from datetime import datetime

from reconnectchina import SearchParameters

try:
    SearchParameters(term=datetime.now(), strategy="body")
except TypeError as e:
    print(e)

A list of values may be a list or a tuple, but not a single string: write type_in=["news"], not type_in="news". A field left as None is not sent, and the API applies its default, which the reference and the field's description give. Whether a value is within the API's limits (a length, a page size, a listed value) is left to the API. It refuses one that is not with a 400 that names the field, by its name in the reference (/typeIn).

Objects are immutable. dataclasses.replace makes a changed copy:

import dataclasses

from reconnectchina import SearchParameters

week = SearchParameters(type_in=["law/decree"], from_="now-7d")
today = dataclasses.replace(week, from_="now-1d/d")

Data that is already written in the API reference's field names, such as a search kept in a file, goes in through from_dict. to_dict() gives back exactly what is sent to the API, in those same names:

import json

from reconnectchina import SearchParameters

with open("search.json") as f:  # {"from": "now-7d", "typeIn": ["law/decree"]}
    parameters = SearchParameters.from_dict(json.load(f))

print(json.dumps(parameters.to_dict(), ensure_ascii=False))

Every class has both methods, and each undoes the other. A field that from_dict does not know goes into extra (see Compatibility) rather than being refused. A misspelt field in a file therefore reaches the API, which refuses it with a 400 naming it.

A response's dates and times are datetime in UTC: a document's dates are instants (see Dates and time zones in the API's documentation for how to show one). Its lists are tuples. A field the API leaves out is None.

Fields left out

Your organization's agreement can leave fields of some documents out of an answer, such as their Chinese text. Such a field is None, and the document's hidden names it and says why, as a HiddenField(field="zh.headline", reason="agreement"). When everything in zh is left out, zh itself is None. The English title and summary are never left out:

from reconnectchina import SearchParameters

page = client.search(
    SearchParameters(term='低空经济 || "low-altitude economy"', strategy="body")
)
for summary in page.summaries:
    if summary.zh is not None and summary.zh.headline is not None:
        print(summary.id, " ".join(summary.zh.headline))
    else:
        print(summary.id, summary.en.title)

A field in hidden is named by its path in the API reference, with a dot between names. hidden may come to name other fields, and give reasons other than agreement. Treat every field it names as left out, whatever the reason. A search still matches the text that is left out, so a Chinese term can find a document whose Chinese it does not show.

Paging

A page that is not the last carries next. Hand it to the matching continue_… method for the following page, until a page comes without one:

from reconnectchina import SearchFirstPage, SearchPage, SearchParameters

page: SearchFirstPage | SearchPage = client.search(
    SearchParameters(type_in=["law/decree"], from_="now-7d")
)
while True:
    for summary in page.summaries:
        print(summary.id)
    if page.next is None:
        break
    page = client.continue_search(page.next)

A run of a saved search pages the same way, starting with run_search. The list of saved searches starts with list_searches and continues with continue_search_list. A type checker tells a search's next from a list's, so handing one to the other method is an error it reports.

hits, and the histograms the search asked for in histograms, come with the first page only. hits counts every match, but a search can be paged only as deep as your organization's agreement allows, and the page that reaches that depth carries no next either. If you walked far fewer summaries than hits, you reached that depth. To reach the rest, narrow the search into shorter spans of from_ and to.

A search's next expires, at the earliest an hour after the search began. The following page raises an ApiError with the type continuation-expired. Start the search again, or narrow it so that it finishes sooner.

Histograms

A search counts its hits over time, issuers, levels and topics when histograms names them, each with its own options. The time histogram's periods are days, months or years in UTC, unless time_zone names another zone:

from zoneinfo import ZoneInfo

from reconnectchina import HistogramRequests, SearchParameters, TimeHistogramOptions

counts = client.search(
    SearchParameters(
        from_="now[Europe/Berlin]-M/M",
        to="now[Europe/Berlin]/M",
        page_size=0,
        histograms=HistogramRequests(time=TimeHistogramOptions(time_zone="Europe/Berlin")),
    )
)
berlin = ZoneInfo("Europe/Berlin")
for bucket in counts.histograms.time if counts.histograms and counts.histograms.time else ():
    print(bucket.from_.astimezone(berlin).date(), bucket.hits)

time_zone takes the name of a zone in the time zone database, which your editor completes and a type checker checks, or a fixed offset from UTC, UtcOffset("+08:00"). An offset is written as a UtcOffset so that a misspelt name is not taken for one. A name follows the zone's daylight saving time and an offset does not. With date_field="published", UtcOffset("+08:00") counts by the day as printed in China (see Dates and time zones in the API's documentation). The bounds of each period are in UTC whatever the zone. A zone in from_ or to (now[Europe/Berlin]) is part of the expression string, which only the API checks.

Detail for many ids

get_details takes at most 32 ids at a time, the limit DetailRequest sets in the API reference. For more, send them in parts:

from reconnectchina import SearchParameters

ids = [s.id for s in client.search(SearchParameters(from_="now-1d/d", page_size=100)).summaries]

for i in range(0, len(ids), 32):
    result = client.get_details(ids[i : i + 32])
    for document in result.detail:
        print(document.id, document.en.title)
    if result.missing:
        print("not found:", ", ".join(result.missing))

detail is in no particular order. Match documents to your ids by id.

Saved searches

A search saved under a name can be run again later by anyone in your organization, with some of its parameters replaced for that run:

from reconnectchina import (
    RunSearchRequest,
    SaveSearchRequest,
    SearchOverrides,
    SearchParameters,
)

client.save_search(
    "semiconductors",
    SaveSearchRequest(
        title="Semiconductors, last 30 days",
        parameters=SearchParameters(
            term="半导体 || semiconductor", strategy="body", from_="now-30d"
        ),
    ),
)

today = client.run_search(
    "semiconductors", RunSearchRequest(parameters=SearchOverrides(from_="now-1d/d"))
)
print(f"{today.hits} hits since midnight UTC yesterday")

parameters is what search takes. title and description are optional, and are for the people who pick a search from a list. A run's parameters replace the saved ones of the same name for that run, each whole: a type_in given for the run is the run's entire type_in.

Saving under a name that exists replaces that search, title and description included. The name itself cannot be changed: to rename a search, save it under the new name and delete the old. Changing a saved search is fetching it and saving a changed copy. What get_search returns holds the parameters as they were saved, possibly by a newer client than yours, so they become SearchParameters through the API's names, and the title and description are carried over:

import dataclasses

from reconnectchina import SaveSearchRequest, SearchParameters

saved = client.get_search("semiconductors")
parameters = SearchParameters.from_dict(saved.parameters.to_dict())
client.save_search(
    "semiconductors",
    SaveSearchRequest(
        title=saved.title,
        description=saved.description,
        parameters=dataclasses.replace(parameters, level_in=["centre"]),
    ),
)

If you save a new name beyond what your organization may keep, the call raises an ApiError with the type saved-search-limit. On the sandbox, a saved search belongs to the login that saved it and is deleted a few days after it was last saved.

Errors

A call either returns what the API answered or raises one of these, all of them subclasses of ReconnectChinaError:

• ApiError: the API answered with an error. status is the HTTP status, and type is the problem type's name ("range-not-entitled", "continuation-expired", …). problem is the whole problem document, so problem.earliest_from, problem.latest_to and problem.server are there when the type carries them. Branch on type; problem.type is the address of its documentation. When a response is not a problem document at all (a proxy's error page, say), type and problem are None, and body holds the start of what came back.
• InvalidResponseError: the API answered with success, but not with what the reference says it answers. path is where in the response the difference is, such as summaries[3].dates.added. This should not happen; if it does, please tell us.
• LoginRequiredError: there is no refresh token, or the login server no longer accepts it: it has expired, been revoked, or was issued by another login server. reason holds what the login server said. Log in again.
• LoginFailedError: a device login ended without a token. error is what the login server said: "access_denied" when the researcher declined, "expired_token" when the link ran out first.
• NetworkError: no answer at all, even after retrying. The error underneath is its __cause__.
• AttemptTimeoutError: one attempt took longer than timeout. It is also a TimeoutError.

from reconnectchina import ApiError, SearchParameters

try:
    client.search(SearchParameters(from_="1800-01-01"))
except ApiError as e:
    if e.type != "range-not-entitled":
        raise
    print(f"The earliest from we may use is {e.problem.earliest_from}")

A mistake in how the client is called, such as a wrong type or a server that is not https, is a TypeError or a ValueError, raised before anything is sent.

The client writes nothing to the console and nothing to logging. Everything it has to report reaches you through an exception, a hook, or on_retry. ConsolePrompt prints, because you asked it to.

Waiting and retrying

The API asks a client to wait when it is busy, when the website needs the capacity, or when your organization or session has reached a limit. It does so with 429 or 503, and gives the number of seconds to wait in Retry-After. The client does as asked: it waits, then sends the same request again, until the call succeeds or its time is up. Only then does it raise the ApiError.

What it retries, and how:

what happened	what the client does
429 or 503	Waits Retry-After seconds, plus up to a tenth more at random, then retries. Never sooner.
401	Refreshes the access token and retries, once.
no answer: refused, reset, name not found	Waits 1 s, doubling to at most 30 s, each wait a random part of that, then retries.
no connection within 10 s	The same: nothing reached the API, so the request is sent again, on a new connection.
timeout passed	Raises AttemptTimeoutError. The API may still be working on the request, so sending it again would only add to the load.
any other status	Raises the ApiError. Waiting does not change a 400, a 403 or a 410, and a 500 is ours to fix.

The login server's token endpoint is retried the same way when it cannot be reached or answers 5xx.

A wait holds back the whole client, not only the call that was refused, because the limit it ran into belongs to your session or organization rather than to one request. Calls started during a wait wait too.

max_retry_wait bounds how long one call may spend waiting: ten minutes by default. When the next wait would take a call past it, the client raises at once rather than waiting only to give up. retry=False turns retrying off. Every 429 and 503 then raises at once, and its ApiError carries retry_after in seconds, for a program that would rather decide itself.

on_retry is told about every wait before it starts, with the attempt, the delay in seconds and the error that caused it:

import logging

from reconnectchina import Client, RetryEvent

log = logging.getLogger("my-script")


def log_retry(event: RetryEvent) -> None:
    log.warning(
        "attempt %d failed (%s); retrying in %.0f s", event.attempt, event.error, event.delay
    )


client = Client(on_retry=log_retry)

Threads and asyncio

A Client can be shared by every thread of a program, and one per program is how it is meant to be used: its waits and its login are shared, and so is its limit of concurrency requests at a time. With the default of 1, calls from several threads queue in the client rather than at the API.

The calls block. In asyncio code, run them in a thread so they do not hold up the event loop:

import asyncio

from reconnectchina import SearchParameters


async def main() -> None:
    first = await asyncio.to_thread(client.search, SearchParameters(from_="now-1d/d"))
    print(f"{first.hits} hits")


asyncio.run(main())

Ctrl-C ends a call on the main thread wherever it is, including in a wait before a retry.

Certificates and proxies

The API's certificates are publicly trusted, so a Python that trusts the usual root certificates needs nothing more. A Python installed from python.org on macOS does not, until you run Install Certificates.command in its folder under Applications. Before that, every call fails with a NetworkError whose cause mentions CERTIFICATE_VERIFY_FAILED.

To use root certificates of your own, give the transport an ssl.SSLContext:

import ssl

from reconnectchina import Client, UrllibTransport

context = ssl.create_default_context(cafile="/etc/ssl/certs/ca-certificates.crt")
client = Client(transport=UrllibTransport(ssl_context=context))

Connecting, the TLS handshake included, may take 10 seconds. A link that needs longer can have it:

from reconnectchina import Client, UrllibTransport

client = Client(transport=UrllibTransport(connect_timeout=30))

Requests go through the proxy that https_proxy names, as with everything else built on Python's urllib. The client never follows a redirect, so a token it sends never goes anywhere but server.

Compatibility

/v1 of the API grows without breaking: a response may gain fields, and some fields may gain values (strategy, date_field, the issuers). The client passes on what it does not know rather than refusing it:

• A value it does not know yet is kept as a string, and its types say so: a response's level is one of the known values, or any other str.
• A field it does not know yet is kept in the object's extra, a mapping from the field's name in the reference to its value. Saving a saved search you fetched keeps such a field.
• You can also set extra yourself, to send a field the API has gained before the client has caught up.

A request is strict the other way. The API refuses a field it does not know with 400, and a request class refuses it sooner, when it is made.

The client's major version changes when its own interface does, not with the API's releases.