kinto-http 10.10.0

Kinto client

Kinto python client

kinto-http is the Python library to interact with a Kinto server.

There is also a similar client in JavaScript.

Installation

Use pip:

$ pip install kinto-http

Usage

Here is an overview of what the API provides:

import kinto_http

client = kinto_http.Client(server_url="http://localhost:8888/v1",
                           auth=('alexis', 'p4ssw0rd'))

records = client.get_records(bucket='default', collection='todos')
for i, record in enumerate(records):
    record['title'] = 'Todo {}'.format(i)
    client.update_record(data=record)

Instantiating a client

The passed auth parameter is a requests authentication policy.

By default, a simple tuple will become a Basic Auth authorization request header, that can authenticate users with Kinto Accounts.

import kinto_http

auth = ('alexis', 'p4ssw0rd')

client = kinto_http.Client(server_url='http://localhost:8888/v1',
                           auth=auth)

It is also possible to pass a bucket ID and/or collection ID to set them as default values for the parameters of the client operations.

client = Client(bucket="payments", collection="receipts", auth=auth)

After creating a client, you can also replicate an existing one and overwrite some key arguments.

client2 = client.clone(collection="orders")

An asynchronous client is also available. It has all the same endpoints as the sync client except for the batch operations.

from kinto_http import AsyncClient

auth = ('alexis', 'p4ssw0rd')

client = AsyncClient(server_url='http://localhost:8888/v1', auth=auth)
info = await client.server_info()
assert 'schema' in info['capabilities'], "Server doesn't support schema validation."

Using a Bearer access token to authenticate (OpenID)

import kinto_http

client = kinto_http.Client(auth=kinto_http.BearerTokenAuth("XYPJTNsFKV2"))

The authorization header is prefixed with Bearer by default. If the header_type is customized on the server, the client must specify the expected type: kinto_http.BearerTokenAuth("XYPJTNsFKV2", type="Bearer+OIDC")

Note

Passing a string containing Bearer will be instantiate a kinto_http.BearerTokenAuth() object automatically.

In other words, kinto_http.Client(auth="Bearer+OIDC XYPJTNsFKV2") is equivalent to kinto_http.Client(auth=kinto_http.BearerTokenAuth("XYPJTNsFKV2", type="Bearer+OIDC"))

Custom headers

Custom headers can be specified in the Client constructor, and will be sent in every request:

import kinto_http

client = kinto_http.Client(server_url="http://server/v1", headers={
    "Allow-Access": "CDN",
    "User-Agent": "blocklist-updater"
})

Getting server information

You can use the server_info() method to fetch the server information:

from kinto_http import Client

client = Client(server_url='http://localhost:8888/v1')
info = client.server_info()
assert 'schema' in info['capabilities'], "Server doesn't support schema validation."

Bucket operations

• get_bucket(id=None, **kwargs): retrieve single bucket

• get_buckets(**kwargs): retrieve all readable buckets

• create_bucket(id=None, data=None, **kwargs): create a bucket

• update_bucket(id=None, data=None, **kwargs): create or replace an existing bucket

• patch_bucket(id=None, changes=None, **kwargs): modify some fields in an existing bucket

• delete_bucket(id=None, **kwargs): delete a bucket and everything under it

• delete_buckets(**kwargs): delete every writable buckets

Groups operations

• get_group(id=None, bucket=None, **kwargs): retrieve single group

• get_groups(bucket=None, **kwargs): retrieve all readable groups

• create_group(id=None, data=None, bucket=None, **kwargs): create a group

• update_group(id=None, data=None, bucket=None, **kwargs): create or replace an existing group

• patch_group(id=None, changes=None, bucket=None, **kwargs): modify some fields in an existing group

• delete_group(id=None, bucket=None, **kwargs): delete a group and everything under it

• delete_groups(bucket=None, **kwargs): delete every writable groups

Collections

• get_collection(id=None, bucket=None, **kwargs): retrieve single collection

• get_collections(bucket=None, **kwargs): retrieve all readable collections

• create_collection(id=None, data=None, bucket=None, **kwargs): create a collection

• update_collection(id=None, data=None, bucket=None, **kwargs): create or replace an existing collection

• patch_collection(id=None, changes=None, bucket=None, **kwargs): modify some fields in an existing collection

• delete_collection(id=None, bucket=None, **kwargs): delete a collection and everything under it

• delete_collections(bucket=None, **kwargs): delete every writable collections

Records

• get_record(id=None, bucket=None, collection=None, **kwargs): retrieve single record

• get_records(bucket=None, collection=None, **kwargs): retrieve all readable records

• get_paginated_records(bucket=None, collection=None, **kwargs): paginated list of records

• get_records_timestamp(bucket=None, collection=None, **kwargs): return the records timestamp of this collection

• create_record(id=None, data=None, bucket=None, collection=None, **kwargs): create a record

• update_record(id=None, data=None, bucket=None, collection=None, **kwargs): create or replace an existing record

• patch_record(id=None, changes=None, bucket=None, collection=None, **kwargs): modify some fields in an existing record

• delete_record(id=None, bucket=None, collection=None, **kwargs): delete a record and everything under it

• delete_records(bucket=None, collection=None, **kwargs): delete every writable records

Permissions

The objects permissions can be specified or modified by passing a permissions to create_*(), patch_*(), or update_*() methods:

client.create_record(data={'foo': 'bar'},
                     permissions={'read': ['group:groupid']})


record = client.get_record('123', collection='todos', bucket='alexis')
record['permissions']['write'].append('leplatrem')
client.update_record(data=record)

Get or create

In some cases, you might want to create a bucket, collection, group or record only if it doesn’t exist already. To do so, you can pass the if_not_exists=True to the create_*() methods:

client.create_bucket(id='blog', if_not_exists=True)
client.create_collection(id='articles', bucket='blog', if_not_exists=True)

Delete if exists

In some cases, you might want to delete a bucket, collection, group or record only if it exists already. To do so, you can pass the if_exists=True to the delete_* methods:

client.delete_bucket(id='bucket', if_exists=True)

Patch operations

The .patch_*() operations receive a changes parameter.

from kinto_http.patch_type import BasicPatch, MergePatch, JSONPatch


client.patch_record(id='abc', changes=BasicPatch({'over': 'write'}))

client.patch_record(id='todo', changes=MergePatch({'assignee': 'bob'}))

client.patch_record(id='receipts', changes=JSONPatch([
    {'op': 'add', 'path': '/data/members/0', 'value': 'ldap:[e-mail]'}
]))

Concurrency control

The create_*(), patch_*(), and update_*() methods take a safe argument (default: True).

If True, the client will ensure that the object doesn’t exist already for creations, or wasn’t modified on the server side since we fetched it. The timestamp will be implicitly read from the last_modified field in the passed data object, or taken explicitly from the if_match parameter.

Batching operations

Rather than issuing a request for each and every operation, it is possible to batch several operations in one request (sync client only).

Using the batch() method as a Python context manager (with):

with client.batch() as batch:
    for idx in range(0, 100):
        batch.update_record(data={'id': idx})

Note

Besides the results() method, a batch object shares all the same methods as another client.

Reading data from batch operations is achieved by using the results() method available after a batch context is closed.

with client.batch() as batch:
    batch.get_record('r1')
    batch.get_record('r2')
    batch.get_record('r3')

r1, r2, r3 = batch.results()

Errors

Failing operations will raise a KintoException, which has request and response attributes.

try:
    client.create_group(id="friends")
except kinto_http.KintoException as e:
    if e.response and e.response.status_code == 403:
        print("Not allowed!")

Requests Timeout

A timeout value in seconds can be specified in the client constructor:

client = KintoClient(server_url="...", timeout=5)

To distinguish the connect from the read timeout, use a tuple:

client = KintoClient(server_url="...", timeout=(3.05, 27))

For an infinit timeout, use None:

client = KintoClient(server_url="...", timeout=None)

See the timeout documentation of the underlying requests library.

Retry on error

When the server is throttled (under heavy load or maintenance) it can return error responses.

The client can hence retry to send the same request until it succeeds. To enable this, specify the number of retries on the client:

client = Client(server_url='http://localhost:8888/v1',
                auth=credentials,
                retry=10)

The Kinto protocol lets the server define the duration in seconds between retries. It is possible (but not recommended) to force this value in the clients:

client = Client(server_url='http://localhost:8888/v1',
                auth=credentials,
                retry=10,
                retry_after=5)

Pagination

When the server responses are paginated, the client will download every page and merge them transparently.

The get_paginated_records() method returns a generator that will yield each page:

for page in client.get_paginated_records():
    records = page["data"]

It is possible to specify a limit for the number of items to be retrieved in one page:

records = client.get_records(_limit=10)

In order to retrieve every available pages with a limited number of items in each of them, you can specify the number of pages:

records = client.get_records(_limit=10, pages=float('inf'))  # Infinity

History

If the built-in history plugin is enabled, it is possible to retrieve the history of changes:

# Get the complete history of a bucket
changes = client.get_history(bucket='default')

# and optionally use filters
hist = client.get_history(bucket='default', _limit=2, _sort='-last_modified', _since='1533762576015')
hist = client.get_history(bucket='default', resource_name='collection')

The history of a bucket can also be purged with:

client.purge_history(bucket='default')

Endpoint URLs

The get_endpoint() method returns an endpoint URL on the server:

client = Client(server_url='http://localhost:8888/v1',
                auth=('token', 'your-token'),
                bucket="payments",
                collection="receipts")

print(client.get_endpoint("record",
                          id="c6894b2c-1856-11e6-9415-3c970ede22b0"))

# '/buckets/payments/collections/receipts/records/c6894b2c-1856-11e6-9415-3c970ede22b0'

Handling datetime and date objects

In addition to the data types supported by JSON, kinto-http.py also supports native Python date and datetime objects.

In case a payload contain a date or a datetime object, kinto-http.py will encode it as an ISO formatted string.

Please note that this transformation is only one-way. While reading a record, if a string contains a ISO formated string, kinto-http.py will not convert it to a native Python date or datetime object.

If you know that a field will be a datetime, you might consider encoding it yourself to be more explicit about it being a string for Kinto.

Command-line scripts

In order to have common arguments and options for scripts, some utilities are provided to ease configuration and initialization of client from command-line arguments.

import argparse
import logging

from kinto_http import cli_utils

logger = logging.getLogger(__name__)

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Download records")
    cli_utils.set_parser_server_options(parser)

    args = parser.parse_args()

    cli_utils.setup_logger(logger, args)

    logger.debug("Instantiate Kinto client.")
    client = cli_utils.create_client_from_args(args)

    logger.info("Fetch records.")
    records = client.get_records()
    logger.warn("{} records.".format(len(records)))

The script now accepts basic options:

$ python example.py --help

usage: example.py [-h] [-s SERVER] [-a AUTH] [-b BUCKET] [-c COLLECTION] [-v]
                  [-q] [-D]

Download records

optional arguments:
  -h, --help            show this help message and exit
  -s SERVER, --server SERVER
                        The location of the remote server (with prefix)
  -a AUTH, --auth AUTH  BasicAuth credentials: `token:my-secret` or
                        Authorization header: `Bearer token`
  -b BUCKET, --bucket BUCKET
                        Bucket name.
  -c COLLECTION, --collection COLLECTION
                        Collection name.
  --retry RETRY         Number of retries when a request fails
  --retry-after RETRY_AFTER
                        Delay in seconds between retries when requests fail
                        (default: provided by server)
  -v, --verbose         Show all messages.
  -q, --quiet           Show only critical errors.
  -D, --debug           Show all messages, including debug messages.

Run tests

In one terminal, run a Kinto server:

$ make run-kinto

In another, run the tests against it:

$ make tests

(Optional) Install a git hook:

therapist install

CHANGELOG

This document describes changes between each past release.

10.10.0 (2022-06-27)

New features

• Use Bearer token Auth object if specified string for auth contains Bearer

• Use Basic Auth if specified string for auth contains :

10.9.0 (2022-02-04)

New features

• Access client configured bucket and collection names at client.bucket_name and client.collection_name respectively.

10.8.0 (2021-12-03)

New features

• Asynchronous client is now available: from kinto_http import AsyncClient (#268)

Internal changes

• Replaced unittest with pytest

10.7.0 (2020-01-09)

New features

• Add ability to specify headers from Client constructor

10.6.1 (2019-11-13)

Bug fixes

• Do not try to parse content on 204 No Content responses

10.6.0 (2019-09-20)

New features

• Specify requests timeout in client constructor

10.5.0 (2019-09-10)

New features

• Add history support (fixes #112), Thanks @FlorianKuckelkorn!

10.4.1 (2019-05-22)

Bug fixes

• Handle bearer tokens without a colon.

10.4.0 (2019-05-09)

• Add support for Bearer tokens in the CLI utilities.

• Use black for code formatting.

10.3.0 (2019-03-07)

New features

• Add support for OAuth access tokens (OpenID) with the BearerTokenAuth() helper. See README. (#197)

10.2.0 (2018-12-17)

New features

• Created new method on client to get paginated records get_paginated_records. (#175)

• Allow additional querystring params in get_*() methods

10.1.1 (2018-11-13)

Bug fixes

• Fix JSON support for in_ and exclude_. (#188)

10.1.0 (2018-11-05)

New feature

• Convert params values as JSON values before passing them to requests. (#185)

10.0.0 (2018-10-15)

Breaking changes

By default, the client now raises an exception when a 4XX error occurs in a batch request (#154)

In order to ignore those errors as before, instantiate the client with ignore_batch_4xx=True.

New feature

• Raise a specific CollectionNotFound exception rather than a generic KintoException.

Bug fixes

• Handle date and datetime object in a Kinto payload. They will be formated as ISO date JSON strings.

Internal changes

• Update tests to work with Kinto 11.0.0.

• Update tests to use stdlib mock module.

9.1.2 (2018-04-17)

Internal changes

• Get rid of six

9.1.1 (2018-02-07)

Bug fixes

• Fix patch methods in batch requests (fixes #171)

9.1.0 (2018-02-05)

Significant changes

• When the server returns a 409 Conflict error response, the request will be retried if the retry parameter is superior to zero (fixes #167)

New Features

• Expose kinto-http and Python module version in the User-Agent (#157)

• Support different PATCH types. Now, instead of settling for the “default” patch method offered by the Kinto server, you can choose by importing a PatchType subclass from kinto_http.patch_type. (Fixes #125.)

Bug fixes

• No longer support method arguments on the update_bucket, update_group, update_collection, and update_record methods. This argument only existed to support the patch_* methods and was never intended to be part of the public API.

9.0.1 (2017-05-30)

Bug fixes

• Fix exception rendering (fixes #153)

9.0.0 (2017-05-25)

Breaking changes

• The client will fail a batch only when a 5XX error occurs (#148)

New Features

• Log all the batch responses (#148)

• Log the request and the batch responses in debug (#148)

• Allow reading responses from batch requests with the results() method. (#146)

8.0.1 (2017-05-16)

Bug fixes

• Fix get_records_timestamp JSONDecode error while trying to decode the body of a HEAD response. (#144)

8.0.0 (2017-05-11)

Breaking changes

• Fetch only one page when _limit is specified and allow to override this with a pages argument (fixes #136)

• Make client methods API consistent by forcing keyword parameters (#119)

• Deduce the id of a resource with the value of id in data if present (#143)

• Drop Python 2.7 support. Now supports Python 3.5+

New Features

• Keep tracks of Backoff headers and raise an BackoffException if we are not waiting enough between two calls. (#53)

• Add --retry and --retry-after to CLI utils helpers (fixes #126)

Bug fixes

• Fix retry behaviour when responses are successful (fixes #129)

• Fix Retry-After value to be read as integer rather than string. (#131)

• Fix No JSON could be decoded ValueError (fixes #116)

Internal changes

• make tests-once to run functional tests in order to calculate coverage correctly (#131)

7.2.0 (2017-03-17)

• Only provide the data JSON field when data is provided. (#122)

7.1.0 (2017-03-16)

Bug fixes

• Method for plural endpoints now return list of objects instead of odict_values.

New features

• Add logging (fixes #36, #110, thanks @sahildua2305)

Documentation

• Fix explanation about safe/if_match/last_modified

• Fix missing methods in docs (#102, thanks @gabisurita)

• Improve contributing guide (#104, #111, thanks @Sayli-Karnik)

• Show how to use the FxABearerTokenAuth auth (#117)

7.0.0 (2016-09-30)

Breaking changes

• Removed if_exists argument from the delete_*s methods for plural endpoints (#98, thanks @mansimarkaur!)

New features

• Added CRUD methods for the group endpoints (#95, thanks @mansimarkaur!)

Documentation

• Add contributing guide (#90, thanks @sahildua2305!)

6.2.1 (2016-09-08)

New features

• Add a if_exists flag to delete methods to avoid raising if the item was already deleted. (#82)

• Improving the clone method to keep all the previous parameters values if missing as parameters. (#91)

6.1.0 (2016-08-04)

New features

• Add a get_records_timestamp method to get the collection ETag. (#81)

6.0.0 (2016-06-10)

Breaking changes

• Rename kinto_client to kinto_http (#74)

5.0.0 (2016-05-12)

Breaking changes

• Rename the last_modified client parameter into if_match (#68)

New features

• Display a better message when having 403 on create_collection and create_record methods (#49)

• Expose get_endpoints as part of the client API (#60)

• Add a server_info method to retrieve the root url info (#70)

Internal changes

• Rename the Batch class into BatchSession (#52)

• Change readthedocs.org urls in readthedocs.io (#71)

4.1.0 (2016-04-26)

New features

• Add new methods get_buckets(), delete_buckets(), delete_bucket(), delete_collections(), delete_records(), patch_record() (#55)

Internal changes

• Functional tests are now tested on Kinto master version (#65)

4.0.0 (2016-03-08)

Breaking changes

• The function cli_utils.set_parser_server_options() was renamed cli_utils.add_parser_options() (#63)

New features

• add_parser_options can now exclude bucket and collection parameters. (#63)

• create_client_from_args can now works even with no bucket or collection arguments (#63)

Bug fixes

• Do not sent body in GET requests. (#62)

3.1.0 (2016-02-16)

New features

• Add CLI helpers to configure and instantiate a Client from command-line arguments (#59)

3.0.0 (2016-02-10)

Breaking changes

• Updated the update_collection() signature: data is now the fisr argument (#47)

New features

• Added a retry option for batch requests (#51)

• Use the “default” bucket if nothing is specified (#50)

• Added a if_not_exists argument to the creation methods (#42)

• Added a replication mechanism in kinto_http.replication (#26)

• Handle the last_modified argument on update or create operations (#24)

Bug fixes

• Do not force the JSON content-type in requests if multipart-encoded files are sent (#27)

• Fail the batch operations early (#47)

• Remove un-needed requirements (FxA) (#43)

• Use max_batch_request from the server to issue more than one batch request (#30)

• Make sure batch raises an error when needed (#28)

• Fix an invalid platform error for some versions of python (#31)

• Do not lowercase valid IDs (#33)

Documentation

• Add documentation about client.batch (#44)

2.0.0 (2015-11-18)

• Added support for pagination in records requests (#13)

• Added support for If-Match / If-None-Match headers for not overwriting existing records (#14)

• Changed the API of the batch support. There is now a client.batch() context manager (#17)

• Added support of the PATCH methods to update records / collections (#19)

1.0.0 (2015-11-09)

Breaking changes

• Rewrote the API to be easier to use (#10)

0.2.0 (2015-10-28)

Breaking changes

• Rename kintoclient to kinto_client (#8)

Features

• Add the endpoints class. (#9)

• Add batching utilities. (#9)

Internal changes

• Add universal wheel configuration.

0.1.1 (2015-09-03)

Initial version

• A client to synchroneously call a Kinto server.