Couchbase Python Client

Python client for Couchbase

NOTE: This is the documentation for the 4.x version of the client. This is mostly compatible with the older 3.x version. Please refer to the release32 branch for the older 3.x version.

Contents

• Prerequisites
• Installing
• Building
• Using the SDK
• Logging
• Building Documentation
• Testing
• Contributing
• Support & Additional Resources
• License
• Appendix

Prerequisites

• Couchbase Server
• You may need a C++ compiler supporting C++ 17 and Python development files, unless a binary wheel is available for your platform. With the 4.0.2 release, wheels are available on Windows, MacOS and Linux (via manylinux) for Python 3.7 - 3.10.
• CMake (version >= 3.18), unless a binary wheel is available for your platform.
• Git, unless a binary wheel is available for your platform.
• OpenSSL is now required for the 4.x Python SDK.
• If using the Twisted Framework and the txcouchbase API, Twisted >= 21.7.0 is required.

Debian and Ubuntu

First-time setup:

$ sudo apt install git-all python3-dev python3-pip python3-setuptools cmake build-essential libssl-dev

NOTE: We have provided Dockerfiles to demonstrate steps to achieve a working setup for various linux platforms. See the dockerfiles folder in the Python SDK examples folder for details.

See Debian and Ubuntu install section to install SDK.

RHEL and CentOS

First-time setup:

$ sudo yum install git-all gcc gcc-c++ python3-devel python3-pip python3-setuptools cmake openssl-devel

❗IMPORTANT❗
Some of the defaults for older operating systems like Centos/RHEL 7 and 8 have defaults to do not meet the 4.x Python SDK minimum requirements. Be sure to update to the minimum requirements prior to installing the SDK. Most notably be sure to check the following:

• The default Python version might be less than 3.7. If so, the Python version will need to be udpated.
• The default OpenSSL version might be less than 1.1.1. If so, the OpenSSL version will need to be updated.
• The gcc version must provide C++17 support. If the installed gcc version does not support C++17, gcc will need to be updated.
• The installed CMake version might be less than 3.17. If so, the CMake version will need to be updated. Check out the steps here to update CMake.

NOTE: We have provided Dockerfiles to demonstrate steps to achieve a working setup for various linux platforms. See the dockerfiles folder in the Python SDK examples folder for details.

See RHEL and Centos install section to install SDK.

Mac OS

It is not recommended to use the vendor-supplied Python that ships with OS X. Best practice is to use a Python virtual environment such as pyenv or venv (after another version of Python that is not vendor-supplied has been installed) to manage multiple versions of Python.

❗IMPORTANT❗
There can be a problem when using the Python (3.8.2) that ships with Xcode on Catalina. It is advised to install Python with one of the following:

• pyenv
• Homebrew
• Install Python via python.org

pyenv

See detailed walk through in Appendix. Also, see pyenv install docs for further details.

NOTE: If using pyenv, make sure the python interpreter is the pyenv default, or a virtual environment has been activiated. Otherwise cmake might not be able to find the correct version of Python3 to use when building.

Homebrew

See Homebrew install docs for further details.

Get the latest packages:

$ brew update

Install Python:

$ brew install python

Update path:

• zsh:

  $ echo 'export PATH="/usr/local/bin:"$PATH' >> ~/.zshrc
  

• bash:

  $ echo 'export PATH="/usr/local/bin:"$PATH' >> ~/.bash_profile
  

Install OpenSSL:

$ brew install openssl@1.1

To get OpenSSL to be found by cmake on macos, find where openssl was installed via homebrew:

brew info openssl@1.1

This will show you how to get it seen by pkg-config. To check that it worked, do this:

pkg-config --modversion openssl

See Mac OS install section to install SDK.

Windows

Wheels are available on Windows for Python 3.7, 3.8, 3.9 and 3.10.

Best practice is to use a Python virtual environment such as venv or pyenv (checkout the pyenv-win project) to manage multiple versions of Python.

If wanting to install from source, see the Windows building section for details.

See Windows install section to install SDK.

Installing

Back to Contents

You can always get the latest supported release version from pypi.

NOTE: If you have a recent version of pip, you may use the latest development version by issuing the following incantation:

pip install git+https://github.com/couchbase/couchbase-python-client.git

NOTE: The Python Client installer relies on PEP517 which older versions of PIP do not support. If you experience issues installing it is advised to upgrade your PIP/setuptools installation as follows:

python3 -m pip install --upgrade pip setuptools wheel

Debian and Ubuntu

First, make sure the prerequisites have been installed.

Install the SDK:

$ python3 -m pip install couchbase

RHEL and CentOS

First, make sure the prerequisites have been installed.

Install the SDK:

$ python3 -m pip install couchbase

Mac OS

First, make sure the prerequisites have been installed.

Install the SDK:

$ python -m pip install couchbase

Windows

First, make sure the prerequisites have been installed.

NOTE: Commands assume user is working within a virtual environment. For example, the following commands have been executed after downloading and installing Python from python.org:
-C:\Users\Administrator\AppData\Local\Programs\Python\Python39\python -m venv C:\python\python39
-C:\python\python39\Scripts\activate

Install the SDK (if using Python 3.7, 3.8, 3.9 or 3.10):

python -m pip install couchbase

Alternative Installation Methods

In order to successfully install with the following methods, ensure a proper build system is in place (see the Windows building section for details).

Source Install (i.e. no wheel)

First, ensure all the requirements for a build system are met.

Install the SDK:

python -m pip install couchbase --no-binary couchbase

Local Install

First, ensure all the requirements for a build system are met.

Clone this Python SDK repository:

git clone --depth 1 --branch <tag_name> --recurse-submodules https://github.com/couchbase/couchbase-python-client.git

Where tag_name is equal to the latest release.
Example: git clone --depth 1 --branch 4.0.0 --recurse-submodules https://github.com/couchbase/couchbase-python-client.git

Move into the directory created after cloning the Python SDK repository:

cd couchbase-python-client

NOTE: If the --recurse-submodules option was not used when cloning the Python SDK repository, run (after moving into the cloned repository directory) git submodule update --init --recursive to recursively update and initialize the submodules.

Install the SDK from source:

python -m pip install .

Anaconda/Miniconda

To use the SDK within the Anaconda/Miniconda platform, make sure the prerequisites for the desired Operating System are met:

• Debian and Ubuntu
• RHEL and Centos
• Mac OS
• Windows

In the Anaconda Prompt, create a new environment:

(base) C:\Users\user1>conda create -n test_env python=3.9

Activate the environment

(base) C:\Users\user1>conda activate test_env

Install the SDK:

(test_env) C:\Users\user1>python -m pip install couchbase

NOTE: If using Windows, and no wheel is available, see the Alternative Install Methods Windows section. The same process should work within the Anaconda/Miniconda platform.

Building

Back to Contents

NOTE: This section only applies to building from source.

Build System Setup

Linux

Make sure the prerequisites have been installed:

• Debian and Ubuntu
• RHEL and Centos

Mac OS

First, make sure the prerequisites have been installed.

Install cmake:

$ brew install cmake

Install command line developer tools:

$ xcode-select --install

NOTE: It is possible that installing or updating to the the latest version of Xcode is needed.

If setuptools is not installed:

$ python -m pip install setuptools

Windows

Requirements

• Download and install Git
• Download and install Visual Studio 2019
  • Check Desktop development with C++ prior to installing
• Download and install CMake >= v 3.18
• Download and install Python

VS2019 Notes

If seeing issues when trying to build (steps in ), some things to check/try:

• Try running the build commands within the Developer Command Prompt for VS2019
• Make sure MSBuild can find the correct VCTargetsPath
  • It is possible the VCTargetsPath environment variable needs to be set. The below example is based on a typical path, but the actual setting should match that of your current environment setup.
    • set VCTargetsPath=C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Microsoft\VC\v160
• Make sure CMake is picking up the correct generator
  • It is possible the CMAKE_GENERATOR environment variable needs to be set
    • set CMAKE_GENERATOR=Visual Studio 16 2019

Build the Python SDK

Clone this Python SDK repository:

git clone --depth 1 --recurse-submodules https://github.com/couchbase/couchbase-python-client.git

NOTE: If the --recurse-submodules option was not used when cloning the Python SDK repository, run (after moving into the cloned repository directory) git submodule update --init --recursive to recursively update and initialize the submodules.

Move into the directory created after cloning the Python SDK repository:

cd couchbase-python-client

The following will compile the module locally:

python setup.py build_ext --inplace

You can also modify the environment CFLAGS and LDFLAGS variables.

❗WARNING: If you do not intend to install this module, ensure you set the PYTHONPATH environment variable to this directory before running any scripts depending on it. Failing to do so may result in your script running against an older version of this module (if installed), or throwing an exception stating that the couchbase module could not be found.

Install

pip install .

❗WARNING: If you are on Linux/Mac OS you may need to remove the build directory: rm -rf ./build before installing with pip: pip3 install ..

Using the SDK

Back to Contents

Connecting

See official documentation for further details on connecting.

# needed for any cluster connection
from couchbase.cluster import Cluster
from couchbase.auth import PasswordAuthenticator

# options for a cluster and SQL++ (N1QL) queries
from couchbase.options import ClusterOptions, QueryOptions

# get a reference to our cluster
cluster = Cluster.connect('couchbase://localhost', ClusterOptions(
  PasswordAuthenticator('Administrator', 'password')))

NOTE: The authenticator is always required.

Basic Operations

See official documentation for further details on Working with Data.

Building upon the example code in the Connecting section:

# get a reference to our bucket
cb = cluster.bucket('travel-sample')

# get a reference to the default collection
cb_coll = cb.default_collection()

# get a document
result = cb_coll.get('airline_10')
print(result.content_as[dict])

# using SQL++ (a.k.a N1QL)
call_sign = 'CBS'
sql_query = 'SELECT VALUE name FROM `travel-sample` WHERE type = "airline" AND callsign = $1'
query_res = cluster.query(sql_query, QueryOptions(positional_parameters=[call_sign]))
for row in query_res:
    print(row)

Async Operations

The Python Couchbase SDK supports asynchronous I/O through the use of the asyncio (Python standard library) or the Twisted async framework.

Asyncio

To use asyncio, import acouchbase.cluster instead of couchbase.cluster. The acouchbase API offers an API similar to the couchbase API.

from acouchbase.cluster import Cluster, get_event_loop
from couchbase.options import ClusterOptions
from couchbase.auth import PasswordAuthenticator


async def write_and_read(key, value):
    cluster = await Cluster.connect('couchbase://localhost',
                      ClusterOptions(PasswordAuthenticator('Administrator', 'password')))
    cb = cluster.bucket('default')
    await cb.on_connect()
    cb_coll = cb.default_collection()
    await cb_coll.upsert(key, value)
    result = await cb_coll.get(key)
    return result

loop = get_event_loop()
rv = loop.run_until_complete(write_and_read('foo', 'bar'))
print(rv.content_as[str])

Twisted

To use with Twisted, import txcouchbase.cluster instead of couchbase.cluster. The txcouchbase API offers an API similar to the couchbase API.

NOTE: The minimum required Twisted version is 21.7.0.

❗WARNING: The 4.x SDK introduced a breaking change where the txcouchbase package must be imported prior to importing the reactor (see example below). This is so that the asyncio reactor can be installed.

# IMPORTANT -- the txcouchbase import must occur PRIOR to importing the reactor
import txcouchbase
from twisted.internet import reactor, defer

from txcouchbase.cluster import TxCluster
from couchbase.options import ClusterOptions
from couchbase.auth import PasswordAuthenticator


def after_upsert(res, key, d):
    print('Set key.  Result CAS: ', res.cas)
    # trigger get_document callback
    d.callback(key)

def upsert_document(key, doc):
    d = defer.Deferred()
    res = cb.upsert(key, doc)
    res.addCallback(after_upsert, key, d)
    return d

def on_get(res, _type=str):
    print('Got res: \n', res.content_as[_type])
    reactor.stop()

def get_document(key):
    res = cb.get(key)
    res.addCallback(on_get)


# create a cluster object
cluster = TxCluster('couchbase://localhost',
                    ClusterOptions(PasswordAuthenticator('Administrator', 'password')))

# create a bucket object
bucket = cluster.bucket('default')
# create a collection object
cb = bucket.default_collection()

d = upsert_document('testDoc_1', {'id': 1, 'type': 'testDoc', 'info': 'fake document'})
d.addCallback(get_document)

reactor.run()

Logging

Back to Contents

The Couchbase Python SDK provides comprehensive logging for debugging and monitoring.

Quick Start

Enable logging via environment variable:

$ export PYCBC_LOG_LEVEL=info
$ python your_app.py

Or configure programmatically:

import logging
from couchbase import configure_logging

configure_logging('my_app', level=logging.DEBUG)

Documentation

For complete logging documentation including configuration options, examples, troubleshooting, and best practices, see LOGGING.md.

For working example scripts demonstrating various logging configurations, see examples/logging/.

Building Documentation

Back to Contents

The documentation is using Sphinx and a number of extensions. To build the documentation be sure to pip install the sphinx_requirements.txt.

python3 -m pip install -r sphinx_requirements.txt

To build the documentation, go into the docs directory and run:

make html

The HTML output can be found in docs/build/html/.

Alternatively, you can also build the documentation from the top-level directory:

python setup.py build_sphinx

Once built, the docs will be in in build/sphinx/html. You can open the index.html file with the following command:

open docs/build/sphinx/html/index.html

Testing

Back to Contents

For running the tests, be sure to pip install the dev_requirements.txt. The Couchbase Python SDK uses pytest for the test suite.

python3 -m pip install -r dev_requirements.txt

The tests need a running Couchbase instance. For this, a test_config.ini file must be present, containing various connection parameters. The default test_config.ini file may be found in the tests directory. You may modify the values of the test_config.ini file as needed.

To run the tests for the blocking API (i.e. couchbase API):

python -m pytest -m pycbc_couchbase -p no:asyncio -v -p no:warnings

To run the tests for the asyncio API (i.e. acouchbase API):

python -m pytest -m pycbc_acouchbase --asyncio-mode=strict -v -p no:warnings

Contributing

Back to Contents

We welcome contributions from the community! Please see follow the steps outlined here to get started.

The Python SDK uses pre-commit in order to handle linting, formatting and verifying the code base. pre-commit can be installed either by installing the development requirements:

python3 -m pip install -r dev_requirements.txt

Or by installing pre-commit separately

python3 -m pip install pre-commit

To run pre-commit, use the following:

pre-commit run --all-files

License

Back to Contents

The Couchbase Python SDK is licensed under the Apache License 2.0.

See LICENSE for further details.

Support & Additional Resources

Back to Contents

If you found an issue, please file it in our JIRA.

The Couchbase Discord server is a place where you can collaborate about all things Couchbase. Connect with others from the community, learn tips and tricks, and ask questions. Join Discord and contribute.

You can ask questions in our forums.

The official documentation can be consulted as well for general Couchbase concepts and offers a more didactic approach to using the SDK.

Appendix

Back to Contents

Mac OS pyenv Install

See pyenv install docs for further details.

Get the latest packages:

$ brew update

For TLS/SSL support:

$ brew install openssl@1.1

Install pyenv:

$ brew install pyenv

NOTE: It is possible that Xcode might need to be reinstalled. Try one of the following:

• Use command xcode-select --install
• Install the latest version of Xcode

For Zsh, run the following commands to update .zprofile and .zshrc. See pyenv install docs for further details on other shells.

$ echo 'eval "$(pyenv init --path)"' >> ~/.zprofile

$ echo 'eval "$(pyenv init -)"' >> ~/.zshrc

NOTE: You need to restart your login session for changes to take affect. On MacOS, restarting terminal windows should suffice.

Install Python version:

$ pyenv install 3.9.7

Set local shell to installed Python version:

$  pyenv local 3.9.7

To use virtualenvwrapper with pyenv, install pyenv-virtualenvwrapper:

$ brew install pyenv-virtualenvwrapper

To setup a virtualenvwrapper in your pyenv shell, run either pyenv virtualenvwrapper or pyenv virtualenvwrapper_lazy

NOTE: If issues with pyenv virtualenvwrapper, using python -m pip install virtualenvwrapper should accomplish the same goal.

Make a virtualenv:

$ mkvirtualenv python-3.9.7-test

Install the SDK:

$ python -m pip install couchbase

Run individual pre-commit commands

To run pre-commit hooks separately, use the following.

autopep8

pre-commit run autopep8 --all-files

bandit

pre-commit run bandit --all-files

clang-format

pre-commit run clang-format --all-files

flake8

pre-commit run flake8 --all-files

isort

pre-commit run isort --all-files

trailing whitespace

pre-commit run trailing-whitespace --all-files