Build and Installation

For users:

pip install libmc

Usage:

import libmc

mc = libmc.Client(['localhost:11211', 'localhost:11212'])
mc.set('foo', 'bar')
assert mc.get('foo') == 'bar'

Under the hood

Under the hood, libmc consists of 2 parts: an internal, fully-functional memcached client implementation in C++ and a Cython wrapper around that implementation. Dynamic memory allocation and memory-copy are slow, so we’ve tried our best to avoid them. libmc also supports the set_multi command, which is not natively supported by the memcached protocol. Some techniques have been applied to make set_multi command extremely fast in libmc (compared to similiar libraries).

Configuration

import libmc
from libmc import (
    MC_HASH_MD5, MC_POLL_TIMEOUT, MC_CONNECT_TIMEOUT, MC_RETRY_TIMEOUT
)

mc = libmc.Client(
    [
    'localhost:11211',
    'localhost:11212',
    'remote_host',
    'remote_host mc.mike',
    'remote_host:11213 mc.oscar'
    ],
    do_split=True,
    comp_threshold=0,
    noreply=False,
    prefix=None,
    hash_fn=MC_HASH_MD5,
    failover=False
)

mc.config(MC_POLL_TIMEOUT, 100)  # 100 ms
mc.config(MC_CONNECT_TIMEOUT, 300)  # 300 ms
mc.config(MC_RETRY_TIMEOUT, 5)  # 5 s

• servers: a list of memcached server addresses. Each address should be formated as hostname[:port] [alias], where port and alias are optional. If port is not given, the default port 11211 will be used. If given, alias will be used to compute the server hash, which would otherwise be computed based on host and port (i.e. whichever portion is given).

• do_split: splits large values (up to 10MB) into chunks (<1MB). The memcached server implementation will not store items larger than 1MB, however in some environments it is beneficial to shard up to 10MB of data. Attempts to store more than that are ignored. Default: True.

• comp_threshold: compresses large values using zlib. If buffer length > comp_threshold > 0 (in bytes), the buffer will be compressed. If comp_threshold == 0, the string buffer will never be compressed. Default: 0

• noreply: controls memcached’s noreply feature. Default: False

• prefix: The key prefix. default: ''

• hash_fn: hashing function for keys. possible values:

  • MC_HASH_MD5

  • MC_HASH_FNV1_32

  • MC_HASH_FNV1A_32

  • MC_HASH_CRC_32

  default: MC_HASH_MD5

  NOTE: fnv1_32, fnv1a_32, crc_32 implementations in libmc are per each spec, but they’re not compatible with corresponding implementions in libmemcached.

• failover: Whether to failover to next server when current server is not available. Default: False

• MC_POLL_TIMEOUT Timeout parameter used during set/get procedure. Default: 300 ms

• MC_CONNECT_TIMEOUT Timeout parameter used when connecting to memcached server in the initial phase. Default: 100 ms

• MC_RETRY_TIMEOUT When a server is not available due to server-end error, libmc will try to establish the broken connection in every MC_RETRY_TIMEOUT s until the connection is back to live. Default: 5 s

NOTE: The hashing algorithm for host mapping on continuum is always md5.

Contributing to libmc

Feel free to send a Pull Request. For feature requests or any questions, please open an Issue.

For SECURITY DISCLOSURE, please disclose the information responsibly by sending an email to security@douban.com directly instead of creating a GitHub issue.

FAQ

from gevent import monkey
monkey.patch_socket()

Acknowledgments

• Thanks to @fahrenheit2539 and the llvm project for the standalone. SmallVector implementation.

• Thanks to @miloyip for the high performance i64toa implementation.

• Thanks to Ivan Novikov for the research in THE NEW PAGE OF INJECTIONS BOOK: MEMCACHED INJECTIONS.

• Thanks to the PolarSSL project for the md5 implementation.

• Thanks to @lericson for the benchmark script in pylibmc.

• Thanks to the libmemcached project and some other projects possibly not mentioned here.

Contributors

• @mckelvin

• @zzl0

• @windreamer

• @lembacon

• @seansay

• @mosasiru

• @jumpeiMano

Who is using

• 豆瓣

• 下厨房

• Some other projects on GitHub

• Want to add your company/organization name here? Please feel free to send a PR!

Documentation

https://github.com/douban/libmc/wiki

Footnotes

LICENSE

Copyright (c) 2014-2020, Douban Inc. All rights reserved.

Licensed under a BSD license: https://github.com/douban/libmc/blob/master/LICENSE.txt