CyberDB 0.9.3

CyberDB is a lightweight Python in-memory database. It is designed to use Python's built-in data structures Dictionaries, Lists for data storage, efficient communication through Socket TCP, and provide data persistence. This module can be used in hard disk database caching, Gunicorn inter-process communication, distributed computing and other fields.

CyberDB

中文版

CyberDB is a lightweight Python in-memory database. It is designed to use Python's built-in data structures Dictionaries, Lists for data storage, efficient communication through Socket TCP, and provide data persistence. This module can be used in hard disk database caching, Gunicorn inter-process communication, distributed computing and other fields.

The CyberDB server uses Asyncio for TCP communication. The client is developed based on Socket, so it supports the Gevent coroutine, but it has not yet adapted to Asyncio. Both the server and the client support PyPy, and it is recommended to use PyPy to run for better performance.

In high concurrency scenarios, the performance bottleneck of traditional databases is mainly hard disk I/O. Even if CyberDB is developed based on the dynamic language Python, the speed is still much faster than that of hard disk databases (such as MySQL), and CyberDB can be used as its cache. In addition, the core of CyberDB lies in programming in a Pythonic way, you can use CyberDB like Dictionaries and Lists.

Installation

1. Enter the command window, create a virtual environment, and enter the following commands in turn

Linux and macOS:

python3 -m venv venv # Create a virtual environment.
. venv/bin/activate # Activate the virtual environment.

Windows:

python -m venv venv # Create a virtual environment.
venv\Scripts\activate # Activate the virtual environment.

2.Install CyberDB, enter

pip install --upgrade pip
pip install cyberdb

If your server and client are running in two different project directories, please install CyberDB in the virtual environment of the server and client respectively.

Links

• GitHub: https://github.com/Cyberbolt/CyberDB
• PyPI: https://pypi.org/project/CyberDB/
• Documentation: https://www.cyberlight.xyz/static/cyberdb
• CyberLight: https://www.cyberlight.xyz/

Quick to Use

In this module, please use CyberDict and CyberList instead of dict and list (a TCP-based Dictionaries-like, Lists-like object).

Server

Run the database server.

import time
import cyberdb

db = cyberdb.Server()
# The data is persistent, the backup file is data.cdb, and the backup cycle is every 900 seconds.
db.set_backup('data.cdb', cycle=900)
# Set the TCP address, port number, and password.
# The start method will not block the operation. If you want the operation to block, please use the run method instead of start, and the parameters remain unchanged.
db.start(host='127.0.0.1', port=9980, password='123456')

while True:
    time.sleep(10000)

After the above server runs, data.cdb and data_backup.cdb (backup files) will be generated (or overwritten) in the project root directory every 900 seconds. The file can be read the next time the database is started using the load method.

Client

Connect to the Database

import cyberdb

# Generate a client instance and connect.
client = cyberdb.connect(host='127.0.0.1', port=9980, password='123456')

Generate proxy Object

# Generate proxy for this request.
proxy = client.get_proxy()
# Automatically obtain database connections from the connection pool.
proxy.connect()

The proxy object is not thread-safe, please generate the proxy object separately in each thread (or coroutine), and obtain the database connection through the connect method. You only need to use the close method to return the connection after the operation is completed, and the returned connection is managed intelligently by the client object.

Manipulate proxy Objects

Create CyberDict and CyberList

# Create dict1 and dict2 tables of type CyberDict and 
# list1 table of type CyberList in the database respectively.
proxy.create_cyberdict('dict1')
proxy.create_cyberdict('dict2')
proxy.create_cyberlist('list1')

dict1 = proxy.get_cyberdict('dict1')
dict2 = proxy.get_cyberdict('dict2')
list1 = proxy.get_cyberlist('list1')

The dict1, dict2, and list1 obtained here are all network objects, and the data is transmitted through TCP. The three objects are controlled by the proxy, and when proxy.close() is called to return the connection, the three objects will also be invalid. Similarly, use proxy.connect() to get connections from the connection pool again, and dict1, dict2, list1 also become available.

Once you understand this operation, you can operate dict1, dict2 like Dictionaries, and list1 like Lists! (CyberDict and CyberList support most methods of Dictionaries, Lists)

Examples are as follows

Common Operations of CyberDict

Add key-value pairs in dict1 and dict2

dict1[0] = 100
dict1['test'] = 'Hello CyberDB!'
dict2[0] = 200

Get the corresponding value

dict1.get(0)

100

dict2[0]

200

View dict1 and dict2 (can also be printed with print )

dict1

{0: 100, 'test': 'Hello CyberDB!'}

dict2

{0: 200}

Get length

len(dict1)

2

Delete key-value pair

del dict1[0]

dict1

{'test': 'Hello CyberDB!'}

Empty dict1

dict1.clear()
dict1

{}

Common Operations of CyberList

Generate the contents of list1

for i in range(5):
    list1.append(99)
    
list1

[99, 99, 99, 99, 99]

Change coordinate values

list1[3] = 100
list1

[99, 99, 99, 100, 99]

To slice

list1[3:]

[100, 99]

Get the length of list1

len(list1)

5

Print each element of list1 by iterating

for v in list1:
    print(v)

99
99
99
100
99

It is strongly recommended to use a for loop to iterate CyberList, each iteration will get v from the server, and the space complexity of the client is o(1). Iteration can also be used for CyberDict. In the iteration of CyberDict, the client space complexity is o(n), and n is the size of CyberDict.keys().

Release the Proxy Object

After the use is complete, return the connection of the proxy to the connection pool.

proxy.close()

The proxy object also supports context managers, such as

with client.get_proxy() as proxy:
    list1 = proxy.get_cyberlist('list1')
    print(list1)

[99, 99, 99, 100, 99]

Generalize

With CyberDB, memory performance can be fully utilized, and different processes (or even different hosts) can communicate through Python's data structures. For more tutorials, please refer to the documentation, Thank you!

Notice

Due to encoding limitations, CyberDB will recognize 0 as None, but it does not affect the calculation, please convert None to 0 in the desired position.

中文版

CyberDB 是一个轻量级的 Python 内存数据库。它旨在利用 Python 内置数据结构 Dictionaries、Lists 作数据存储，通过 Socket TCP 高效通信，并提供数据持久化。该模块可用于 硬盘数据库缓存、Gunicorn 进程间通信、分布式计算 等领域。

CyberDB 服务端使用 Asyncio 进行 TCP 通信。客户端基于 Socket 开发，所以支持 Gevent 协程，暂未适配 Asyncio。服务端和客户端均支持 PyPy，推荐使用 PyPy 运行，以获得更好的性能。

高并发场景下，传统数据库的性能瓶颈主要在硬盘 I/O，即便 CyberDB 基于动态语言 Python 开发，速度仍然远快于硬盘数据库(如 MySQL)，CyberDB 可以作为它的缓存。此外，CyberDB 的核心在于使用 Pythonic 的方式编程，你可以像使用 Dictionaries 和 Lists 一样使用 CyberDB。

安装方法

1.进入命令窗口，创建虚拟环境，依次输入以下命令

Linux 和 macOS:

python3 -m venv venv # 创建虚拟环境
. venv/bin/activate # 激活虚拟环境

Windows:

python -m venv venv # 创建虚拟环境
venv\Scripts\activate # 激活虚拟环境

2.安装 CyberDB，依次输入

pip install --upgrade pip
pip install cyberdb

如果你的服务端和客户端在两个不同的项目目录运行，请分别在服务端、客户端的虚拟环境中安装 CyberDB。

链接

• GitHub: https://github.com/Cyberbolt/CyberDB
• PyPI: https://pypi.org/project/CyberDB/
• 文档: https://www.cyberlight.xyz/static/cyberdb-chn
• 电光笔记: https://www.cyberlight.xyz/

快速使用

该模块中请使用 CyberDict 和 CyberList 替代 dict 和 list （一种基于 TCP 的类 Dictionaries、类 Lists 对象）。

服务端

服务端初始化，设置备份和 TCP 监听地址。

import time
import cyberdb

db = cyberdb.Server()
# 数据持久化，备份文件为 data.cdb，备份周期 900 秒一次。
db.set_backup('data.cdb', cycle=900)
# 设置 TCP 地址、端口号、密码，生产环境中密码建议使用大小写字母和数字的组合。
# start 方法不会阻塞运行，若希望该操作阻塞，请使用 run 方法代替 start，参数不变。
db.start(host='127.0.0.1', port=9980, password='123456')

while True:
    time.sleep(10000)

上述服务端运行后，每 900 秒将在此项目根目录生成（或覆盖）data.cdb。下次启动数据库可使用 load 方法读取该文件。

客户端

连接数据库

import cyberdb

# 生成客户端实例并连接。
client = cyberdb.connect(host='127.0.0.1', port=9980, password='123456')

生成 proxy 对象

# 生成本次请求的 proxy。
proxy = client.get_proxy()
# 从连接池自动获取数据库连接。
proxy.connect()

建议在每个线程(或协程)中单独生成 proxy 对象，并通过 connect 方法获取数据库连接。你只需在操作完成后使用 close 方法归还连接即可，归还后的连接由 client 对象智能管理。

操作 proxy 对象

创建 CyberDict 和 CyberList

# 在数据库中分别创建类型为 CyberDict 的 dict1、dict2 表，
# 类型为 CyberList 的 list1 表。
proxy.create_cyberdict('dict1')
proxy.create_cyberdict('dict2')
proxy.create_cyberlist('list1')

dict1 = proxy.get_cyberdict('dict1')
dict2 = proxy.get_cyberdict('dict2')
list1 = proxy.get_cyberlist('list1')

此处获取的 dict1、dict2、list1 均为网络对象，数据通过 TCP 传输。三个对象受 proxy 控制，当调用 proxy.close() 归还连接后，三个对象也会失效。同样，使用 proxy.connect() 可重新从连接池获取连接，dict1、dict2、list1 也变为可用。

了解此操作后，你便可以像操作 Dictionaries 一样操作 dict1、dict2，像操作 Lists 一样操作 list1 了！（CyberDict 和 CyberList 支持 Dictionaries、Lists 的大部分方法）

示例如下

CyberDict 常用操作

在 dict1 和 dict2 中新增键值对

dict1[0] = 100
dict1['test'] = 'Hello CyberDB!'
dict2[0] = 200

获取对应的值

dict1.get(0)

100

dict2[0]

200

查看 dict1 和 dict2 (也可以使用 print 打印)

dict1

{0: 100, 'test': 'Hello CyberDB!'}

dict2

{0: 200}

获取长度

len(dict1)

2

删除键值对

del dict1[0]
dict1

{'test': 'Hello CyberDB!'}

清空 dict1

dict1.clear()
dict1

{}

CyberList 常用操作

生成 list1 的内容

for i in range(5):
    list1.append(99)
    
list1

[99, 99, 99, 99, 99]

更改坐标值

list1[3] = 100
list1

[99, 99, 99, 100, 99]

进行切片

list1[3:]

[100, 99]

获取 list1 的长度

len(list1)

5

通过迭代打印 list1 每个元素

for v in list1:
    print(v)

99
99
99
100
99

强烈推荐使用 for 循环迭代 CyberList，每次迭代将从服务端获取 v，客户端的空间复杂度为 o(1)。迭代同样可用于 CyberDict，CyberDict 的迭代中，客户端空间复杂度为 o(n), n 为 CyberDict.keys() 的长度。

释放 proxy 对象

使用完成，将 proxy 的连接归还至连接池即可。

proxy.close()

proxy 对象同样支持上下文管理器，如

with client.get_proxy() as proxy:
    list1 = proxy.get_cyberlist('list1')
    print(list1)

[99, 99, 99, 100, 99]

概括

有了 CyberDB，便能充分利用内存性能，不同进程(甚至不同主机)能通过 Python 的数据结构通信。更多教程请参考文档，感谢你的支持！

注意

由于编码限制，CyberDB 会将 0 识别为 None，但并不影响计算，请在所需位置将 None 转为 0。