Skip to main content

The gddriver is an efficient tool for operating the storage service such as oss and ftp.

Project description

gddriver

pipeline status coverage report

存储驱动:封装oss/ftp等底层存储服务的api

快速开始

使用oss驱动进行操作

import gddriver

from gddriver import models
from gddriver import providers


host = "https://oss-cn-shenzhen.aliyuncs.com"
port = None  # oss无需填端口号
container_name = 'bucket name'
credential = models.Credential(
    access_key_id="",
    access_key_secret="",
    access_key_token=""
)


# operator 支持使用with的上下文,在每次使用之后释放资源
with gddriver.GenericOperator(
                provider=providers.Provider.OSS, credential=credential,
                host=host,port=port) as operator:
    obj_list, next_marker = operator.list_container_objects(container_name=container_name)
    for i in obj_list:
        print(i)

使用ftp驱动进行操作

import gddriver

from gddriver import models
from gddriver import providers


host = "127.0.0.1"
port = 21
container_name = 'bucket name'
credential = models.Credential(
    user="",
    password="",
)


# operator 支持使用with的上下文,在每次使用之后释放资源
with gddriver.GenericOperator(
                provider=providers.Provider.FTP, 
                credential=credential,host=host,port=port) as operator:
    obj_list, next_marker = operator.list_container_objects(container_name=container_name)
    for i in obj_list:
        print(i)

自定义分片上传

# -*- coding: utf-8 -*-

from gddriver import Provider
from gddriver import transmission
from gddriver import models


container_name = "your bucket or ftp directory"
credential = models.Credential(
    access_key_id="your access key id",
    access_key_secret="your access key secret",
    user='user name',
    password='password'
)

operator = transmission.GenericOperator(
    host="endpoint",
    credential=credential,
    port=None,
    provider=Provider.OSS  # Provider.FTP
)


object_name = "sample/test-multi"
upload_id = operator.init_multipart_upload(
        container_name=container_name,
        object_name=object_name
)

with open("/path/to/file", "rb") as f:
    part_num = 0
    parts = []
    while True:
        part_num += 1
        # oss 最小分片要求100KB
        c = f.read(1024 * 100)
        if not c:
            break

        part_upload_request = models.PartUploadRequest(
            container_name=container_name,
            stream=c,
            object_name=object_name,
            part_number=part_num,
            upload_id=upload_id
        )
        # 注意:ftp等存储介质下,operator不是线程安全的
        result = operator.upload_part(request=part_upload_request)
        # 等价
        # parts.append({"etag": result.part_info.etag, "part_number": result.part_info.part_number})
        parts.append(result.part_info)
    print("Parts: {}".format(parts))

    request = models.CompleteMultipartUploadRequest(
        container_name=container_name,
        object_name=object_name,
        upload_id=upload_id,
        parts=parts
    )
    operator.complete_multipart_upload(request=request)

模块说明

models.Object

文件/对象的原信息,包含name, size等基础信息以及extra字典中包含的额外信息

extra中(可能包含)的信息在不同的存储中实现如下:

  • oss:
    • last_modified: UNIX TIMPSTAMP
    • etag: etag
    • object_type: Normal Multipart Appendable ...
    • storage_class: Standard Archive ...
    • restore: 是否为恢复/恢复中的归档文件 (get_object_meta时获得)
    • restore_finished: 是否恢复完成 (get_object_meta时获得)
    • expiry_date: 归档文件恢复的过期时间 (get_object_meta时获得)
  • ftp:
    • parent: /root/prefix
    • is_file: 是否为文件
    • short_name: 对象名,没有前缀

storage_class属性获取文件在对象存储中的类型,如 Standard Archived等,依据存储介质而定,默认为Standard

models._Request

存储服务操作请求

  • _Request
    • StreamTransferRequest 流式(上传)请求
      • FTPStreamUploadRequest ftp流式上传
      • OSSStreamUploadRequest oss流式上传
      • AppendRequest 追加上传
      • PartUploadRequest 分片上传
    • FileTransferRequest 文件传输请求
      • OSSUploadRequest oss文件上传
      • FTPUploadRequest ftp文件上传
      • OSSDownloadRequest oss文件下载
      • FTPDownloadRequest ftp文件下载
    • CopyRequest 复制请求
      • OSSCopyRequest oss文件复制请求
    • StreamDownloadRequest 流式下载请求
      • FTPStreamDownloadRequest ftp流式下载
      • OSSStreamDownloadRequest oss流式下载

models.OperationResult

部分请求返回的操作结果,以下接口返回的值为OperationResult的子类对象:

  • upload_file
  • upload_object_via_stream
  • append_object
  • download_file
  • download_object_as_stream
  • restore_object
  • upload_part
  • complete_multipart_upload

部分模型的介绍:

  • DownloadResult 下载结果,包含server_checksum client_checksum checksum_type等信息
  • StreamDownloadResult 一个可迭代对象,同时包含stream(类File的流)和client_checksum(需要完成流的读取之后才能获得)属性
  • UploadResult 上传结果,包含checksum checksum_type等信息
    • AppendResult 追加上传的结果, checksum为追加上传这一部分的校验码
  • RestoreResult 归档文件恢复的结果,包含finished(是否完成)和expiry_date(过期时间)属性
  • PartUploadResult 分片上传结果,包含part_info属性,用于完成分片上传请求

models.Credential

Credential用于管理存储服务的认证信息,user, password, access_key_id, access_key_secret, access_key_token 等信息

base.Connection

存储服务的连接信息,使用驱动执行操作时,需要通过Connection对象创建连接、发送请求等,不同的服务有不同的实现方式

  • drivers.ftp.FTPConnection
  • drivers.ftp.OSSConnection

base.StorageDriver

定义了访问存储介质时的操作接口:

  • 上传
    • 上传本地文件
    • 通过流上传
    • 追加上传(追加上传只能以流的形式上传)
  • 下载
    • 下载到本地文件
    • 以流的形式下载
  • 删除
    • 删除对象
    • 通过前缀删除(部分实现)
    • 批量删除(部分实现)
  • 冷备、恢复
  • 获取前面链接
  • 遍历container中的对象/文件
  • 初始化Connection实例

StorageDriver对象是线程安全的:StorageDriver的所有实现,都只是"定义"了在这种存储介质下操作的行为方式,所有的操作都基于函数传入的connection参数,并不会管理用户的Connection信息

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

gddriver-0.2.10.tar.gz (39.8 kB view hashes)

Uploaded Source

Built Distribution

gddriver-0.2.10-py2.py3-none-any.whl (45.2 kB view hashes)

Uploaded Python 2 Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page