流式处理数据

利用生成器、协程等流式处理数据。

对数据按多个函数的组合进行过滤、去重，对数据列进行转换，对数据进行计算。

对数据进行质量控制（QC）检查，确保最后的数据与预期相符。

将数据写入csv等。

原理和特点说明

编程主要要到了生成器，各个类用for循环从上游抽取数据，用yield给下游提供数据。通过改写or规则，利用|操作符将各个组建组合起来。

同时支持协程，构建广播、路由等数据节点。

特点：

高拓展性。
低内存占用。
近依赖python基础包。

参考项目

整体思路主要参考了这个项目：https://github.com/sandabuliu/python-stream。

安装

>>> pip install file-stream

使用

从CSV文件读取数据，按条件筛选后输出到屏幕。

reader = CsvReader('/home/hetao/Data/p5w/数据分析/IPO_RoadShow.txt', delimiter='\t', encoding='gbk')
fit = Filter(lambda x: True)
writer = ScreenOutput(end='\r')
p = reader | fit | writer
p.output()

更多范例参见test文件夹。

Metadata

Release files for file-stream 0.2.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for file-stream 0.2.4
File	Size	Uploaded	Provenance	Actions
file_stream-0.2.4.tar.gz	9.7 kB	Oct 15, 2020	Without provenance	Details for file_stream-0.2.4.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for file-stream 0.2.4
File	Interpreter	ABI	Platform	Actions Reset
file_stream-0.2.4-py3-none-any.whl 13.0 kB Oct 15, 2020 Without provenance	Python 3	none	any	Details for file_stream-0.2.4-py3-none-any.whl

Total release size: 22.7 kB

Release files / file_stream-0.2.4.tar.gz

Download URL	file_stream-0.2.4.tar.gz
Size	9.7 kB
Tags	Source
SHA-256 checksum How to use checksums	`92aaec15786a4bd5517e78130b4e6bc1ab8353b971d95fb32d91e49a074f2d67`
BLAKE2b-256 checksum How to use checksums	`545751a6ee27f14db59e25d51090508dd92bea2af3db483097ec45155343bf55`
Upload date	Oct 15, 2020
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/36.5.0.post20170921 requests-toolbelt/0.9.1 tqdm/4.50.2 CPython/3.6.3`

Release files / file_stream-0.2.4-py3-none-any.whl

Download URL	file_stream-0.2.4-py3-none-any.whl
Size	13.0 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`bd021141c61f63790b6bd0f7c98f1bc1c1bfedfb81bf7c11daea608b9b8e74c1`
BLAKE2b-256 checksum How to use checksums	`d465d8e3acd29808ff032cc62fe6bf3b49ffa906fba00b30744bd059c58913e7`
Upload date	Oct 15, 2020
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/36.5.0.post20170921 requests-toolbelt/0.9.1 tqdm/4.50.2 CPython/3.6.3`

Security for file-stream 0.2.4

Report project as malware

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

file-stream 0.2.4