Skip to main content

Flownet: A distributed dataflow orchestration system.

Project description


SAGE-Flownet

SAGE-Flownet 是一个轻量、原生 Python 的动态数据流执行引擎。它以“嵌套动态图”为核心模型,把用户写下的普通 Python 函数编译为可执行的 FlowTask DAG,并由运行时按需调度执行,让复杂的工作流像调用函数一样自然。

它不是另一个固定模板的流水线框架。Flownet 将远程调用、闭包、回调以及子图嵌套视为一等公民,在不牺牲 Python 语义的前提下,支持灵活组合和横向扩展。

安装与使用

pip install isage-flownet

⚠️ PyPI 包名 vs. import 名

SAGE 生态约定:PyPI 包名使用 isage- 前缀(因为 sage 在 PyPI 已被占用), 但 Python import 名使用 sage.* 命名空间。

操作 命令
安装 pip install isage-flownet
import from flownet import flow, init

import isage_flownet 不存在,请始终使用 flownet

from flownet import flow, init, create_actor

@flow
def my_pipeline(init_stream):
    return init_stream.map(lambda x: x * 2)

init()  # 启动运行时

✧ 项目如何工作(概述)

  • Flow 定义与编译:通过 @flow 装饰器将函数包装为 Flow 定义(FlowDef),编译为包含 Transformation 的 FlowTask 图。
  • 数据流建模DataStream.map/flatmap 等算子创建上下游关系,形成可追踪的依赖图和执行边。
  • 运行时执行FlownetRuntime 负责处理请求栈(TransformationFrame),并由 operator handler 决定目标调用与转发策略。
  • Actor 调用统一化ActorRef / ActorMethodRef 将本地与远程调用统一抽象,运行时根据目标地址完成派发。
  • FlowRun 输出通道:运行时通过 FlowRunRef 的输出通道暴露“写入输入—等待输出”的调用方式,负责 fork/join 逻辑与结果收集。

✧ 预期目标与效果

  • 保持原生 Python 开发体验:无需额外 DSL,工作流像写函数一样简单。
  • 支持动态与嵌套的复杂流程:运行时编译、依赖跟踪与延迟执行适配不确定结构。
  • 可扩展的分布式执行:为本地/远程统一执行提供清晰路径,便于水平扩展。
  • 更可靠的并发与结果聚合:由运行时完成 fork/join、收集与异常处理,降低业务代码复杂度。

Intellectual Property

All Rights Reserved applies to repository contents authored by Jingyuan Tian, including code, documentation, benchmarks, and related assets.

The following copied lab projects are excluded from this claim and retain their original ownership/license context: applications/SAGE, applications/sage-studio, applications/sageData.

  1. License terms: LICENSE
  2. Ownership details: COPYRIGHT.md

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fluty-0.1.1.tar.gz (284.7 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

fluty-0.1.1-py3-none-any.whl (377.2 kB view details)

Uploaded Python 3

File details

Details for the file fluty-0.1.1.tar.gz.

File metadata

  • Download URL: fluty-0.1.1.tar.gz
  • Upload date:
  • Size: 284.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.2.0 CPython/3.11.14

File hashes

Hashes for fluty-0.1.1.tar.gz
Algorithm Hash digest
SHA256 6611877b65eeb40c474fe2f07a81c724be0004668e7fd735ba80535fae2d2311
MD5 7740070552ed1ada26d7d1789f3dd177
BLAKE2b-256 26d5c718d287901f5189e911b52b417ee6383e6dede19864f41fc3c5e27c32ff

See more details on using hashes here.

File details

Details for the file fluty-0.1.1-py3-none-any.whl.

File metadata

  • Download URL: fluty-0.1.1-py3-none-any.whl
  • Upload date:
  • Size: 377.2 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.2.0 CPython/3.11.14

File hashes

Hashes for fluty-0.1.1-py3-none-any.whl
Algorithm Hash digest
SHA256 3e044d1a0c5b48ea2e58acfb3bd3153882abf2fc3a806cfee25d25394ee778f0
MD5 75b5b466dbbf64cc97a8d0974aadbae5
BLAKE2b-256 7e684e050034b02be80b8600f494cc5e4a6729177e8f05b836c747721b1836f5

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page