Pass directories between metaflow steps.
Project description
Magic Directories
An experimental plugin for passing data in directories in between steps.
Warning: this package is highly experimental.
Installation
pip install metaflow-plugin-magicdir
Usage
You can use @magicdir
to pass local directories between metaflow steps. This will also work remotely.
#flow.py
from metaflow import FlowSpec, step
from metaflow.plugins import magicdir
class MagicDirFlow(FlowSpec):
@magicdir(dir='mydir')
@step
def start(self):
with open('mydir/output1', 'w') as f:
f.write('hello world')
with open('mydir/output2', 'w') as f:
f.write('hello world again')
self.next(self.end)
@magicdir(dir='mydir')
@step
def end(self):
print('first', open('mydir/output1').read())
print('second', open('mydir/output1').read())
if __name__ == "__main__":
MagicDirFlow()
If you run the above flow, you will see the following output:
> python flow.py run --with batch
2022-02-05 12:49:51.503 Workflow starting (run-id 11):
2022-02-05 12:49:51.511 [11/start/1 (pid 65038)] Task is starting.
2022-02-05 12:49:52.157 [11/start/1 (pid 65038)] Task finished successfully.
2022-02-05 12:49:52.165 [11/end/2 (pid 65046)] Task is starting.
2022-02-05 12:49:52.724 [11/end/2 (pid 65046)] first hello world
2022-02-05 12:49:52.799 [11/end/2 (pid 65046)] second hello world
2022-02-05 12:49:52.800 [11/end/2 (pid 65046)] Task finished successfully.
2022-02-05 12:49:52.801 Done!
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for metaflow_plugin_magicdir-0.0.3.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | c2dbb93c4cdfa5e490eff5661ce6c2d780b72523b1bb9bbd28e6c66ed5f1365d |
|
MD5 | 911b3b43f2a0d6be33160782872af632 |
|
BLAKE2b-256 | 2fc6cc31db4275519b7613a47cec8f694c1a2831d63fb53585da428c1ceec2aa |
Close
Hashes for metaflow_plugin_magicdir-0.0.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fd74dbdaa9cc741be832f580009059cb26ec4261e6ad9f0bae8dd6bb97e44ce7 |
|
MD5 | e9fda4d189d3bfa9b88f388617049a7c |
|
BLAKE2b-256 | 08eb5b68a69c4817e09ac8ea61ec66b0e866712d486ef3d1552afd0878afd07f |