Simple-Process-REPL 1.0.0

An extensible application framework with REPL for creating processes with side effects.

particle board interface (pbi) program.

This is the Readme for the application in this repository. for instructions on how to set up a chromebook to run this application under linux see the file: READMEchromebook.md

So, this is my take on making it possible to do a repeatable process using the particle.io cli, 'particle-cli on Arch Linux', to interface with a board, verify it's life, update it, claim/register it, flash it, test it, etc.

This is a generic script which should work with any particle board. It has an interactive mode for processing many boards in a loop, as well as the ability to execute different steps individually or in a group from the command line or interactively in a REPL.

All of these things could be almost doable in a chain of 'do this', 'do that'. But the boards take time in between events. The USB device comes up and down constantly, it's not reliable just because you know that's where the board was. I have read about other gnu/linuxs which change the device on occasion or always. Arch does not. Once I have it, I have it. However, it comes and goes...

So we have to wait, watch and listen.

But, as a whole, it's just a module of things we'd like to do. So we wrap those up to make life easier. and life is easier. At some point, making life is easier is just listing all the previous things that made live easier. And so it goes.

Dependencies

• Python >= 3.9: There is a dictionary merge in the code using 3.9 syntax.

• pythondialog -> pip install pythondialog will take care of that.

Starting out

├── pbi
│   ├── docs
│   ├── pbi
│   │   ├── core.py       --- Particle board/app interface layer.
│   │   ├── __init__.py
│   │   ├── logs.py      --- Setup logging.
│   │   ├── options.py   --- cli argument parsing.
│   │   ├── particle.py  --- Particle cli interface.
│   │   ├── repl_core.py --- Generic Application and function layer 
│   │   └── repl.py      --- The REPL and interpreter.
│   ├── pbi-defaults.yaml  --- the default configuration file.
│   ├── PBI.log          --- default logfile.
│   ├── pbi.py           --- application.
│   ├── pbi-readline.rc  --- An attempt to automate readline configuration.
│   ├── R5Bin.bin
│   ├── R5Test.bin
│   ├── setup.py
│   └── tinker-0.8.0-rc.27-boron.bin
├── READMEchromebook.md
└── README.md

Getting Help

Help with the command line can be obtained with pbi.py -h Additionally, Help with the symbols which are available for programming in the yaml files or in the REPL are obtained with pbi.py help

• pbi.py -h for cli help.
• pbi.py help for internal help.
• pbi.py particle-help for internal application layer help.

The easiest way to understand this is system is by using the REPL. It will show you how it works. pbi.py -r

Then type help, particle-help and/or showin.

Once in the REPL at the prompt; PBI:>, There are two help commands. help and particle-help. Help shows all the commands known with their documentation. particle-help, is a text in core.py, Intended to help with understanding how to use all of the things that are available to do.

Particle help is included with the Help also, But the help text is getting a bit long at this point, so scrolling up is necessary. The mechanism that it works by also requires it to be a command.

The REPL is simple, it has history, it has tab completion and the help is decent, assuming you know why you are here in the first place. Set the loglvl debug if you want to see more. Try some specials, def a msgbox., save a new config.

The first thing to do is a list and a get or just a get. From there the device id board type and usb device should be known. They will be used for other commands in the process that is created.

If you know particle commands then those should make sense, it's a small subset of the possibilities. Others are wrappers to make life easier, or to do some other task like wifi, or dialog message windows.

The REPL will do whatever you ask, so help, show, list, identify, update, set-setup-done, etc. Some which require a bit more, such as entering dfu or listen mode, are wrapped up together for convenience, but also available as commands themselves.

In the REPL, show-all will show you everything there is to know about the state of things in yaml format. The REPL is meant to be as introspective as it is programmable.

Get

It's the first command you'll want to do when you plug in.

This is the command we use to populate our usb device, board type and device-id. Various particle commands need the id, and we need the usb device so we know who to wait for. It uses particle serial list for it's data.

A map of all things that matter

is called AS - Application State.

It is a merge of the Repl's Application state and the dictionary given by the core layer.

The command: show-all or showin in the REPL will give it to you in yaml. help will give you the documentation for every command you can do, even the ones you just created. Inside repl_core.py, AS is the name of the Application State structure. AS['config'] is the name of the loaded configuration. AS['device'] is the dictionary of device information. The easiest way to access it is showin device or showin config serial with showin key1 key2,... is the command to find sub-section or attributes in the REPL. showin config , showin defaults, or just showin which is the same as show-all. showin config files or showin config files logfile.

So, If you need something, a function we don't have, Add an actual function to core.py, and put an entry in one of the symbol tables.

If it can be created from a combination of pieces then do it with a new symbol. It could be def'd in the REPL, once it worked correctly. Then save config, that will sync the symbol tables between the interpreter and the config, then the config will save with whatever you've got. If you 'know it', code the new functionality directly in the yaml.

The save-config command automatically syncs your functions from the REPL before the save.

You can sync the symbols to your config at anytime with the sync-funcs command. However, you still must save the configuration you have in memory if you want to keep them.

Be warned, that def's in the REPL are ephemeral unless saved. make it up, throw it away, or save it. -- this is not a step. --

lights.

Something very important for knowing the state of your Particle.io boron.

The meaning of the lights on a Particle.io Boron.

Data and symbol driven

This program is actually a very simple interpreter with an interactive REPL. Everything you want to do must be a python function which is registered in the interpreter's symbol table. From there, everything is composable from symbol/words from the interpreter's symbol table, ie, your symbols. Those composed symbols can also be added to the interpreter's symbol table to create increasingly complex sets of processes, which are executed in order. These user functions can also be defined in the YAML config file. which defaults to 'pbi-defaults.yaml'

A poor mans lisp

At the lowest level, in core.py, the symbols/commands are directly connected to python functions. But symbols/commands can also be lists of known symbols instead of a function. This allows for the creation of sub-groups which can be referenced by other symbols. There are no parentheses, only the ability to associate lists of symbols with a new symbol.

import repl as r
symbols = [
    ['wifi',       connect_wifi,    'Connect to wifi using nmtui if not connected.']
    ['list',       P.list_usb,      'List the particle boards connected to USB.']
    ['start',      'wifi list',     'Connect wifi and list the boards.']
    ['domore',     'start identify', 'Start then identify']
    ['doevenmore', 'domore setup',   'Start identify and setup.']
]
r.init_symbol_table(symbols)

The symbols domore and doevenmore can be defined in the YAML configuration file, it is not necessary to modify python code unless new functionality needs to be introduced.

Special Symbols

The interpreter is not very bright and has no way of grouping things together which makes it difficult to execute commands which take arguments. Specials are symbols at the beginning of a command which will eat the rest of the line, in attempt to do what they are supposed to do.

The interpreter has the concept of special symbols, These are symbols which take arguments and will consume the entire REPL command. The REPL itself has one function in this symbol table, def which allows for the creation of a new symbol with the following syntax.

def <symbol> 'helpstr' symbol1 symbol2 symbol3...

Special symbols that can be useful, but must be implemented outside of the REPL are commands to set options, and save or load a configuration file. These are implemented in core.py along with all the other symbols.

Special symbols have a number of args which can be set. If positive the command will be checked for compliance. Here is an example which creates symbols for saving and loading configurations from a given filename. These obviously must be the responsibility of the application, ie. core.py.

specials = [
    "Commands we want in the repl which can take arguments."
    ['save-config', save-config, 1,
    "Save the configuration; save-config 'filename'"]

    ['load-config', load-config, 1,
    "Load a configuration; save-config 'filename'"]
]

Currently dialog windows are wrapped in individual python functions, but they could be a special that takes an argument. The eval command in the reple will do the right thing if a special is part of your process.

Weird fact: You can put a completed specials command in the regular symbols table and it will do the right thing. It's just going to be static with it's parameters.

Functionality

Aside from basic particle board commands, there are many symbols builtin which do special things. There is wait which just waits for the usb device to come online with a timeout. There is pause which just sleeps for a few seconds, as set in the configuration. The wifi function checks the wifi with linux's network manager, and uses nmcli to create a connection if one does not exist. Functionality is easy to add with a new function and an entry in the symbol table.

Configuration

PBI uses YAML for it's configuration files. Everything is specified there, there is very little in common with the cli. If no config file is given, the default will be loaded. The primary purpose of the cli is to designate the fashion you would like for pbi to run. The default configuration file is pbi_defaults.yaml.

4 modes of running

• Run in a loop for doing a process over and over,
• Run the default process once,
• Run a list of command/symbols from the command line,
• As an interactive REPL

Running in the REPL allows for the preservation of state as well as introspection and the interactive manipulation of a board. It is possible to create new symbols/functions as well as saving and loading of configurations and functions.

The REPL

The REPL is very convenient as it saves state, and can be used to interactively create/execute a process step by step. Symbols have a documentation string associated with them. It is possible to get a list of symbols and their help by typing help at the REPL prompt. The doc strings for the functions, and the source code for compound functions are also included.

Everything is driven by the two symbol tables and the yaml config file. Additional functionality can be added by adding to the functions to symbol table in core.py. User functions, ie, lists of known symbols, can be defined in the REPL or in the configuration file. With the limitation that those functions are ultimately composed of known symbols as defined in core.py If symbols are defined within the REPL, they should be saved or they will be lost upon exit.

The default process

In the configuration there is an autoexec attribute. This should be a symbol name or list of symbol names to run as the default process. This is the process that will run when running cli in interactive loop mode, or when run once.

If symbols are given on the cli after the option then that list is executed once automatically instead of the symbol in autoexec.

Symbols/Commands/functions

We've got three kinds.

• Symbols which point at directly at parameter-less functions
• Symbols which are lists of symbols, compound commands.
• Symbols which are special because they take parameters.

symbol/functions.

These commands are just python functions, whatever it is they do.

Compound commands

Compound commands are commands defined outside of python code. They are strings which can be parsed and evaluated by the REPL/interpreter. The core and particle functions are very specific, so creating more complex process is a matter of creating compound commands. The setup command is an example of this.

The setup command is actually composed of 3 commands. update wait set-setup-done Compound commands can be built from other compound commands and even special commands. Compound commands can be defined in yaml, in python code, or interactively in the REPL.

It is also possible to define a command that executes a special command.

Specials

These are also pointers to python functions, but which take some arguments. These go on a line by themselves since we have no way of knowing them unless the line starts with them, and then the special gobbles up the rest of the line.

It is possible to make compound commands of specials which can then be used in other compound commands. The specials are commands like def, save-config, load-config, msgbox, msgcli, loglvl, log-info, showin, etc. Many of them have no real use outside of the development environment.

The Particle.io layer

There is very little here, in the particle.py module. These are all of the basic commands I've used so far, as long as they are as close to bare particle-cli commands. I combined some things, like flash always does dfu first, identify always does a listen.

The more complex functions are in core.py, These are functions which save values, and interact with the configuration. The rest of the functions can actually live in the configuration file. It is only necessary to modify python code if there is a desire for more base functionality.

Handshake function

This is a generic function in repl-core.py. It manages the interaction with the test procedure. Everything handshake does is defined in the configuration. If anything fails, or doesn't match, an exception is raised and the device is considered failed.

Here are the steps that handshake() does.

• Wait for the start_string, match it.
• Respond with the response_string.
• Look in the output for:
  • fail_regex,
  • done_regex,
  • do_qqc_regex.
• If fail, raise an exception.
• if done, exit quietly with true.
• if do_qqc, then call the do_qqc_function and send the return value to the serial device. qqc = quelque chose = something. It's a common .fr abbrev. :-)

In our case, the do_qqc_function is input-serial, which prompts for a serial number, validates it, and returns it. This function must be listed in the symbol table as that is where handshake() will look for it. Makes it easy to test. serial-input at the pbi:> prompt.

Current state

PBI is working very well. It is a pleasure to work in the REPL to create a new process.

get setup wait pause testit or get testit work well, notice it skips register which has been a problem for me.

There are two commands that could be suitable full processes, one with dialogs everywhere and one without. They can be changed however we like.

The help seems to be ever growing, and now has an application layer of help which can be set. The command particle-help is set in the Application/particle layer, and it is included in the output of the regular help command at the top.

flash-test, flash-image, flash-tinker, are working, but through os.popen().read() instead of subprocess like everything else. I don't have an explanation, subrocess needs more configuration for these commands. I've tried shell=True with no change. So it's going be down in the details somewhere.

If a command fails at any point in a process, the entire process stops and the board is considered a failure.

Invoking.

Two different kinds of help are built in. * pbi.py -h * pbi.py help

This is one way to run, get and setup. It works from the command line. pbi.py get setup

It can be run as a REPL. pbi.py -r

It will run the autoexec once, automatically. pbi.py

It will run the autoexec In a loop, with start and again dialogs. pbi.py -i

Logging: In good shape.

REPL: It's getting to be a pleasure.

• Builtin help
• show, showin, and show-all are quite handy.
• Working, persistent, history and tab completion.
• Symbols, Specials, and compound symbols are working as designed.
• Seems to be handling exceptions and displaying good errors.
• Defining new symbols works. -> specials work.
• The loglvl command can can change the logging level interactively.
• Defining a symbol of a special works. - Super cool.
  • msgbox "Hello World"
  • def foo mymsg "my special msg" msgbox "Hello World"
  • That means it works in yaml too.
• log-info and log-debug allow sending of arbitrary messages to the log.
• sh for running shell commands.

Repl Core: Everything generic in functionality

This is where all of the basic functionality like dialogs, command prompts, saving and loading of configurations and in general looking around and manipulating data etc. handshaking, waiting, pausing etc.

There are a number of key features that are working.

• particle commands - all of them from the doc, and then some.
• dialogs - There is a pythondialog interface, and cli dialog interface.
• serial waiting, reading, and sending,
• wifi, - Uses network manager (nmcli) for linux. Non-functional on other platforms.
• Waiting and handshaking are working.
  • wait looks for the actual usb device with a timeout.
  • pause sleeps for pause_time.
  • get now works with a timeout loop. This is pretty much the first step to any process, It's nice to have the chance to re-plug if it starts without you.
  • handshake does a blocking serial.read/readline for both the initial string, and the test results after. See above for more details.

Note: wait for device is literally a poll to see if the device file exists. Once it appears there is some time before the udev rules make the file accessible by non-root users. A pause helps everything go smoothly. The next command will actually have access to the device. So now I have a habit of following a wait with a pause.

I had thought that perhaps using the particle.get_w_wait function to wait for the device could work nicely, but it does not. So what we have are the best of what I've thought of.

Core.py - The device interface layer.

Everything here has to do with the particle board cli. This is where additional application functionality is defined. A handful of functions and their entries into a symbol table is about all of it.