cdm-flowparser

cdm-flowparser project is a Python 3 (Python 2 support is limited) extension module. The module provided functions can takes a file with a python code or a character buffer, parse it and provide back a hierarchical representation of the code in terms of fragments. Each fragment describes a portion of the input: a start point (line, column and absolute position) plus an end point (line, column and absolute position).

Comments are preserved too.

The module is used in the Codimension Python IDE to generate a flowchart-like diagrams for an arbitrary Python code as the user types it. Basically the IDE detects a pause in typing and regenerates the diagram.

Python 3 Installation and Building

The master branch contains code for Python 3 (3.5/3.6/3.7/3.8/3.9 grammar is covered).

The module can be installed using pip:

pip install cdmcfparser

You can also retrieve the full source code which in addition has some utilities. In order to do that you can follow these steps:

git clone https://github.com/SergeySatskiy/cdm-flowparser.git
cd cdm-flowparser
make
make check
make localinstall

Python 3: Visualizing Parsed Data

Suppose there is ~/my-file.py file with the following content:

#!/bin/env python
import sys

# I like comments
a = 154
for x in range(a):
    print("x = " + str(x))

sys.exit(0)

Then you can run a test utility (if you have a local repository clone):

~/cdm-flowparser/utils/run.py ~/my-file.py

The output will be the following:

Running control flow parser version: trunk
Module location: /home/swift/cdm-flowparser/cdmcfparser.cpython-35m-x86_64-linux-gnu.so
<ControlFlow [0:113] (1,1) (9,11)
 Body: [18:113] (2,1) (9,11)
 LeadingComment: None
 SideComment: None
 LeadingCMLComments: n/a
 SideCMLComments: n/a
 isOK: true
 Errors: n/a
 Warnings: n/a
 BangLine: [0:16] (1,1) (1,17)
 EncodingLine: None
 Docstring: None
 Suite: <Import [18:27] (2,1) (2,10)
         Body: [18:27] (2,1) (2,10)
         LeadingComment: None
         SideComment: None
         LeadingCMLComments: n/a
         SideCMLComments: n/a
         FromPart: None
         WhatPart: [25:27] (2,8) (2,10)
        >
        <CodeBlock [30:54] (4,1) (5,7)
         Body: [48:54] (5,1) (5,7)
         LeadingComment: <Comment [30:46] (4,1) (4,17)
                          Parts: <Fragment [30:46] (4,1) (4,17)
                                 >
                         >
         SideComment: None
         LeadingCMLComments: n/a
         SideCMLComments: n/a
        >
        <For [56:100] (6,1) (7,26)
         Body: [56:73] (6,1) (6,18)
         LeadingComment: None
         SideComment: None
         LeadingCMLComments: n/a
         SideCMLComments: n/a
         Async: None
         For: [56:58] (6,1) (6,3)
         Iteration: [60:72] (6,5) (6,17)
         Suite: <CodeBlock [79:100] (7,5) (7,26)
                 Body: [79:100] (7,5) (7,26)
                 LeadingComment: None
                 SideComment: None
                 LeadingCMLComments: n/a
                 SideCMLComments: n/a
                >
         ElsePart: None
        >
        <CodeBlock [103:113] (9,1) (9,11)
         Body: [103:113] (9,1) (9,11)
         LeadingComment: None
         SideComment: None
         LeadingCMLComments: n/a
         SideCMLComments: n/a
        >
>

Python 2 Installation and Building

Attention: Python 2 version is not supported anymore. There will be no more Python 2 releases.

The latest Python 2 release is 1.0.1. Both pre-built modules and source code are available in the releases area on github: latest Python 2 release 1.0.1.

To build a Python 2 module from sources please follow these steps:

cd
wget https://github.com/SergeySatskiy/cdm-flowparser/archive/v1.0.1.tar.gz
gunzip v1.0.1.tar.gz
tar -xf v1.0.1.tar
cd cdm-flowparser-1.0.1/
make
make localinstall
make check

Python 2: Visualizing Parsed Data

Suppose there is the following file ~/my-file.py with the following content:

#!/usr/bin/python
import sys

# I like comments
a = 154
for x in xrange( a ):
    print "x = " + str( x )

sys.exit( 0 )

Then you can run a test utility:

~/cdm-flowparser/utils/run.py ~/my-file.py

The output will be the following:

Running control flow parser version: trunk
<ControlFlow [0:119] (1,1) (9,13)
 isOK: true
 Errors: n/a
 Warnings: n/a
 BangLine: [0:16] (1,1) (1,17)
 EncodingLine: None
 Docstring: None
 Suite: <Import [18:27] (2,1) (2,10)
         Body: [18:27] (2,1) (2,10)
         LeadingComment: None
         SideComment: None
         LeadingCMLComments: n/a
         SideCMLComments: n/a
         FromPart: None
         WhatPart: [25:27] (2,8) (2,10)
        >
        <CodeBlock [30:54] (4,1) (5,7)
         Body: [48:54] (5,1) (5,7)
         LeadingComment: <Comment [30:46] (4,1) (4,17)
                          Parts: <Fragment [30:46] (4,1) (4,17)
                                 >
                         >
         SideComment: None
         LeadingCMLComments: n/a
         SideCMLComments: n/a
        >
        <For [56:104] (6,1) (7,27)
         Body: [56:76] (6,1) (6,21)
         LeadingComment: None
         SideComment: None
         LeadingCMLComments: n/a
         SideCMLComments: n/a
         Iteration: [60:75] (6,5) (6,20)
         Suite: <CodeBlock [82:104] (7,5) (7,27)
                 Body: [82:104] (7,5) (7,27)
                 LeadingComment: None
                 SideComment: None
                 LeadingCMLComments: n/a
                 SideCMLComments: n/a
                >
         ElsePart: None
        >
        <SysExit [107:119] (9,1) (9,13)
         Body: [107:119] (9,1) (9,13)
         LeadingComment: None
         SideComment: None
         LeadingCMLComments: n/a
         SideCMLComments: n/a
         Argument: [117:117] (9,11) (9,11)
        >
>

Under the Hood

Basically the ‘run.py’ utility has the following essential lines for the example above:

from cdmcfparser import getControlFlowFromFile
controlFlow = getControlFlowFromFile("my-file.py")

# Serializes nicely the controlFlow object
# See the run.py for the details of how it is done

The run.py is available in a local clone at ~/cdm-flowparser/utils/run.py or you can see the source code online

Essential Links

• Codimension Python IDE home page

• latest Python 2 release 1.0.1

• Python 3 Pypi package page

• Visualization Technology