calmjs 3.4.4

A Python framework for building toolchains and utilities for working with the Node.js ecosystem from within a Python environment.

calmjs

A Python framework for building toolchains and utilities for working with the Node.js ecosystem from within a Python environment.

Introduction

Calmjs defines an extensible framework for interoperability between Python and Node.js runtime for Python packages, to provide their developers a well defined interface for bi-directional access between Node.js/Javascript development tools and the JavaScript code within their Python packages, such that a proper, formal integration with Node.js/JavaScript environment from a Python environment can be facilitated. The goal of the Calmjs framework is to aid the development, testing, and deployment of Python packages that also include and/or integrate with external JavaScript code.

Methodology

First, this is achieved by providing Python packages the ability to declare dependencies on Node.js/JavaScript packages or source files that are required to complete their functionality. This common framework will ensure the accessibility of these metadata under a common protocol, to avoid incompatible declarations that are not portable between different projects and environments, or being otherwise scattered across different tools or locations or be duplicated within the same working environments by different sets of tools. Without a common framework, the result is that Python packages will be unable to properly communicate their non-Python requirements and states between each other, resulting in difficulties in building, development and deployment of the software stack, and/or becoming a source of errors for those processes.

Second, by offering a set of tools built on top of this extensible framework to work with these declarations for generating the configuration files for required Node.js tools, so that they can construct the required the build and/or runtime environment for their functionality.

Ultimately, this permits better Node.js integration with a given Python environment, lowering the amount of effort needed to achieve continuous integration and/or delivery of Python packages in conjunction with Node.js/JavaScript packages in a reproducible manner.

Implementation

In order to achieve this, the Calmjs framework provides a set of extension to setuptools that assists with the tracking and management of dependencies of JavaScript or Node.js based packages (such as ones through npm) for a given Python package. It also provides a number of base classes that can be used to build custom toolchains that implement different strategies for managing and compiling required JavaScript code and related assets into the deployment artifacts that an application server may use, or to generate test harnesses to ensure correctness under both the development and production environment. These extra functionalities will be provided by other Python packages under the calmjs namespace in order to realize this modular architecture.

The name Calmjs was originally derived from the steps in the first iteration of the toolchain which involves the steps compile, assemble, and linkage into a module of JavaScript using the namespace from the host Python package. The m in the logo is the ear of a rabbit. The reason this animal is chosen as the mascot for this project is because of their dietary habits, as it’s analogous to how JavaScript code is typically turned into a minimally usable level by other tools and framework.

Features overview

A framework for integration with Node.js based package managers

Through setuptools command hooks, calmjs provides Python packages with the ability to declare and manage manifest definition files for Node.js based package management systems (e.g. such as package.json for npm). Under typical usage, this means the declaration of dependencies or devDependencies for the JavaScript packages needed by a given Python package can be tracked, all within the setuptools framework through the extensions provided by calmjs.

The other part of this infrastructure is that these declarations follow the Python package dependency graph. Developers and users can make use of the calmjs console command entry point, or through setuptools, to generate a manifest file to facilitate the installation of Node.js packages required by the Python packages within the completed application stack, tailored for all the packages at hand.

calmjs includes the integration support for both npm and yarn by default.

Export JavaScript code out of Python packages with the same namespace

A given Python package that included associated JavaScript source code within the same Python module and namespace structure alongside Python source code within the source tree, will be able to declare those namespaces as the root for those JavaScript modules under the exact same Python package namespace through setuptools entry points.

These declarations will be available through registries exposed by the calmjs module registry system for other packages to turn those declarations through the API provided by the framework into working JavaScript code following the same declared module and namespace structures. The default module registry will make use of the / character (instead of the . character like in Python) as the separator for the names due to established naming conventions in JavaScript (and in ES6 towards the future).

Other tools that works with the Calmjs framework can then make use of these raw JavaScript source files in conjunction with the local Node.js environment, or generate artifacts for deployment over the web. This leads to…

Better integration of JavaScript toolchains into Python environments

This is achieved by providing a framework for building toolchains for working with tools written in JavaScript for Node.js/JavaScript environments that integrates properly with existing Python packages and environments.

There are no limitations as to how or what can be done with the tools or the source files, as this is left as an implementation detail. For an example please refer to the calmjs.rjs Python package, which allows the production of AMD (Asynchronous Module Definition) artifacts from JavaScript packages embedded inside Python packages, or calmjs.webpack which integrates with webpack for the production of another commonly used bundled artifact format.

Generally, toolchains can be built to find and load all Python packages (through the calmjs registry system) that have any JavaScript source files, and those will be extracted, go through the appropriate transpilers (if any) in order to build deployable artifacts. Test harnesses can be set up to aid with running of unit tests, functional testing and naturally the final integration tests needed for a successful deployment.

Well-defined modular architecture to ensure code reuse and extensibility

The features described so far are built upon a foundation of generic classes and modules, so that the support for additional JavaScript tools or custom process for handling transpilation can be as simple as creating a new module for a couple of classes with additional parameters with the relevant setuptools entry points.

In fact, calmjs out of the box only ships with just the core framework plus the npm/yarn interfacing part, with the support for tools like Bower or r.js as completely separate packages (as calmjs.bower and calmjs.rjs respectively), such that projects, environments or sites that do not need the functionality those packages provide can simply opt to not have them installed.

Installation

As the goal of calmjs is to integrate Node.js and npm (or yarn) into a Python environment, they need to be available within the environment; if they are not installed please follow the installation steps for Node.js appropriate for the target operating system/environment/platform.

To install calmjs into a given Python environment, the following command can be executed to install directly from PyPI:

$ pip install calmjs

Alternative installation methods (for developers, advanced users)

Development is still ongoing with calmjs, for the latest features and bug fixes, the development version can be installed through git like so:

$ # standard installation mode
$ pip install git+https://github.com/calmjs/calmjs.git#egg=calmjs
$ # for an editable installation mode; note the upgrade flag
$ pip install -U -e git+https://github.com/calmjs/calmjs.git#egg=calmjs

Note that the -U flag for the editable installation is to ensure that setuptools be upgraded to the latest version to avoid issues dealing with namespaces for development packages, which is documented in the next paragraph.

Alternatively, the git repository can be cloned directly and execute python setup.py develop while inside the root of the source directory, however if this development installation method is done using any version of setuptools earlier than v31, there will be inconsistent errors with importing of modules under the calmjs namespace. Various symptoms of namespace import failures are documented under the troubleshooting section of this document.

Testing the installation

To ensure that the calmjs installation is functioning correctly, the built-in testsuite can be executed by the following:

$ python -m unittest calmjs.tests.make_suite

If there are failures, please file an issue on the issue tracker with the full traceback, and/or the method of installation. Please also remember to include platform specific information, such as Python version, operating system environments and version, and other related information related to the issue at hand.

Usage and description of key features

When installed to a particular Python environment, the calmjs command-line utility will become available within there.

$ calmjs
usage: calmjs [-h] [-d] [-q] [-v] [-V] <command> ...

positional arguments:
  <command>
    artifact     helpers for the management of artifacts
    npm          npm support for the calmjs framework
    yarn         yarn support for the calmjs framework

optional arguments:
  -h, --help     show this help message and exit

global options:
  -d, --debug    show traceback on error; twice for post_mortem
                 '--debugger' when execution cannot continue
  -q, --quiet    be more quiet
  -v, --verbose  be more verbose
  -V, --version  print version information

As mentioned, npm support is built-in so it is always available; to access its help, simply execute calmjs npm -h, which will then list the options available for that particular subcommand. If other subcommands are available (which will be provided by other calmjs integration packages) they will be listed as a <command> and their specific help messages will be accessible in the same manner.

Declare a package.json for a given Python package

If a package wish to declare dependencies on packages hosted by npm, it may do something like this in its setup.py:

from setuptools import setup

package_json = {
    "dependencies": {
        "jquery": "~3.0.0",
        "underscore": "~1.8.0",
    }
}

setup(
    name='example.package',
    # ...
    setup_requires=[
        'calmjs',
        # plus other setup_requires ...
    ],
    package_json=package_json,
    # ...
)

Note that setup_requires section must specify calmjs in order to enable the package_json setup keyword for the generation of the package.json metadata file for the given package whenever python setup.py egg_info is executed (directly or indirectly), so that even if calmjs is not already installed into the current Python environment, it will be acquired from PyPI and be included as part of the setuptools setup process, and without being a direct dependency of the given package. The package.json will be generated if the provided data is either a valid JSON string or a dictionary without incompatible data types. For example:

$ python setup.py egg_info
running egg_info
writing package_json to example.package.egg-info/package.json
...
$ cat example.package.egg-info/package.json
{
    "dependencies": {
        "jquery": "~3.0.0",
        "underscore": "~1.8.0"
    }
}

The key reason for using setup_requires is to not force a given package’s dependents to have calmjs as part of their dependencies, as typically this is a requirement only for developers but not for end-users. This also mean that for developers that want to use calmjs and utilities they must install that separately (i.e. pip install calmjs), or declare calmjs as a development dependency through the usage of extras_require flag, for example:

setup(
    name='example.package',
    # ...
    setup_requires=[
        'calmjs',
        # ...
    ],
    extras_require={
        'dev': [
            'calmjs',
            # ... plus other development dependencies
        ],
    },
    # ...
)

Then to fully install the package as an editable package with the dependencies listed under the dev extras:

$ pip install -e .[dev]
Obtaining file://example/package
...
Installing collected packages: ..., calmjs, ...
  Running setup.py develop for example.package
Successfully installed ...

Note that now the calmjs package remains installed in the Python environment, and the utilities they provide may now be used, covered by the following sections.

Using the package.json across Python package dependencies

With the package.json file written to the package metadata directory, utilities such as calmjs make make use of it. One method to do this is through that package’s setup.py. By invoking setup.py npm --init from there, a new package.json, combined with all the dependencies and devDependencies declared by the Python package dependencies of the given package, will be written to the current directory. This is akin to running npm init, with the difference being that the dependencies are also being resolved through the Python package dependency tree for the given Python package.

Do note that this requires the package.json creation and handling capability be available for the given package (refer to previous section on how to achieve this) and all dependencies must be correctly installed and be importable from the current Python environment.

Alternatively, the calmjs npm utility may be used. Invoking calmjs npm --init example.package from the command line will achieve the same thing, anywhere on the file system, provided that both calmjs and example.package are installed and available through the current Python environment’s import system. For more details and information about this utility, please refer to the integration with npm through calmjs npm section.

Dealing with npm dependencies with Python package dependencies

Flat is better than nested. So all dependencies (and devDependencies) declared by any upstream Python package will be automatically inherited by all its downstream packages, but they have the option to override it with whatever they want through the mechanism as described above. They can set a JavaScript or Node.js package to whatever versions desired, or even simply remove that dependency completely by setting the version to None.

Whenever an actual package.json is needed by calmjs, the calmjs npm utility flattens all Node.js dependencies needed by the Python packages into a single file, which is then passed into the respective JavaScript package manager for consumption. This process is also done when a calmjs toolchain or utility make use of these declared information to to generate the desired artifacts to achieve whatever desired task at hand.

Of course, if the nested style of packages and dependency in the same style as npm is desired, no one is forced to use this, they are free to use whatever tools to interpret or make use of whatever data files and dependencies available, and/or to split their packages up to Python and JavaScript bits and have them be deployed and hosted on both PyPI (for pip) and npm respectively and then figure out how to bring them back together in a coherent manner. Don’t ask (or debate with) the author on how the latter option is better or easier for everyone (developers, system integrators and end-users) involved.

Declare explicit dependencies on paths inside node_modules

Given that the dependencies on specific versions of packages sourced from npm is explicitly specified, build tools will benefit again from explicit declarations on files needed from those packages. Namely, the compiled packages could be declared in the extras_calmjs section in JSON string much like package_json, like so:

extras_calmjs = {
    'node_modules': {
        'jquery': 'jquery/dist/jquery.js',
        'underscore': 'underscore/underscore.js',
    },
}

setup(
    name='example.package',
    ...
    extras_calmjs=extras_calmjs,
    ...
)

Since node_modules is declared to be an extras_key, conflicting declarations between packages within the environment will be resolved and merged in the same manner as dependencies conflicts declared in package_json.

Please do note that complete path names must be declared (note that the .js filename suffix is included in the example); directories can also be declared. However, as these declarations are done from within Python, explicit, full paths are required thus it is up to downstream integration packages to properly handle and/or convert this into the conventions that standard Node.js tools might expect (i.e. where the .js filename suffix is omitted).

Export JavaScript code from Python packages

Furthering the previous example, if the files and directories inside example.package are laid out like so:

.
├── example
│   ├── __init__.py
│   └── package
│       ├── __init__.py
│       ├── content.py
│       ├── form.py
│       ├── ui.js
│       ├── ui.py
│       └── widget.js
└── setup.py

To declare the JavaScript source files within ./example/package as JavaScript modules through calmjs, an entry point can be declared like so in the setup.py file:

setup(
    ...
    entry_points="""
    ...
    [calmjs.module]
    example.package = example.package
    """,
    ...
)

Note that the name of the entry point is not relevant; that entry point name is ignored, as the intention of the default module registry is to provide a module name that maps directly to the same import namespace as the source Python module, but with the ES5 namespace separator /, instead of the . character as in Python. If an explicit mapping is required, a new module registry class may be defined that uses the provided name as the CommonJS import name from the JavaScript code.

The default method will expose the two source files with the following names:

- 'example/package/ui'
- 'example/package/widget'

For some projects, it may be undesirable to permit this automated method to extract all the available JavaScript source files from within the given Python module.

To get around this, it is possible to declare new module registries through the Calmjs framework. Provided that the ModuleRegistry subclass was set up correctly to generate the desired modules from a given package, simply declare this as a calmjs.registry entry point like so:

setup(
    ...
    entry_points="""
    ...
    [calmjs.registry]
    example.module = example.package.registry:ExampleModuleRegistry
    """,
    ...
)

Do note that while the names permitted for an entry point name is quite unrestricted, these registry names should be of a standard dotted namespace format to ensure maximum tool compatibility, as these can be specified from the command line through tools that utilizes this system.

Once the registry was declared, simply replace calmjs.module with the name of that, along with a calmjs_module_registry attribute that declare this example.module registry is the default registry to use with this package.

setup(
    ...
    calmjs_module_registry=['example.package'],
    entry_points="""
    ...
    [example.module]
    example.package = example.package
    """,
    ...
)

Within the Calmjs framework, tools can be explicitly specified to capture modules from any or all module registries registered to the framework. One other registry was also defined. If the entry point was declared like so:

setup(
    ...
    entry_points="""
    ...
    [calmjs.py.module]
    example.package = example.package
    """,
    ...
)

The separator for the namespace and the module will use the . character instead of /. However given that the . character is a valid name for a JavaScript module, the usage of this may create issues with certain JavaScript tools. While AMD based module systems can generally handle . characters in imports without issues, allowing somewhat more Python-like feel importing using dotted names within the JavaScript environment, however, this may lead to incompatibilities with other JavaScript libraries thus the usage of this naming scheme is not recommended.

By default, another registry with the .tests suffix is also declared as a compliment to the previously introduced registries, which packages can make use of to declare JavaScript test code that accompanies the respective modules that have been declared. For example:

setup(
    ...
    entry_points="""
    ...
    [calmjs.module]
    example.package = example.package

    [calmjs.module.tests]
    example.package.tests = example.package.tests
    """,
    ...
)

Much like the first example, this declares example.package as a Python namespace module that exports JavaScript code, with the subsequent declaration section being the module that contains the tests that accompanies the first.

Make available the accompanied resource files to loaders

Certain Node.js build tools and frameworks support the concept of “loaders” for other resource files through the same import/require system for JavaScript modules. To provide the most basic support to ease the effort required by package creators to expose other resource files to the Node.js tooling import systems, a subsidiary loader registry may be declared and used in conjunction with a parent module registry. To extend on the previous example, if following entry points are defined:

[calmjs.module]
example.package = example.package

[calmjs.module.loader]
json = json[json]
text = text[txt,json]

The calmjs.module.loader registry will reference its parent registry calmjs.module for the specific modules that have been exposed, so that it will source all the relevant filenames with the declared file name extensions to a specific loader by name. Unlike the base module registry, the module loader registry will ignore the Python module name section, while the name (on the left-hand side) is the desired loader, with the extras (comma-separated tokens enclosed between the []) are the file name extensions to be acquired from the package. Thus with the previously defined entries for the example.package, the following require statements should resolve if the target resource files exist for the package:

// resolved through "json = ...[json]"
var manifest_obj = require('json!example/package/manifest.json');
// resolved through "text = ...[txt,json]"
var manifest_txt = require('text!example/package/manifest.json');
var index_txt = require('text!example/package/index.txt');

As the values generated by the registry follow the standard toolchain spec compile entry grammar, this should satisfy the most basic use cases and can be included directly as a calmjs_module_registry for the package. However, the actual usage in the provided JavaScript code conjunction with the actual toolchain packages that integrates/interacts with with their respective Node.js packages may have interactions that will require special handling (such as inclusion/exclusion of the generated segments targeted for the final artifact, or how those sources are aliased or made available to the system, or whether or not the registry itself requires manual integration); please consult the documentation for the specific integration package with regards to this specific registry type.

One final note: due to the limitations of the Python entry point system, file name extensions are assumed to be all lower-case.

Integration with npm through calmjs npm

As mentioned, it is possible to make use of the package.json generation capabilities from outside of setuptools. Users can easily do the same through the built-in calmjs npm tool:

$ calmjs npm --help
usage: calmjs npm [-h] [-d] [-q] [-v] [-V] [--view] [--init] [--install]
                  [-i] [-m] [-w] [-E] [-P] [-D]
                  <package> [<package> ...]

npm support for the calmjs framework

positional arguments:
  <package>          python packages to be used for the generation of
                     'package.json'

optional arguments:
  -D, --development  explicitly specify development mode for npm install
  -E, --explicit     explicit mode disables resolution for dependencies;
                     only the specified Python package(s) will be used.
  -h, --help         show this help message and exit
  -i, --interactive  enable interactive prompt; if an action requires an
                     explicit response but none were specified through
                     flags (i.e. overwrite), prompt for response;
                     disabled by default
  -m, --merge        merge generated 'package.json' with the one in
                     current directory; if interactive mode is not
                     enabled, implies overwrite, else the difference will
                     be displayed
  -P, --production   explicitly specify production mode for npm install
  -w, --overwrite    automatically overwrite any file changes to current
                     directory without prompting

Naturally, the same --init functionality shown above with the setuptools framework is available, however package names can be supplied for generating the target package.json file from anywhere on the filesystem, provided that the Python environment has all the required packages installed. For instance, if the Node.js packages for example.package is to be installed, this can be invoked to view the package.json that would be generated:

$ calmjs -v npm --view example.package
2016-09-01 16:37:18,398 INFO calmjs.cli generating a flattened
'package.json' for 'example.package'
{
    "dependencies": {
        "jquery": "~3.0.0",
        "underscore": "~1.8.0",
    },
    "devDependencies": {},
    "name": "example.package"
}

If there is an existing package.json file already in the current directory, using the -i flag with --init or --install will show what differences there may be between the generated version and the existing version, and prompt for an action.

Toolchain

Documentation on how to extend the Toolchain class to support use cases is currently incomplete. This is usually combined together with a calmjs.runtime.DriverRuntime to hook into the calmjs runtime.

Unfortunately at this time a detailed guide on how to create a complete implementation is not completed (only documentation within the class are, however). For a working example on how this may be achieved please refer to the implementations provided by calmjs.rjs or calmjs.webpack.

Toolchain Advice

For package developers that need to provide additional instructions to a toolchain execution (e.g. for compatibility between RequireJS and also webpack for specific use case of features to a given package), the toolchain system will also make use of the advice system such that additional instructions may be created and registered for use and reuse by their dependents. Much like the Toolchain, this feature is currently lacking in documentation outside of the test cases.

Pre-defined artifact generation through setuptools

It is possible to define the artifacts to be generated for a given package and the rule to do so. Simply define a function that return an instance of a calmjs.toolchain.Toolchain subclass that have integrated with the desired tool, and a calmjs.toolchain.Spec object with the rules needed. These specific functions are often provided by the package that offers them, please refer to the toolchain packages listed and linked in the previous section for further details on how these might be used.

As these are also implemented through the registry system, the entry points generally look like this:

setup(
    ...
    build_calmjs_artifacts=True,
    entry_points="""
    ...
    [calmjs.artifacts]
    complete.bundle.js = example.toolchain:builder
    """,
    ...
)

In the example, the builder function from the module example.toolchain is used to generate the complete.bundle.js file. The generated artifact files will reside in the calmjs_artifacts directory within the package metadata directory (one that ends with either .dist-info or .egg-info) for that package. An accompanied calmjs_artifacts.json file will also be generated, listing the versions of the various Python packages that were involved with construction of that artifact, and the version of binary that was used for the task.

When the build_calmjs_artifacts is set to True, the hook for automatic generation of these artifacts through the setup.py build step will enabled. This is useful for automatically bundling the artifact file with a release such as Python wheels (e.g. running setup.py bdist_wheel will also build the declared artifacts. Otherwise, this step can be manually invoked using setup.py build_calmjs_artifacts or through the calmjs artifact build tool.

Troubleshooting

The following may be some issues that may be encountered with typical usage of calmjs.

CRITICAL calmjs.runtime terminating due to a critical error

If calmjs encounters any unexpected situation, it may abort like so:

$ calmjs npm --install calmjs.dev
CRITICAL calmjs.runtime terminating due to a critical error

If no useful ERROR message is listed before, please try running again using a debug flag (either -d or --debug).

$ calmjs -d npm --install calmjs.dev
CRITICAL calmjs.runtime terminating due to exception
Traceback (most recent call last):
...

Specifying the debug flag twice will enable the post_mortem mode, where a debugger will be fired at the point of failure. Authors of packages that implement runtime classes that provide subcommands to the calmjs command may find this useful during their development cycles. Do note that the default debugger is only triggered if the top level runtime class enable the usage (which the runtime class that implement the calmjs command does) and if the failure occur during the invocation of the runtime class. Any other errors or exceptions that occur during the setup stage of the calmjs runtime will simply be logged at a lower priority level (e.g. to make warnings generated during the setup stage visible, additional verbose flags must be provided).

ERROR bad ‘calmjs.runtime’ entry point

ImportError

This is typically caused by improper removal of locally installed packages that had an entry point registered, an addon package to calmjs registered entry points pointing to bad import locations, or conflicting installation methods was used for the current environment as outlined in the installation section of this document. Either reinstall the broken package again with the correct installation method for the environment, or fully uninstall or remove files belonging to the packages or sources that are triggering the undesirable error messages.

bad entry point

This is caused by packages defining malformed entry point. The name of the package triggering this error will be noted in the log; the error may be reported to its developer.

Random ImportError when trying to import from the calmjs namespace

As calmjs is declared as both namespace and package, there are certain low-level setup that is required on the working Python environment to ensure that all modules within can be located correctly. However, versions of setuptools earlier than v31.0.0 does not create the required package namespace declarations when a package is installed using a development installation method (e.g. using python setup.py develop) into the Python environment in conjunction with another package that was installed through pip within the same namespace. Failures can manifest as inconsistent import failures for any modules under the calmjs namespace. As an example:

>>> from calmjs import tests
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name tests
>>> from calmjs import parse  # calmjs.parse was installed via pip
>>> from calmjs import tests
>>> # no failure, but it was failing just earlier?

It could also manifest differently, such as an AttributeError, which may be triggered through the execution of unittests for calmjs:

$ coverage run --include=src/* -m unittest calmjs.tests.make_suite
Traceback (most recent call last):
  ...
    parent, obj = obj, getattr(obj, part)
AttributeError: 'module' object has no attribute 'tests'
$ python -m calmjs.tests.make_suite
/usr/bin/python: No module named 'calmjs.tests'

To resolve this issue, ensure that setuptools is upgraded to v31 or greater, which may be installed/upgraded through pip like so:

$ pip install --upgrade setuptools

Then reinstall all the required packages that are under the calmjs namespace to resolve this import issue.

Environmental variables being ignored/not passed to underlying tools

Generally speaking, the Calmjs framework filters out all environmental variables except for the bare minimum by default, and only passes a limited number to the underlying tool. These are the PATH and the NODE_PATH variables, plus platform specific variables to enable execution of scripts and binaries.

Runtime reporting ‘unrecognized arguments:’ on declared arguments

This issue should be fully resolved for calmjs>=3.1.0.

The default behavior in the ArgumentParser defaults to uselessly blaming the root parser for any unrecognized arguments caused by its subparsers. The original workaround prior to calmjs-3.1.0 had the failure as documented below as its subparser resolver implementation was incomplete. Either of these misleading behaviors impede the end users from being able to quickly locate the misplaced argument flags.

For instance, if the calmjs command was executed like so resulting in error message may look like this:

$ calmjs subcmd1 subcmd2 --flag item
usage: calmjs subcmd1 ... [--flag FLAG]
calmjs subcmd1: error: unrecognized arguments: --flag

This means that --flag is unrecognized by the second subcommand (i.e. the calmjs subcmd1 subcmd2 command) as that was placed after subcmd2, but the subparser for subcmd1 flagged that as an error. Unfortunately there are a number of issues in the argparse module that makes it difficult to fully resolve this problem, so for the mean time please ensure the flag is provided at the correct subcommand level (i.e. in this case, calmjs subcmd1 --flag item subcmd2), otherwise consult the help at the correct level by appending -h to each of the valid subcommands.

Module registry not locating files from namespace packages

There are a number of edge cases associated with namespace packages in Python, especially if they are provided on the system through different methods (i.e. mix of zipped eggs, wheels and development packages). While workarounds for handling of namespace modules for the given packages are provided, there are limitations in place. One such cause is due to complexity in dealing with zipped eggs; if this is an issue, please ensure that the affected package has zip_safe declared as false, or alternatively generate a Python wheel then install that wheel, if the target Python environment has that as the standard installation format.

UserWarning: Unknown distribution option: ‘package_json’

This also applies to other relevant options, as it is caused by the execution of setup.py without calmjs being available to setuptools, such that the handling method for these keywords remain undefined. This can be corrected by providing calmjs as part of the setup_requires section. Further information on this may be found in the using package_json section of this document.

Contribute

• Issue Tracker: https://github.com/calmjs/calmjs/issues

• Source Code: https://github.com/calmjs/calmjs

Legal

The Calmjs project is copyright (c) 2016 Auckland Bioengineering Institute, University of Auckland. calmjs is licensed under the terms of the GPLv2 or later.

Changelog

3.4.4 (2023-03-07)

• Upgrading to setuptools-65.6.0 and beyond will result in a new version of distutils that “fixed” how logging works, but now the hooks and expected workarounds no longer work. Given that the deprecation of distutils is in the works, no fixes will be done until setuptools fully replaces it beyond the release of Python 3.12, as the author no longer trust Python to provide a stable platform to develop software on. Thus, any future “fixes” will only be provided on an even more reactive (rather than proactive) basis. [ #66 ]

3.4.3 (2023-03-02)

• This is a maintenance release for Python 3.11

• The registration of subparser of the same name will now be blocked, rather than cascade to result into a ArgumentError that cannot be recovered from. [ #64 ]

3.4.2 (2021-10-09)

• This is a maintenance release for Python 3.10; no substantial changes were made.

• Provided a check for disabling integration tests using the CALMJS_SKIP_INTEGRATION environment variable, when set to a non- empty string, the integration tests found in test_dist will be skipped. [ #60 ]

3.4.1 (2019-05-23)

• If a toolchain execution raised the abort or cancel exception, there will now be an appropriate debug log entry if debug mode is enabled.

• Avoid breaking an existing manner of assignment of advice_packages in related projects - ensure that a None value that was provided be treated as an empty list. [ #57 ]

3.4.0 (2019-05-22)

• Clean up a failing test case on Windows due to failure to use normcase to normalize the path for comparison between test result and expected answers. [ #53 #54 ]

• Provide a means to enable/disable the invocation of the post_mortem debugger at the class level, such that this feature is not forcibly enabled for all subclasses of Runtime. [ #55 ]

• Provide a registry for packages to specify which requirement (package) to acquire toolchain advice from to apply by default for the relevant toolchains. This also necessitated some changes to where the optional advices are applied, from the previous location applied as a setup level advice applied by the standard toolchain runtime, to the default sequence within the default toolchain itself. [ #56 ]

  • As a consequence of this feature, any setup advice that trigger a toolchain abort or cancel will not leave the cleanup advices not handled.

  • This feature is implemented in a manner that allow manual invocation of a toolchain with a list of manually provided advice_package be able to always override the default specified ones.

3.3.1 (2018-08-20)

• Correct the implementation of the helper function which such that path arguments to valid executables are accepted and returned. [ #52 ]

3.3.0 (2018-07-23)

• Implement the features required to simplify the process of exposing auxiliary resource files provided by Python packages to the Node.js build tools. [ #46 #48 #50 ]

  • Provide a standardised base child module registry and some helper functions for their usage.

  • Provide a loader module registry with a restrictive naming scheme that directly references a single parent module registry.

  • This also necessitated exposing the mapper of the parent registry in a way that is reusable for other filename extensions, so that the default mapper will also accept the globber and the filename extension arguments.

  • Refactor a number of registry classes so that they may be more easily extended.

  • The self-referential property of the root registry is now properly implemented.

• For Node.js packages that didn’t have a main or browser section defined in their package.json, make use of the default entry file index.js if it exists. [ #49 ]

3.2.1 (2018-05-16)

• Pack related helpers for specific sets of package metadata files into functions that return them. Naturally existing ones are provided, with the keys/filenames parameterized for reuse by dependants. [ #43 ]

• Use the ast module for parsing the es5 String Node value instead of the unicode-escape method as it encompasses more cases, including the line continuation escape sequence which can show up. [ #44 ]

3.1.0 (2018-04-30)

• Fix the modgen function in calmjs.indexer by actually not using the marked as deprecated indexer functions by default, but instead use the pkg_resources version as originally intended. [ #30 #33 ]

• Ensure lookups on package names that have been normalized internally by pkg_resources can still be resolved by their original name. [ #31 ]

• Correctly return an unsuccessful exit code on various partial success while running calmjs artifact build command and for the distutils build_calmjs_artifacts command. [ #27 #38 ]

• Correctly locate the subparser(s) that were responsible for whatever arguments they cannot recognize; includes cleaning up the interactions between the runtime and argparser classes and Python 3.7 compatibility fixes. [ #41 ]

• Fix handling of working directory flag as the validation should be done in the beginning rather than later. Also clean up various logging/error messages surrounding that, plus a fix to toolchain test case isolation. Note that downstream packages that did not set up the export target as an absolute part will result in a warning. [ #42 ]

3.0.0 (2018-01-10)

• The yarn subcommand is now provided as an alternative to npm.

• Also decreased the log verbosity during the bootstrap runtime stage, so that for systems that don’t have the required binaries available, the default calmjs command won’t show those pile of warnings for that (increasing verbosity with -v will restore those warnings).

• Some confusing internal (but public) identifiers which are used in the Toolchain and Spec system have been renamed to better reflect their intended use and purpose. Deprecation code is applied to aid transition, and these will be removed in 4.0.0.

  • For Spec:

    • *_source_map -> *_sourcepath (except for the key that really amplified the confusion which was generate_source_map)

    • *_targets -> *_targetpaths (to be consistent with paths on the filesystem).

  • On the Toolchain, for the similar reasons as above:

    • sourcemap_suffix -> sourcepath_suffix

    • target_suffix -> targetpath_suffix

• Changed the order of binary resolution for Driver instances with configured NODE_PATH and current working directory to align with how Node.js inject them internally (in module.paths, current working directory has higher order of precedence over NODE_PATH), for the method BaseDriver.find_node_modules_basedir.

• Framework for predefined artifact generation for packages through the calmjs.artifacts registry.

• Also split off the directory resolution from the above method to BaseDriver.which_with_node_modules.

• Deprecated the existing toolchain.transpiler function as a standard callable. The new version must be an instance of BaseUnparser provided by the calmjs.parse package. The NullToolchain will retain the usage of the legacy transpiler.

• Generation of the full transpile targetpaths will be normalized. Note that targetpath is still toolchain specific.

• Removed most of the vlqsm module as the functionality is now provided by calmjs.parse.vlq and calmjs.parse.sourcemap. Only the legacy SourceWriter class remain, which is deprecated.

• Provide generic first class support for loader plugins, such that downstream packages should no longer need to explicitly declare extras_calmjs to specify the location of loaders for all the different toolchains (which inevitably collide and cause conflicts). Toolchains downstream will need to implement support for this.

• Artifact production support, including integration with setuptools.

2.1.0 (2016-11-29)

• Namespace packages that have a module explicitly provided should still be able to be looked up if a valid entry point is provided; naturally if the module isn’t declared correctly then the behavior remains unchanged (github issue #5)

• Name field in package.json should contain project names standard to Node.js, i.e. if extras are specified, it should be stripped. This is done so that that npm will not choke on it with a warning and die. (github issue #4)

2.0.0 (2016-11-16)

• Expose the indexer module functions mapper and modgen as public.

• Completely refactored the Toolchain class to have much more consistent method naming convention and argument lists.

• The compile method now reads from an instance specific list of methods which allow very customizable compilation steps.

• Specific ways for a toolchain to skip specific names based.

• Fixed copying of bundle sources to targets nested in subdirectories.

• The Spec callback system is now renamed to advice system and more comprehensively implemented; every step within the toolchain will execute advices before and after for each respective step that have been registered under the matching identifiers. The identifiers for advices are are formalized as constants that can be imported from the calmjs.toolchain module.

• The advice system has dedicated exceptions which can be raised to signal an abort or cleanly stop a run.

• A couple spec keys were formalized, which are BUILD_DIR and CONFIG_JS_FILES, reserved for the build directory and marking out configuration JavaScript files.

• On a successful toolchain call, all advices registered to the spec under the key calmjs.toolchain.SUCCESS will now be invoked.

• Dedicated runtime provided for Toolchain subclasses, joining the ranks of a few other BaseDriver subclasses. This is implemented as calmjs.runtime.ToolchainRuntime.

• calmjs.runtime.Runtime can be subclassed and nested as it will now nest all BaseRuntime. Also the init method is removed, just use __init__ and standard subclassing super usage rules.

• The default ArgumentParser instance for every Runtime will no longer be created until accessed, as it is now a property.

• Provide a way for packages to declare the primary module registry or registries it declared packages for through a new setup keyword calmjs_module_registry, if required and desired.

• The default set of module registry names have been changed. Registry calmjs.pythonic is renamed to calmjs.py.module; the related testing related registry is renamed to include the full name of its implied target.

• Reserved a small set of core (already defined) registries for the calmjs framework, which is formally defined and enforce by the registry itself.

• Corrected interactive-mode detection.

• Helpers for source map generation for simple transpilation.

1.0.3 (2016-09-07)

• Fixed the issue with bad environmental variables for subprocess.Popen for Windows under Python 2.7.

• Other minor testing fixes on CI platforms.

1.0.2 (2016-09-04)

• Fixed invocation of binaries on the Windows platform.

• Corrected some minor wording.

1.0.0 (2016-09-02)

• Initial release of the calmjs framework foundation.

• A cli runtime entry point is provided, named calmjs.

• Provide core integration with npm and generation of package.json through the setuptools framework; this is accessible as a setuptool command or through calmjs runtime.

• Provide a registry framework for use within calmjs.

• Provide core registries for registering packages for use by calmjs through the predefined setuptools entry points and groups.

• Provide a cli tool driver framework for interacting with node and other Node.js based or other command line tools.

• Provide the base toolchain framework, built on top of the tool driver framework.

• Provide modules for doing integration testing for modules that build on top of the calmjs framework.