django-npm-mjs 4.0.0

A Django package to use npm.js dependencies and transpile ES2015+

django-npm-mjs

A Django package to use npm.js dependencies and transpile ES2015+

This package is used by Fidus Writer to bundle JavaScript. We try to keep it as generic as possible, so if there is something that seems very odd and specific to Fidus Writer, it is likely just an oversight from us. Please contact us and we'll see what we can do about it.

This package similar to django-compressor in that it treats JavaScript files before they are served to the user. But there are some differences:

• It does not mix different JavaScript module entry files. It only bundles everything imported from one entry file. With ES2015+ there is not as much need to have lots of JavaScript files operating in the global namespace.

• It allows importing from one django app in another app within the same project as if they were in the same folder similar to how static files and templates are handled by Django.

• It includes handling of npm.js imports.

• The JavaScript entry files' base names do not change and an automatic version query is added to be able to wipe the browser cache (/js/my_file.mjs turns into /js/my_file.js?v=239329884). This way it is also possible to refer to the URL from JavaScript (for example for use with web workers).

• It allows for JavaScript plugin hooks between django apps used in cases when a django project can be used both with or without a specific app, and the JavaScript from one app needs to import things from another app.

• The transpiler collects JavaScript from all discoverable Python packages, not only INSTALLED_APPS. This lets you compile a single bundle before packaging (e.g. in a Debian package or Docker image) and enable or disable plugins later at runtime simply by adding or removing apps from INSTALLED_APPS. The production server never needs to transpile again.

Quick start

1. Install "npm_mjs"

    pip install django-npm-mjs
   

2. Add "npm_mjs" to your INSTALLED_APPS setting like this::

    INSTALLED_APPS = [
        ...
        'npm_mjs',
    ]
   

3. Define a PROJECT_PATH in the settings as the root folder of the project (PROJECT_DIR will also be accepted)::

    PROJECT_PATH = os.path.realpath(os.path.join(os.path.dirname(__file__), '..'))
   

4. Define a SETTINGS_PATHS in the settings to contain the paths of all setting files (settings.py + any local_settings.py or similar file you may have defined) - this is to transpile again whenever settings have changed::

    SETTINGS_PATHS = [os.path.dirname(__file__), ]
   

5. Add the static-transpile folder inside the PROJECT_PATH to the STATICFILES_DIRS like this::

    STATICFILES_DIRS = (
        os.path.join(PROJECT_PATH, 'static-transpile'),
        ...
    )
   

6. Load transpile, and use static template tags to your templates to refer to JavaScript files. All entry files to ES2015+ modules need to have *.mjs endings. Entries can look like this::

    {% load transpile %}
    ...
    <script type="text/javascript" src="{% static "js/index.mjs" %}"></script>
   

You can continue to load other resources such as CSS files as before using the static template tag::

    <link type="text/css" rel="stylesheet" href="{% static "css/fonts.css" %}" />

7. Run ./manage.py transpile.

8. Run ./manage.py runserver. Your ES2015+ modules will be served as browser compatible JS files and all static files will have a versioned ending so that you can set your static server to let browsers cache static files indefinitely as long as DEBUG is set to False.

NPM.JS dependencies

1. Add package.json or package.json5 files into one or more of your apps. All package files will be merged.

2. Import in your JS files from any of the npm modules specified in your package files.

3. Run ./manage.py transpile.

4. Run ./manage.py runserver.

Plugins

django-npm-mjs provides a built-in plugin system for projects that ship with optional Django apps. Instead of editing core code to register a plugin, you drop a JavaScript file into a convention-based directory and the transpiler wires it up automatically.

How it works

1. Discovery at compile time — transpile walks all Python packages on sys.path and copies every static/js/plugins/<type>/*.js file into the build cache. Files named init.js are ignored (they are legacy placeholders).

2. Index generation — For each <type> directory the transpiler writes an index.js that imports every discovered module and exports them as a plugins array of [app_name, module_namespace] tuples:

   import * as my_plugin from "./my_plugin"
   import * as another from "./another"
   export const plugins = [
     ['my_plugin', my_plugin],
     ['another', another],
   ]
   

3. Runtime filtering — Core pages import {plugins} from ../../plugins/<type> and iterate with:

   plugins.forEach(([app, plugin]) => {
       if (!this.app.settings.APPS.includes(app)) {
           return
       }
       Object.values(plugin).forEach(pluginExport => {
           if (typeof pluginExport === "function") {
               this.plugins[pluginExport.name] = new pluginExport(this)
               this.plugins[pluginExport.name].init()
           }
       })
   })
   

   The APPS filter means plugins are compiled into the bundle regardless of whether their Django app is in INSTALLED_APPS, but they are only instantiated when the app is enabled.

Why compile everything?

This design lets you build a single JavaScript bundle before packaging (e.g. during CI or in a Debian build) and ship it to production. End users can then enable or disable bundled plugins by editing INSTALLED_APPS in their configuration.py without ever running the transpiler on the production server.

Writing a plugin

Create a JavaScript file inside your Django app's static/js/plugins/<type>/ directory (choose the hook point that matches your use case):

// my_plugin/static/js/plugins/app/my_plugin.js
export class MyPlugin {
    constructor(app) {
        this.app = app
    }

    init() {
        // Add a route, register a menu item, attach an event listener, etc.
    }
}

You can place files in multiple hook points if needed:

my_plugin/static/js/plugins/app/my_plugin.js
my_plugin/static/js/plugins/menu/my_plugin.js

Each exported class will be instantiated automatically when the containing app is present in INSTALLED_APPS.

Referring to the transpile version within JavaScript sources

In your JavaScript sources, you can refer to the version string of the last transpile run like this::

    transpile.VERSION

For example::

    let downloadJS = `download.js?v=${transpile.VERSION}` // Latest version of transpiled version of download.mjs

ManifestStaticFilesStorage

If you use ManifestStaticFilesStorage, import it from npm_mjs.storage like this:

from npm_mjs.storage import ManifestStaticFilesStorage

If you use that version, you can refer to other static files within your JavaScript files using the staticUrl() function like this:

const cssUrl = staticUrl('/css/document.css')

Note that you will need to use absolute paths starting from the STATIC_ROOT for the staticUrl() function. Different from the default ManifestStaticFilesStorage, our version will generally interprete file urls starting with a slash as being relative to the STATIC_ROOT.

Translations

Commands such as ./manage.py makemessages and ./manage.py compilemessages will work as always in Django, with some slightly different defaults. Not specifying any language will default to running with --all (all languages). Not specifying any domain will default to running for both "django" and "djangojs" (Python and Javascript files). The static-transpile directory will also be ignored by default.

NOTE: JavaScript files that contain template strings will require at least xgettext version 0.24 or higher. See below for installation instructions.

Install xgettext 0.24

First check which xgettext version your OS comes with:

xgettext --version

If it is below version 0.24, you will need to install a newer version. For example in your current virtual environment:

Step 1: Activate Your Virtual Environment

source /path/to/venv/bin/activate

Step 2: Install Build Dependencies Install tools required to compile software:

sudo apt-get update
sudo apt-get install -y build-essential libtool automake autoconf

Step 3: Download and Extract Gettext 0.24

wget https://ftp.gnu.org/pub/gnu/gettext/gettext-0.24.tar.gz
tar -xzf gettext-0.24.tar.gz
cd gettext-0.24

Step 4: Configure and Install into the Venv Install to your venv's directory using --prefix:

./configure --prefix=$VIRTUAL_ENV
make
make install

Step 5: Verify Installation Ensure the new xgettext is in your venv and check the version:

which xgettext  # Should output a path inside your venv
xgettext --version  # Should show 0.24

Step 6: Cleanup

cd ..
rm -rf gettext-0.24 gettext-0.24.tar.gz

Testing

The package includes a comprehensive test suite to ensure reliability and prevent regressions.

Running Tests

Run all tests using the provided test runner:

python runtests.py

Or use Python's unittest directly:

python -m unittest discover npm_mjs/tests

If you have pytest installed:

pytest

Critical Regression Tests

The test suite includes specific tests for previously encountered bugs:

• Double slashes in strings ("path//to//file" should not be treated as comments)
• Single quotes inside double-quoted strings (proper quote conversion)
• Long lines (handling files with line 7, column 178 errors)
• URLs in strings and comments (preserving https:// patterns)

For more details, see npm_mjs/tests/README.md.

Debugging JSON5 Parse Errors

If you encounter errors when parsing package.json5 files, the parser provides helpful debug output using Python's logging module. The Django management command automatically enables debug mode, which shows:

• The processed content after comment removal and quote conversion
• The exact line and column where parsing failed
• The specific error message

Example debug output:

================================================================================
JSON5 Parser Error - Processed content that failed to parse:
================================================================================
 -->   4:   "key": "value with problem",
================================================================================
Error at line 4, column 26: Expecting ',' delimiter
================================================================================

When using the parser directly in your code, you can enable debug output:

import logging
from npm_mjs.json5_parser import parse_json5

# Configure logging to see debug output
logging.basicConfig(level=logging.ERROR, format='%(message)s')

# Enable debug output for troubleshooting
try:
    result = parse_json5(content, debug=True)
except json.JSONDecodeError as e:
    # Debug info logged as ERROR before exception is raised
    pass

By default, debug=False to keep the output clean in automated scripts and tests. When debug=True, error details are logged at the ERROR level using Python's standard logging module.