A addon for Plone which allow you to track requests from AI Chatbots to the server in Matomo.

Features

Tracks the requests of AI bots in Matomo, also when Varnish serves them from its cache, without slowing them down:

• Varnish classifies AI bots by User-Agent as user (an AI assistant fetching a page for a user, like Claude-User or ChatGPT-User), search (AI search crawlers) or training (training data crawlers), and writes that to its log.
• varnishncsa writes these requests to a log file, and a small shipper sends them in batches to Plone. When Plone or Matomo is down, tracking is delayed, not lost.
• Plone forwards them to Matomo's bulk tracking API, with the original time of the request. Requests for the resources of pages, like images, styles and scripts, are skipped, and documents like PDFs are tracked as downloads:
  • user requests go to Matomo's AI Chatbots report (Matomo 5.8 or later).
  • Optionally all AI bot requests go to a separate Matomo site as visits, with the bot category, bot name and the Varnish cache status (hit, miss, pass) as custom dimensions.

The control panel "Matomo AI Chatbot Tracking" has the Matomo settings. See the add-on's README for setting up Varnish, varnishncsa and the shipper.

Installation

Install collective.matomoaitracker with pip:

pip install collective.matomoaitracker

And to create the Plone site:

make create-site

This Addon expects you to run a varnish, varnishnsca and shipper Docker image. Examples can be found in the subfolder for each service.

Contribute

• Issue tracker
• Source code

Prerequisites ✅

• An operating system that runs all the requirements mentioned.
• uv
• Make
• Git
• Docker (optional)

Development Installation 🔧

1. Clone this repository, then change your working directory.

   git clone git@github.com:collective/collective.matomoaitracker.git
   cd collective.matomoaitracker
   

2. Install this code base. This also builds the Docker images for the full stack.

   make install
   

Docker stack 🐳

The full stack runs in Docker:

requests:  Varnish (8001) -> Nginx VirtualHostMonster (8002) -> Plone site "Plone" (8003)
tracking:  Varnish log -> varnishncsa -> ai-bots.log -> shipper -> Plone -> Matomo (8004)

Varnish's configuration (varnish/default.vcl) is based on the one of Plone's cookieplone project templates, tuned for plone.app.caching: Plone purges changed content from Varnish, logged-in users are not cached. For the tracking, Varnish only classifies AI bots and writes the result to its log, it makes no HTTP calls. varnishncsa writes the AI bot requests to a log file, which the shipper sends in batches to Plone, and Plone forwards them to Matomo. By default that is a fake Matomo, which records what it receives for the tests.

make stack-start

On start the Plone site is created, the Matomo settings are applied and the service user for the shipper is created. Settings can be overridden with environment variables or in a .env file, see .env.example:

cp .env.example .env

Note: if you change anything to the create_site.py script, you must remove the matomoaitracker-plone image, otherwise this has no effect.

Simulate an AI chatbot visit, and see what reached the fake Matomo a few seconds later:

curl -A "Claude-User/1.0" http://localhost:8001/
curl http://localhost:8004/_requests

Test the VCL and the varnishncsa format with varnishtest:

make varnish-test

Test the whole tracking chain against the running stack with the fake Matomo: every AI bot is tracked once in the right category, also from the cache, other requests are not, bots are not slowed down, and outages of Matomo or the shipper delay tracking without losing it. These are the pytest tests in tests/stack, which make test skips because they need the stack:

make stack-test

The tests run with Plone's caching policy, and again with pages cached in Varnish for 60 seconds (moderateCaching), restoring the policy afterwards. They log in to Plone's REST API as admin:admin for that, set PLONE_ADMIN=user:password for other credentials.

Other targets: make stack-logs, make stack-stop and make stack-remove-data.

Add features using plonecli or bobtemplates.plone

This package provides markers as strings (<!-- extra stuff goes here -->) that are compatible with plonecli and bobtemplates.plone. These markers act as hooks to add all kinds of subtemplates, including behaviors, control panels, upgrade steps, or other subtemplates from plonecli.

To run plonecli with configuration to target this package, run the following command.

make add <template_name>

For example, you can add a content type to your package with the following command.

make add content_type

You can add a behavior with the following command.

make add behavior

You can check the list of available subtemplates in the [`bobtemplates.plone` `README.md` file](https://github.com/plone/bobtemplates.plone/?tab=readme-ov-file#provided-subtemplates).
See also the documentation of [Mockup and Patternslib](https://6.docs.plone.org/classic-ui/mockup.html) for how to build the UI toolkit for Classic UI.

License

The project is licensed under GPLv2.

Credits and acknowledgements 🙏

Generated using Cookieplone (2.0.0) and cookieplone-templates (fa8eca4) on 2026-09-30 17:16:12.816436. A special thanks to all contributors and supporters!

Changelog

1.0.0a1 (2026-10-08)

Breaking

• Track AI bots from the Varnish log instead of with a synchronous libvmod-curl call, so bots are no longer slowed down and cache hits are tracked without any HTTP call. Varnish classifies bots as user, search or training and logs that, varnishncsa writes the requests to a file, and a new shipper (shipper/matomo_ai_shipper.py) sends them in batches to @@matomoaitracker, which forwards them to Matomo's bulk tracking API with the original request time. AI assistants go to Matomo's AI Chatbots report; optionally all AI bots go to a separate Matomo site with the bot category and cache status as custom dimensions. Outages of Plone or Matomo delay tracking instead of losing it.

  @@matomoaitracker now only accepts POSTed JSON batches from users with the new collective.matomoaitracker: Submit tracking events permission (role "Matomo AI Tracker"), and needs the Matomo token in the MATOMO_AI_TOKEN_AUTH environment variable. varnish/varnish.vcl is replaced by varnish/matomoaitracker.vcl, which is included at the top level of the VCL.

Feature

• Add a Docker stack for development: Varnish in front of Nginx doing the VirtualHostMonster rewrites to Plone, with varnishncsa, the shipper and a fake Matomo for the tracking. make install builds the images, make stack-start runs them, make varnish-test and make stack-test test them. Settings can be put in a .env file, see .env.example. Varnish uses the configuration of Plone's cookieplone templates.
• Add an optional "Bot name" custom dimension to the AI bots site, with the name of the bot like GPTBot or ClaudeBot, to see which crawlers cause load peaks.
• Add the Gemini-Deep-Research and Google-NotebookLM AI assistants, skip the resources of pages (configurable as "Excluded URLs" in the control panel), track documents like PDFs as downloads based on their content type, and send source=Varnish with requests for the AI Chatbots report.
• Give site administrators more control in the control panel: switch AI bot tracking off, choose which bot categories are tracked in the AI bots site, check the settings against Matomo with "Test connection", and see when the shipper last delivered, how many requests were tracked, and the last error.
• The Matomo token can be pasted in the control panel. It is never sent back to the browser, saving the form with an empty token field keeps it, and "Remove token" removes it. The MATOMO_AI_TOKEN_AUTH environment variable still works and overrides the stored token.
• The MATOMO_AI_TRACKING_ENABLED environment variable overrides the "Track AI bots" setting, which the control panel then shows read-only. Use it to track only in production, also when its database is copied to other environments.