roundup 2.4.0

A simple-to-use and -install issue-tracking system with command-line, web and e-mail interfaces. Highly customisable.

I’m proud to release version 2.4.0 of the Roundup issue tracker. This release is a bugfix and feature release, so make sure to read docs/upgrading.txt to bring your tracker up to date.

The 79 changes, as usual, include some new features and many bug fixes.

Version 2.4.0 will be the last release to support Python 2. The next minor release, planned for mid 2025, will occur 5 years after Roundup started supporting Python 3.

Note that you should run roundup-admin ... migrate to update the database schema version. Do this before you use the web, command-line or mail interface and before any users access the tracker.

You can install it with:

pip install roundup

(preferably in a virtual environment). To download it, use:

pip download roundup

then unpack and test/install from the tarball.

Among the notable improvements in 2.4.0 from the 2.3.0 release are:

• three CVE’s have been fixed. One requires changes to your tracker’s home directory. The other two are fixed by installing 2.4.0. See https://www.roundup-tracker.org/docs/security.html for details and instructions on how to fix these in 2.4.0 and earlier releases.

• new classhelper component thanks to a team of students from CS682 at U-Mass Boston. This fixes many issues with the old classhelper. It is implemented as a web-component and needs REST interface access. It will fall back to the classic classhelper if REST is not available or if the browser does not support web-components.

• fix Windows Python installation using pip. It used to go into an infinite loop during install or download. Also fix installation of shared files (templates) so roundup-admin can find them.

• using @current_user as a value in a search URL for a user property will use the current logged in user. Now you can share searches like: “My issues” as “my” will become the current logged in user.

• login failures to the REST/XML-RPC interfaces are now rate limited to limit password guessing attacks.

• utf8mb4 is the default charset for MySQL. This requires migrating your database using the mysql client. You can choose to keep the older character set in config.ini.

• PostgreSQL services defined in pg_service.conf can be used. PostgreSQL schemas are supported to eliminate the need for the roundup user to have database creation/deletion privileges.

• fix out of memory issue when importing larger trackers into PostgreSQL.

• multiple roundup-admin improvements: display protected properties (like creation date), better formatting of output, command history. Also on windows, pyreadline3 is supported to provide an editable interactive command line.

• an experimental wsgi performance improvement in 2.3.0 is now now the default and is opt-out.

• new template functions: utils.readfile and utils.expandfile. Javascript that is included in the Python core will be moved to external files and be able to have values from Roundup substituted in the Javascript.

• allow content-type of a template to be set from inside the template. This allows returning json or xml from a template without a .json or .xml extention.

• fix import/export on windows to use Unix style line endings fixing export/import on Windows and making exports portable across platforms.

• various other Windows platform fixes including test suite fixes.

• sqlite version 1 and StructuredText support removed.

The file CHANGES.txt has a detailed list of feature additions and bug fixes for each release. The most recent changes from there are at the end of this announcement. Also see the information in doc/upgrading.txt.

If you find bugs, please report them to issues AT roundup-tracker.org or create an account at https://issues.roundup-tracker.org and open a new ticket. If you have patches to fix the issues they can be attached to the email or uploaded to the tracker.

Upgrading

If you’re upgrading from an older version of Roundup you must follow all the “Software Upgrade” guidelines given in the doc/upgrading.txt documentation.

Note that you should run roundup-admin ... migrate for all your trackers to update the database schema version. Do this before you use the web, command-line or mail interface and before any users access the tracker.

Roundup requires Python 2 newer than version 2.7.12 or Python 3 newer than or equal to version 3.6 for correct operation. (Python 3.4 or 3.5 may work, but are not tested.) Note that Roundup 2.4.0 will be the last release to support Python 2. You should deploy new trackers with Python 3 and plan on upgrading older trackers from Python 2 to Python 3. See the upgrade guide.

To give Roundup a try, just download (directions above), unpack and run:

python demo.py

then open the url printed by the demo app.

Release info and download page:

https://pypi.org/project/roundup/

Source and documentation is available at the website:

https://www.roundup-tracker.org/

Mailing lists - the place to ask questions:

https://sourceforge.net/p/roundup/mailman/

About Roundup

Roundup is a simple-to-use and install issue-tracking system with command-line, web and e-mail interfaces. It is based on the winning design from Ka-Ping Yee in the Software Carpentry “Track” design competition.

Roundup manages a number of issues (with flexible properties such as “description”, “priority”, and so on) and provides the ability to:

1. submit new issues,

2. find and edit existing issues, and

3. discuss issues with other participants.

The system facilitates communication among the participants by managing discussions and notifying interested parties when issues are edited. One of the major design goals for Roundup that it be simple to get going. Roundup is therefore usable “out of the box” with any Python 3.6+ installation. It doesn’t even need to be “installed” to be operational, though an install script is provided.

It comes with five basic issue tracker templates

• a classic bug/feature tracker

• a more extensive devel tracker for bug/features etc.

• a responsive version of the devel tracker

• a jinja2 version of the devel template (work in progress)

• a minimal skeleton

and supports four database back-ends (anydbm, sqlite, mysql and postgresql).

Recent Changes

From 2.3.0 to 2.4.0

Fixed:

• CVE-2024-39124 - The classhelpers (_generic.help.html) are vulnerable to an XSS attack. A specially crafted URL that used that endpoint would result in running a script embedded in the URL. (Found/reported by Alec Romano (4rdr), fix/tests John Rouillard)

• CVE-2024-39125 - If the Referer header is set to a script tag, it will be executed when the error in the Referer header is reported. (Found/reported by Alec Romano (4rdr), fix/tests John Rouillard)

• CVE-2024-39126 - PDF, XML and SVG files attached to an issue can contain embedded JavaScript. This JavaScript was executed when the file was accessed. PDF files are now downloaded and not displayed in the browser. A content security policy is added for all download files which prevents code execution in SVG files. (Found/reported by Alec Romano (4rdr), fix/tests John Rouillard)

• issue2551282 - MySQL utf8mb4 issues and issue2551115 - Use utf8mb4 as a default for MySQL instead of utf8 The default database type and collations have been set to: utf8mb4, utf8mb4_unicode_ci and utf8mb4_0900_bin. They are (sadly) configurable from config.ini. Require directions on upgrading the MySQL db have been documented in upgrading.txt.

• issue2551063 - Rest/Xmlrpc interfaces needs failed login protection. Failed API login rate limiting with expiring lockout added. (John Rouillard)

• issue2551184 - improve i18n handling. Patch to test to make sure it uses the test tracker’s locale files and not other locale files. (Marcus Priesch)

• issue2551283 - fail if version 2.4.9 of markdown2 is used, it broke [issue1](issue1) style links. Support markdown2 2.4.8 and earlier and 2.4.10 with its new schema filtering method. (John Rouillard)

• multiple flake8 fixes (John Rouillard)

• rename loop variable in ‘for sendto in sendto:’ (John Rouillard)

• issue2551193 - Fix roundup for removal of cgi and cgitb standard python modules (and FieldStorage/MiniFieldStorage). Replaced imports from cgi to use roundup.anypy.cgi_ which will load the system cgi unless it is missing. Then it will load roundup.anypy.vendored.cgi and make *FieldStorage symbols available. Roundup uses its own cgitb.py and not the system cgitb.py. It looks like it’s the precursor to the system cgitb.py. (John Rouillard)

• issue2551278 - datetime.datetime.utcnow deprecation. Replace calls with equivalent that produces timezone aware dates rather than naive dates. (John Rouillard)

• when using “roundup-admin display” indent the listing only if headers or protected fields are requested. This makes the output look like it did previously to 2.3.0 if the new features aren’t used. Roundup-admin output was never meant to be machine parsed, but don’t break it unless required. (John Rouillard)

• issue2551290 - pip install roundup Hangs on Windows 10 The install under windows goes into an infinite loop using pip or source install. (John Rouillard)

• Document use of pyreadline3 to allow roundup-admin to have CLI editing on windows. (John Rouillard)

• issue2551293 - remove schema_hook from Tracker instance. Looks like it was an obsolete hook used for testing. Never documented and not accessible from schema.py.

• Fix roundup-admin security command. Lowercase its optional argument. Roles are indexed by lower case role name. So ‘security User’ and ‘security user’ should generate the same output. (John Rouillard from issue on mailing list by Chuck Cunningham)

• make roundup-server exit more quickly on ^C. This seems to be limited to windows. (John Rouillard)

• Fix error handling so failure during import of a non-user item doesn’t cause a second traceback. (Found by Norbert Schlemmer, fix John Rouillard)

• Handle out of memory error when importing large trackers in PostgreSQL. (Found by Norbert Schlemmer, extensive testing by Norbert, fix John Rouillard)

• use unittest.mock rather than mock for test/test_hyperdbvals.py. (found by Ralf Schlatterbeck. Fix John Rouillard)

• disable proxy with wget in roundup_healthcheck. (Norbert Schlemmer Noschvie on github.com)

• support dicttoxml2.py for Roundup running on 3.7 and newer. dicttoxml uses a type alias: collection.Iterator that is dropped in Python 3.10. (found by Norbert Schlemmer, fix John Rouillard)

• fix duplicate html id ‘password’ in user.item.html in all templates except jinja2. (John Rouillard)

• fix unclosed file when saving index in indexer_dbm.py. (John Rouillard)

• fix task index in devel tracker so it doesn’t cause a crash if all fields are selected. (John Rouillard)

• fix windows install. When using pip share directory is installed in a directory tree under the lib directory. Fix it so that Lib/share is used to install the share tree. The lets Roundup find tracker templates and translation files. (Found by Simon Eigeldinger, fix John Rouillard)

• fix roundup-demo, interactive mode would nuke an existing tracker. (Found Tonu Mikk, fix John Rouillard)

• fix detection/reporting when using a SQLite3 library without FTS5 support. Install docs updated to state that FTS5 support is required when using SQLite for back end. (Found Tonu Mikk, fix John Rouillard)

• issue2551320: user.help-search.html doesn’t respect properties. Setting url parameter properties when using the classhelp for users now shows the requested properties. (Found by Patel Malav and Nikunj Thakkar of the UMass-Boston CS682 Spring 2024 class; fix John Rouillard)

• use ast.eval_literal() rather than eval() to turn CSV exported string values into Python object/values.

• use template’s guess at Content-Type in headers only if Content-Type is not already set. This allows a template to set its own content type. For example: _generic.translate can set content type (via request.client.additional_headers) to application/json and return json from the template. This json could access the 1i18n functions for a javascript helper. (John Rouillard)

• when template processing raises an exception the line number is sometimes missing. This causes cgitb to raise a second exception which clobbers the info about the template issue. As a stop-gap set the line number to -1 so the original traceback can be seen. This could be a bug in ZopeTAL. (John Rouillard)

• issue2551328 - REST results show next link if number of results is a multiple of page size. There should be no next link. (Found by Patel Malav and Bharath Kanama of the UMass-Boston CS682 Spring 2024 class; fix John Rouillard)

• issue2551264 - REST X-Total-Count header and @total_size count incorrect when paginated - correct values are now returned. (John Rouillard)

• issue2551331 - Fix repeat first/last methods. (John Rouillard)

• Fix import/export on windows. Use unix line terminating characters. (John Rouillard)

• Fix anydbm session/otks clear() method on windows when backed by dumbdbm. Also make anydbm detect the initialized database when using dumbdbm. (John Rouillard)

• Use of ‘-’ directory in static_files config option under windows Python fixed. (John Rouillard)

• issue2551334 - number of test bugs that prevented test suite from running under Windows Python are fixed. WIP. (John Rouillard)

• issue2551302 - Remove support for sqlite version 1 from back_sqlite.py. We have been using sqlite3 for over a decade. (John Rouillard)

• issue2551285 - Remove StructuredText support. reStructuredText is still supported. (John Rouillard)

• Use roundup-demo -p option to set listening port. Was ignored before. (John Rouillard)

• issue2551346 - Classic tracker’s statusauditor raises error if detectors/config.ini missing STATUSAUDITOR_CHATTING_REQUIRES_TWO_USERS. The statusauditor.py for jinja2 and classic templates has been changed to assume that this option is off when the setting is missing from detectors/config.ini. Other templates do not implement this option. (John Rouillard)

• issue2551350 - Python changes for 3.12 with roundup 2.3.0. Fixes for cgitb.py crash due to pydoc.html.header() signature change. (Patch by Andrew (kragacles), applied John Rouillard)

• issue2551350 - Python changes for 3.12 with roundup 2.3.0. Fixes for mailer.py crash due to change in starttls signature change. (Patch by Andrew (kragacles), modified and applied John Rouillard)

• make classhelper link open in a new window by setting target=”_blank”. This prevents overwriting of current page with the classhelper if javascript is disabled. (John Rouillard)

• issue2551341 - if @columns missing from an index url, the group headers colspan property = 0. Add “or 100” in stanza’s so headers span all rows (up to 100).

• fix roundup-server response requiring a 301 redirect. Did not set content length leading to hang/error. (John Rouillard)

• report basename of filename when template file is invalid rather than reporting a TypeError. (John Rouillard)

• Make Last-Modified header use GMT not -0000 timezone. Fix error reported by redbot testing. (John Rouillard)

• Send Vary: Accept-Encoding on any file that could be compressed even if the file is not encoded/compressed. Found by Redbot testing. (John Rouillard)

• make If-None-Match work for static file (@@file) case. Found by Redbot testing (John Rouillard)

• Send vary: accept-encoding for if-modified-since conditional requests where the file is not modified. (John Rouillard)

• Update JWT example in rest.py to use replacement for datetime.datetime.utcnow(). (John Rouillard)

• issue2551219 - document requirements of PEM file when using roundup-server in SSL/TLS mode. Report better error messages when PEM file is missing certificate or private key. (John Rouillard)

• Cleanup tracker index generation by roundup-server. Send correct Content-Length headers so HTTP/1.1 connections don’t hang. (John Rouillard)

• Fix delay when using csv export actions. The CSV file is written incrementally, so we can’t determine the Content-Length. When using HTTP/1.1, this causes a delay while the browser waits for a timeout. Forcing the connection to close after the CSV file is written removes the delay. (John Rouillard)

Features:

• issue2551323 - Remove XHTML support. Disabled option to set html_version to xhtml. Running roundup commands with html_version set to xhtml will result in an “Invalid value for HTML_VERSION: ‘xhtml’” error. (John Rouillard)

• issue2551103 - add pragma ‘display_protected’ to roundup-admin. If true, print protected attributes like id, activity, actor… when using display or specification subcommands. (John Rouillard)

• add -P pragma=value command line option to roundup-admin. Allows setting pragmas when using non-interactive mode. (John Rouillard)

• issue685275 - add pragma show_retired to control display of retired items when using list/table. Add pragma display_header to print headers for display command. Header displays designator and retired/active status. (John Rouillard)

• issue2551299 - support config.ini rdbms option ‘service’. Allow use of a PostgreSQL connection service file (pg_service.conf) for configuring database on a per-tracker basis. Also replaces use of PGSERVICE env variable for single instance trackers. (From ML question by ivanov. John Rouillard)

• issue2550852 - support for specifying a PostgreSQL schema to use for the Roundup database. (Patch by Stuart McGraw; slight modifications, tests, docs: John Rouillard).

• issue2551274: add configurable logging for REST API when something fails, we now log status code and error message. (Ralf Schlatterbeck)

• issue2551317 - add some Jinja2 examples to customizing.txt document. (John Rouillard)

• multiple scripts/… updates - Python3, linting, enhancements: weekly-report,schema-dump.py, roundup-reminder, copy-user.py, dump_dbm_sessions_db.py, contributors.py (John Rouillard)

• roundup/msgfile.py can now be called as ‘python msgfmt.py de.po de.mo’ or ‘python msgfmt.py -o de.mo de.po’ to compile a translation file if GNU msgfmt is missing. (John Rouillard)

• save roundup-admin history between sessions. Load ~/.roundup_admin_rlrc file to set history-size persistently. Add pragma history_length to override for a session. (John Rouillard)

• the roundup-admin history command now dumps the journal entries in a more human readable format. Use the raw option to get the older machine parsible output. (John Rouillard)

• Multiple JWT secrets are supported to allow key rotation. See an updated config.ini for details. (John Rouillard)

• issue2551212 - wsgi performance improvement feature added in 2.2.0 is active by default. Can be turned off if needed. See upgrading.txt for info. (John Rouillard)

• issue2551270 - Better templating support for JavaScript. Add utils.readfile(file, optional=False) and utils.expandfile(file, token_dict=None, optional=False). Allows reading an external file (e.g. JavaScript) and inserting it using tal:contents or equivalent jinja function. expandfile allows setting a dictionary and tokens in the file of the form “%(token_name)s” will be replaced in the file with the values from the dict. (John Rouillard)

• add @group to rest interface collection queries. Useful when using optgroup in select elements. (John Rouillard)

• roundup-demo can set the hostname in the URL using the -H parameter. So you can start a demo tracker that is available from your network using ‘roundup-demo … -B hostname -H hostname’. (John Rouillard)

• issue2551347 - make _generic.help.html work without property settings. THis applies to classic or minimal trackers. It allows use of classhelp without the property seting for informtion only (e.g. description of what a priority or status means) without being able to select the property in the classhelper. Good for adding help for Link properties. (John Rouilllard)

• issue1525113 - notation to filter by logged-in user. Use @current_user with properties that are a Link to the ‘user’ class to match the currently logged in user. Allows sharing of queries like “Issues I created” or “Issues I am assigned to” by removing the hard coded user id number and replacing it with the current user’s id. Tracker templates updated to use it. (John Rouillard from a patch by Jon C. Thomason)

• Add a /rest/data/user/roles REST endpoint. (John Rouillard)

• issue2551353 - Add roundup-classhelper for 2.4.0 release. Integrate new classhelper web component to wrap existing classhelper link. This fixes a number of outstanding bugs against the current classhelper using current web features. (Patel Malav, Nikunj Thakkar, Bharath Kanama with integration by John Rouillard)

• disable spellcheck on all password fields to try to prevent browser from exposing passwords to external servers. (John Rouillard)