Zope Object Database: object database and persistence
Project description
Zope Object Database: object database and persistence
The Zope Object Database provides an object-oriented database for Python that provides a high-degree of transparency. Applications can take advantage of object database features with few, if any, changes to application logic. ZODB includes features such as a plugable storage interface, rich transaction support, and undo.
ZODB 3.8
Introduction
The ZODB package provides a set of tools for using the Zope Object Database (ZODB). The components you get with the ZODB release are as follows:
Core ZODB, including the persistence machinery
Standard storages such as FileStorage
The persistent BTrees modules
ZEO, for scalability needs
documentation (needs more work)
Our primary development platforms are Linux, Mac OS X, and Windows XP. The test suite should pass without error on all of these platforms, although it can take a long time on Windows – longer if you use ZoneAlarm. Many particularly slow tests are skipped unless you pass –all as an argument to test.py.
Compatibility
ZODB 3.8 requires Python 2.4.2 or later.
ZEO servers and clients are wholly compatible among 3.3, 3.4, 3.5, 3.6 and 3.7; a ZEO client from any of those versions can talk with a ZEO server from any. ZODB 3.8 ZEO clients require ZODB 3.8 servers and later. ZODB 3.8 ZEO Servers will work with ZODB 3.2 clients and later.
Prerequisites
You must have Python installed. If you’ve installed Python from RPM, be sure that you’ve installed the development RPMs too, since ZODB builds Python extensions. If you have the source release of ZODB, you will need a C compiler.
You also need the ZConfig, zdaemon, zope.interface, zope.proxy and zope.testing packages. If you are using easy_install or zc.buildout to install ZODB, then these will be installed for you automatically.
Installation
ZODB is released as a distutils package. The easiest ways to build and install it are to use easy_install, or zc.buildout.
To install by hand, first install the dependencies, ZConfig, zdaemon, zope.interface, zope.proxy and zope.testing. These can be found either in the Python Package Index, or at http://download.zope.org/distribution/.
To build it, run the setup script:
% python setup.py build
The 64-bit support for the BTrees package may be enabled by using this build command instead:
% python setup.py build_ext -DZODB_64BIT_INTS build
To test the build, run the test script:
% python test.py
For more verbose test output, append one or two ‘-v’ arguments to this command.
If all the tests succeeded, you can install ZODB using the setup script:
% python setup.py install
This should now make all of ZODB accessible to your Python programs.
Testing for Developers
The ZODB checkouts are buildouts. When working from a ZODB checkout, first run the bootstrap.py script to initialize the buildout:
% python bootstrap.py
and then use the buildout script to build ZODB and gather the dependencies:
% bin/buildout
This creates a test script:
% bin/test -v
This command will run all the tests, printing a single dot for each test. When it finishes, it will print a test summary. The exact number of tests can vary depending on platform and available third-party libraries.:
Ran 1182 tests in 241.269s OK
The test script has many more options. Use the -h or --help options to see a file list of options. The default test suite omits several tests that depend on third-party software or that take a long time to run. To run all the available tests use the --all option. Running all the tests takes much longer.:
Ran 1561 tests in 1461.557s OK
Maintenance scripts
Several scripts are provided with the ZODB and can help for analyzing, debugging, checking for consistency, summarizing content, reporting space used by objects, doing backups, artificial load testing, etc. Look at the ZODB/script directory for more informations.
History
The historical version numbering schemes for ZODB and ZEO are complicated. Starting with ZODB 3.4, the ZODB and ZEO version numbers are the same.
In the ZODB 3.1 through 3.3 lines, the ZEO version number was “one smaller” than the ZODB version number; e.g., ZODB 3.2.7 included ZEO 2.2.7. ZODB and ZEO were distinct releases prior to ZODB 3.1, and had independent version numbers.
Historically, ZODB was distributed as a part of the Zope application server. Jim Fulton’s paper at the Python conference in 2000 described a version of ZODB he called ZODB 3, based on an earlier persistent object system called BoboPOS. The earliest versions of ZODB 3 were released with Zope 2.0.
Andrew Kuchling extracted ZODB from Zope 2.4.1 and packaged it for use by standalone Python programs. He called this version “StandaloneZODB”. Andrew’s guide to using ZODB is included in the Doc directory. This version of ZODB was hosted at http://sf.net/projects/zodb. It supported Python 1.5.2, and might still be of interest to users of this very old Python version.
Zope Corporation released a version of ZODB called “StandaloneZODB 1.0” in Feb. 2002. This release was based on Andrew’s packaging, but built from the same CVS repository as Zope. It is roughly equivalent to the ZODB in Zope 2.5.
Why not call the current release StandaloneZODB? The name StandaloneZODB is a bit of a mouthful. The standalone part of the name suggests that the Zope version is the real version and that this is an afterthought, which isn’t the case. So we’re calling this release “ZODB”. We also worked on a ZODB4 package for a while and made a couple of alpha releases. We’ve now abandoned that effort, because we didn’t have the resources to pursue ot while also maintaining ZODB(3).
License
ZODB is distributed under the Zope Public License, an OSI-approved open source license. Please see the LICENSE.txt file for terms and conditions.
The ZODB/ZEO Programming Guide included in the documentation is a modified version of Andrew Kuchling’s original guide, provided under the terms of the GNU Free Documentation License.
More information
We maintain a Wiki page about all things ZODB, including status on future directions for ZODB. Please see
and feel free to contribute your comments. There is a Mailman mailing list in place to discuss all issues related to ZODB. You can send questions to
or subscribe at
and view its archives at
Note that Zope Corp mailing lists have a subscriber-only posting policy.
Andrew’s ZODB Programmers Guide is made available in several forms, including DVI and HTML. To view it online, point your browser at the file Doc/guide/zodb/index.html
Bugs and Patches
Bug reports and patches should be added to the Launchpad:
Whats new in ZODB 3.8.6 (2010-09-21)
Bugs Fixed:
Updating blobs in save points could cause spurious “invalidations out of order” errors. https://bugs.launchpad.net/zodb/+bug/509801
(Thanks to Christian Zagrodnick for chasing this down.)
BTree sets and tree sets didn’t correctly check values passed to update or to constructors, causing Python to exit under certain circumstances.
Passing keys or values outside the range of 32-bit ints on 64-bit platforms led to undetected overflow errors. Now these cases cause TypeErrors to be raised.
Fixed a serious bug that caused cache failures when run with Python optimization turned on.
Whats new in ZODB 3.8.5 (2009-12-16)
Bug Fixed:
A ZEO threading bug could cause transactions to read inconsistent data. (This sometimes caused an AssertionError in Connection._setstate_noncurrent.)
New Feature:
The standard storages, FileStorage, ClientStorage, DemoStorage, and MappingStorage now allow the version argument to history and load to be ommitted. This is to make it easier to write application code that works with ZODB 3.8 and later versions, which don’t support versions.
Whats new in ZODB 3.8.4 (2009-10-01)
Bugs Fixed:
Conflict errors committing blobs caused ZEO servers to stop committing transactions.
Whats new in ZODB 3.8.3
New Feature:
There’s a new utility script, strip_versions that strips version data from storages. This is needed to prepare databases containing version records for using ZODB 3.9, which no-longer supports versions.
Bugs Fixed:
CVE-2009-2701: Fixed a vulnerability in ZEO storage servers when blobs are available. Someone with write access to a ZEO server configured to support blobs could read any file on the system readable by the server process and remove any file removable by the server process.
Fixed NameError in cases where a directory cannot be created, e.g. when the necessary permissions are missing.
Fixed a pack test that was not compatible with storages that always return an object count of 0.
Calling __setstate__ on a persistent object could under certain uncommon cause the process to crash.
Whats new in ZODB 3.8.2
Bugs Fixed:
Fixed vulnerabilities in the ZEO network protocol that allow:
CVE-2009-0668 Arbitrary Python code execution in ZODB ZEO storage servers
CVE-2009-0669 Authentication bypass in ZODB ZEO storage servers
The vulnerabilities only apply if you are using ZEO to share a database among multiple applications or application instances and if untrusted clients are able to connect to your ZEO servers.
Limit the number of object ids that can be allocated at once to avoid running out of memory.
Whats new in ZODB 3.8.1
Bugs Fixed:
(3.8.1) Reset _cache on a connection’s _reader object when resetting the cache, to prevent reads from the old cache object, e.g. during Zope2’s auto-refresh of products. (https://bugs.launchpad.net/zodb/+bug/142667).
(3.8.1) An exception would be raised when an error occured attempting to lock a file and logging of said error was enabled.
(3.8.1) Fixed a bug to allow opening of deep-copied blobs.
(3.8.1) Fixed bug #189542 by prepending the module to an undefined name.
(3.8.1) Failures in tpc_finish of client-storages weren’t handled correctly, leaving the client storage in an inconsistent state.
(3.8.1) If there is a failure while FileStorage is finalizing a transaction, the file storage is closed because it’s internal meta data may be invalid.
(3.8.1) FileStorages previously saved indexes after a certain number of writes. This was done during the last phase of two-phase commit, which made this critical phase more subject to errors than it should have been. Also, for large databases, saves were done so infrequently as to be useless. The feature was removed to reduce the chance for errors during the last phase of two-phase commit.
(3.8.1) File storages previously kept an internal object id to transaction id mapping as an optimization. This mapping caused excessive memory usage and failures during the last phase of two-phase commit. This optimization has been removed.
(3.8.1) Fixed a bug that caused deep copying of blobs to fail.
(3.8.1) Refactored handling of invalidations on ZEO clients to fix a possible ordering problem for invalidation messages.
(3.8.1) An ZEO cache internal data structure can get out of sync with the data in a way that prevents data from being loaded into the cache. We don’t yet know why, but added an exception handler to prevent this error from being fatal.
(3.8.1) Fixed setup.py use of setuptools vs distutils, so .c and .h files are included in the bdist_egg.
(3.8.1) On many systems, it was impossible to create more than 32K blobs. Added a new blob-directory layout to work around this limitation.
(3.8.1) Fixed a bug, introduced in an earlier beta, that allowed clients to connect to out of date servers.
(3.8.1) Fixed bug that could lead to memory errors due to the use of a Python dictionary for a mapping that can grow large.
(3.8.1) Fixed bug #251037: Made packing of blob storages non-blocking.
(3.8.1) Fixed a bug that could cause InvalidObjectReference errors for objects that were explicitly added to a database if the object was modified after a savepoint that added the object.
(3.8.1) Fixed several bugs that caused ZEO cache corruption when connecting to servers. These bugs affected both persistent and non-persistent caches.
(3.8.1) Improved the the ZEO client shutdown support to try to avoid spurious errors on exit, especially for scripts, such as zeopack.
(3.8.1) Packing failed for databases containing cross-database references.
(3.8.1) Cross-database references to databases with empty names weren’t constructed properly.
(3.8.1) The cache used an excessive amount of memory, causing applications with large caches to exhaust available memory.
Fixed a number of bugs in the handling of persistent ZEO caches:
Cache records are written in several steps. If a process exits after writing begins and before it is finishes, the cache will be corrupt on restart. The way records are writted was changed to make cache record updates atomic.
There was no lock file to prevent opening a cache multiple times at once, which would lead to corruption. Persistent caches now use lock files, in the same way that file storages do.
A bug in the cache-opening logic led to cache failure in the unlikely event that a cache has no free blocks.
When using ZEO Client Storages, Errors occured when trying to store objects too big to fit in the ZEO cache file.
Fixed bug in blob filesystem helper: the isSecure check was inversed.
Fixed bug in transaction buffer: a tuple was unpacked incorrectly in clear.
Fixed bug in Connection.TmpStore: load() would not defer to the backend storage for loading blobs.
Fixed bug #190884: Wrong reference to POSKeyError caused NameError.
Completed implementation of ZEO authentication. This fixes issue 220856.
What’s new on ZODB 3.8.0
General
The ZODB Storage APIs have been documented and cleaned up.
ZODB versions are now officially deprecated and support for them will be removed in ZODB 3.9. (They have been widely recognized as deprecated for quite a while.)
Changed the automatic garbage collection when opening a connection to only apply the garbage collections on those connections in the pool that are closed. (This fixed issue 113923.)
(3.8.0b3) Document conflict resolution (see ZODB/ConflictResolution.txt).
(3.8.0b3) Bugfix the situation in which comparing persistent objects (for instance, as members in BTree set or keys of BTree) might cause data inconsistency during conflict resolution.
(3.8.0b3) Support multidatabase references in conflict resolution.
(3.8.0b3) Make it possible to examine oid and (in some situations) database name of persistent object references during conflict resolution.
(3.8.0b3) Added missing data attribute for conflict errors.
(3.8.0b5) Fixed bug 153316: persistent and BTrees gave errors on x86_64 Intel XEON platforms.
ZEO
(3.8.0b6) Bug #98275: Made ZEO cache more tolerant when invalidating current versions of objects.
(3.8.0b4, 3.8.0b5) Fixed a serious bug that could cause client I/O to stop (hang). This was accomonied by a critical log message along the lines of: “RuntimeError: dictionary changed size during iteration”. (In b4, the bug was only partially fixed.)
(3.8a1) ZEO’s strategoes for avoiding client cache verification were improved in the case that servers are restarted. Before, if transactions were committed after the restart, clients that were up to date or nearly up to date at the time of the restart and then connected had to verify their caches. Now, it is far more likely that a client that reconnects soon after a server restart won’t have to verify its cache.
(3.8a1) Fixed a serious bug that could cause clients that disconnect from and reconnect to a server to get bad invalidation data if the server serves multiple storages with active writes.
(3.8a1) It is now theoretically possible to use a ClientStorage in a storage server. This might make it possible to offload read load from a storage server at the cost of increasing write latency. This should increase write throughput by offloading reads from the final storage server. This feature is somewhat experimental. It has tests, but hasn’t been used in production.
Transactions
(3.8a1) Add a doom() and isDoomed() interface to the transaction module.
First step towards the resolution of http://www.zope.org/Collectors/Zope3-dev/655
A doomed transaction behaves exactly the same way as an active transaction but raises an error on any attempt to commit it, thus forcing an abort.
Doom is useful in places where abort is unsafe and an exception cannot be raised. This occurs when the programmer wants the code following the doom to run but not commit. It is unsafe to abort in these circumstances as a following get() may implicitly open a new transaction.
Any attempt to commit a doomed transaction will raise a DoomedTransaction exception.
(3.8a1) Clean up the ZODB imports in transaction.
Clean up weird import dance with ZODB. This is unnecessary since the transaction module stopped being imported in ZODB/__init__.py in rev 39622.
(3.8a1) Support for subtransactions has been removed in favor of save points.
Blobs
(3.8b5) Fixed bug #130459: Packing was broken by uncommitted blob data.
(3.8b4) Fixed bug #127182: Blobs were subclassable which was not desired.
(3.8b3) Fixed bug #126007: tpc_abort had untested code path that was broken.
(3.8b3) Fixed bug #129921: getSize() function in BlobStorage could not deal with garbage files
(3.8b1) Updated the Blob implementation in a number of ways. Some of these are backward incompatible with 3.8a1:
o The Blob class now lives in ZODB.blob
o The blob openDetached method has been replaced by the committed method.
(3.8a1) Added new blob feature. See the ZODB/Blobs directory for documentation.
ZODB now handles (reasonably) large binary objects efficiently. Useful to use from a few kilobytes to at least multiple hundred megabytes.
BTrees
(3.8a1) Added support for 64-bit integer BTrees as separate types.
(For now, we’re retaining compile-time support for making the regular integer BTrees 64-bit.)
(3.8a1) Normalize names in modules so that BTrees, Buckets, Sets, and TreeSets can all be accessed with those names in the modules (e.g., BTrees.IOBTree.BTree). This is in addition to the older names (e.g., BTrees.IOBTree.IOBTree). This allows easier drop-in replacement, which can especially be simplify code for packages that want to support both 32-bit and 64-bit BTrees.
(3.8a1) Describe the interfaces for each module and actually declare the interfaces for each.
(3.8a1) Fix module references so klass.__module__ points to the Python wrapper module, not the C extension.
(3.8a1) introduce module families, to group all 32-bit and all 64-bit modules.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
