zope.sqlalchemy 1.1

Minimal Zope/SQLAlchemy transaction integration

• Introduction

• Running the tests

• Example

  • Long-lasting session scopes

• Registration Using SQLAlchemy Events

• Development version

• Changes

  • 1.1 (2019-01-03)

  • 1.0 (2018-01-31)

  • 0.7.7 (2016-06-23)

  • 0.7.6 (2015-03-20)

  • 0.7.5 (2014-06-17)

  • 0.7.4 (2014-01-06)

  • 0.7.3 (2013-09-25)

  • 0.7.2 (2013-02-19)

  • 0.7.1 (2012-05-19)

  • 0.7 (2011-12-06)

  • 0.6.1 (2011-01-08)

  • 0.6 (2010-07-24)

  • 0.5 (2010-06-07)

  • 0.4 (2009-01-20)

  • 0.3 (2008-07-29)

  • 0.2 (2008-06-28)

  • 0.1 (2008-05-15)

Introduction

The aim of this package is to unify the plethora of existing packages integrating SQLAlchemy with Zope’s transaction management. As such it seeks only to provide a data manager and makes no attempt to define a zopeish way to configure engines.

For WSGI applications, Zope style automatic transaction management is available with repoze.tm2 (used by Turbogears 2 and other systems).

This package is also used by pyramid_tm (an add-on of the Pyramid) web framework.

You need to understand SQLAlchemy and the Zope transaction manager for this package and this README to make any sense.

Running the tests

This package is distributed as a buildout. Using your desired python run:

$ python bootstrap.py $ ./bin/buildout

This will download the dependent packages and setup the test script, which may be run with:

$ ./bin/test

or with the standard setuptools test command:

$ ./bin/py setup.py test

To enable testing with your own database set the TEST_DSN environment variable to your sqlalchemy database dsn. Two-phase commit behaviour may be tested by setting the TEST_TWOPHASE variable to a non empty string. e.g:

$ TEST_DSN=postgres://test:test@localhost/test TEST_TWOPHASE=True bin/test

Example

This example is lifted directly from the SQLAlchemy declarative documentation. First the necessary imports.

>>> from sqlalchemy import *
>>> from sqlalchemy.ext.declarative import declarative_base
>>> from sqlalchemy.orm import scoped_session, sessionmaker, relation
>>> from zope.sqlalchemy import ZopeTransactionExtension
>>> import transaction

Now to define the mapper classes.

>>> Base = declarative_base()
>>> class User(Base):
...     __tablename__ = 'test_users'
...     id = Column('id', Integer, primary_key=True)
...     name = Column('name', String(50))
...     addresses = relation("Address", backref="user")
>>> class Address(Base):
...     __tablename__ = 'test_addresses'
...     id = Column('id', Integer, primary_key=True)
...     email = Column('email', String(50))
...     user_id = Column('user_id', Integer, ForeignKey('test_users.id'))

Create an engine and setup the tables. Note that for this example to work a recent version of sqlite/pysqlite is required. 3.4.0 seems to be sufficient.

>>> engine = create_engine(TEST_DSN, convert_unicode=True)
>>> Base.metadata.create_all(engine)

Now to create the session itself. As zope is a threaded web server we must use scoped sessions. Zope and SQLAlchemy sessions are tied together by using the ZopeTransactionExtension from this package.

>>> Session = scoped_session(sessionmaker(bind=engine,
... twophase=TEST_TWOPHASE, extension=ZopeTransactionExtension()))

Call the scoped session factory to retrieve a session. You may call this as many times as you like within a transaction and you will always retrieve the same session. At present there are no users in the database.

>>> session = Session()
>>> session.query(User).all()
[]

We can now create a new user and commit the changes using Zope’s transaction machinery, just as Zope’s publisher would.

>>> session.add(User(id=1, name='bob'))
>>> transaction.commit()

Engine level connections are outside the scope of the transaction integration.

>>> engine.connect().execute('SELECT * FROM test_users').fetchall()
[(1, ...'bob')]

A new transaction requires a new session. Let’s add an address.

>>> session = Session()
>>> bob = session.query(User).all()[0]
>>> str(bob.name)
'bob'
>>> bob.addresses
[]
>>> bob.addresses.append(Address(id=1, email='[e-mail]'))
>>> transaction.commit()
>>> session = Session()
>>> bob = session.query(User).all()[0]
>>> bob.addresses
[<Address object at ...>]
>>> str(bob.addresses[0].email)
'[e-mail]'
>>> bob.addresses[0].email = 'wrong@wrong'

To rollback a transaction, use transaction.abort().

>>> transaction.abort()
>>> session = Session()
>>> bob = session.query(User).all()[0]
>>> str(bob.addresses[0].email)
'[e-mail]'
>>> transaction.abort()

By default, zope.sqlalchemy puts sessions in an ‘active’ state when they are first used. ORM write operations automatically move the session into a ‘changed’ state. This avoids unnecessary database commits. Sometimes it is necessary to interact with the database directly through SQL. It is not possible to guess whether such an operation is a read or a write. Therefore we must manually mark the session as changed when manual SQL statements write to the DB.

>>> session = Session()
>>> conn = session.connection()
>>> users = Base.metadata.tables['test_users']
>>> conn.execute(users.update(users.c.name=='bob'), name='ben')
<sqlalchemy.engine...ResultProxy object at ...>
>>> from zope.sqlalchemy import mark_changed
>>> mark_changed(session)
>>> transaction.commit()
>>> session = Session()
>>> str(session.query(User).all()[0].name)
'ben'
>>> transaction.abort()

If this is a problem you may tell the extension to place the session in the ‘changed’ state initially.

>>> Session.remove()
>>> Session.configure(extension=ZopeTransactionExtension('changed'))
>>> session = Session()
>>> conn = session.connection()
>>> conn.execute(users.update(users.c.name=='ben'), name='bob')
<sqlalchemy.engine...ResultProxy object at ...>
>>> transaction.commit()
>>> session = Session()
>>> str(session.query(User).all()[0].name)
'bob'
>>> transaction.abort()

Long-lasting session scopes

The default behaviour of the transaction integration is to close the session after a commit. You can tell by trying to access an object after committing:

>>> bob = session.query(User).all()[0]
>>> transaction.commit()
>>> bob.name
Traceback (most recent call last):
DetachedInstanceError: Instance <User at ...> is not bound to a Session; attribute refresh operation cannot proceed...

To support cases where a session needs to last longer than a transaction (useful in test suites) you can specify to keep a session when creating the transaction extension:

>>> Session = scoped_session(sessionmaker(bind=engine,
... twophase=TEST_TWOPHASE, extension=ZopeTransactionExtension(keep_session=True)))

>>> session = Session()
>>> bob = session.query(User).all()[0]
>>> bob.name = 'bobby'
>>> transaction.commit()
>>> bob.name
u'bobby'

The session must then be closed manually:

>>> session.close()

Registration Using SQLAlchemy Events

The zope.sqlalchemy.register() function performs the same function as the ZopeTransactionExtension, except makes use of the newer SQLAlchemy event system which superseded the extension system as of SQLAlchemy 0.7. Usage is similar:

>>> from zope.sqlalchemy import register
>>> Session = scoped_session(sessionmaker(bind=engine,
... twophase=TEST_TWOPHASE))
>>> register(Session, keep_session=True)
>>> session = Session()
>>> jack = User(id=2, name='jack')
>>> session.add(jack)
>>> transaction.commit()
>>> engine.execute("select name from test_users where id=2").scalar()
u'jack'

Development version

GIT version

Changes

1.1 (2019-01-03)

• Add support to MySQL using pymysql.

1.0 (2018-01-31)

• Add support for Python 3.4 up to 3.6.

• Support SQLAlchemy 1.2.

• Drop support for Python 2.6, 3.2 and 3.3.

• Drop support for transaction < 1.6.0.

• Fix hazard that could cause SQLAlchemy session not to be committed when transaction is committed in rare situations. See https://github.com/zopefoundation/zope.sqlalchemy/pull/23

0.7.7 (2016-06-23)

• Support SQLAlchemy 1.1.

  See: https://github.com/zopefoundation/zope.sqlalchemy/issues/15

0.7.6 (2015-03-20)

• Make version check in register compatible with prereleases.

0.7.5 (2014-06-17)

• Ensure mapped objects are expired following a transaction.commit() when no database commit was required.

  See: https://github.com/zopefoundation/zope.sqlalchemy/issues/8

0.7.4 (2014-01-06)

• Allow session.commit() on nested transactions to facilitate integration of existing code that might not use transaction.savepoint().

  See: https://github.com/zopefoundation/zope.sqlalchemy/issues/1

• Add a new function zope.sqlalchemy.register(), which replaces the direct use of ZopeTransactionExtension to make use of the newer SQLAlchemy event system to establish instrumentation on the given Session instance/class/factory. Requires at least SQLAlchemy 0.7.

  See: https://github.com/zopefoundation/zope.sqlalchemy/issues/4

• Fix keep_session=True doesn’t work when a transaction is joined by flush and other manngers bug.

  See: https://github.com/zopefoundation/zope.sqlalchemy/issues/5

0.7.3 (2013-09-25)

• Prevent the Session object from getting into a “wedged” state if joining a transaction fails. With thread scoped sessions that are reused this can cause persistent errors requiring a server restart.

  See: https://github.com/zopefoundation/zope.sqlalchemy/issues/2

0.7.2 (2013-02-19)

• Make life-time of sessions configurable. Specify keep_session=True when setting up the SA extension.

• Python 3.3 compatibility.

0.7.1 (2012-05-19)

• Use @implementer as a class decorator instead of implements() at class scope for compatibility with zope.interface 4.0. This requires zope.interface >= 3.6.0.

0.7 (2011-12-06)

• Python 3.2 compatibility.

0.6.1 (2011-01-08)

• Update datamanager.mark_changed to handle sessions which have not yet logged a (ORM) query.

0.6 (2010-07-24)

• Implement should_retry for sqlalchemy.orm.exc.ConcurrentModificationError and serialization errors from PostgreSQL and Oracle. (Specify transaction>=1.1 to use this functionality.)

• Include license files.

• Add transaction_manager attribute to data managers for compliance with IDataManager interface.

0.5 (2010-06-07)

• Remove redundant session.flush() / session.clear() on savepoint operations. These were only needed with SQLAlchemy 0.4.x.

• SQLAlchemy 0.6.x support. Require SQLAlchemy >= 0.5.1.

• Add support for running python setup.py test.

• Pull in pysqlite explicitly as a test dependency.

• Setup sqlalchemy mappers in test setup and clear them in tear down. This makes the tests more robust and clears up the global state after. It caused the tests to fail when other tests in the same run called clear_mappers.

0.4 (2009-01-20)

Bugs fixed:

• Only raise errors in tpc_abort if we have committed.

• Remove the session id from the SESSION_STATE just before we de-reference the session (i.e. all work is already successfuly completed). This fixes cases where the transaction commit failed but SESSION_STATE was already cleared. In those cases, the transaction was wedeged as abort would always error. This happened on PostgreSQL where invalid SQL was used and the error caught.

• Call session.flush() unconditionally in tpc_begin.

• Change error message on session.commit() to be friendlier to non zope users.

Feature changes:

• Support for bulk update and delete with SQLAlchemy 0.5.1

0.3 (2008-07-29)

Bugs fixed:

• New objects added to a session did not cause a transaction join, so were not committed at the end of the transaction unless the database was accessed. SQLAlchemy 0.4.7 or 0.5beta3 now required.

Feature changes:

• For correctness and consistency with ZODB, renamed the function ‘invalidate’ to ‘mark_changed’ and the status ‘invalidated’ to ‘changed’.

0.2 (2008-06-28)

Feature changes:

• Updated to support SQLAlchemy 0.5. (0.4.6 is still supported).

0.1 (2008-05-15)

• Initial public release.