MOAI, A Open Access Server Platform for Institutional Repositories

## MOAI, an Open Access Server Platform for Institutional Repositories

MOAI is a platform for aggregating content from different sources, and publishing it through the Open Archive Initiatives protocol for metadata harvesting. It’s been built for academic institutional repositories dealing with relational metadata and asset files.

### What does it do?

The MOAI software can aggregate content from different sources, transform it and store it in a database. The contents of this database can then be published in many separate OAI feeds, each with its own configuration.

The MOAI software has a very flexible system for combining records into sets, and can use these sets in the feed configuration. It also comes with a simple yet flexible authentication scheme that can be easily customized. Besides providing authentication for the feeds, the authentication also controls the access to the assets.

### Why MOAI

MOAI has been specifically developed for universities, and contains a lot of hard-earned wisdom. The software has been in production use since 2007, and new features have been continually added. In late 2008, the software was completely refactored and packaged under the name “MOAI”. You can read more about this on the MOAI History page.

MOAI is a standalone system, so it can be used in combination with any repository software that comes with an OAI feed such as Fedora Commons, EPrints or DSpace. It can also be used directly with an SQL database or just a folder of XML files.

The MOAI project takes the philosophy that every repository is different and unique, and that an institutional repository is a living thing. It is therefore never finished. Metadata is always changes, improving, and evolves. We think this is healthy.

Because of this viewpoint, the MOAI software makes it as easy as possible to add or modify parts of your repository (OAI) services stack. It tries to do this without sacrificing power, and encouraging the re-use of components.

### Features

MOAI has some interesting features not found in most OAI servers. Besides serving OAI, it can also harvest OAI. This makes it possible for MOAI to work as a pipe, where the OAI data can be reconfigured, cached, and enriched while it passes through the MOAI processing.

More specifically MOAI has the ability to:

• Harvest data from different kinds of sources

• Serve many OAI feeds from one MOAI server, each with their own configuration

• Turn metadata values into OAI sets on the fly, creating new collections

• Use OAI sets to filter records shown in a feed, configurable for each feed

• Work easily with relational data (e.g. if an author changes, the publication should also change)

• Simple and robust authentication through integration with the Apache webserver

• Serve assets via Apache while still using configurable authentication rules

## Installing MOAI

The MOAI Software can be run in any wsgi compliant server.

MOAI comes with a development server that can be used for testing. In production mod_wsgi can be used to run Moai in the apache webserver.

## Installation Steps

MOAI is a normal python package. It is tested with python2.5 and 2.6. I recommend creating a virtualenv to install the package in.

http://pypi.python.org/pypi/virtualenv/

This makes development and deployment easier. Instructions below are for unix, but MOAI should also work on Windows

Go into the MOAI directory with the setup.py, and run the virtualenv command

> cd moai > virtualenv .

Now, activate the virtualenv

> source bin/activate

Install MOAI in the virtualenv using pip

> pip install -e .

(this will take a while)

When this process finishes, Moai and all its dependencies will be installed.

## Running in development mode

The development server should never be used for production, it is convenient for testing and development. Note that you should always activate the virtualenv otherwise the dependecies will not be found

> cd moai > source bin/activate > ./bin/paster serve settings.ini

This will print something like:

Starting server in PID 7306. Starting HTTP server on http://127.0.0.1:8080

You can now visit localhost:8080/oai to view the moai oaipmh feed.

## Configuring MOAI

Configuration is done in the settings.ini file. The default settings file uses the Paste#urlmap application to map wsgi applications to a url.

In the composite:main section there is a line:

/oai = moai_example

Which maps the /oai url to a Moai instance. This makes it easy to run many Maoi instances in one server, each with it’s own configuration.

The app:moai_example configuration let’s you specify the following options:

name

The name of the oai feed (returned in Identify verb)

url

The url of the oai feed (returned in oaipmh xml output)

The email adress of the amdin (returned in Identify verb)

formats

disallow_sets

List of setspecs that are not allowed in the output of this feed

allow_sets

If used, only sets listed here will be returned

database

SQLAlchemy uri to identify the database for used for storage

provider

Provider identifier where moai retrieves content from

content

Class that maps metadata from provider format to moai format

The Moai system is designed to periodically fetch content from a provider, and convert it to Moai’s internal format, which can then be translated to the different metadata formats for the oaipmh feed.

Moai comes with an example that shows this principle:

In the moai/moai directory there are two XML files. Let’s pretend these files are from a remote system, and we want to publish them with MOAI.

In the settings.ini file, the following option is specified:

provider = file://moai/example-*.xml

This tells moai that we want to use a file provider, with some files located in moai/example-*.xml.

The following option points to the class that we want to use for converting the example content xml data to Moais internal format.

content = moai_example

The last option tells Moai where to store it’s data, this is usually a sqlite database:

database = sqlite:///moai-example.db

Now let’s try to add these two xml files, let’s first visit the oaipmh feed to make sure nothing is allready being served:

This should return a noRecordsMatch error.

To add the content, run the update_content script, with the section name from the settings.ini as argument

> ./bin/update_moai moai_example

This will produce the following output:

/ Updating content provider: example-2345.xml Content provider returned 2 new/modified objects

100.0%[====================================================================>] 2 Updating database with 2 objects took 0 seconds

Now when you visit the oaipmh feed again you should see the two records:

When you run the update_moai script again, it will create a new database with all the records (in this case moai_example.db). It is also possible to specify a data with the –date switch. When a data is specified, only records that were modified after this date will be added. The update_moai script can be run from a daily or hourly cron job to update the database

It’s possible and most of the time, needed, to extend Moai for your use-cases. The Provider and Content classes from the example might be a good starting point if you want to do that. All your customizations should be registered with Moai through entry_points. Have a look at Moais setup.py for more information. The best approach would be to create your own python package with setup.py and install this in the same environment as Moai. This will let Moai find your customizations. Note that when you change something in your setup.py, you have to reinstall the package, for Moai to pick up the changes.

Note that the moai.interfaces file contains documentation about the different classes that you can implement.

Instead of writing your own provider/content classes, you can also register your own custom database. Implementing a replacement for moai.database.SQLDatabase can be more complicated then writing a provider/content class, but it has the advantage that Moai is always up to date, and you don’t need a second sqlite database.

Have a look at the setup.py file from the MOAI code, it registers several databases. You could use this mechanism to register your own database from your own python package.

In the settings.ini you configuration you can then reference your database (‘mydb://some+config+variables’).

For the database, have a look at the generic database provider in database.py. The only methods that you need to implement are: oai_sets, oai_earliest_datestamp and oai_query. The oai_query method returns dictionaries with record data. The keys of these dictionaries are defined in the metadata files (for example metadata.py), have a look at the source.

For oai_dc there are the following names:

‘title’, ‘creator’, ‘subject’, ‘description’, ‘publisher’, ‘contributor’, ‘type’, ‘format’, ‘identifier’, ‘source’, ‘language’, ‘date’, ‘relation’, ‘coverage’, ‘rights

So a return value would look like:

{'id': <oai record id>,
'deleted': <bool>,
'modified': <utc datetime>,
'sets': <list of setspecs>,
'title': [<list with publication title>],
'creator': [<list of creator names>],
...}
}

## MOAI Changes

### MOAI 2.0.0 (2013-02-28)

• First public release in the 2.x series, switched to mod_wsgi, smaller simpler codebase, pluggability through entry_points instead of ZCA.

• New docs and examples, simpler direct integration with external databases

• Update mods format and support for EU GAL metadata MODS extension

• Several documentation and tool improvements by Filipe Correia

### MOAI 1.1.2 (2010-09-30)

• Changed contact info, and switched repository from svn to mercurial this release has no code changes

### MOAI 1.1.1 (2010-09-02)

Changes

• Fixed ID/IDref attributes in MODS format

• Fixed ZeroDivisionError in progressbar drawing of updatedb script

### MOAI 1.1.0 (2010-04-08)

#### Changes

• Added new config option deleted_sets which can be used to make all resources within a specific set oai deleted within the context of a specific feed. In many cases this is preferable compared to disallowed_sets since resources will just dissapear when they are not allowed without leaving a proper deleted resource behind.

• Disable ranged HTTP requests in Apache mod_python backend. This was giving problems with acrobat plugin and its “Fast Web View” feature.

• Show records sorted on descending datestamp, instead of ascending record id, in sqlite backend.

### MOAI 1.0.10 (2010-03-25)

Changes

• Changed DIP:objectType to rdf:type in the didl metadata format, as required by darenet/driver/didl 3.0 format

• Added support for modification dates and accessrights in the didl format

### MOAI 1.0.9 (2010-02-16)

Changes

• Added checking on XML compatibility in update.py, update_database_iterate(). This prevents the OAI server from crashing on invalid data.

• Fixed a minor bug in tools.py. The -d (debug) flag for the update_database script didn’t halt on errors.

• Added a flush_threshold attribute to the DatabaseUpdater. This can be used to make the database flush after n records. By default the value is -1, and flushing only happens at the end of the update.

• Fixed assertion statements in content.py (gave warnings in python2.6)

### MOAI 1.0.8 (2009-16-18)

Changes

• Added a MANIFEST.in file to make sure all files are included in the python package.

### MOAI 1.0.7 (2009-16-11)

Changes

• Changed DIP namespace in didl metadata extension

• Fixed bug in ListMetadataFormats verb

• Uses pyoai 2.4.1, which when specifying a date (not a datetime) for the until parameter, defaults to 23:59:59 instead of 00:00:00.

### MOAI 1.0.6 (2009-08-12)

Changes

• Fixed metadataPrefix value in didl:Resource URL. This was using didl instead of nl_didl

### MOAI 1.0.5 (2009-08-12)

• Unicode bug in Fedora provider

• Added support for mods:location/url Make url prefered identifier in oai_dc:identifier instead of uri

• Update method of Content Objects can now return False to prevent further processing of the object. The object will be logged as ignored

### MOAI 1.0.4 (2009-07-15)

Changes

• Fixed misspelled mimeType attribute in DIDL metadata prefix

### MOAI 1.0.3 (2009-06-23)

Changes

• Bug in ListMetadataFormats when using nl_didl prefix

• Added simple dc example profile

### MOAI 1.0.2 (2009-05-20)

Changes

• Location in mod_python config is dynamicly extracted from Metro Server class.

### MOAI 1.0.1 (2009-05-19)

Changes

• Fixed URL error in didl metadata prefix

• Seperated darenet specific didl extensions in nl_didl prefix

### MOAI 1.0 (2009-05-04)

Changes

• Improved documentation

• More complete mods name roles

• Require newer pyoai version that gives better error responses when wrong datatime values are used in OAI requests

### MOAI 0.9.x (2009-02-01)

#### Changes

• Added basic authentication for fedora provider

• Made fedora provider datastream argument optional, if not specified the whole foxml file is returned

• Ported SQLite database backend from EUR RePub

• Made Fedora provider not write all files in same directory

#### Changes

• OAI ListRecords with from and until dateparams where not inclusive in SQLite engine

• OAI ListSets did not return dynamic sets (added test)

• OAI GetRecord returning the same record (added test)

### MOAI 0.9 (2008-11-18)

• Initial development borrowing code and ideas from RepubXML, a repository environment build for the Erasmus Univiersity Rotterdam

## Project details

Uploaded source