django-phpBB3 0.2.1

django database models of phpBB3

Django-phpBB3

Django-phpBB3 provides the database models of a existing phpBB3 installation. This is usefull for all people who like to access the phpbb forum database from a django application. It’s not a phpBB clone or a forum!

Django-phpBB3 has also a script for migrate a phpBB3 forum into a DjangoBB forum. See below.

Notes:

• There are a few mismatches between the origin phpBB3 Database schema and the models.py form Django-phpBB3. Because of this: Creating the phpBB3 tables via syncdb will not work with phpBB3.

  • phpBB3 used often 0 for a not set value, but we need a real null value in db (to get None back)

• phpBB3 Models with composite keys are unsupported, but stored in django_phpBB3/unsupported_models.py

Warning: Write access to the original phpBB database is not tested, yet. So it’s not guaranteed that changes trough the django admin will work with phpBB3! So, in the current state, you should only access ‘read-only’ to the origin phpBB3 data ;)

Patches are welcome! Please send pull requests.

example projects

We add two example projects:

`django_phpBB3_project`_	minimal project without extra depencies
`phpBB2DjangoBB_project`_	DjangoBB example project for use the phpBB2DjangoBB migration command

setup

~$ virtualenv DjangoBB_env
~$ cd DjangoBB_env/
~/DjangoBB_env$ source bin/activate
(DjangoBB_env)~/DjangoBB_env$ pip install django-phpBB3 # Will also install Django

(If default python interpreter is <2.7: Use: virtualenv -p python2.7 DjangoBB_env - see below)

Or if you will use the most recent git version:

# install git read-only clone (Will also install Django):
(DjangoBB_env)~/DjangoBB_env$ pip install -e git+git://github.com/jedie/django-phpBB3.git#egg=django-phpBB3

If you would like to test/use the DjangoBB migration install this:

(DjangoBB_env)~/DjangoBB_env$ pip install -e git+git://github.com/slav0nic/DjangoBB.git#egg=DjangoBB
(DjangoBB_env)~/DjangoBB_env$ pip install Whoosh

Maybe you want my dev branch of DjangoBB (More info in My DjangoBB github wiki page):

(DjangoBB_env)~/DjangoBB_env$ pip install -e git+git://github.com/jedie/DjangoBB.git@dev#egg=DjangoBB
(DjangoBB_env)~/DjangoBB_env$ pip install Whoosh

local_settings.py example

# database table prefix (default is: "phpbb3_")
#PHPBB_TABLE_PREFIX = "phpbb_"

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
        'NAME': 'phpbb3', # Or path to database file if using sqlite3.
        'USER': 'phpbb3', # Not used with sqlite3.
        'PASSWORD': 'foo bar', # Not used with sqlite3.
        'HOST': '', # Set to empty string for localhost. Not used with sqlite3.
        'PORT': '', # Set to empty string for default. Not used with sqlite3.
    }
}

# settings for phpBB3 migration:

# filesystem path to the /files/ sub directory of the phpBB installation:
# Set to None, if your phpBB3 forum has no attachments
PHPBB_ATTACHMENT_PATH = "/path/to/phpBB/files/"

init database

If you will test the two example projects, you need to initialises the database, e.g.:

~$ cd DjangoBB_env/
~/DjangoBB_env$ source bin/activate
(DjangoBB_env)~/DjangoBB_env$ cd src/django-phpbb3/phpBB2DjangoBB_project
(DjangoBB_env).../phpBB2DjangoBB_project$ # create local_settings.py here, see above!
(DjangoBB_env).../phpBB2DjangoBB_project$ ./manage.py syncdb --all
(DjangoBB_env).../phpBB2DjangoBB_project$ ./manage.py migrate --fake

test django-phpBB3

We added a test project for easy start in: /django_phpBB3_project/

Create a local_settings.py into /django_phpBB3_project/ see example above.

You must initialises the database (see above).

You can startup a test server with, e.g:

./manage.py runserver --traceback --insecure

migrate a phpBB3 installation to DjangoBB

We add a example project for a phpBB3 migration in: /phpBB2DjangoBB_project/

The migration from phpBB3 to DjangoBB is implement as a migration command here: /django_phpBB3/management/commands/phpbb2djangobb.py

To run the migration you can use the minimal phpBB2DjangoBB example project example. This project includes the DjangoBB forum app and django-phpBB3 app. Before start, you must install the dependencies!

You must create a local_settings.py into /phpBB2DjangoBB_project/ see example above.

Before you start migration: You should check if database encoding settings are ok. To check, login and check in admin / Django_phpBB3 / **Posts** if non-ascii characters are ok.

Typical problem are: e.g. in default ubuntu/debian setup a mysql server will use latin-1 ! So before you start, you should correct this. e.g.:

ALTER DATABASE test DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci

Make a SQL Backup before doing anything!!!

Run the migration with ./manage.py phpbb2djangobb, e.g:

~$ cd DjangoBB_env/
~/DjangoBB_env$ source bin/activate
(DjangoBB_env)~/DjangoBB_env$ cd src/django-phpbb3/phpBB2DjangoBB_project
(DjangoBB_env).../phpBB2DjangoBB_project$ ./manage.py phpbb2djangobb

after migration

Important: No phpBB forum permissions would be migrated, yet. So: Before you make a migrated Forum available to public, you must manually hide any internal forums, if exists! To limit a forum in DjangoBB, e.g.:

• Create a new user group in django admin panel

• Create a seperate djangobb category and add the creates user group

• Move internal forums to the new category

• To give a user the permissions to the internal forums: Edit the user and add the created user group.

You must update the haystack search index with e.g.:

(DjangoBB_env).../phpBB2DjangoBB_project$ ./manage.py update_index --verbosity=2

more information: http://django-haystack.readthedocs.org/en/latest/management_commands.html

Depend on the used database you must reset sequences.

Sequences are indexes used by some database engines to track the next available number for automatically incremented fields.

more info: https://docs.djangoproject.com/en/dev/ref/django-admin/#sqlsequencereset-appname-appname

options

cleanup_users

Add --cleanup_users=x to setup which users are migrate, where x is a number:

0	all users
1	users with email addresses (skip bots)
2	users with email and has a ‘lastvisit’ date (default)
3	email + lastvisit and has created at least one post

migration limitations

Things that would not be transfered, yet:

• Forum moderators

• Category groups

• sort of Category / Forum would not be converted to position number

Handling missing features in DjangoBB:

• We transfer phpBB Users who are in the user group ADMINISTRATORS or GLOBAL_MODERATORS to all forum moderators.

• global posts and announcement converted to normal sticky posts

• sub forums would be flatten

• ghost moved posts are ignored (only the moved post would be created)

• ForumWatch are ignored

• All user must do a passwort reset. (We can’t reuse the phpBB password hash)

• Guest posts would be mapped to a new “Anonymous” user (has unusable password and no email address). Note: guest posts are planed in DjangoBB, see: http://support.djangobb.org/topic/98/

TODO:

• Handle forum permissions: Now all forums will be public, after migration!

• Bugfix for user post count migration - https://github.com/jedie/django-phpBB3/issues/6

• migrate user avatar - https://github.com/jedie/django-phpBB3/issues/7

• add a separate command for migrate phpBB private messages to django-messages (very low priority, patches are welcome)

migration - notes

The migration script copied the phpBB username 1:1 and doesn’t cleanup anything. The django forms for creating/change a user will only limit with: Letters, digits and @/./+/-/_ only But the model fields is a simple CharField. A nick will be accepted in login depend on the login form you use in your project.

migration - open questions

• Are the phpBB times in phpbb_posts in UTC oder in local time from user?

troubleshooting

HTMLParseError while post migration

Maybe you used a older Python Version than 2.7.3 In older versions the HTMLParser is not so robust as in 2.7.3 Maybe default Python interpreter is 2.6, but 2.7 is installed to, than try:

# Display the exsiting Python version:
~$ python --version # Default python
Python 2.6.6
~$ python2.7 --version # Try if 2.7 is available
Python 2.7.3

# Create env with 2.7:
~$ virtualenv -p python2.7 DjangoBB_env

Work-a-round if you can’t create a virtualenv with a newer Python interpreter: set: settings.DJANGOBB_SMILES_SUPPORT = False More Information here: https://github.com/jedie/django-phpBB3/issues/14

UnicodeEncodeError in the terminal while running migration

Solution: You must set either your locale, or the ​PYTHONIOENCODING environment variable, according to your terminal’s encoding — in most cases, utf-8. More info can be found in the comment of issues 9

How to debug migration

We use python built-in pdb module for debug a migration traceback. pudb is used, if installed. To install it, run e.g.:

~$ cd DjangoBB_env/
~/DjangoBB_env$ source bin/activate
(DjangoBB_env)~/DjangoBB_env$ pip install pudb

phpBB3 links

Here some links for phpBB documentations:

• http://wiki.phpbb.com/Tables

• http://www.phpbbdoctor.com/doc_tables.php

The SQL schema files from phpBB can be found here:

• https://github.com/phpbb/phpbb3/tree/develop/phpBB/install/schemas

history

• v0.2.1 - 21.09.2012

  • Add “is_staff” flag to all “global moderators”

  • Set “is_active” flag depent on “user_inactive_time”

  • Add --only_users for only migrate/update users.

  • Use for getting the pseudo “Anonymous” user account

• v0.2.0 - 21.09.2012

  • Speedup migration by disable haystack search index creation.

  • Bugfix: Handle not existing phpBB users -> Map to Anonymous user.

  • Add support for pudb debugger (optional) and fallback to pdb

  • Bugfix in migration: clean user signature, too.

  • Don’t check PHPBB_ATTACHMENT_PATH if no attachments exists in phpBB3 forum.

  • Avoid UnicodeEncodeError in migration output

  • Speedup user migration by stats only output

  • Add --flush_djangobb and --max_entries in migration command for testing.

• v0.1.6 - 29.08.2012

  • Add more information in README

  • Fix unicode errors

• v0.1.5 - 31.07.2012

  • migrate user language, too.

• v0.1.4 - 30.07.2012

  • many bugfixes in models.

• v0.1.3 - 27.07.2012

  • catch error if a post can’t be migrated and continue migration

  • remove django_authopenid and django_messages from phpbb2djangobb project settings

  • nicer output while migrating with eta time

• v0.1.2 - 27.07.2012

  • remove date from version string

  • add this histroy list ;)

• v0.1.0 - 24.07.2012

  • migrate post subscriptions, too.

• 23.07.2012

  • migrate phpBB3 file attachments, too.

  • bugfixes in bbcode cleanup

  • handle html entities in cleanup

• 20.07.2012

  • bbcode cleanup contributed by nitely

• 19.07.2012

  • start with phpBB3 migration command

• 16.07.2012

  • project starts by Jens Diemer

Project links

Homepage	https://github.com/jedie/django-phpBB3
PyPi	http://pypi.python.org/pypi/django-phpBB3/

donation

• Flattr this!

• Send Bitcoins to 1YJk2JTMyDrZ8oTHkMhTaQtRWqSgrHPiS

contact

Come into the conversation, besides the github communication features:

Forum	http://www.pylucid.org/en/forum/11/
IRC	#pylucid on freenode.net (Yes, the PyLucid channel…)
webchat	http://webchat.freenode.net/?channels=pylucid