py3to2 2009.01.03

run python3.0 & 2to3 generated scripts w/ 2.6 extension support

this project is being superseded by asciiporn at http://code.google.com/p/asciiporn/

AUTHOR
kai zhu
[e-mail]

REQUIREMENTS
- posix/unix os (Windows currently unsupported)
- w/ python2.6 & python3.0 installed

INSTALL
$ python2.6 setup.py build
$ python2.6 setup.py install
$ python2.6 setup.py dev --quicktest
$ python2.6 setup.py dev --py2to3test ## takes awhile to finish

the above will build & install 3 files:
- extended python2.6 interpreter: bin/py3to2
- initialization script: lib/python2.6/site-packages/py3to2_init.py
- python3.0 bytecode compiler: lib/python2.6/site-packages/py3to2.py

################################################################################
ABSTRACT

py3to2 is a python2.6 interpreter w/ extended python3.0 opcodes, allowing it to
natively run python3.0 & 2to3 generated scripts. it should b
mostly backwards-compatible w/ cpython2.6 & its extensions.

the intended purpose is to allow developers to migrate python2.6 scripts to
python3.0 while retaining backwards compatibility w/ existing extension modules.
py3to2 coexists w/ ur existing python2.6 installation (it consists of 3 files)

for a real-world py3to2 app (python3.0 script using 2.6 extension modules),
checkout asciiporn: http://pypi.python.org/pypi/asciiporn

MECHANISM

py3to2 has 3 components:
- py3to2
python interpreter. can evaluate python2.6 bytecode containing additional
python3.0 opcode instructions

- py3to2_init.py
initialization script. sets up import hook for recognizing python3.0 scripts

- py3to2.py
bytecode compiler. the compile process takes 2 steps:
- a persistent python3.0 process is created for compiling scripts into
python3.0 code
- py3to2.py then converts the code from python3.0 to python2.6 format

MANIFEST

./patch/ - patched files
./py3to2.diff - summary of patches (maybe out-of-date)

################################################################################
MAGIC
simply add the MAGIC LINE:

from __future__ import py3k_syntax

to make py3to2 aware that a script is using python3.0 syntax

PSEUDOMETHOD
py3to2 supports ".." syntax notation for pseudomethods
goto: http://pypi.python.org/pypi/pseudomethod
for more details about this feature

2TO3
py3to2 includes convenience functions for automatically generating &
testing scripts using 2to3:
- class py2to3:
- __call__ - overwrites file w/ one generated by 2to3
- overwrite_and_load_module - overwrites file & then attempt to import it
- test_stdlib - given a directory containing a copy of python2.6's
standard library, it will overwrite them using 2to3 & then
attempts to load each file

API: try help(py3to2) ^_-

py3to2 module:
- class codetree - mutable codeobj & disassembler/assembler/debugger
- class compiler - compiling tools
- python3.0 wrappers:
- py3k_compile() - compile python3.0 src
- py3k_eval() - eval py3thon3.0 src
- py3k_exec() - exec python3.0 src

USAGE
start up the py3to2 interpreter by typing "py3to2" in ur shell:
$ py3to2

Python 2.6.py3to2 (r26:66714, Nov 18 2008, 00:56:43)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-10)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

try out this simple python3.0 script:
################################################################
## PEP3132 Extended Iterable Unpacking
## copy this to file test_pep3132.py

from __future__ import py3k_syntax

a,*b = 1,2,3
assert a == 1 and b == [2,3]
print(a,b)
################################################################
>>>
>>> import test_pep3132
created...
py3k server starting...
...py3k server started w/...

1 [2, 3]

here's another python3.0 script using scipy (python2.6) extension module:
################################################################
## u must have scipy installed for this script to work
## copy this to file test_pep3132_scipy.py

from __future__ import py3k_syntax

import scipy
a,*b = scipy.array([1,2,3])
assert a == 1 and b == [2,3]
print(a,b)
################################################################
>>>
>>> import test_pep3132_scipy
1 [2, 3]

another simple, but more thorough test script, test_py3k,
is included w/ this distribution:
>>>
>>> import test_py3k
testing PEP3102 Keyword-Only Arguments
testing PEP3104 Access to Names in Outer Scopes
testing PEP3105 Make print a function
testing PEP3107 Function Annotations
testing PEP3112 Bytes literals in Python 3000
testing PEP3113 Removal of Tuple Parameter Unpacking
testing PEP3114 Renaming iterator.next() to .__next__()
testing PEP3115 Metaclasses in Python 3000
testing PEP3120 Using UTF-8 as the default source encoding
testing PEP3127 Integer Literal Support and Syntax
testing PEP3129 Class Decorators
testing PEP3131 Supporting Non-ASCII Identifiers
testing PEP3132 Extended Iterable Unpacking
testing PEP3135 New Super
testing pseudomethod example 0
testing pseudomethod example 1
testing pseudomethod example 2
testing pseudomethod example 3
testing numpy example

FEATURES
PEP3102 Keyword-Only Arguments
PEP3104 Access to Names in Outer Scopes
PEP3105 Make print a function
PEP3107 Function Annotations
PEP3111 Simple input built-in in Python 3000
PEP3112 Bytes literals in Python 3000
PEP3113 Removal of Tuple Parameter Unpacking
PEP3114 Renaming iterator.next() to .__next__()
PEP3115 Metaclasses in Python 3000
PEP3120 Using UTF-8 as the default source encoding
PEP3127 Integer Literal Support and Syntax
PEP3129 Class Decorators
PEP3131 Supporting Non-ASCII Identifiers
PEP3132 Extended Iterable Unpacking
PEP3135 New Super

UNICODE SUPPORT
py3to2 will only load ascii & utf8-encoded scripts
(utf8 is the default encoding in python3.0).

although they're illegal in python3.0, for backwards-compatibility sake,
py3to2 supports unicode literals for explicit <unicode> obj creation:

u"\u1234" is an explicit <unicode> obj (note unicode literal in front)
"\u1234" is NOT a <unicode> obj when converted back to python2.6

note also the following r equivalent under python2.6:

u"\u1234" <==> "\u1234".decode("raw_unicode_escape")

so u MUST do either u"..." or "...".decode("raw_unicode_escape")
to create explicit <unicode> obj in py3to2.

LIMITATIONS (FEATURES NOT FULLY SUPPORTED)
except for the aforementioned unicode issue, from a migration standpoint,
py3to2 is mostly feature complete in terms of python3.0's language syntax,

language issue aside, python3.0 scripts will still behave differently b/c of
internal differences between python2.6 & python3.0:
- exception handling. py3to2 implements python3.0 syntax for raising &
catching exceptions. but the underlying behavior is still python2.6
- builtin functions / types. a few of these have become different beasts
under python3.0

################################################################################
PYTHON2.6 COMPATIBILITY TEST
$ python setup.py dev --maketest
...
test_sys
test test_sys failed -- Traceback (most recent call last):
File ".../Python-2.6/Lib/test/test_sys.py", line 487, in test_objecttypes
check(get_cell().func_code, size(h + '4i8Pi2P'))
File ".../Python-2.6/Lib/test/test_sys.py", line 423, in check_sizeof
self.assertEqual(result, size, msg)
AssertionError: wrong size for <type 'code'>: got 128, expected 120
...
325 tests OK.
1 test failed:
test_sys
35 tests skipped:
test_aepack test_al test_applesingle test_bsddb185 test_bsddb3
test_cd test_cl test_codecmaps_cn test_codecmaps_hk
test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_curses
test_gdbm test_gl test_imgfile test_kqueue test_linuxaudiodev
test_macos test_macostools test_normalization test_ossaudiodev
test_pep277 test_py3kwarn test_scriptpackages test_socketserver
test_startfile test_sunaudiodev test_tcl test_timeout
test_urllib2net test_urllibnet test_winreg test_winsound
test_zipfile64
2 skips unexpected on linux2:
test_tcl test_gdbm

2TO3 COMPATIBILITY TEST
$ python setup.py dev --py2to3test
...
tested 200 2to3 generated scripts from 2.6.1.py3to2 (r261:67515, Jan 4 2009,
01:28:21)
[GCC 4.2.3 20071123 (prerelease) (Debian 4.2.2-4)] standard library

0 skipped:


28 couldn't import required modules:
BaseHTTPServer CGIHTTPServer cgi cookielib copy DocXMLRPCServer
dummy_threading HTMLParser httplib _LWPCookieJar macurl2path mimetools mimetypes
_MozillaCookieJar os pdb pickle pydoc re robotparser sgmllib SimpleHTTPServer
SimpleXMLRPCServer _strptime tempfile threading urllib2 urllib

6 were non-utf8 compliant scripts:
base64 getopt heapq shlex smtpd tarfile

8 failed import due to other reasons:
anydbm dbhash doctest imputil sets trace UserList UserString

159 passed import:
_abcoll abc aifc ast asynchat asyncore atexit audiodev Bastion bdb binhex
bisect calendar cgitb chunk cmd codecs codeop code collections colorsys commands
compileall ConfigParser contextlib Cookie copy_reg cProfile csv decimal difflib
dircache dis dumbdbm dummy_thread filecmp fileinput fnmatch formatter fpformat
fractions ftplib functools __future__ genericpath getpass gettext glob gzip
hashlib hmac htmlentitydefs htmllib ihooks imaplib imghdr inspect io keyword
linecache locale macpath mailbox mailcap markupbase md5 mhlib MimeWriter mimify
modulefinder multifile mutex netrc new nntplib ntpath nturl2path numbers opcode
optparse os2emxpath __phello__.foo pickletools pipes pkgutil platform plistlib
popen2 poplib posixfile posixpath pprint profile pstats pty pyclbr py_compile
pydoc_topics Queue quopri random repr rexec rfc822 rlcompleter runpy sched sha
shelve shutil site smtplib sndhdr socket SocketServer sre_compile sre_constants
sre_parse sre ssl stat statvfs StringIO stringold stringprep string struct
subprocess sunaudio sunau symbol symtable tabnanny telnetlib textwrap this
_threading_local timeit toaiff tokenize token traceback tty types unittest
urlparse UserDict user uuid uu warnings wave weakref webbrowser whichdb xdrlib
xmllib xmlrpclib zipfile

################################################################################
RECENT CHANGES:
20090103 updated to python 2.6.1
20081129 - major revision
PyCodeObject now has kwonlyargcount attr
- breaks one regression test but greatly simplifies patch & prevents many
future bugs
- fixes pydoc bug
PyFunctionObject now has __kwdefaults__ & __annotations__ attr
ceval.c re-patched in light of above changes (much simpler)
20081128
more documentation
added 2to3 convenience functions
added unicode utf-8 support
20081123
moved pseudomethod syntax handling to py3k server
added more checks during setup
added more documentation
backported patch r67299 fixing an issue w/ super()
cleaned up py3to2.compiler class
20081120
fixed package importing bug - py3to2 failed to import foo.bar
20081119
created self-installing distutils distribution
20081019
ported to python-2.6
consolidate & simplify patches to 1 file: ceval.c
created extension module builtins_py3k
revamped import hook again
removed unicode support & restrict source code to ascii-only
20080727
revampled import hook
20080911
consolidate patches to 2 files: bltinmodule.c & ceval.c
20080828
add kwonlyargcount 'attr' to codeobj
add __annotations__ & __kwdefaults__ attr to funcobj
add __pseudomethod__ feature to baseobj
20080819
pure python import hook - removed magic comment & use magic path instead
revamped str & bytes handling
revamped py3k .pyc file handling
20080802
pep3135 New Super
20080717
pep3107 Function Annotations
pep3120 Using UTF-8 as the default source encoding
pep3131 Supporting Non-ASCII Identifiers
20080713
import / reload works transparently on py3k scripts using a magic comment
added pep3102 Keyword-Only Arguments
20080709 added a py3k preparser
20080702
rewrote py3k server's pipe io. implemented partial bytearray & bytes class.
wrote a few simple tests
20080630
__build_class__ function to bltmodule.c. tested class decorators to b
working.
################################################################################