RegEx Tokenizer

The follow code snippet is adapted from Fredrik Lundh's effbot.org entry
Using Regular Expressions for Lexical Analysis

Say you want to tokenize an expression such as "(3+5)*10":

#!/usr/bin/env python
'''
Use regex to tokenize a string expression.
adapted from:
http://effbot.org/zone/xml-scanner.htm
'''
import re

reg_token = re.compile(r"""
\s* #skip whitespace
([0-9\.]+| #one or more digits or '.'
aka floats or ints
\w+| #words
[+\-*/!^%&|]{1,2}| #operators
.) #any character except newline
""",
re.VERBOSE)

def tokenize(expr):
'''
Returns a list of tokens for an expression string.
Allows operators +-*/!^%&|
Treats doubled operator e.g., **, ++ as single token
'''
def v_token(obj):
try:
if '.' in obj:
return float(obj)
else:
return int(obj)
except:
return obj

return [v_token(tkn.group()) for tkn
in reg_token.finditer(expr)]

Let's test on some expressions

expr = ["(3+7)*90", # basic
"(3+7.1)*90", # has floats
"(3+7.1)*90*alpha", # has variables
"(3+7.1)*90*alpha, g", # invalid expression, tokenize and leave to parser
"(5.0 - 3.2)/6*9", # other forms
"b = 2 + a*10",
"x = \n x**2", #picks up **, ++ as a token
"i++",
""
]

for exp in expr:
tkns = tokenize(exp)
print("\nExpression: %s\nTokens: %s " % (exp, tkns))

Gives us...

Expression: (3+7)*90
Tokens: ['(', 3, '+', 7, ')', '*', 90]

Expression: (3+7.1)*90
Tokens: ['(', 3, '+', 7.0999999999999996, ')', '*', 90]

Expression: (3+7.1)*90*alpha
Tokens: ['(', 3, '+', 7.0999999999999996, ')', '*', 90, '*', 'a', 'l', 'p', 'h', 'a']

Expression: (3+7.1)*90*alpha, g
Tokens: ['(', 3, '+', 7.0999999999999996, ')', '*', 90, '*', 'a', 'l', 'p', 'h', 'a', ',', ' g']

Expression: (5.0 - 3.2)/6*9
Tokens: ['(', 5.0, ' -', 3.2000000000000002, ')', '/', 6, '*', 9]

Expression: b = 2 + a*10
Tokens: ['b', ' =', 2, ' +', ' a', '*', 10]

Expression: x =
x**2
Tokens: ['x', ' =', ' \n x', '**', 2]

Expression: i++
Tokens: ['i', '++']

Expression:
Tokens: []

code snippet is at dzone: http://snippets.dzone.com/user/bondgeek

Speaking of timing

Using list comprehension is much faster than not:

In [35]: ll = [(x,x*x) for x in range(100)]

In [36]: def f1(obj):
....: for row in obj:
....: x = row[0]
....: y = row[1]
....:

In [37]: def f2(obj):
....: for row in obj:
....: x,y = row
....:

In [38]: def f3(obj):
....: for row in obj:
....: x,y = (row[0],row[1])
....:

In [39]: timing(f1,10000,ll)
f1 2.04

In [40]: timing(f2,10000,ll)
f2 0.8

In [41]: timing(f3,10000,ll)
f3 2.41

Function calls always add a bit of overhead:

def f1(obj):
x = min(obj,0.0)

def f3(obj):
if obj <= 0.0:
x = obj
else:
x = 0.0

In [56]: timing(f1,10000,5.)
f1 0.06

In [57]: timing(f1,10000,-5.)
f1 0.05

In [67]: timing(f3,10000,5.)
f3 0.04

In [68]: timing(f3,10000,-5.)
f3 0.04

Useful to know.

Timing is everything

Python Patterns - An Optimization Anecdote

The above link is to an article by Guido van Rossum on the Python website.   Needless to say, anything Guido says on the topic of Python is worth looking at (he is the author of the Python programming language). 

I thought of this particular article while reading a post, trying to decide how to check if an object is a sequence.  While all the contributors to the discussion are helpful, none actually checks the performance of the proposed solutions.  This is typical of the posts you see on various forums. 

Guido's article highlights how straightforward it is to do basic testing most of the time.  Here is a quick summary of performance of the proposed solutions in the above link:

import time

# Guido's timing function
def timing(f, n, a):
    print f.__name__,
    r = range(n)
    t1 = time.clock()
    for i in r:
        f(a); f(a); f(a); f(a); f(a); f(a); f(a); f(a); f(a); f(a)
    t2 = time.clock()
    print round(t2-t1, 3)

if __name__ == "__main__":
    #For Example
    # some functions to check if an object is a sequence
    def isit(obj):
        try:
            it = iter(obj)
            return True
        except TypeError:
            return False

    isit2 = lambda obj: isinstance(obj,basestring) or    \ getattr(obj,'__iter__',False)

    def isit3(obj):
        return (isinstance(obj,basestring) or getattr(obj,'__iter__',False))

    #...then:
    '''
    >>> timing(isit3, 100000, [])
    isit3 0.99
    >>> timing(isit2, 100000, [])
    0.99

    >>> timing(isit, 100000, [])
    isit 0.53
    '''

Choosing a Python GUI api

I narrowed the choice to Tkinter and wxPython fairly quickly--based on Tkinter being the de facto alternative and wxPython being the most discussed alternative on a basic Google search of "Python GUI".

Also wxPython has the largest widget collection, including a spreadsheet and since what I'm doing will involve a spreadsheet-like interface I decided that further investigation had quickly diminishing returns.

PyQt looks very powerful, and drives the incredibly impressive Orange application--but failed the "can I install and use it without much brain damage?" test, as did pyGTK  (also known as the "can an idiot install it?" test--me being the idiot--if something requires more than one or steps to install, it generally fails this test).   I would not be surprised to need revisit pyQt and pyGTK for larger scale projects.

The following, very helpful discussion walks through a very simple app in Tkinter and wxPython side-by-side.  Good for understanding the basic differences of the two packages and for understand the basics of GUI programming.

Building a basic GUI application in Python with Tkinter and wxWidgets

NB:  One fact that might help clear confusion as you surf GUI related posts-- wxWidgets == wxWindows.   The name of the underlying C++ library was changed to wxWidgets at some point (no doubt copyright/trademark related).

I'm also starting to look at wxGlade, a GUI builder wrapper for wxPython.

NB, re Editors:  I am using Eclipse with Pydev, with good results. 

Python: Static Methods versus Class Methods

The best discussion I've seen regarding the differences between static methods and class methods in Python is an old post at Miya's blog.    I like the way Miya approaches it.  Rather than going into the technical discussion found in the Python docs, he asks why would one use static methods, if it seems that class methods can do everything static methods can but not vice versa.

The key to understanding the difference between the two is in the comments.  A commentor points out

"classmethod give[s] you access to the class's attributes. static method does not so..."

Modifying the commentors example slightly:

>>> class MyClass(object):

        myattribute = 'spam'

        @classmethod
        def eggs(cls):
            return cls.myattribute

        @staticmethod
        def static_eggs():
            self.myattribute   # Will this work??
       
>>> MyClass.eggs()             # O.K. for class method
'spam'
>>> MyClass.static_eggs()     # ...not so much for static method
Traceback (most recent call last):
  File "<pyshell#33>", line 1, in <module>
    MyClass.static_eggs()
  File "<pyshell#31>", line 8, in static_eggs
    self.myattribute
NameError: global name 'self' is not defined

To recap:

• Both static and class methods can be called from the class without an instance:

>>>MyClass.static_method_that_says_hi()
"HI"
>>>MyClass.class_method_that_says_hi()
"HI"
>>>x = MyClass()
>>>x.static_method_that_says_hi()
"HI"

• Both can be inherited by sub-classes and maintain their identity (i.e., both are actually static).

• Class methods give you access to a class attributes and static methods do not.

So, why use one instead of the other?  Why not just use class methods since they're more powerful?

For me the principle is to use the simplest structure that handle's problem.   Class methods can do more, and therefore using them should signal that you're class does fairly complicated stuff.  Having a bias to using static methods means that you've thought about parsimony in your design. 

I'll come up some examples of each and be back.