Home Original page

Basic tokenizer

Andrea Griffini agriff at tin.it
Sat Sep 4 02:05:39 EDT 2004 
On Thu, 02 Sep 2004 13:02:09 GMT, "Paul McGuire"
<ptmcg at austin.rr._bogus_.com> wrote:

>I'd be interested in your feedback on this pyparsing example:
>http://www.geocities.com/ptmcg/python/fourFn.py.txt
>
>(The "fourFn.py" title is a legacy - this now does 5-function arithmetic
>plus trig functions.)

pyparsing impressed me... but positively :-)

I don't like the example you're talking about that much;
first there are at least a couple of problems in the
specific calculator logic... the solution found for "^"
right-association is broken (try evaluating 2^3+4) and
there is no provision for unary minus.

The solution for "^" is however very simple; instead of
using "expr" you can use as structure

  factor = Forward()
  factor << (atom + Optional( expop + factor ));

Usually when I (hand) write a parser I've a single function
for parsing all binary operations that reads from a
table the operator and associativity. For left-association
the rule is

  expr(n) = expr(n-1) + ZeroOrMore( op+expr(n-1) )

while for right-association is

  expr(n) = expr(n-1) + ZeroOrMore( op+expr(n) )

(in the latter ZeroOrMore could be Optional instead,
but ZeroOrMore is fine anyway and that way the code
is simpler).

Also I don't like the idea of writing to the global
exprstack and the way parsing expression is used
(things like (lpar + expr + rpar).suppress() look
really weird).

The module seems to me very easy to use, however, and in
little time and very few lines of code I was able to change
fourFn to parse the five binary operations, unary minus,
comparisions, the C++ "?:" ternary opertor, variable
reading and writing (but with assignment being "statement"
and not an operator like in python) and generating as
output a list of opcodes for a stack based VM including
jump codes for "?:" (and this without global variables,
and so handling correctly the backtracking).

I've yet to see how nicely is possible to handle error
reporting (another no-brainer for hand-coded parsers);
but so far I didn't read the documentation :-).

Andrea
More information about the Python-list mailing list