Guido van Rossum guido at python.org
Mon Aug 9 22:21:57 CEST 2004

• Previous message: [Python-Dev] tokenize.py (generate_tokens yielding @ as OP)
• Next message: [Python-Dev] Another test_compiler mystery
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

---

> This is a question regarding the tokenize.generate_tokens function.
> 
> As it is in CVS HEAD, generate_tokens yields the '@' token as
> having type OP, not type AT. Is this how it should be?

Without looking at the code, I believe tokenize treats all operators
as OP, for a rather liberal definition of "operator".

(A more serious problem that I just encountered in a different
context: when you feed it input containing unsupported characters, the
spaces before those unsupported characters are also returned as
tokens.  I think that's a bug.)

--Guido van Rossum (home page: http://www.python.org/~guido/)

---

• Previous message: [Python-Dev] tokenize.py (generate_tokens yielding @ as OP)
• Next message: [Python-Dev] Another test_compiler mystery
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

---

More information about the Python-Dev mailing list