Home Original page

Parsing a potentially corrupted file

Chris Angelico rosuav at gmail.com
Wed Dec 14 07:57:03 EST 2016 
On Wed, Dec 14, 2016 at 10:43 PM, Paul  Moore <p.f.moore at gmail.com> wrote:
> This is a messy format to parse, but it's manageable. However, there's a catch. Because the logging software involved is broken, I can occasionally get a log record prematurely terminated with a new record starting mid-stream. So something like the following:
>
> [2016-11-30T20:04:08.000+00:00] [Component] [le[2016-11-30T20:04:08.000+00:00] [Component] [level] [] [] [id] Description of the issue goes here
>
> I'm struggling to find a "clean" way to parse this. I've managed a clumsy approach, by splitting the file contents on the pattern [ddd-dd-ddTdd:dd:dd.ddd+dd:dd] (the timestamp - I've never seen a case where this gets truncated) and then treating each entry as a record and parsing it individually. But the resulting code isn't exactly maintainable, and I'm looking for something cleaner.
>

Is the "[Component]" section something you could verify? (That is - is
there a known list of components?) If so, I would include that as a
secondary check. Ditto anything else you can check (I'm guessing the
[level] is one of a small set of values too.) The logic would be
something like this:

Read line from file.
Verify line as a potential record:
    Assert that line begins with timestamp.
    Verify as many fields as possible (component, level, etc)
    Search line for additional timestamp.
    If additional timestamp found:
        Recurse. If verification fails, assume we didn't really have a
corrupted line.
        (Process partial line? Or discard?)
    If "[[" in line:
        Until line is "]]":
            Read line from file, append to description
            If timestamp found:
                Recurse. If verification succeeds, break out of loop.

 Unfortunately it's still not really clean; but that's the nature of
working with messy data. Coping with ambiguity is *hard*.

ChrisA
More information about the Python-list mailing list