At least in CPython, format strings can be given as bytes, as an alternative to str. E.g.

>>> struct.unpack(b'>hhl', b'\x00\x01\x00\x02\x00\x00\x00\x03')
(1, 2, 3)

Looking at the source code [1], this appears to be consciously accounted for. But it doesn't seem to be mentioned in the documentation. I think the docs should either say it's a possibility, or warn that it's an implementation detail.

[1] http://hg.python.org/cpython/file/cde4b66699fe/Modules/_struct.c#l1340

Also it would be nice to clarify if struct.Struct.format is meant to be a byte string. Reading the documentation and examples I expected a character string. It was an issue for me when embedding one structure within another:

HSF_VOL_DESC = Struct("< B 5s B")

# Python 3.2.3's "Struct.format" is actually a byte string
NSR_DESC = Struct(HSF_VOL_DESC.format.decode() + "B")

For 3.3, I verified that adding b prefix to first three doc examples gives same output as without, but also discovered that example outputs are wrong, at least on windows, because of byte ordering issues.

>>> pack('hhl', 1, 2, 3)
b'\x01\x00\x02\x00\x03\x00\x00\x00'
>>> pack(b'hhl', 1, 2, 3)
b'\x01\x00\x02\x00\x03\x00\x00\x00'
>>> unpack(b'hhl', b'\x00\x01\x00\x02\x00\x00\x00\x03')
(256, 512, 50331648)
>>> unpack('hhl', b'\x00\x01\x00\x02\x00\x00\x00\x03')
(256, 512, 50331648)

> but also discovered that example outputs are wrong

That's documented to some extent:  there's a line in the docs that says:
"All examples assume a native byte order, size, and alignment with a big-endian machine".

Given that little-endian machines are much more common that big-endian these days, it may be worth rewriting the examples for little-endian machines.

> Also it would be nice to clarify if struct.Struct.format is meant to be > a byte string.

Hmm.  That seems wrong to me.  After all, the format string is supposed to be a piece of human-readable text rather than a collection of bytes.  I think it's borderline acceptable to allow a bytes instance to be passed in for the format (practicality beats purity and all that), but I'd say that the output format should definitely be unicode.

I think the example should be switched *and* the formats should specify the endianess so the examples work on all systems.

I'm happy to put together a docs patch, but I don't have any indication of the right answer (is it a safe feature to use, or an implementation detail?) Is there another venue where I should raise the question?

Python 2 supports only str. Support for unicode objects has been added in r59687 (merged with other unrelated changes in changeset 13aabc23cf2e). Maybe Raymond can explain why the type for the Struct.format was chosen bytes, not unicode.

No, this is not r59687. I can't found from which revision in 59680-59695 it came.

The issue of Struct.format being a byte string has been raised separately in Issue 21071.

Actually the “struct” module doc string seems to already hint that format strings can be byte strings:

“Python bytes objects are used to hold the data representing the C struct and also as format strings . . .”

Assuming it is intended to support byte strings, here is a patch that documents them being allowed, and adds a test case

I think the direction to take for this depends on the outcome of Issue 21071. First we have to decide if the “format” attribute is blessed as a byte string (and not deprecated), or whether it is deprecated or changed to a text string.

Serhiy pointed out that it is not entirely “safe” because mixing equivalent byte and text formats can generate ByteWarning.

messages: + msg176701

components: + Extension Modules, - Library (Lib)