coder package - github.com/apache/beam/sdks/go/pkg/beam/core/graph/coder

coder package - github.com/apache/beam/sdks/go/pkg/beam/core/graph/coder - Go Packages

Package coder contains coder representation and utilities. Coders describe how to serialize and deserialize pipeline data and may be provided by users.

This section is empty.

ErrVarIntTooLong indicates a data corruption issue that needs special handling by callers of decode. TODO(herohde): have callers perform this special handling.

DecodeBool decodes a boolean according to the beam protocol.

DecodeByte decodes a single byte.

DecodeBytes decodes a length prefixed []byte according to the beam protocol.

DecodeDouble decodes a float64 in big endian format.

DecodeEventTime decodes an EventTime.

DecodeInt32 decodes an int32 in big endian format.

DecodeStringUTF8 decodes a length prefixed UTF8 string.

DecodeUint32 decodes an uint32 in big endian format.

DecodeUint64 decodes an uint64 in big endian format.

DecodeVarInt decodes an int64.

DecodeVarUint64 decodes an uint64.

DecoderForSlice returns a decoding function that decodes the beam row encoding into the given type.

Returns an error if the given type is invalid or not decodable from a beam schema row.

EncodeBool encodes a boolean according to the beam protocol.

EncodeByte encodes a single byte.

EncodeBytes encodes a []byte with a length prefix per the beam protocol.

EncodeDouble encodes a float64 in big endian format.

EncodeEventTime encodes an EventTime as an uint64. The encoding is millis-since-epoch, but shifted so that the byte representation of negative values are lexicographically ordered before the byte representation of positive values.

EncodeInt32 encodes an int32 in big endian format.

EncodeStringUTF8 encodes a UTF8 string with a length prefix.

EncodeUint32 encodes an uint32 in big endian format.

EncodeUint64 encodes an uint64 in big endian format.

EncodeVarInt encodes an int64.

EncodeVarUint64 encodes an uint64.

EncoderForSlice returns an encoding function that encodes a struct type or a pointer to a struct type using the beam row encoding.

Returns an error if the given type is invalid or not encodable to a beam schema row.

IsCoGBK returns true iff the coder is for a CoGBK type.

IsFieldNil examines the passed in packed bits nils buffer and returns true if the field at that index wasn't encoded and can be skipped in decoding.

IsKV returns true iff the coder is for key-value pairs.

IsW returns true iff the coder is for a WindowedValue.

ReadRowHeader handles the field header for row decodings.

This returns the number of encoded fileds, the raw bitpacked bytes and any error during decoding. Each bit only needs only needs to be examined once during decoding using the IsFieldNil helper function.

If there are no nil fields encoded,the byte array will be nil, and no encoded fields will be nil.

ReadSimpleRowHeader is a convenience function to read Beam Schema Row Headers for values that do not have any nil fields. Reads and validates the number of fields total (returning an error for mismatches, and checks that there are no nils encoded as a bit field.

func RegisterCoder(t reflect.Type, enc, dec interface{})

RegisterCoder registers a user defined coder for a given type, and will be used if there is no beam coder for that type. Must be called prior to beam.Init(), preferably in an init() function.

Coders are encoder and decoder pairs, and operate around []bytes.

The coder used for a given type follows this ordering:

Coders for Known Beam types.
Coders registered for specific types
Coders registered for interfaces types
Default coder (JSON)

Types of kind Interface, are handled specially by the registry, so they may be iterated over to check if element types implement them.

Repeated registrations of the same type overrides prior ones.

func RegisterSchemaProviders(rt reflect.Type, enc, dec interface{})

RegisterSchemaProviders Register Custom Schema providers.

func RequireAllFieldsExported(require bool)

RequireAllFieldsExported when set to true will have the default coder buildings using RowEncoderForStruct and RowDecoderForStruct fail if there are any unexported fields. When set false, unexported fields in default destination structs will be silently ignored when coding. This has no effect on types with registered coder providers.

RowDecoderForStruct returns a decoding function that decodes the beam row encoding into the given type.

Returns an error if the given type is invalid or not decodable from a beam schema row.

RowEncoderForStruct returns an encoding function that encodes a struct type or a pointer to a struct type using the beam row encoding.

Returns an error if the given type is invalid or not encodable to a beam schema row.

Types returns a slice of types used by the supplied coders.

WriteRowHeader handles the field header for row encodings.

WriteSimpleRowHeader is a convenience function to write Beam Schema Row Headers for values that do not have any nil fields. Writes the number of fields total and a 0 len byte slice to indicate no fields are nil.

Coder is a description of how to encode and decode values of a given type. Except for the "custom" kind, they are built in and must adhere to the (unwritten) Beam specification.

func CoderFrom(c *CustomCoder) *Coder

CoderFrom is a helper that creates a Coder from a CustomCoder.

NewBool returns a new bool coder using the built-in scheme.

NewBytes returns a new []byte coder using the built-in scheme. It is always nested, for now.

func NewCoGBK(components []*Coder) *Coder

NewCoGBK returns a coder for CoGBK elements.

NewDouble returns a new double coder using the built-in scheme.

func NewI(c *Coder) *Coder

NewI returns an iterable coder in the form of a slice.

func NewKV(components []*Coder) *Coder

NewKV returns a coder for key-value pairs.

func NewPW(c *Coder, w *WindowCoder) *Coder

NewPW returns a ParamWindowedValue coder for the window of elements.

NewR returns a schema row coder for the type.

NewString returns a new string coder using the built-in scheme.

func NewT(c *Coder, w *WindowCoder) *Coder

NewT returns a timer coder for the window of elements.

NewVarInt returns a new int64 coder using the built-in scheme.

func NewW(c *Coder, w *WindowCoder) *Coder

NewW returns a WindowedValue coder for the window of elements.

func SkipW(c *Coder) *Coder

SkipW returns the data coder used by a WindowedValue, or returns the coder. This allows code to seamlessly traverse WindowedValues without additional conditional code.

func (c *Coder) Equals(o *Coder) bool

Equals returns true iff the two coders are equal. It assumes that functions with the same name and types are identical.

CustomCoder contains possibly untyped encode/decode user functions that are type-bound at runtime. Universal coders can thus be used for many different types, but each CustomCoder instance will be bound to a specific type.

LookupCustomCoder returns the custom coder for the type if any, first checking for a specific matching type, and then iterating through registered interface coders in reverse registration order.

NewCustomCoder creates a coder for the supplied parameters defining a particular encoding strategy.

Equals returns true iff the two custom coders are equal. It assumes that functions with the same name and types are identical.

type ElementDecoder interface {
	Decode(r io.Reader) (interface{}, error)
}

ElementDecoder encapsulates being able to decode an element from a reader.

type ElementEncoder interface {
	Encode(element interface{}, w io.Writer) error
}

ElementEncoder encapsulates being able to encode an element into a writer.

Kind represents the type of coder used.

const (
	Custom             Kind = "Custom" 
	Bytes              Kind = "bytes"  
	String             Kind = "string" 
	Bool               Kind = "bool"
	VarInt             Kind = "varint"
	Double             Kind = "double"
	Row                Kind = "R"
	Timer              Kind = "T"
	WindowedValue      Kind = "W"
	ParamWindowedValue Kind = "PW"
	Iterable           Kind = "I"
	KV                 Kind = "KV"
	LP                 Kind = "LP" 

	Window Kind = "window" 

	
	
	
	
	
	
	
	
	
	CoGBK Kind = "CoGBK"
)

Tags for the various Beam encoding strategies. https://beam.apache.org/documentation/programming-guide/#coders documents the usage of coders in the Beam environment.

type RowDecoderBuilder struct {

	
	
	
	
	RequireAllFieldsExported bool
	
}

RowDecoderBuilder allows one to build Beam Schema row encoders for provided types.

Build constructs a Beam Schema coder for the given type, using any providers registered for itself or it's fields.

When decoding values, decoder functions produced by this builder will first check for exact type matches, then interfaces implemented by the type in recency order of registration, and then finally the default Beam Schema encoding behavior.

TODO(BEAM-9615): Add final factory types. This interface is subject to change. Currently f must be a function func(reflect.Type) (func(io.Reader) (interface{}, error), error)

type RowEncoderBuilder struct {

	
	
	
	
	RequireAllFieldsExported bool
	
}

RowEncoderBuilder allows one to build Beam Schema row encoders for provided types.

Build constructs a Beam Schema coder for the given type, using any providers registered for itself or it's fields.

When generating encoding functions, this builder will first check for exact type matches, then against interfaces with registered factories in recency order of registration, and then finally use the default Beam Schema encoding behavior.

TODO(BEAM-9615): Add final factory types. This interface is subject to change. Currently f must be a function of the type func(reflect.Type) func(T, io.Writer) (error).

type WindowCoder struct {
	Kind    WindowKind
	Payload string 
}

WindowCoder represents a Window coder.

func NewGlobalWindow() *WindowCoder

NewGlobalWindow returns a window coder for the global window.

func NewIntervalWindow() *WindowCoder

NewIntervalWindow returns a window coder for interval windows.

Equals returns whether passed in WindowCoder has the same Kind and Payload as this WindowCoder.

WindowKind represents a kind of window coder.

const (
	GlobalWindow   WindowKind = "GWC"
	IntervalWindow WindowKind = "IWC"
)

Available window coders. The same coder could be used for more than one kind of windowing strategy.