Home Original page

dstore package - github.com/streamingfast/dstore - Go Packages

This section is empty.

View Source

var ErrMetadataNotSupported = errors.New("metadata is not supported by local file system store")

Deprecated: Use NewStoreFromFileURL

ReadObject directly reads the giving file URL by parsing the file url, extracting the path and the filename from it, creating the store interface, opening the object directly and returning all this.

This is a shortcut helper function that make it simpler to get store from a single file url. 

type AzureStore struct {
	
}
func (c AzureStore) Overwrite() bool
func (c AzureStore) SetMeter(meter Meter)

Deprecated: Use the Options to add callbacks to inject metering from the upstream code instead 

func (c AzureStore) SetOverwrite(in bool)
type BufferedFileReadCloser struct {
	
}
func NewBufferedFileReadCloser(file *os.File) *BufferedFileReadCloser
func (c GSStore) Overwrite() bool
func (c GSStore) SetMeter(meter Meter)

Deprecated: Use the Options to add callbacks to inject metering from the upstream code instead 

func (c GSStore) SetOverwrite(in bool)
type LocalStore struct {
	
}
func (c LocalStore) Overwrite() bool
func (c LocalStore) SetMeter(meter Meter)

Deprecated: Use the Options to add callbacks to inject metering from the upstream code instead 

func (c LocalStore) SetOverwrite(in bool)
type MemoryStore struct {
	
}

MemoryStore is a store that keeps all data in memory. Useful for unit testing where a store is required. Work in progress, not all methods are implemented 

func (c MemoryStore) Overwrite() bool
func (c MemoryStore) SetMeter(meter Meter)

Deprecated: Use the Options to add callbacks to inject metering from the upstream code instead 

func (c MemoryStore) SetOverwrite(in bool)
type Meter interface {
	AddBytesRead(int)
	AddBytesWritten(int)
}

Deprecated: This interface will no longer be used, use the Options to inject all metering logic from the upstream code instead 

type MockStore struct {
	OpenObjectFunc       func(ctx context.Context, name string) (out io.ReadCloser, err error)
	WriteObjectFunc      func(ctx context.Context, base string, f io.Reader, metadataKeyValues ...string) error
	CopyObjectFunc       func(ctx context.Context, src, dest string) error
	DeleteObjectFunc     func(ctx context.Context, base string) error
	FileExistsFunc       func(ctx context.Context, base string) (bool, error)
	ObjectAttributesFunc func(ctx context.Context, base string) (*ObjectAttributes, error)
	SetMetadataFunc      func(ctx context.Context, base string, metadata map[string]string) error
	ListFilesFunc        func(ctx context.Context, prefix string, max int) ([]string, error)
	WalkFunc             func(ctx context.Context, prefix string, f func(filename string) error) error
	WalkFromFunc         func(ctx context.Context, prefix, startingPoint string, f func(filename string) error) error
	WalkFromToFunc       func(ctx context.Context, prefix, startingPoint, exclusiveEndPoint string, f func(filename string) error) error
	PushLocalFileFunc    func(ctx context.Context, localFile string, toBaseName string) (err error)

	Files    map[string][]byte
	Metadata map[string]map[string]string
	
}
func (s *MockStore) Overwrite() bool

SetFile sets the content of a file. Set the value "err" to trigger an error when reading this file. 

func (_ *MockStore) SetMeter(_ Meter)

Deprecated: SetMeter on MockStore no longer does anything 

func (s *MockStore) SetOverwrite(in bool)

WriteFiles dumps currently know file 

type Option interface {
	
}
func AllowOverwrite() Option

AllowOverwrite allow files to be overwritten when already exist at a given location. 

func Compression(compressionType string) Option

Compression defines which kind of compression to use when creating the store instance.

Valid `compressionType` values: - <empty> No compression - zstd Use ZSTD compression - gzip Use GZIP compression

WithCompressedReadCallback allows you to set a callback function that is invoked when a compressed read operation is performed.

WithCompressedWriteCallback allows you to set a callback function that is invoked when a compressed write operation is performed.

WithUncompressedReadCallback allows you to set a callback function that is invoked when an uncompressed read operation is performed.

WithUncompressedWriteCallback allows you to set a callback function that is invoked when an uncompressed write operation is performed. 

func (c S3Store) Overwrite() bool
func (c S3Store) SetMeter(meter Meter)

Deprecated: Use the Options to add callbacks to inject metering from the upstream code instead 

func (c S3Store) SetOverwrite(in bool)
type Store interface {
	OpenObject(ctx context.Context, name string) (out io.ReadCloser, err error)
	FileExists(ctx context.Context, base string) (bool, error)

	ObjectPath(base string) string
	ObjectURL(base string) string
	ObjectAttributes(ctx context.Context, base string) (*ObjectAttributes, error)
	SetMetadata(ctx context.Context, base string, metadata map[string]string) error

	WriteObject(ctx context.Context, base string, f io.Reader, metadataKeyValues ...string) (err error)
	PushLocalFile(ctx context.Context, localFile, toBaseName string) (err error)

	CopyObject(ctx context.Context, src, dest string) error
	Overwrite() bool
	SetOverwrite(enabled bool)

	
	
	
	
	
	
	Walk(ctx context.Context, prefix string, f func(filename string) (err error)) error
	WalkFrom(ctx context.Context, prefix, inclusiveFrom string, f func(filename string) (err error)) error
	WalkFromTo(ctx context.Context, prefix, inclusiveFrom, exclusiveTo string, f func(filename string) (err error)) error

	ListFiles(ctx context.Context, prefix string, max int) ([]string, error)

	DeleteObject(ctx context.Context, base string) error

	
	
	BaseURL() *url.URL
	SubStore(subFolder string) (Store, error)

	
	SetMeter(meter Meter)
}
func NewStore(baseURL, extension, compressionType string, overwrite bool, opts ...Option) (Store, error)

NewStore creates a new Store instance. The baseURL is always a directory, and does not end with a `/`.

NewStoreFromFileURL works against a full file URL to derive the store from it as well as the filename it points to. Use this method **only and only if** the input points to a file directly, if your input is to build a store, use NewStore instead.

This is a shortcut helper function that make it simpler to get store from a single file url.

OpenObject directly opens the giving file URL by parsing the file url, extracting the path and the filename from it, creating the store interface, opening the object directly and returning all this.

This is a shortcut helper function that make it simpler to get store from a single file url.