S3InputSource Class Reference

Source for reading XML files from Amazon S3 (s3://bucket/key), with Arrow's S3 filesystem. More...

#include <OpenMS/FORMAT/DATAACCESS/S3InputSource.h>

Public Member Functions
	S3InputSource (const std::string &uri)
	Opens the S3 object uri.
	~S3InputSource ()
std::unique_ptr< InputStream >	makeStream () const
	A stream of the (decompressed) content of the object, for the parser.
	S3InputSource (const S3InputSource &)=delete
S3InputSource &	operator= (const S3InputSource &)=delete

Static Public Member Functions
static bool	isS3URI (const std::string &uri)
	Whether uri is an S3 URI (starts with "s3://")
static bool	isAvailable ()
	Whether OpenMS can read S3 URIs: it was built with an Arrow library that has S3 support.
static Int64	objectSize (const std::string &uri)
	The size of the S3 object uri, in bytes.
static std::string	readRange (const std::string &uri, Int64 offset, Int64 length)
	The bytes [offset, offset + length) of the S3 object uri (fewer at its end).

Private Attributes
std::unique_ptr< Impl >	impl_
	The opened object (Arrow types stay out of this header)

Detailed Description

Source for reading XML files from Amazon S3 (s3://bucket/key), with Arrow's S3 filesystem.

The object is streamed to the parser; gzip and bzip2 compressed objects (recognized by their magic bytes) are decompressed on the fly.

Credentials, region and endpoint are those of Arrow's S3 filesystem: the AWS default credential chain (e.g. the environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_DEFAULT_REGION, ~/.aws) and the query parameters of the URI, e.g. s3://bucket/key?region=eu-central-1, or s3://bucket/key?endpoint_override=localhost:9000&scheme=http for an S3-compatible server.

Needs Arrow with S3 support (as in OpenMS' vcpkg build, arrow[s3]); see isAvailable().

Constructor & Destructor Documentation

S3InputSource() [1/2]

S3InputSource ( const std::string &  uri )

explicit

Opens the S3 object uri.

Parameters

[in]

uri

An S3 URI (s3://bucket/key, optionally with query parameters, see above)

Exceptions

Exception::IllegalArgument	if uri is no S3 URI or OpenMS has no S3 support (see isAvailable())
Exception::FileNotFound	if the object cannot be opened (e.g. it does not exist, or access is denied)

~S3InputSource()

~S3InputSource

(

)

S3InputSource() [2/2]

S3InputSource ( const S3InputSource &  )

delete

Member Function Documentation

isAvailable()

static bool isAvailable ( )

static

Whether OpenMS can read S3 URIs: it was built with an Arrow library that has S3 support.

isS3URI()

static bool isS3URI ( const std::string &  uri )

static

Whether uri is an S3 URI (starts with "s3://")

makeStream()

std::unique_ptr< InputStream > makeStream

(

)

const

A stream of the (decompressed) content of the object, for the parser.

Read errors of the stream throw Exception::IOException; corrupt compressed data throws Exception::ParseError.

Exceptions

Exception::IOException

if the object cannot be read

objectSize()

static Int64 objectSize ( const std::string &  uri )

static

The size of the S3 object uri, in bytes.

Exceptions

Exception::IllegalArgument,Exception::FileNotFound

as the constructor

operator=()

S3InputSource & operator= ( const S3InputSource &  )

delete

readRange()

static std::string readRange ( const std::string &  uri, Int64  offset, Int64  length  )

static

The bytes [offset, offset + length) of the S3 object uri (fewer at its end).

Exceptions

Exception::IllegalArgument,Exception::FileNotFound	as the constructor
Exception::IOException	if the bytes cannot be read

Member Data Documentation

impl_

std::unique_ptr<Impl> impl_

private

The opened object (Arrow types stay out of this header)