HTTPInputSource Class Reference

Source for reading XML files from HTTP, HTTPS and FTP URLs, with libcurl. More...

#include <OpenMS/FORMAT/DATAACCESS/HTTPInputSource.h>

Public Member Functions
	HTTPInputSource (const std::string &url)
	A source for url.
std::unique_ptr< InputStream >	makeStream () const
	Starts a transfer of the resource and returns a stream of its (decompressed) content, for the parser.

Static Public Member Functions
static Int64	objectSize (const std::string &url)
	The size of the resource at url, in bytes, as reported by the server.
static std::string	readRange (const std::string &url, Int64 offset, Int64 length)
	The bytes [offset, offset + length) of the resource at url (fewer at its end), with a range request.
static bool	isURL (const std::string &url)
	Whether url is an HTTP, HTTPS or FTP URL (starts with "http://", "https://" or "ftp://")
static std::string	redactURL (const std::string &url)
	url without user info ("user:password@"), query and fragment, for log and error messages.
static std::string	redactURLs (const std::string &text)
	text with every "scheme://" URL in it redacted as by redactURL(), for diagnostics of other libraries.
static std::string	redactURLs (const std::string &text, const std::string &url)
	text redacted as by redactURLs(text), and also without the parts of url that redactURL() removes.

Private Attributes
std::string	url_

Detailed Description

Source for reading XML files from HTTP, HTTPS and FTP URLs, with libcurl.

The resource is streamed to the parser; gzip and bzip2 compressed resources (recognized by their magic bytes) are decompressed on the fly. HTTP transfer compression is requested as gzip (Content-Encoding) and decompressed the same way, as the stream is read, so a highly compressed response is never held in memory at once. A compressed resource (e.g. .mzML.gz) sent with transfer compression is decompressed in both layers.

Redirects to HTTP, HTTPS and FTP URLs are followed. Proxies come from the usual environment variables (http_proxy, https_proxy, no_proxy). file:// URLs are accepted too, but XMLFile reads local files directly.

Constructor & Destructor Documentation

HTTPInputSource()

HTTPInputSource ( const std::string &  url )

explicit

A source for url.

Parameters

[in]

url

An HTTP, HTTPS, FTP or file URL

Exceptions

Exception::IllegalArgument

if url is no such URL

Member Function Documentation

isURL()

static bool isURL ( const std::string &  url )

static

Whether url is an HTTP, HTTPS or FTP URL (starts with "http://", "https://" or "ftp://")

Parameters

[in]

url

A URL or local file name

makeStream()

std::unique_ptr< InputStream > makeStream

(

)

const

Starts a transfer of the resource and returns a stream of its (decompressed) content, for the parser.

Transfer errors of the stream throw Exception::IOException; corrupt compressed data throws Exception::ParseError.

Exceptions

Exception::FileNotFound	if the server reports that the resource does not exist (e.g. HTTP 404)
Exception::IOException	if the transfer fails (e.g. the host cannot be reached)

objectSize()

static Int64 objectSize ( const std::string &  url )

static

The size of the resource at url, in bytes, as reported by the server.

For HTTP and HTTPS, from the response to a request of the first byte, not a HEAD request: a presigned URL (e.g. of S3) is signed for one request method, usually GET.

Parameters

[in]

url

An HTTP, HTTPS, FTP or file URL

Exceptions

Exception::IllegalArgument,Exception::FileNotFound,Exception::IOException

as the constructor and makeStream(); also Exception::IOException if the server does not report the size

readRange()

static std::string readRange ( const std::string &  url, Int64  offset, Int64  length  )

static

The bytes [offset, offset + length) of the resource at url (fewer at its end), with a range request.

From an HTTP server that ignores the range, the bytes before offset are received and discarded.

Parameters

[in]	url	An HTTP, HTTPS, FTP or file URL
[in]	offset	The first byte to read
[in]	length	The number of bytes to read

Exceptions

Exception::IllegalArgument,Exception::FileNotFound,Exception::IOException

as the constructor and makeStream()

redactURL()

static std::string redactURL ( const std::string &  url )

static

url without user info ("user:password@"), query and fragment, for log and error messages.

These parts can hold credentials, e.g. ftp://user:password@host/run.mzML or the signature of a presigned URL. Applies to any "scheme://" URL (e.g. s3:// too). Local file names are returned unchanged, also when they contain '?', '#' or '@'.

Parameters

[in]

url

A URL or local file name

redactURLs() [1/2]

static std::string redactURLs ( const std::string &  text )

static

text with every "scheme://" URL in it redacted as by redactURL(), for diagnostics of other libraries.

Arrow's or libcurl's messages can repeat the URL they were given, credentials included. A URL ends at whitespace, a quote or an angle bracket.

Parameters

[in]

text

A message that may contain URLs

redactURLs() [2/2]

static std::string redactURLs ( const std::string &  text, const std::string &  url  )

static

text redacted as by redactURLs(text), and also without the parts of url that redactURL() removes.

A library can also repeat a part of the URL it was given on its own, e.g. Arrow's "Invalid S3 connection scheme '...'" for the value of the query parameter "scheme". The user name, the password, each query value and the fragment of url (as given and percent-decoded) are replaced by "***" wherever they are a whole word of text, i.e. not part of a longer word.

Parameters

[in]	text	A message about url
[in]	url	The URL the message is about (local file names have no such parts)

Member Data Documentation

url_

std::string url_

private