ParquetTableStream Class Reference

A Parquet file written table by table: the merged QPX psm file of a multi-file search, appended run by run while the next run is searched. More...

#include <OpenMS/FORMAT/ArrowIOHelpers.h>

Public Member Functions
	ParquetTableStream (std::string filename, const ParquetWriteConfig &config, std::shared_ptr< const arrow::KeyValueMetadata > metadata)
	~ParquetTableStream ()
	ParquetTableStream (const ParquetTableStream &)=delete
ParquetTableStream &	operator= (const ParquetTableStream &)=delete
bool	append (const std::shared_ptr< arrow::Table > &table, int threads=0)
bool	close (const std::map< std::string, std::string > &keys={})
	Ends the file with keys added to its footer. False (logged) if that failed (file removed, stream failed).
void	abandon ()
	Ends an unfinished stream as failed and removes its file (nothing after close())
Size	rows () const
	The rows appended so far.
bool	failed () const
	A write was refused or failed (the file is gone)
bool	closed () const
	The file is written (close() succeeded)
const std::string &	filename () const
	The file.

Private Attributes
std::unique_ptr< Impl >	impl_

Detailed Description

A Parquet file written table by table: the merged QPX psm file of a multi-file search, appended run by run while the next run is searched.

append() opens the file at the first table, with that table's schema and metadata as its footer (built once, e.g. by qpxFileMetadata()), and writes the rows of every table into the row groups writeTableToParquet() would cut for the concatenation of the tables (row_group_size rows per group, a group continued across tables), the columns of a group encoded in parallel (as writeTableToParquet() with threads > 1, on a pool of the stream whose size follows the threads of each append()). The later tables must have the schema of the first (metadata aside). A QPX table (the metadata names the view) is validated per table by one QPXValueValidation, which keeps the primary keys of the tables before: a key twice across tables is refused as it is in the concatenated table. close() adds keys to the footer (keys known only after the last table, e.g. scan_format) and ends the file. A failed append() or close() removes the file, logged; the stream is then done (failed()). Without an append(), close() writes nothing. The destructor removes an unfinished file. One thread at a time.

Constructor & Destructor Documentation

ParquetTableStream() [1/2]

ParquetTableStream	(	std::string	filename,
		const ParquetWriteConfig &	config,
		std::shared_ptr< const arrow::KeyValueMetadata >	metadata
	)

The file, the writer configuration (threads: of the validation and encoding until an append() says otherwise) and the footer (the schema metadata of the first table is replaced by it; null: the first table's own)

~ParquetTableStream()

~ParquetTableStream

(

)

ParquetTableStream() [2/2]

ParquetTableStream ( const ParquetTableStream &  )

delete

Member Function Documentation

abandon()

void abandon

(

)

Ends an unfinished stream as failed and removes its file (nothing after close())

append()

bool append	(	const std::shared_ptr< arrow::Table > &	table,
		int	threads = 0
	)

Appends the rows of table (validated, encoded with threads; 0: the threads of the configuration). False (logged) if refused, null, or the write failed (also by an exception, caught here): the file is removed, the stream has failed.

close()

bool close

(

const std::map< std::string, std::string > &

keys = {}

)

Ends the file with keys added to its footer. False (logged) if that failed (file removed, stream failed).

closed()

bool closed

(

)

const

The file is written (close() succeeded)

failed()

bool failed

(

)

const

A write was refused or failed (the file is gone)

filename()

const std::string & filename

(

)

const

The file.

operator=()

ParquetTableStream & operator= ( const ParquetTableStream &  )

delete

rows()

Size rows

(

)

const

The rows appended so far.

Member Data Documentation

impl_

std::unique_ptr<Impl> impl_

private