Public helpers for writing and concatenating Arrow tables to Parquet files. More...

Classes
struct	ParquetColumn
	One column for withColumns(), built without Arrow types. More...

class	ParquetTableStream
	A Parquet file written table by table: the merged QPX psm file of a multi-file search, appended run by run while the next run is searched. More...

struct	QPXRunFileNameKeys
	MetaInfo indices qpxPsmRunFileName() consults, resolved once. More...

Functions
std::string	generateUuidV4 ()
	Generate a lowercase hyphenated RFC 4122 version-4 UUID string.

Size	tableRowCount (const std::shared_ptr< arrow::Table > &table)
	Number of rows in an Arrow table, or 0 when it is null.

bool	writeTableToParquet (const std::shared_ptr< arrow::Table > &table, const std::string &filename, const ParquetWriteConfig &config=ParquetWriteConfig{})
	Write an Arrow table to a Parquet file.

std::shared_ptr< arrow::Table >	withColumns (const std::shared_ptr< arrow::Table > &table, const std::vector< ParquetColumn > &columns)
	The table with `columns` added: a column of the name of an existing one replaces it (in its place), any other is appended. The schema metadata of `table` is kept.

std::shared_ptr< arrow::Table >	tableOfColumns (const std::vector< ParquetColumn > &columns, Size rows)
	A table of `columns` (in their order, without schema metadata), for callers outside libOpenMS that cannot use Arrow themselves (TOPP tools), e.g. a QPX metadata view (run, provenance)

bool	concatenateAndWriteToParquet (const std::vector< std::shared_ptr< arrow::Table > > &tables, const std::string &filename, const ParquetWriteConfig &config=ParquetWriteConfig{}, const std::shared_ptr< const arrow::KeyValueMetadata > &metadata=nullptr)
	Concatenate a vector of Arrow tables and write the result to a Parquet file.

bool	validateQPXTable (const std::shared_ptr< arrow::Table > &table, const std::string &filename, const ParquetWriteConfig &config=ParquetWriteConfig{}, const std::shared_ptr< const arrow::KeyValueMetadata > &metadata=nullptr)
	The QPX value validation of writeTableToParquet() alone (QPXValueValidation of the view the table's `file_type` metadata names; a table without QPX metadata passes)

std::shared_ptr< const arrow::KeyValueMetadata >	qpxFileMetadata (const std::string &file_type, const ParquetWriteConfig &config=ParquetWriteConfig{}, const std::map< std::string, std::string > &extra={})
	Build the canonical QPX file-level key-value metadata.

std::string	qpxScanFormat (const std::string &native_id)
	Classify a spectrum native ID into a QPX `scan_format` token.

std::string	qpxScanFormat (const std::vector< std::string > &native_ids)
	Derive a single QPX `scan_format` token for a set of native IDs.

std::string	qpxScanFormat (const std::vector< std::string > &native_ids, int threads)

std::string	qpxRunFileName (const std::string &ms_run_path)
	Reduce an MS run path to the QPX `run_file_name` form.

std::vector< Int32 >	qpxScanComponents (const std::string &spectrum_reference)
	The QPX `scan` column value for one spectrum reference.

std::string	qpxPsmRunFileName (const MetaInfoInterface &hit, const MetaInfoInterface &identification, const std::string &resolved_run_file, const QPXRunFileNameKeys &keys)
	The QPX `run_file_name` the psm view writes for one identification and hit.

bool	qpxWarnOnRunNameCollisions (const std::string &context, const std::vector< std::string > &ms_run_paths)
	Warn once per export about source files that collapse onto one `run_file_name`.

std::string	qpxIntensityLabel (const std::string &column_label, const std::string &channel_name)
	QPX `intensities`[].label for a ConsensusMap column header.

std::map< std::uint64_t, std::string >	qpxIntensityLabels (const ConsensusMap &cmap)
	Derive canonical QPX intensity labels for every ConsensusMap column.

bool	qpxIsCanonicalIntensityLabel (const std::string &label)
	Test whether a string is in the canonical SDRF/QPX intensity-label vocabulary.

std::vector< std::pair< std::string, std::string > >	qpxCvParams (const MetaInfoInterface &hit)
	The meta values of an identification that QPX carries as `cv_params`.

std::vector< std::pair< std::string, std::string > >	qpxCvParams (const MetaInfoInterface &hit, const MS1LabelState::Keys &keys)
	qpxCvParams() with the registry indices resolved once (MS1LabelState::Keys), for a loop over many hits

std::shared_ptr< arrow::Array >	getColumn (const std::shared_ptr< arrow::Table > &table, const std::string &name, bool required, const char *context)
	Fetch a named column from a table, combining chunks if needed.

std::shared_ptr< arrow::Array >	getColumn (const std::shared_ptr< arrow::Table > &table, const std::string &name, bool required=true)
	getColumn() with "ArrowIOHelpers" as the log context

std::string	getStringValue (const std::shared_ptr< arrow::Array > &array, int64_t row)
	Read a string at `row`, or "" if null/out-of-bounds.

double	getDoubleValue (const std::shared_ptr< arrow::Array > &array, int64_t row, double default_val=0.0)
	Read a double at `row`, or `default_val` if null.

float	getFloatValue (const std::shared_ptr< arrow::Array > &array, int64_t row, float default_val=0.0f)
	Read a float at `row`, or `default_val` if null.

int32_t	getInt32Value (const std::shared_ptr< arrow::Array > &array, int64_t row, int32_t default_val=0)
	Read an int32 at `row`, or `default_val` if null.

int64_t	getInt64Value (const std::shared_ptr< arrow::Array > &array, int64_t row, int64_t default_val=0)
	Read an int64 at `row`, or `default_val` if null.

bool	getBoolValue (const std::shared_ptr< arrow::Array > &array, int64_t row, bool default_val=false)
	Read a bool at `row`, or `default_val` if null.

bool	isNull (const std::shared_ptr< arrow::Array > &array, int64_t row)
	Whether `array` is null at `row` (or unset)

std::string	metaValueString (const DataValue &value)
	The text a metavalue is written as in a list<struct{name,value,value_type}> column.

void	readMetaValues (const std::shared_ptr< arrow::Array > &array, int64_t row, MetaInfoInterface &target, const std::unordered_set< std::string > &excluded_keys={})
	Read metavalues from a list<struct{name,value,value_type}> column.

Detailed Description

Public helpers for writing and concatenating Arrow tables to Parquet files.

TOPP tools link against libOpenMS (which exports these helpers) but not directly against Arrow/Parquet. These wrappers keep all Arrow/Parquet API calls inside libOpenMS so downstream binaries don't need to import Arrow symbols.

Function Documentation

◆ concatenateAndWriteToParquet()

bool concatenateAndWriteToParquet	(	const std::vector< std::shared_ptr< arrow::Table > > &	tables,
		const std::string &	filename,
		const ParquetWriteConfig &	config = `ParquetWriteConfig{}`,
		const std::shared_ptr< const arrow::KeyValueMetadata > &	metadata = `nullptr`
	)

Concatenate a vector of Arrow tables and write the result to a Parquet file.

All tables must share the same schema. An empty input vector is a no-op (returns true without writing).

Parameters

[in]	tables	Vector of Arrow tables to concatenate (must share schema)
[in]	filename	Output file path
[in]	config	Parquet writer configuration
[in]	metadata	Schema key-value metadata for the merged file. Concatenation otherwise inherits the first input's metadata, i.e. that table's identity; pass a fresh one (see qpxFileMetadata()) so the merged file is its own. `nullptr` keeps whatever the concatenation produced.

Returns: true on success (or if tables is empty), false on error

◆ generateUuidV4()

std::string generateUuidV4 ( )

Generate a lowercase hyphenated RFC 4122 version-4 UUID string.

Used by QPX Parquet exporters when attaching file metadata.

Returns: UUID string, e.g. "550e8400-e29b-41d4-a716-446655440000"

◆ getBoolValue()

bool getBoolValue	(	const std::shared_ptr< arrow::Array > &	array,
		int64_t	row,
		bool	default_val = `false`
	)

Read a bool at row, or default_val if null.

◆ getColumn() [1/2]

std::shared_ptr< arrow::Array > getColumn	(	const std::shared_ptr< arrow::Table > &	table,
		const std::string &	name,
		bool	required,
		const char *	context
	)

Fetch a named column from a table, combining chunks if needed.

Returns nullptr if the column is missing or contains no chunks. When required is true, missing columns are logged as errors. context (e.g. the calling class) prefixes the log messages.

◆ getColumn() [2/2]

std::shared_ptr< arrow::Array > getColumn	(	const std::shared_ptr< arrow::Table > &	table,
		const std::string &	name,
		bool	required = `true`
	)

getColumn() with "ArrowIOHelpers" as the log context

◆ getDoubleValue()

double getDoubleValue	(	const std::shared_ptr< arrow::Array > &	array,
		int64_t	row,
		double	default_val = `0.0`
	)

Read a double at row, or default_val if null.

◆ getFloatValue()

float getFloatValue	(	const std::shared_ptr< arrow::Array > &	array,
		int64_t	row,
		float	default_val = `0.0f`
	)

Read a float at row, or default_val if null.

◆ getInt32Value()

int32_t getInt32Value	(	const std::shared_ptr< arrow::Array > &	array,
		int64_t	row,
		int32_t	default_val = `0`
	)

Read an int32 at row, or default_val if null.

◆ getInt64Value()

int64_t getInt64Value	(	const std::shared_ptr< arrow::Array > &	array,
		int64_t	row,
		int64_t	default_val = `0`
	)

Read an int64 at row, or default_val if null.

◆ getStringValue()

std::string getStringValue	(	const std::shared_ptr< arrow::Array > &	array,
		int64_t	row
	)

Read a string at row, or "" if null/out-of-bounds.

◆ isNull()

bool isNull	(	const std::shared_ptr< arrow::Array > &	array,
		int64_t	row
	)

Whether array is null at row (or unset)

◆ metaValueString()

std::string metaValueString ( const DataValue & value )

The text a metavalue is written as in a list<struct{name,value,value_type}> column.

The same as DataValue::toString(), except for doubles (also in lists): they are written in their shortest form that reads back as the same double, so that readMetaValues() restores them exactly. DataValue::toString() rounds values between 0.01 and 10000 to 15 decimals, which loses the last digits of e.g. 1.2345678901234567.

◆ qpxCvParams() [1/2]

std::vector< std::pair< std::string, std::string > > qpxCvParams ( const MetaInfoInterface & hit )

The meta values of an identification that QPX carries as cv_params.

QPX reserves cv_params, a list of {cv_name, cv_value} pairs, on the feature and psm views for annotations outside the fixed schema. OpenMS uses it for the label state of an MS1-labeled identification, i.e. the MS1LabelState meta values MS1Label:labeled_sequence, MS1Label:removed_labels and MS1Label:channel that MS1LabeledWorkflow records on the PeptideHit.

Only these keys are exported, so identifications without a label state (label-free, isobaric) keep a null cv_params, as before.

Parameters

[in] hit The PeptideHit (or any MetaInfoInterface) to read the keys from

Returns: (cv_name, cv_value) pairs in the order above, only for the keys present; empty if none is

◆ qpxCvParams() [2/2]

std::vector< std::pair< std::string, std::string > > qpxCvParams	(	const MetaInfoInterface &	hit,
		const MS1LabelState::Keys &	keys
	)

qpxCvParams() with the registry indices resolved once (MS1LabelState::Keys), for a loop over many hits

◆ qpxFileMetadata()

std::shared_ptr< const arrow::KeyValueMetadata > qpxFileMetadata	(	const std::string &	file_type,
		const ParquetWriteConfig &	config = `ParquetWriteConfig{}`,
		const std::map< std::string, std::string > &	extra = `{}`
	)

Build the canonical QPX file-level key-value metadata.

Writes the keys defined by the QPX serialization spec: qpx_version, file_type, creator, software_provider, creation_date (ISO 8601), compression_format and uuid, plus any extra keys.

Build this once per output file and reuse the returned object for both the writer schema and every batch — each call mints a fresh uuid and creation_date, so calling it per batch would produce mismatched schemas.

For a known file_type it also stamps primary_key and identity_composite, the declaration a reader needs to re-derive the view's opaque row ids (see QPXIdentity). Done here rather than per exporter so that a producer cannot emit ids without saying what they were derived from.

Parameters

[in]	file_type	QPX view token: `"psm_file"`, `"feature_file"` or `"pg_file"`
[in]	config	Write configuration; supplies `compression_format`
[in]	extra	Additional keys, e.g. `{{"scan_format", "scan"}}`

Returns: The metadata, or nullptr if config selects a compression QPX does not define (LZ4). Callers must treat nullptr as a write failure.

◆ qpxIntensityLabel()

std::string qpxIntensityLabel	(	const std::string &	column_label,
		const std::string &	channel_name
	)

QPX intensities[].label for a ConsensusMap column header.

The label is a join key: QPX matches intensities[].label against the unnested run.samples[].label of run.parquet (docs/spec/views.md), so it must be a canonical channel token, not a channel index or a file name.

Isobaric channels resolve to the reporter name prefixed by the method family, using OpenMS' own channel names — "TMT126", "TMT131" (TMT10-plex channel 10), "ITRAQ114". Label-free maps resolve to "LFQ". Recognizable SILAC modification labels resolve by mass class; callers that have a whole ConsensusMap should prefer qpxIntensityLabels(), which can distinguish a two-plex second channel (heavy by role) from a three-plex medium channel.

Parameters

[in]	column_label	ConsensusMap::ColumnHeader::label. IsobaricChannelExtractor writes `"<methodname>_<channelname>"` (e.g. `"tmt10plex_126"`); ProteomicsLFQ writes `"label-free"`.
[in]	channel_name	The header's `channel_name` meta value (e.g. `"126"`); when it is absent from an isobaric header, the validated `column_label` suffix is used.

Returns: The label, or "" when the isobaric method, non-isobaric vocabulary token, or SILAC role cannot be identified — writing a guessed token into a join key is worse than writing none, so callers must handle the empty result.

◆ qpxIntensityLabels()

std::map< std::uint64_t, std::string > qpxIntensityLabels ( const ConsensusMap & cmap )

Derive canonical QPX intensity labels for every ConsensusMap column.

Unlike the scalar overload, this sees all columns of one source run. It recognizes the FeatureFinderMultiplex SILAC shapes and maps them to the active SDRF/QPX vocabulary: "SILAC light" / "SILAC heavy" for two-plex and "SILAC light" / "SILAC medium" / "SILAC heavy" for three-plex. The mapping is by channel role, so a two-plex Arg6 channel is correctly called heavy even though the same modification is the medium channel in the standard three-plex.

Parameters

[in] cmap ConsensusMap whose column headers are labelled

Returns: Map index to canonical label. An unrepresentable header maps to "" and is logged; exporters must refuse it.

◆ qpxIsCanonicalIntensityLabel()

bool qpxIsCanonicalIntensityLabel ( const std::string & label )

Test whether a string is in the canonical SDRF/QPX intensity-label vocabulary.

Covers every isobaric method supported by OpenMS plus the QPX LFQ, SILAC, mTRAQ, and dimethyl plex labels. Intended for write-time value validation, after producer-specific labels have been normalized.

Parameters

[in] label Candidate label

Returns: true if label is a canonical QPX channel token

◆ qpxPsmRunFileName()

std::string qpxPsmRunFileName	(	const MetaInfoInterface &	hit,
		const MetaInfoInterface &	identification,
		const std::string &	resolved_run_file,
		const QPXRunFileNameKeys &	keys
	)

The QPX run_file_name the psm view writes for one identification and hit.

Prefers a per-hit or per-identification file reference over the identification run's resolved path, then reduces it with qpxRunFileName().

Shared with the feature view for the same reason as qpxScanComponents(): the feature view has to reproduce this exactly to decide which of its rows a given PSM belongs to.

The metavalue keys are passed as pre-resolved MetaInfoRegistry indices rather than looked up by name: MetaInfoRegistry::getIndex() takes an omp critical lock, and this runs once per row inside the exporters' parallel batch build, where a per-row lock would serialize it. Resolve them once outside the row loop with QPXRunFileNameKeys.

Parameters

[in]	hit	The PSM's hit; consulted for `reference_file_name` / `run_file_name`
[in]	identification	The PSM; consulted for `reference_file_name`
[in]	resolved_run_file	Fallback path from IdentifierMSRunMapper::getPrimaryMSRunPath()
[in]	keys	Pre-resolved metavalue indices

Returns: The bare run name, or empty when no source could be resolved at all

◆ qpxRunFileName()

std::string qpxRunFileName ( const std::string & ms_run_path )

Reduce an MS run path to the QPX run_file_name form.

QPX defines run_file_name as the raw data file name without extension, and uses it as a primary-key component in the psm, feature and pg views (and to join them against run.parquet). The full name with extension belongs in run.file_name.

Parameters

[in] ms_run_path Source path, e.g. /data/proj/S1_Frontal_1.mzML

Returns: The stem, e.g. S1_Frontal_1; empty input yields an empty result.

Referenced by QuantificationUnits::QuantificationUnits().

◆ qpxScanComponents()

std::vector< Int32 > qpxScanComponents ( const std::string & spectrum_reference )

The QPX scan column value for one spectrum reference.

QPX stores scan as a list of integer components. OpenMS derives it from the spectrum's native ID, whose format is auto-detected.

Shared by the psm and feature views on purpose. scan is part of both views' identity composites, so the id linking a PSM to its feature is only reproducible while both extract the same number from the same reference – two verbatim copies of this would be one edit away from a collection whose cross-references silently stop resolving.

Parameters

[in] spectrum_reference Native ID, e.g. controllerType=0 controllerNumber=1 scan=2075

Returns: The components, in order; empty when the reference is empty or carries no recognizable scan number (which is a legitimate value, e.g. for an unidentified feature)

◆ qpxScanFormat() [1/3]

std::string qpxScanFormat ( const std::string & native_id )

Classify a spectrum native ID into a QPX scan_format token.

Parameters

[in] native_id A spectrum native ID (e.g. controllerType=0 ... scan=1234)

Returns: "index" for index= IDs, "scan" for other recognized native IDs, or "" when the convention cannot be determined.

See also: qpxScanFormat(const std::vector<std::string>&)

◆ qpxScanFormat() [2/3]

std::string qpxScanFormat ( const std::vector< std::string > & native_ids )

Derive a single QPX scan_format token for a set of native IDs.

Unrecognized IDs are ignored. Returns "" when no ID is recognized, or when the inputs disagree — mixed conventions are reported once via the log rather than guessed at, so an ambiguous export omits scan_format instead of mislabeling it.

◆ qpxScanFormat() [3/3]

std::string qpxScanFormat	(	const std::vector< std::string > &	native_ids,
		int	threads
	)

qpxScanFormat(native_ids) checking the IDs in threads ranges in parallel (OpenMP; 1: serial): the same result and the same warning (the first pair of conventions in ID order)

◆ qpxWarnOnRunNameCollisions()

bool qpxWarnOnRunNameCollisions	(	const std::string &	context,
		const std::vector< std::string > &	ms_run_paths
	)

Warn once per export about source files that collapse onto one run_file_name.

QPX defines the column as the file name without path or extension, so two distinct paths sharing a stem (/a/run1.mzML and /b/run1.mzML) become indistinguishable as a join and partition key. Same-named files in different directories are a legitimate layout and the origin is known – only the spec's representation cannot express it – so this warns rather than failing, matching the policy the PSM exporter already applies.

Parameters

[in]	context	Caller name for the log line
[in]	ms_run_paths	Source paths participating in one QPX collection

Returns: true if every distinct path yields a distinct run name

◆ readMetaValues()

void readMetaValues	(	const std::shared_ptr< arrow::Array > &	array,
		int64_t	row,
		MetaInfoInterface &	target,
		const std::unordered_set< std::string > &	excluded_keys = `{}`
	)

Read metavalues from a list<struct{name,value,value_type}> column.

Decodes typed entries (int, double/float, *_list, string) and assigns them to target. Keys in excluded_keys are skipped.

◆ tableOfColumns()

std::shared_ptr< arrow::Table > tableOfColumns	(	const std::vector< ParquetColumn > &	columns,
		Size	rows
	)

A table of columns (in their order, without schema metadata), for callers outside libOpenMS that cannot use Arrow themselves (TOPP tools), e.g. a QPX metadata view (run, provenance)

Parameters

[in]	columns	The columns, each with `rows` entries
[in]	rows	The number of rows

Returns: the table, or null on an Arrow error (logged)

Exceptions

Exception::IllegalArgument if a column has another number of rows

◆ tableRowCount()

Size tableRowCount ( const std::shared_ptr< arrow::Table > & table )

Number of rows in an Arrow table, or 0 when it is null.

Exists so that a TOPP tool can ask "did this table come out empty?" without calling an Arrow member function itself. Arrow is deliberately confined to libOpenMS' implementation (see the note on ParquetTableComparator), and on Windows its symbols are dllimport, so a tool that calls arrow::Table::num_rows() directly fails to link there while building fine on Linux.

Parameters

[in] table the table to measure; may be null

Returns: the row count, or 0 for a null table

◆ validateQPXTable()

bool validateQPXTable	(	const std::shared_ptr< arrow::Table > &	table,
		const std::string &	filename,
		const ParquetWriteConfig &	config = `ParquetWriteConfig{}`,
		const std::shared_ptr< const arrow::KeyValueMetadata > &	metadata = `nullptr`
	)

The QPX value validation of writeTableToParquet() alone (QPXValueValidation of the view the table's file_type metadata names; a table without QPX metadata passes)

For a table that is written more than once (a run's own file and the merged file of a single run): validated here once, with config.threads, then written with config.validated set. A refused table is logged as by the write.

Parameters

[in]	table	The table (must not be null)
[in]	filename	The file it is for (in the log)
[in]	config	`qpx_extra_columns` and `threads`
[in]	metadata	The footer the table is written with (concatenateAndWriteToParquet()): it names the view (`file_type`); a table built by an exporter carries none itself, so without it nothing is checked

Returns: true if the table may be written

◆ withColumns()

std::shared_ptr< arrow::Table > withColumns	(	const std::shared_ptr< arrow::Table > &	table,
		const std::vector< ParquetColumn > &	columns
	)

The table with columns added: a column of the name of an existing one replaces it (in its place), any other is appended. The schema metadata of table is kept.

For callers outside libOpenMS that cannot use Arrow themselves (TOPP tools), e.g. to add optional columns to a QPX view before writing it.

Parameters

[in]	table	The table (must not be null)
[in]	columns	The columns, each with one entry per row of `table`

Returns: the new table, or null on an Arrow error (logged)

Exceptions

Exception::IllegalArgument if a column has another number of rows than table

◆ writeTableToParquet()

bool writeTableToParquet	(	const std::shared_ptr< arrow::Table > &	table,
		const std::string &	filename,
		const ParquetWriteConfig &	config = `ParquetWriteConfig{}`
	)

Write an Arrow table to a Parquet file.

Parameters

[in]	table	The Arrow table to write (must not be null)
[in]	filename	Output file path
[in]	config	Parquet writer configuration (compression, row group size, ...)

Returns: true on success, false on error (errors are logged)

Classes

Functions

Detailed Description

Function Documentation

◆ concatenateAndWriteToParquet()

◆ generateUuidV4()

◆ getBoolValue()

◆ getColumn() [1/2]

◆ getColumn() [2/2]

◆ getDoubleValue()

◆ getFloatValue()

◆ getInt32Value()

◆ getInt64Value()

◆ getStringValue()

◆ isNull()

◆ metaValueString()

◆ qpxCvParams() [1/2]

◆ qpxCvParams() [2/2]

◆ qpxFileMetadata()

◆ qpxIntensityLabel()

◆ qpxIntensityLabels()

◆ qpxIsCanonicalIntensityLabel()

◆ qpxPsmRunFileName()

◆ qpxRunFileName()

◆ qpxScanComponents()

◆ qpxScanFormat() [1/3]

◆ qpxScanFormat() [2/3]

◆ qpxScanFormat() [3/3]

◆ qpxWarnOnRunNameCollisions()

◆ readMetaValues()

◆ tableOfColumns()

◆ tableRowCount()

◆ validateQPXTable()

◆ withColumns()

◆ writeTableToParquet()