OpenMS
Loading...
Searching...
No Matches
ArrowIOHelpers.h File Reference
#include <OpenMS/config.h>
#include <OpenMS/CONCEPT/Types.h>
#include <OpenMS/DATASTRUCTURES/StringUtils.h>
#include <OpenMS/FORMAT/MSExperimentArrowExport.h>
#include <cstdint>
#include <map>
#include <memory>
#include <optional>
#include <string>
#include <utility>
#include <variant>
#include <unordered_set>
#include <vector>

Go to the source code of this file.

Classes

struct  ParquetColumn
 One column for withColumns(), built without Arrow types. More...
 
struct  ParquetColumn::Score
 An entry of a SCORE_LIST row: the struct of the QPX additional_scores (QPXPSMSchema::additionalScoresType()) More...
 
struct  ParquetColumn::StructField
 
struct  ParquetColumn::StructValue
 
class  ParquetTableStream
 A Parquet file written table by table: the merged QPX psm file of a multi-file search, appended run by run while the next run is searched. More...
 
struct  QPXRunFileNameKeys
 MetaInfo indices qpxPsmRunFileName() consults, resolved once. More...
 

Namespaces

namespace  arrow
 
namespace  OpenMS
 Main OpenMS namespace.
 
namespace  OpenMS::MS1LabelState
 The label state of an identification in MS1-labeled (SILAC, Dimethyl, ...) data.
 
namespace  OpenMS::ArrowIOHelpers
 Public helpers for writing and concatenating Arrow tables to Parquet files.
 

Functions

std::string generateUuidV4 ()
 Generate a lowercase hyphenated RFC 4122 version-4 UUID string.
 
Size tableRowCount (const std::shared_ptr< arrow::Table > &table)
 Number of rows in an Arrow table, or 0 when it is null.
 
bool writeTableToParquet (const std::shared_ptr< arrow::Table > &table, const std::string &filename, const ParquetWriteConfig &config=ParquetWriteConfig{})
 Write an Arrow table to a Parquet file.
 
std::shared_ptr< arrow::Table > withColumns (const std::shared_ptr< arrow::Table > &table, const std::vector< ParquetColumn > &columns)
 The table with columns added: a column of the name of an existing one replaces it (in its place), any other is appended. The schema metadata of table is kept.
 
std::shared_ptr< arrow::Table > tableOfColumns (const std::vector< ParquetColumn > &columns, Size rows)
 A table of columns (in their order, without schema metadata), for callers outside libOpenMS that cannot use Arrow themselves (TOPP tools), e.g. a QPX metadata view (run, provenance)
 
bool concatenateAndWriteToParquet (const std::vector< std::shared_ptr< arrow::Table > > &tables, const std::string &filename, const ParquetWriteConfig &config=ParquetWriteConfig{}, const std::shared_ptr< const arrow::KeyValueMetadata > &metadata=nullptr)
 Concatenate a vector of Arrow tables and write the result to a Parquet file.
 
bool validateQPXTable (const std::shared_ptr< arrow::Table > &table, const std::string &filename, const ParquetWriteConfig &config=ParquetWriteConfig{}, const std::shared_ptr< const arrow::KeyValueMetadata > &metadata=nullptr)
 The QPX value validation of writeTableToParquet() alone (QPXValueValidation of the view the table's file_type metadata names; a table without QPX metadata passes)
 
std::shared_ptr< const arrow::KeyValueMetadata > qpxFileMetadata (const std::string &file_type, const ParquetWriteConfig &config=ParquetWriteConfig{}, const std::map< std::string, std::string > &extra={})
 Build the canonical QPX file-level key-value metadata.
 
std::string qpxScanFormat (const std::string &native_id)
 Classify a spectrum native ID into a QPX scan_format token.
 
std::string qpxScanFormat (const std::vector< std::string > &native_ids)
 Derive a single QPX scan_format token for a set of native IDs.
 
std::string qpxScanFormat (const std::vector< std::string > &native_ids, int threads)
 
std::string qpxRunFileName (const std::string &ms_run_path)
 Reduce an MS run path to the QPX run_file_name form.
 
std::vector< Int32 > qpxScanComponents (const std::string &spectrum_reference)
 The QPX scan column value for one spectrum reference.
 
std::string qpxPsmRunFileName (const MetaInfoInterface &hit, const MetaInfoInterface &identification, const std::string &resolved_run_file, const QPXRunFileNameKeys &keys)
 The QPX run_file_name the psm view writes for one identification and hit.
 
bool qpxWarnOnRunNameCollisions (const std::string &context, const std::vector< std::string > &ms_run_paths)
 Warn once per export about source files that collapse onto one run_file_name.
 
std::string qpxIntensityLabel (const std::string &column_label, const std::string &channel_name)
 QPX intensities[].label for a ConsensusMap column header.
 
std::map< std::uint64_t, std::string > qpxIntensityLabels (const ConsensusMap &cmap)
 Derive canonical QPX intensity labels for every ConsensusMap column.
 
bool qpxIsCanonicalIntensityLabel (const std::string &label)
 Test whether a string is in the canonical SDRF/QPX intensity-label vocabulary.
 
std::vector< std::pair< std::string, std::string > > qpxCvParams (const MetaInfoInterface &hit)
 The meta values of an identification that QPX carries as cv_params.
 
std::vector< std::pair< std::string, std::string > > qpxCvParams (const MetaInfoInterface &hit, const MS1LabelState::Keys &keys)
 qpxCvParams() with the registry indices resolved once (MS1LabelState::Keys), for a loop over many hits
 
std::shared_ptr< arrow::Array > getColumn (const std::shared_ptr< arrow::Table > &table, const std::string &name, bool required=true)
 Fetch a named column from a table, combining chunks if needed.
 
std::string getStringValue (const std::shared_ptr< arrow::Array > &array, int64_t row)
 Read a string at row, or "" if null/out-of-bounds.
 
double getDoubleValue (const std::shared_ptr< arrow::Array > &array, int64_t row, double default_val=0.0)
 Read a double at row, or default_val if null.
 
float getFloatValue (const std::shared_ptr< arrow::Array > &array, int64_t row, float default_val=0.0f)
 Read a float at row, or default_val if null.
 
int32_t getInt32Value (const std::shared_ptr< arrow::Array > &array, int64_t row, int32_t default_val=0)
 Read an int32 at row, or default_val if null.
 
int64_t getInt64Value (const std::shared_ptr< arrow::Array > &array, int64_t row, int64_t default_val=0)
 Read an int64 at row, or default_val if null.
 
bool getBoolValue (const std::shared_ptr< arrow::Array > &array, int64_t row, bool default_val=false)
 Read a bool at row, or default_val if null.
 
bool isNull (const std::shared_ptr< arrow::Array > &array, int64_t row)
 Whether array is null at row (or unset)
 
std::string metaValueString (const DataValue &value)
 The text a metavalue is written as in a list<struct{name,value,value_type}> column.
 
void readMetaValues (const std::shared_ptr< arrow::Array > &array, int64_t row, MetaInfoInterface &target, const std::unordered_set< std::string > &excluded_keys={})
 Read metavalues from a list<struct{name,value,value_type}> column.
 

Class Documentation

◆ OpenMS::ArrowIOHelpers::ParquetColumn::Score

struct OpenMS::ArrowIOHelpers::ParquetColumn::Score

An entry of a SCORE_LIST row: the struct of the QPX additional_scores (QPXPSMSchema::additionalScoresType())

Class Members
optional< bool > higher_better none: unknown (null)
string name
double value = 0.0