The parts of a QPX dataset that a search engine adds to the views of QPXFile. More...

#include <OpenMS/FORMAT/QPXDatasetExport.h>

Classes
struct	MzView

struct	PeakFilter
	The peak filter of the mz view: the `top` most intense peaks per `window` Th (`top` 0: all peaks), and the annotated ones (keptPeaks()) More...

struct	ProvenanceStep
	A step of the processing (a row of the provenance view, provenanceView()) More...

struct	Run
	A run of the dataset (a row of the run view, runView()) More...

struct	RunSample
	A sample of a run (the run view's `samples`) More...

struct	Sample
	A sample of the dataset (a row of the sample view, sampleView()) More...

struct	View
	A view of the dataset for writeDataset() More...

Public Types
using	Column = ArrowIOHelpers::ParquetColumn
	A column of a QPX view, without Arrow types.

Static Public Member Functions
static std::string	scoreName (const std::string &name)
	The name of a score or meta value in the snake_case of the QPX `additional_scores`.

static std::vector< Column >	psmColumns (const PeptideIdentificationList &peptide_ids, const std::vector< bool > &has_spectrum, bool isotope_observed_minus_theoretical)
	The optional columns of the QPX psm view, one entry per row of the view (the best hit of each element of `peptide_ids` with hits; elements without hits have no row)

static std::vector< Size >	keptPeaks (const MSSpectrum &spectrum, const std::vector< PeptideHit::PeakAnnotation > &annotations, int precursor_charge, const PeakFilter &filter)
	The indices of the peaks of `spectrum` that `filter` keeps: the most intense ones per window and those of the fragment annotation `annotations`; all peaks if `filter.top` is 0.

static MzView	mzView (const MSExperiment &spectra, const PeptideIdentificationList &peptide_ids, const std::string &run_file_name, const PeakFilter &filter)
	The QPX mz view of the MS2 spectra of the PSMs of one run, with the peaks `filter` keeps.

static std::shared_ptr< arrow::Table >	runView (const std::vector< Run > &runs)
	The QPX run view: one row per run, in the order of `runs`.

static std::shared_ptr< arrow::Table >	sampleView (const std::vector< Sample > &samples)
	The QPX sample view: one row per sample, in the order of `samples`.

static std::shared_ptr< arrow::Table >	provenanceView (const std::vector< ProvenanceStep > &steps)
	The QPX provenance view: one row per step, in the order of `steps`.

static std::string	paramToJSON (const Param &param)
	`param` as one JSON object of its full parameter names (e.g. "Search:precursor:mass_tolerance_lower") and values, in the order of the parameters (e.g. for the `config` of a ProvenanceStep)

static std::string	formatNumber (double value, const std::string &non_finite="null")
	`value` as a locale-independent number: the shortest of 15 to 17 significant digits that reads back as `value` (e.g. 0.1 -> "0.1", 100.0 -> "100")

static bool	writeDataset (const std::string &directory, const std::string &prefix, const std::map< std::string, View > &views, std::set< std::string > *failed=nullptr)
	Write the QPX dataset `views` to `directory:` `directory/prefix.VIEW.parquet per` view (VIEW: its name)

Detailed Description

The parts of a QPX dataset that a search engine adds to the views of QPXFile.

A tool that writes a QPX dataset (e.g. ProSE with -out_qpx) takes the mandatory columns of the psm view from QPXFile and adds the optional ones of this class with ArrowIOHelpers::withColumns(). It writes the mz view of its identified spectra with mzView(), the run, sample and provenance views with runView(), sampleView() and provenanceView(), and all views of the dataset with writeDataset(). Columns are ArrowIOHelpers::ParquetColumn and tables are only forward-declared, so the caller needs no Arrow headers.

All members are static.

Experimental classes:: This API is experimental and may change in future versions.

Class Documentation

◆ OpenMS::QPXDatasetExport::MzView

struct OpenMS::QPXDatasetExport::MzView

The QPX mz view of the spectra of the PSMs of one run (mzView()). It is of that run only: its key id, the native ID, is unique within the run, not across runs (writeDataset() refuses an mz view with an id twice)

Class Members
vector< bool >	has_spectrum	Per element of the peptide identifications whether the view has the row of its spectrum (false for an element without hits, and for all if `table` is null)
Size	peaks_kept = 0	The peaks of the rows (kept by the peak filter)
Size	peaks_read = 0	The peaks of the spectra of the rows.
shared_ptr< Table >	table	The mz view (null if it could not be built; the error is logged)
Size	unresolved = 0	The PSMs whose native ID has the row of a spectrum that is not theirs: several spectra share the native ID (at different RTs), and the row is of another one than the PSM's, or none of them is at the PSM's RT
Size	without_spectrum = 0	The PSMs without a spectrum of their native ID (and RT), whose native ID has no row either.

◆ OpenMS::QPXDatasetExport::PeakFilter

struct OpenMS::QPXDatasetExport::PeakFilter

The peak filter of the mz view: the top most intense peaks per window Th (top 0: all peaks), and the annotated ones (keptPeaks())

Class Members
bool	cap_at_precursor = true	The deisotoper capped the fragment charge at the precursor's charge.
bool	deisotoped = false	The fragment annotation is of deisotoped spectra (e.g. ProSE's 'Search:fragment:deisotope' with a high-resolution fragment tolerance): the deisotoper moved each fragment of charge z (up to `max_charge`, or the precursor's charge with `cap_at_precursor`) to its 1+ m/z, which is no peak of the spectrum as the file has it.
int	max_charge = 3	The highest fragment charge of the deisotoper.
Size	top = 100	The most intense peaks kept per window; 0: all peaks.
double	window = 100.0	The width of a window (Th)

◆ OpenMS::QPXDatasetExport::ProvenanceStep

struct OpenMS::QPXDatasetExport::ProvenanceStep

A step of the processing (a row of the provenance view, provenanceView())

Class Members
string	category	The `step_category` (e.g. "database_search")
string	config	The `config` (e.g. paramToJSON() of the tool's parameters; "": null)
string	name	The `step_name` (e.g. "peptide_search")
vector< string >	output_views	The `output_views` (e.g. "psm"; empty: null)
vector< pair< string, string > >	parameters	The `parameters`, as key and value, in this order.
string	tool	The `tool_name` (e.g. "ProSE")

◆ OpenMS::QPXDatasetExport::Run

struct OpenMS::QPXDatasetExport::Run

A run of the dataset (a row of the run view, runView())

Class Members
string	accession	The `run_accession`.
optional< string >	acquisition_method	The `acquisition_method` (none or "": null)
optional< string >	dissociation_method	The `dissociation_method` (none or "": null)
vector< string >	enzymes	The `enzymes` (empty: null)
string	file_name	The `file_name` (e.g. the base name of the input file; "": null)
vector< string >	fixed_modifications	The fixed modifications of the search, by their OpenMS names (e.g. "Carbamidomethyl (C)")
optional< string >	fraction	The `fraction` (none or "": null)
optional< string >	instrument	The `instrument` (none or "": null)
string	run_file_name	The `run_file_name` (as in the psm and mz views)
vector< RunSample >	samples	The samples of the run.
vector< string >	variable_modifications	The variable modifications of the search, by their OpenMS names (e.g. "Oxidation (M)")

◆ OpenMS::QPXDatasetExport::RunSample

struct OpenMS::QPXDatasetExport::RunSample

A sample of a run (the run view's samples)

Class Members
string	accession	The sample's `sample_accession` (of the sample view)
optional< Int32 >	biological_replicate	The biological replicate (none: null)
string	label = "LFQ"	The label of the sample in the run ("LFQ" for label-free)
optional< Int32 >	technical_replicate	The technical replicate (none: null)

◆ OpenMS::QPXDatasetExport::Sample

struct OpenMS::QPXDatasetExport::Sample

A sample of the dataset (a row of the sample view, sampleView())

Class Members
string	accession	The `sample_accession`.
string	organism = "not available"	The `organism` ("not available", as SDRF has it, if unknown)
string	organism_part = "not available"	The `organism_part` ("not available", as SDRF has it, if unknown)

◆ OpenMS::QPXDatasetExport::View

struct OpenMS::QPXDatasetExport::View

A view of the dataset for writeDataset()

Class Members
ParquetWriteConfig	config	The writer configuration.
map< string, string >	metadata	Extra keys of the file's footer (ArrowIOHelpers::qpxFileMetadata())
vector< shared_ptr< Table > >	tables	The tables of the view, concatenated (same schema); empty: the view is not written; a null table: the view fails

Member Typedef Documentation

◆ Column

using Column = ArrowIOHelpers::ParquetColumn

A column of a QPX view, without Arrow types.

Member Function Documentation

◆ formatNumber()

static std::string formatNumber	(	double	value,
		const std::string &	non_finite = `"null"`
	)

static

value as a locale-independent number: the shortest of 15 to 17 significant digits that reads back as value (e.g. 0.1 -> "0.1", 100.0 -> "100")

Parameters

[in]	value	The number
[in]	non_finite	The text of a non-finite `value` (NaN, infinity)

Returns: the text

◆ keptPeaks()

static std::vector< Size > keptPeaks	(	const MSSpectrum &	spectrum,
		const std::vector< PeptideHit::PeakAnnotation > &	annotations,
		int	precursor_charge,
		const PeakFilter &	filter
	)

static

The indices of the peaks of spectrum that filter keeps: the most intense ones per window and those of the fragment annotation annotations; all peaks if filter.top is 0.

The windows are jumping, as WindowMower's (a window starts at its first peak, the next at the first peak past it), but the last window keeps the full quota too. An annotated m/z keeps the peak nearest to it (within 0.01 Th). Of deisotoped spectra (filter.deisotoped) it also keeps, for each charge z from 2 to the deisotoper's maximum (filter.max_charge, or precursor_charge if lower and filter.cap_at_precursor), the peak nearest to (m/z + (z - 1) * proton) / z (within 0.01 Th): the monoisotopic peak of a fragment of charge z, which the deisotoper moved to the annotated 1+ m/z. Every annotated fragment thus keeps its peak in the spectrum as the file has it; a 1+ fragment may keep an unrelated peak at its 2+ or 3+ position too.

Parameters

[in]	spectrum	The spectrum, sorted by m/z
[in]	annotations	The fragment annotation of a PSM of `spectrum`
[in]	precursor_charge	The spectrum's recorded precursor charge (as in the search's preprocessing); 0 if unknown (then up to `filter.max_charge`)
[in]	filter	The peak filter

Returns: the indices of the kept peaks, ascending

◆ mzView()

static MzView mzView	(	const MSExperiment &	spectra,
		const PeptideIdentificationList &	peptide_ids,
		const std::string &	run_file_name,
		const PeakFilter &	filter
	)

static

The QPX mz view of the MS2 spectra of the PSMs of one run, with the peaks filter keeps.

The spectrum of a PSM (the best hit of an element of peptide_ids with hits) is the MS2 spectrum of spectra of its native ID; of several spectra of its native ID the one at its RT (within 1e-6 s). The view has one row per native ID (id is its key), in the order of the first PSM of each: the row of the first spectrum of a native ID that a PSM has. A PSM whose spectrum is not found has none, nor has one whose spectrum shares its native ID with the spectrum of an earlier row (unresolved). The peaks of a row are those that keptPeaks() keeps for any of the PSMs of its spectrum (each keeps its own annotated peaks), with the precursor charge of the spectrum's first precursor (not the charge of a PSM: a spectrum of unknown charge was deisotoped up to filter.max_charge, whatever the charge of its PSMs). A spectrum not sorted by m/z is sorted (in a copy).

The view is of one run: its id (the native ID) is unique within the run, but runs share native IDs (e.g. "scan=1"), so the views of several runs are not one mz view. writeDataset() refuses an mz view whose tables have an id twice; write each run's view to a dataset (file) of its own.

The columns, in this order: id, run_file_name, scan (ArrowIOHelpers::qpxScanComponents()), ms_level, centroid, scan_start_time (minutes), inverse_ion_mobility (null unless in Vs/cm^2), ion_injection_time (null without MS:1000927), total_ion_current, precursors, mz, intensity, cv_params (null).

Parameters

[in]	spectra	The spectra of the run (the MS2 spectra of its PSMs suffice; other spectra are left out)
[in]	peptide_ids	The identifications of the run, in the order of the rows of the psm view
[in]	run_file_name	The run's `run_file_name` (e.g. ArrowIOHelpers::qpxRunFileName() of its file)
[in]	filter	The peak filter

Returns: the view, has_spectrum with one entry per element of peptide_ids (for psmColumns()), and its counts

◆ paramToJSON()

static std::string paramToJSON ( const Param & param )

static

param as one JSON object of its full parameter names (e.g. "Search:precursor:mass_tolerance_lower") and values, in the order of the parameters (e.g. for the config of a ProvenanceStep)

Integers and doubles (formatNumber(), non-finite: null) are JSON numbers, lists JSON arrays, an empty value null, and all other values JSON strings.

Parameters

[in] param The parameters

Returns: the JSON object

◆ provenanceView()

static std::shared_ptr< arrow::Table > provenanceView ( const std::vector< ProvenanceStep > & steps )

static

The QPX provenance view: one row per step, in the order of steps.

The columns, in this order: step_order (1, 2, ...), step_category, step_name, tool_name, tool_version (VersionInfo::getVersion()), tool_uri (null), parameters (structs of key and value), config, output_views.

Parameters

[in] steps The steps of the processing

Returns: the view, or null on an Arrow error (logged)

◆ psmColumns()

static std::vector< Column > psmColumns	(	const PeptideIdentificationList &	peptide_ids,
		const std::vector< bool > &	has_spectrum,
		bool	isotope_observed_minus_theoretical
	)

static

The optional columns of the QPX psm view, one entry per row of the view (the best hit of each element of peptide_ids with hits; elements without hits have no row)

The columns, in this order:

additional_scores (SCORE_LIST): the main score and every numeric meta value of the hit, named by scoreName() (each name once, the posterior error probability and non-score meta values such as target_decoy left out), with higher_better for the scores whose direction is known;
spectrum_reference (UTF8): the native ID; null if the element has no spectrum (has_spectrum) or no native ID;
isotope_error (INT16): the meta value isotope_error (or IsotopeError) as observed minus theoretical (also so in additional_scores); null without one, and if it is neither an integer nor a double of an integral value (as read back from a pin file) in the INT16 range;
protein_mappings (STRUCT_LIST of accession, start, end, aa_before, aa_after): one entry per peptide evidence, 1-based positions (null if unknown), "-" for a protein terminus, null for an unknown residue.

Parameters

[in]	peptide_ids	The identifications of one run, in the order of the rows of the psm view
[in]	has_spectrum	Per element of `peptide_ids` whether the mz view has its spectrum; empty: all true
[in]	isotope_observed_minus_theoretical	Whether the isotope error meta values are observed minus theoretical (false: theoretical minus observed, their sign is flipped)

Returns: the four columns

Exceptions

Exception::IllegalArgument if has_spectrum is neither empty nor of the size of peptide_ids

◆ runView()

static std::shared_ptr< arrow::Table > runView ( const std::vector< Run > & runs )

static

The QPX run view: one row per run, in the order of runs.

The columns, in this order: run_accession, run_file_name, file_name, samples (structs of sample_accession, label, biological_replicate, technical_replicate), fraction, instrument, enzymes, dissociation_method, acquisition_method, additional_terms (always null), modification_parameters (QPX MODIFICATION structs of accession, name, fixed, position, target_amino_acid: the fixed, then the variable modifications; accession "UNIMOD:<n>" or, without a UniMod accession, the full ID; a modification unknown to ModificationsDB has its name as accession and name and null position and target_amino_acid). A tool adds optional columns with ArrowIOHelpers::withColumns().

Parameters

[in] runs The runs

Returns: the view, or null on an Arrow error (logged)

◆ sampleView()

static std::shared_ptr< arrow::Table > sampleView ( const std::vector< Sample > & samples )

static

The QPX sample view: one row per sample, in the order of samples.

The columns, in this order: sample_accession, organism, organism_part (all non-null).

Parameters

[in] samples The samples

Returns: the view, or null on an Arrow error (logged)

◆ scoreName()

static std::string scoreName ( const std::string & name )

static

The name of a score or meta value in the snake_case of the QPX additional_scores.

Percolator's scores get their names of bigbio/qpx#355 (e.g. "MS:1001492" -> "percolator_score"), "q-value" is "global_qvalue", others are converted ("ln(hyperscore)" -> "ln_hyperscore", "DeltaMass" -> "delta_mass", "COMET:xcorr" -> "comet_xcorr").

Parameters

[in] name The score type or meta value key

Returns: the QPX name; "" for the posterior error probability, which has a column of its own

◆ writeDataset()

static bool writeDataset	(	const std::string &	directory,
		const std::string &	prefix,
		const std::map< std::string, View > &	views,
		std::set< std::string > *	failed = `nullptr`
	)

static

Write the QPX dataset views to directory: directory/prefix.VIEW.parquet per view (VIEW: its name)

Creates directory if needed. Writes the views at the same time (each file on a thread of its own), each with its tables concatenated and the footer of ArrowIOHelpers::qpxFileMetadata() of the file type "VIEW_file", its config and its metadata. A view without tables is not written; a view with a null table fails. The mz view is not kept (logged; its file is removed) if a psm view was given and could not be written, as its rows are keyed by the spectra of the PSMs. The mz view is of one run (mzView()): its key id, the native ID, is unique only within a run, so an mz view whose tables together have an id twice (e.g. "scan=1" of two runs) fails (the error names the id). A view that fails is logged and the others are still written.

Parameters

[in]	directory	The folder of the dataset
[in]	prefix	The common prefix of the files (e.g. "quantms")
[in]	views	The views by name (e.g. "psm", "run")
[out]	failed	If given, receives the names of the views that failed (or were not kept)

Returns: true if every view with tables was written; false if the folder could not be created or a view failed

Classes

Public Types

Static Public Member Functions

Detailed Description

Class Documentation

◆ OpenMS::QPXDatasetExport::MzView

◆ OpenMS::QPXDatasetExport::PeakFilter

◆ OpenMS::QPXDatasetExport::ProvenanceStep

◆ OpenMS::QPXDatasetExport::Run

◆ OpenMS::QPXDatasetExport::RunSample

◆ OpenMS::QPXDatasetExport::Sample

◆ OpenMS::QPXDatasetExport::View

Member Typedef Documentation

◆ Column

Member Function Documentation

◆ formatNumber()

◆ keptPeaks()

◆ mzView()

◆ paramToJSON()

◆ provenanceView()

◆ psmColumns()

◆ runView()

◆ sampleView()

◆ scoreName()

◆ writeDataset()