ParquetTableComparator.h File Reference

#include <OpenMS/CONCEPT/Types.h>
#include <OpenMS/OpenMSConfig.h>
#include <string>
#include <vector>

Go to the source code of this file.

Classes
struct	ParquetDiffSettings
	Settings for ParquetTableComparator. More...
struct	ParquetDiffResult
	Outcome of a table comparison. More...
class	ParquetTableComparator
	Primary-key-aware, order-insensitive, tolerant comparison of Parquet tables. More...

Namespaces
namespace	OpenMS
	Main OpenMS namespace.

---

Class Documentation

OpenMS::ParquetDiffSettings

struct OpenMS::ParquetDiffSettings

Settings for ParquetTableComparator.

The tolerance pair mirrors FuzzyStringComparator: a numeric cell is accepted when either acceptable_absdiff or acceptable_ratio is satisfied. Use acceptable_absdiff for the "zero vs. epsilon" case, where no ratio is meaningful.

Class Members
double	acceptable_absdiff = 0.0	Maximum acceptable |a-b| of two numbers.
double	acceptable_ratio = 1.0	Maximum acceptable ratio max(|a|,|b|)/min(|a|,|b|) of two numbers. 1.0 = must be identical.
bool	compare_identity_columns = false	Compare (and dump) the QPX identity and cross-reference columns as ordinary values. Off by default, and the default is the correct one for a reference comparison. The spec is explicit that "identity is meaningful within a file only; distinct QPX files must not be joined on @c feature_id alone" – and comparing a file against a committed reference is exactly that join. The values are also hostile to it: feature_id hashes rt and observed_mz, so one ULP of platform drift does not shift a digit, it produces a wholly different int64 and turns a tolerable numeric difference into "row missing on both sides". What is asserted about the ids instead is stronger and portable: -schema mode checks that they are present, non-null and unique, and QPXIdentity_test pins the derivation itself against the reference implementation's own vectors.
vector< string >	ignore_columns	Columns excluded from value comparison (they are still schema-checked).
Size	max_reported = 25	Stop collecting messages of any one category after this many (0 = unlimited).
vector< string >	primary_key	Columns forming the primary key. Empty: derive from the file's QPX file_type metadata.
bool	schema_only = false	Compare schemas only; skip key building and value comparison.
bool	unordered_lists = false	Compare list-valued cells as multisets rather than sequences.

OpenMS::ParquetDiffResult

struct OpenMS::ParquetDiffResult

Outcome of a table comparison.

The three message vectors are populated independently, so a run can report a schema mismatch and a value mismatch at once. equal is true only when all three are empty.

Class Members
bool	equal = false	True when no schema, key or value difference was found.
vector< string >	key_errors	Duplicate primary keys, and keys present in only one of the two tables.
double	max_absdiff = 0.0	Largest absolute difference observed between two numeric cells.
double	max_ratio = 1.0	Largest ratio observed between two numeric cells (1.0 when all were identical).
vector< string >	primary_key_used	The primary key actually used, whether supplied or auto-detected.
Size	rows_1 = 0	row count of the first table
Size	rows_2 = 0	row count of the second table
Size	rows_compared = 0	rows matched by primary key and compared cell-by-cell
vector< string >	schema_errors	Missing/extra columns, differing Arrow types, differing nullability.
Size	suppressed = 0	Number of messages suppressed by ParquetDiffSettings::max_reported.
vector< string >	value_errors	Per-cell differences, prefixed by the primary key of the row they occur in.