Intermediate format specification#
This page specifies the internal tabular formats ProteoBench produces while processing a benchmark
submission. Scoring, plotting, datapoint generation, the submission-validation layer, and the
intermediate_hash that identifies a dataset all depend on the column names, types, and semantics
defined here. The intermediate format is specific to each module, though modules can share formats
where applicable; anything module-specific below is purely illustrative.
Note
Status: descriptive specification of current behavior (format version 1, implicit). It documents what the code produces today. Proposed changes (an explicit version field, a canonical serialization for hashing) are described under “Reproducibility and the intermediate hash” below.
Scope#
There are two distinct tables:
Artifact |
Produced by |
Persisted |
Consumed by |
|---|---|---|---|
Standard format (pre-scoring) |
|
No (in-memory) |
Scoring; the submission-validation layer |
Intermediate format (post-scoring) |
|
Yes, as |
Plotting, datapoint metrics, the “View Single Result” table |
The intermediate format is the primary subject of this page. The standard format is documented as the declared input format that any module or tool parser must produce.
Standard format#
Produced by convert_to_standard_format() in proteobench/io/parsing/parse_settings.py. It’s a
long table with one row per (e.g. precursor, run) pair for the precursor quantification module: a
precursor quantified in six runs yields six rows. The table isn’t projected to a fixed column set
(unmapped input columns may also be present); only the columns below are relied on downstream, and
this description is specific to quantification modules — it may differ for other modules.
Column |
Type |
Required |
Meaning |
|---|---|---|---|
|
str |
yes |
Protein or protein-group identifier(s). Multiple proteins joined by |
|
str |
yes |
Plain (unmodified) peptide sequence. |
|
int |
yes |
Precursor charge. |
|
str |
conditional |
ProForma-encoded modified sequence. Present only when the tool TOML defines a |
|
str |
conditional |
|
|
str |
conditional |
Equal to |
|
str |
yes |
Cleaned run name (extensions and known suffixes stripped by |
|
float |
yes |
Quantitative intensity for the (precursor, run) pair. Rows with |
|
str |
yes |
Condition label, |
|
bool |
yes |
Result of matching |
species flags |
bool |
yes |
One boolean column per species in the module |
|
bool |
yes |
|
run dummy columns |
bool |
yes |
One boolean column per distinct |
The intermediate format table#
The table used to calculate all metrics in a given module. It should hold every value needed for metric calculation in a standardized fashion, but isn’t limited to these columns — it can include more, even tool-specific ones. It does need defined (documented) and “normalized” (comparable across submissions) columns for metric calculation.
For quantification modules, this is produced by QuantScoresHYE.generate_intermediate() in
proteobench/score/quantscoresHYE.py. One row per precursor (or peptidoform). Only
single-species precursors are retained: compute_epsilon() keeps rows where exactly one
species flag is set (unique == 1), so multi-species precursors don’t appear.
Column catalog#
Column |
Type |
Meaning |
|---|---|---|
|
str |
Precursor key. Name depends on the module |
|
float |
Mean linear intensity across the runs of condition A / B. |
|
float |
Standard deviation of linear intensity per condition. |
|
float |
Mean of the log2 intensities per condition. |
|
float |
Standard deviation of log2 intensities per condition. |
|
float |
Coefficient of variation per condition, |
|
float |
|
per-run intensity columns |
float |
One column per distinct run name (cleaned |
|
int |
Number of runs in which the feature (precursor, peptidoform) was quantified. |
species flags |
bool |
One boolean per species (HYE: |
|
int |
Sum of the species flags. Always |
|
str |
The single species name for the precursor. |
|
float |
|
|
float |
Accuracy: |
|
float |
Per-species empirical center of |
|
float |
Precision: |
Notes on semantics#
All logarithms are base 2.
epsilonrequires ground truth (the expected ratio) and measures accuracy;epsilon_precision_*requires no ground truth and measures reproducibility.The per-run intensity columns aren’t a fixed part of the schema — their names and count follow the run names of the submitted experiment.
Reproducibility and the intermediate hash#
The dataset identity is intermediate_hash, computed in proteobench/datapoint/quant_datapoint.py
as the SHA1 of the intermediate DataFrame rendered with pandas.DataFrame.to_string().
Because the hash is taken over the rendered text, it depends on:
the set of columns and their order
the row order
the floating-point formatting used by
to_string()the run-specific intensity column names
Any change to the scoring code, the column order, or pandas’ rendering can change the hash for identical inputs. Column order and row order are therefore part of the format.
Module variants#
HYE (Human-Yeast-Ecoli): the format described above. Species set
HUMAN,YEAST,ECOLI.PYE (Plasma,
quant_lfq_DIA_ion_plasma):QuantScoresPYE.generate_intermediate()is currently a pass-through over the HYE implementation, so the intermediate columns are the same. The plasma-specific quantities (spike-in fold-change error, dynamic range, human-plasma epsilon) are computed later, inQuantDatapointPYE, not in the intermediate table. The plasma datapoint also usesmax_nr_observed = 12instead of 6.De novo (
denovo_DDA_HCD): a different schema entirely, carrying match-type and amino-acid-level columns (e.g.match_type,aa_matches_dn/aa_matches_gt,pep_match) rather than the quantitative columns above. A detailed catalog for the de novo intermediate format will be added alongside the module’s maturation.
Persistence and external resources#
The intermediate format table is written to result_performance.csv inside the dataset archive on
the public datasets server, keyed by
intermediate_hash. It’s reloaded for the “View Single Result” tab and for regenerating plots.
Reference example#
test/data/intermediate_files/result_performance_MaxQuant_20241216_120952.csv is a committed
sample of the (version-1, pre-precision) intermediate for a MaxQuant DDA submission.
Non-goals#
This page doesn’t define the per-tool native output formats; those are handled by the format-specific loaders in
io/parsing/parse_ion.py.It doesn’t define the datapoint JSON stored in the results repositories; that’s a separate structure built by the
QuantDatapoint*classes.