What the evaluation says and does not say#
Ingence makes proposals and readings, not scientific conclusions, so it is fair to ask how often they are right. This page says how that is checked today, what the check can show and what it cannot. Read it before you decide how far to trust what Ingence shows you.
In short
Ingence is checked on invented laboratories whose right answers are written down. The check gives counts for each kind of question. It never gives one score.
Saying nothing is not counted as a mistake, so how many answers the engine found is always shown beside how many of its answers were right.
There is no held-out measurement on real laboratories yet: one made against answers written down beforehand and kept apart from the engine’s development. There is no figure for how often Ingence is right on a real drive, and none is claimed.
A free sample collection of an invented laboratory, for anyone to try and test against, is generated by a command that ships with Ingence (below).
How the engine is checked today#
The check has two halves. The public half is described here, and anyone can run it. The private half is the same scorer pointed at a real folder, which a laboratory can do for itself (see the end of this page).
Invented laboratories. A generator writes a pool of invented files with the shapes laboratories
have: dated experiment folders holding raw, processed and analysis files; replicates; copies and
backups; a folder that moved; an installed program, laid out as programs are, among the experiments;
a readme that names other files; scripts and their outputs; empty outputs; a file whose ending says
one thing and whose contents say another; dates written in several forms; and results that two sets
of raw data could each have produced. Each name is an invented word or an ordinary word such as raw
or analysis. Every file is built from its format’s structure with invented values, and nothing in
one is a measurement.
An answer key. Beside the pool, and never inside it, the generator writes what a person who knew the laboratory would say from the files and their names: which stage each file is at, which folders are experiments, which files are replicates, copies or made from one another, which files name one another and which folder is a program. Where the files alone cannot settle an answer, the key says so, and the right output is to abstain. The engine never opens a key while it reads or infers.
A scorer. It runs the engine over the pool and compares what each rule concluded with the key. It reports each kind of question on its own: stage, experiment folder, program, empty file, an ending contradicted by the contents, the date a name states, replicates, copies, groups of experiments, references and derivations.
How a result is counted#
Each answer is counted in one of these ways, and the counts are never added together.
Right: it matches the key. This includes the engine saying there is none where the key says none, and, for pairs, refusing a pair the key does not have. Those two are counted, but they sit outside precision, which looks only at the answers that could be wrong.
Wrong: a different value; a value where the key says there is none; or the engine saying there is none where the key has a value. For a pair of files, a pair the engine drew that the key does not have, or a pair the engine refused that the key does have.
No claim: the engine did not conclude, because it does not know or has not looked, where the key has an answer. This is never counted as wrong (How Ingence thinks explains “unknown” and “unexamined”).
Conflicting: the engine looked and found answers that disagree, and said so.
Missed, for pairs: a pair the key has and the engine did not draw. A key pair that the engine refused or found conflicting is missed too.
Outside and ignored: a key for a real folder names only some paths, so what the engine says about the paths a key never names is counted apart as outside the key and is never wrong. What the key itself leaves out of the score, where any answer is acceptable, is counted as ignored.
Undecidable: where the key marks a question that the files alone cannot settle, the right output is to abstain. Whatever the engine does there, an answer, a refusal or a conflict, is counted apart and is never wrong.
From these the scorer reports two proportions for each rule.
Precision: of the answers that are right or wrong, the share that are right.
Recall: of the answers the key expects, the share the engine got right.
Silence is not counted as an error, because abstaining is a valid answer. That has a cost, which is why both proportions are always shown: an engine that said nothing would make no mistakes and would find nothing. Read precision and recall together, and read the “no claim” count beside them.
There is no single score. Every figure is a count with its total: “right in 18 of 20”, never a bare percentage. Every proportion carries a Wilson interval at the 95 percent level, the range of rates that the evidence cannot rule out given only this many cases. A small number of cases gives a wide range, and the picture below shows why a figure is never read without it.
Each row is one rule’s record on a scale from 0 (never right) to 1 (always right). The ring is the share that were right in the cases seen: all of 5, nine in ten of 20 and nine in ten of 200. The bar is the interval around it. The 5 of 5 row reaches 1 but its bar starts at 0.57: five cases cannot rule out a rule that is wrong more than four times in ten. The 180 of 200 row has the same share as the 18 of 20 row, and its bar runs only from 0.85 to 0.93, because ten times as many cases were seen. The cases here are made up, to show how to read the picture. They are not a result.
The full report, report.json, also splits the counts by the standing the engine gave each
conclusion, so a reader can check whether the conclusions it holds more firmly are right more often.
The short summary does not show the split, and for rules whose conclusions are classes of files, such
as replicates and copies, every conclusion carries one standing, so the split says little.
What the evaluation says#
It says whether the rules do what their authors meant, on drives of the kind the authors imagined, counted rule by rule. That is useful in a particular way. A rule that stops working shows up the moment its behavior changes. A rule’s wrong answers are listed, so the rule can be fixed or its wording corrected.
The pool is an alarm for change and not a target. It sounds when something changes, and nothing should be tuned to it. It is a stated rule of the generator and its tests that a shape is never changed because the engine and the key disagree. People keep that rule; the software does not enforce it.
What it does not say#
There is no held-out measurement on real laboratories yet. A held-out check is one where the answers are written down by a person who knows the folder, before anyone tunes a rule against it, and where the folder was not used to build the rules. No such check has been reported. Until one is, there is no precision or recall figure for any real drive, and a figure from the invented pools must not be read as one.
The invented laboratories are the authors’ picture of a laboratory drive. The people who wrote the rules also wrote the generator and the key. A pass shows that the engine reads the drive that was imagined. It does not show that the engine is right about yours. The key is one person’s judgment of what the files alone can settle, and another reader could draw some of its lines elsewhere. The pools have also partly served as development data and not only as a test: the written reasons of a few rules name cases that first showed up in them, so those rules were shaped with them in view.
Abstaining makes precision look good. An engine that declines a hard case is not wrong about it, and also does not help. This is why recall and the “no claim” count sit beside precision.
A small count says little. A row built on a few cases has a wide interval. Do not compare two figures without comparing their totals.
Some questions are not scored yet. The sample collection’s key lists what a person would also say and the scorer does not yet score. Whole parts of the engine, such as themes and the list of experiments with no analysis found, are not part of this check.
It makes no claim of scientific validity. The check is about files: whether a table was linked to the raw data it came from, whether two folders are copies. It says nothing about whether an experiment was sound or a result is true. Ingence does not interpret science.
A demonstration is not a scored result. Reading a public archive, or any worked example on the site, shows what Ingence does. It is not a measurement. Only a run scored against a key counts as one, and it is reported with its counts and its interval.
The sample collection#
The invented pools above are small and plain. A larger collection, the sample collection, is generated by a command that ships with Ingence. It writes a shared drive of an invented research laboratory over several years, with several invented people, in a folder you name:
python -m ingence.evaluation sample-lab sample-collection --tier S --seed 1
The drive is in the new folder’s pool folder, and you can read it like any other root. The
collection holds what real drives hold: instrument files in the shapes of their formats, tables,
scripts, notebooks, documents and slides; names in several conventions that change over the years;
files saved again as _v2, _final and _final_v2; folders copied whole into a backup; links that
point at nothing; folders that moved; experiments that were abandoned; analysis whose raw data is
gone and raw data that was never analyzed; a program installed inside a data folder; and the folder
of a member who left.
It comes in three sizes.
Size |
About how many files |
What it is for |
|---|---|---|
S |
five hundred |
The tutorial, the examples in the documentation and tests. |
M |
ten thousand |
A demonstration and screenshots. |
L |
one hundred thousand |
A test of speed and memory. |
A size and a seed are all that is needed to write a collection: the same size and seed give the same files, byte for byte, with the same names and modification times. Beside the drive, never inside it, the command writes:
its answer key, in the format the scorer reads;
a record of the generator’s version, the size and the seed;
a datasheet that answers the questions of “Datasheets for Datasets” (a standard form for saying what a collection is for, how it was made and what it cannot do).
Everything in it is invented. The people, projects and conditions are made up. The words in a name, such as techniques, stages, cell lines and stains, are ordinary ones. No file comes from anybody’s disk. The files shaped like a Zeiss CZI hold only a header and are not readable CZI files.
The collection is planned for release as a dataset under CC0, which waives all rights, with a permanent identifier (a DOI) so that it can be cited. It has not been published as a dataset.
What it cannot test. It cannot tell you how Ingence does on your drive. It is one model of a laboratory drive. It holds no real instrument data and no real habits of any one laboratory, and its key shares the limits described above.
Run the public check yourself#
The scorer ships with Ingence. Run it in the environment where Ingence is installed:
python -m ingence.evaluation run evaluation-run --seed 1 --size 1
This writes a pool and its key into the folder evaluation-run, reads the pool with a project of
its own, and prints the counts for each question. It also saves them as report.json and
summary.txt in that folder. Name a folder that does not exist yet: the command will not write over
a pool that is already there. The seed picks one draw of the pool, and the size is how many times
each shape is written, each with words of its own. A different seed gives the same words to other
shapes, with new dates and numbers. It is not data of a new kind, and it cannot show how the engine
does on a drive that the shapes do not describe.
To make a pool and its key and nothing else, use generate in place of run.
A laboratory can also score Ingence on a real folder of its own. Someone who knows the folder writes a
key for part of it, in the same format, and the project command scores a finished project against
it. It takes the key, the project and a folder for the report:
python -m ingence.evaluation project --key KEY --project PROJECT --out REPORT-FOLDER
The key must be kept outside the folder it describes, and the scorer refuses one saved inside it. Run this way, the check stays on that laboratory’s machine.