Methods
This page describes what Ingence does with a folder of files, step by step, and why. It is written so that a methods section can be drawn from it. How Ingence works gives the ideas on one page. Science says what the project claims and what it does not.
Everything below is about how Ingence reads, names and relates files. It doesn’t say anything about the experiments the files came from.
How each step is described
Each step says what the engine does and why. Where a published method stands behind the step, it also says where the engine departs from that method. A departure is a parameter the engine fixes, a step it leaves out, or a reading it adds. Where the engine doesn’t follow a published method, the step says so. The reason is then an engineering one, and the step doesn’t give a citation.
The numbers are the defaults of Ingence v0.4.0. Most are settings or rule parameters, which the settings reference and the rules reference list. A few are fixed in the code. Words in code font are the engine’s own names for commands, settings, rules and values. An author and a year in brackets link to the References at the end of the page.
Listing the files
What it does. The first step of a scan lists every folder and file under each data folder. Each step of a scan
saves its results as a run. This step is a census run, and the command ingence index runs it on its own. For
each item, the engine records what the filesystem says: its name, size, kind and times. It uses directory listings and
stat calls only, and it never opens a file. It never follows a link. It records the link. To decide whether the
target lies inside the data folder, it reads the link one step at a time.
The engine lists several folders at a time, eight by default (the setting scan.workers). So records don’t arrive in a
fixed order. It sorts every table into one fixed order before it writes the table. So two file lists of the same files
are the same, byte for byte. When a folder or a file’s facts cannot be read, the engine records an error with its
reason. The listing then goes on. Privacy and your data says what the engine reads
and what it writes.
Why. A count over a complete listing is arithmetic, and the engine can state it without reading any data. Listing is slow on a network share, so the engine spreads it over threads. A scan that quietly skipped a locked file would report a wrong count. So every failure is a record.
This step doesn’t follow a published method.
Names are kept in one form
What it does. The engine compares paths after it converts them to Unicode normalization form C (NFC). It keeps the bytes as read beside the converted form, so the engine doesn’t lose anything about the original name.
Why. Systems differ in how they store an accented letter. Some store it as one code point, and some as a letter and a separate accent. Without one form, the same file scanned from two machines would have two identities.
Where it departs. It does not depart. The annex says that equivalent strings have a unique binary representation when they are kept in a normalized form. It defines NFC as a canonical decomposition followed by a canonical composition (Unicode Standard Annex #15). The engine uses NFC to decide identity. It never uses NFC to change what it stores or shows.
Case is read from the names a data folder holds
What it does. For each data folder, the engine decides if the filesystem treats names differing only in case as one name. It decides from the names the data folder already holds, and it doesn’t write anything there. Two names in one folder that differ only in case show that the data folder is case-sensitive. A name looked up with its case flipped may answer with the same device and file number. That shows the data folder is not case-sensitive. The engine reads at most 32 folders, breadth first. If it finds neither sign, the label is unknown.
Why. Merging two files that are not one cannot be undone. Keeping apart two spellings of one file can be undone. So unknown keeps spellings apart.
This step doesn’t follow a published method.
Each run is saved whole
What it does. A run writes its tables into a staging folder and checks them. It checks row counts, foreign keys and ten publication checks (see Checks before a run is published). It then flushes the tables to disk and renames the folder into place. Last, it replaces one small pointer file with an atomic rename. Each run is complete. A run that reuses nine tenths of an earlier run’s entries still writes all of them. Earlier runs are never edited, and they stay readable.
Why. If a run is interrupted, the earlier run stays current and whole. The current run’s tables are the Index: everything the engine has observed about the files. Reading the Index is one read of one folder, and it doesn’t replay any history. And any result can always be rebuilt from the run that made it.
Where it departs. Table formats use the same idea. Their data files are never edited. A record lists which files make up each version. One step makes a new version current. Delta Lake is one such format. Its log gives it transactions and time travel (reading an earlier version) over cloud object stores (Armbrust et al. 2020). Ingence takes the idea, but it doesn’t take any of the machinery. It writes from one machine. A manifest for each run and a pointer file take the place of a transaction log.
Files that moved
What it does. The engine matches a file at a new path to a file that has gone from its old path. It matches them by name, size and modification time to the second. Where both file lists give a device and file number, the same number confirms a move. A different number marks a copy. Where a number is missing, birth times decide. Equal birth times confirm a move, and a later birth time on the new file marks a copy. A copy whose original has gone is read as a move and labeled as a copy. A move keeps what was read about the file and what a person decided about it.
Why. A moved folder should not look like new data. A person’s decisions should stay with the files they were about.
This step doesn’t follow a published method.
The engine’s own files
What it does. The listing recognizes the engine’s installation, any project folder and any run by the names of their children. It never opens a file to do this. These folders are left out of the Index and listed in the run’s record.
Why. A run that indexed its own output would describe itself.
This step doesn’t follow a published method.
Reading names
A name is the first evidence there is, and the cheapest to read. One reader, the name reader, reads every name. So a name means the same thing wherever it is read. The search box reads a typed query with the same name reader, at the same version.
Cutting a name into parts
What it does. The reader splits off the extension first. The extension is matched only against the Dictionary’s words for file formats. The Dictionary is the list of words Ingence knows, with their meanings (see The Dictionary).
The rest of the name is cut into atoms: words, numbers, quantities with units, dates, times, indexed labels and
identifiers. Parts that belong together are held together before any splitting. In HeLa_DMSO_500nM_2.5h_r2.tif, the
reader finds two words, a concentration (500nM), a duration (2.5h) and an indexed label (r2). A run of letters
and digits, without a separator and beginning with a letter, is one word. Examples are a strain, a construct or a formula. A
hash, a universally unique identifier (UUID) or an accession is one atom of its own kind. It is never looked up.
A capital letter divides a run of letters by one of two rules:
Where a lower-case letter meets a capital (
meadowOrchard), the run divides when each piece has three letters or more. This rule looks only at length. A shorter piece stays with its neighbor, somRNAstays whole.Where capitals meet lower-case letters (
NASAharbor), the run divides only where the Dictionary knows a piece. It also divides where the Dictionary knows the capitals as a two-letter tag such asWT. Otherwise the run stays whole. SoWTmeadowdivides andNASAharbordoes not.
Under either rule, a known two-letter tag such as WT or KO is always a piece of its own.
Why. Splitting at every change between a letter and a digit would cut 500nM into a number and letters. The
quantity would be lost. Reading one thing as several is the costlier mistake, because the pieces no longer know about
each other.
This step doesn’t follow a published method. The reader has a version number. Any change that could give different parts for the same name raises the version. Every stored part records the version that made it, so an earlier run is never silently rewritten.
Quantities and units
What it does. A quantity is a number with a unit. Each unit in the Dictionary has a factor to the base unit of its
dimension. So 500nM and 0.5uM reduce to one value. The reader first matches a unit symbol exactly as written, and
only then ignores case. The longest unit wins, and an exact match wins a tie. So nM (nanomolar) is not read as nm
(nanometer), which measures a different kind of quantity. A one-character symbol must always match with its exact
case. It makes a quantity only where it ends its part of the name. So 2.5h is a duration, and the d inside a hash
is not a unit.
A unit entry may name a code from the Unified Code for Units of Measure (UCUM) as a reference. A test checks every code the Dictionary names against the copy of the UCUM table kept with Ingence’s source.
Why. Names write concentrations, times and lengths in many ways. The engine can’t compare them until they are one value. A code from a shared standard says what a unit is, apart from any laboratory’s own spelling.
Where it departs. UCUM is a code system for units. Its codes are case-sensitive, so a quantity and its unit can be
sent between systems without ambiguity. Ingence uses UCUM as a reference only, and it does not parse UCUM expressions.
Each unit also lists the spellings that names use (µm, micron, mcg), and those are not UCUM codes. UCUM’s
license does not allow its table to be changed or used to make a derivative work. So the Dictionary points to UCUM’s
codes and does not copy them.
Dates and times
What it does. A name can write a date in many ways: 20260304, 2026-03-04, 04Mar2026. The reader takes a run
of digits as a date only where the calendar holds it. The year must also fall inside the setting dictionary.years,
which is 2013 to the year of the run by default. A date that two field orders allow, such as 01-02-2026, is recorded
with both readings and marked ambiguous. The reader does not guess. The sibling names in one folder may all fit one
order, as when one has a first field over 12. That order then settles the others, and the reading names the sibling
that settled it. The reader reads a year and month without a day (2026-03) as two numbers, so it doesn’t give a date.
Dates read from inside files follow a stricter rule. A date without a day doesn’t give a time. Reading 2026-03 as 1
March would add a day that the date doesn’t state. In a document’s own properties, the engine reads a time
without a zone as UTC. An instrument’s
clock outside the years 1980 to 2200 is set aside. An unset clock field reads as 1601 or 1970.
Why. A wrong date looks like an answer, and every order built on it carries the error. A date the reader leaves unread shows as missing, so a person can see the gap.
Where it departs. ISO 8601 is the standard for written dates and times (ISO 8601-1). Names use
forms outside it, such as 04Mar2026, and the reader accepts those too. Inside files, the engine reads W3C-DTF, the
profile of ISO 8601 that the web uses (Wolf and Wicksteed 1998). That profile allows a year alone and a
year with a month, and it writes a zone after every time. The engine doesn’t give a time for a year or a month alone,
because a time must name a day. Document properties often leave the zone out, so the engine reads a missing zone as
UTC. The local zone of the machine that did the scan would move every document’s clock.
Words that run together
What it does. Two Dictionary words written together without a separator divide when one division is plainly the
best. For example, dapichannel becomes dapi and channel. This applies only where the whole is not a Dictionary
word and each half has at least four letters.
A spelling that a capital divides is read as one word or as its pieces, once for the whole run. A person’s answer decides first. Failing that, the Dictionary decides, if it knows the whole spelling. Failing that, the run’s own counts decide. The spelling is one word if names hold it at least half as often as its rarest piece alone. The reader reads a regular plural as its singular where the singular is a Dictionary word.
Why. Whether actinGFP is one name or two words depends on how the people who wrote it use it. The counts show
how they use it, and a person can overrule them.
This step doesn’t follow a published method. The reading shows the counts, so a person can check them by hand.
The Dictionary
What it does. The Dictionary says what a word means in this project’s data. It is made of senses. A sense is one meaning of a word. Each sense has a term, a canonical name, a kind, its spellings, an origin and a scope. The kind is technique, reagent, instrument, date, unit and so on. The words are data in YAML files, and they are never in the engine’s code. The Dictionary reference lists the shipped words. A pack is a file of words for one field. Packs of words says what a pack adds to the Dictionary. Every project loads every shipped pack, and a person does not need to accept one.
The binder is the part of the engine that matches words to senses. It looks up every word of every name. It records the sense it chose, or that it didn’t choose one, and why. This record is the word’s binding.
Scope decides first which sense wins, and origin decides second. A narrower scope beats a wider one. So a sense a person gave for one folder beats a sense given for the whole project. At equal scope, a person’s answer beats everything that ships. Among what ships, the core words and the words each field’s pack adds stand level. Two senses that stand level are a tie, and the folder’s own files then decide it:
The binder counts the files of the nearest folder that has at least
dictionary.domain_skew.minimum_filesfiles that vote.A file votes by its type or by a word only one field spells. It can also vote by the other words at the same place in its name.
The leading field wins where it holds at least
dictionary.domain_skew.minimum_shareof the votes. It must also holddictionary.domain_skew.minimum_ratiotimes the votes of the field in second place.
If the files do not decide, a preference a person gave for one field wins. Without one, the binder assumes the word’s
general meaning and marks it as an assumption. If the word doesn’t have a general meaning, the tie stays. The binding names every
sense it did not choose. The rules that need the word then don’t give a result, and they say why. A tie in a word that more than
dictionary.domain_skew.question_above_files files carry becomes a question. A person’s answer settles it. Every
match is graded as exact, case-folded (matched with case ignored) or loose.
One entry alone never lets a rule conclude anything. Where one word is weak evidence, a rule that reads the word asks for more. For example, a file is about the techniques its name mentions only when its name mentions at least two.
Why. A wrong entry should never cause a wrong result that somebody then has to search for. One entry alone doesn’t conclude anything. So the shipped Dictionary can be large, and an irrelevant entry never leads to a result. If one shipped pack won a tie silently, installing Ingence would make an interpretation on the user’s behalf. The folder’s own files are the evidence a person would use. A tie that is reported can be settled in one answer.
This step doesn’t follow a published method.
Typing slips
What it does. Some words are not in the Dictionary but are one or two slips from a word it knows. The engine offers the known word beside the unknown word’s question, as a likely spelling. A slip is a letter added, dropped or changed, or two neighboring letters swapped. The distance is the optimal string alignment distance, which doesn’t edit any part of the word twice. Both the word and the known spelling must have six letters or more. At most three candidates are offered, nearest first. Where two are equally near, both are shown. A plural is a spelling of its word and never a slip. The engine finds candidates through an index of the known words with letters deleted. So it can compare a word with a whole Dictionary quickly.
A proposed spelling is never bound to the word. The binder corrects a slip only under stricter limits, which are not settings:
For one edit, the word has 5 letters or more.
For two edits, the word has 7 letters or more, and the edits only put its letters in another order.
Common English words, units, dates and short words in capitals are never corrected.
The binding says the spelling was corrected. The rule spelling_read_as lists each spelling, the spelling it is read
as, and the number of files that write it. A correction a person refuses does not come back.
Why. People type a letter twice, drop one or swap two, and the word becomes one the Dictionary does not hold. A word one slip from a known word is as often a different word as a slip. A construct’s name is one example. So the choice stays with the person.
Where it departs. Damerau’s technique for correcting spelling errors assumes that a word missing from the dictionary has at most one error. The four kinds of error are a wrong letter, a missing letter, an extra letter and a transposition (Damerau 1964). Ingence counts the same four edits. It differs in four ways:
It allows two edits, and a setting allows one to three.
It does so only for words of six letters or more. A shorter word is two edits from too many others.
It proposes, and it corrects only under the stricter limits above.
It shows the nearest candidates, up to three, instead of choosing one.
Reading inside files
What it does. Listing never opens a file. Reading contents is the later step that does, within limits and against
a budget of work. The command ingence dive runs it on its own, and the engine records it as a dive run.
The first scan that the Dashboard starts for a new project reads contents at Standard depth. It does so whatever the project’s depth, unless that depth is Quick, which doesn’t open any file. This first scan reads one file of every series before it reads a second file of any series. A series is a group of files with the same folder, name shape, type and size class. Where the files read from a series disagree, the scan reads more of that series. An unread file of a series is counted as described by a read file of its series. The rules don’t carry what the engine read to the unread file. Any other reading of contents takes the file types in turn. It reads one file of every type before a second of any, smallest first.
Where a file’s header reader has answered, the engine does not read the rest of a binary file for text. Several worker processes read files in parallel. Each worker starts once and reads many files. Only the parent process writes the results, so the results are the same for any number of workers. The number of workers follows the machine’s cores and memory. It is lower on a network share, and it falls when a worker dies or the reading slows.
Three limits bound the step. One clock bounds the reading of each file. A budget of work bounds the reading time of all files, summed over the workers. A wall clock bounds the whole step. These are the defaults:
At Standard,
ingence divemay spend 900 seconds of reading time, summed over the workers.At Deep, the wall clock stops the step after 4 hours.
The Dashboard’s first scan stops reading contents after 900 seconds on the wall clock.
A person can set other limits, with ingence dive --budget or in the settings. What the step does not reach is marked
not read yet (the status unexamined), and the run is still complete. Each file’s failure is recorded against that
file, and the step goes on. A file that passes its limit is read in a worker that is stopped and replaced. The engine
opens archives in memory and never writes them to disk. It reads every member like a file, under limits on depth,
expansion ratio, member count, bytes and time. It reads scripts and notebooks and never runs them.
Type detection records every detector’s opinion and how well the detectors agree. The detectors include the extension and a table of file signatures. Detection also profiles the bytes as empty, text, wide text, binary or near-random. The engine reports a file whose extension contradicts its content. A detected type only chooses which reader opens the file. What the reader finds replaces the detected type.
Each value read from a file records how far it sits from the bytes. The levels, from nearest to farthest, are these:
the bytes themselves;
structure revealed without interpretation;
a reversible decoding;
a parser of a format’s specification;
a rule of thumb over content;
a meaning.
Two more states stand outside these levels: a file that could not be read, and a file not read yet.
Why. Opening files is where cost and risk are. So the engine bounds it and states what it leaves undone. A detector that is wrong should not decide a result. A value from a parser and a value from a rule of thumb should not look alike.
This step doesn’t follow a published method. The readers for Nikon ND2, Leica LIF and Lambert LI-FLIM files take the acquisition time from each file’s own header. Other readers read a PNG or JPEG header, a TIFF’s first page, an Office document’s properties or a PDF’s metadata. They also read a table’s columns, or a text file’s encoding and frequent words. They work within the same limits.
The Rulebook and inference
A rule is data
What it does. A rule is a record, and the Rulebook is the set of all rules. A rule has an identifier and a version, a sentence in plain words and a written reason. It names the registered function that evaluates it. It lists its parameters, with allowed ranges and defaults. It states its effect: it classifies, groups, proposes, constrains, flags or enumerates. What a rule publishes is a claim: one result, with the rule and the evidence behind it.
A rule that asserts a relation names the relation. For an equivalence, it also names the respect in which things are alike. A rule names its tier, the kind of reasoning it uses (see Four kinds of reasoning). It also names the rules that would defeat it. The rule’s sentence and the evaluating function read each parameter through one function. So what a person reads and what the engine does cannot disagree. Revisions of the Rulebook only add, and they never change an earlier revision. A replaced rule is marked inactive from a revision on, and any earlier run can be replayed.
Why. The interpretive rules are the product. Anyone can read a rule, change a parameter or switch a rule off. Every claim traces to a named rule with a version. A rule’s effect fixes where in a run it works. So the order of rules comes from their effects, and a person doesn’t keep a list of it.
This step doesn’t follow a published method. The rules are listed in the rules reference.
The order of an inference run
What it does. The inference step reads the finished Index and never the data folders. The command ingence infer
runs it on its own. It runs the rules by their effect:
Classifying rules go first.
Grouping rules go second. A grouping rule gives every entry a key, and entries with equal keys form a class. A class of one doesn’t group or publish anything.
Proposing rules go third. They draw pairs within the classes the grouping rules made. For each pair, they consult the constraining rules.
Flagging rules go last. They raise findings: matters for a person’s attention that don’t conclude anything.
A derivation is one file made from another. A file’s stage is the step of the work it belongs to, such as raw data or analysis. The engine finds a derivation before it reads any stage (see What references what). Each inference run replaces what the previous one concluded. Every claim a rule supports is computed again from the Index. So a rule that is switched off stops concluding on the next run.
Why. A grouping is an equivalence relation, and a key states it without comparing anything. Two entries are in one class where their keys are equal. So the engine forms classes in one pass over the entries. Comparing every pair would not finish on a real data collection. Running the rules by effect means a rule’s effect fixes the order. So the order cannot depend on where a rule stands in a file.
This step doesn’t follow a published method.
Five outcomes of one evaluation
What it does. Evaluating a rule on a subject ends in one of five outcomes, and the run records which one.
What happened |
Status |
Failure class |
|---|---|---|
The rule concluded |
|
none |
No result: there wasn’t anything to read |
|
none |
No result: what it needed could not be read |
|
|
No result: what it needed was never produced |
|
none |
What it read pointed more than one way |
|
|
Only the function that evaluates a rule knows which of its inputs mattered. So that function explains why it didn’t give a result. How Ingence works describes the three kinds of not-knowing in plain words.
Why. A rule that didn’t have an input must not look like a rule that was wrong. One empty value would hide which kind of gap it is: unknown, not read yet or conflicting. Each kind asks for something different from a person.
This step doesn’t follow a published method.
Four kinds of reasoning
What it does. Every claim carries its tier, the kind of reasoning that produced it. The tier comes from the rule, and nothing else can set it higher.
Kind |
Meaning |
|---|---|
Entailed |
Follows from the observations, and cannot be wrong if they are right. Examples are a count over a complete listing, or two whole-file digests that agree. |
Defeasible |
Follows by default, and is retracted when a rule it names concludes against it. Most rules are of this kind. |
Abductive |
The best explanation of the facts, which another explanation could replace. Seven shipped rules are of this kind, among them the copy-and-paste signal. |
Testified |
A person said so. It counts only inside the scope the person addressed. The record format has this kind. In Ingence v0.4.0, a person’s answers act through the answer log, as the next sections describe. The shipped rules don’t publish claims of this kind. |
Why. The kinds differ in what could make a claim wrong. So the engine names them, and it does not score them. A number that looked like a probability would need to be calibrated against something. The engine doesn’t have anything to calibrate it against.
Where it departs. The word defeasible comes from the study of reasoning whose conclusions can be withdrawn. Philosophers call it defeasible reasoning. Pollock treats it as roughly the same as nonmonotonic reasoning, the term used elsewhere (Pollock 1987). Ingence uses the word narrowly and doesn’t build a theory of defeat. A defeater is a named rule. When the defeater concludes about the same subject, the claim is published as conflicting. The engine does not weigh arguments against each other, and it does not rank defeaters.
A defeater is a rule
What it does. A defeasible or abductive rule must name what would defeat it. It names other rules, which a person can open and read. A rule that doesn’t name one still runs, and its outcomes and failures are recorded. But it doesn’t publish a claim, and it says why. On a claim, a defeater is recorded as fired where the named rule concluded about the same subject. Otherwise it is recorded as what would retract the claim. A claim whose defeater fired is still published, as conflicting, so the retraction can be seen.
Why. A retraction can be trusted only when its reason was written in advance, in a form a person can read. A reason invented afterward would only fill in the record.
This step doesn’t follow a published method.
What a person adds
What it does. Everything a run reads from a person goes through the answer log. It lists every answer and correction a person gave. The engine only ever adds to it. A run reads the answer log as it stood when the run began.
A person’s word about a spelling becomes a sense in the Dictionary, which wins over the shipped senses.
A person’s yes to a proposal is read by a rule that names the answer. A set of files a person called copies is grouped by such a rule. A group of experiments a person approved groups every file beneath them.
A person’s “these are unrelated” does not delete a copy claim. It is recorded as a fired defeater on each claim that can be retracted. That claim is then published as conflicting. An entailed claim, such as equal whole-file digests, cannot be retracted.
Answers don’t change what the Index observed.
Why. A claim needs four inputs: the Index, the Rulebook, the Dictionary and the answer log. Given the same four, the same claims follow.
This step doesn’t follow a published method.
What a claim carries
What it does. A claim names its subject, what it asserts, the rule and the rule’s version. It also names the scope it holds in, the records it rests on and its defeaters. One place in the engine builds every claim, so a claim’s tier is exactly its rule’s tier. Any claim can be traced back to the bytes it rests on.
Why. An inference cannot be checked unless its provenance, the record of where it came from, stays with it.
Where it departs. The nearest standard is the W3C’s provenance ontology (PROV-O). Ingence’s claims carry the same kind of information, but the engine does not write PROV. The relations later on this page name the PROV terms they correspond to.
The tier decides and a score breaks ties
What it does. A claim’s standing is its tier, and never a number. Where an order is needed, the tier decides it. A number only orders items inside one tier, and it is never shown as a probability. A rule that proposes pairs draws its candidates only within a stated set. The set is the classes of a grouping rule, or the files of one folder. The rule never draws candidates from all of the data. At most 25 partners are proposed for any one file. A file whose list was cut is marked as cut. A candidate is never a claim. It is a proposal for a person to decide. If a constraint refuses it, it is blocked, and the conflict is recorded. Otherwise it is circumstantial. At this step the engine does not grade a candidate as anchored or direct. It never adds scores into a confidence.
Why. Three views of one fact do not become three pieces of evidence by being added. A threshold on a sum cannot be defended. A search of all pairs never finishes on a real data collection. A list marked as cut is a different answer from a complete list.
This step doesn’t follow a published method.
Checks before a run is published
What it does. Ten checks run on the graph a run is about to publish, and a failure stops the publication. The checks come from the declared properties of the relations, which The eight relations describes:
Containment forms a forest: an item is inside at most one container, and never inside itself, directly or through others.
Derives doesn’t have a cycle.
Identical is a proper equivalence.
Equivalent closes only within one respect.
A series is totally ordered.
Concerns is never chained.
A fact transfers across identical, and it doesn’t transfer across any other relation.
Every embedded part carries an expansion state.
Every asserted edge names a rule and a version.
More evidence never lowers a claim’s tier.
A check that lacks the context to run is recorded as not applicable. It does not stop the publication.
The last check compares two runs, so one publication cannot run it. The engine keeps it by construction. A claim has its rule’s tier. More evidence can add a defeater, which is recorded and shown, but it cannot lower a tier. The engine’s tests hold a function that compares two runs and fails when a tier fell.
Why. A property that is only warned about can fail without anyone seeing it. So a failed check stops the publication.
This step doesn’t follow a published method.
Copies and sameness
People can mean several different things by “the same file”. Ingence keeps them apart, because they answer different questions. They also cost very different amounts to settle.
The same bytes
What it does. Two files are the same bytes when their whole-file digests are equal. A digest is a short value computed from a file’s bytes. The digest is BLAKE3, read in 1 MiB pieces. Equal digests give identical, the one relation that lets a fact about one file transfer to the other. Different digests prove that the files are not identical. The engine keeps that proof and never computes it again. A quick digest hashes the first 64 KiB, the last 64 KiB and the size. It can show that two files probably differ. It never shows that they are the same, and it only nominates candidates.
Why. A digest of the whole file is the only evidence that proves identity. A quick digest costs little and removes candidates early.
Where it departs. BLAKE3 is a cryptographic hash function built as a tree. The tree lets it use any number of processor cores and vector lanes in parallel (O’Connor et al.). The specification also provides extendable output, which can be of any length. Ingence uses the default 256-bit output for a file’s digest. Each of Ingence’s own records has an identifier. It is the record’s kind, then the first 128 bits of a BLAKE3 hash. That hash is of the text that identifies the record, and it is not a digest of any file.
Hashing only what a comparison needs
What it does. By default, the engine hashes only what comparing the files requires (the setting hash.mode,
need). It takes a sample digest of each file that shares its size with another file. It reads the whole of each file
whose sample digest another file shares. The engine doesn’t need anything more to tell apart a file whose size is unique.
Why. Hashing is limited by disk and network reads, and people notice a long first scan. Reading a file that can’t have a duplicate doesn’t gain anything.
This step doesn’t follow a published method.
Same shape, same columns, same text
What it does. Two tables have the same columns when their column names match, in order and ignoring case. Their delimiter must also match, and each table must have two columns or more. The comparison doesn’t hold any cell values. Two arrays have the same shape when their shape, element type and storage order match. Two folders are alike when the kinds of things directly inside them match, and each holds at least three things. Each of these has its own grouping rule. These rules don’t say anything about being one experiment.
Two optional readers add similarity by content:
A sketch of a text file is a MinHash over overlapping 9-character windows of the file’s first 256 KiB. The amount read is a setting. The case is folded, the spacing is collapsed, and the sketch uses 128 permutations.
A difference hash of a downscaled grayscale image survives resizing and recompression. It does not survive cropping or rotation.
Both readers only nominate candidates.
Why. A re-saved table, the same measurements in another format and a figure exported twice don’t share any bytes. A person would still call each pair the same thing.
Where it departs. The text sketch takes the approach of Broder. He reduced the resemblance of two documents to a set intersection that random sampling can estimate. The sample is taken independently for each document (Broder 1997). Ingence fixes the window at nine characters and the sketch at 128 permutations. It sketches a sample of the file, and not the whole file. It reads the result as a nomination for something stronger to settle, and never as a claim. The image hash doesn’t follow a published method.
A copy-and-paste signal
What it does. Files that agree in name, size, modification time and kind are grouped as probably copied from one another. Files inside an installed program are left out. Birth time is not compared, because a copy rarely keeps it. The rule is abductive. It explains the facts, and it does not claim that the files hold the same bytes.
Why. Equal name, size and modification time are what a copy and paste leaves behind. That is worth raising for a person. Only a digest, and never this rule, supports a claim of copying. The rule names installed programs as its defeater.
This step doesn’t follow a published method.
The eight relations
What it does. Everything the engine asserts is built from eight relations: contains, embeds, derives, identical, equivalent, succeeds, references and concerns. The engine doesn’t have a general “related to”. If the engine can’t say how two things relate, it doesn’t give a result. Seven properties declare each relation:
whether it is directed;
whether it is symmetric;
whether it is transitive, as asserted;
whether a cycle is a contradiction;
what shape of order it makes;
whether the engine can know it found them all;
whether it lets a fact move from one end to the other.
Each pair of relations differs in at least one of the seven, and a test checks this. A relation that differs from another only in meaning is a specialization. A specialization has a name, a parent relation and fixed parameters. It inherits the parent’s properties and may not contradict them. The table below gives each relation in plain words. The relations reference lists them with their properties.
Relation |
In plain words |
|---|---|
contains |
This folder holds that file or folder. After a full listing, this is complete. |
embeds |
This file holds that part inside it, such as a member of a zip archive. Never known to be complete. |
derives |
This was made from that. It never runs in a circle. Only the direct step is asserted. All the steps back to the first file are the file’s provenance. |
identical |
These are the same byte for byte. The only relation that lets you treat one as the other. |
equivalent |
These are the same in one stated respect, such as the same kind of experiment. Every grouping is built from it. |
succeeds |
This comes after that in an ordered series, such as versions one, two and three. It is not the same as being made from. |
references |
This mentions or points at that. The only relation that may form a circle. |
concerns |
These are about the same subject. It works both ways but never chains: if A concerns B and B concerns C, that doesn’t say anything about A and C. |
Why. A relation without properties is not a relation. Each relation has its place because of a formal property that the other relations don’t have. The engine checks these properties (see Checks before a run is published). Treating concerns as transitive would connect everything to everything after a few steps. Similarity graphs fail in this way.
Where it departs. The practice comes from a study of one relation that looks like a single thing. Winston, Chaffin and Herrmann showed that “part of” is several relations. An apparent failure of transitivity in reasoning about parts and wholes comes from treating them as one (Winston et al. 1987). Ingence takes the practice and not the taxonomy. It separates relations by their formal properties. Each relation also lists its nearest terms in other vocabularies:
PROV-O (Lebo et al. 2013);
the Dublin Core metadata terms (DCMI);
the OBO Relation Ontology (Smith et al. 2005);
SKOS (Miles and Bechhofer 2009);
a few others.
A reader who knows those vocabularies can see what each relation corresponds to. The lists are for reading, and the engine does not write them out.
The record of a succeeds edge has a field for a relation between time intervals. There are thirteen such relations: before, meets, overlaps and so on. The shipped rules don’t set the field.
Experiments and groups
What an experiment is
What it does. Two rules decide which folders are experiments. One reads what a folder holds, and the other reads its name. A policy combines them. By default, a dated name comes first, then what the folder holds. Each file belongs to the highest experiment folder above it. A program’s folders don’t belong to any experiment.
Experiment folder from contents counts what each folder holds, from names and declared formats alone. An acquisition is a file whose extension the Dictionary declares as one an instrument writes while it records data. A file named for an acquisition is another file whose name holds the acquisition’s whole name, parts in order. Examples are a table or a corrected copy. The rule asks how far this material is bound together under one folder. It also asks whether the folder’s subfolders are linked by the acquisition names they share.
Experiment folder from dated name takes a folder whose name holds a date beside other words. Files in one folder whose names differ only by a counting number are repeats of one condition. The rule is replicates of one condition.
Why. A dated folder name is the clearest statement a person makes about where one experiment begins, so it comes first. Reading what a folder holds doesn’t need any word’s meaning, so it still works where names say little.
This step doesn’t follow a published method.
Which experiments belong together
What it does. The engine proposes groups of experiments. Each experiment becomes a set of features of four kinds:
the words of its files’ names, counted once for the experiment (a folder of 5,000 frames says
tifonce);the words of its own folder’s name;
the names of its files;
the folder it sits in.
A word whose sense is a stage, a replicate or a connective doesn’t link anything. Each feature is weighted by how few experiments hold it, so a word that only two experiments share weighs most. The engine cuts a feature held by more than a quarter of the experiments. In a small project, it keeps a feature held by no more experiments than one experiment and its three nearest.
Experiments are compared by cosine similarity. Two experiments are joined when each is among the other’s three nearest. Their similarity must also be at least 0.2, and they must share at least two features. The groups are the connected pieces of this graph. A piece of more than eight experiments is cut again at a higher similarity, until every piece has eight experiments or fewer. The method doesn’t use random numbers, so it gives the same groups every time. Each proposed group is a question, and the engine doesn’t group anything until a person says yes.
Why. Whether two experiments go together is a question at a larger scale than replicate files. A rare shared word links experiments better than a common one. Every link can be shown with the features it rests on.
Where it departs. Weighting a feature by how rarely it occurs is the idea of Spärck Jones. She argued that terms should be weighted by how widely they occur across documents. Matches on less frequent, more specific terms are then worth more (Spärck Jones 1972). The engine applies the idea to experiments in place of documents, and it uses a smoothed form. The rest of the procedure doesn’t follow a published method: the mutual nearest neighbors, the connected pieces and the cut.
Groups made of words
What it does. A group is a set of words that some experiments share exactly. It is never a date or a number. The engine finds the closed sets of words. A closed set is exactly the words that all the experiments holding it share. The engine finds them by closing the intersections of the experiments’ word sets, in a fixed order. It orders the sets by inclusion. It asks about the most general sets first, when five or more experiments hold them. It asks about a set that adds words to an approved set only once every set above it is approved. Approval passes only to what contains the approved set, and never to a subset of it. Sets of fewer than two words are not kept. The lattice, the sets ordered by inclusion, stops at 5,000 sets and says so when it does. Where it has stopped, the engine doesn’t ask anything.
Why. A group is a set of files with common words. Approving one says that the words the engine saw together are really linked. Every experiment holding all of them is then in the group. Asking from the general to the specific keeps the number of questions small. Each question is a set of words a person can read.
Where it departs. In formal concept analysis, a context of objects and attributes gives a lattice of formal concepts (Ganter and Wille 1999). Here the objects are experiments, the attributes are words, and the closed sets are the concepts’ shared attributes. The engine does not analyze the lattice for its own sake. It uses the lattice to order questions. It keeps only sets of at least two words, and it caps their number. It asks only about sets that enough experiments hold.
How complete an experiment is
What it does. For each experiment, the engine counts the raw acquisitions it found. It also counts how many of them are analyzed. A raw acquisition is analyzed when the engine found an analysis file or a results table for it. That file was found to be made from the acquisition or named for it, or a person confirmed the link. The Dashboard’s experiment rows show this as “Raw files analyzed”. The engine leaves the share empty, never zero, where it didn’t find a raw acquisition.
An experiment is listed under No analysis found when all of these hold:
it has at least five raw acquisitions;
under half of them are analyzed;
it has fewer than five analysis files and fewer than three results tables of its own.
The reason comes from the experiment’s theme: experiments whose names use similar words (see Themes and the map). If at least half of the theme’s other raw acquisitions are analyzed, the reason is “Perhaps forgotten”. If under a fifth are analyzed, the reason is “A line that stopped”. Otherwise the engine doesn’t give a reason. The Dashboard groups the list under the headings Forgotten, A theme that stopped and No reason given.
Why. A missing file does not mean that the file does not exist. It means the engine did not find it. So the count is always of what was found in what was read.
This step doesn’t follow a published method.
What references what, and what was made from what
What it does. When a file mentions a path, the engine records the path as the text it is. It resolves the path later against the Index. A code include is never resolved against data. An absolute path that isn’t under any of the project’s data folders names a place the Index doesn’t hold. That is an answer, and it is not a failure. The engine looks for a path relative to the mentioning file’s folder first, and then in the data folders. It looks for a bare name only in its own folder. If several files match, the engine doesn’t give a result. It never takes the nearest, the newest or the first. A match gives a references relation, and never a derivation. The engine records a script’s own read and write calls that name a file literally, with their direction.
Rules propose derivations, and each rule rests on one kind of evidence. They include a script’s calls, a file named for an acquisition, and one name in two folders of an experiment. Each derivation is found and directed by its own evidence or by time. A file’s stage is not used to find a derivation or to direct one. Stages are read afterward, and a stage that disagrees with a derivation is raised for a person. Constraints limit derivations in three ways:
a derivation is never cyclic;
a program is never made from a file;
two files whose names give different subjects or samples are never counted as made one from the other.
A file’s time is the earliest time it is known to have existed (earliest credible time). Clocks are ranked by whether they survive a copy, in this order:
an instrument’s own acquisition time;
a document’s own metadata;
a date in the name;
the modification time;
the filesystem’s change and birth times.
The ranking is by how well a clock survives copying, and not by precision. An impossible time is set aside.
Why. Knowing what was made from what is the purpose of reading a laboratory’s folders. A wrong arrow is worse than a missing one. A script is the one place where a laboratory states these facts itself. A stage word in a name should not decide which way a derivation runs.
This step doesn’t follow a published method.
Themes and the map
A theme is a set of experiments whose names use similar words. Ingence finds themes itself. Themes are a projection: the engine computes them again from a finished run, the Dictionary and the settings. A theme is never a claim, a question or an input to any rule.
What each experiment says
What it does. Each experiment is treated as a document. The words of its own folder’s name count three times, and
the words of the folders inside it count once. A word on its files counts ln(1 + the number of files that carry it).
A word the Dictionary binds is replaced by its sense, so all its spellings become one word. Senses whose kind says how
a file was made are left out, because they don’t say anything about the subject. Such kinds are a stage, a format, a date, a
replicate and so on.
A word the Dictionary does not know is read as stems. It stays whole where it is a construct, such as letters followed by a number. Otherwise it is cut at its numbers. A regular plural of such a word is read as its singular.
A word is kept only if an acquisition’s name or a folder name at or above an experiment holds it. A sense must appear
in at least two experiments. An unknown word must appear in at least three experiments of one data source, each time
beside another word. A data source is each data folder by default, or each area (the setting
themes.collection_level). A word in more than half the weighted experiments, and in more than two, is context and is
dropped.
Why. Words that say how a file was made would join every experiment to every other. A word typed in two experiments could be a habit, and one typed in three is a pattern.
This step doesn’t follow a published method.
Weighting data sources
What it does. A project may hold a few experiments made by hand and many made by a machine. An experiment’s weight
is sp − 1, where s is the number of experiments in its data source. The power p is one half (the
setting themes.collection_weight_power). So a data source as a whole weighs its size to the power one half. A
thousand-fold difference in size becomes about thirty-fold. A data source of fewer than ten experiments is weighed as
if it had ten.
Why. Thousands of near-identical experiments made by a machine should not outweigh a small set made by people. And one experiment alone should not weigh as much as dozens.
This step doesn’t follow a published method.
Rarity weights and a word space
What it does. Experiments whose names hold the same words become one point that carries all their weight. Each
word’s count in an experiment is multiplied by a rarity weight, ln((1 + N) / (1 + df)) + 1. Here N is the weight of
all the points, and df is the weight of the points holding the word. Each point is then scaled to unit length.
A truncated singular-value decomposition reduces the matrix to at most max(50, 2⌈1.5√n⌉) dimensions. Here n is the
number of distinct documents. Before the fit, the rows are scaled by the square root of their weight. Every point is
then projected into the space and scaled to unit length again. So cosine similarity is a dot product.
Why. The decomposition places experiments that use related words close together, even when they share few exact words. A theme needs that. The dimensions grow with the number of distinct documents, so that every theme the method may choose has room.
Where it departs. The rarity weight rests on the argument that widely used terms should weigh less (Spärck Jones 1972). The decomposition is latent semantic analysis. It factors a large term-by-document matrix by singular-value decomposition. The original paper kept about a hundred orthogonal factors (Deerwester et al. 1990). Four things differ:
The documents are the names of experiments, and not texts.
The number of dimensions grows with the data, and it is never fixed at a hundred.
The rows carry weights, so a document repeated many times counts as one point of larger weight.
The decomposition is the randomized one in scikit-learn (Pedregosa et al. 2011). It finds a subspace by random sampling of the matrix (Halko et al. 2011).
A seed is the number that starts the random sampling, so each seed gives a slightly different space. The engine uses this on purpose, as the next steps show.
The graph, and what chance gives
What it does. Each point is joined to its nearest neighbors by cosine similarity. The number of neighbors is the square root of the number of points, and at least five. Each edge has the similarity as its weight. An edge between points of different data sources is kept only if its similarity beats what chance gives.
A null is what the similarities would be by chance, and the engine uses a row-shuffle null. Each shuffled document keeps its number of distinct words and their counts. It draws which words they are without replacement, in proportion to how many documents use each word. The shuffled documents are embedded in the same way. The cutoff is the three-quarters point of the similarities between each shuffled document and its ten nearest shuffled documents.
Why. One data source’s words should never pull another’s experiments in by chance. A null that shuffles columns only renames the words, and every similarity stays as it was. So it doesn’t measure any chance at all. Shuffling within rows keeps each document’s size and each word’s overall frequency. It breaks the pairing of words.
This step doesn’t follow a published method. It is a permutation-style null, chosen to keep what should be kept.
Communities
What it does. Themes are the communities of this graph, found by the Louvain method. The method visits nodes in a seeded random order. Each node moves to the neighboring community where it gains most. The communities then become the nodes of a smaller graph, and the steps repeat until every node stays in its community.
The quality the method maximizes is modularity with a resolution parameter. A node’s gain from joining a community is the weight of its links to that community, less a chance term. The chance term is the resolution times the weight chance would put there. That weight depends on the node’s and the community’s total link weights. Ties go to the lowest-numbered community. Afterward, every community is split into its connected pieces.
Why. Modularity rewards a division with more weight inside communities than chance would give. Chance here is a graph with the same total link weight at each node. The resolution sets how fine the communities are. The seed makes the result repeatable. It also lets the engine measure how much the result depends on the order of visits.
Where it departs. The method is Blondel et al. (2008), a heuristic based on modularity optimization. Newman and Girvan (2004) proposed the quality function. Ingence differs in three ways.
First, the paper optimizes modularity itself. The engine adds a resolution in the form of Reichardt and Bornholdt (2006). Their single parameter weights the chance term against the links present. A value of one gives Newman and Girvan’s modularity.
Second, the engine makes every community connected. Traag et al. (2019) show that Louvain may yield arbitrarily badly connected communities. They also show that the Leiden algorithm guarantees connected ones. The engine does not run Leiden. Leiden’s reference implementation is under the GNU General Public License (GPL). The engine keeps its required dependencies under permissive licenses. So the engine runs Louvain, implemented from the published description over sparse arrays. It then splits each community into its connected pieces. It does not claim the other properties the Leiden paper proves.
Third, the engine’s Louvain visits nodes in an order drawn from a seed, so each seed gives a different result.
How many themes
What it does. The engine cuts the graph at seven resolutions, with eight seeds each. The resolutions are 0.3, 0.5,
0.7, 1, 1.4, 2 and 3. A resolution is a candidate when its number of themes lies between max(3, ½√n) and max(4, 1½√n). Here n is the number of distinct documents. If the window doesn’t hold any resolution, every resolution
is a candidate.
Among the candidates, the engine takes the one on which the seeds agree most. On a tie, it takes the one with more themes. Agreement is the adjusted mutual information between the first seed’s communities and each other seed’s. It is averaged and rounded to two places. The best resolution may be the finest or the coarsest tried, and beat the one beside it. Then the engine tries one step past it, at most twice each way. A step that only ties ends the search. A community of fewer than three experiments, or of fewer than two different documents, is not a theme.
Why. A cut that is too coarse gives themes too broad to name. A cut that is too fine gives themes that do not repeat. Seeds that agree show a cut that the data support, and not one that the order of visits produced.
Where it departs. Mutual information compares two ways of dividing the same items. The adjusted form removes the agreement that chance gives (Vinh et al. 2010). This matters most when the number of items is small beside the number of groups. The engine uses that measure. The window, the rounding and the step past the edge are its own, and they don’t follow a published method.
Experiments that are not placed
What it does. After the cut, some experiments are not placed. Their best similarity to the rest of their community is no better than the chance cutoff. A community left under the minimum size isn’t a theme. Each experiment left out carries one of these reasons, in these words:
“no theme words”;
“its group is smaller than a theme”;
“its words are no closer to any other experiment’s than chance”;
“its words are no closer to the rest of its group than chance”;
“its experiments read the same words, which is no theme on its own”.
Why. The engine flags what it cannot place, and it never forces an experiment into a neighboring theme.
This step doesn’t follow a published method.
Which seed is shown, and how firm each theme is
What it does. The same final steps are applied to every seed’s communities. The engine compares every pair of seeds by the adjusted Rand index of their placements. It shows the medoid, the seed with the highest mean agreement. For each theme, it also reports how well the other seeds keep it:
For each other seed, the engine finds the best Jaccard index that one of its communities reaches with the theme. The Jaccard index is the share of the union that is shared. The engine reports the mean of these over the other seeds.
It reports the theme’s core: the experiments every seed keeps with it.
A seed does not hold a theme where its best match is below 0.8. The engine counts the seeds that do not hold the theme, and the other themes those seeds split it with.
Why. The engine shows the most typical placement, so the one shown is never an arbitrary choice. A person can see where a theme is firm and where it is not.
Where it departs. The Rand index counts the pairs of items that two partitions treat alike. The adjusted form corrects it for chance (Hubert and Arabie 1985). The engine applies it to seeds. The best Jaccard index resembles the cluster-wise stability measure of Hennig. He resamples the data. For each original cluster, he takes the mean Jaccard similarity to its most similar cluster in the resamples (Hennig 2007). The same description is in the documentation of his R package, which a reader can open without a subscription (clusterboot). The engine changes the seed and not the sample. The mark of 0.8 is the engine’s own, and it doesn’t follow a published method. Themes are not part of the scored evaluation (see Evaluation). These measures say how firm a division is, and they do not say that it is right.
Names for themes
What it does. A theme is named by its most distinctive words. The default ranking is a class-based TF-IDF (term frequency times inverse document frequency). A word’s score is the share of the theme’s weight that the word holds, times a log. The log is of one plus a product. The product is the number of themes times the share of the word’s weight in the theme.
Words that are Dictionary senses lead the label where at least half of the sense’s weight lies in the theme. They are
shown by their canonical names. A label is three words, drawn from six. An option ranks words instead by a log-odds
score (the setting themes.label_method). The score is the z-score of the log-odds ratio of a word between the theme
and all other experiments. It uses an informative Dirichlet prior drawn from all the experiments.
To show how firm a label is, the engine draws the experiments again with replacement, 100 times. Every experiment stays in its theme. Each draw ranks the words again. A label word’s support is the share of the draws in which it stays in the label.
Why. A person can’t read a theme without a name. A name taken from the commonest words would name every theme alike.
Where it departs. The option is the method of Monroe, Colaresi and Quinn (Monroe et al. 2008). It ranks words by the z-scores of their log-odds ratios. It puts an informative Dirichlet prior on word use, proportional to the expected distribution of words. The engine changes three things:
It counts the experiments that hold a word, and not the word’s occurrences.
It takes the expected distribution from all experiments of the run.
It sets the prior’s total strength at 100 counts, a starting value that has not been measured.
The option is not the default. The class-based TF-IDF is the default, because it has been measured on the engine’s test collections. The log-odds ranking has not been measured. The redrawing is the bootstrap, which estimates the sampling distribution of a quantity from the observed data alone (Efron 1979). The engine redraws only the counts of words inside fixed themes, and it does not redraw the themes. So the support figure says how firm a label is, and the seeds say how firm the themes are.
Themes over time
What it does. Each experiment has one year. The year is the earliest date a name gives for any of its files. Failing that, it is the earliest credible time of any of its files. An experiment with neither is undated and counted apart. For each theme, the engine reports its first year, its last year and its peak. The peak is the year with the most of its experiments, or the earlier year on a tie.
Each theme also gets a direction:
A theme is going if its last year is the latest year or the one before. A theme that is not going is stopped.
A going theme that began in the last two years is new.
Any other going theme is rising where its recent share is at least 1.5 times its earlier share. The recent share is its share of all themed experiments over the last two years.
It is falling where that share is at most two thirds of the earlier share, and steady otherwise.
For each year with at least two experiments, the engine also lists four things:
the themes over and under their usual share that year (the share in the year divided by the share overall);
the themes that began;
the themes last seen the year before;
the words first written.
Why. A theme’s history shows what a laboratory was working on, and when. The engine uses shares and not counts, so a busy year does not read as growth in every theme.
This step doesn’t follow a published method. The ratio of 1.5 is a stated choice and has not been measured.
Words used in each other’s place
What it does. For the By year mode of Overlap, the engine lists pairs of words. In each pair, a later word seems to replace an earlier one. It counts how often each pair of words appears in the same experiment. It keeps the pairs that appear together more often than chance gives (positive pointwise mutual information). It then describes each word by its profile of those values. It compares only words of three letters or more. Each word must be in at least three experiments, and in at most half of them. Two words are a pair when all of these hold:
their profiles have a cosine similarity of at least 0.35;
the experiments holding both are at most a tenth of the experiments holding either;
the later word’s median year is at least a year after the earlier one’s;
the later word first appears no earlier than the earlier word first appears;
the earlier word’s last year is at most a year after the later word’s first year.
Why. A laboratory changes its words over the years. Two words used for one thing at different times show a change of habit. A reader would want to see it.
Where it departs. Pointwise mutual information compares how often two words occur together with how often chance would put them together. It is the logarithm of their joint probability over the product of their separate probabilities (Church and Hanks 1990). Bullinaria and Levy compared ways to build word meaning from counts of words that occur together. Positive values of this measure, compared by cosine, did best on their semantic tasks (Bullinaria and Levy 2007). The engine takes that form. It counts experiments, and not words in a window. The thresholds and the requirements on years are its own.
The map
What it does. The map places each experiment in a plane, so that experiments whose names use similar words lie near each other. It uses the vectors of the shown seed’s space. From 8 to 6,000 distinct documents, the placement is Barnes-Hut t-SNE with these parameters:
a perplexity of at most 30, and about a third of the points;
1,000 iterations;
a starting layout from the first two principal components;
a fixed seed and one thread.
From 3 to 7 documents, or above 6,000, the engine uses the first two principal components alone. Below 3 documents it doesn’t draw a map, and it gives the reason. Coordinates are rounded to six places. Experiments whose names hold the same words share a point. The engine still places an experiment that isn’t in a theme, because a map should show every experiment.
Why. A picture lets a person see which experiments lie close. The map says that distances are approximate. It also says that t-SNE keeps close neighbors close and does not keep larger distances. So the gaps between groups say little about how different the groups are.
Where it departs. The t-SNE method maps data of many dimensions into two or three dimensions (van der Maaten and Hinton 2008). The Barnes-Hut variant approximates the gradient that t-SNE learns from, in O(N log N) time (van der Maaten 2014). Kobak and Berens wrote a protocol for single-cell data that preserves global structure (Kobak and Berens 2019). Starting from the principal components, and not from a random layout, is part of that protocol. The engine takes that start and leaves the other parameters at scikit-learn’s defaults. The fixed seed, the single thread and the rounding are the engine’s own. They make the same run give the same places on the same machine.
Search
What it does. The search box reads a query with the same name reader as the listing. Each typed word becomes a concept: the sense it names, that sense’s spellings, and the variants the data use. A result is a unit: an experiment, an analysis folder, a product file or a plain folder. Every match carries the route by which it was found. The routes are these:
the word as typed;
another spelling of its sense;
a plural the data use;
the start of a word;
the inside of a name;
a near spelling, only for a concept that the other routes didn’t match.
Units are ranked by a tuple of counts, and never by a weighted sum. A tuple is a list of counts, compared in order. The counts include how many typed concepts a unit covers, and how broadly the weakest was reached. They also include the kind of unit, how many concepts its own name carries, and others. A tiebreak number made from rarity weights is not shown as a percentage.
Beside the results, the engine lists words that come with the query. A word’s score is its pointwise mutual
information with the query over units: log2(N × both / (matched × holding)). In this formula:
N is the number of units in the project;
matched is the number of units holding the query;
holding is the number of units holding the word;
both is the number of units holding the query and the word.
A word is listed only if its score is above zero and it occurs in at least two matched units. At most twelve words are listed, each with its counts.
Why. A person cannot question a list ordered by one number such as 0.87. A person can question every element of a tuple. So a person can see why a result came first.
Where it departs. Church and Hanks estimate word association from a body of text, with a measure based on mutual information (1990). They count words within a window. They leave out pairs counted five times or fewer, because the measure becomes unstable when counts are very small. The engine counts units and not words in a window. Its floor is two units, because two is the least count that is a pattern and not one name. That reason is the only support for the floor, and the floor has not been measured. The ranking by a tuple doesn’t follow a published method.
Comparing two file lists of the same files
What it does. The engine compares two file lists of what may be the same files. It decides that a file is the same file by these tests, strongest first:
its place, size and time;
a digest, where reading contents computed one;
its name, size and modification time;
weakest of all, its size and time alone.
The engine reports counts and never a score. It reports how many files kept their path, and whether one rewrite of a path prefix explains the moves. It also reports how many pairs of files that shared a folder still share one. It computes the adjusted Rand index from the same table and shows it beside the counts. It never replaces them.
Why. A reorganized data folder should not look like new data. The question to answer is whether the organization changed.
Where it departs. The adjusted Rand index is that of Hubert and Arabie (1985). The engine applies it to the folders the files were in and the folders they are in now. The pair counts come from one table of how many files went from each old folder to each new one. So the engine never lists pairs one by one.
Evaluation
What it does. Ingence is scored on invented laboratories against a written ground truth. The program that generates each invented laboratory writes its ground truth beside the folder, never inside it. The ground truth marks the questions that the files alone cannot settle. Each rule’s answers are counted as right, wrong, no claim or conflicting, and, for pairs, missed.
Precision is the share right among the answers that could be wrong. Recall is the share right among the answers the ground truth expects. Not giving a result isn’t a mistake, and the engine doesn’t hide it. The number of answers found is shown beside the number right.
Every proportion is a count with its total. It carries a 95% confidence interval, the Wilson score interval (z = 1.959964). The lower bound is exactly zero when none of the cases succeeded. The upper bound is exactly one when all of them did. The engine doesn’t give an interval when there aren’t any cases. The same interval is used for the share of a rule’s output that people confirmed. What the evaluation says and does not say gives the method and its limits. It also gives a command to run the public check yourself.
Why. A single score hides whether it rests on six cases or six hundred. An interval shown with its counts does not hide this. An engine that didn’t say anything wouldn’t make mistakes, and it wouldn’t find anything. So precision and recall are always read together.
Where it departs. The interval is named for Wilson (Wilson 1927). Brown, Cai and DasGupta show that the standard interval for a proportion behaves erratically. They also show that the usual textbook guidance on its safety is misleading. They recommend the Wilson interval, or the Jeffreys prior interval, for small samples (Brown et al. 2001). The engine does not change the formula, but it departs from the setting the formula assumes. The interval treats each case as an independent draw. Answers about files in one folder are often not independent. So the real uncertainty can be larger than the interval shows. Themes, and several other parts of the engine, are not part of the scored check.
References
Papers and books come first, by author. Standards and specifications follow, then software documentation. Each DOI is a link that resolves to the publisher.
Papers and books
Armbrust, M., Das, T., Sun, L., Yavuz, B., Zhu, S., Murthy, M., Torres, J., van Hovell, H., Ionescu, A., Łuszczak, A., Świtakowski, M., Szafrański, M., Li, X., Ueshin, T., Mokhtar, M., Boncz, P., Ghodsi, A., Paranjpye, S., Senster, P., Xin, R. and Zaharia, M. (2020). Delta Lake: high-performance ACID table storage over cloud object stores. Proceedings of the VLDB Endowment 13(12), 3411-3424. doi:10.14778/3415478.3415560
Blondel, V. D., Guillaume, J.-L., Lambiotte, R. and Lefebvre, E. (2008). Fast unfolding of communities in large networks. Journal of Statistical Mechanics: Theory and Experiment 2008, P10008. doi:10.1088/1742-5468/2008/10/P10008
Broder, A. Z. (1997). On the resemblance and containment of documents. Proceedings, Compression and Complexity of SEQUENCES 1997, 21-29. doi:10.1109/SEQUEN.1997.666900
Brown, L. D., Cai, T. T. and DasGupta, A. (2001). Interval estimation for a binomial proportion. Statistical Science 16(2), 101-133. doi:10.1214/ss/1009213286
Bullinaria, J. A. and Levy, J. P. (2007). Extracting semantic representations from word co-occurrence statistics: a computational study. Behavior Research Methods 39(3), 510-526. doi:10.3758/BF03193020
Church, K. W. and Hanks, P. (1990). Word association norms, mutual information, and lexicography. Computational Linguistics 16(1), 22-29. aclanthology.org/J90-1003
Damerau, F. J. (1964). A technique for computer detection and correction of spelling errors. Communications of the ACM 7(3), 171-176. doi:10.1145/363958.363994
Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K. and Harshman, R. (1990). Indexing by latent semantic analysis. Journal of the American Society for Information Science 41(6), 391-407. doi:10.1002/(SICI)1097-4571(199009)41:6<391::AID-ASI1>3.0.CO;2-9
Efron, B. (1979). Bootstrap methods: another look at the jackknife. The Annals of Statistics 7(1), 1-26. doi:10.1214/aos/1176344552
Ganter, B. and Wille, R. (1999). Formal Concept Analysis: Mathematical Foundations. Springer, Berlin. doi:10.1007/978-3-642-59830-2
Halko, N., Martinsson, P. G. and Tropp, J. A. (2011). Finding structure with randomness: probabilistic algorithms for constructing approximate matrix decompositions. SIAM Review 53(2), 217-288. doi:10.1137/090771806
Hennig, C. (2007). Cluster-wise assessment of cluster stability. Computational Statistics & Data Analysis 52(1), 258-271. doi:10.1016/j.csda.2006.11.025
Hubert, L. and Arabie, P. (1985). Comparing partitions. Journal of Classification 2, 193-218. doi:10.1007/BF01908075
Kobak, D. and Berens, P. (2019). The art of using t-SNE for single-cell transcriptomics. Nature Communications 10, 5416. doi:10.1038/s41467-019-13056-x
Monroe, B. L., Colaresi, M. P. and Quinn, K. M. (2008). Fightin’ words: lexical feature selection and evaluation for identifying the content of political conflict. Political Analysis 16(4), 372-403. doi:10.1093/pan/mpn018
Newman, M. E. J. and Girvan, M. (2004). Finding and evaluating community structure in networks. Physical Review E 69, 026113. doi:10.1103/PhysRevE.69.026113
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M. and Duchesnay, É. (2011). Scikit-learn: machine learning in Python. Journal of Machine Learning Research 12, 2825-2830. jmlr.org/papers/v12/pedregosa11a.html
Pollock, J. L. (1987). Defeasible reasoning. Cognitive Science 11(4), 481-518. doi:10.1207/s15516709cog1104_4
Reichardt, J. and Bornholdt, S. (2006). Statistical mechanics of community detection. Physical Review E 74, 016110. doi:10.1103/PhysRevE.74.016110
Smith, B., Ceusters, W., Klagges, B., Köhler, J., Kumar, A., Lomax, J., Mungall, C., Neuhaus, F., Rector, A. L. and Rosse, C. (2005). Relations in biomedical ontologies. Genome Biology 6, R46. doi:10.1186/gb-2005-6-5-r46
Spärck Jones, K. (1972). A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation 28(1), 11-21. doi:10.1108/eb026526
Traag, V. A., Waltman, L. and van Eck, N. J. (2019). From Louvain to Leiden: guaranteeing well-connected communities. Scientific Reports 9, 5233. doi:10.1038/s41598-019-41695-z
van der Maaten, L. (2014). Accelerating t-SNE using tree-based algorithms. Journal of Machine Learning Research 15, 3221-3245. jmlr.org/papers/v15/vandermaaten14a.html
van der Maaten, L. and Hinton, G. (2008). Visualizing data using t-SNE. Journal of Machine Learning Research 9, 2579-2605. jmlr.org/papers/v9/vandermaaten08a.html
Vinh, N. X., Epps, J. and Bailey, J. (2010). Information theoretic measures for clusterings comparison: variants, properties, normalization and correction for chance. Journal of Machine Learning Research 11, 2837-2854. jmlr.org/papers/v11/vinh10a.html
Wilson, E. B. (1927). Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association 22(158), 209-212. doi:10.1080/01621459.1927.10502953
Winston, M. E., Chaffin, R. and Herrmann, D. (1987). A taxonomy of part-whole relations. Cognitive Science 11(4), 417-444. doi:10.1207/s15516709cog1104_2
Standards and specifications
O’Connor, J., Aumasson, J.-P., Neves, S. and Wilcox-O’Hearn, Z. BLAKE3: one function, fast everywhere. Specification, version of 2 November 2021. github.com/BLAKE3-team/BLAKE3-specs
Dublin Core Metadata Initiative. DCMI Metadata Terms. dublincore.org/specifications/dublin-core/dcmi-terms
International Organization for Standardization. ISO 8601-1:2019, Date and time: representations for information interchange, Part 1: basic rules. The standard’s committee, ISO/TC 154, describes it at isotc154.org/standards/iso-8601-1-2019
Lebo, T., Sahoo, S. and McGuinness, D. (editors) (2013). PROV-O: The PROV Ontology. W3C Recommendation, 30 April 2013. w3.org/TR/prov-o
Miles, A. and Bechhofer, S. (editors) (2009). SKOS Simple Knowledge Organization System Reference. W3C Recommendation, 18 August 2009. w3.org/TR/skos-reference
Schadow, G. and McDonald, C. J. The Unified Code for Units of Measure (UCUM), specification, version 2.2. ucum.org/ucum
Whistler, K. (editor). Unicode Standard Annex #15: Unicode Normalization Forms. Unicode Consortium. unicode.org/reports/tr15
Wolf, M. and Wicksteed, C. (1998). Date and Time Formats. W3C Note. w3.org/TR/NOTE-datetime
Software documentation
Hennig, C. clusterboot: clusterwise cluster stability assessment by resampling. Documentation of the R package fpc, version 2.2-12. search.r-project.org/CRAN/refmans/fpc/html/clusterboot.html