Public archives#

Ingence can read a public archive’s list of files as if it were a folder, without downloading the files. This guide explains how, using the BioImage Archive as the example. It describes the mechanism. For what Ingence made of 263 studies from that archive, see Ingence on the BioImage Archive.

What a listing root is#

A root does not have to be a folder on your disk. It can be an archive’s listing, written

listing:<archive>:<identifier>

<archive> is the short name of a source Ingence knows how to read, and <identifier> names one study in that archive. The one source that ships is biostudies, which reads studies from the BioStudies service, where the BioImage Archive’s studies are kept. A BioImage Archive study’s identifier is its accession number, which begins S-BIAD. So a root reads listing:biostudies: followed by that accession number.

A listing holds what the archive says about each file: its path, its declared size and the descriptions the depositor wrote. It holds no file contents. A BioStudies listing carries no checksum for a file and usually no date of its own. A listing made by another converter may declare more.

The steps#

ingence init archive-project --root listing:biostudies:ACCESSION
ingence index -p archive-project --fetch
ingence infer -p archive-project
ingence see -p archive-project

Replace ACCESSION with the study’s accession number.

index --fetch is the one step that uses the network. It fetches each listing root’s list of files from its archive into the project, once. It is polite: one request at a time, a pause between requests (half a second unless you change listing.pause_seconds), a request that says what it is, and every request logged beside the listing. If the archive answers “too many requests” or “service unavailable,” Ingence stops asking that archive for the run and does not retry. Any other failure is recorded against that study and the fetch moves on to the next. It prints how many requests it made and how many bytes came back.

After the first fetch, every index reads the listing from what was fetched. Without --fetch, nothing touches the network. Add --fetch again to fetch a fresh copy.

Several roots can be given, a folder on your disk among them, by repeating --root.

What a dive does with a listing#

Nothing. A listing’s files are not on your computer, so ingence dive leaves them as the listing stated them: unexamined, which is different from “nothing found.” What Ingence reads comes from names, paths, declared sizes and the depositor’s own descriptions.

Other archives#

An archive with no source built in can still be read. A converter outside Ingence writes the archive’s listing in Ingence’s listing format into the project’s cache, and index reads it from there. A new archive is a converter, not a change to the engine.

Before you use an archive’s data#

  • Check the archive’s terms. Listing metadata and the studies’ own licenses can differ, and the files’ licenses vary by study. Ingence does not check licenses.

  • An archive is somebody else’s server. Fetch what you need, not everything. If you change the pause between requests (listing.pause_seconds), make it longer rather than shorter.

  • A listing is what the depositor wrote. A link between files that the depositor declared is kept away from the rules by default (listing.declared_links). Such a link is an answer key for the engine’s own guesses about what was made from what, and a rule that could read the key would be marking its own work.