Metadata Harvesting Endpoint¶

Protocol — instantiates Open Reuse Publication Infrastructure

A machine endpoint that lets external catalogs, search engines, and aggregators pull an artifact's metadata in bulk, so it can be discovered without anyone ever visiting its home site.

An artifact nobody can find might as well be private, and expecting the world to browse your site is not a discovery strategy. Metadata Harvesting Endpoint is the federation surface: a queryable, exportable endpoint that serves an artifact's structured metadata in bulk, in a shared vocabulary, so that other catalogs, search systems, and aggregators can index it and route their own users to it. Its defining move is to invert discovery — instead of waiting for users to come to the artifact, it pushes the artifact's metadata out to wherever users already search. It is not the record itself and not the place the artifact is stored; it is the access-and-export channel that lets many downstream systems mirror and refresh a catalog cheaply.

Example¶

A university's open-access repository holds thousands of datasets and papers, but few outside researchers know the repository exists. It exposes an OAI-PMH endpoint (a real, long-standing harvesting protocol; DCAT, sitemaps, and Schema.org/JSON-LD serve similar roles) that emits a Dublin Core record for every deposit, supports "list everything changed since date T," and hands back a resumption token to page through large result sets. A national research aggregator and a discipline-specific search engine harvest the endpoint on a schedule. Months later, a researcher on another continent — who has never heard of that university's repository — finds a dataset through the aggregator's search, follows the persistent link in the harvested record, and downloads it. The repository's reach multiplied far beyond its own front page, and it did so through an open, automated pull that cost the aggregators almost nothing to set up.

How it works¶

The endpoint's distinguishing property is that it serves metadata in a harvestable form: bulk list-and-get operations, a shared descriptive vocabulary that aggregators already understand, and — critically — incremental harvesting ("give me only records changed since my last pull") so that mirroring a large, active catalog stays cheap and current. Because it is open and machine-driven, any catalog can subscribe without a bespoke integration or a negotiated data-sharing deal. The result is federation: one artifact's metadata comes to live, kept fresh, inside every catalog its potential users actually consult.

Tuning parameters¶

Protocol and vocabulary — OAI-PMH, DCAT, Schema.org/JSON-LD, or plain sitemaps versus a custom API. A protocol harvesters already speak maximizes reach; a bespoke one gives control at the cost of adoption.
Metadata richness — minimal Dublin Core versus full descriptive, rights, and provenance metadata. Richer records improve discovery and filtering downstream but cost more mapping effort and go stale faster.
Incremental harvesting support — full-dump-only versus selective "changed since" harvesting. Incremental is what lets large, frequently updated catalogs be mirrored without re-pulling everything.
Rate, quota, and openness — anonymous open harvesting versus keyed or throttled access. Open maximizes reach; throttling protects the server from aggressive harvesters.
Identifier binding — whether each harvested record carries a persistent identifier, so the downstream links survive even as hosting changes.

When it helps, and when it misleads¶

Its strength is reach with equity: it multiplies an artifact's discoverability far beyond its own site and, because it is open and automated, puts that discoverability within reach of small catalogs, regional search tools, and users who would never find the origin directly — findability stops being a privilege of those who already know where to look.

Its failure mode is that it is an amplifier, so it amplifies whatever quality you feed it. A thin or stale metadata record does not stay in one place; it replicates into every harvester, and now the wrong or outdated description is everywhere at once. Broken persistent identifiers are worse: harvested records become dead ends scattered across a dozen catalogs. The classic misuse is standing up an endpoint and then letting the metadata rot, so aggregators dutifully surface records that no longer reflect the artifact. The discipline that keeps it honest is to keep records rich, fresh, and identifier-anchored, support incremental harvest so mirrors actually update, and watch what aggregators ingest — the F and A of the FAIR principles (Findable, Accessible) are exactly what a harvesting endpoint is for.[^fair]

How it implements the components¶

Metadata Harvesting Endpoint fills the open-access and reach side of the archetype — the components that get an artifact's metadata out to where it can be found:

open_access_channel — it is the open, machine-facing route by which metadata leaves the origin in bulk, without a negotiated integration for each consuming catalog.
access_equity_accommodation — by federating metadata into many catalogs and search tools, it puts discovery within reach of users and aggregators everywhere, not only those who already know the origin exists.

It serves metadata; it does not author the record — that is the Machine-Readable Manifest — nor host the artifact's files (Public Artifact Registry, Package Manager Distribution) or mint the identifiers it exposes (Persistent Identifier Minting).

Instantiates: Open Reuse Publication Infrastructure — the channel that turns a hosted artifact into a discoverable one across the wider ecosystem.
Consumes: Machine-Readable Manifest supplies the metadata records it serves; Persistent Identifier Minting supplies the stable links those records carry.
Sibling mechanisms: Public Artifact Registry · Machine-Readable Manifest · Persistent Identifier Minting · Package Manager Distribution · Open License Declaration · Schema or API Specification Publication · Semantic Versioning or Release Scheme · Changelog and Release Notes · Deprecation Notice Feed · Integrity Checksum or Signature · Example Corpus or Test Fixture · Reference Implementation Repository · Community Contribution Guidelines