graystack

Projects / Archival Index

MAC · BOX 1–18Archival Index · a records QA workbench

From a shoebox of lists to an archive you can trust.

Three volunteers, three ways of typing a box list. Archival Index turns them into consistent records, makes two different people check each one, keeps the history of every change, and publishes only what passed.

Role
Data standards, records QA, product
Platform
Web app, plus a SQLite export
Built with
JavaScript, SQL (SQLite), Python
Status
Concept study · working app

01 / The problem

Same archive, three volunteers, three formats.

Small archives run on volunteers, and every volunteer writes a box list their own way. Before anyone can search the collection, those lists have to agree, and someone has to be able to say who checked what.

Box list A · a spreadsheetBox,Folder,Description,Date 3,5,Memo re: insurance,05/06/1974 4,3,Misc. letters,undated
Box list B · typed notesBOX 8 f1 Photographs, opening of community hall (b&w, 12 prints) c.1968 f2 undated photograph - unidentified group
Box list C · shorthandBx 13 / Fdr 1 / Oral history w/ J. Harlan, founding member / 1979-80 / 2 cassettes Bx 18 / Fdr 1 / Misc / ?
Dates in every shape

"c.1968", "1972-74", "1970s", "05/06/1974". People can read them; a search box can't.

One pair of eyes

Whoever typed the list also checks it, so their mistakes go straight into the catalogue.

No trail

When a title changes, nobody can say who changed it, when, or what it said before.

A concept study, honestly framed. The Millbrook Arts Council, its volunteers and its records are invented, and there was no formal research. The workflow follows common archival practice: controlled vocabularies, two-person checking and recorded provenance.

02 / The normaliser

Type a line the way a volunteer would.

This runs the app's own normaliser. It reads all three formats, builds a call number, cleans the title, picks a record type from a controlled list, turns the date into something a computer can sort, and raises the checks a cataloguer has to deal with.

Try one

03 / The workflow

Four stages, and nobody checks their own work.

Every record moves from a raw line to the public finding aid, and every step is recorded in its history.

  1. Intake preview of box list C
    Draft

    Intake

    Preview a list as written beside the normalised records. Nothing is added until you confirm, and unreadable lines are named.

  2. Cataloguing a record with open checks
    Catalogued

    First pass

    A cataloguer fixes each record against its source line. Every check must be fixed or acknowledged with a note.

  3. Verifying a record against its source
    Verified

    Second pass

    A different person compares record and source, then verifies it or returns it with a reason.

  4. The public finding aid
    Published

    Finding aid

    Only verified records appear, arranged by series, box and folder, ready to search or print.

The two-person ruleThe app refuses to let a cataloguer verify their own record, and the exported database refuses it too.

04 / Standards

Dates people can read and computers can sort.

Each date keeps a readable form for the finding aid and a start and end for searching and sorting. This table is produced live by the same code as the app.

As typedShown asMachine rangeFlags

05 / The data layer

A database that enforces the same rules.

The workbench exports a SQLite script: series, records and every history event. Its constraints repeat the app's rules, and a short Python script reads it to produce a QA report.

export.js · part of the schemaView source
  status        TEXT NOT NULL CHECK (status IN ('draft', 'catalogued', 'verified', 'flagged')),
  catalogued_by TEXT,
  verified_by   TEXT,
  -- Two-pass QA: a verified record names a verifier who is not its cataloguer.
  CHECK (status <> 'verified' OR (verified_by IS NOT NULL AND catalogued_by IS NOT NULL AND verified_by <> catalogued_by)),
  CHECK (date_start IS NULL OR date_end IS NULL OR date_start <= date_end)

CREATE VIEW finding_aid AS
  SELECT s.title AS series, r.box, r.folder, r.call_number, r.title, r.date_display, r.extent
  FROM records r JOIN series s ON s.id = r.series_id
  WHERE r.status = 'verified'
python qa_report.py mac-archive.sqlreal output, trimmed
Archival Index QA report: 36 records

Status
  verified      16  ###########.............
  catalogued     8  #####...................
  flagged        1  #.......................
  draft         11  #######.................

Boxes (verified / listed)
  Box  1   5 / 5   ############
  Box  2   3 / 5   #######.....
  Box  3   4 / 5   ##########..
  …

Dates: 3 undated, 7 approximate
Two-pass rule: OK

Work by person
  maya   catalogued  23   verified   0   returned  0
  sam    catalogued   2   verified  16   returned  1

Finding aid: 16 of 36 records published

06 / Screens

The workbench, captured from the app.

Select any screen to see it larger.

The shelfEach box shows how far its folders have got, and empty boxes show what's still to be listed.
CatalogueSource line on top, fields below, and checks that block the first pass until they're dealt with.
VerifySource beside record. V verifies, R returns with a note.
RecordsSearch every field, including the original wording, with filters that count as you narrow.
ProvenanceEvery import, edit, check, pass and return, with before and after.
Finding aidThe public face of the collection, built only from verified records.
IntakeA new list, previewed line by line before anything is added.
DataCSV, SQLite and JSON exports, the schema, and the Python report.

07 / Try it

The real workbench, with the sample collection.

Switch between Maya, the cataloguer, and Sam, the verifier, at the top right. This demo starts fresh each time; the full app keeps your work in your browser.

  1. 1

    Import box list C. Intake → Preview, then add it, and spot the duplicate folder.

  2. 2

    Catalogue a record. Fix or acknowledge its checks, then mark it catalogued.

  3. 3

    Try to verify it as Maya. You can't. Switch to Sam and verify it, then find it in the finding aid.

Everything stays in your browser. Export a SQLite file from the Data page whenever you like.

Open the full app

08 / How it's built

Rules in small pure modules, a thin interface on top.

The normaliser, the workflow and the exporters know nothing about the page, so the same code runs the workbench, the demo above, and the script that generated the sample export.

  • ~1,390lines across JavaScript, CSS and Python
  • 3box-list formats read without being told which
  • 8checks a cataloguer must fix or acknowledge
  • 0records published without two people
  1. A normaliser with no side effectsnormalize.js reads lists, parses dates, guesses types from a controlled vocabulary and lists checks. Text in, records out.
  2. One door for every changeworkflow.js is the only code that edits a record, and each edit adds a history event with who, when, and before and after.
  3. Checks that block, not nagA record can't be catalogued while a check is open. Acknowledging one needs a note, which is saved in the history too.
  4. Rules repeated in the databaseThe SQLite schema's CHECK constraints match the app, so an export edited by hand still can't publish an unchecked record.
  5. Standard library Pythonqa_report.py loads the export into an in-memory SQLite database and needs nothing installed.
  6. Searchable finding aidSearch covers every field, including the volunteers' original wording, and the finding aid prints cleanly.

09 / What I'd test next

What a real archive would teach it.

The workflow is built on common practice, but it hasn't met real volunteers or real boxes yet.

How many lines does the normaliser read correctly?

Collect real box lists and measure how often the type, date and extent come out right first time.

Test: a corpus of real lists
Is two-person checking sustainable?

Small archives may have one regular volunteer. A lighter rule, such as sample checking, might be needed.

Test: interviews with archive managers
Do checks help or slow people down?

Eight checks catch real problems, but too many acknowledgements could become box-ticking.

Test: time-on-task with and without checks
Can it export to archival standards?

EAD or a catalogue system's import format would let records move into a real collection system.

Next: an EAD export