Foundations

Overview

Questions

• what best practice and generic skills underpin your encounters with data and research?

Objectives

• identify and use best practice in data structures
• identify and understand a data-driven mindset

Foundations

---

In the last episode, we discussed what we each think of as data. We came up with a lot of different ideas of what data looks like and how it can be used. Before we crack on with using the computational tools at our disposal, I want to spend some time on some foundation level stuff - a combination of best practice and generic skills that frame what you’ll encounter across Archive Carpentry.

Trainer Note: we recommend using this section as an opportunity to discuss foundational skills that you think are relevant.

Data are Collected Through Research

To summarize the brainstorming session that we had in the last episode, data are information collected through research. As archivists, we support research. When we start to think of our collections as data, we can start to support new methods of providing access to our data. Data can be manipulated using automated or computational methods, allowing us to improve our workflows. When approaching our work with a data-aware mindset, we should think of the systems that we are using to do our work.

The computer and the systems inside it are stupid

This does not mean that the computer isn’t useful. Given a repetitive task, an enumerative task, or a task that relies on memory, it can produce results faster, more accurately, and less grudgingly than you or I. Rather when I say that you should keep in mind that the computer is stupid, I mean to say that computer only does what you tell it to. If it throws up an error, it is often not your fault; in most cases, the computer has failed to interpret what you mean because it can only work with what it knows (ergo, it is bad at interpreting). This is not to say that the people who told the computer what to tell you when it doesn’t know what to do couldn’t have done a better job with error messages – they could. So keep in mind as we go along that if you find an error message frustrating, it isn’t the computer’s fault that it is giving you an archaic and incomprehensible error message, it is a human person’s.

• The correct language to learn is the one that works in your local context. There truly isn’t a best language, just languages with different strengths and weaknesses, all of which incorporate the same fundamental principles;
• Knowing the structure of the interface that you are using will assist you in learning. Databases and computer systems can seem opaque. Knowing what data structures they were built to support can help you to troubleshoot
• Automate to make the time to do something else! Taking the time to gather together even the most simple programming skills can save time to do more interesting stuff! (even if often that more interesting stuff is learning more programming skills …)
• Understanding the interface can help you to communicate with developers and engineers Taking the time to gather together even the most simple programming skills can help you to better communicate your needs to developers.

Beyond the Interface

Much of the work that you do with data may be completed through a software interface. Your archival catalog and Excel spreadsheets are interfaces that allow you to view your data more easily. The data itself is organized into structures that many of you will be familiar with, but is much more text-heavy and may not be as simple for humans to read.

Plain text formats are your friend

Why? Because computers can process them! Structures and formats that may be easier for humans to read often cannot be read by computers.

If you want computers to be able to process your stuff, try to get into the habit of using platform-agnostic formats where possible, such as .txt for notes and .csv or .tsv for tabulated data (the latter pair are just spreadsheet formats, separated by commas and tabs respectively). These plain text formats are preferable to the proprietary formats used as defaults by Microsoft Office because they can be opened by many software packages and have a strong chance of remaining viewable and editable in the future. Most standard office suites include the option to save files in .txt, .csv and .tsv formats, meaning you can continue to work with familiar software and still take appropriate action to make your work accessible. Compared to .doc or .xls, these formats have the additional benefit of containing only machine-readable elements.

Whilst it is common practice to use bold, italics, and colouring to signify headings or to make a visual connection between data elements, these display-orientated annotations are not (easily) machine-readable, and hence can neither be queried and searched nor are appropriate for large quantities of information (the rule of thumb is, if you can’t find it by CTRL+F, it isn’t machine readable). It is preferable to use standards that signify heading levels, as these standards are not only machine-readable, but also translate easily across web browsers and potential future content migrations.

In archival practice, standards have been developed in order for computers to understand the methods that we use to describe our collections. ISAD(G) – General International Standard Archival Description – has helped archivists to determine how to describe their collections but EAD – Encoded Archival Description – has given archivists a standard way to format their description.

Key Points

• data are used in research
• archival collections and archival description are data
• data structures should be consistent and predictable
• consider the standards and structures used in your own data
• identify and use computational methods in your work
• identify how standards and structures can be used in research