Organizing corpora at the Stanford Literary Lab. Balancing simplicity and flexibility in metadata management

Konferenzbeitrag

This article describes a series of ongoing efforts at the Stanford Literary Lab to manage a large collection of literary corpora (~40 billion words). This work is marked by a tension between two competing requirements – the corpora need to be merged together into higher-order collections that can be analyzed as units; but, at the same time, it’s also necessary to preserve granular access to the original metadata and relational organization of each individual corpus. We describe a set of data management practices that try to accommodate both of these requirements – Apache Spark is used to index data as Parquet tables on an HPC cluster at Stanford. Crucially, the approach distinguishes between what we call “canonical” and “combined” corpora, a variation on the well-established notion of a “virtual corpus” (Kupietz et al., 2014; Jakubíek et al., 2014; van Uytvanck, 2010).

Object Details

• Language: Englisch

Classification and Topics

• Subject: Korpus <Linguistik> ; Englisch ; Texttechnologie ; Datenmanagement ; Metadaten ; Sprache

Contributors, Places and Time

• Event: Geistige Schöpfung
• (who): McClure, David ; Algee-Hewitt, Mark ; Douris, Steele ; Fredner, Erik ; Walser, Hannah
• Event: Veröffentlichung
• (who): Mannheim : Institut für Deutsche Sprache
• (when): 2017-07-05

Further information

• URN: urn:nbn:de:bsz:mh39-62617
• Last update: 06.03.2025, 9:00 AM CET