Konferenzbeitrag
Evaluating DBMS-based Access Strategies to Very Large Multi-layer Corpora
Linguistic query systems are special purpose IR applications. As text sizes, annotation layers, and metadata schemes of language corpora grow rapidly, performing complex searches becomes a highly computational expensive task. We evaluate several storage models and indexing variants in two multi-processor/multi-core environments, focusing on prototypical linguistic querying scenarios. Our aim is to reveal modeling and querying tendencies – rather than absolute benchmark results – when using a relational database management system (RDBMS) and MapReduce for natural language corpus retrieval. Based on these findings, we are going to improve our approach for the efficient exploitation of very large corpora, combining advantages of state-of-the-art database systems with decomposition/parallelization strategies. Our reference implementation uses the German DeReKo reference corpus with currently more than 4 billion word forms, various multi-layer linguistic annotations, and several types of text-specific metadata. The proposed strategy is language-independent and adaptable to large-scale multilingual corpora.
- Sprache
-
Englisch
- URN
-
urn:nbn:de:bsz:mh39-48124
- Letzte Aktualisierung
-
14.09.2023, 08:26 MESZ
Objekttyp
- Konferenzbeitrag
Beteiligte
- Schneider, Roman
- Paris : European Language Resources Association (ELRA)
Entstanden
- 2016-04-26
Ähnliche Objekte (12)
A Functional Database Framework for Querying Very Large Multi-Layer Corpora
KoGra-DB: Using MapReduce for language corpora
Example-based querying for linguistic specialist corpora
GenitivDB - a corpus-generated database for German genitive classification
A Corpus Linguistic Perspective on Contemporary German Pop Lyrics with the Multi-Layer Annotated "Songkorpus"
Decision Tree-Based Evaluation of Genitive Classification – An Empirical Study on CMC and Text Corpora. Language Processing and Knowledge in the Web
Challenges in the Alignment, Management and Exploitation of Large and Richly Annotated Multi-Parallel Corpora
E-VALBU: Advanced SQL/XML processing of dictionary data using an object-relational XML database
A database-driven ontology for German grammar
memasysco: XML schema based metadata management system for speech corpora
SPLICR: A Sustainability Platform for Linguistic Corpora and Resources
A Web-Platform for Preserving, Exploring, Visualising, and Querying Linguistic Corpora and other Resources
A Functional Database Framework for Querying Very Large Multi-Layer Corpora
KoGra-DB: Using MapReduce for language corpora
Example-based querying for linguistic specialist corpora
GenitivDB - a corpus-generated database for German genitive classification
A Corpus Linguistic Perspective on Contemporary German Pop Lyrics with the Multi-Layer Annotated "Songkorpus"
Decision Tree-Based Evaluation of Genitive Classification – An Empirical Study on CMC and Text Corpora. Language Processing and Knowledge in the Web
Challenges in the Alignment, Management and Exploitation of Large and Richly Annotated Multi-Parallel Corpora
E-VALBU: Advanced SQL/XML processing of dictionary data using an object-relational XML database
A database-driven ontology for German grammar
memasysco: XML schema based metadata management system for speech corpora
SPLICR: A Sustainability Platform for Linguistic Corpora and Resources
A Web-Platform for Preserving, Exploring, Visualising, and Querying Linguistic Corpora and other Resources
A Functional Database Framework for Querying Very Large Multi-Layer Corpora
KoGra-DB: Using MapReduce for language corpora
Example-based querying for linguistic specialist corpora
GenitivDB - a corpus-generated database for German genitive classification
A Corpus Linguistic Perspective on Contemporary German Pop Lyrics with the Multi-Layer Annotated "Songkorpus"
Decision Tree-Based Evaluation of Genitive Classification – An Empirical Study on CMC and Text Corpora. Language Processing and Knowledge in the Web
Challenges in the Alignment, Management and Exploitation of Large and Richly Annotated Multi-Parallel Corpora
E-VALBU: Advanced SQL/XML processing of dictionary data using an object-relational XML database
A database-driven ontology for German grammar
memasysco: XML schema based metadata management system for speech corpora
SPLICR: A Sustainability Platform for Linguistic Corpora and Resources