Konferenzbeitrag
Matrix and double-array representations for efficient finite state tokenization
This paper presents an algorithm and an implementation for efficient tokenization of texts of space-delimited languages based on a deterministic finite state automaton. Two representations of the underlying data structure are presented and a model implementation for German is compared with state-of-the-art approaches. The presented solution is faster than other tools while maintaining comparable quality.
- Sprache
-
Englisch
- Thema
-
Algorithmus
Endlicher Zustandsraum
Datenstruktur
Deutsch
Korpus <Linguistik>
Sprache
- Ereignis
-
Geistige Schöpfung
- (wer)
-
Diewald, Nils
- Ereignis
-
Veröffentlichung
- (wer)
-
Paris : European Language Resources Association (ELRA)
Mannheim : Leibniz-Institut für Deutsche Sprache (IDS)
- (wann)
-
2022-07-01
- URN
-
urn:nbn:de:bsz:mh39-111091
- Letzte Aktualisierung
-
06.03.2025, 09:00 MEZ
Datenpartner
Dieses Objekt wird bereitgestellt von:
Leibniz-Institut für Deutsche Sprache - Bibliothek. Bei Fragen zum Objekt wenden Sie sich bitte an den Datenpartner.
Leibniz-Institut für Deutsche Sprache - Bibliothek. Bei Fragen zum Objekt wenden Sie sich bitte an den Datenpartner.
Objekttyp
- Konferenzbeitrag
Beteiligte
- Diewald, Nils
- Paris : European Language Resources Association (ELRA)
- Mannheim : Leibniz-Institut für Deutsche Sprache (IDS)
Entstanden
- 2022-07-01
Ähnliche Objekte (12)
Matrix and double-array representations for efficient finite state tokenization
Linear representations of finite groups
Linear representations of finite groups
Orthogonal representations of finite groups
Linear representations of finite groups
Representations of finite dimensional algebras
Efficient operation execution on multidimensional array data
Efficient operation execution on multidimensional array data
Integrality of representations of finite groups
Stratifying modular representations of finite groups
Tonhöhenempfindung und Sprachverstehen mit dem Nucleus Double Array Implantat
Representations of finite Chevalley groups : a survey
Matrix and double-array representations for efficient finite state tokenization
Linear representations of finite groups
Linear representations of finite groups
Orthogonal representations of finite groups
Linear representations of finite groups
Representations of finite dimensional algebras
Efficient operation execution on multidimensional array data
Efficient operation execution on multidimensional array data
Integrality of representations of finite groups
Stratifying modular representations of finite groups
Tonhöhenempfindung und Sprachverstehen mit dem Nucleus Double Array Implantat
Representations of finite Chevalley groups : a survey
Matrix and double-array representations for efficient finite state tokenization
Linear representations of finite groups
Linear representations of finite groups
Orthogonal representations of finite groups
Linear representations of finite groups
Representations of finite dimensional algebras
Efficient operation execution on multidimensional array data
Efficient operation execution on multidimensional array data
Integrality of representations of finite groups
Stratifying modular representations of finite groups
Tonhöhenempfindung und Sprachverstehen mit dem Nucleus Double Array Implantat