Objective vs. Search: Decomposing What Makes a Good Tokeniser

Problem

This work addresses the confounding of optimization objectives and search procedures in tokenization algorithms, a gap in the literature that has implications for the design and evaluation of tokenizers. The authors highlight that existing tokenization methods may not adequately separate these two components, leading to suboptimal performance in various applications. This paper is a preprint and has not yet undergone peer review.

Method

The authors investigate two tokenization algorithms: BottomUpLL, which is a bottom-up likelihood-based approach, and TopDownComp, a top-down compression-based method. They conduct experiments by varying model sizes, vocabulary sizes, and domains, specifically comparing English-only and multilingual settings. The evaluation metrics employed include bits-per-byte, which measures the efficiency of the tokenization, and performance on the BLiMP task, which assesses linguistic acceptability.

Results

The results indicate that BottomUpLL tokenizers achieve lower bits-per-byte in most settings compared to TopDownComp tokenizers, suggesting a superior efficiency in the former. However, the authors report no consistent relationship between design choices and performance on the BLiMP task, indicating that the effectiveness of tokenization strategies may not be straightforwardly linked to their underlying design principles. The available text does not report quantitative results for the BLiMP task.

Limitations

The authors do not flag any evident limitations in their study. However, the lack of a consistent relationship between design choices and performance on the BLiMP task may suggest potential areas for further investigation, particularly in understanding the nuances of tokenization performance across different linguistic contexts.

Why it matters

This research has significant implications for downstream work in natural language processing, particularly in the development of more effective tokenization strategies. By clarifying the roles of optimization objectives and search procedures, the findings can guide future research in designing tokenizers that are both efficient and effective across diverse applications, potentially improving model performance in tasks such as language modeling and machine translation.