Alta-tokenizer is a tool designed for converting Kinyarwanda language text into tokens, it can also tokenizer other languages like English, French or similar languages but with low compression rate since the tokenizer was trained on Kinyarwanda text only. There is an option for training your own custom tokenizer using defined function or method. It is covered in the section of training your own tokenizer. Hence, you can use that way to train your own tokenizer using dataset for a different language. It is based on the Byte Pair Encoding (BPE) algorithm. It can both encode and decode text in Kinyarwanda.

Tokenization
Tokenization is the first step in preparing dataset raw text for processing by foundational models, especially in NLP tasks. Human language is inherently unstructured sentences can vary in length, contain punctuation, and follow complex grammatical rules. Machines, however, require structured, numerical input to process and understand language. Tokenization bridges this gap by converting text into a format that models can work with. Tokenization is a critical and fundamental process in natural language processing (NLP) that serves as the critical first step in preparing raw text data for analysis or modeling by converting unstructured language into a structured format that machines can process.

Metrics
The metric used to measure the accuracy of this tokenizer is the compression rate and ability to encode and decode texts Compression rate is the ratio of the total original number of characters in the text to the number of tokens in the encoded text.

For example the sentence: "Nagiye gusura abanyeshuri."
The sentence has 26 characters. Suppose the sentence is tokenized into the following tokens:[78, 1760, 32, 5256, 32, 1845, 46]. The total number of tokens is 7. So, the compression rate is 3.714X(where X indicates that the number is approximate).