maze_dataset.tokenization.modular
implements ModularMazeTokenizer
and related code
1"""implements `ModularMazeTokenizer` and related code""" 2 3__all__ = [ 4 # modules 5 "all_tokenizers", 6 "element_base", 7 "elements", 8 "fst_load", 9 "fst", 10 "hashing", 11 "maze_tokenizer_modular", 12 "save_hashes", 13]