docs for maze-dataset v1.3.0
View Source on GitHub

maze_dataset.tokenization.modular

implements ModularMazeTokenizer and related code


 1"""implements `ModularMazeTokenizer` and related code"""
 2
 3__all__ = [
 4	# modules
 5	"all_tokenizers",
 6	"element_base",
 7	"elements",
 8	"fst_load",
 9	"fst",
10	"hashing",
11	"maze_tokenizer_modular",
12	"save_hashes",
13]