Tokenizer Choice For LLM Training: Negligible or Crucial?
Ali, Mehdi; Fromm, Michael; Thellmann, Klaudia; Rutmann, Richard; Lübbering, Max; Leveling, Johannes; Klug, Katrin; Ebert, Jan; Doll, Niklas; Schulze Buschhoff, Jasper; Jain, Charvi; Weber, Alexander Arno; Jurkschat, Lena; Abdelwahab, Hammam; John, Chelsea; Suarez, Pedro Ortiz; Ostendorff, Malte; Weinbach, Samuel; Sifa, Rafet; Kesselheim, Stefan; Flores-Herr, Nicolas, 2023