LnL-AI
/

dbrx-base-tokenizer

Transformers.js

Model card Files Files and versions

dbrx-base-tokenizer / tokenizer.json

Commit History

add pad and extra_N special tokens

3a0d911

Qubitium commited on Apr 3, 2024

Upload tokenizer.json with huggingface_hub

a66b082
verified

Qubitium commited on Apr 3, 2024