@g6 XddlmZmZdZdZdZdZdZdZdZ d Z d Z d Z ee e e e d Z d S))data_gym_to_mergeable_bpe_ranksload_tiktoken_bpez <|endoftext|>z<|fim_prefix|>z<|fim_middle|>z<|fim_suffix|>z<|endofprompt|>cBtdd}ddd|tdidS) NzIhttps://openaipublic.blob.core.windows.net/gpt-2/encodings/main/vocab.bpezLhttps://openaipublic.blob.core.windows.net/gpt-2/encodings/main/encoder.json)vocab_bpe_fileencoder_json_filegpt2QJ's|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+Pnameexplicit_n_vocabpat_strmergeable_ranksspecial_tokens)r ENDOFTEXTrs N/var/www/piapp/venv/lib/python3.11/site-packages/tiktoken_ext/openai_public.pyrr sA5bhO !d*$e,   c>td}ddd|tdidS)NzGhttps://openaipublic.blob.core.windows.net/encodings/r50k_base.tiktoken r50k_baser r r r rrrs rrr9'QO!d*$e,   rc>td}ddd|tdidS)NGhttps://openaipublic.blob.core.windows.net/encodings/p50k_base.tiktoken p50k_baseir r r rrs rrr%rrcjtd}tdtdtdtdi}dd||dS) Nrr rijik p50k_editr r rrr)rr FIM_PREFIX FIM_MIDDLE FIM_SUFFIXrrs rrr2sH'QO  E:ujZ_`Nd*(   rc xtd}tdtdtdtdt di}dd||d S) NzIhttps://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktokeniiiii cl100k_basezs(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+r )rrr!r"r# ENDOFPROMPTr$s rr&r&?sV'SO 6FFFV NN*(   r)rrrrr&N) tiktoken.loadrrrr!r"r#r'rrrrr&ENCODING_CONSTRUCTORSrrr+sLLLLLLLL                  (  r