Add Tokenizer support for Sagemaker Endpoint Backend#213

Closed

shinglyu wants to merge 7 commits intosobelio:mainfrom shinglyu:sagemaker-endpoint-tokenizer

+139-23

Commits on Aug 29, 2023

feat: added tokenizer for sagemaker-endpoint
Shing Lyu
committed

Commits on Aug 30, 2023

feat: added max_token_allowed information
Shing Lyu
committed
feat: make answer prefix None
Shing Lyu
committed
feat: added tokens_used info
Shing Lyu
committed
fix: share the fomatter between falcon 7b and 40b
Shing Lyu
committed
fix: cargo fmt
Shing Lyu
committed
fix: minor review comment fixes
Shing Lyu
committed