Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TensorRT的采样可以和QWen官方generation_config.json里面提供的采样参数对齐吗？ #69

Open
Hukongtao opened this issue Jan 26, 2024 · 4 comments
Open

TensorRT的采样可以和QWen官方generation_config.json里面提供的采样参数对齐吗？ #69

Hukongtao opened this issue Jan 26, 2024 · 4 comments

Comments

@Hukongtao
Copy link

image
@Tlntin
Copy link
Owner

Tlntin commented Jan 27, 2024

可以的

@Hukongtao
Copy link
Author

Hukongtao commented Jan 29, 2024 •
edited
Loading

image
我跑的Qwen官方的例子，每次出来的结果都是不一样的，采样参数如问题里面的截图所示。
但是我使用tensorrt-llm跑，每次的结果都是一样的。参数应该和hf对齐的
image
不太理解为啥tensor-llm每次推理的结果都是一样的

@Tlntin
Copy link
Owner

Tlntin commented Jan 29, 2024

可能是要每次都传一个不一样的随机种子给SamplingConfig

@Hukongtao
Copy link
Author

可能是要每次都传一个不一样的随机种子给SamplingConfig

试了一下，确实是这样。默认random_seed为None，竟然不具有随机性

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Hukongtao @Tlntin