This repository has been archived by the owner on Oct 17, 2024. It is now read-only.
LogicKor V2 Update #37
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- add PROMPT_STRATEGY - change judge_template - change evaluation template
boxqkrtm
approved these changes
Jun 28, 2024
|
#34 잘 들어갔네요 |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
LogicKor V2 Update
업데이트 동기
업데이트 사항
judge_template 수정
Template 내에서 사용 되는 용어 및 요구하는 답변의 조건들이 바뀌었습니다.
모델 템플릿 사용 방식 수정
generator에서 생성시 모델의 chat_template 파라미터를 사용합니다.
evaluator 수정
특정 폴더에 있는 generation 결과들을 한번에 평가합니다.
Merge 이전 Todo
Further Work
템플릿이 수정되고 모델들의 성능이 좋아짐에 따라 상당수의 모델이 최고점에 가까운 점수를 얻고 있습니다. LogicKor-Hard등의 어려운 데이터를 포함한 데이터셋이 필요하다고 생각됩니다.
본 Repo를 확장하여 LogicKor-Hard를 포함한 여러 데이터를 기반으로 평가 할 수 있도록 하는 작업이 필요할 것 같습니다.
Contact
추가적인 이야기를 함께 나누고 싶으시다면,
https://instruct.kr/