Feature/support qwenvl glm4-v (tested) #4377
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
终于还差一个image的padding处理就能做好训练支持了。 |
|
@hiyouga 改的比较多捏，有空帮忙看看这个实现思路行不行。谢谢。 |
marko1616
commented
Jun 20, 2024
| Image.fromarray(image).convert("RGB").save(image_path) | ||
| messages[-1]["content"] = template.format_image.apply(content=os.fspath(image_path))[0] + messages[-1]["content"] | ||
| elif image is not None and model_args.visual_inputs_type == "vision_message_embed": | ||
| messages[-1]["content"] = template.format_image.apply()[0] + messages[-1]["content"] |
There was a problem hiding this comment.
如果不是内嵌在文本的image url默认放在最后一个的开头（Qwenvl如果不是开头效果不好）
marko1616
commented
Jun 20, 2024
| if model_args.visual_inputs_type == "vision_message_embed": | ||
| dataset = dataset.rename_column("image_inputs","images") | ||
| print(dataset["images"]) | ||
|
|
There was a problem hiding this comment.
dataset.map不能重用删除的column_name
marko1616
commented
Jun 20, 2024
| transforms.Normalize((0.48145466, 0.4578275, 0.40821073), (0.26862954, 0.26130258, 0.27577711)), | ||
| ] | ||
| ) | ||
| return transform(images[0]) if len(images) != 0 else transform(Image.new("RGB", (1120, 1120), (255, 255, 255))) |
marko1616
commented
Jun 20, 2024
| if model_args.visual_inputs and finetuning_args.freeze_vision_tower: | ||
| target_modules = "^(?!.*vision_tower).*(?:{}).*".format("|".join(target_modules)) | ||
| if model_args.visual_inputs and finetuning_args.freeze_vision: | ||
| target_modules = f"^(?!.*{VISION_FREEZE_MAP[model_args.visual_inputs_type]})."+"*(?:{}).*".format("|".join(target_modules)) |
There was a problem hiding this comment.
其实还有点小问题，可能会把GLM4的视觉模块附加了
|
成功跑了训练。 |
marko1616
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes #4375
Before submitting