How to use rendered PKL in Blender?

[gqbay]

Thank you for FrankMocap!!!!

I've just made my first try. And it's pretty successful!
But how do I use PKL motion capture data? Please, help me! =)
I'm a Blender guy

[penincillin]

@gqbay
Sorry, we are too busy to add new features in recent days. Would you mind refer to this issue to see if it can resolve your problems ?
Thanks !

[gqbay]

@gqbay
Sorry, we are too busy to add new features in recent days. Would you mind refer to this issue to see if it can resolve your problems ?
Thanks !

Unfortunately, there's nothing that could help me. I was already red that issue before you sent me.
Maybe any link, info for research? Thanks a lot!

[carlosedubarreto]

Hi @gqbay , the PKL file generated from Frankmocap seems a lot with the one generated by Vibe.
So, I think you could tha a look at the vibe documentation to be easier to understand the PKL data

https://github.com/mkocabas/VIBE/blob/master/doc/demo.md#output-format

and maybe this link from frankmocap can help you too

https://github.com/facebookresearch/frankmocap/blob/master/docs/joint_order.md

[gqbay]

Thank you so much!🙏🙌

…

On Mon, Feb 8, 2021, 4:00 PM Carlos Barreto ***@***.***> wrote: Hi @gqbay <https://github.com/gqbay> , the PKL file generated from Frankmocap seems a lot with the one generated by Vibe. So, I think you could tha a look at the vibe documentation to be easier to understand the PKL data https://github.com/mkocabas/VIBE/blob/master/doc/demo.md#output-format and maybe this link from frankmocap can help you too https://github.com/facebookresearch/frankmocap/blob/master/docs/joint_order.md — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#75 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APTY3FKLDYXZEHVKN3HEY7DS56Y4BANCNFSM4XDGJDSQ> .

[gqbay]

Hi @gqbay , the PKL file generated from Frankmocap seems a lot with the one generated by Vibe.
So, I think you could tha a look at the vibe documentation to be easier to understand the PKL data

https://github.com/mkocabas/VIBE/blob/master/doc/demo.md#output-format

and maybe this link from frankmocap can help you too

https://github.com/facebookresearch/frankmocap/blob/master/docs/joint_order.md

Nothing that I cloud understand there for my sad =(
Did you reach some success in PKL to FBX or BHV addon?

[carlosedubarreto]

I had to drop it, for 2 reasons.

Frankmocap is an amazing posing estimation tool, but it won't solve my problems. So I can't go deeper on it.
Another problem, is, that, to make a decent conversion, I would have to figure out how to do the hand pose estimation conversion, that VIBE (the original code I was using as reference [99% of the code 😅] ) didn´t have, So I would have to figure it out too....

As I was delaying my project learning how to build frankmocap on windows for almost a month, I saw that I could not delay my project anymore devoting time to frankmocap or vibe, since I wont be using them.

Basically, I stopped trying to make one conversion code. I also had very little knowledge to be able to build a decent fbx/bvh converter, sadly.

[gqbay]

Thanks for the feedback! Hope someone's will bring us desirable addon 🙌🙏

…

On Tue, Feb 9, 2021, 6:21 PM Carlos Barreto ***@***.***> wrote: I had to drop it, for 2 reasons. Frankmocap is an amazing posing estimation tool, but it won't solve my problems. So I can't go deeper on it. Another problem, is, that, to make a decent conversion, I would have to figure out how to do the hand pose estimation conversion, that VIBE (the original code I was using as reference [99% of the code 😅] ) didn´t have, So I would have to figure it out too.... As I was delaying my project learning how to build frankmocap on windows for almost a month, I saw that I could not delay my project anymore devoting time to frankmocap or vibe, since I wont be using them. Basically, I stopped trying to make one conversion code. I also had very little knowledge to be able to build a decent fbx/bvh converter, sadly. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#75 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APTY3FMA554PMILCZ2SHTVLS6ESEZANCNFSM4XDGJDSQ> .

[BluePrintRandom]

so - in blender the bones each have a matrix - update this matrix you update the bone

[BluePrintRandom]

bone = bpy.data.objects['armature'].pose.bones['BoneName']
bone.matrix = data
I think***

[mariusl04]

bone = bpy.data.objects['armature'].pose.bones['BoneName']
bone.matrix = data
I think***

Have you been able to use frankmocap's mocap data in blender?

[BluePrintRandom]

I have yet to attempt it - but I have a deep understanding of blender python, but tend to be busy.

basically, if we create a hand rig and it's armature bones are mapped the same / named the same as frankmocap,
we can set each bones transform each frame to match the real hands.

[gqbay]

I have yet to attempt it - but I have a deep understanding of blender python, but tend to be busy.

basically, if we create a hand rig and it's armature bones are mapped the same / named the same as frankmocap,
we can set each bones transform each frame to match the real hands.

My man, with all respect, seems like you're our the one and only hope 🙏🙌

[mariusl04]

I have yet to attempt it - but I have a deep understanding of blender python, but tend to be busy.

basically, if we create a hand rig and it's armature bones are mapped the same / named the same as frankmocap,
we can set each bones transform each frame to match the real hands.

I found this method that is similar and has a to .fbx converter. The output is not perfect and I don't think it has hand tracking but I thought it might help you.

https://github.com/mkocabas/VIBE

[UtsaChattopadhyay]

bone = bpy.data.objects['armature'].pose.bones['BoneName']
bone.matrix = data
I think***

Have you been able to use frankmocap's mocap data in blender?

Hey..So I used a b3d addon in blender and it has import frankmocap option there where if u give the .pkl it will make theanimation and let u export to bvh format as well..
https://github.com/carlosedubarreto/b3d_mocap_import
Just download the repo and add in the add-on in blender

[carlosedubarreto]

Oh yeah, thanks for pointing it out @UtsaChattopadhyay, I forgot to say that I finished the addon for blender.
Besides that github link you gave, I also have it in gumroad (to be honest I dont remember if githubhas the latest version.)
But I tend to focus on gumroad.
The most updated for sure will be there.

And over there its possible to see many other things that I share for free, (actually I have only one paid addon, there are others that has a value needed, but they have coupon codes on the page, I just had to do that because of the limitation on the size of the free files)

the link
https://carlosedubarreto.gumroad.com/l/mocap_import

[UtsaChattopadhyay]

Oh yeah, thanks for pointing it out @UtsaChattopadhyay, I forgot to say that I finished the addon for blender. Besides that github link you gave, I also have it in gumroad (to be honest I dont remember if githubhas the latest version.) But I tend to focus on gumroad. The most updated for sure will be there.

And over there its possible to see many other things that I share for free, (actually I have only one paid addon, there are others that has a value needed, but they have coupon codes on the page, I just had to do that because of the limitation on the size of the free files)

the link https://carlosedubarreto.gumroad.com/l/mocap_import

Hi @carlosedubarreto . Thank you for the reply. I checked them out. I have one thing to clarify though to check if it's same for you. So the output of frankmocap, the video file, has some jitters in it. Similarly the 3D animation in blender from the predicted.pkl files got the jitter as well. https://drive.google.com/drive/folders/1-81vwkmxHiCWbcDSN78GI69ZjtrvRZ6E
Did u encounter this thing also? Any suggestions/ technique on how to get rid of it?
Thanks

[carlosedubarreto]

Hello @UtsaChattopadhyay
Yes, Its very hard not to get jitters. The other one that made less jitters to me was Easymocap, but it need at least 2 cameras, calibration and so on... but the results are usually more stable.

But, one simple way that you could clean the jitters, is using the smooth option on the graph editor. It´s not perfect, but can help a lot

[UtsaChattopadhyay]

Hello @UtsaChattopadhyay Yes, Its very hard not to get jitters. The other one that made less jitters to me was Easymocap, but it need at least 2 cameras, calibration and so on... but the results are usually more stable.

But, one simple way that you could clean the jitters, is using the smooth option on the graph editor. It´s not perfect, but can help a lot

Oh great. Thanks a lot @carlosedubarreto . It really improves . Better than the previous one. And true, for easy mocap, lot clearer but it takes more time and also atleast 2 cameras to make the video first. Thanks again. :)

[haiderasad]

@UtsaChattopadhyay @carlosedubarreto i imported the pkl files into blender but
1)its not showing hands in blender output while the pkl file was of both hand+body
2)the overall blender output is pretty worse(the skeleton keeps on flipping upside down) than the video file generated through repo which gives a pretty good hand plus body avatar

any suggestions?

[carlosedubarreto]

Hello @haiderasad , If you are using mocap import to use the pkl in blender, i must say that, it has been a log time that I dont use it, and what I remember is that the import of the pkl data as made using coordinate points instead of angles.

Its easier to import, but maybe you need to adjust the size of the skeleton.

There is an option to choose the size of the skeleton for that reason, and I guess that maybe it can help you.

If its not the problem, I dont have any other idea to suggest, sorry.