Question about training the agent

[MitchellBrovarnik]

so when I run

python -m tmrl --worker in my powershell i get this over and over and over again

INFO:root:02/06/25 01:06:28 copying buffer for sending
INFO:root:02/06/25 01:06:28 checking for new weights
INFO:root:02/06/25 01:06:28 collecting train episode
INFO:root:02/06/25 01:06:50 copying buffer for sending
INFO:root:02/06/25 01:06:50 checking for new weights
INFO:root:02/06/25 01:06:50 collecting train episode
INFO:root:02/06/25 01:06:57 copying buffer for sending
INFO:root:02/06/25 01:06:57 checking for new weights
INFO:root:02/06/25 01:06:57 collecting train episode
INFO:root:02/06/25 01:07:05 copying buffer for sending
INFO:root:02/06/25 01:07:05 checking for new weights

I also did record the reward for the map prior to running this, Is this the output I should be getting/how do I get the agent to follow the path I gave it during the reward run?

When I run python -m tmrl --test in my powershell nothing happens to my car, it doesn't move but the game still randomly resets itself? I did add the weights_only=True in actor.py

How do I also save the model so it can resume from what it already learned next time I run it or does it automatically do that/how can I tell from the data if it's learning? Because every time I restart it the memory len gets small.

Also, I have an amd gpu and not nvidia, so currently the training is defaulting to my cpu, is there a way for me to edit the code to use direct ml so i can use my amd gpu for training?

Lastly, how do I know if the agent is using lidar or sac?

Thanks in advance and sorry for the load of questions, I'm new to this!

[yannbouteiller]

Hi, I moved your question to the discussions section (issues are for reporting bugs in the code).

• When using python -m tmrl --test, you simply rollout your current AI in standalone mode, without doing any training. If the game doesn't move but randomly resets itself, it means that your current AI just stays there and does nothing. If you tried to train your own AI, this is normal, but if you are testing the AI that is provided with tmrl on the tmrl-test track, the car should move and complete the track (before trying anything, you should make sure that the default AI works on tmrl-test. If you messed something up you can delete or rename the TmrlData folder and do python -m tmrl --install to get it back to its initial state)
• When using python -m tmrl --worker, you are creating a Rollout Worker whose role it to collect training samples from the game to send them to a server (which you need to launch in another terminal with python -m tmrl --server). This server will transfer these samples to a trainer (which you need to launch with python -m tmrl --trainer) that will use them to train your AI. However, if you have an AMD GPU, training will not use it as it expects CUDA (i.e. NVIDIA) at the moment. tmrl does not support AMD's ROCm technology yet (as it only has beta support from PyTorch). If you want to use an AMD GPU, you will need to implement support for it in the tmrl codebase, which requires moderate proficiency in Python.
• The model is saved each time your trainer sends new weights. However, if you are training on CPU, this will take an unreasonable amount of time. As for the memory, it is checkpointed by the trainer after each training epoch, which is even much longer by default. You can change the length of training epochs in config.json
• SAC is the training algorithm used by default, regardless of whether you use LIDAR or full images. By default, tmrl uses full images (and resizes the TrackMania window to a very small window so that screenshots are small.)

[yannbouteiller]

It is a model trained to complete the tmrl-test track specifically. Have you tested the policy on this track?

[yannbouteiller]

For cuda yes, if you look for all the occurences of "cuda" in the code you should find all the parts that need to be adapted. I haven't tried using pytorch with AMD personnally, so I cannot help much there

[yannbouteiller]

For your last question, yes it does. Recording a reward saves your reward function in the TmrlData folder, and then this is the reward that gets used by tmrl.

[MitchellBrovarnik]

It is a model trained to complete the tmrl-test track specifically. Have you tested the policy on this track?

Yes I have, on a fresh new install, it wasn't able to complete the track, but it did get through some of it.

For your last question, yes it does. Recording a reward saves your reward function in the TmrlData folder, and then this is the reward that gets used by tmrl.

This gets used when training, correct? But first i'd need to set the ENV to TM20FULL for it to use those rewards, yes?

[yannbouteiller]

Yes I have, on a fresh new install, it wasn't able to complete the track, but it did get through some of it.

Something is wrong then, it should complete the track without too much difficulty. Do you get spammed with many "time step timed out" warnings? If not, perhaps you are not using the same camera angle as the one the model was trained on, or something like this.

This gets used when training, correct? But first i'd need to set the ENV to TM20FULL for it to use those rewards, yes?

Yes this is the reward used when training, and also to detect when the car goes off-track or fails to move forward. Those rewards will be used regardless of the ENV that you set.

[MitchellBrovarnik]

Something is wrong then, it should complete the track without too much difficulty. Do you get spammed with many "time step timed out" warnings? If not, perhaps you are not using the same camera angle as the one the model was trained on, or something like this.

Nope, I don't see that many "time step timed out" warnings, sure I get a few, but not that often. Maybe I didn't uninstall it properly and that messed something up, i'll try to again and see what happens.

thanks for your help thus far though!!

edit: I just tried agian doing a clean install, I did pip uninstall tmrl and then deleted the tmrldata folder and then did pip install tmrl and then finally python -m tmrl --install the first thing i did was load up the test map and run python -m tmrl --test it still hasnt been able to finish the map at all, if its supposed to look anything like what you showed in the video; it doesn't, it some reason really strugles... I don't understand what I did wrong. I put the lowest possible graphics setting and it opens in a tiny window when i run the command, i have fps set to 30, if it has anything to do with not being in first person then that's my answer, me clicking 3 does absolutely nothing. Also, when running the test, it looks as though the agent is unsure of what to do, it goes side to side a lot and does random stuff. I could record it and upload it so you can see, but if it's because of me not being in first person then I'm not sure why pressing 3 doesn't do anything.

edit 2: when i run with the "SAC_4_LIDAR_pretrained" it's even worse at completing the map, it just auto turns right and gets stuck and keeps reseting itself.

[MitchellBrovarnik]

Perhaps something is weird in how TrackMania maps gamepad inputs?

I set gamepad to false and nothing really changed in the test run.

You can also refer to these instructions to double-check that the camera angle is properly setup for the model pretrained in the Full environment.

My settings in config are identical to the ones you have for the SAC_4_imgs_pretrained I did a fresh install yesterday and didn't change anything, and everything was the same as what you had. So I'm really confused why it's not working, it doesn't make sense but here we are.

[yannbouteiller]

I am also very confused, this should work rightaway after you delete the TmrlData folder. In theory when you delete this folder, tmrl detects that it is not where it should be and downloads a clean version, including clean models and a clean config.json that by default work on the tmrl-test track in 3rd person view.

So if this happens properly and your timesteps don't timeout like crazy, the only thing I can possibly see would be an issue with how the game it setup.

[yannbouteiller]

BTW do you have this issue with the version installed from PyPI with pip install tmrl ?

[yannbouteiller]

Also for the "full" environment, the camera angle affects the performance of the model, like if you select the view that is too high or too low above the car, the model doesn't work well, maybe that's your issue with this environment, idk?

For the LIDAR environment though, if you properly hide the car in 1st person view so that you see only the road, it should quite reliably complete the tmrl-test track (unless something has changed in the game colors... I'll double-check because I haven't tested the LIDAR environment in a while)

[MitchellBrovarnik]

BTW do you have this issue with the version installed from PyPI with pip install tmrl ?

I am not sure what you mean by this, I just followed the installation instructions, ran the commands there needed to get everything setup, but still it doesn't work how it should. I'm using the Canada skin and everything too.