How to setting local file location on server side, how to delete local files there?

[wztdream]

Hi,
I am using wandb-local, it is great. but during training my computer says my disk is full.
I checked the disc usage it shows my system hard drive is used up by wandb

I have a small hard disc only used for system files, but wandb used up this disc. I have a much larger hard disc to store other data exept system fiels, and mount it under /home folder, but wandb did not use it. So there are two questions need help:

1. how to setting wandb-local to store local data on specific folder, for example under my large /home folder
2. how to delete the large local files

[exalate-issue-sync]

Luis Bergua commented:
Hi wztdream,

Thanks for writing in!

1. You can specify the argument dir in wandb.init(). This is an absolute path to a directory where metadata will be stored. When you call download() on an artifact, this is the directory where downloaded files will be saved. By default, this is the ./wandb directory.
2. You can delete files under an artifact. Here you have all the information about this.
   Please let me know if this would be useful for you.

Best,
Luis

[wztdream]

1. You can specify the argument `dir` in `wandb.init()`. This is an absolute path to a directory where metadata will be stored. When you call `download()` on an artifact, this is the directory where downloaded files will be saved. By default, this is the `./wandb` directory.

First let me clarity the setting on my side.

1. I have a disc(120G) mounted under root directory and wandb-local was installed there.
2. I have another disc(1T) mounted under /home directory and my project was there.

I did not setting dir in wandb.init() , so wandb used the default setting, and indeed there is an ./wandb directory under my project folder. But the problem here seems not related with dir argument, as shown in above picture, it seems wandb used up my space under the root /var/lib/docker/... folder, which is were wandb-local was installed, and this seems to me is consumed by the server not the client. So setting dir argument seems will not solve the problem.

2. You can delete files under an artifact. Here you have all the information about this.

I did not use artifact actually, these data are mainly just logged image data, so is it save I directly deleted these folders, for example delete seme-gan folder above if I did not want to keep this project anymore?

[exalate-issue-sync]

Luis Bergua commented:
Hi wztdream,

Sorry for my late response and thanks for the explanation! After exploring this, I think that using the environment variable WANDB_DIR should work for you (I have checked that, after setting it, all my files are stored there). Here is the link to our documentation about environment variables. Could you check if setting this variable to /home solves your issue? In order to delete folder, I think the easiest way could be using the Unix command rm -r seme-gan and so this folder and all its contents will be deleted. Please let me know if this would be useful, thanks!

Best,
Luis

[wztdream]

Hi Luis Bergua,
Thank you for looking at this. But setting WANDB_DIR did not work, this only changed the file path for the "client" side, not the server side, this is same by setting dir in wandb-init.

I did the experiment:

1. setting WANDB_DIR to /home
2. run experiments

the runs are stored on two locations:

1. client part: /home/wandb/my-runs
2. server part: /var/lib/docker/volumes/wandb/_data/minio/local-files/user-name/project-name/my-runs

you can see that setting WANDB_DIR only changed the file location for client side( where wandb sync files from), i.e. change from ./my-project/wandb to WANDA_DIR/wandb, but it did not change the server part,(where wandb sync files to) which is my problem here.

I think maybe I need to reinstall wandb-local under /home, and I suppose if the docker is under /home, the server local files will be under /home correspondingly, what do you think and how to install wandb-local under specific location?

[exalate-issue-sync]

Luis Bergua commented:
Hi wztdream,

Thanks for sharing your process! I have been able to investigate this in depth and, since you are using wandb-local, this is the expected behaviour. The reason of this is because the purpose of local is having a remote machine where docker is installed, so files will be always stored there and in your /home directory. In your case, since you only have one machine, you will always have duplicated files unless you manually delete the content of /var/lib/docker/volumes/wandb/_data/minio/local-files/user-name/project-name/my-runs. The behaviour that you would like to have is available when using our wandb public cloud so may I ask if using this would be useful for you? Please let me know if this would be useful for you. Thanks!

Best,
Luis

[wztdream]

Hi Luis Bergua:
Thank you again:)

Thanks for sharing your process! I have been able to investigate this in depth and, since you are using wandb-local, this is the expected behavior.

I understand current behavior is expected, as I asked previously if it is possible to install wandb-local under my /home directly? In that case both the remote and local files will be under my /home folder, there will be duplicate files, but it is OK as I have a large disc there.

The behavior that you would like to have is available when using our wandb public cloud so may I ask if using this would be useful for you?

Actually, I had used wandb public for a while, and I switch to wand-local due to the concern that I need log huge mounts of image data(mainly for debug purpose) and sending large mounts of images to remove server will be a bottleneck.

[vanpelt]

Hey @wztdream when using the default wandb server command we use a named docker volume (wandb) by default. The default location for these volumes is /var/lib/docker/volumes/wandb. There are some clever ways to alter the path of the named volume by manually creating it, but the simplest would likely be to just run the container manually and specify a bind mount. The command to run the container with a bind mount is:

docker run --rm -d -v /home/wandb-local:/vol -p 8080:8080 --name wandb-local wandb/local

If you want to migrate the data from your existing volume to /home, you should run the following commands before starting a container with the bind mount:

docker stop wandb-local
mkdir /home/wandb-local
cp -r /var/lib/docker/volumes/wandb/* /home/wandb-local

We release new versions of the server ~every 2 weeks so you'll want to upgrade the server by running:

docker pull wandb/local:latest
docker stop wandb-local
docker run --rm -d -v /home/wandb-local:/vol -p 8080:8080 --name wandb-local wandb/local

[vanpelt]

Oh, and to free up the disk space from the original docker volume after copying the files you can run docker volume rm wandb

[wztdream]

@vanpelt
Thank you for you detail explain:)

I follow bellow steps:

docker stop wandb-local
mkdir /home/wandb-local
cp -r /var/lib/docker/volumes/wandb/* /home/wandb-local

and removed the old volume

docker volume rm wandb

Then:

docker stop wandb-local
docker run --rm -d -v /home/wandb-local:/vol -p 8080:8080 --name wandb-local wandb/local

localhost:8080 lead me to a sign up page, after sign up again(using same email and username as my old account), I can log in but there is You need a license for this instance of W&B Local

But when I click the Get a free license it says: Your account already has a deployment

I have no idea what is going on here

[exalate-issue-sync]

WandB Internal User commented:
wztdream commented:
Hi Luis Bergua:
Thank you again:)

Thanks for sharing your process! I have been able to investigate this in depth and, since you are using wandb-local, this is the expected behavior.

I understand current behavior is expected, as I asked previously if it is possible to install wandb-local under my /home directly? In that case both the remote and local files will be under my /home folder, there will be duplicate files, but it is OK as I have a large disc there.

The behavior that you would like to have is available when using our wandb public cloud so may I ask if using this would be useful for you?

Actually, I had used wandb public for a while, and I switch to wand-local due to the concern that I need log huge mounts of image data(mainly for debug purpose) and sending large mounts of images to remove server will be a bottleneck.

[wztdream]

Update, after restart the computer I can see my previous license from Depolyer, So now the problem solved. Thank you all:)

[exalate-issue-sync]

WandB Internal User commented:
wztdream commented:
Update, after restart the computer I can see my previous license from Depolyer, So now the problem solved. Thank you all:)