Provide a more memory-efficient NIO-based file system watcher in DevTools

[dsu]

File Watcher thread takes about of 98% of entire application memory.Memory consumption is still growing after application start.
It is also impossible to exclude log files from third party libraries that need to write to files under resources directory.
It would be nice to be able to exclude files by location or by file extension. Something similar to restart.include/restart.exclude properties.

[wilkinsona]

Sorry, I don't understand your description of the problem.

File Watcher thread takes about of 98% of entire application memory

Roughly speaking, a thread consumes a fixed amount of memory that's determined by the JVM's configured thread stack size. A thread itself shouldn't consume 98% of the available memory. Can you please clarify what you mean here?

It is also impossible to exclude log files from third party libraries that need to write to files under resources directory.

Why would a library need to write its output to a location alongside your application's resources?

Before we can assess the need for an enhancement, we'll need to have a better understanding of the problem that you're facing and why it's occurring. To help with that, can you please provide a small sample that we can clone or unzip that reproduces the problem?

[dsu]

It just takes more memory than it should - heapdump. I would like to reduce the number of files that are watched.
Probably a few thousands of files under resources directory can help to reproduce the problem.

Why would a library need to write its output to a location alongside your application's resources?

All file paths rely on some configuration property. Library stores all the files and logs under a common root directory. Some of this files need to be accessible as a resources. This is just an example. What does it matter?

[wilkinsona]

Thanks.

What does it matter?

It matters because we can't make a change or enhancement without first fully understanding the problem that we're trying to solve. On that note, as I said above, can you please provide a small sample that we can clone or unzip that reproduces the problem?

[dsu]

Imagine an app that needs to generate thousands of images that needs to be accessible by a reverse proxy..

package com;

import java.io.BufferedWriter;
import java.io.File;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.nio.file.Files;
import java.nio.file.Path;

import javax.annotation.PostConstruct;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.stereotype.Component;

@SpringBootApplication
public class WatcherKillApp {

	public static void main(String[] args) {
		System.out.println("WatcherKillApp");
		SpringApplication.run(WatcherKillApp.class, args);
	}

	@Component
	public static class FileCloner {

		@PostConstruct
		public void cloneFiles() throws IOException {
			String root = getClass().getResource(".").getFile() + File.separator + ".." + File.separator + "static";
			for (int i = 0; i <= 500000; i++) {
				Path fullPath = new File(new File(root), i + ".jpg").toPath();
				// make sure file exists
				Files.createDirectories(fullPath.getParent());

				try (BufferedWriter bw = Files.newBufferedWriter(fullPath)) {
					bw.write("test");
				}

				System.out.println("new file " + fullPath.toFile().getAbsolutePath());
			}

		}
	}

}

[philwebb]

I think I see what you're saying. You have an application that has a very large number of static resources. The FileSystemWatcher class stores a map containing FolderSnapshot objects. A FolderSnapshot includes a FileSnapshot for each monitored file (so we know the date/length etc). Since you have so many files, the memory consumption is quite large.