"flutter.setSubscriptions" call never forgets a file (aka will ask about an ever growing number of files)

[jensjoha]

In IntelliJ, opening a project with flutter in the environment in pubspec.yaml, e.g. something like

name: issue_whatnot
publish_to: "none"

environment:
  flutter: ">=3.29.0 <4.0.0"
  sdk: ">=3.7.0 <4.0.0"

opening a file sends a flutter.setSubscriptions request looking something like this:

{
  "id": "42",
  "method": "flutter.setSubscriptions",
  "params": {
    "subscriptions": {
      "OUTLINE": [
        "file:///path/to/file_1.dart"
      ]
    }
  }
}

Say we close the file and open another, it now sends something like this:

{
  "id": "42",
  "method": "flutter.setSubscriptions",
  "params": {
    "subscriptions": {
      "OUTLINE": [
        "file:///path/to/file_1.dart",
        "file:///path/to/file_2.dart"
      ]
    }
  }
}

closing that and opening a third, closing that and opening a forth etc a bunch of times and you'll end up with something like

{
  "id": "42",
  "method": "flutter.setSubscriptions",
  "params": {
    "subscriptions": {
      "OUTLINE": [
        "file:///path/to/file_1.dart",
        "file:///path/to/file_2.dart",
        ...
        "file:///path/to/file_n.dart",
      ]
    }
  }
}

at which point the analyzer will start to become slow:

5 ms (+ 22045 ms)	completion.getSuggestions2
23121 ms (+ 100 ms)	edit.getAssists

Notice that if I open a file I've already opened once it doesn't send the request, I believe because of this:

flutter-intellij/flutter-idea/src/io/flutter/dart/FlutterDartAnalysisServer.java

Lines 140 to 150 in 38e78f0

	private void addSubscription(@NotNull final String service, @NotNull final String filePath) {
	if(!isServerConnected()) {
	return;
	}
	final List<String> files = subscriptions.computeIfAbsent(service, k -> new ArrayList<>());
	final String filePathOrUri = getAnalysisService().getLocalFileUri(filePath);
	if (!files.contains(filePathOrUri)) {
	files.add(filePathOrUri);
	sendSubscriptions();
	}
	}

.

(i.e. sendSubscriptions is only called i it actually adds the path)

I'm guessing something goes wrong in these lines:

flutter-intellij/flutter-idea/src/io/flutter/editor/ActiveEditorsOutlineService.java

Lines 125 to 152 in 38e78f0

	final List<String> obsoletePaths = new ArrayList<>();
	synchronized (outlineListeners) {
	for (final String path : outlineListeners.keySet()) {
	if (!newPaths.contains(path)) {
	obsoletePaths.add(path);
	}
	}
	for (final String path : obsoletePaths) {
	final String filePathOrUri = getAnalysisServer().getAnalysisService().getLocalFileUri(path);
	final FlutterOutlineListener listener = outlineListeners.remove(filePathOrUri);
	if (listener != null) {
	getAnalysisServer().removeOutlineListener(FileUtil.toSystemDependentName(path), listener);
	}
	}
	// Register new outline listeners.
	for (final String path : newPaths) {
	final String filePathOrUri = getAnalysisServer().getAnalysisService().getLocalFileUri(path);
	if (outlineListeners.containsKey(filePathOrUri)) {
	continue;
	}
	final FlutterOutlineListener listener = new OutlineListener(filePathOrUri);
	outlineListeners.put(filePathOrUri, listener);
	getAnalysisServer().addOutlineListener(FileUtil.toSystemDependentName(path), listener);
	}
	}

Without having looked into it it seems suspect that it takes a key from outlineListeners (final String path : outlineListeners.keySet()) and then does something to it before trying to remove it again (final String filePathOrUri = getAnalysisServer().getAnalysisService().getLocalFileUri(path);, final FlutterOutlineListener listener = outlineListeners.remove(filePathOrUri);), but I'm guessing.

[jensjoha]

/cc @jwren

[bwilkerson]

@jwren Is it true that IntelliJ should never need to get flutter outline information for more than one file at a time?

[jwren]

I am not familiar enough with the Flutter setSubscriptions protocol (https://htmlpreview.github.io/?https://github.com/dart-lang/sdk/blob/main/pkg/analysis_server/doc/api.html#request_flutter.setSubscriptions) to be able to come in and determine if the right things are being done here from the IJ side without lots of manual testing and digging into the protocol. It looks like the feature was largely added by Konstantin and Jacob. Thanks for starting to look into this issue to get better performance.

[pq]

Thanks for this great report @jensjoha!

Taking a look at this now.

[alexander-doroshko]

The the fileOutlineListeners map grows indefinitely because

• addOutlineListener uses getLocalFileUri(filePath) as a map key
• removeOutlineListener assumes that a plain filePath is used as a key

If all code related to the "flutter.setSubscriptions" call is not needed anymore, this map may be completely removed along with a bunch of other obsolete code

[pq]

If all code related to the "flutter.setSubscriptions" call is not needed anymore, this map may be completely removed along with a bunch of other obsolete code

Right! I'm trying to establish exactly this. It looks like CommonTestConfigUtils uses this service (see, for example, getTestsFromOutline) but I'm not sure if this is being used (or works as intended). Will dig a little more.

Thanks!

[pq]

On the topic of managing the file map, it seems like the issue is introduced (or at least manifest) here:

flutter-intellij/flutter-idea/src/io/flutter/editor/ActiveEditorsOutlineService.java

Lines 134 to 136 in 38e78f0

	for (final String path : obsoletePaths) {
	final String filePathOrUri = getAnalysisServer().getAnalysisService().getLocalFileUri(path);
	final FlutterOutlineListener listener = outlineListeners.remove(filePathOrUri);

The rub is that the path in obsoletePaths is already a file path (for example file:///Users/pq/StudioProjects/fluttery/lib/main.dart). When such a path is passed to getLocalFileUri it gets turned into something like file://file:///Users/pq/StudioProjects/fluttery/lib/main.dart, which is incorrect and so the listener is never removed.

@alexander-doroshko: it seems like getLocalFileUri could be improved in the Dart plugin DartAnalysisServerService to do the right thing in case a valid file path is passed into it?

In the meantime, I think using the obsolete path as-is addresses the problem:

for (final String path : obsoletePaths) {
  final FlutterOutlineListener listener = outlineListeners.remove(path);
  ...

[pq]

As for the use of the flutter outline service in general, even though we are no longer displaying a flutter outline view, it does seem like we're using the the service to figure out where to contribute test line markers.

See:

flutter-intellij/flutter-idea/src/io/flutter/run/common/TestLineMarkerContributor.java

Line 63 in a801049

final TestType testCall = testConfigUtils.asTestCall(dartCallExpression);

which calls here:

flutter-intellij/flutter-idea/src/io/flutter/run/common/CommonTestConfigUtils.java

Line 59 in 9a6aff3

public TestType asTestCall(@NotNull PsiElement element) {

I've confirmed that this code is alive and does contribute to markers showing up

but I'm not sure it's entirely working as intended. Either way, in the absences of a flutter outline view, it's likely this isn't the best way to calculate test line markers regardless but I'm missing a lot of context and need to do some more digging but maybe we should create another issue for that and just focus on getting the subscribed file list to stop growing infinitely.

[alexander-doroshko]

The rub is that the path in obsoletePaths is already a file path (for example file:///Users/pq/StudioProjects/fluttery/lib/main.dart)

You mean 'already a URI' :)

@alexander-doroshko: it seems like getLocalFileUri could be improved in the Dart plugin DartAnalysisServerService to do the right thing in case a valid file path is passed into it?

You mean 'to do the right thing in case a valid file URI is passed into it' :)

I'd say passing URI to DartAnalysisServerService.getLocalFileUri() is an incorrect API usage. I think getLocalFileUri() shouldn't try to fix caller mistake. I'd rather make it throw IllegalArgumentException. If it had done that, we'd have caught the problem a year ago.