[Tracing] Management Grain call Filtering? Internal management grain messages are noisy

[rwkarg]

I've turned on tracing for "Microsoft.Orleans" and it's great for my application's grain calls, but there's a bunch of other management grain calls. For example, IDeploymentLoadPublisher/UpdateRuntimeStatistics appears to get called every second. There are also IMembershipTable, IRemoteGrainDirectory, and IReminderTableGrain calls included.

Perhaps there could be separate sources for the management related activities and "the rest".
Ex.
"Microsoft.Orleans.Management" or "Microsoft.Orleans.Internal" for the internal/management related calls.
"Microsoft.Orleans" for the remainder of the grain calls.

It looks like ISystemTarget may be a marker interface that could do the differentiation of sources.
Perhaps ActivityPropagationGrainCallFilter can encapsulate that logic in a pair of Start/CreateActivity protected members that look at the call context and provide an Activity from the appropriate ActivitySource.

orleans/src/Orleans.Core/Diagnostics/ActivityPropagationGrainCallFilter.cs

Line 11 in 4702095

internal abstract class ActivityPropagationGrainCallFilter

[rwkarg]

Adding something like this to ActivityPropagatorGrainCallFilter works, but I'm not sure if namespace name is the best way to filter out the system grains. It seems to be accurate but unclear on what ongoing maintenance would be required (how often are new assemblies, like Orleans.Reminder, really added?).

I also tried keeping the current Microsoft.Orleans ActivitySourceName and adding Microsoft.Orleans.Internal but that ended up with OpenTelemetry listening to both ActivitySources when I did .AddSource("Microsoft.Orleans"), even though there wasn't any wildcard specified. Still need to dig in to why that is happening.

        public const string ActivitySourceName = "Microsoft.Orleans.User";
        public const string InternalActivitySourceName = "Microsoft.Orleans.Internal";

        private static readonly ActivitySource InternalSource = new(InternalActivitySourceName);
        private static readonly ActivitySource Source = new(ActivitySourceName);

        protected static Activity? StartClientActivity(IOutgoingGrainCallContext context)
        {
            return context.Request.InterfaceType.Assembly.GetName().Name switch
            {
                "Orleans.Core" => InternalSource.StartActivity(context.Request.ActivityName, ActivityKind.Client),
                "Orleans.Runtime" => InternalSource.StartActivity(context.Request.ActivityName, ActivityKind.Client),
                "Orleans.Reminders" => InternalSource.StartActivity(context.Request.ActivityName, ActivityKind.Client),
                _ => Source.StartActivity(context.Request.ActivityName, ActivityKind.Client)
            };
        }

        protected static Activity? CreateServerActivity(IIncomingGrainCallContext context, string traceParent)
        {
            return context.Request.InterfaceType.Assembly.GetName().Name switch
            {
                "Orleans.Core" => InternalSource.CreateActivity(context.Request.ActivityName, ActivityKind.Server, traceParent),
                "Orleans.Runtime" => InternalSource.CreateActivity(context.Request.ActivityName, ActivityKind.Server, traceParent),
                "Orleans.Reminders" => InternalSource.CreateActivity(context.Request.ActivityName, ActivityKind.Server, traceParent),
                _ => Source.CreateActivity(context.Request.ActivityName, ActivityKind.Server, traceParent)
            };
        }