Invocations refactoring #85

nerzhulart · 2016-12-20T16:48:01Z

This refactoring originally was started to fix multiple deadlocks in AsyncOperationManager.
After some investigation I've decided to rewrite all of this code to .net Task API to avoid syncronization issues caused by bad desing and wrong using of ManualResetEvents everywhere.
The main changes:

I'ive introduced a new EvaluatorExceptionThrownException that wraps target exception object and allows to inspect this object in watch pad when your eval has thrown an exception.
AsyncOperationManager.Invoke is explicitly synchronous now and returns OperationResult. Also it was synchronous before as well, but returned void and the call site had to obtain the result in other field and it was complicated.
I've added a lot of logging into this code

After this PR merged I'm going to add the second counterpart PR in monodevelop repo to supply these changes for CorDebug

nerzhulart · 2016-12-30T22:07:20Z

Hey, guys! Any updates on this? Tests on windows go fine.

slluis · 2017-01-10T12:00:44Z

Mono.Debugging/Mono.Debugging.Evaluation/AsyncOperationManager.cs

-						operationsToCancel.Remove (methodCall);
-						ST.Monitor.PulseAll (operationsToCancel);
-					}
-					if (wasAborted)


Why remove this check?

there is no Aborted property on AsyncOperation now, Cancellation is controlled by Task API

slluis · 2017-01-10T12:12:45Z

Mono.Debugging/Mono.Debugging.Evaluation/AsyncOperationManager.cs

+				if (!op.RawTask.Wait (ShortCancelTimeout)) {
+					try {
+						BusyStateChanged (this, new BusyStateEventArgs {IsBusy = true, Description = desc});
+						op.RawTask.Wait (LongCancelTimeout);


There is an important change in the logic around here. In the old implementation, an invocation abort attempt (InternalAbort) doesn't return until the invocation is aborted or the debug session ends, and keeps trying to abort meanwhile. In this implementation it just tries to cancel once, and returns after a "long" timeout. That's not right.

I've made a lot of experiments and found that many calls of Abort doesn't aborts evaluation if the first call didn't abort successfully. It's right for CorDebug and SBD as well. You may try to run eval of method with loop
void RunLoop() { int i = 0; while (true) { Console.WriteLine(i++); } }
And try to abort it many times, you will fail with timeout always,

Maybe in some cases will always timeout, maybe in others will not. In any case, we can't leave the "Busy" state until the debugger really stopped executing the method.

There is a big issue with your logic (that was used in MD now): if we try to abort some calculation that cannot be aborted gracefully (e.g. endless loop) the debugger may NOT to fire any signal to us - finished or aborted - and in this case we can wait forever. I've already faced with this that's why I've rewritten this to use timeouts. It's absolutely right for CorDebug, I'm not sure about SDB. Also ICorDebugEval has IsActive property, but after any times of Abort() on endless loop or some other long calculation it says us 'true'. I've read sources of .net core (because ICD impl in .net core is pretty the same as full framework) and they just wait 500 ms after Abort call and only after that your the ICD goes to consistent state. So waiting for IsActive == false may lead to forever awaiting.

if we try to abort some calculation that cannot be aborted gracefully (e.g. endless loop) the debugger may NOT to fire any signal to us - finished or aborted - and in this case we can wait forever.

That's is correct, and it is the whole point of having the debugger Busy mode. In SDB, if an invocation is in progress, we can't do anything else with the debugger. So, if the invocation can't be aborted, the only thing we can do is wait forever until the debuggee is able to end the invocation, or until the user stops the debugging session. When we enter the Busy mode, a dialog is shown to the user asking what to do (keep waiting for the call to end or stop the debugging session).

If this is not a problem for CorDebug, then you can add a flag to avoid waiting forever, but we need to keep the current behavior for SDB.

I've restored the inifinite waiting. Now it should work as before. Also I've improved exception handling in SDB invocations. See commit 102fa7a

slluis · 2017-01-10T14:00:58Z

Mono.Debugging/Mono.Debugging.Evaluation/IAsyncOperation.cs

@@ -0,0 +1,22 @@
+namespace Mono.Debugging.Evaluation
+{
+	public interface IAsyncOperation


This seems to be unused.

That's right! Thanks

Still unused...

slluis · 2017-01-10T14:07:05Z

Mono.Debugging/Mono.Debugging.Evaluation/AsyncOperationBase.cs

+
+	public static class OperationResultEx
+	{
+		public static OperationResult<TValue> ThrowIfException<TValue> (this OperationResult<TValue> result, EvaluationContext ctx)


Why not just make it a member of OperationResult?

I guess this approach makes the code more simple. OperationResult is just a data class without any internal logic

That looks unnecessary to me. In any case, up to you.

In this case ThrowIfException is more helper method rather than required to be in the class. It's IMHO

slluis · 2017-01-10T14:22:51Z

Mono.Debugging.Soft/SoftDebuggerAdaptor.cs

-			} catch (InvocationException ex) {
-				if (!Aborting && ex.Exception != null) {
-					string ename = ctx.Adapter.GetValueTypeName (ctx, ex.Exception);
-					var vref = ctx.Adapter.GetMember (ctx, null, ex.Exception, "Message");


Why is this logic gone?

I've replaced simple Exception message to full exception object. So now you can walk through it, but before you can see only exception message

How is that object shown to the user?

It's an ObjectValue with its children that has IsException flag. It's shown in watchpad as ordinary object

slluis · 2017-01-10T14:29:05Z

Mono.Debugging/Mono.Debugging.Evaluation/AsyncOperationManager.cs

-			Disposing = true;
-			lock (operationsToCancel) {
-				foreach (AsyncOperation op in operationsToCancel) {
-					op.InternalShutdown ();


All the Shutdown logic is gone, why?

Shutdown had nearly the same logic as the Cancel. I've unified them into Cancel.

Nearly the same logic, but not the same. Let's say an invocation is made. While the caller is waiting for the invocation to end, the manager is disposed, so the call is canceled. But if the call can't really be cancelled, the caller will wait forever.

Why do you think so? After Dispose is called the Invoke method on another thread will throw EvaluationAbortedException

If the .Abort() call on the invocation fails, or can't be completed (it can happen, that's why there is all that abort retry and debugger busy mode infrastructure), then the invocation will never end, and EvaluationAbortedException will never be thrown.

Fixed. See below

slluis · 2017-01-10T14:32:08Z

Mono.Debugging.Soft/SoftDebuggerAdaptor.cs

-			}
-		}
-
-		string GetInfo ()


More logic that has been removed for no apparent good reason.

If you mean logic of getting exception message so look at line 2230. SoftOperationResult wraps exception object to show it like ordinary object for user

I mean GetInfo(), which is used to log the method and type name when there is an invocation error.

Seems that GetInfo used only for logging. Isnt't it enough to use Description for this? Now I log all this states using this property

I'm not sure if it is enough or not, it needs to be checked. I just don't want code to be removed if there isn't a good reason.

The main reasons to remove are simplifying and avoiding duplications.

Let's bring it back then.

slluis · 2017-01-10T14:35:40Z

Mono.Debugging/Mono.Debugging.Evaluation/AsyncEvaluationTracker.cs

@@ -41,7 +41,7 @@ namespace Mono.Debugging.Evaluation
 	/// will then be made asynchronous and the Run method will immediately return an ObjectValue
 	/// with the Evaluating state.
 	/// </summary>
-	public class AsyncEvaluationTracker: RemoteFrameObject, IObjectValueUpdater, IDisposable


Why this change?

There were no usages of AsyncEvaluationTracker by this interface.

If you used by some dynamic way I can revert this change

The important bit is that RemoteFrameObject is a MarshalByRefObject, and AsyncEvaluationTracker needs to be remotable.

Can you please explain how is this working? where to look at the remoting?

The library is designed so that the debugging engine can run in a remote process, which communicates with the client using remoting. That's really up to the debugger engine implementation. In such scenario, the AsyncEvaluationTracker could be executing in the remote process and generating ObjectValue instances that are serialized into the client process and which may need to hold a reference to that remote object (

debugger-libs/Mono.Debugging/Mono.Debugging.Evaluation/AsyncEvaluationTracker.cs

Line 85 in ccadee5

return ObjectValue.CreateEvaluating (this, new ObjectPath (id, name), flags);

). This model was necessary with the old Mono debugger, but it is not with the soft debugger. However, the infrastructure is still there, in case a specific debugger implementation needs it.

Please restore the RemoteFrameObject base class.

slluis · 2017-01-10T14:36:26Z

Mono.Debugging/Mono.Debugging.Evaluation/AsyncOperationBase.cs

+	{
+		Task RawTask { get; }
+		string Description { get; }
+		void AfterCancelled (int elapsedAfterCancelMs);


What is AfterCancelled for? it doesn't seem to have any implementation.

It's used in CorDebug (monodevelop repo). It's done to ensure that we awaited enough amount of time after Abort() call. Otherwise we can get exceptions in CorDebug.

nerzhulart · 2017-01-10T15:55:23Z

I have to say that this code works in Rider since November and the evaluation experience was greatly improved as well as performance. Before this we faced with deadlocks many times per day. Also we didn't notice any evaluation issues after we merged this change into our production code.,

slluis · 2017-01-10T17:29:45Z

Even if it works for Rider, I need to make sure that it doesn't cause any regression in XS. It is an important change in a critical piece of code, so we need to be very careful about it.

nerzhulart · 2017-01-10T17:40:13Z

Yes, of course! I just wanted to say that this code was not written a day ago and pushed without any testing

nerzhulart · 2017-01-10T17:41:38Z

I've removed IAsyncOperation.cs and restore AsyncOperationTracker to be RemoteFrameObject.

nerzhulart · 2017-01-18T13:55:00Z

Do you have any new suggestions for this PR?

nerzhulart · 2017-01-20T16:50:58Z

@slluis Hi! Any news on this?

slluis · 2017-01-20T16:53:04Z

Sorry, it has been a busy week. I'll review it asap.

nerzhulart · 2017-01-20T16:56:54Z

Ok! Thanks. Will wait.

nerzhulart · 2017-01-23T19:45:38Z

Added a couple of commits. Now the Abort() is being called in loop as it was before.

nerzhulart · 2017-03-16T12:53:27Z

Hi! Any news on this?
I have new fixes for value modifications, but they are dependent on this

…that wraps target process exception that occurred during runtime call. This allows to inspect exception objects in watch window if eval thrown an exception.

…lows to properly show an exception occurred during property evaluation.

Ignore eval exceptions occured during ToString() calls in type presentation

Better handling of exeptions in Soft invocations

…dateSessionState() is invoked (in finally) (cherry picked from commit 3befe19)

… repo)

nerzhulart · 2017-03-16T19:06:39Z

Added commits for Win32 debugger (moved from MD)

… for many times. Maybe this may help to avoid hanging evaluations. (Moved from MD repo)

slluis · 2017-03-17T11:20:20Z

Mono.Debugging/Mono.Debugging.Evaluation/AsyncEvaluationTracker.cs

@@ -41,7 +41,7 @@ namespace Mono.Debugging.Evaluation
 	/// will then be made asynchronous and the Run method will immediately return an ObjectValue
 	/// with the Evaluating state.
 	/// </summary>
-	public class AsyncEvaluationTracker: RemoteFrameObject, IObjectValueUpdater, IDisposable


Please restore the RemoteFrameObject base class.

slluis · 2017-03-17T11:37:00Z

Mono.Debugging/Mono.Debugging.Evaluation/AsyncOperationManager.cs

+			if (!op.RawTask.Wait (ShortCancelTimeout)) {
+				try {
+					ChangeBusyState (true, desc);
+					while (true) {


This loop should gracefully exit if AsyncOperationManager is disposed.

Logic for switching to busy state is now different. Please use old logic.

This loop should gracefully exit if AsyncOperationManager is disposed.

Done

Logic for switching to busy state is now different. Please use old logic.

Can you please explain more detailed?
For me the current logic is quite the same:

StartInvoke

Wait normally for given timeout

If timed out -> Abort()

Wait for Short timeout

If timed out -> Enter busy

Repeat from 4

If awaited -> Exit busy

Seems something like that was before and implemented on magic counters. And the code was very confusing

slluis · 2017-03-17T11:42:55Z

Mono.Debugging/Mono.Debugging.Evaluation/AsyncOperationManager.cs

-			lock (operationsToCancel) {
-				foreach (AsyncOperation op in operationsToCancel) {
-					op.InternalShutdown ();
+			var desc = op.Description;


It doesn't take into account that the method may already be being aborted, like the old implementation did.

slluis · 2017-03-17T12:07:34Z

Mono.Debugging/Mono.Debugging.Evaluation/IAsyncOperation.cs

@@ -0,0 +1,22 @@
+namespace Mono.Debugging.Evaluation
+{
+	public interface IAsyncOperation


Still unused...

nerzhulart · 2017-03-17T15:10:24Z

It doesn't take into account that the method may already be being aborted, like the old implementation did.

Fixed

slluis self-assigned this Dec 20, 2016

slluis self-requested a review December 20, 2016 18:27

nerzhulart mentioned this pull request Dec 21, 2016

[CorDebug] Invocations rewritten to .net Task API mono/monodevelop#1726

Closed

slluis requested changes Jan 10, 2017

View reviewed changes

nerzhulart force-pushed the invocationsRefactoring branch 2 times, most recently from 0657baa to 9809e72 Compare January 16, 2017 17:38

nerzhulart and others added 11 commits March 16, 2017 20:15

Special evaluator exception type (EvaluatorExceptionThrownException) …

90a1f51

…that wraps target process exception that occurred during runtime call. This allows to inspect exception objects in watch window if eval thrown an exception.

EvaluatorExceptionThrownException handling in ValueReference. This al…

b72dfaf

…lows to properly show an exception occurred during property evaluation.

Evaluation exception wrapping in CreateObjectValue.

cac578c

Ignore eval exceptions occured during ToString() calls in type presentation

Separate lock object. Get rid of useful Monitor.Pulse()

54a8ed0

Async operations rewritten to Tasks

f3e1fee

Proper logging: before the exception body was lost, now it's logged.

ad8187f

Invocation is awaited infinitely as it was before.

5be3b5b

Better handling of exeptions in Soft invocations

Restore GetInfo() and detailed logging on invocation

1d4a9c4

More proper evaluation aborting.

d8ef7b8

Get rid of OperationData

3226be3

Move checking for cancelled token into try-catch to guarantee that Up…

e9ee3d0

…dateSessionState() is invoked (in finally) (cherry picked from commit 3befe19)

nerzhulart force-pushed the invocationsRefactoring branch from c574f03 to e9ee3d0 Compare March 16, 2017 17:16

CorDebug Invocations rewritten to .Net Task API (commit moved from MD…

5db72cd

… repo)

nerzhulart added 2 commits March 16, 2017 21:56

CorDebug Checking for aborted state in CorMethodCall

0aec667

More proper evaluation aborting

e1f7852

nerzhulart mentioned this pull request Mar 16, 2017

Set value refactoring #105

Open

nerzhulart added 2 commits March 16, 2017 23:28

Trying to continue all threads if eval Abort() and RudeAbort() failed…

ae0eb0f

… for many times. Maybe this may help to avoid hanging evaluations. (Moved from MD repo)

Handle exceptions in AbortImpl() (Moved from MD repo)

29b6a90

slluis requested changes Mar 17, 2017

View reviewed changes

nerzhulart added 5 commits March 17, 2017 16:13

Restore RemoteFrameObject for AsyncEvaluationTracker

c594ffc

Remove unused file IAsyncOperation.cs

45b3c96

Break loop if disposed.

3f9b165

Line endings fix

c6f1148

Don't call Abort() if operation is alredy in aborting state

e2cf427

Base automatically changed from master to main March 4, 2021 12:37

Invocations refactoring #85

Are you sure you want to change the base?

Invocations refactoring #85

Conversation

nerzhulart commented Dec 20, 2016 • edited Loading

nerzhulart commented Dec 30, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nerzhulart Jan 17, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nerzhulart Jan 10, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nerzhulart commented Jan 10, 2017 • edited Loading

slluis commented Jan 10, 2017

nerzhulart commented Jan 10, 2017

nerzhulart commented Jan 10, 2017

nerzhulart commented Jan 18, 2017 • edited Loading

nerzhulart commented Jan 20, 2017

slluis commented Jan 20, 2017

nerzhulart commented Jan 20, 2017

nerzhulart commented Jan 23, 2017

nerzhulart commented Mar 16, 2017

nerzhulart commented Mar 16, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nerzhulart Mar 17, 2017 • edited Loading

Choose a reason for hiding this comment

nerzhulart Mar 17, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nerzhulart commented Mar 17, 2017

nerzhulart commented Dec 20, 2016 •

edited

Loading

nerzhulart Jan 17, 2017 •

edited

Loading

nerzhulart Jan 10, 2017 •

edited

Loading

nerzhulart commented Jan 10, 2017 •

edited

Loading

nerzhulart commented Jan 18, 2017 •

edited

Loading

nerzhulart Mar 17, 2017 •

edited

Loading

nerzhulart Mar 17, 2017 •

edited

Loading