Evaluating Debugged-Process Memory in a Visual Studio Extension

I’ve recently been working on a Visual Studio extension that acts a bit like the Visual Studio memory window, but visualises the memory as a texture. Like the memory window, there’s a box for the user to type in an address expression. The plug-in then needs to evaluate the expression in the context of the debugger’s current thread and stack frame, and then retrieve the contents of the debuggee process’s memory space at that address.

This is a relatively common operation for the debugger — it’s done by the memory window, as well as the watch windows, of course. I therefore assumed it would be fairly straightforward. I was very wrong. After much hair-pulling, however, I was finally able to get it to work, so I thought I’d document it here in case anyone else needs to do something similar.

It’s worth noting that the plug-in I’m writing is written in C#, so all the code snippets will be C# too. The VSX API is COM, though, so the steps should be basically the same whatever the language. And I’m targeting VS 2012, so all of this may or may not work with other versions. The API doesn’t look to have changed between 2010 and 2012, so there’s a good chance it will work there, but I haven’t tried it.

Evaluating Expressions

The Visual Studio automation API provides a way to evaluate expressions in the current stack frame (EnvDTE.DTE.Debugger.GetExpression), which is of course exactly what we want. However, the returned type doesn’t allow direct access to the debuggee’s memory, just the evaluated value as a string.

So, we need to use the full ‘package’ VSX API. The main interface that represents a property in the debugger in the VSX API is IDebugProperty2. Once you have one of these, you can fairly easily use it to get the contents of the debuggee’s memory. To get an IDebugProperty2, you can call IDebugExpression2.EvaluateSync. You can get an IDebugExpression2 by calling ParseText on an IDebugExpressionContext2 interface. And you can get one of those by calling GetExpressionContext on an IDebugStackFrame2. So the challenge becomes to get an IDebugStackFrame2 presenting the ‘current’ stack frame — i.e. the one that is used to evaluate watch expressions.

Registering for Debug Events

Despite much searching, I couldn’t find a way in the VSX API to directly get the current stack frame (or the current thread, or even the current process, for that matter). However, there is a way to register for debug events, which enables the caller to receive a callback when the state of the debugger changes, including, for example, hitting a breakpoint. And the callback takes as parameters various interfaces representing the new state of the debugger.

This is achieved by implementing the IDebugEventCallback2 interface, and then passing it to IVsDebugger.AdviseDebugEventCallback. This method takes an object (IUnknown in the underlying COM gubbins), but it will magically be cast to an IDebugEventCallback2 underneath. Needless to say, this took me some time to realise! Anyway, here’s some code that does this.

// Get the Debugger service.
IVsDebugger debugService = Microsoft.VisualStudio.Shell.Package.GetGlobalService(typeof(SVsShellDebugger)) as IVsDebugger;
if (debugService != null)
{
    // Register for debug events.
    // Assumes the current class implements IDebugEventCallback2.
    debugService.AdviseDebugEventCallback(this);
}

The debugger indicates an event by calling the Event method of the IDebugEventCallback2 interface. The parameters include the GUID of the event that has fired, along with a load of interfaces representing the state of the debugger. I needed an event that gets fired when you hit a breakpoint (or the debugger breaks for some other reason, like an exception), and also when the user changes the stack frame or thread. Fortunately there is one event that covers all these scenarios. Its GUID is {ce6f92d3-4222-4b1e-830d-3ecff112bf22}, but I have no idea what event it represents — I could find no mention of it in the docs or on Google. But it gets fired at just the right time, so I’m not going to lose much sleep over that!

Getting the Current Stack Frame

Unfortunately, the parameters to IDebugEventCallback2.Event do not include the current stack frame (that would be too easy!) It does give you the current thread (as IDebugThread2), though. With this, you can enumerate all the stack frames for that thread using the EnumFrameInfo method. This still gives you no information about what the current stack frame is, however.

Fortunately, we can get this information from the automation API. EnvDTE.DTE.Debugger.CurrentStackFrame returns a StackFrame object. This in and of itself isn’t much use to us, but the object is actually an instance of StackFrame2 (obviously!) So, casting to this type, we can get its Depth property, which indicates how deep in the stack the current frame is. It’s worth noting that the top frame has a depth of 1, not 0. Here’s some code to do all that.

// Get the automation API DTE object.
EnvDTE.DTE DTE = Microsoft.VisualStudio.Shell.Package.GetGlobalService(typeof(SDTE)) as EnvDTE.DTE;
if (DTE == null)
{
    Debug.WriteLine("Could not get DTE service.");
    return;
}
if (DTE.Debugger.CurrentStackFrame == null)
{
    // No current stack frame.
    return;
}

// Cast to StackFrame2, as it contains the Depth property that we need.
StackFrame2 currentFrame2 = DTE.Debugger.CurrentStackFrame as StackFrame2;
if (currentFrame2 == null)
{
    Debug.WriteLine("CurrentStackFrame is not a StackFrame2.");
    return;
}

// Depth property is 1-based.
uint currentFrameDepth = currentFrame2.Depth - 1;

So, we now have a way of enumerating the stack frames of the current thread, and we know the depth the current frame, so it’s simple to put these two together and get the current stack frame. Here’s a bit more code that does just that.

// Get frame info enum interface.
IEnumDebugFrameInfo2 enumDebugFrameInfo2;
if (VSConstants.S_OK != thread.EnumFrameInfo(enum_FRAMEINFO_FLAGS.FIF_FRAME, 0, out enumDebugFrameInfo2))
{
    Debug.WriteLine("Could not enumerate stack frames.");
    return;
}

// Skip frames above the current one.
enumDebugFrameInfo2.Reset();
if (VSConstants.S_OK != enumDebugFrameInfo2.Skip(currentFrameDepth))
{
    Debug.WriteLine("Current stack frame could not be enumerated.");
    return;
}

// Get the current frame.
FRAMEINFO[] frameInfo = new FRAMEINFO[1];
uint fetched = 0;
int hr = enumDebugFrameInfo2.Next(1, frameInfo, ref fetched);

if (hr != VSConstants.S_OK || fetched != 1)
{
    Debug.WriteLine("Failed to get current stack frame info.");
    return;
}

IDebugStackFrame2 stackFrame = frameInfo[0].m_pFrame;
if (stackFrame == null)
{
    Debug.WriteLine("Current stack frame is null.");
    return;
}

Accessing Debuggee Memory

Now that we have the sacred IDebugStackFrame2, we can use it to evaluate the address expression and access the debuggee memory, as described above. For completeness, here’s the code that does that.

// Get a context for evaluating expressions.
IDebugExpressionContext2 expressionContext;
if (VSConstants.S_OK != stackFrame.GetExpressionContext(out expressionContext))
{
    Debug.WriteLine("Failed to get expression context.");
    return;
}

// Parse the expression string.
IDebugExpression2 expression;
string error;
uint errorCharIndex;
if (VSConstants.S_OK != expressionContext.ParseText(addressExpressionString,
    enum_PARSEFLAGS.PARSE_EXPRESSION, 10, out expression, out error, out errorCharIndex))
{
    Debug.WriteLine("Failed to parse expression.");
    return;
}

// Evaluate the parsed expression.
IDebugProperty2 debugProperty = null;
if (VSConstants.S_OK != expression.EvaluateSync(enum_EVALFLAGS.EVAL_NOSIDEEFFECTS,
    unchecked((uint)Timeout.Infinite), null, out debugProperty))
{
    Debug.WriteLine("Failed to evaluate expression.");
    return;
}

// Get memory context for the property.
IDebugMemoryContext2 memoryContext;
if (VSConstants.S_OK != DebugProperty.GetMemoryContext(out memoryContext))
{
    // In practice, this is where it seems to fail if you enter an invalid expression.
    Debug.WriteLine("Failed to get memory context.");
    return;
}

// Get memory bytes interface.
IDebugMemoryBytes2 memoryBytes;
if (VSConstants.S_OK != DebugProperty.GetMemoryBytes(out memoryBytes))
{
    Debug.WriteLine("Failed to get memory bytes interface.");
    return null;
}

// The number of bytes to read.
uint dataSize = GetMemorySizeToRead();

// Allocate space for the result.
byte[] data = new byte[dataSize];
uint writtenBytes = 0;

// Read data from the debuggee.
uint unreadable = 0;
int hr = memoryBytes.ReadAt(memoryContext, dataSize, data, out writtenBytes, ref unreadable);

if (hr != VSConstants.S_OK)
{
    // Read failed.
}
else if (writtenBytes < dataSize)
{
    // Read partially succeeded.
}
else
{
    // Read successful.
}

Other Links

Advertisements

About Simon

I'm a lead developer at Geomerics working on the integration of Enlighten into various game engines. This blog is for random musing about real-time rendering and games development.
This entry was posted in Uncategorized and tagged , , . Bookmark the permalink.

4 Responses to Evaluating Debugged-Process Memory in a Visual Studio Extension

  1. frank koch says:

    Thanks for sharing this information, it was very useful. I have found a way to get the CurrentThread without listening to an event:
    IDebuggerInternal11 debuggerServiceInternal = (IDebuggerInternal11)debugService;
    IDebugThread2 thread = debuggerServiceInternal.CurrentThread;
    To compile this, you need to copy an Interface from the great ExceptionBreaker add-in:
    https://github.com/ashmind/ExceptionBreaker/blob/master/ExceptionBreaker/ComImports/IDebuggerInternal11.cs

  2. Simon says:

    Cool, that is a lot easier! And thanks for the heads up about ExceptionBreaker, hadn’t seen that before.

    • Robben says:

      Hey, quick question here, EvaluateSync makes the debugee crash. Any idea why that might be the case ? Thanks

  3. Simon says:

    No, sorry, I’ve never seen that. It’s a while since I wrote this code and I’ve kind of forgotten how it all worked I’m afraid!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s