.NET corrupts heap when marshalling LPSTR

(Hi Lizzie)

I've just spent the best part of a week trying to track down why my bridge from C# to C++ was working on Windows XP but crashing sporadically on Windows 7, and the answer is that .NET marshalling is trickier than you think for strings.

Essentially, I had this:

[DllImport("mydll.dll",CharSet=Ansi,CallingConvention=Cdecl)]
[MarshalAs(UnmanagedType.LPStr)]
private static extern String LookupCorrespondingString(Int32 key);

and in the DLL, I had

__declspec(dllexport) const char *LookupCorrespondingString(int key);

Whenever I called this, it would get all the way into my DLL, I could tell it was going to return a value, but during the return operation, it would crash. When I ran it in the debugger, I got output messages about how memory was being free'd into the wrong heap.

Eventually I found this article:
https://blogs.msdn.com/dsvc/archive/2009/06/22/troubleshooting-pinvoke-related-issues.aspx
which contained the useful quote (emphasis mine):

When a string buffer allocated by native code is marshaled to managed code, CLR Interop marshaller will allocate a managed string object and copy the contents of native buffer to the managed string object. Now in order to prevent a potential memory leak, CLR Interop Marshaller will try to free allocated native memory. It does so by calling CoTaskMemFree. The decision to call CoTaskMemFree is by-design. This can at times lead to crash, if memory was allocated by the called-native function using any API other than CoTaskMemAlloc family of API’s as custom allocators may allocate on different heaps.

And there was the answer. .NET was freeing the block of memory I had passed to it to help me "prevent a potential leak". The problem being the lack of ability to communicate the 'const-ness' of the underlying DLL entry point's return value in the MarshalAs() attribute.

The solution was to declare the entry point differently:

[DllImport("mydll.dll",CharSet=Ansi,CallingConvention=Cdecl)]
private static extern IntPtr LookupCorrespondingString(Int32 key);

and then when I call it, do the marshalling explicitly.

String s = Marshal.PtrToStringAnsi( LookupCorrespondingString(k) );

After I bitched and moaned about how this stuff isn't documented anywhere, Manish Jawa kindly pointed out that it is, in fact, documented in the very first sentence on this page: http://msdn.microsoft.com/en-us/library/f1cf4kkz.aspx - you can't ask for more than that.

Actually, you can and they give it to you here: http://msdn.microsoft.com/en-us/library/x3txb6xc.aspx - the problem I was experiencing and its solution spelled out.

So, the moral of the story is: if you pass native strings to .NET via the marshalling interface, make sure you use IntPtr and PtrToStringAnsi() unless you want them to be free'd for you.