Monday, February 13, 2012

HOWTO: Replace a VB6 dll with a C# dll when methods have optional string parameters

My quest to replace my VB6 COM dll with a C# dll continues. In my last post I resolved the problems I was having where early binding to the DLL didn't work, but late binding did.

The problem I need to address this time is handling a VB6 method that has an optional String parameter. Something like:

Public Sub TestMethod(ByVal sArgIn As String, Optional sArgOut As String)

Looking in the TypeLibrary for the DLL (the *.IDL file we get from the OLEVIEW tool), we see this:

HRESULT TestMethod(
[in] BSTR sArgIn,
[in, out, optional] BSTR* sArgOut);

Creating the TLB via the MIDL command works, but it gives the following warning:

.\MyDll13.IDL(30) : warning MIDL2400 : for oleautomation, optional parameters should be VARIANT or VARIANT * : [optional
] [ Parameter 'sArgOut' of Procedure 'TestMethod' ( Interface '_TestClass' ) ]

I'll now go ahead and create my .Net DLL. When I use .Net to implement the interface, it stubs the C# method as:

using System.Runtime.InteropServices;

namespace MyDllNet
public class MyProxyClass : MyDll13ModifiedTA.TestClass
public string GetVersion()

public void TestMethod(string sArgIn, [OptionalAttribute]ref string sArgOut)

This looks good. After completing the code and compiling it, I run my original VB6 app which uses the TestMethod method from a VB6 app via late binding. It works fine when I pass two parameters, but if I only pass one parameter it falls over with "Run-time error '13' Type mismatch". Interestingly, the VB6 app which uses early binding works with both one or two parameters passed!

C# DLL implements IDL with optional BSTR* argument
VB6 BindingPass One ArgPass Two Args
Early BindingOKOK
Late BindingOKError "Type mismatch"

Plan B. Plan B is to change the optional parameter in the IDL from BSTR* to VARIANT*. According to the MIDL error message, VARIANT* is the right type to use for an optional parameter in COM. Note that I'm not changing the original VB6 COM component, I'm just changing the definition in the Type Library.

Creating the TLB via the MIDL command no longer raises a warning; implementing this new IDL/TLB in Visual Studio yields the following method signature:

public void TestMethod(string sArgIn, [OptionalAttribute]ref object sArgOut) {}

But does it work? No, it's worse! Now it crashes the CLR!!

C# DLL implements IDL with optional VARIANT* argument
VB6 BindingPass One ArgPass Two Args
Early BindingError clr.dll APPCRASHError clr.dll APPCRASH
Late BindingOKOK

OK, this confirms that I don't know what I'm doing! But wait, if I remove all of the code from my "TestMethod" in the .Net DLL then everything works!?! The APPCRASH must be caused by the way I'm handling the object argument in my .Net code. Time to debug into the .Net dll component and see what's going on!

And it turns out that if I try to return anything at all (apart from null) in the optional parameter, I get a System.StackOverflowException in an Unknown Module. Almost time for me to give this one away. Just one last thing to try... VB.Net!

Plan C. VB.Net VB.Net offers more comprehensive COM support, presumably because it is supposed to be a migration path for people who have VB6. So I'll create a new VB.Net class project, and add a reference to the "optional BSTR*" assembly (create in Plan A above), and implement the interface in VB.Net. The VB.Net code looks something like:

Imports System.Runtime.InteropServices

Public Class MyProxyClass
Implements MyDll13TldAssembly.TestClass

Public Function GetVersion() As String Implements MyDll13TldAssembly._TestClass.GetVersion
End Function

Public Sub TestMethod(ByVal sArgIn As String, Optional ByRef sArgOut As String = Nothing) Implements MyDll13TldAssembly._TestClass.TestMethod
End Sub
End Class

And guess what? It works! Late-binding, early-binding, the optional supplied, the optional not supplied. All combinations work!. Job done.

To Conclude: If you want to implement a VB6 COM interface, which has an optional String parameter, in .Net, then use VB.Net! (with some provisos)

No comments: