by ref parameters

Here is a starting patch (byref1.patch), but it still doesn't work.

Here are the docs on system.reflection.emit and byref parameters: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpref/html/frlrfsystemreflectionemitparameterbuilderclasstopic.asp

And here is a test script:

def testref(ref refparam as int):
print "refparam", refparam, refparam.GetType()
refparam = 3

def testout(out outparam as int):
print "outparam", outparam, outparam.GetType()
outparam = 5

x as int
y as int

x = 1
testref
print x //should be 3, not 1

testout
print y //should be 5, not 0

2nd version of patch. Still incomplete though. Missing some necessary features and missing proper test cases.

I'm attaching a test script next showing areas where it still fails.

-biggest bug: directly passing a byref param on to another method (or superclass method) by ref fails. Also returning a byref param directly fails.
-not yet implemented but easy to do: passing byref doubles or decimals and other types (need to use ldind_R8 and stind_R8 for example instead of ldind_I4/stind_I4)
-possible bug: passing nested types by reference. Haven't tested this. I've only tested external types and internal types known by the module.

-"feature" - two overloaded methods that are the same except one has a byref param and the other a regular param will appear the same to boo. I believe C# distinguishes the two. I don't think it will be possible to support this in boo, but it is evil anyway.
-"feature" - Since boo doesn't complain about using an initialized variable, there is really no difference between using ref and out params. For example C# would complain about this:
uninitialized2 as MyClass
print uninitialized2

Returning a byref param works actually. It is only passing on a byref param to another method as a byref param that still doesn't work.

The patch in byref3.zip solves the problem of passing on byref parameters directly to another method with byref parameters. byref.boo was expanded to include more tests.

byref4.zip adds support for passing other value types by reference such as double, short, decimal, etc. All my test cases are working now.

Nothing significant changed in byref5, but it uses a clearer logic in emitassembly, adds more tests, adds support for ref and out params in booprintervisitor, adds "ref" and "out" keywords to the JEdit and other syntax files, and fixes a name conflict in the #develop addin (had a param named "ref").

I understand the need to support calling methods with byref arguments. But I don't understand the need to support their definition in boo.

What problem are we trying to solve?

Basically, for P/Invoke declarations in Boo.

I recall several months go, before Boo Explorer adopted the SharpDevelop Editor (and was just using a TextEdit control), that it needed to have one C# file that P/Invoked the location of the caret in a text box so it could display the primitive code completion window. That was it--the entire file was just one class with a static P/Invoke method in it, and for that you had to invoke an entirely different compiler to make it an assembly which you would then reference with Boo in order to get the GetCaretPos() functionality.

I also forgot to mention that if you're going to write an API that P/Invokes, you may need to expose byref in your public API if you're going to be passing an incoming primitive to a P/Invoked that takes a ref.

There's also a small usecase someone mentioned in the irc channel (something about passing integers by ref?); does anyone remember what it was?

I recently tried to make some ASP.NET 2.0 stuff in Boo, but had to give up writing providers for membership etc., since I couldn't override methods like this one in the MembershipProvider class:

public abstract MembershipUser CreateUser(
stringusername,
stringpassword,
stringemail,
stringpasswordQuestion,
stringpasswordAnswer,
boolisApproved,
objectproviderUserKey,
out MembershipCreateStatusstatus
);

(http://msdn2.microsoft.com/library/ms152040(en-us,vs.80).aspx)

Providers are essential in ASP.NET 2.0 development, and not being able to make a membership provider in Boo is a pity. And this is probably not the only case where you would need to implement interfaces or abstract classes having methods with ref or out parameters!

ok. it makes total sense.

byref6 adds support for overriding external methods with byref parameters.
I did this by slightly changing TypeSystemServices.CheckOverrideSignature, but more work may be necessary (like in ProcessMethodBodies.ResolveCallableReference). I was hoping to avoid changing the typesystem as much as possible.

Also, I tested passing array elements (arr[0]) and list elements by reference. Passing array elements by reference works, whereas list elements do not. This limitation existed before my patch I believe.

Reopen when it is decided to add this. There never was any feedback on the patch.

Hello gentlemen,
as you are probably aware of we are working on exploring the options of making boo the new integrated programming and scripting language inside MS Office products. One major obstacle in our ability to deliver a VBA/C# comparable user experience is the missing ref capability. Ref parameters are used sometimes in event handler signatures within Excel (for which we have a crude workaround), and all over the map within MSWord. Since our resources are very limited, this issue has become a crucial decision point for our roadmap moving forward.

Please let us know what the chances are to get this issue re-opened and resolved?

Best wishes and thanks for making Boo one of the greatest things we've had the pleasure to work with.

Albert Ullrich
Trelliswerk LLC
www.trelliswerk.com

Thanks for the feedback, Albert.

I'm sure everyone would love to see boo being the language of choice for office automation. Could you provide a sample delegate definition for us to test the implementation with?

Doug, are these patches still valid for the current svn revision?

Yes but you may have to manually make the changes to ParameterDeclaration.cs. Tortoisesvn doesn't like the header on that file.

Everything is working, but there are two peverify issues that Rodrigo might want to tackle because they involve code he is more familiar with. I've forgotten half of this IL stuff anyway.

One is when you define and use callable types with byref parameters, like so:
callable TestByrefCallable(ref x as int, y as int) as int

They work as expected, but peverify shows errors like:
[IL]: Error: [....exe : TestByrefCallable::Call] [offset 0x0000000F] [opcode callvirt] [found address of objref 'System.Object'] [expected address of Int32] Unexpected type on the stack.

Probably need minor changes to the InjectCallableConversions or EmitAssembly to account for byref parameters in delegates, but I am unfamiliar with the code in InjectCallableConversions.

A second peverify issue is passing decimals by reference to an internal method. It will work fine, peverify shows errors like below. EmitAssembly needs to do some extra work when the type is decimal and GetStoreRefParamCode is OpCodes.Stind_Ref.
peverify output:
[IL]: Error: [....exe : Boo.ByRefTest.Byref-main-testsModule::storerefdecimal] [offset 0x0000000F] [opcode stind.ref] Expected an Objref on the stack.
[IL]: Error: [....exe : Boo.ByRefTest.Byref-main-testsModule::storerefdecimal] [offset 0x0000000F] [opcode stind.ref] Expected address of an Objref on the stack.

The problem is more general than passing decimals by reference. I wasn't properly testing structs. So extra work needs to be done when passing complex value types by reference, but OpCodes.Stind_Ref may still be the correct opcode to use in those cases.

struct MyStruct:
X as int
Y as int
def constructor(x as int, y as int):
X = x
Y = y

def teststruct2(ref s as MyStruct):
s = MyStruct(11,21)

s2 = MyStruct(1,2)
teststruct2(s2)
print s2.X, s2.Y, s2.X==11, s2.Y==21 //<--error, s2.Y is still 2

For callables/delegates, apparently you have to pass the byref parameters to EndInvoke, followed by the iasyncresult parameter.

I partially implemented this by changing TransformCallableDefinitions.CreateEndInvokeMethod (below). But
TypeSystemServices.CreateEndInvokeMethod also needs to be changed. Since that creates the EndInvoke method in an entirely different way than the one in TransformCallableDefinitions, it may require adding byref information to the type system, which is what I was hoping to avoid.

TransformCallableDefinitions.CreateEndInvokeMethod:

Method CreateEndInvokeMethod(CallableDefinition node)
{
Method method = CreateRuntimeMethod("EndInvoke", node.ReturnType);

foreach(ParameterDeclaration p in node.Parameters)
{
if (p.IsByRefOrOut)

}

method.Parameters.Add(
new ParameterDeclaration("asyncResult",
CodeBuilder.CreateTypeReference(typeof(IAsyncResult))));
return method;
}

Here's an updated patch that fixes a number of issues.

At least two of the remaining issues include:

Doesn't check and raise an error if you pass a non-lvalue to a byref method:
def doit(ref y as int):
y = 4
doit(10) //no complaint
//also doesn't complain if you cast a byref method to a nonbyref callable or vice versa

Also events with handlers that have byref parameters do not work:

import System
class Observable:

static _value = null

static Value:
get:
return _value
set:
old = _value
_value = value
Changed(old, _value)
print "after changed:", _value

static event Changed as callable(object, ref object)

def handler(old, ref new):
print("$

, $

")
new = "wuh"

Observable.Changed += handler
Observable.Value = "foo"
Observable.Value = "bar"
Observable.Changed -= handler
Observable.Value = "baz"

//should see "after changed: wuh?"

Dug, we got a problem with "out." From the Boo perspective, "out" is useless.

C#:
an out parameter of a type must be assigned before it is used; that is, it must be assigned by the callee
a ref parameter of an array type must be definitely assigned by the caller.

What does this mean?
Method A declares a variable FOO.

OUT
===
If method B takes FOO with OUT, method A does not have to initialize FOO before it passes it to B.
FOO gets initialized in B. The C# compiler checks that it is initialized in B before any operations occur. It also checks that it gets initialized before B returns.

REF
===
If method B takes FOO by REF, method A must initialize FOO before it passes it to B, otherwise an error occurs.

How is this a problem?
===================
Boo automatically initializes variables. That is why you don't get "unassigned variable errors like you do in C#."
foo as int
print foo #<-- no error; prints ZERO

So, either we make Boo force you to initialize variables, which helps by killing a few bugs, but most of the times it sucks, or we remove "out" support, and let those who want to support C# use the [Out] attribute like VB does. To tell you the truth, I don't even remember seeing an OUT parameter in .NET. I did see it in p/invoke code, however, since Boo does initialize the variable, it's not required in Boo.

byref9.patch fixes all the issues I've found so far. However there still is a CDATA error with nant test. There's not any information about what file is causing the error, but it's not any of my byref tests.

Events with byref parameters work, and it thows an error if you pass something that isn't a reference expression or an array slicing expression to a byref parameter:

def doit(ref y as int):
y = 4

//these all throw errors now:
doit(10)

l = [1,2,3]
doit(cast(int,l[1]))

doit(9+3)

//these still work:
x = 3
doit
print x

arr = (1,2,3)
doit(arr[0])
print arr[0]

Right, there is essentially no difference between out and ref in boo because boo initializes variables. I kept it there for those who want compatibility with C#. An attribute might work, but [Out] is used for COM interop. out params use a different kind of attribute.

I hope we won't add too many of these. I don't want Boo to become polluted for the sake of other languages.

byref10.patch fixes the nant test issues. All tests are working now.

Removed out params.

Patch applied. Thanks, Doug!!!