Expression Trees - Part I

Expression trees are the representation of code as data structure instead of executable code. We have used expression trees for the discussion regarding INotifyPropertyChanged but never got a chance to discuss the concept itself.

The feature was introduced in .net framework to support meta-programming in .net framework i.e. to be able to treat code as data. The feature enables us to analyze the code at runtime. It also allows the run-time creation of code. In INotifyPropertyChanged example, we used Expression trees to generate new code based on the expression passed as argument. This involved code analysis and generation which was only possible by the use of expression trees.

Expression tree works by creating a semantic model of code. This semantic model can be used by compilation tools to generate runnable code.

What does it enable me?
Expression trees enables various feature both in the framework and to the developers. It can be used for:

1. Dynamic modification of executable code.
2. The execution of LINQ query.
3. Creation of dynamic queries.

Expression Trees In .Net Framework
The expression tree types were introduced in .net 3.5. They are further enhanced in .net 4.0. The types can be found in System.Linq.Expression namespace in System.Core assembly. The class hierarchy of the types can be presented as follows:

Amongst the class in the hierarchy, there are three sealed classes:

1. BlockExpression
2. GotoExpression
3. Expression<TDelegate>

How to create Expression Trees?
Expression trees can be created using the following:

1. Using a lambda expression
2. Manual creation using the available API

We will be discussing both of them in this post.

Assignment of Lambda Expression
A lambda expression can be assigned to an expression tree or a delegate. It cannot be assigned to an implicitly typed variable. Basically C# compiler emits an expression tree or delegate based on the destination variable type. If we are using an implicitly typed variable then it would not be possible for the compiler to decide and hence the error.

We can also base our expression tree off of Action delegate if the expression is not returning any data. In the following example we are just creating an Expression Tree which just prints a string on the console. It doesn't return any data so we are using Action delegate here. We compile the expression into delegate in the same way as we did before. Now we can use it like a regular delegate.


When a lambda expression / anonymous function is assigned to a delegate it creates a reference for the executable code which might be used to call the executable function. On the other hand, assigning to expression tree creates a representation of the code of the lambda expression (anonymous function). This can not be executed directly but it may be used to find out how the executable code would look like. We can also tweak the code by playing with the nodes of expression tree. We can also execute other code based on the nodes in expression tree. That is exactly what we did with the example referred in first paragraph. In the example code, we determine the property (member expression) in the expression tree and then we raise PropertyChanged event for the said property. Doing this relieved us from using the magic string in the setters of our models and view models.

Conversion between Expression Tree & Delegate
An expression tree can be converted into a delegate by compiling it. It must be remembered that the reverse is not true i.e. a delegate can not be converted to an expression tree.

If a conversion exists from an anonymous function to a delegate type D, a conversion also exists to the expression tree type Expression<D>. Based on the same principle, the compiler converts the lambda expression, passed as argument, to the type of parameter. The parameter type might be either ExpressionTree or Delegate.

Building Expression Trees:
Expression trees are a lot easier to build if we think bottom-up. We can start building from leaf and work our way towards the top of the tree. It is a lot easier if we make a rough sketch of how the tree structure would be and then build the expression tree.


Let's first build a tree to represent the expression in the above algorithm.

If we were using lambda expression, then we can easily create a lambda to assign to a variable of Expression type. We can also use this to create a delegate. We can also use the delegate to generate some result.


Here the lambda expression returns and integer based on the evaluation of an expression. The lambda expression is assigned to an Expression type of variable. This is then compiled into a delegate as described above. The result is computed by the delegate and result is returned. This is rather confusing and not recommended use of expression tree. Why in the world we created the expression when we just needed an executable code. We could have rather assigned the lambda expression to a Func delegate and use that for computing. Or we could just return the result of Lambda expression and the framework would take care of creating an implicit anonymous function for the lambda expression. We would certainly never use this for production code. But in order to understand the feature, we would continue to use this to check the correctness of our expressions.

How to introduce variables:
More than usual we need expressions involving variables. In Expression Trees, variables are specified using Parameter Expressions. We have two options to declare parameter expressions.

1. We can instantiate ParameterExpression directly.
2. We can use Expression's factory method i.e. Expression.Variable to get the instance of parameter expression.

I prefer using the 2nd option and that seems to be a recommendation in documentation as well.


We can also use Expression.Parameter to create a ParameterExpression. In the following example, above example is recreated just changing Expression.Variable to Expression.Parameter.

This seems similar to the previous example except that we introduced Expression.Parameter instead of Expression.Variable.

Debugging Expression Trees
Visual Studio has little support of debugging expression tree. We can see how expression tree structure looks like while debugging. Just hover over an expression tree variable and use TextVisualizer to see DebugView.

The debug view shows the expression tree in a particular format. Different Node types are represented differently. The details of the format might be found here [http://msdn.microsoft.com/en-us/library/ee725345%28VS.100%29.aspx]

We can also use Html Visualizer to see the expression tree in the same format as the Text Visualizer.

Instead of hovering over the expression variable, we can use quick watch to see the same expression tree.

We can also use Immediate Window to see the same result.

Expression Trees & Dynamic Language Runtime
.Net CLR makes use of Dynamic Language Runtime [DLR] to provide support to dynamic languages or dynamic features of a language. The runtime uses expression trees for the dynamic code. It passes the expression tree to the appropriate runtime binder. The runtime binder selection would depend on the dynamic code including C# dynamic binder, Iron Python dynamic binder or for legacy COM Objects.


The runtime binder maps the request to the target's object call structure.

While binding, it might not find the expected properties / operations on the target object which might result in some exceptions. Now you realize the Microsoft.CSharp assembly reference to your .net 4.0 projects.

The use of expression trees by DLR happens on backend by the runtime. All you need to care about is the exception which could result.

Limitations:
It must be remembered that lambda statements cannot be used to build expression tree. These are lambdas specified in terms of code blocks [in c# they are defined within the braces {lambda_stamement}] which might return some values.

Hello, this is C# method. Who is Calling me?

And there was no easier answer other than going through the stack trace and scanning the frames yourself. Well this is just made easier in Visual Studio 2011 Beta and we will be discussing the same feature today. This provides a method details about the caller.

Why should I care?
It seems that you haven't been involved in chasing and tracing some weird flow in which your code is executed. This has always been a pain for most of us. Sometimes, the only option left is to write a bunch of Trace.WriteLine() with some text to identify the method. Then at the end of usecase, you look at the trace to find out exactly what was the flow of execution of your code. This feature is provided to further support this tracing, debugging and diagnostics.

What Information does it provide?
This is an optional details that you can add to all or some of your methods. With the new features, you can get the following information easily:

1. The compile time path of the source file of the caller.
2. Line Number in the source file where the method is called
3. The name of member [Property or Method] of the caller.

How is this feature supported in the framework?
This feature is supported by providing new attributes in System.Runtime.CompilerServices namespace in mscorlib.

How can I use them in my C# code?
These attributes are targeted for methods. We need to decorate the parameters with any of these attributes and the run-time would populate them with the relevant information as promised. This is built on top of another great C# feature. It is called optional parameters. This would have all the limitations of optional parameter i.e. these parameters must be the last parameters of the methods and they should have some default values assigned based on their types. CallerMemberNameAttribute and CallerFilePathAttribute may be assigned a string value. CallerLineNumberAttribute must be specified with some integer value.

Let's create a C# console application and update Program.cs as follows:


Here we have used all three of the newly available attributes in CallerInfoMethod definition. We are calling the method from within the same class from Main method. Let's run the application and see what it prints on the console.

Can I use more than One Attribute for a Parmeter?
I don't know why you would do that but you might be a bigger creative thinker than me. You might want to combine the information from those attributes. So to answer that, Yes you can use CallerFilePathAttribute and CallerMemberNameAttribute with the same parameter. They cannot be used with CallerLineNumberAttribute because of different data types. But the information is not combined instead, as I have noticed, CallerFilePathAttribute takes precedence and overrides CallerMemberNameAttribute. Let's update the same method as follows:


When we run the above code, we notice this override behavior:

Possible Improvement
I have seen most of the developers shooting down any and all ideas which involve System.Runtime.CompilerServices, and there is a reason for that. This is based on Microsoft's recommendation:

The classes in System.Runtime.CompilerServices are for compiler writers' use only.

I don't buy to this and clearly we use other things from the namespace too, like when unit testing InternalsVisibleTo is used very often. I think that it would be better if these new attributes are moved to System.Diagnostics instead. This should keep everyone happy.

Microsoft Zindabad!!!

Fusion Log Viewer [Fuslogvw] for Assembly Binding Issues

This is the best advice you can give to an engineer.

"Remember, the better you know about your tools and components the better engineer you are."

It is the best advice I can give to anyone. One of the habit of highly acclaimed writing of Stephen Covey "7 Habits of Highly Effective People", he also includes one of the habit as "Sharpen your Saw". We get so much involved with our day-to-day activities at work that we don't realize that our saw is being rusted. If we don't keep it sharpened than it wouldn't be of any use after some time. After sometime, we might also cry "Who stole my cheese?". Actually nobody did... It's us who kept eating our cheese, lived our life in the comfortable zone and now the cheese is finished. We should have been on new adventures of exploration of new avenues of cheese building. There is no benefit crying over spilled milk.

Have you ever got this message when you try to launch an application? Clearly, it can't find one of the libraries is missing which is required for the application to start up. It can't find ClassLibrary1. Well, It definitely is a weird name for your library but still can't find it. But why? Where is it trying to load this assembly from? Why does it even need this? What other libraries it is using. There are several other questions you want to ask but can't answer. In order to better get the root cause of the problem and better troubleshooting, there are several tools to help. One such tools is Fusion Log Viewer [Fuslogvw].

Fusion Log Viewer provides details about assembly binding in a .net application. It works as an .net app assembly binding monitor. It can shows details about every assembly loaded in a .net application.

Installing Fusion Log Viewer
Fusion Log Viewer is not a separate install. It is rather a .net framwework tool which is installed with Visual Studio and Widnows SDK.

Launching the Viewer:
There are various options to launch the tool. First of all, we can launch it using Visual Studio Command Prompt installed with Visual Studio. Doing that, you wouldn't have to navigate to the folder containing the tool. Just type the name of the viewer and hit enter. It should be loaded for you. You can find Visual Studio Command Prompt from Windows Start up menu.

As you see above, it is important that we run the tool as administrator. If you don't like to hold your mouse like most developers then you don't have to right click the shortcut and select "run as administrator". Just type the name of application in Start menu and select it using the navigation UP / DOWN keys like you always do. Now instead of right clicking just hit CTRL + SHIFT + ENTER instead of just ENTER. You should see the following view of Visual Studio Command prompt.

Otherwise, It would launch successfully but it wouldn't allow us to see the binding details. It greets the user with greyed out options and gives no freaking idea about the cause of this hurtful behavior.

Running as administrator, it enables the options as follows:

Using The Viewer
After launching the viewer, the tool's setting would determine what kind of log do you want to see for assembly binding. You have the following options:

1. Log Disabled: No logging for assembly binding.
2. Login exception text: Logs minimum binding information to the disk.
3. Log assembly binding failures: Generally used options. Logs in only those libraries which have binding issues.
4. Log all binds to disk: Even those assemblies which don't have any binding failures.

Let's select to log only those bindings which have failures. Now we run an application which have some library binding issues. This should be logged as follows:

The number of entries in the list for an application would depend upon the number of assemblies which can not be loaded. If we had selected the option to log all bindings then an entry would represent an assembly loaded in an application.

Where are these logs created?
By default, these logs are created in Temporary Internet folders. It is also the recommended way because you wouldn't have to worry about cleaning this up. Alternatively, we can change the location in Settings view as follows:

Example Log

LOG: This bind starts in default load context.
LOG: Using application configuration file: C:\Users\shujaat\Documents\Visual Studio 2010\Projects\WpfTestProjcect\WpfTestProjcect\bin\Debug\WpfTestProjcect.exe.Config
LOG: Using host configuration file: 
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework\v4.0.30319\config\machine.config.
LOG: Policy not being applied to reference at this time (private, custom, partial, or location-based assembly bind).
LOG: Attempting download of new URL file:///C:/Users/shujaat/Documents/Visual Studio 2010/Projects/WpfTestProjcect/WpfTestProjcect/bin/Debug/ClassLibrary1.DLL.
LOG: Attempting download of new URL file:///C:/Users/shujaat/Documents/Visual Studio 2010/Projects/WpfTestProjcect/WpfTestProjcect/bin/Debug/ClassLibrary1/ClassLibrary1.DLL.
LOG: Attempting download of new URL file:///C:/Users/shujaat/Documents/Visual Studio 2010/Projects/WpfTestProjcect/WpfTestProjcect/bin/Debug/ClassLibrary1.EXE.
LOG: Attempting download of new URL file:///C:/Users/shujaat/Documents/Visual Studio 2010/Projects/WpfTestProjcect/WpfTestProjcect/bin/Debug/ClassLibrary1/ClassLibrary1.EXE.
LOG: All probing URLs attempted and failed.

So the run-time probes for the assembly and you have the above details.

Note: It is not just fusion log viewer which can give you details about the assemblies being loaded as part of an application. There are also other tools. Let's look at them briefly.

Visual Studio Modules Debug Window
When you are developing an application and want to know the details about the loaded assembly. Just go to the Debug Windows menu and select "Modules". It gives the details of all the loaded assemblies. So if you can't find an assembly which you think should have been loaded already then you have an issue. We can also load symbols of these assemblies to take debugging to the next level.

If you don't see an assembly which you think should have been loaded then there is definitely a problem.

Process Explorer
This is another tool which can help you in this regard. It is a Sysinternals tool which can give details about various details about a running process. It includes the details about the assemblies being used by the process.

If you are running the Process Explorer with Admin Privileges then you could see the associated assemblies used with the process.

Process Explorer is an important tool in the developer toolbox. This can be downloaded from sysinternals [http://technet.microsoft.com/en-us/sysinternals/bb896653]

List DLLs
This is another sysinternals tool specialized for listing the dlls in a process. It is also needed to be launched as administrator. We can use "runas" for this purpose.

Obviously you need to have Secondary Logon Service running for this to work.

If you are already running something with credentials of other account than in that case Secondary Logon Service cannot be stopped. So don't get frustrated if you see the "Stop" option as disabled in Services.msc. Just close the other application and then the option would be available to stop the service.

Visual Studio 2010 Commands Cheat Sheet

I use Visual Studio Commands on everyday basis but couldn't find some one-pager cheat sheet which I could pin against my cubicle wall. It is created from the commands available in msdn here: [http://msdn.microsoft.com/en-us/library/c3a0kd3x.aspx]

You can download it here:

Handling Arithmetic Overflows in C#

In this post we will be discussing rather unknown feature of C#. This is related to the behavior of integer arithmetic in the event of overflow.

Compile Time Overflow Checks:
C# compiler checks for overflow of assignments and expressions involving only numerical literals. Let's see the follows:

If we build the project, we get the same error in the error list.

With decimal, the same overflow issue is reported as follows:

The situation gets interesting when the expression results are being assigned to a variable of a type which clearly can hold the value as follows:

Here "ticks" can clearly hold the result of the expression as it is a long type but since the members involved in the expression are default Int32, that is why, the compiler thinks that the result can not be computed without an overflow. In order to fix that we can clearly specify the types of members of expression as follows:

That is why widening conversions are implicit but we have to clearly state narrowing conversions. This could have easily worked if, in this case, we just had specified one of the operands involved as of long type. In a case like following, the types of the operands are promoted as the operand type which can hold largest value (if such implicit promotion) are allowed and then the operator (multiplication) is applied. This is also the type of result of such expression. In the following case, we are just updating one of the involved operands to be explicitly of type long. This fixes the compiler concern of possible overflow because of the operands type promotions.

Please remember that there are no such compile or run-time overflow checks for floating point numbers. As in the following example, this seems to be an overflow by assigning a value which is greater than the maximum value that the type can hold, but it compiles fine and there are also no run-time issues.

Overflows during Program execution:
Different numeric types have different behaviors at run-time if one of the operand is non-literal. Float never cause any issue when the expression is apparently resulting in an overflow. Decimal always cause overflow exception. These behaviors cannot be controlled.

Running the above code causes the following run-time exception:

Let's see if the behavior of integral types is the same. Let's write the following code and run it.


First, please don't pull your hair who have understood what seems wrong here. For those who missed, look at the above code again and see if this is a case of overflow. Isn't it? But why the exception is not generated? The above is really a case of overflow but it runs fine but the following is displayed on the console:

Basically this is the default behavior. Integral overflows don't result in overflow exception. Now we are going to discuss how we can change the behavior so that the integral overflows also result in exceptions.

Compiler Option:
C# compiler support /checked compiler option [http://technet.microsoft.com/en-us/query/h25wtyxf]. If we compile our code with /checked+, it would cause overflow exception for any overflow involving integer types. /checked- is the default behavior where overflow exceptions are not generated. Don't worry, you wouldn't have to go back to using command prompt to compile your code using csc.exe directly. Visual Studio IDE supports setting this. Just open the property window of a C# project and go to Build tab. Clicking Advanced button should open the following window. Here, through the specified check box, we can control the runtime behavior of integral overflows. If checked, an exception is caused at runtime.

Setting this option would affect code generation in such a way that an Overflow exception is generated in case of overflows. Let's check the above checkbox and run the code again.

Updating the above setting modifies the build configuration. CheckForOverflowUnderflow is updated as true for the configuration.


If we want to update this behavior for release as well then we can update the release configuration in the same way. We can also create a new configuration using Configuration Manager based on an already existing build configuration. [http://msdn.microsoft.com/en-us/library/kwybya3w.aspx]

Checked / Unchecked Keyword:
We can even control the behavior for each statement or a sub-statement scope. C# also supports introducing a smaller checked context for a single statement or statement block. Let's un-check the "check for arithmetic overflow / underflow" from project settings and update the code as follows:


Here we have defined the arithmetic expression under checked context. In this scope, any integral arithmetic overflow would result in OverflowException.

The counter keyword is "unchecked". If the project is set to check for integral overflow then we can define a scope with unchecked integral overflow as follows:


When we execute the above code then there is no reported overflow exception and the control just moves to the next line of code as follows:

Please remember that checked keyword doesn't affect code generation of nested calls. Like in the following code:


When we run the above code, although the summation seems to be causing an overflow and it is enclosed in checked scope, still the generated code would not include overflow exception support for this case.

It must also be noted that checked / unchecked contexts are only applicable to ++,--, - (unary), +, -, *, / operators. It is also applicable to explicit numeric conversion. So other operations,e.g. Shift, would not cause Overflow exception even if they execute in checked context no matter shift is used for multiplication.