Thursday, October 31, 2013

EventSource & Performance for High Volume Events

.net framework 4.5 introduced EventSource API to write events using ETW infrastructure. The choice of ETW infrastructure has made the writing of events lightning fast. Now they can be consumed by any ETW consumer. If the event provider is not enabled, then the events just fall on the floor. On the other hand, if a session is already established, they are written to ETW session buffers. Now they can be consumed by the consumers. The event provider (EventSource) doesn't really have to wait until the messages are consumed. This is independent on the number of consumers registered for a particular type of events. We have discussed about ETW tools and their usage with EventSource API [Reference].

As we discussed above, the choice of ETW infrastructure has made EventSource API extremely fast. But we can even improve on it further. But remember that these recommendations would be useful in the case for high volume events generally. For low frequency events, although you would have performance improvement, but this might not be very noticeable.

Optimization # 1: Use IsEnabled() before WriteEvent()
ETW controllers can enable a provider before registration of provider. So when a provider registers itself and starts emitting events, a session is automatically created and they are forwarded to the buffer. In order to prove that, we can start Semantic Logging Service (from Application Block) and then run our application. Our service should still be able to consume the events. On the other hand, if the provider is not enabled, the events just fall on the floor by the ETW infrastructure. In order to improve it, we can check if the event provider is enabled so that we don't even use the WriteEvent() definition in base class. It does check the same thing in the base class as well.

When an Event Provider is enabled (generated for EventSource), it receives a command to enable itself. We can even use this command in our custom EventSource by overriding OnEventCommand() method. You don't need to use the definition of base class method as there is an empty implementation of the virtual method.

It caches this in a local variable. So EventSource can use this field to check if it is enabled. There is no property available in EventSource type to check that but we can find IsEnabled() method to do just that. Now a method might cause some confusion for you thinking it might be slow as it might do some other stuff and could slow it down further. As the development team suggests, it is just a field fetch. We can also verify this by dotPeeking into the framework assembly from Windows folder.

As a matter of fact there are two overloads of the method. One of these overloads is parameter less. The other one is more interesting, it lets us check if the provider is enabled for a particular level and keyword, which makes it more interesting.

Optimization # 2: Convert static field fetch to local member fetch
This suggestion is not for EventSource implementation but it refers to the use of EventSource instance in the logging site. The logging site is the code from where we call EventSource's methods. As we have seen several examples in this blog, EventSource implementation is singleton based [See singleton], it is a static variable fetch at the call site. Here is an example of the usage:

As we know static variable fetch is more expensive than an instance member fetch, we can assign the singleton instance to a local instance member. In order to improve the usage of EventSource's method, we can assign this to a local variable. Then we can use the local member at the logging site.

Optimization #3: Minimize the number of EventSources in the application
In a previous post, we discussed about the mapping between EventSource and ETW Event providers. Actually the framework registers an Event Provider in ETW infrastructure for every EventSource in the application. This is done using ETW EventRegister API. This allows the framework to pass a command to the EventSource from external controllers.

The framework also maintains a list of EventSources in the application domain. We can get the list of all EventSource (s) in the application domain using the static GetSources() method in EventSource type.

Both of these tasks are performed at startup. Since this would depend on the number of EventSource (s) in the applications so the higher number for event sources in your application would mean slower startup. This can be resolved by minimizing the number of EventSource in your application. I would definitely not suggest an EventSource for each type in your application. There can be two options to resolve this.

The first option is using Partial types. We can span the same type across different files in the same assembly. All of these definitions are combined to generate a consolidated type by the compiler. We generally organize our types in different folders in a project. Here each folder generally represents types belonging to the same group. We can provide all the event methods definition for the method group for the types in this folder.

Most of the real life projects span more than one projects. In this case, we should still be able to minimize the number of EventSource (s), by defining on for each group, in order to improve the startup performance. Here each event source can take care of instrumentation requirement for a group of types in your application. You can group them logically or the way you want.

Optimization # 4: Avoid fallback WriteEvent method with Object Array Parameter
There are various overloads of WriteEvent() methods in EventSource type. We need to call WriteEvent() method from our [Event] methods in EventSource type. One of these overloads is an overload with params array. If none of the overload matches your call then the compiler automatically falls back to this overload.

We should be avoiding the fallback method as much as possible for performance reasons as it is reported to be 10-20 times more expensive. This is because the arguments need to cast to object, an array needs to be allocated and these casted arguments are added to the array. Then calling the methods with these arguments as serialized.

In order to avoid using the fallback overload, we can introduce new WriteEvent() method by overriding it.

Wednesday, October 30, 2013

Adding PCL Reference to Mono Touch and Mono for Android Applications

October has been a fun month. I got to travel across the country to three different places, met with amazing people and shared the common enthusiasm about technology. It's like our souls know each others. Code camps are opportunities for learning for everyone; including presenters and attendees. For one, presenters in one session like to go to other sessions as attendees, the other is, when they are asked thought provoking questions covered with practical problems of the individual developers. Most of the times, it makes the presenter think about the specialized area; it makes them see it in a different perspective.

I am planning a separate post about the experiences about the October. I think this would be a good idea to discuss some questions on the blog. Let's discuss first question in this post. The question is:

Can we add PCL references in Montouch and Mono for Android applications?

The short answer is Yes; and we can also do that in Visual Studio. I have Xamarin Starter edition installed on my machine. It also includes a trial version of Visual Studio extension for Xamarin tools.

Xamarin Studio
Here I have a sample solution called NameReflectorApp. It has a portable class library project Xamarin.Core. It also has an android application project NameReflectorApp.Android.

The portable class library project is targeting a number of frameworks as shown below:

This is targeting Profile 158 of .net portable subset. There are still some issues with Xamarin Studio like you see a warning message at the bottom of the target frameworks. But don't worry, this should still work. We cannot see here Mono for Android as one of the target frameworks but we can still add such references. In Xamarin's term, this is called Edit References, funny...

Visual Studio Support
Xamarin has support for Visual Studio. But this is not available for all the editions. Generally speaking, it is available for more expensive ones. But we can still play with the extension during the trial period available when we install the starter edition. The best thing is we can decide when we want to start the trial period. We can use this option to first familiarize ourselves with the technology and when we need to see how well it integrates with our current Microsoft technologies, we can start the trial period.

Visual Studio is not as forgiving in adding references in such case. There are issues with the Xamarin's Visual Studio extension. As we add our portable class library's reference to Android application, it shows up the following error dialog.

Xamarin developer community terms this situation as a lie. In order to fix this, we need to use some xml hack. I would totally recommend the following post where we discussed how we can add additional frameworks options for Portable Class Library projects.

As we discussed in the above post, we can add options for additional target frameworks by adding files the Supported Frameworks folder for a target profile. Let's add two new Xml files to our selected profile as follows:

The contents of these files are as follows:

After adding the above files, we need to restart Visual Studio. After that, as we go to the PCL's property page, we see that there are already Xamarin.iOS and Xamarin.Android in the target framework list. If we hit Change button, we see additional options for Xamrin.iOS and Xamarin.Android. These options are checked.

Adding the reference to the Android application project goes through just fine. Here is the portable library's reference added to the project's references list.

If you have Xamarin Studio running then restart this and open the same solution again. You can see that these new frameworks are also available in Xamarin, which is amazing.


Sunday, October 20, 2013

SLAB, EventSource and Standard ETW Tools

As we have been discussing through the past few posts that we can direct event log data generated through EventSource to be saved in different destinations. EntLib6 provides a number of sinks provided just for that purpose out of the box. They include support for console, files, Sql Server database and Windows Azure Table storage. We have also seen how we can direct the event's data to Windows Event Log [Discussion]. It would be interesting to see how we can use the existing ETW tools to view the event's data generated using EventSource API in .net framework 4.5.

EventSource API is based on registration free ETW [Event Tracing for Windows] data. This means we don't have to register an ETW event provider for generation and consumption of these events. The existing tools including LogMan and Windows Performance Analyzer are based on registered event providers. But since EventSource API is still based on the same ETW infrastructure, we can register the Event Provider manually and direct the events' data to an ETL file. Since these tools can work with this format, we can use our expertise in these tools to troubleshoot and analyzer situations we need this data for.

It must be remembered that EventSource API uses ETW infrastructure only in the case of out-proc listeners.

Performance Monitor [Perfmon.exe]
Since EventSource establishes an ETW session for out-proc consumers, we should be able to use PerfMon to see the details of the session.

As an ETW provider, each event source is assigned with a GUID. We can note the unique identifier from the properties of the provider. This identifier is generally used by ETW controllers to start / stop a session. We can also verify that the streaming mode for the configured EventSource as Real time.

Like PerfView, PerfMonitor is also based on TraceEvent library. The major benefit of PerfMonitor over Perfmon is that the former can be used as an ETW controller.

Internally Logman seems to use Performance Logs & Alerts service. If the service is not running, the utility just starts this. Make sure that we are running the command prompt with Administrative privileges.

Here we are creating a trace data collector using Logman utility.

After running the above command, we should be able to find the specified collector when queried. This can also be done using Logman. The utility supports a verb "query" which list all the data collectors.

We need to start the data collection for the events generated from the event provider. In order to do that we can use Logman's start verb with the provider's name as follows:

Querying the provider again should show the status of the provider as Running. This should ensure that the command ran successfully.

Stopping the trace session should flush the trace data in the ETL file. Here is the file generated for our provider in the same folder as we specified while creating trace data collection for the provider.

Logman also registers a user-defined trace which can be viewed with Performance Monitor.

This is another useful Windows Utility for ETW data. One of the usage for this utility could be to process the ETL files generated using any other tool. It can be used to generate human readable files from the binary data. Let's see the following usage:

The above command should generate dumpfile.xml and summary.txt in the same folder. For the events generated for our event provider, the following files are generated. You can open them to have an idea about the expected format for the generated files. We should note that the utility allows us to control the names and format of dump and summary file names and formats. The supported formats for dump file include XML (default), CSV and EVTX.


We can also generate xml (default) or html based report for the generated events data. Here we are generating the report in html format. It generates the report based on the etl file provided in the same command.

You can have a look at the following report generated for events generated from our event source.

Tracerpt can also use ETW data from real time sessions. We can use PerfMon to determine the session we are interested in. We can then use the same name with -rt switch for TraceRpt. Here is the session details for our event source.

Thursday, October 10, 2013

VS2013 RC: Portable Class Libraries Target Frameworks

In this post we are going to discuss the changes in target framework selections in Visual Studio 2013 RC. Up until, Visual Studio 2012, Visual Studio has supported the following framework selections:
  1. .Net framework 4.0, 4.0.3 & 4.5
  2. Silverlight 4 & 5
  3. Windows Phone 7.x & 8.0
  4. Windows Store Apps
  5. Xbox 360
With Visual Studio 2013 RC, now Portable Class Libraries can also support targeting .net framework 4.5.1 and Windows Store Apps (Windows 8.1). But Visual Studio 2013 has mainly reduced other target frameworks for PCL. The following is the list of frameworks which are not supported in Visual Studio 2013 for PCL development.
  1. Silverlight 4
  2. Windows Phone 7.x
  3. Xbox 360

Supported Framework Xml Changes
In order to support a combination of target frameworks for PCLs, Visual Studio keeps a list of xml documents in Supported framework folders for each profile. Actually, a profile is selected based on target framework selection matching the profile.

Now we can develop PCLs in three different versions of Visual Studio. They are Visual Studio 2010 SP1, 2012 and 2013 RC. It appears that in order to limit target frameworks for PCL development in Visual Studio 2013 RC there are some changes in the Xml schema for these documents. Let's see the document for Silverlight 4 in Profile44 in Visual Studio 2012.

Visual Studio 2013 has added a few new attributes. They are FamilyName, MinimumVisualStudioVersion and MaximumVisualStudioVersion. We must remember that the version of Visual Studio 2012 and 2013 RC are 11.x and 12.x respectivly.

The above is the Xml document for Silverlight 4 in the Supported Framework folder in the same profile after the installation of Visual Studio 2013 RC. We can notice that the maximum version for the framework is "11.0", so the framework is not supported in Visual Studio 2013 RC. Below is the framework xml for Windows Store apps (Windows 8.1) in \v4.6\Profile\Profile44\SupportedFrameworks folder. From the xml document, we can clearly figure out that this profile can only be used for a Visual Studio version 2013 RC at minimum.

Converting Project for Unsupported Frameworks
Opening a PCL project in Visual Studio can convert the target frameworks to one of the supported frameworks. A Silverlight4 project is converted to Silverlight5 project.

We can opt to not migrate the project to a new framework version but we wouldn't be able to work on it in Visual Studio 2013 RC then. The project is shown in Solution Explorer as follows:

There seems to be a bug in the current version, which shows the following error message if we don't opt to convert, by not checking the check box. It is shown as follows:

Wednesday, October 9, 2013

Silicon Valley Code Camp 2013 Session Material

It was a great experience speaking at Silicon Valley Codecamp this year. As promised, here is the recording of my session.

Cross Platform Development with Portable Class Libraries from Muhammad Siddiqi on Vimeo.

This is the slide deck from my session:

You can get the code from my sky drive. Here is the download link:

Following, you can see further details about the event and sessions. You can also evaluate the session.