Enterprise Library 6 - SLAB - Supporting Complex Types for Payload

Previously we have discussed there are only certain primitive types supported by EventSource introduced in .net framework 4.5. This post is an attempt how we can work around this limitation. Actually, we can decorate EventSource methods with [Event] and [NonEvent] attributes. The other most important point is the following:

[Event] methods can be private.

We can use this to only include private [Event] methods which would be used to generate ETW manifest from the EventSource. Additionally we can introduce public methods in the EventSource type supporting the non-primitive custom type as parameters. Hence these object-oriented types make writing the logs easier without putting any load on the underlying infrastructure supporting the event mechanism.

Let's create a sample C# project targeting .net framework 4.5. We add a type EventSourceWithComplexType to the project. As you can see below this is an EventSource. Here we are using Lazy initialization introduced in .net framework 4.0 for singleton implementation [Please refer here for singleton implementation]. It has a private constructor to make sure that it cannot be instantiated from outside. The most noticeable thing is its two methods. Here we have a method StudentCreatedCore() decorated with [Event] attribute, has a list of parameters including types supported by the ETW infrastructure. It has a corresponding StudentCreated() method with [NonEvent] attribute. It has a public access modified and has a non-primitive type for parameter. The most important thing is that it is using the StudentCreatedCore() method.


Can The implementation satisfy EventSourceAnalyzer?
We have discussed that we can use EventSourceAnalyzer for testing our EventSource implementation [Discussion]. Let's see if we can satisfy EventSourceAnalyzer for the above implementation. We just need to call InspectAll() method with an instance of the EventSource. Here we are using xUnit for testing supported by resharper. You need to install the nuget package. You would also need to get EnterpriseLibrary.SemanticLogging package for using EventSourceAnalyzer.


As a matter of fact, EventSourceAnalyzer has no complaints and the unit test passes. Zindabad!



Listening for Events
Now let's see if we can listen for events using this EventSource implementation. Here we are attaching the EventSource with ConsoleSink using the ConsoleLog [Discussion]. We are using the same EventLister to enable the events. Here we are just using the public method on EventSource type with an instance of Student type. Let's see if we could see the event's data on the console [keep the spirit high, friend!!!].


And we do see that event's data is being printed on the console in the same format as we provided in the StudentCreatedCore() method. Hence the runtime was able to do it from the private method. It has no problem with generating the ETW manifest for private [Event] methods and using it at run-time. Zindabad!



Download


Love for other people what you love for yourself.    [Prophet Muhammad - Salallahu-Alaihe-Wasallam]

EntLib6 - Semantic Logging using Windows Azure Sink - Table Storage

This post is part of series of posts discussing about Semantic Logging Application Block [SLAB]. Earlier we have discussed about how we can push the log data to file and SQL Server storage. In this post we are discussing how we can use Windows Azure Table Storage as a destination of these logs.

Windows Azure Sink Configuration
As we have discussed updating destination and format of the event data requires no changes in the application generating these ETW events if we are using out-proc consumption of these events. SLAB provides Semantic Logging Service. We can use it out-of-the-box to make all these necessary changes without event bringing the application down. The configuration is done in SemanticLogging-Svc.xml configuration file for the service. Here we are using Windows Azure Storage Emulator.


Storage Emulator Not Running / Installed
For our development machine, we first need to install Windows Azure SDK to run the above. We specially need Windows Azure Storage Emulator running for this example. The emulator uses a default account key to test the application logic. In absence of an emulator running on your machine, with the above configuration, you would see the following error in your Semantic Logging Service console.



Installing Windows Azure SDK for .Net 2.1
The SDK is currently on the version 2.1. We can download it from Microsoft Download Center. You can make a selection based on Visual Studio edition you are running on your development machine.



It uses Web PI (Platform Installer) for the installation.



Running Storage Emulator
Running the emulator is easy. We can find it in the start menu. In order to run it from the Windows Azure Command Prompt, we need to use its application name i.e. dsserviceldb.



You should be able to see this running in the System Tray. I have it running on my Windows 7 machine as follows:



Just make sure that, it is running Table Storage service. We need it for Azure Sink as used in the example post.



Now let us run the same application as used in the previous post. The application is generating two ETW events with an Id 1 and 2. As we run it, we should be able to see the SLAB table Development storage. Here we are using the extensions installed in Visual Studio 2012 by Windows Azure SDK.



The extension also supports viewing the storage data. For our example the payload is recorded as follows:



But this seems a bit different than SQL Server sink data. Here it has added additional columns for the payload fields. It is a union of all the fields for all the event types as used by the ETW Event provider defined with the help of EventSource in the source application.

Entlib6 SLAB - Custom IEventTextFormatter Implementation

We have seen how we can use IEventTextFormatter to further format the data from the source sinks using Semantic Application Block. SLAB provides three implementations of the interface and we discussed all three of them. You might be wondering what if we want to format this differently to be consumed by a certain stakeholder. So the main question is if we can introduce and utilize custom implementation of IEventTextFormatter. This post is a continuation of previous post discussing the custom event text formatter.

Let's create a simple class library project and install EnterpriseLibrary.SemanticLogging nuget package to the project. Let's name the project as EventSourceCustomizations. We add the following implementation of IEventTextFormatter to the project.


This is a very simple implementation of the interface. Here the formatter is just formatting the event entry data in CSV format and writes using the TextWriter passed as method argument. This is enough to define a custom event text formatter.

In-Proc Usage of Custom IEventTextFormatter implementation
In order to use the IEventTextFormatter in-process where the ETW events are originating, we need to add the assembly reference of the library to the source project. As discussed before, we can use ObservableEventListener to in-process consume the events. SLAB provides extension methods to the type to log to console which specifies the IEventTextFormatter as one of the parameter. Here we can use our custom implementation of IEventTextFormatter.


Running the above code with our sample ApplicationEventSource shows the data being pushed to the console as follows:



Semantic Logging Service - Custom IEventTextFormatter
We can appreciate the custom IEventTextFormatter more if we are able to use it out-proc. In this case, we need to make no change in the source application. It is possible that during the life time of the application, we realize that we need to format the event data in a different format. We don't need to make a single line of code change to the source application in this case. Semantic Logging Service configuration completely supports this change. We can add the new custom IEventTextFormatter to the SemanticLoggingSvc configuration as follows:


When the application logs the ETW events using the same source, Semantic Logging Service uses the new custom event text formatter and writes the data to the console as follows:



Just make sure that you have copied the assembly containing the custom IEventTextFormatter to the service folder:



Please note that this can be further improved by incorporating the intellisense support. That requires the schema definition for the custom IEventTextFormatter. We also need to provide an implemenation of IFormatterElement for the element. But we should be able to use the type directly instead of through CustomEventTextFormatter. Please refer development guide for Enterprise Library 6 for such an example.

Download

EntLib Semantic Logging App Block - Event Text Formatting

In the discussion around Reactive Event Sourcing , we discussed briefly about text formatting for ETW messages. In this post, we are trying to understand the text formatting in a little more details discussing the different options available for text formatting for event data provided by EventSource as supported by Semantic Logging Application Block [SLAB].

In the examples for this post, we are going to be using Semantic Logging Service. We have discussed about it in the past. You can get yourself started with the installation and configuration of the service here: [Semantic Logging Service]

Sample Event Source
Let's create a sample console application EventSourceFormatting.csproj. We are adding the following EventSource to the project. It has only a single logging method which can be used when there is a request for password hashing.


Message Formatting
EventSource allows us to add a formatted message to the payload. Just look at the definition of RequestForPasswordHashing() method. Here we are decorating the event method with the Message property. You can notice that we are using it like string.Format() where we are expecting the arguments. The arguments are used from the Event method parameters with the same indexing. So for the example here, it would be using req for {0} and userName for {1}. The runtime would format the message using them and assign to event's Message.



The message and payload for the above event should be logged as follows:



Here we have called the event method as follows:


If you are using Semantic Logging Service to test this code with Sql LocalDB, you can use the following service configuration:


IEventTextFormatter
The message formatting as we described above, can format the message adding additional details from the payload. But we might need more than that. We might need to add some text in header and footer of the log being generated if we are using a file or console based sink.

SLAB supports text, JSON and XML based text formatting. The types are available in Microsoft.Practices.EnterpriseLibrary.SemanticLogging.Formatters namespace available in Microsoft.Practices.EnterpriseLibrary.SemanticLogging assembly. The assembly is downloaded using EnterpriseLibrary.SemanticLogging nuget package.



EventTextFormatter
EventTextFormatter is the simplest of all text formatters. It supports qualifying the event's data with additional details including header and footer. These are not the header and footer for the whole log file but each event data would be enveloped in them. Here We can add the following FlatFileSink to our service configuration. It has a simple EventTextFormatter with some specified header and footer details.


The Event Level specified on the sink is the minimum level. If there are error level events generated by one of the event source specified by the sink then they are also logged. You can notice how our event data is decorated with header and footer as specified in the above configuration using the EventTextFormatter by the FlatFileSink.


There is one more amazing option with EventTextFormatter which is missing in the other supported IEventTextFormatters (s). The option is called VerbosityThreshold. This is to further qualify the event for which we desire to apply this formatting. Let's first update our events as follows:


In the following example, we are interested in all the events from the specified EventSource which are at least set as [ Level = Informational]. But we want the additional verbose details only in the case of events with Informational level.


So all the events with level less than Informational wouldn't have the detailed verbose event detail. By default this is set as Error. That is why we were already seeing the detailed event data for Error level event. For our example, the data would be pushed to the sink as follows:


XmlEventTextFormatter
We can use XmlEventTextFormatter without any changes to the source application. Let's see below where we are updating the configuration of Semantic Logging Service. We are using the same Event source, still interested in the same event level. We are even logging to the same file but just in a different format i.e. Xml.


Here is how the same event data is logged in a different format. We don't have the header and footer data as we had for EventTextFormatter.


You can notice that message is still formatted in the same way.

JsonEventTextFormatter
Similar is the case for pushing the event data to the sink in JSON format. In the following Semantic Logging Service configuration, we have used JsonEventTextFormatter with FlatFileSink.

And now the same event data is pushed out to the file in JSON formatter. We don't even need to restart the source application for this change. Just look at the same data in a different format. This time this is JSON.


Why DataTimeFormat Property?
In our EventSourceAnalyzer discussion we mentioned that DateTime data type is not supported for event payload. If you look at the definition of IEventTextFormatter, it has a DateTimeFormatter property. Why in the world do you need to format this when it is not even supported. Just Don't get too excited!!!

The DateTimeFormat is not for the payload data. It is for the event data. If you look at the log file, you can find that all the events are logged with TimeStamp. This formatting is to applied for that Timestamp. Let's update the service configuration as follows:


Just notice the TimeStamp format. You can compare this with the TimeStamp format before.


Source Sinks & Supporting EventTextFormatters
It must be remembered that all sinks don't support every text formatter. There are only a few which support these formatters. In the following image, I have tried to list sinks and the supporting Event Text formatters.



Source Sinks support a maximum of a single text formatter. It can be any of the supported text formatter as presented above.

Changing the Semantic Logging Service Configuration
It must be remembered that we don't need to restart the service after every changes in the service configuration including additional / removal of an event sink. Here we are running the service as a console application. Just look at the detailed message when we changed the configuration. Isn't that great? Zindabad!!!



Download

Event Source Analyzer for Potential Run-time Errors

You might write the best EventSource with all the required methods for logging. But we must remember that your event source would generate manifest and the manifest is used at run-time to write ETW events. This might result in unexpected runtime errors. One wonders how those issues can be found out earlier in the development cycle. Semantic Logging Application Block provides EventSourceAnalyzer for just this purpose. It is used to verify the run-time checks for an EventSource instance.



The type provides Inspection methods. In case of a possible failure during verification, it throws EventSourceAnalyzerException. I think the best usage of this type is to write unit tests to verify such checks for the defined EventSource.



Checks Performed
Now what checks are performed by EventSourceAnalyzer. It performs a 3-point check on EventSource. The tests are as follows:

1. Is there any error when the given EventSource is enabled by an EventListener?
2. Can event schema be requested from the EventSource?
3. Can all Event based methods be invoked in the specified EventSource class?

If the answer of all the above questions is Yes, the unit test is passed!

Let's Start Testing Our EventSource
Here is an method from an EventSource. The compiler generates manifest for ETW events based on the method parameters of Event method. So at run-time calling WriteEvent with a different parameter list than the generated manifest would cause failure. The same is the case with the following EventSource method.


If we use EventSourceAnalyzer for an EventSource containing this method, this should result in the an exception providing the details about the possible issue at runtime.



Generating an ETW event with a non-existent EventId would also result in failure at runtime. Let's see the following method where the Event method is decorated with an EventId which would be used to generate the manifest for the associated event.


Now at runtime it could generate an event with an Id in the manifest just because of an incorrect argument to WriteEvent method. We can determine that earlier in the cycle. The analyzer would result in the following exception in this case.



There is a typo here with the exception message. I have logged a connect item for this. If you really dislike this then you can vote for this.



There are only a handful of types which could be used for ETW events. Someone has specified a unofficial list here as follows: [ Boolean, SByte, Byte, Int16, UInt16, Int32, UInt32, Int64, UInt64, Single, Double, String, Guid]. Since WriteEvent accepts an object array as method argument, the compiler wouldn't have any issue if we pass an instance of an unsupported type. This would only cause failure at runtime. Look at the following method which is using DateTime type.


The Analyzer wouldn't like this and would result an error. Here is the generated exception for the above case:



The analyzer even checks for the order of the parameters in the WriteEvent and the EventSource method. They should be in the same order. In the following we are passing arguments to WriteEvent in a different order than EventSource method.


EventSourceAnalyzer can be a little flexible about this. We can use the following property so that the order is not checked.



Because of the way EventSourceAnalyzer inspects the EventSource, we might need to set ExcludeEventListenerEmulation as well. So that EventListener is not used, otherwise, it would mimic the runtime and would result in failure. The order is important at run-time so we should avoid setting this property to true generally.


The analyzer can also be used to verify that not only the parameters count and order should match, but the types of EventSource method and WriteEvent should also be matching. Here is the exact case in the following:


This would not be checked by EventSourceAnalyzer by default. We can set the EventSourceAnalyzer as follows so that this is also checked.


Since the methods in the EventSource are used to define Event Types for the resulting Event Provider. There should be no non-event methods in the EventSource class. But we do need helper methods sometime. All such helper methods should be clearly marked as NonEventAttribute. Yes, even if they are private.



Writing Unit Tests
As you might have noticed, EventSourceAnalyzer is provided by Semantic Logging Application Block [SLAB] in Enterprise Library 6.0. On the other hand EventSource is provided by .net framework 4.5. SLAB extends the EventSource by providing the amazing ways we can utilize the event data but it is not a requirement to generate ETW event data. We can even have an out-proc event listener which could utilize the generated data and direct it to any sink. The source project doesn't need any reference of the application block.

But we still want to test the logic, we can just install the package to our unit test project. In the unit test we can verify all these checks pretty early. These unit tests can be added to our continuous integration build. Let's install the application block nuget package to our unit test project.



We can use xUnit for our unit tests. As we have discussed before, we can add its support in IDE by installing a Resharper extension. Here we are installing a release version of the package.



Let's add the following unit test to inspect EventSource for runtime checks.


In case a test fails, it shows it as follows:



Download

Enterprise Library 6 is out!

Enterprise Library provides an excellent way to keep the cross-cutting concerns outside your main application code. Microsoft Pattern & Practices team released new version of Enterprise Library last week. This was released on 25th April and this is Version 6 of the library. Full version of the library and source code can be downloaded from Microsoft's Download Center.



The documentation can be downloaded directly form Pattern & Practices team's codeplex page. The documentation is still in preview version and is not updated on MSDN. Still this can be downloaded from the main page. In addition to the Enterprise Library documentation, it also includes the migration guide for migrating applications from Enterprise Library 5. You can also find the reference PDF for Unity Application Block version 3 which is provided as part of the new version of Enterprise Library.



List of Application Blocks in Enterprise Library 6
The library seems to be released with exciting new features in existing application block. It includes a new application block with the name Semantic Logging Application Block. Transient Fault Handling application Block seems to have been added from Azure Integration Pack for Enterprise Library 5. Enterprise Library 6 includes the following application blocks.

1. Data Access Application Block
2. Exception Handling Application Block
3. Logging Application Block
4. Policy Injection Application Block
5. Semantic Logging Application Block
6. Transient Fault Handling Application Block
7. Unity Application Block
8. Validation Application Block

As discussed above, Semantic Application Block is the newest in the list. This is supposed to be built on top of EventSource provided in .net framework 4.5 for structured logging. EventSource provides structured logging support for ETW events. Semantic Application Block is supposed to add support structured logging by specializing EventSource, where the logs can be sinked to Azure or SQL Server database.

There are other application blocks which used to be in Enterprise Library 5 and haven't been migrated over to EntLib6. They include the following:

1. Cryptography Application Block
2. Caching Application Block
3. Validation Application Block

Back in 2010, we did discuss about how we can use Validation Application Block in an MVVM based application.

Unity 3.0
The release also includes a new version of unity application block. This is version 3 of Unity. As we know Unity provides the support of Dependency Injection for .net applications. This is a container based approach to Inversion of Control. Unity 3 extends Dependency Injection Windows store Apps and deeper integration with ASP.net MVC and Web API.

Requirements for Enterprise Library 6
Enterprise Library 6 only works with .net framework 4.5 and .net for Windows Store Apps (for Unity and Transient fault handling application block. It cannot be used side by side with existing Enterprise Library 5 binaries. If you are still on .net framework 4.0, you will need to make a decision about migrating to .net framework 4.5 first before moving to the new version of the enterprise library.

Using Enterprise Library Application Blocks
You can download the list of the binaries from the link provided in download section to use enterprise library's application blocks in your code. Pattern & Practices team suggests using Nuget for this purpose. For ease of developers, Pattern & Practices team has provided these application blocks in the form of Nuget packages. These Nuget packages can be used individually as per the requirement. Just add the appropriate package to your project using Nuget



Still To Come
This is the view of the release from top. There are other minor improvements and updates which require more detailed discussions. I am planning a few posts about the updates in Enterprise Library 6. You might find me blogging about them next month. You can find release notes here.