EntLib6 SLAB - Events Versioning

Semantic Logging Application Block's main focus is event consumption. It is based on EventSource introduced in .net framework 4.5. It allows these events to be consumed In-Proc and Out-Proc. But no system is free from any changes. Logging should also accommodate system changes. EventSource allows its events to be tagged with version numbers. This is part of metadata defined for the [Event] methods.

Let's assume we deploy the example application as developed in previous example. After two months the client calls and asks if we could add an additional fields to Students. The field would be used to record Age of student. Since student creation is being logged, we would need to log this information as well. In this post we will be trying to accommodate this change.


It is also possible that we might need to add additional methods to the EventSource due to additional logging needs. We need to make sure that we are assigning them appropriate keywords and levels as expected by the downstream consumption system. As discussed before, they can also be outside the system.

Versioning & EventSource
We must remember that these versions cannot co-exist in an overloaded fashion in an EventSource as in the following code. As we have discussed before, these methods map to the Event Types for ETW infrastructure, it would generate a duplicate key, which is not allowed.


Here we have just overloaded StudentCreatedCore() method decorated with EventAttribute. It has an additional parameter studentName. Here we have kept the Id as same for the Event but included Version with some value assigned. We have also added the similar property in the other overloaded method. In the public method, we are deciding which method to use based on certain condition. The world is really a happy place to live...

Let's see how our unit test behave in this case. Hmm, it fails with the details about possible issue at runtime because of more that one events with the same name.



Let me tell you that the code compiles just fine in this case. This is to save us from experiencing an exception at runtime and then cursing ourselves for missing the obvious.



Let's update the definition of the EventSource removing the earlier version of the [Event] method.


Versioning & Events Consumption
Since Events are generated to be consumed In-Proc or Out-Proc systems. The events detail can be used by automation system to generate reports and notifications for the infrastructure and management teams responsible for the health and monitoring for the system. They expect a certain payload and message format. Since they can be persisted, the data for historic and current payload structure would co-exist in the persistence storage. They need some way to decide what format to use based on some data. Version is the best candidate for this. The reports can use versioning to decide which payload format should be used by the automation system. We must remember that we cannot enable / disable the events based on versions or even [Event].

Let's update the configuration of Semantic Logging Service for the update. Here we are adding an Azure Sink. The sink is enabled for the EventSource defined above for all the events with a maximum level set as Informational.


As we generate these events in the source system. They are provided to the Semantic Logging Service by the ETW infrastructure. TracingService hooks up to the ETW infrastructure for these events. As the events are received they are passed to the avaiable sinks. These sinks are utilized based on their configuration of the events source and keywords they are interested in. For our events, they are pushed to the azure development table storage. Just notice the payload for the new version and old version. In this way, they can be easily identified and operated on.



Download

Enterprise Library 6 - SLAB - Supporting Complex Types for Payload

Previously we have discussed there are only certain primitive types supported by EventSource introduced in .net framework 4.5. This post is an attempt how we can work around this limitation. Actually, we can decorate EventSource methods with [Event] and [NonEvent] attributes. The other most important point is the following:

[Event] methods can be private.

We can use this to only include private [Event] methods which would be used to generate ETW manifest from the EventSource. Additionally we can introduce public methods in the EventSource type supporting the non-primitive custom type as parameters. Hence these object-oriented types make writing the logs easier without putting any load on the underlying infrastructure supporting the event mechanism.

Let's create a sample C# project targeting .net framework 4.5. We add a type EventSourceWithComplexType to the project. As you can see below this is an EventSource. Here we are using Lazy initialization introduced in .net framework 4.0 for singleton implementation [Please refer here for singleton implementation]. It has a private constructor to make sure that it cannot be instantiated from outside. The most noticeable thing is its two methods. Here we have a method StudentCreatedCore() decorated with [Event] attribute, has a list of parameters including types supported by the ETW infrastructure. It has a corresponding StudentCreated() method with [NonEvent] attribute. It has a public access modified and has a non-primitive type for parameter. The most important thing is that it is using the StudentCreatedCore() method.


Can The implementation satisfy EventSourceAnalyzer?
We have discussed that we can use EventSourceAnalyzer for testing our EventSource implementation [Discussion]. Let's see if we can satisfy EventSourceAnalyzer for the above implementation. We just need to call InspectAll() method with an instance of the EventSource. Here we are using xUnit for testing supported by resharper. You need to install the nuget package. You would also need to get EnterpriseLibrary.SemanticLogging package for using EventSourceAnalyzer.


As a matter of fact, EventSourceAnalyzer has no complaints and the unit test passes. Zindabad!



Listening for Events
Now let's see if we can listen for events using this EventSource implementation. Here we are attaching the EventSource with ConsoleSink using the ConsoleLog [Discussion]. We are using the same EventLister to enable the events. Here we are just using the public method on EventSource type with an instance of Student type. Let's see if we could see the event's data on the console [keep the spirit high, friend!!!].


And we do see that event's data is being printed on the console in the same format as we provided in the StudentCreatedCore() method. Hence the runtime was able to do it from the private method. It has no problem with generating the ETW manifest for private [Event] methods and using it at run-time. Zindabad!



Download


Love for other people what you love for yourself.    [Prophet Muhammad - Salallahu-Alaihe-Wasallam]

Semantic Logging Application Block In-Proc Usage

Semantic Logging Application Block provides various types of sinks. They include Console, File based, SQL Server and Windows Azure Table based sinks. We have been discussing how we can use these sinks out-proc. In this post we will be trying to focus how we can use these sinks in-proc i.e. in the same process where the ETW events are generated from. This is specially useful for scenarios where it is not feasible for running Semantic Logging Service as a separate process. I think the most common use would be desktop applications.

Let's create a sample console application SLABInProcApp targeting .net framework 4.5.



Since we would be using the sinks in process. We need to use the necessary nuget packages. As we know FlatFile, RollingFlatFile and Console sinks are directly available in EnterpriseLibrary.SemanticLogging package. The SQL Server and Windows Azure Table sinks are available in EnterpriseLibrary.SemanticLogging.Database and EnterpriseLibrary.SemanticLogging.WindowsAzure packages respectively. Since these packages have a dependency on EnterpriseLibrary.SemanticLogging package, installing these two packages should take care of all the necessary dependencies. These are the packages and their dependencies as shows in Package Visualizer.



And this is how the packages.config should look like for the project.

Now let us introduce our custom EventSource we have been using in these posts. As we have been seeing, it must inherit EventSource type available in System.Diagnostic.Tracing namespace. We can also introduce the necessary types for keywords, tasks. Here we have two logging methods with Event Ids set as 1 and 2.


IObservable<EventEntry> Implementation
SLAB provides ObservableEventListener. This is a specialized EventListener available (.net framework 4.5). As its name suggests, this is an observable type implementing IObservable<EventEntry>. Now SLAB provides various extension method to hook this observable type to the available sinks. In order to keep the world simple, for a given sink type X, the available extension is named as LogToX().



Here we seem to be interested in pushing the logs to Console, SQL Server and Windows Azure table. So we would be needing ConsoleSink, SqlDatabaseSink and WindowsAzureTableSink. These are provided through ConsoleLog, SqlDatabaseLog and WindowsAzureTableLog types respectively through their corresponding LogToX() methods. The extension methods for logging to SQL Server and Windows Azure Table are made available through installing their respective nuget packages.


There is some pre-requisites before we can push data to SqlServer and Windows Azure table here for this example. We need to create database (including the required objects). We also need to make sure that Storage Emulator is running on the machine running this application. This is only required for development. For the production application we can provide the necessary connection details. We have discussed the pre-requisites for SQL Server and Windows Azure table here.

And That's it! Ain't that simple!!!

Now we can just run the application. Since we have subscribed to the ConsoleSink through ConsoleLog.LogToConsole() extension method for IObservable<EventEntry> type, we should be seeing the following on the console.



Subscribing to WindowsAzureTableSink should push the following in the development table storage.



And finally we should be able to see the following data in Traces table in the SQL Server database used here:



Simplifying it Further
We can simplify the code even further. We don't need to instantiate ObservableEventListener ourselves. XLog types also provide the necessary CreateListener() methods. In addition to returning a new instance of ObservableEventListener, they also hook up the EventListener instance returned to the respective source sink.


Event Listeners and Application Life Cycles
Please note that this is a very simple example where you see these subscriptions and logging in the same methods. This would not be the same in the real-life application. You can consider this like Trace Listeners, they are added in the beginning of application life cycle. In the same way, EventListener subscriptions can also be done at the similar stage. They would be more suitable for application seams (Mark Seemann).

There are still some limitations about implementing interface type by the EventSource.

Subscribing Custom IObserver<EventEntry>
Since ObservableEventListener implements IObservable<EventEntry>, we can subscribe any IObserver<EventEntry> with this. Let's implement a sample IObserver<EventEntry> implementation.

We can use the CustomObserver to subscribe to the ObservableEventListener now. This should be using OnNext() method for any new event from the listener.


Download Code

Event Source & Event Provider Mapping

We have been discussing that EventSource is used to generate ETW event provider. I thought it would be good to discuss how it maps out to ETW world. In this brief post, we are going to discuss just that.

As we have discussed compiler generates ETW manifest based on the definition of our EventSource. This is a type which inherits from EventSoure class from System.Diagnostics.Tracing namespace in mscorlib assembly. There is a one-to-one correspondence between the EventSource in our code and resulting EventProvider. Let's see how the elements in EventSource customized type maps out to ETW event provider.



Naming Conventions & Battle b/w Dev & IT
Defining your own Event Source can get you in battle against naming conventions in your code and ETW. Since ETW infrastructure is not maintained by development team, they might be following some standard that they want you to adhere to. This is specially true if your ETW events are destined for Windows Event Log. Here we have discussed how we can target our EventSource based events to Windows Event Log.

In order to make it easier for you Microsoft has introduced EventSource allows specifying a friendly name of the EventSource using EventSourceAttribute. This is a class-level attribute. The compiler uses this name to create ETW provider for this EventSource. Now your system management team might require you to generate events in a folder structure based on their policy. In order to support that we can use dashes with EventSourceAttribute.Name. In the following example we would define the attribute as [EventSource(Name = "Samples-EventSourceDemo-EventLog")]. By default, it uses the class name for the EventProvider name. For the unique identifier for the EventProvider, EventSourceAttribute provides Guid property.



References

• http://technet.microsoft.com/en-us/library/jj714799.aspx

Event Source Analyzer for Potential Run-time Errors

You might write the best EventSource with all the required methods for logging. But we must remember that your event source would generate manifest and the manifest is used at run-time to write ETW events. This might result in unexpected runtime errors. One wonders how those issues can be found out earlier in the development cycle. Semantic Logging Application Block provides EventSourceAnalyzer for just this purpose. It is used to verify the run-time checks for an EventSource instance.



The type provides Inspection methods. In case of a possible failure during verification, it throws EventSourceAnalyzerException. I think the best usage of this type is to write unit tests to verify such checks for the defined EventSource.



Checks Performed
Now what checks are performed by EventSourceAnalyzer. It performs a 3-point check on EventSource. The tests are as follows:

1. Is there any error when the given EventSource is enabled by an EventListener?
2. Can event schema be requested from the EventSource?
3. Can all Event based methods be invoked in the specified EventSource class?

If the answer of all the above questions is Yes, the unit test is passed!

Let's Start Testing Our EventSource
Here is an method from an EventSource. The compiler generates manifest for ETW events based on the method parameters of Event method. So at run-time calling WriteEvent with a different parameter list than the generated manifest would cause failure. The same is the case with the following EventSource method.


If we use EventSourceAnalyzer for an EventSource containing this method, this should result in the an exception providing the details about the possible issue at runtime.



Generating an ETW event with a non-existent EventId would also result in failure at runtime. Let's see the following method where the Event method is decorated with an EventId which would be used to generate the manifest for the associated event.


Now at runtime it could generate an event with an Id in the manifest just because of an incorrect argument to WriteEvent method. We can determine that earlier in the cycle. The analyzer would result in the following exception in this case.



There is a typo here with the exception message. I have logged a connect item for this. If you really dislike this then you can vote for this.



There are only a handful of types which could be used for ETW events. Someone has specified a unofficial list here as follows: [ Boolean, SByte, Byte, Int16, UInt16, Int32, UInt32, Int64, UInt64, Single, Double, String, Guid]. Since WriteEvent accepts an object array as method argument, the compiler wouldn't have any issue if we pass an instance of an unsupported type. This would only cause failure at runtime. Look at the following method which is using DateTime type.


The Analyzer wouldn't like this and would result an error. Here is the generated exception for the above case:



The analyzer even checks for the order of the parameters in the WriteEvent and the EventSource method. They should be in the same order. In the following we are passing arguments to WriteEvent in a different order than EventSource method.


EventSourceAnalyzer can be a little flexible about this. We can use the following property so that the order is not checked.



Because of the way EventSourceAnalyzer inspects the EventSource, we might need to set ExcludeEventListenerEmulation as well. So that EventListener is not used, otherwise, it would mimic the runtime and would result in failure. The order is important at run-time so we should avoid setting this property to true generally.


The analyzer can also be used to verify that not only the parameters count and order should match, but the types of EventSource method and WriteEvent should also be matching. Here is the exact case in the following:


This would not be checked by EventSourceAnalyzer by default. We can set the EventSourceAnalyzer as follows so that this is also checked.


Since the methods in the EventSource are used to define Event Types for the resulting Event Provider. There should be no non-event methods in the EventSource class. But we do need helper methods sometime. All such helper methods should be clearly marked as NonEventAttribute. Yes, even if they are private.



Writing Unit Tests
As you might have noticed, EventSourceAnalyzer is provided by Semantic Logging Application Block [SLAB] in Enterprise Library 6.0. On the other hand EventSource is provided by .net framework 4.5. SLAB extends the EventSource by providing the amazing ways we can utilize the event data but it is not a requirement to generate ETW event data. We can even have an out-proc event listener which could utilize the generated data and direct it to any sink. The source project doesn't need any reference of the application block.

But we still want to test the logic, we can just install the package to our unit test project. In the unit test we can verify all these checks pretty early. These unit tests can be added to our continuous integration build. Let's install the application block nuget package to our unit test project.



We can use xUnit for our unit tests. As we have discussed before, we can add its support in IDE by installing a Resharper extension. Here we are installing a release version of the package.



Let's add the following unit test to inspect EventSource for runtime checks.


In case a test fails, it shows it as follows:



Download

EntLib6 Semantic Logging with SQL Server Sink

In the last couple of months we started our discussion about EventSource and Semantic Logging Application Block. You can refer to the previous posts here:

• EventSource & PerfView
• Semantic Out-Process Logging using Semantic Logging Application Block - File Sink
• SLAB - Reactive Event Sourcing
• To Log or Not to Log...Really???
• .net 4.5 EventSource & Windows Event Log - Microsoft's Nuget Package
• EventSource and Implenting Interfaces

In the above posts, we discussed how to create an Event Source and why do we actually need structured logging. We presented some examples of consuming the events generated by EventSource using tools including PerfView and SLAB's Semantic Logging Service. Event Sinks and listeners allow us to consume those events and pushed them out externally, including file system and Windows Event log. This post is about taking the discussion a little further to see how we can push out the events generated using an EventSource to SQL Server Database.

System Requirements
SLAB supports SQL Server 2008 and higher, SQL Server LocalDB and Windows Azure SQL Database. We will be using SQL Server Local DB for this example.

Sample Event Source
Let's create a sample console application SLABSqlEventSinkEx. We add the following EventSource to the project.


Let's try to verify if our EventSource has no issues by using PerfView. We discussed about using PerfView for ETW events here.


And we do see the event being generated. Please notice the selection below:



SLAB provides a number of Event sinks. Not all of them are in the same assembly.



Pattern & Practices team has distributed them among different Nuget packages. If you want to utilize the events generated by an event source and push them out to a SQL Server database then you would need to use the following package:



After packages installation, this is how my packages.config looks like.


Creating Database
The nuget package provides the necessary script to create database schema. We can find the scripts under ...\packages\EnterpriseLibrary.SemanticLogging.Database.1.0.1304.0\scripts folder. Just run the cmd file in the folder. It should run both sql files passing the required parameters.



The script creates a new database Logging with a single table, stored procedure and a couple of user defined types. It assumes installation of SqlLocalDB installation. Here we have connected with the database using Visual Studio SQL Server Object Explorer in Visual Studio 2012.



We can use SqlLocalDB command line tool to check the state of sql server instance used.



If you are using out-proc Semantic Logging Service to consume the events then you really don't need to download the package. The script and command file is copied into the service's folder when you install the nuget package. You can find them here:



Using SQL Server Sink with Semantic Logging Service
As we have discussed before SLAB allows us for out-proc logging using Semantic Logging Service. The service can run as a console application or windows service. You can see the this post where we introduced the service and its use with file system sink for event sources. Let's update the service configuration [SemanticLogging-svc.xml] to push the events from SqlApplicationEventSource to SQL Server database created above.


After updating the configuration, we can run the service as a console application as follows:



Run the application writing ETW Events. As the events are pushed out, the service writes the details to the specified database. Here is the same data as we saw in PerfView earlier in this post.



Using SLAB's In-Proc SQL Server Sink
We can also enable the events In-Proc. SqlDatabaselog can be used to create the listener to be used to enable the events. We can add the following code to the Main method of our console application created above.


But I have noticed an issue logging the data in SQL Server. I have discussed the issue here: [Discussion about Keyword issue]. The issue is based on using Keyword with the event method, which might seems related.

Download

Enterprise Library 6 is out!

Enterprise Library provides an excellent way to keep the cross-cutting concerns outside your main application code. Microsoft Pattern & Practices team released new version of Enterprise Library last week. This was released on 25th April and this is Version 6 of the library. Full version of the library and source code can be downloaded from Microsoft's Download Center.



The documentation can be downloaded directly form Pattern & Practices team's codeplex page. The documentation is still in preview version and is not updated on MSDN. Still this can be downloaded from the main page. In addition to the Enterprise Library documentation, it also includes the migration guide for migrating applications from Enterprise Library 5. You can also find the reference PDF for Unity Application Block version 3 which is provided as part of the new version of Enterprise Library.



List of Application Blocks in Enterprise Library 6
The library seems to be released with exciting new features in existing application block. It includes a new application block with the name Semantic Logging Application Block. Transient Fault Handling application Block seems to have been added from Azure Integration Pack for Enterprise Library 5. Enterprise Library 6 includes the following application blocks.

1. Data Access Application Block
2. Exception Handling Application Block
3. Logging Application Block
4. Policy Injection Application Block
5. Semantic Logging Application Block
6. Transient Fault Handling Application Block
7. Unity Application Block
8. Validation Application Block

As discussed above, Semantic Application Block is the newest in the list. This is supposed to be built on top of EventSource provided in .net framework 4.5 for structured logging. EventSource provides structured logging support for ETW events. Semantic Application Block is supposed to add support structured logging by specializing EventSource, where the logs can be sinked to Azure or SQL Server database.

There are other application blocks which used to be in Enterprise Library 5 and haven't been migrated over to EntLib6. They include the following:

1. Cryptography Application Block
2. Caching Application Block
3. Validation Application Block

Back in 2010, we did discuss about how we can use Validation Application Block in an MVVM based application.

Unity 3.0
The release also includes a new version of unity application block. This is version 3 of Unity. As we know Unity provides the support of Dependency Injection for .net applications. This is a container based approach to Inversion of Control. Unity 3 extends Dependency Injection Windows store Apps and deeper integration with ASP.net MVC and Web API.

Requirements for Enterprise Library 6
Enterprise Library 6 only works with .net framework 4.5 and .net for Windows Store Apps (for Unity and Transient fault handling application block. It cannot be used side by side with existing Enterprise Library 5 binaries. If you are still on .net framework 4.0, you will need to make a decision about migrating to .net framework 4.5 first before moving to the new version of the enterprise library.

Using Enterprise Library Application Blocks
You can download the list of the binaries from the link provided in download section to use enterprise library's application blocks in your code. Pattern & Practices team suggests using Nuget for this purpose. For ease of developers, Pattern & Practices team has provided these application blocks in the form of Nuget packages. These Nuget packages can be used individually as per the requirement. Just add the appropriate package to your project using Nuget



Still To Come
This is the view of the release from top. There are other minor improvements and updates which require more detailed discussions. I am planning a few posts about the updates in Enterprise Library 6. You might find me blogging about them next month. You can find release notes here.