Amazon S3 Sink for Semantic Logging Service – Enterprise Library 6

Event Source API was introduced in .net framework 4.5 to support semantic logs. It allows an application to author events. These events are generated as ETW events which can be consumed in-proc or out-proc. For a general in-proc consumption, the API has introduced EventSourceListener type.

Semantic Logging Application Block (SLAB) has developed on top of EventSource API with more focus on event’s consumption. It has introduced sinks for consumption which are utilized by ETW event listener in the application block. These events are destination specific. The Pattern & Practices (p&p) team...

October 2013, What a month this has been...

October is about to pass. But what an amazing months it has been...

The greatest skill a developer has to have is to adjust within a group of people. Believe me, this is even difficult than developing a technical skill. Interpersonal communication is a craft and one needs to master it in order to better survive for selling the ideas. Since we are in the business of consulting for software services, we need to move between organizations every few months and years. Not all the people are welcoming, they have different psychologies and motivations. These events prove to be a practice. You need to go to the venue, introduce yourself, present in the presence of total strangers - you have no idea about their backgrounds.

Few Advices & Lessons Learned
Humor is an interpersonal lubricant. Since we are interacting with the audience for the first time, we need to break the barrier of being strange. Humor is the universal lubrication of friction between people. You are never strange for the people who smile with you. Although this is the most effective technique, but this can go either way. You need to pull off the joke really well. Like an actor, the punch line has to be effective. Practice it, tell it a few times to your friends no matter how annoying they become.

One must go through the session material before the session. I have realized that we try to be very detail oriented when are creating slides for our presentation. These conferences and code camps are not class room sessions. People are interested in links and ideas they can take home. This give them direction for their technical development. We need to give them as much as possible in the limited time we have. But the most important thing is to motivate them enough about the stuff that they study further after the event. We need to be fresh for the event, so take a nap before the event.

We can also record the session if possible. This is one of the take away item for yourself. A recorded session can be useful for preparing yourself if you are speaking about the same stuff later on. This is specially useful if you have been away focusing on some other technological stuff for a few months. You can also publish them for general audience if you think this might help them learning about the topic.

We must time box our session. A session has a list of agenda items. We need to make sure that we time box the agenda so that we are able to cover each item and still be able to finish our session in time. If there is an item which is taking more time, tell about some pointers the audience might take away with them. We must also be ourselves, confident and to the point. Call your family or friend for a little chit chat in the morning.

Session Slides
I have presented my thoughts and ideas over two areas from a huge list of Microsoft technology stack. They are as follows:

1. Cross Platform Development using Portable Class Libraries
2. Event Source API, ETW and Semantic Logging Application Block

I have uploaded the session slides on my sky drive. You can view it here or download it if you want to view them offline.



Session Videos
Here are the session videos from the events in Silicon Valley, Boston and Southwest Florida Code camps.

semanticLogs from Muhammad Siddiqi on Vimeo.

Portable Class Libraries - Boston Codecamp 20 from Muhammad Siddiqi on Vimeo.

slab from Muhammad Siddiqi on Vimeo.

Cross Platform Development with Portable Class Libraries from Muhammad Siddiqi on Vimeo.

Download Code
You can also download the code from these sessions. I even uploaded that to my sky drive.

EntLib6 - EventSource & Keywords for Semantic Logs

In this post we are going to discuss two very common details for defining event source. They are EventLevel and EventKeyword. They are so central for defining and enabling events. Yet, it is very easy to make mistake in using or expecting a particular behavior with the values used for them. The value assigned to them might include other values as well, which might come as a surprise for others.

Event Level
As we discussed, the event level selected for enabling a listener is the maximum value of the level which would be forwarded to the subscribed sinks. All events assigned with levels equal or lesser than the assigned enumeration values are considered enabled.

I have really felt that intellisence is not really useful in this case. The levels are ordered in the ascending order of their names. It could have been really useful if they could be available in the order of their values [but there are some things in life you cannot get, hence the void... :)]



Let us try to make it a little easier. We can come up with an acronym like from top to bottom, it is VIWECL. In order to remember this, we can come up with some catchy phrase like Very Intelligent Wife so Extremely Cool Life. Here whatever level we specify to enable the event source, all the lower levels are automatically included. I think we should not forget it now. When someone special is involved, it is better to try not to forget...



Keywords As Power of 2
It must be remembered that keywords must be assigned values in power of 2. It makes it easier to enable and disable the events. Here is an inner class Keywords defined in an EventSource. You can see that the values are 1, 2, 4, 8. They are powers of 0, 1, 2, 3 respectively.


In order to understand how we can use these keywords, we need to understand the bit mask for these keywords. We have the bit masks for Creation, AdmissionApproved, FeeDeposited and TransferCreditHours.



Definition of EventSource
Let us introduce an EventSource. The event source has four [Event] based methods. The methods are using the values of keywords defined above. They are also assigned with particular


Special Event Level & Keyword
EventLevel.LogAlways has special meaning when we use it to enable the events. It means to ignore levels when filtering is being applied i.e. all levels are considered. Since this is a perfectly fine EventLevel value which could be assigned to an Event method, it might get confusing when you would just want to enable the events assigned to this level, as it has a specific mean in this scenario. I think we should better avoid it when defining our events. I still have this in my EventSource just to give you an example.

Similar is the case for Keywords.All. It means to include all keywords. Basically, when we don't specify an Keyword to enable a scenario, all those events which are assigned with a specific value of keywords are just ignored. If we want to consider all of them instead, we can use this value for EventKeyword when enabling events. It has a value -1. We can use this value for enabling all keywords with xml configuration in the case of Semantic Logging Service configuration.

Configuration for In-Proc usage
As we have discussed before we can use SLAB both In-Proc and Out-Proc scenarios. For the In-Proc scenarios, we can enable the events using ObservableEventListener specifying the details of event's levels and keywords we are specially interested in. Let's first consider the case of specials for level and keyword. Here we are using EventLevel.Always and Keyword.All, which means to say, I want to log everything.


And it does have all the logs, no matter what Level or Keywords they are assigned with. Here is the output for the above code:



EventLevel.LogAlways with other keywords
Let's see the following combination. Here we are using EventLevel.LogAlways with two keywords.


If you go back and look at the definition of our EventSource, we don't have this combination of Level and Keywords for any event. There is only one event with Level assigned as EventLevel.LogAlways, but that has different keyword assigned.

So does it mean, it should include none of the events? Acutally No!!!

As we have discussed before, it has a special meaning the case when we are enabling event. It means to ignore whatever value of level assigned to events. In other words, just use the keywords specified for enabling the events. So it should just enable the two events, defined with the specified keywords. If we run this code, it shows the following output:



Level is the Max Level for Events
As we have discussed in the stairs diagram for Events. The event level assigned is the maximum value for the EventLevel considered for logging. In the following example, we are using EventLevel.Error for enabling the event. If we look at the diagram, we should realize, it would automatically include EventLevel.Critical and EventLevel.LogAlways. Let's change the configuration as follows:


As expected, it includes events assigned with levels as EventLevel.Error, EventLevel.Critical and EventLevel.LogAlways. Here is the output:



-matchAnyKeyword for Out-Proc Configuration
We can use a similar configuration for Out-Proc scenario. In the Semantic Logging service provided by Semantic Logging Application Block, matchAnyKeyword attribute is provided for source configuration. The value assigned to this keyword is treated differently. It is the sum of all keywords we want to enable. Here we want to enable Keyword.Creation and Keyword.AdmissionApproved. We need to just add the two numbers to calculate the value which should be assigned to the attribute. We can also use bit calculation by masking the two keyword bits. We can then convert the value in decimal as follows:



Let's use this in the configuration for the service. Here we are interested in the events with maximum level assigned as EventLevel.Informational. We are specially interested in the two keywords specified above.


Since we are using Windows Azure Sink with Storage Simulator, we should be able to see the following events logged in SLAB table.



Download

Semantic Logging Application Block In-Proc Usage

Semantic Logging Application Block provides various types of sinks. They include Console, File based, SQL Server and Windows Azure Table based sinks. We have been discussing how we can use these sinks out-proc. In this post we will be trying to focus how we can use these sinks in-proc i.e. in the same process where the ETW events are generated from. This is specially useful for scenarios where it is not feasible for running Semantic Logging Service as a separate process. I think the most common use would be desktop applications.

Let's create a sample console application SLABInProcApp targeting .net framework 4.5.



Since we would be using the sinks in process. We need to use the necessary nuget packages. As we know FlatFile, RollingFlatFile and Console sinks are directly available in EnterpriseLibrary.SemanticLogging package. The SQL Server and Windows Azure Table sinks are available in EnterpriseLibrary.SemanticLogging.Database and EnterpriseLibrary.SemanticLogging.WindowsAzure packages respectively. Since these packages have a dependency on EnterpriseLibrary.SemanticLogging package, installing these two packages should take care of all the necessary dependencies. These are the packages and their dependencies as shows in Package Visualizer.



And this is how the packages.config should look like for the project.

Now let us introduce our custom EventSource we have been using in these posts. As we have been seeing, it must inherit EventSource type available in System.Diagnostic.Tracing namespace. We can also introduce the necessary types for keywords, tasks. Here we have two logging methods with Event Ids set as 1 and 2.


IObservable<EventEntry> Implementation
SLAB provides ObservableEventListener. This is a specialized EventListener available (.net framework 4.5). As its name suggests, this is an observable type implementing IObservable<EventEntry>. Now SLAB provides various extension method to hook this observable type to the available sinks. In order to keep the world simple, for a given sink type X, the available extension is named as LogToX().



Here we seem to be interested in pushing the logs to Console, SQL Server and Windows Azure table. So we would be needing ConsoleSink, SqlDatabaseSink and WindowsAzureTableSink. These are provided through ConsoleLog, SqlDatabaseLog and WindowsAzureTableLog types respectively through their corresponding LogToX() methods. The extension methods for logging to SQL Server and Windows Azure Table are made available through installing their respective nuget packages.


There is some pre-requisites before we can push data to SqlServer and Windows Azure table here for this example. We need to create database (including the required objects). We also need to make sure that Storage Emulator is running on the machine running this application. This is only required for development. For the production application we can provide the necessary connection details. We have discussed the pre-requisites for SQL Server and Windows Azure table here.

And That's it! Ain't that simple!!!

Now we can just run the application. Since we have subscribed to the ConsoleSink through ConsoleLog.LogToConsole() extension method for IObservable<EventEntry> type, we should be seeing the following on the console.



Subscribing to WindowsAzureTableSink should push the following in the development table storage.



And finally we should be able to see the following data in Traces table in the SQL Server database used here:



Simplifying it Further
We can simplify the code even further. We don't need to instantiate ObservableEventListener ourselves. XLog types also provide the necessary CreateListener() methods. In addition to returning a new instance of ObservableEventListener, they also hook up the EventListener instance returned to the respective source sink.


Event Listeners and Application Life Cycles
Please note that this is a very simple example where you see these subscriptions and logging in the same methods. This would not be the same in the real-life application. You can consider this like Trace Listeners, they are added in the beginning of application life cycle. In the same way, EventListener subscriptions can also be done at the similar stage. They would be more suitable for application seams (Mark Seemann).

There are still some limitations about implementing interface type by the EventSource.

Subscribing Custom IObserver<EventEntry>
Since ObservableEventListener implements IObservable<EventEntry>, we can subscribe any IObserver<EventEntry> with this. Let's implement a sample IObserver<EventEntry> implementation.

We can use the CustomObserver to subscribe to the ObservableEventListener now. This should be using OnNext() method for any new event from the listener.


Download Code

EntLib6 - Semantic Logging using Windows Azure Sink - Table Storage

This post is part of series of posts discussing about Semantic Logging Application Block [SLAB]. Earlier we have discussed about how we can push the log data to file and SQL Server storage. In this post we are discussing how we can use Windows Azure Table Storage as a destination of these logs.

Windows Azure Sink Configuration
As we have discussed updating destination and format of the event data requires no changes in the application generating these ETW events if we are using out-proc consumption of these events. SLAB provides Semantic Logging Service. We can use it out-of-the-box to make all these necessary changes without event bringing the application down. The configuration is done in SemanticLogging-Svc.xml configuration file for the service. Here we are using Windows Azure Storage Emulator.


Storage Emulator Not Running / Installed
For our development machine, we first need to install Windows Azure SDK to run the above. We specially need Windows Azure Storage Emulator running for this example. The emulator uses a default account key to test the application logic. In absence of an emulator running on your machine, with the above configuration, you would see the following error in your Semantic Logging Service console.



Installing Windows Azure SDK for .Net 2.1
The SDK is currently on the version 2.1. We can download it from Microsoft Download Center. You can make a selection based on Visual Studio edition you are running on your development machine.



It uses Web PI (Platform Installer) for the installation.



Running Storage Emulator
Running the emulator is easy. We can find it in the start menu. In order to run it from the Windows Azure Command Prompt, we need to use its application name i.e. dsserviceldb.



You should be able to see this running in the System Tray. I have it running on my Windows 7 machine as follows:



Just make sure that, it is running Table Storage service. We need it for Azure Sink as used in the example post.



Now let us run the same application as used in the previous post. The application is generating two ETW events with an Id 1 and 2. As we run it, we should be able to see the SLAB table Development storage. Here we are using the extensions installed in Visual Studio 2012 by Windows Azure SDK.



The extension also supports viewing the storage data. For our example the payload is recorded as follows:



But this seems a bit different than SQL Server sink data. Here it has added additional columns for the payload fields. It is a union of all the fields for all the event types as used by the ETW Event provider defined with the help of EventSource in the source application.

Entlib6 SLAB - Custom IEventTextFormatter Implementation

We have seen how we can use IEventTextFormatter to further format the data from the source sinks using Semantic Application Block. SLAB provides three implementations of the interface and we discussed all three of them. You might be wondering what if we want to format this differently to be consumed by a certain stakeholder. So the main question is if we can introduce and utilize custom implementation of IEventTextFormatter. This post is a continuation of previous post discussing the custom event text formatter.

Let's create a simple class library project and install EnterpriseLibrary.SemanticLogging nuget package to the project. Let's name the project as EventSourceCustomizations. We add the following implementation of IEventTextFormatter to the project.


This is a very simple implementation of the interface. Here the formatter is just formatting the event entry data in CSV format and writes using the TextWriter passed as method argument. This is enough to define a custom event text formatter.

In-Proc Usage of Custom IEventTextFormatter implementation
In order to use the IEventTextFormatter in-process where the ETW events are originating, we need to add the assembly reference of the library to the source project. As discussed before, we can use ObservableEventListener to in-process consume the events. SLAB provides extension methods to the type to log to console which specifies the IEventTextFormatter as one of the parameter. Here we can use our custom implementation of IEventTextFormatter.


Running the above code with our sample ApplicationEventSource shows the data being pushed to the console as follows:



Semantic Logging Service - Custom IEventTextFormatter
We can appreciate the custom IEventTextFormatter more if we are able to use it out-proc. In this case, we need to make no change in the source application. It is possible that during the life time of the application, we realize that we need to format the event data in a different format. We don't need to make a single line of code change to the source application in this case. Semantic Logging Service configuration completely supports this change. We can add the new custom IEventTextFormatter to the SemanticLoggingSvc configuration as follows:


When the application logs the ETW events using the same source, Semantic Logging Service uses the new custom event text formatter and writes the data to the console as follows:



Just make sure that you have copied the assembly containing the custom IEventTextFormatter to the service folder:



Please note that this can be further improved by incorporating the intellisense support. That requires the schema definition for the custom IEventTextFormatter. We also need to provide an implemenation of IFormatterElement for the element. But we should be able to use the type directly instead of through CustomEventTextFormatter. Please refer development guide for Enterprise Library 6 for such an example.

Download