Having fun coding a DirectSmile Cross Media Data Relations

The Crossmedia version 5 comes with many new features and today I picked one that totally thrills me. Actually, I think that it’s way too cool to name it Data Relations, what is rather obfuscating the real power of this new feature. To show you why I figure Crossmedia Data Relations really rock, I want to share with you an example I made. I will spread the example over two posts, and this post here is the first one that demonstrates the implementation of a Data Relation. The second post will cover how to access a Data Relation from inside a Crossmedia campaign. But first, let me explain what a Data Relation is.

Crossmedia Data Relations

A Data Relation is a relation between the campaign database, you add to your cross media template, and other table(s) from a different data source. Such a data source can be almost everything you have in mind, for example a third party database or a web service. The data relation is the logic that glues tables from those two data sources together.

A Data Relation is a .Net assembly that comes with a pluggable interface based on the Managed Extensibility Framework (MEF). It has the charming advantage that such an assembly already has everything it needs to be accessible by DSMX. All you need to add is code to communicate with your data source to return data tables back to DSMX (which is pretty easy, as you will see).

What you need to create your own Data Relation

Because a Data Relation is a .Net Framework assembly you need Microsoft Visual Studio to build your sources. A Visual Studio Express is fair enough. It doesn’t matter if your are friend with C# or prefer Visual Basic.NET, you should have basic experiences in one of those .NET languages though. My sample is written in VB.NET by the way.

Data Relations example

Sharpen your tools is important in almost every field and especially in software development. That’s why I have one side project for over ten years now that I continuously rewrite every now and then. The application is a web based DVD management software that keeps track of your movies, if they are in the shelf or borrowed by a friend. Adding new movies is done by typing the title of the movie into a web form. A request is then sent to amazon to get all movie details. Those details are then persisted in a database.

To show you how a DSMX data relation could be implemented I extracted the part that queries Amazon web services. The data relation in other words can do a search for a title against Amazon and return a table that contains detailed information about movies you searched for. If you search for Batman, for instance, Amazon will return certainly a long list of movies.

Implementation. Setting up the project

I will not go to much into detail, because Christoph wrote a very good documentation which you will find pretty straight forward, I promise. It describes thoroughly all methods and properties the interface provides and an top it includes sample code.

Enough talk, let’s start with an empty Class Library project in Visual Studio and call it SearchAmazon. We need the following references to be added to out project:

  • System.ComponentModel.Composition (what is in fact the Extensibility Framework)
  • DataRelationInterfaces.dll ( a component that provides us with the necessary interfaces by DirectSmile)

After that we can remove the class or module that came with the Visual Studio project template a add new class and name it SearchAmazon as well.

Because our class should be exposed to DSMX later on, we must decelerate the class with the Export and the ExportMetaData attribute.

We also want to implement two interfaces called IDataRelation and IDataRelation_LoadTableNameWithParameters.

All that done the body of your class should look like this:

clip_image002

Implementation. Adding logic to your class

First we need a display name for our relation that is shown later in the Data Relations dialog of the DSMX designer.

Public ReadOnly Property DisplayName As String Implements DataRelationInterfaces.IDataRelation.DisplayName
        Get
            Return "Amzon Movie Search"
        End Get
End Property

Then we have to decide what parameters we like to pass from our campaign to the Data Relation:

  • the movie title we search for
  • the Amazon Web Services Developer Id
  • the secret developer for Amazon Web Services
Public Sub GetMetaData(AccountID As Integer, Language As String, MetaData As DataRelationInterfaces.Objects.IhtDataProviderMetaData) Implements DataRelationInterfaces.IDataRelation.GetMetaData
        MetaData.AddParameter("AWSSKeyId", "Amazon Web Services Developer ID", "", True)
        MetaData.AddParameter("AWSSecretKey", "Amazon Web Services Secret Key", "", True)
        MetaData.AddParameter("title", "Movie title", "", False)
End Sub

Later, in the Data Relations dialog, those parameters are exposed by name and description and we can add values to them. The last parameter of the AddParameter method is of interest, if that boolean is set to true a password text field is shown and typed-in characters are hidden.

Next we implement the GetTableNames method. Although this method allows a list of tables, as the method name implies, we just add a single table to the collection, holding our search result.

Public Sub GetTableNames(AccountID As Integer, Language As String, IhtDataProviderTablesList As DataRelationInterfaces.Objects.IhtDataProviderTablesList) Implements DataRelationInterfaces.IDataRelation_LoadTableNamesWithParameters.GetTableNames
        IhtDataProviderTablesList.AddTable("Amazon search result")
End Sub

There a lot more properties we can implement what makes the navigation in our Data Relation much more convenient, like support for column selection, column filtering, order by or paging. But this would extend this little example, but please refer to the documentation if you are interested. I, for the moment, return just false in all those properties, which means it’s not supported.

Public ReadOnly Property supportsColumnSelection As Boolean Implements DataRelationInterfaces.IDataRelation.supportsColumnSelection
        Get
            Return False
        End Get
End Property

Public ReadOnly Property SupportsDistinct As Boolean Implements DataRelationInterfaces.IDataRelation.SupportsDistinct
        Get
            Return False
        End Get
End Property

Public ReadOnly Property SupportsFilter As Boolean Implements DataRelationInterfaces.IDataRelation.SupportsFilter
        Get
            Return False
        End Get
End Property

Public ReadOnly Property SupportsOrderBy As Boolean Implements DataRelationInterfaces.IDataRelation.SupportsOrderBy
        Get
            Return False
        End Get
End Property

Public ReadOnly Property SupportsPaging As Boolean Implements DataRelationInterfaces.IDataRelation.SupportsPaging
        Get
            Return False
        End Get
End Property

Now to the most important method, the LoadData method. As the name implies, the method loads the data into our Data Relation table. The implementation does three things:

  • It validates the parameters passed from the campaign manager
  • It sends a search request to the Amazon web services
  • It adds the appropriate fields to our Data Relations table and fills the table with the search result
Public Sub LoadData(Table As DataRelationInterfaces.Objects.IhtDataTable) Implements DataRelationInterfaces.IDataRelation.LoadData
        Dim title As String = Table.GetParameterValue("title")
        Dim secretdeveloperKey As String = Table.GetParameterValue("AWSSecretKey")
        Dim developerId As String = Table.GetParameterValue("AWSSKeyId")

        if String.IsNullOrEmpty(title)
            Throw New ArgumentException("Movie title parameter must have a value.") 
        End If

        if String.IsNullOrEmpty(developerId)
            Throw New ArgumentException("AWS Developer Id parameter must have a value") 
        End If

        if String.IsNullOrEmpty(secretdeveloperKey)
            Throw New ArgumentException("AWS Secret Key parameter must have a value.") 
        End If

        Dim result As List(Of Movie)
        Dim itmLookUp As New Amazon.ItemLookup(developerId,secretdeveloperKey)

        result = itmLookUp.SearchByTitle(title)

        If result IsNot Nothing Then

            AddFields(Table)

               For Each m In result
                    Addrow(m, Table)
               Next

         End If

 End Sub

 Private Sub AddFields(ByRef table As IhtDataTable)
        table.AddField("ID")
        table.AddField("Name")
        table.AddField("Actor1Name")
        table.AddField("Actor3Name")
        table.AddField("Actor2Name")
        table.AddField("DirectorName")
        table.AddField("EAN")
        table.AddField("ASIN")
        table.AddField("PublishDate")
        table.AddField("Publisher")
        'Todo: Add more fields here ...
End Sub

 Private Sub Addrow(m As Movie, byref table As IhtDataTable)

        Dim r As IhtDataTableRow
        r = table.AddRow
        r.AddValue(table.RowCount + 1) 
        r.AddValue(m.Name)
        r.AddValue(m.Actor1Name)
        r.AddValue(m.Actor3Name)
        r.AddValue(m.Actor2Name)
        r.AddValue(m.DirectorName)
        r.AddValue(m.EAN)
        r.AddValue(m.ASIN)
        r.AddValue(m.PublishDate)
        r.AddValue(m.Publisher)
        'Todo: Add more values here ...
    End Sub

The actual call to Amazon is abstracted into a helper class called ItemLookup. That calls takes our AWS logon credentials in the constructor and provides one simple method called SearchByTitle. That method returns a list of Movie, which is a simple type that contains a set of properties. The names of the properties will become the field names of our table, which are added in the AddFields method. While iterating through the search result the AddRow method takes the Movie type and adds it the row collection of the Data Relation table.Throwing an exception inside a Data Relation is bubbled through the interface and directly visible in the DSMX Designer. The LoadData method takes advantage of this while validating the parameters. Very handy by the way.

Conclusion

With Data Relations we have a powerful tool in hand that offers us new opportunities to merge data from different data sources into our campaign data.

As we will see in the next post, this feature can really play it’s strength in building rich client applications using the Crossmedia Designer. In my opinion, as a software developer, this flexibility must not be underestimated. Software development is in a process of moving from the ivory tower where just a few people were able to aggregate data and build reporting applications, to a point where the architectural plumbing is more and more available or generated automatically and a majority of people can start building applications using those automatically aggregated data. The Crossmedia Designer is an excellent example!

Advertisements

Verschlüsseln einer virtuellen Harddisk mit Bitlocker

Ich bin ein Laptop-Benutzer. Ich habe nur einen Laptop, doch den benutze ich am Arbeitsplatz, zu Hause, in der Bahn zur Arbeit und eigentlich auch sonst überall wo ich mich gerade aufhalte. Diese hohe Flexibilität hat einen gewissen Preis, denn ein Laptop ist schnell gestohlen. Und damit auch alle darauf enthaltenen Daten. Ehrlich gesagt, sind es oft die Daten die wichtiger sind als das Gerät. Es wäre zwar schade, wenn mein Lenovo W520 gestohlen würde, doch ist ein Datendiebstahl um einiges schlimmer.

Darum schütze ich meine Daten in einem verschlüsselten Container. Dieser Container ist üblicherweise eine virtuelle Festplatte, also eine Datei, die einem Verzeichnis liegt und zur Laufzeit als Laufwerk vom Betriebssystem eingebunden wird.

Für solche verschlüsselten Laufwerke gibt es verschiedene Tools, zum Beispiel von der PGP Corperation. Ich benutze allerdings ein Feature aus der Windows 7 Ultimate Edition, das BitLocker heißt. Man braucht zwar leider die Ultimate Edition, denn nur diese enthält Bitlocker, doch dann hat man alles an Boardmitteln parat was man braucht, um eine verschlüsselte virtuelle Harddisk zu erstellen.

Erstellen einer virtuellen Disk mit Windows 7

Das Erstellen einer virtuellen Disk ist sehr einfach. Alles was man dazu braucht ist die Datenträgerverwaltung.

  1. Rechte Maustaste auf Computer und Computerverwaltung auswählen
  2. Datenspeicher expandieren
  3. Rechts auf Datenträgerverwaltung klicken
  4. Virtuelle Festplatte erstellen auswählen

image

Im darauffolgenden Assistenten muss man den Speicherort und die Größe der virtuellen Disk angeben. Die Größe kann dynamisch oder fest sein. Hat man den Assistenten erfolgreich beendet erscheint ein neues Volume in der Datenträger Übersicht. Auf diesem Datenträger kann man nun eine primäre Partition erstellen, diese dann wie üblich formatieren und einen Laufwerksbuchstaben zuweisen.

Danach steht das Laufwerk zu Verfügung.

Bit Locker Verschlüsselung des Laufwerks

Im Kontextmenü, durch Rechtsklick auf das Laufwerk, im Explorer, gibt es in einer Windows 7 Ultimate die Möglichkeit diese Partition mit BitLocker zu verschlüsseln.

Den Schlüssel, den Bitlocker erzeugt sollte man ausdrucken oder in einer Textdatei speichern, vorzugsweise auf einem zweiten Computer, damit er nicht verloren geht.

Abhängig von der Größe dauert die Verschlüsselung einige Minuten.

Erneutes Einhängen des virtuellen Laufwerks nach Neustart

Ein virtuelles Laufwerk steht nach einem Neustart nicht sofort zu Verfügung. Es gibt nun zwei Wege das Laufwerk wieder anzumelden. Zum Beispiel kann man das Laufwerk wieder über die Datenträgerverwaltung einbinden. Das erfordert allerdings das Laden der Computerverwaltung usw.

Für mich ist das ein wenig zu unbequem, deshalb habe ich mir eine Kommandozeilen-Batch Datei geschrieben, die das Laufwerk automatisch einhängt. Alles was ich dann noch machen muss, ist meine Bitlocker Passphrase einzugeben.

Die Batchdatei steuert das Kommandozeilen Tool DISKPART fern. Dazu braucht es nur DISKPART, mit einem Verweis auf eine Scriptdatei, aufzurufen:

diskpart /s “<PATH TO SCRIPT>\attachDisk.txt”

Das übergebene Script mit den Befehlen für DISKPART ist ebenfalls sehr einfach:

select vdisk file=”<PATH TO VHD>\SecureDisk.vhd”
attach vdisk

Hier wird unsere virtuelle Harddisk ausgewählt und dann per attach Befehl eingebunden.

Womit sie dann so zur Verfügung stehen sollte:

image

Leicht erkennbar an dem Bitlocker-Schlosssymbol.

Sending DirectSmile Integration Server perfomance counters to Amazon CloudWatch

This is the third post of a series of blog posts about new cloud computing features in DirectSmile Integration Server version 5.

In my last post about DirectSmile Integration Server 5 performance counters, which you can find here, I showed how the DirectSmile Intergration Server backend collects windows performance counters regarding the performance of DSMI. This helps most of all IT departments to monitor if the system is working properly and sufficient.

In a cloud environment like Amazon’s EC2 this isn’t enough though, because Amazon provides us a very good platform, called Amazon CloudWatch, which is capable of monitoring all instances at once. CloudWatch can aggregate performance data and present it in nice graphs. On top of that, CloudWatch lets us add alerts and actions depending on the monitoring values.

To get the performance counters written by the backend into CloudWatch I wrote a little CloudWatch client that runs on the machine that has the DSMI backend running. This client is a simple console application that can also be started as a Windows service, which is then running in the background.

Once the client sends the performance counter data, the counters are visible in Amazon CloudWatch.

Here’s a screenshot taken from Amazon CloudWatch, showing the production server load, of two DSMI machines, while job processing.

clip_image002

Based on metrics we can setup alerts. We can for example define an alert that notifies us if the DSMIBusyProdserver value is hitting 100 % continuesly for 3 hours in a row. Or if the DSMIFailedJobsInPercent is rising above 5%. Depending on the DSMIJobInQueue you could spin up more EC2 instances to cope with the extra job load.

New Performance counters in DirectSmile Integration Server 5

This is the second post of a series of blog posts about new cloud computing features in DirectSmile Integration Server version 5.

Following DSMI installations of our customers in the past I mentioned that those installations became bigger and bigger. Not only that those installations involve more machines holding production servers, no, even DSMIs systems are extended by load balancing and replication.

The built-in DirectSmile Integration Server backend monitor is quite a handy tool to monitor the state of the jobs queues and the production servers. The disadvantage though is, that especially the backend monitor needs a range of ports to be open to allow the Silverlight application, hosted in the browser, to access those data. Another disadvantage, from an IT perspective is, that this highly-valued data cannot be aggregated easily by common tools in IT administration.

Having demands of IT departments in mind, we added performance counters to the DirectSmile Integration Server backend. The backend exposes now those three counters:

dsmiperf

  • DSMIBusyProdServer shows the load of busy production server in percent
  • DSMIFailedJobsInPercent shows the amount of failed jobs in percent
  • DSMIJobQueue shows the amount of jobs waiting in the In-queue.

I think that these three values can give IT administration a better understanding of the current state of a DSMI. Bottlenecks can be easily, and in an automated way, identified. For instance, you probably have not enough Production Servers running, if the Job-In Queue is always filled. Or, deeper investigation is necessary, if DSMIFailedJobsInPercent hits a defined level.

Have fun.

Hosting DirectSmile Integration Server 5 in the cloud

This is the first post of a series of blog posts about new cloud computing features in DirectSmile Integration Server version 5.

I ‘m very excited to let you know that the version 5 of the DirectSmile Integration Server can be hosted in the cloud.

Cloud computing for the DirectSmile Integration Server means full flexibility in image and document rendering power. The demands for personalized printing in our industry depends often on seasons in the year, there is, for instance, the calendar production in the end of the year. The postcards production increases drastically around Mother’s or Valentine day, what is a total difference to the average production over the rest of the year. Wouldn’t it be great to ensure extra capacity to cope with those extra amounts on a daily or even hourly basis? And keep your system small enough for the rest of the year to save energy and hardware costs?

In the beginning of 2012 we and some DirectSmile Beta customers started an approach to move an onsite installation of the DirecSmile Integration Server into the cloud. Having a strong relationship with local ISPs we began with private cloud solutions. That was very successful and is in production now.

In parallel we extended our relationship with Amazon and started test installations in the EC2 Amazon cloud environment. Against the private cloud scenarios, what typically depend on the customer and the customer’s ISP, we are following a more general approach with Amazon. Amazon’s global infrastructure, datacenters and monitoring software made it more easy for us to find basic solution that enables our customers to move their existing servers into the cloud or start a new installation from scratch. Because installations are generalized in virtual machine templates, increasing and decreasing the amount of available machines is much more easy.

I’m planning a series of blog posts around DSMI and cloud computing and this is the first. Let me finish with a really excellent best practice guide written by Andreas Ostermann, who put together all his experience from helping customers establishing large DSMI installations in the Amazon AWS cloud environment in the last months.

DSM AWS Hosting Best Practice

have fun.