DirectSmile Integration Server commandline tool DStrip

In the past I wrote about one or the other command line tool related to DSMI, but this one is quite handy and easy to use. DStrip makes it very easy to unpack and add a bunch of DirectSmile Set- or Document archives, in dZip format, to DSMI at once.

dZip

The DirectSmile Document and Set export format is typically a Zip container that includes a lot of resources, like clip-, pic- or system fonts, a background image and more. While the Set preperation for instance, we usually add several scalings of the Set as well. The Zip Archive keeps the folder hirarchie and that enures that the Set is unpacked at the right place on your machine, which is typically the working directory.

DSMI and dZips

The DirectSmile Integration Server provides several methods to import Sets and Documents, I blogged about this here. Most of them, except the whole catalog import, have in common that you can import just one archive at once. The import can deal with dZips though, but one at a time.

DStrip

That’s where you can use DStrip. It can import as many dZips as it can find in a source folder and it can remove all scalings before the import to DSMI. Scalings are igrnored by DSMI, because of the built-in autoscaling.

DStrip is a little commandline tool that you simply tell:

  • a source directory that contains dZip archives
  • a target directory were the unpacked Sets and Documents should be stored in, typically the DSMI account folder

On top of that you can pass three commandline switches, but DStrip works just fine with the defaults:

  • IgnoreScaled True/False (Default True)] Shortform: /i:
  • FlattenOutput:True/False (Default False)] Shortform: /f:
  • Overwrite:True/False (Default True)] Shortform: /o:
    Here’s an example, where I ran DStrip from c:\temp. I copied a few dZips into the temp\dzips folder before and ran the tool. All I need to specfify was the source- and the destintation directory.

image

Have fun!

Advertisements

Deny access from internet to DirectSmile Integration Server settings

The DirectSmile Integration Server front end is using https and Forms authentication to authenticate and authorize DSMI accounts. That is a common way to ensure security and to allow access to specific users.

On top of that you can add extra security to restrict access to specific parts in DSMI.

Using IP Address Domain Restrictions in IIS 7/8

To restrict the access to the server settings from the internet for instance, you can add the IIS feature called IP Address Domain Restrictions.

image

This feature enables client IP based  restrictions for a whole web site or a specific sub folder of an IIS applications. For the server settings example, this means that you want to add a restriction just for the ServerSettings subfolder.

 image

Setting up such a restriction is easy, all you need to deny the access from unspecified clients in the feature settings.

image

Now we are denying every request, what is secure. Now we add an exception to allow requests from the local hostm only.

image

Clients trying to access the DSMI server settings, will now recieve a 403:

image

While the server settings are still accessible from the local machine.

Autoscaling activated automatically on Image set checkin

With the next version of the DirectSmile Integration Server we will change a minor thing in the Image Set check-in process. Formerly if you checked in a DirectSmile Set the auto scaling was turned of and must have been activated manually. We changed that and now auto scaling is activated for a fresh uploaded Set automatically.

In case you don’t want to update your DSMI but need to have the auto scaling activated you can do that by adding an INSERT Trigger to the items table in the dsmodb database. The trigger simply sets the auto scaling flag to true on new check-ins. The SQL code example to add such a trigger might look like this:

CREATE TRIGGER tblItems_Set_Autoscale
ON [dbo].[tblItems] FOR INSERT
AS
BEGIN
— SET NOCOUNT ON added to prevent extra result sets from
— interfering with SELECT statements.
SET NOCOUNT ON;
DECLARE @Id int
DECLARE @ContentType char(3)
SELECT @Id = inserted.[ID] FROM inserted
SELECT @ContentType= inserted.[ContentType] FROM inserted
if @ContentType =’SET’
Update [dbo].[tblItems] set AutoScale=1 WHERE ID=@Id
END
GO

It checks if the user checked in a SET and updates the autoscale flag.

Sending DirectSmile Integration Server perfomance counters to Amazon CloudWatch

This is the third post of a series of blog posts about new cloud computing features in DirectSmile Integration Server version 5.

In my last post about DirectSmile Integration Server 5 performance counters, which you can find here, I showed how the DirectSmile Intergration Server backend collects windows performance counters regarding the performance of DSMI. This helps most of all IT departments to monitor if the system is working properly and sufficient.

In a cloud environment like Amazon’s EC2 this isn’t enough though, because Amazon provides us a very good platform, called Amazon CloudWatch, which is capable of monitoring all instances at once. CloudWatch can aggregate performance data and present it in nice graphs. On top of that, CloudWatch lets us add alerts and actions depending on the monitoring values.

To get the performance counters written by the backend into CloudWatch I wrote a little CloudWatch client that runs on the machine that has the DSMI backend running. This client is a simple console application that can also be started as a Windows service, which is then running in the background.

Once the client sends the performance counter data, the counters are visible in Amazon CloudWatch.

Here’s a screenshot taken from Amazon CloudWatch, showing the production server load, of two DSMI machines, while job processing.

clip_image002

Based on metrics we can setup alerts. We can for example define an alert that notifies us if the DSMIBusyProdserver value is hitting 100 % continuesly for 3 hours in a row. Or if the DSMIFailedJobsInPercent is rising above 5%. Depending on the DSMIJobInQueue you could spin up more EC2 instances to cope with the extra job load.

New font installation feature in DirectSmile Integration Server Part 2

This is a follow up of a post I wrote in back in 2011 regarding font installation on multiple servers in DSMI. While in this post I tried to paint a more general picture I now want to go a bit more into detail and shine a little light into configuration.

Setup

If you upload a DirectSmile Set to DSMI, the VDP Studio archives all fonts from your workstation and uploads them together with other resources to DSMI. DSMI unpacks the archive into the dsmusers/sets directory. While DSMI is unpacking the files DSMI recognizes the font files and stores an installation request in the [dsmodb].[dbo].[tblFonts] table. A typical record looks like this:

image

This table is constantly scanned by the DSMWatchDog application that runs on all DSMI machines that provide DirectSmile Production Servers.

Although we can have several watchdogs registered for the same DSMI, we can have just one watchdog per machine. So, each watchdog is responsible for the font installation on each machine.

To ensure that each watchdog can persist the installation result of the fonts, each watchdog gets a unique id in the system. This ID is later on persisted to each font installation record. In this way the watchdog can check if it already installed the font and we can check the result of the font installation at any time.

Installation Result

Installation results are persisted by the ID of the watchdog that tried to install the font in the InstallationResult[dsmodb].[dbo].[tblFonts] table. Here’s an example:

[100] -> CFR: OK, AFR: OK, SMR: 1, WPS: OK
[200] -> CFR: OK, AFR: OK, SMR: 1, WPS: OK
[201] -> CFR: OK, AFR: OK, SMR: 1, WPS: OK

In the example we see the result from three watchdogs (100,200,201) that all successfully installed the font.

This is what one a single InstallationResult line consists of:

[100] CFR AFR SMR WPS
Description WatchDog ID File system copy result Font Resource Add Result Send Message result Registry access result
Return Value N/A OK or exception message OK, or exception message Must be greater zero OK, or exception message

WatchDog configuration

The watchdog needs information to access the dsmodb database, where to find the dsmusers directory and a unique ID . We also can define a path where to write a log file. If we let the log file value empty, no log file will be written. Those settings are stored in the .net based application configuration file. You can find both files in the DirectSmile Online Backend directory.

image

Here’s an example of the user settings area of the DSMWatchDog.Exe.Config file:

image

Conclusion

Giving the different instances of the DSMWatchDog unique IDs we can run the font installation on different machines in a DSMI environment. The limitation that the watchdog can only install fonts on DSMI servers (master or client in replication scenarios) is over. That makes it possible that even servers that just provide Production Servers to a DSMI can profit from font installation. Despite the information in the application icon balloon, the InstallationResult gives the DSMI operator a detailed hint about what and where failed in case a font installation went wrong.

Continuous deployment of DirectSmile products

We at DirectSmile love our products

Because we love our products everyone of our team wants to get the latest version installed to benefit from latest changes or simply to play around with new features.

This is great! But it means a lot of maintenance though, doing all those setups and configuration in the morning when there’s a new nightly build available.

From continuous builds to continuous deployments

Generally, an installation is a process that has quite simple rules. You provide all the necessary information the installer file needs, beginning with the target directory, database connection strings or IIS website names and the windows installer services does the rest.

The good thing is, that those information normally don’t change if you do a software update. All installers support a parameter system that allows you to add those settings as arguments to the installer execution call.

That’s a good start for the DirectSmile Installation Service. The general purpose of this service is to run installations and uninstallation of a DirectSmile product in an automated way.

In development we coupled the DirectSmile Installation Service with Jenkins, the continuous integration server, we are using. Using this technology for a while now enables us to do builds on each check-in and run tests immediately on staging servers, where even the new installer is deployed to.

Application architecture

The DirectSmile Installation Service application consists of two parts. A server component which is installed on the machine that should be automatically maintained. And a client component that has all the client specific configuration data and automates the installation service remotely.

The client and the server communicate through SSL secured tunnel. The communication can only be established if the client identifies the server and (much more important) if the server identifies the client. This is covered by client and server certificates.

The installation file itself can be either downloaded by the service from a trusted http download server, or from the local file system.

image

Figure 1. DirectSmile Installation Service

While the installation service is a windows service component the client is a windows command line application.

Installation remote commands

The client comes with a few commands that initiate an installation, uninstallation or simply shows you the current installed version of an installation. Here’s a short description of the commands:

Usage: DSMInstallationClient <version|install|uninstall|installfont|getlog|backup|help>
Params:   [/url:<
http://downloadurl>]
[/productCode:<{GUID}>]
[/processesToKill:<Process1;Process2>]
[/servicename:<servicename>]
[/source:<src ath to backup>]
[/destination:<backup dest path>]
[/endpoint:<service endpoint url]

About ProductCodes

Usually a installation product code is a GUID. To make it more convenient to deal with product code we created some shortcuts for the DirectSmile products and you can use Product Name representational string instead.
Those are: dsmi, dsmg, dsmx, dsmstore, smartstream.

Version

Here’s an example how to retrieve the version number of DSMX remotely:

version /ProductCode=dsmx

 

Installation

 

Running an installation is quite easy. All you need to provide the Url, where the installation file can be downloaded and all parameters that are needed by the installation process. 

install /url=http://... /WEBSITES=”c:\inetpub\wwwroot\dsmstore” /watchdog=yes

This command would download the installation file for the DirectSmile Card & Giftshop and execute the installation. While this installation is running the DirectSmile Watchdog would be stopped.

Uninstallation

To do an uninstallation the product code is needed, but we can use our product name shortcut here again. You can also pass a list of processes and services that should be stopped before the uninstallation.

uninstall /ProductCode=dsmi
/ProcessesToKill=”DirectSmile Generator;DSMWatchDog”
/ServiceName=”DSMOnlineBackend”

This command would do an uninstallation of DSMI, but first it would stop all running ProductionServer instances and the DirectSmile Watchdog. It also would stop the DSMOnlineBackend service.

 

Doing a backup of directory remotely

The installation service can do a backup a directory. This directory will be zipped automatically and place i n the destination directory you name.

backup /source=”c:\inetpub\wwwroot\dsmstore” /desitination=”c:\temp”

The command would create a backup of the web application directory of the DirectSmile Shop and copy the zipped archive to c:\temp.

Endpoint parameter

The client comes with a config file that includes a default endpoint url to the installation service. Usually you don’t want to change that, but if you have several products installed on different machines, you might want to use the same client to handle all of those installationes. In this case you could pass the service endpoint url as an argument in the installation client call.

DSMInstallationClient version /ProductCode=dsmx /endpoint=”https://<SERVER>/DSMInstallationSevice.svc”

This sample would would retrieve the DSMX product version from a specific service running on a server called <server>.

Log files

The DirectSmile Installation service writes a log file. This log file can be downloaded to the client.

getlog > filename.log

This command would download the log file from the server and stores it locally on the client machine.

What to do if the DirectSmile Integration Server services are inaccessible

Today I would like to list some of the typical reasons why DSMI services fail to initialize or why you can’t access those services. Below you will find a short to-do or what-to-check list, that I’m going through when ever I have problem. Those are all things that you can easily do.

Multiple SSL Certificates

DirectSmile relevant certificates are stored in the localmachine/my certificate store. Please check if you have just one certificate with the common name that is addressed in the web.config file of DSMI. If you have multiple certificates, then it’s impossible for the web server to distinguish between the certificates and the initialization of the WCF services will fail. This error is often the case if you requested a new certificate from your CA, because the old expired and you forgot to remove the old one from the store.

Compilation error

Open the workflow URL in the browser to check if the typical WCF generated info screen appears. The URL would look like: http://SERVER/dsmo/workflow.svc

If successful you should see something like:

clip_image001

If not, you receive a HTTP 500 (Internal Server Error) Status Code.

Fiddler Test

Run Fiddler on your client and browse to the workflow or repository manager. Check if fiddler reports any errors. If you don’t have Fiddler installed go check: http://www.fiddler2.com/fiddler2/

Server Event log

If you have access to the server, a check in the event log is always worthwhile. Typically you will find errors indicating why a service could not be started.

Server Time vs Client Time

The time difference between the client and server is immanent to establish an SSL Connection. The difference must be less than 5 minutes between the server and the client. This is independent from time zones, because the server and client local time is always mapped to GMT.

Crosssite Scripting

To allow the Silverlight Clients like the Workflow and the Repository Manager to connect to the DSMI server the server must provide a so called ClientAccessPolicy.xml file in the web servers root folder. This file contains a straight forward set of rules, that tells the webs server to allow access to http/https applications or not.

Important in our case is, that it explicitly allows https access, like this file:

clip_image002

WCF Tracing

If all those checks from above were fruitless, you could activate WCF event tracing in the web.config file of DSMI. But maybe this is something you want to do together with the DirectSmile Support Zwinkerndes Smiley