DirectSmile Integration Server commandline tool DStrip

In the past I wrote about one or the other command line tool related to DSMI, but this one is quite handy and easy to use. DStrip makes it very easy to unpack and add a bunch of DirectSmile Set- or Document archives, in dZip format, to DSMI at once.

dZip

The DirectSmile Document and Set export format is typically a Zip container that includes a lot of resources, like clip-, pic- or system fonts, a background image and more. While the Set preperation for instance, we usually add several scalings of the Set as well. The Zip Archive keeps the folder hirarchie and that enures that the Set is unpacked at the right place on your machine, which is typically the working directory.

DSMI and dZips

The DirectSmile Integration Server provides several methods to import Sets and Documents, I blogged about this here. Most of them, except the whole catalog import, have in common that you can import just one archive at once. The import can deal with dZips though, but one at a time.

DStrip

That’s where you can use DStrip. It can import as many dZips as it can find in a source folder and it can remove all scalings before the import to DSMI. Scalings are igrnored by DSMI, because of the built-in autoscaling.

DStrip is a little commandline tool that you simply tell:

  • a source directory that contains dZip archives
  • a target directory were the unpacked Sets and Documents should be stored in, typically the DSMI account folder

On top of that you can pass three commandline switches, but DStrip works just fine with the defaults:

  • IgnoreScaled True/False (Default True)] Shortform: /i:
  • FlattenOutput:True/False (Default False)] Shortform: /f:
  • Overwrite:True/False (Default True)] Shortform: /o:
    Here’s an example, where I ran DStrip from c:\temp. I copied a few dZips into the temp\dzips folder before and ran the tool. All I need to specfify was the source- and the destintation directory.

image

Have fun!

Deny access from internet to DirectSmile Integration Server settings

The DirectSmile Integration Server front end is using https and Forms authentication to authenticate and authorize DSMI accounts. That is a common way to ensure security and to allow access to specific users.

On top of that you can add extra security to restrict access to specific parts in DSMI.

Using IP Address Domain Restrictions in IIS 7/8

To restrict the access to the server settings from the internet for instance, you can add the IIS feature called IP Address Domain Restrictions.

image

This feature enables client IP based  restrictions for a whole web site or a specific sub folder of an IIS applications. For the server settings example, this means that you want to add a restriction just for the ServerSettings subfolder.

 image

Setting up such a restriction is easy, all you need to deny the access from unspecified clients in the feature settings.

image

Now we are denying every request, what is secure. Now we add an exception to allow requests from the local hostm only.

image

Clients trying to access the DSMI server settings, will now recieve a 403:

image

While the server settings are still accessible from the local machine.

Customizing the default image for PictureInPicture Sets

Application Default image
Usually, if you pass a URL to a DirectSmile Set that is contains a Picture In Picture frame, the image is downloaded and placed while the image is rendered. Sometimes, if the download failes for instance, the image can’t be placed and a default image is taken instead. Typically the image would look like this:

image

As you can see the text field does not contain a valid URL and the default image, built into the application, is taken.

Providing your own default image
The default image can easily be customized by placing a JPG file in the DirectSmile Set directory. Let’s take the following image.

image

It is important to name the image default.jpg, otherwise the VDP Studio won’t find the image.

image

The VDP Studio recognizes the file and automatically replaces the default image by the customized image.

image

Autoscaling activated automatically on Image set checkin

With the next version of the DirectSmile Integration Server we will change a minor thing in the Image Set check-in process. Formerly if you checked in a DirectSmile Set the auto scaling was turned of and must have been activated manually. We changed that and now auto scaling is activated for a fresh uploaded Set automatically.

In case you don’t want to update your DSMI but need to have the auto scaling activated you can do that by adding an INSERT Trigger to the items table in the dsmodb database. The trigger simply sets the auto scaling flag to true on new check-ins. The SQL code example to add such a trigger might look like this:

CREATE TRIGGER tblItems_Set_Autoscale
ON [dbo].[tblItems] FOR INSERT
AS
BEGIN
— SET NOCOUNT ON added to prevent extra result sets from
— interfering with SELECT statements.
SET NOCOUNT ON;
DECLARE @Id int
DECLARE @ContentType char(3)
SELECT @Id = inserted.[ID] FROM inserted
SELECT @ContentType= inserted.[ContentType] FROM inserted
if @ContentType =’SET’
Update [dbo].[tblItems] set AutoScale=1 WHERE ID=@Id
END
GO

It checks if the user checked in a SET and updates the autoscale flag.

Sending DirectSmile Integration Server perfomance counters to Amazon CloudWatch

This is the third post of a series of blog posts about new cloud computing features in DirectSmile Integration Server version 5.

In my last post about DirectSmile Integration Server 5 performance counters, which you can find here, I showed how the DirectSmile Intergration Server backend collects windows performance counters regarding the performance of DSMI. This helps most of all IT departments to monitor if the system is working properly and sufficient.

In a cloud environment like Amazon’s EC2 this isn’t enough though, because Amazon provides us a very good platform, called Amazon CloudWatch, which is capable of monitoring all instances at once. CloudWatch can aggregate performance data and present it in nice graphs. On top of that, CloudWatch lets us add alerts and actions depending on the monitoring values.

To get the performance counters written by the backend into CloudWatch I wrote a little CloudWatch client that runs on the machine that has the DSMI backend running. This client is a simple console application that can also be started as a Windows service, which is then running in the background.

Once the client sends the performance counter data, the counters are visible in Amazon CloudWatch.

Here’s a screenshot taken from Amazon CloudWatch, showing the production server load, of two DSMI machines, while job processing.

clip_image002

Based on metrics we can setup alerts. We can for example define an alert that notifies us if the DSMIBusyProdserver value is hitting 100 % continuesly for 3 hours in a row. Or if the DSMIFailedJobsInPercent is rising above 5%. Depending on the DSMIJobInQueue you could spin up more EC2 instances to cope with the extra job load.

New Performance counters in DirectSmile Integration Server 5

This is the second post of a series of blog posts about new cloud computing features in DirectSmile Integration Server version 5.

Following DSMI installations of our customers in the past I mentioned that those installations became bigger and bigger. Not only that those installations involve more machines holding production servers, no, even DSMIs systems are extended by load balancing and replication.

The built-in DirectSmile Integration Server backend monitor is quite a handy tool to monitor the state of the jobs queues and the production servers. The disadvantage though is, that especially the backend monitor needs a range of ports to be open to allow the Silverlight application, hosted in the browser, to access those data. Another disadvantage, from an IT perspective is, that this highly-valued data cannot be aggregated easily by common tools in IT administration.

Having demands of IT departments in mind, we added performance counters to the DirectSmile Integration Server backend. The backend exposes now those three counters:

dsmiperf

  • DSMIBusyProdServer shows the load of busy production server in percent
  • DSMIFailedJobsInPercent shows the amount of failed jobs in percent
  • DSMIJobQueue shows the amount of jobs waiting in the In-queue.

I think that these three values can give IT administration a better understanding of the current state of a DSMI. Bottlenecks can be easily, and in an automated way, identified. For instance, you probably have not enough Production Servers running, if the Job-In Queue is always filled. Or, deeper investigation is necessary, if DSMIFailedJobsInPercent hits a defined level.

Have fun.

Hosting DirectSmile Integration Server 5 in the cloud

This is the first post of a series of blog posts about new cloud computing features in DirectSmile Integration Server version 5.

I ‘m very excited to let you know that the version 5 of the DirectSmile Integration Server can be hosted in the cloud.

Cloud computing for the DirectSmile Integration Server means full flexibility in image and document rendering power. The demands for personalized printing in our industry depends often on seasons in the year, there is, for instance, the calendar production in the end of the year. The postcards production increases drastically around Mother’s or Valentine day, what is a total difference to the average production over the rest of the year. Wouldn’t it be great to ensure extra capacity to cope with those extra amounts on a daily or even hourly basis? And keep your system small enough for the rest of the year to save energy and hardware costs?

In the beginning of 2012 we and some DirectSmile Beta customers started an approach to move an onsite installation of the DirecSmile Integration Server into the cloud. Having a strong relationship with local ISPs we began with private cloud solutions. That was very successful and is in production now.

In parallel we extended our relationship with Amazon and started test installations in the EC2 Amazon cloud environment. Against the private cloud scenarios, what typically depend on the customer and the customer’s ISP, we are following a more general approach with Amazon. Amazon’s global infrastructure, datacenters and monitoring software made it more easy for us to find basic solution that enables our customers to move their existing servers into the cloud or start a new installation from scratch. Because installations are generalized in virtual machine templates, increasing and decreasing the amount of available machines is much more easy.

I’m planning a series of blog posts around DSMI and cloud computing and this is the first. Let me finish with a really excellent best practice guide written by Andreas Ostermann, who put together all his experience from helping customers establishing large DSMI installations in the Amazon AWS cloud environment in the last months.

DSM AWS Hosting Best Practice

have fun.

New font installation feature in DirectSmile Integration Server Part 2

This is a follow up of a post I wrote in back in 2011 regarding font installation on multiple servers in DSMI. While in this post I tried to paint a more general picture I now want to go a bit more into detail and shine a little light into configuration.

Setup

If you upload a DirectSmile Set to DSMI, the VDP Studio archives all fonts from your workstation and uploads them together with other resources to DSMI. DSMI unpacks the archive into the dsmusers/sets directory. While DSMI is unpacking the files DSMI recognizes the font files and stores an installation request in the [dsmodb].[dbo].[tblFonts] table. A typical record looks like this:

image

This table is constantly scanned by the DSMWatchDog application that runs on all DSMI machines that provide DirectSmile Production Servers.

Although we can have several watchdogs registered for the same DSMI, we can have just one watchdog per machine. So, each watchdog is responsible for the font installation on each machine.

To ensure that each watchdog can persist the installation result of the fonts, each watchdog gets a unique id in the system. This ID is later on persisted to each font installation record. In this way the watchdog can check if it already installed the font and we can check the result of the font installation at any time.

Installation Result

Installation results are persisted by the ID of the watchdog that tried to install the font in the InstallationResult[dsmodb].[dbo].[tblFonts] table. Here’s an example:

[100] -> CFR: OK, AFR: OK, SMR: 1, WPS: OK
[200] -> CFR: OK, AFR: OK, SMR: 1, WPS: OK
[201] -> CFR: OK, AFR: OK, SMR: 1, WPS: OK

In the example we see the result from three watchdogs (100,200,201) that all successfully installed the font.

This is what one a single InstallationResult line consists of:

[100] CFR AFR SMR WPS
Description WatchDog ID File system copy result Font Resource Add Result Send Message result Registry access result
Return Value N/A OK or exception message OK, or exception message Must be greater zero OK, or exception message

WatchDog configuration

The watchdog needs information to access the dsmodb database, where to find the dsmusers directory and a unique ID . We also can define a path where to write a log file. If we let the log file value empty, no log file will be written. Those settings are stored in the .net based application configuration file. You can find both files in the DirectSmile Online Backend directory.

image

Here’s an example of the user settings area of the DSMWatchDog.Exe.Config file:

image

Conclusion

Giving the different instances of the DSMWatchDog unique IDs we can run the font installation on different machines in a DSMI environment. The limitation that the watchdog can only install fonts on DSMI servers (master or client in replication scenarios) is over. That makes it possible that even servers that just provide Production Servers to a DSMI can profit from font installation. Despite the information in the application icon balloon, the InstallationResult gives the DSMI operator a detailed hint about what and where failed in case a font installation went wrong.

Continuous deployment of DirectSmile products

We at DirectSmile love our products

Because we love our products everyone of our team wants to get the latest version installed to benefit from latest changes or simply to play around with new features.

This is great! But it means a lot of maintenance though, doing all those setups and configuration in the morning when there’s a new nightly build available.

From continuous builds to continuous deployments

Generally, an installation is a process that has quite simple rules. You provide all the necessary information the installer file needs, beginning with the target directory, database connection strings or IIS website names and the windows installer services does the rest.

The good thing is, that those information normally don’t change if you do a software update. All installers support a parameter system that allows you to add those settings as arguments to the installer execution call.

That’s a good start for the DirectSmile Installation Service. The general purpose of this service is to run installations and uninstallation of a DirectSmile product in an automated way.

In development we coupled the DirectSmile Installation Service with Jenkins, the continuous integration server, we are using. Using this technology for a while now enables us to do builds on each check-in and run tests immediately on staging servers, where even the new installer is deployed to.

Application architecture

The DirectSmile Installation Service application consists of two parts. A server component which is installed on the machine that should be automatically maintained. And a client component that has all the client specific configuration data and automates the installation service remotely.

The client and the server communicate through SSL secured tunnel. The communication can only be established if the client identifies the server and (much more important) if the server identifies the client. This is covered by client and server certificates.

The installation file itself can be either downloaded by the service from a trusted http download server, or from the local file system.

image

Figure 1. DirectSmile Installation Service

While the installation service is a windows service component the client is a windows command line application.

Installation remote commands

The client comes with a few commands that initiate an installation, uninstallation or simply shows you the current installed version of an installation. Here’s a short description of the commands:

Usage: DSMInstallationClient <version|install|uninstall|installfont|getlog|backup|help>
Params:   [/url:<
http://downloadurl>]
[/productCode:<{GUID}>]
[/processesToKill:<Process1;Process2>]
[/servicename:<servicename>]
[/source:<src ath to backup>]
[/destination:<backup dest path>]
[/endpoint:<service endpoint url]

About ProductCodes

Usually a installation product code is a GUID. To make it more convenient to deal with product code we created some shortcuts for the DirectSmile products and you can use Product Name representational string instead.
Those are: dsmi, dsmg, dsmx, dsmstore, smartstream.

Version

Here’s an example how to retrieve the version number of DSMX remotely:

version /ProductCode=dsmx

 

Installation

 

Running an installation is quite easy. All you need to provide the Url, where the installation file can be downloaded and all parameters that are needed by the installation process. 

install /url=http://... /WEBSITES=”c:\inetpub\wwwroot\dsmstore” /watchdog=yes

This command would download the installation file for the DirectSmile Card & Giftshop and execute the installation. While this installation is running the DirectSmile Watchdog would be stopped.

Uninstallation

To do an uninstallation the product code is needed, but we can use our product name shortcut here again. You can also pass a list of processes and services that should be stopped before the uninstallation.

uninstall /ProductCode=dsmi
/ProcessesToKill=”DirectSmile Generator;DSMWatchDog”
/ServiceName=”DSMOnlineBackend”

This command would do an uninstallation of DSMI, but first it would stop all running ProductionServer instances and the DirectSmile Watchdog. It also would stop the DSMOnlineBackend service.

 

Doing a backup of directory remotely

The installation service can do a backup a directory. This directory will be zipped automatically and place i n the destination directory you name.

backup /source=”c:\inetpub\wwwroot\dsmstore” /desitination=”c:\temp”

The command would create a backup of the web application directory of the DirectSmile Shop and copy the zipped archive to c:\temp.

Endpoint parameter

The client comes with a config file that includes a default endpoint url to the installation service. Usually you don’t want to change that, but if you have several products installed on different machines, you might want to use the same client to handle all of those installationes. In this case you could pass the service endpoint url as an argument in the installation client call.

DSMInstallationClient version /ProductCode=dsmx /endpoint=”https://<SERVER>/DSMInstallationSevice.svc”

This sample would would retrieve the DSMX product version from a specific service running on a server called <server>.

Log files

The DirectSmile Installation service writes a log file. This log file can be downloaded to the client.

getlog > filename.log

This command would download the log file from the server and stores it locally on the client machine.

DirectSmile Integration Server installation scenarios

I am very happy to see more and more complex installations of the Directsmile Integration Server (DSMI) out in the field. Although you can fairly run DSMI on a single server, in scenarios where you depend on 100% up-time or need to capture extremely high load peeks it can be interesting for you to think about failover scenarios and scalability. In this blog post I tried to capture all the different scenarios I saw in the field as well as their pros and cons.

DSMI replication master vs. slave

Before we go into detail here’s a short reminder about the replication service and the difference between a DSMI master and a slave.

In a DSMI environment that is using replication you have always one server that is the master and one to many slaves. The difference between master and slave is, that the master is the only DSMI were you can create new users, accounts and upload repository items like sets and documents. Clients can request images and documents from all slaves and the master though. This simple architecture is based on the convention that it is acceptable that you cannot upload items or create new users while the master is getting maintained, but this must not effect client calls and you system keeps to be accessible for client calls.

Read more about the DirectSmile Replication Service in those two blog posts:

https://odehne.wordpress.com/2010/09/10/synchronizing-two-or-many-directsmile-integration-servers/
https://odehne.wordpress.com/2011/09/27/changes-in-dsmi-replication-service/

Scalability

The DirectSmile Integration Server is easy to scale. You simply run a base installation of DSMI on, at least, two machines. While the installation you assign one machine as the DSMI master and the other as DSMI slaves (see chapter DSMI replication master vs. slave) . Finally you install the DSMI replication service on one of your machines und your scaled DSMI is ready to rock. The DSMI replication service ensures that changes you make in the master’s file system and the DSMI database are tracked and the slave machines are updated.

To take advantage from your scaled DSMI you place a load balancer in front of your DSMI machines that routes the incoming requests in a schema you define.

Here’s a short graphic showing how such a simple approach would look like.

image

Failover

Based on the architecture shown in the diagram above you see that in this scenario it’s quite easy to run one DSMI as a failover server for the other. From a service client perspective those two DSMI installations look like one. Actually the client can’t access a single machine because the client does it’s calls against a single domain/IP registered on the load balancer. The load balancer then routes the requests to one or the other machine. Theoretically there is no limit in DSMI slave installations in this scenario, typically you would start with one slave and depending on your load you would increase the amount of installations.In a failover scenario it is important that it is ensured that your client requests are served, even if one machine is down.

Single data center installation vs. replication between data centers

Usually you want to install your replicated DSMI in one data center or on your site. Replicating over different data centers can become necessary if you split preview and print data. For example, if you host a web server farm in a data center that only serves preview images and documents for your websites. On the other hand, you have a second DSMI system that is located on-premise in your LAN next to your presses where print data is rendered in high resolution. The advantage of spitting preview and print data reduces the load of data send over the wire from your DSMI servers to your presses.

You can easily escalate such a scenario depending on your failover and load balancing needs. Let’s say you start with a replicated DSMI farm for your previews and a single DSMI on-premise. The on-premise system can consist of several machines again, where one machine would have DSMI installed and the other machines provide just production server instances, like shown in the diagram.

image

As you can see there is a bit more IT involved here. We have to deal with complex routing, and firewall policies must be set to allow the communication between those two systems. This scenario depends on a good connection between different locations. To ensure that the replication is done in a reasonable amount of time your connection bandwidth must be 2MBit or higher in up- and down streaming.

In the end you would have one machine, the DSMI master (which can be located where ever you want, either on the on-premises system or on the web, that replicates the changes to the other server. Though DirectSmile sets and documents are easy maintained exposed to the public).

On top of that you could certainly replicate the on-premises servers as well again and built a second failover cluster.

Another typical scenario where you want to replicate between datacenters is if your company has different branches in different countries and you need to provide preview and print data in all those different  branches. In this case you would replicate internally and externally but still could take advantage from a single point to upload your items and manage the DSMI users and accounts.

Network storage (NAS) vs. server storage

All scenarios described above have one thing in common, that is that the DSMI user folders, containing sets and documents are decentralized. In other words, the same user folder exists on all DSMI servers. This effects first of all the initial replication when you add a new slave to your system though. But, depending on the amount of items you update or add in daily business, this can easily become a number in the equation regarding the load on the wire.

Using a NAS or SAN solution may help you to overcome the problem of a decentralized storage, by providing a single point to store the DSMI user folders.

image

Unfortunately there are some limitations when using a centralized storage system. DSMI relies on Windows file sharing and Windows authentication, that’s why the operating system providing the shares must be Windows based. DirectSmile strongly recommends to use Windows Storage Server 2008, which makes it possible to configure a single Windows user account that is used as a service-user for the DSMI services and can read and write to the shared folders.

Attention:
Providers tend to offer you a standard ISCSI/SAN storage. Those system can’t be mounted simultaneously on more then one Windows Server. This means you will need a Windows Server in between to share the storage. After all you will loose read/write performance and gain a single point of failure.

Conclusion

Running a replicated DSMI cluster for failover and/or deal with a heavy load is possible although the initial setup requires profound understanding of IT related topics like routing and load balancing. Because those scenarios differ strongly based on the network topology it’s important to invest time to plan what’s the best for your situation. Often those scenarios are even bound to the workflows of the campaigns and projects you plan to run.

I hope this article could help you paint an overall picture of what is possible. If you have any questions don’t hesitate to call us. We at DirectSmile are very excited about these kind of projects and love to assist you setting up those installations.