Skip to main content

Windows Workflow, child workflows and parallel loops

I recently needed to create a pretty complex workflow to process a large volume of records as quickly as possible. We needed to process 500k records in an hour. We have a scheduled process that triggers the initial workflow that looks for scheduled work. When work is found a call is made out to another system to load the data we need per order (so this can be 1-X order records). Each order record can have 1 to X  number of customer records we need to process. Along the way we have a few user approvals we wait for. In trying to create a Workflow process for this we ran into a few challenges and learnings. Hopefully, this helps some others out there.

At a high level here is what we put together. 

image

The idea here is the Management Workflow is the workflow that is fired off by the schedule. It then can fire off the processing workflow. The green workflows can have multiple instances of them created by its parent.

The Processing Workflow is responsible for finding all the customers that need to be processed for a given order.

The Send workflow is responsible for packaging up all the data for a given customer (calls to CMS system or any other external system) and sending that data to the delivery system.

With these processes able to spin up multiple instances and load balances via IIS we are able to processes thousands of records a second. 

Since these are Windows Workflow Services hosted in IIS we load balance new workflow instances across servers. This approach allows us to scale both out and up to deal with demand. There are some things to keep in mind when setting Windows Workflow up this way though.

Duplex Communication

The first thing we needed in this model is a way for the Processing workflow to spin up child (Send) workflows and to have those child workflows report back on there status.

We used an approach called out on MSDN for Parent-Child Workflow Pattern Using Durable Duplex. Based on this code we were able to create our setup and get this parent child relationship working. To make this work you need to make sure you get your WCF bindings configured correctly and make sure your workflow objects are setup with Callback Correlation correctly. The easily way to get start on this is to download the example (direct link) and really review it.

Throttles

One of the main issues we ran into once we got this setup and running was throttling issues. In each workflow we had a couple Parallel ForEach loops and some async activities. This was one of those situation where we actually slowed ourselves down by trying to go to fast. The workflow would spin so many parallels up and connections to other servers that we killed our active connections causing an app pool reset.

To solve this we had to do a couple of things. First, realize that just because you can parallel does not mean you should parallel. We reduced the number of parallel for each loop we had or manually throttled how many could be created at one time (wish Workflow designer gave you a setting for this). Second, we realized that in production there is some tuning that needs to be done on WCF (these are all Windows Workflow services so WCF). There is a great blog post for Tweaking your WCF apps for high throughput workloads that helped. It calls out how to troubleshoot this issue and the config changes you cam make for MaxConcurrentCalls, MaxConcurrentInstances and MaxConcurrentSessions.

MaxConcurrentInstances (when you follow the steps in the blog) will stop additional ones from spinning up. But we also needed to limit how many items we passed into the parallel so it did not spin up to much at a time. We did this by using a while and simply picking a certain number from our collection and passing those to a parallel for each. As long as the while still had items to pick it kept going back to the parallel.

Tracing

Now that you have these workflows up and running how do you keep track of what they are doing, what state they are in and if the child workflows are healthy? If the child workflows are calling back to the parent what do you do when one dies on you?

I am sure there are a number of ways to tackle this but here is what we did. We used Windows Workflow Foundation tracking we created a custom tracking service so we can track and log the data we need about the workflow to our database. This allows us to track each state of the workflow we care about. So now we can log when a workflow goes into an aborted or unhandledexception state. Keep in mind that if you track everything this will track A LOT of data and write A LOT to the database. So in production it is important to create a custom tracking profile to limit what is logged to only what is required.

Now that you have your workflows logging information to trace you can have your workflows start to monitor that data. In our case we have the parent workflow keep track of what workflows it spins up. Then for each workflow it spins up in goes into a pick activity. One pick branch is triggered when the child calls back. The other pick branch has a heartbeat (a delay followed by a check on the child’s workflow state). If the child workflow state is found to be in an bad state (based on the state logged by the tracker) a flag is set to mark that workflow as unhealthy and the pick branch’s action is triggered. If the child workflow is healthy the heartbeat loops and wait for a configured amount of time before firing off again. If the child workflow stays healthy and calls back the first pick branch’s action is triggered and we go on about our marry way.

Now you have an approach to creating a parent child workflow and monitoring the states of the children. Next up is moving this solution to a load balanced environment (post in progress).

Comments

Popular posts from this blog

Uniting Testing Expression Predicate with Moq

I recently was setting up a repository in a project with an interface on all repositories that took a predicate. As part of this I needed to mock out this call so I could unit test my code. The vast majority of samples out there for mocking an expression predicate just is It.IsAny<> which is not very helpful as it does not test anything other then verify it got a predicate. What if you actually want to test that you got a certain predicate though? It is actually pretty easy to do but not very straight forward. Here is what you do for the It.IsAny<> approach in case someone is looking for that. this .bindingRepository.Setup(c => c.Get(It.IsAny<Expression<Func<UserBinding, bool >>>())) .Returns( new List<UserBinding>() { defaultBinding }.AsQueryable()); This example just says to always return a collection of UserBindings that contain “defaultBinding” (which is an object I setup previously). Here is what it looks like when you want to pass in an exp

Password Management

The need to create, store and manage passwords is a huge responsibility in modern day life. So why is it that so many people do it so poorly? This is a loaded questions with answers ranging from people being uneducated, to lazy, to educated but not affective in their methods and many more. This blog is to help those (in some way even myself) around me strengthen their online security. Why does it matter? To answer this let's look at a few numbers. According to the US Department of Justice (DOJ)’s most recent study , 17.6 million people in the US experience some form of identity theft each year. Ok fine but that is identity theft that has nothing to do with password management. What is one way someone can start getting information about who you are? How do they get access to steal your money? From Cyber Security Ventures 2019 report : "Cybersecurity Ventures predicts that healthcare will suffer 2-3X more cyberattacks in 2019 than the average amount for other industries. W

Excel XIRR and C#

I have spend that last couple days trying to figure out how to run and Excel XIRR function in a C# application. This process has been more painful that I thought it would have been when started. To save others (or myself the pain in the future if I have to do it again) I thought I would right a post about this (as post about XIRR in C# have been hard to come by). Lets start with the easy part first. In order to make this call you need to use the Microsoft.Office.Interop.Excel dll. When you use this dll take note of what version of the dll you are using. If you are using a version less then 12 (at the time of this writing 12 was the highest version) you will not have an XIRR function call. This does not mean you cannot still do XIRR though. As of version 12 (a.k.a Office 2007) the XIRR function is a built in function to Excel. Prior version need an add-in to use this function. Even if you have version 12 of the interop though it does not mean you will be able to use the function. The

Experience Profile Anonymous, Unknown and Known contacts

When you first get started with Sitecore's experience profile the reporting for contacts can cause a little confusion. There are 3 terms that are thrown around, 1) Anonymous 2) Unknown 3) Known. When you read the docs they can bleed into each other a little. First, have a read through the Sitecore tracking documentation to get a feel for what Sitecore is trying to do. There are a couple key things here to first understand: Unless you call " IdentifyAs() " for request the contact is always anonymous.  Tracking of anonymous contacts is off by default.  Even if you call "IdentifyAs()" if you don't set facet values for the contact (like first name and email) the contact will still show up in your experience profile as "unknown" (because it has no facet data to display).  Enabled Anonymous contacts Notice in the picture I have two contacts marked in a red box. Those are my "known" contacts that I called "IdentifyAs"

Anatomy of Sitecore Business Rule - Macros

In previous posts, we talked about  field syntax and the basic structure of business rules . This time we are going to dive into macros in the business rules. Macros are used as part of the business rule syntax. The syntax looks like this and calls for 4 parameters. [Property to set, Operator/Macro, AdditionalParameters, Display text]. When I first started working with business rules the difference between operator and macro was confusing. To add to this confusion some of the out-of-the-box macros are named with the term "operator" (like ListOperator who's configuration points to a class called ListMacro and the class implements IRuleMacro). Anything under the path /sitecore/system/Settings/Rules/Definitions/Macros should be a macro and should implement IRuleMacro. Macros have the follow characteristics: They inherit the IRuleMacro interface The interface requires this execute method void Execute(XElement element, string name, UrlString parameters, string value)