I recently needed to create a pretty complex workflow to process a large volume of records as quickly as possible. We needed to process 500k records in an hour. We have a scheduled process that triggers the initial workflow that looks for scheduled work. When work is found a call is made out to another system to load the data we need per order (so this can be 1-X order records). Each order record can have 1 to X number of customer records we need to process. Along the way we have a few user approvals we wait for. In trying to create a Workflow process for this we ran into a few challenges and learnings. Hopefully, this helps some others out there.
At a high level here is what we put together.
The idea here is the Management Workflow is the workflow that is fired off by the schedule. It then can fire off the processing workflow. The green workflows can have multiple instances of them created by its parent.
The Processing Workflow is responsible for finding all the customers that need to be processed for a given order.
The Send workflow is responsible for packaging up all the data for a given customer (calls to CMS system or any other external system) and sending that data to the delivery system.
With these processes able to spin up multiple instances and load balances via IIS we are able to processes thousands of records a second.
Since these are Windows Workflow Services hosted in IIS we load balance new workflow instances across servers. This approach allows us to scale both out and up to deal with demand. There are some things to keep in mind when setting Windows Workflow up this way though.
Duplex Communication
The first thing we needed in this model is a way for the Processing workflow to spin up child (Send) workflows and to have those child workflows report back on there status.
We used an approach called out on MSDN for Parent-Child Workflow Pattern Using Durable Duplex. Based on this code we were able to create our setup and get this parent child relationship working. To make this work you need to make sure you get your WCF bindings configured correctly and make sure your workflow objects are setup with Callback Correlation correctly. The easily way to get start on this is to download the example (direct link) and really review it.
Throttles
One of the main issues we ran into once we got this setup and running was throttling issues. In each workflow we had a couple Parallel ForEach loops and some async activities. This was one of those situation where we actually slowed ourselves down by trying to go to fast. The workflow would spin so many parallels up and connections to other servers that we killed our active connections causing an app pool reset.
To solve this we had to do a couple of things. First, realize that just because you can parallel does not mean you should parallel. We reduced the number of parallel for each loop we had or manually throttled how many could be created at one time (wish Workflow designer gave you a setting for this). Second, we realized that in production there is some tuning that needs to be done on WCF (these are all Windows Workflow services so WCF). There is a great blog post for Tweaking your WCF apps for high throughput workloads that helped. It calls out how to troubleshoot this issue and the config changes you cam make for MaxConcurrentCalls, MaxConcurrentInstances and MaxConcurrentSessions.
MaxConcurrentInstances (when you follow the steps in the blog) will stop additional ones from spinning up. But we also needed to limit how many items we passed into the parallel so it did not spin up to much at a time. We did this by using a while and simply picking a certain number from our collection and passing those to a parallel for each. As long as the while still had items to pick it kept going back to the parallel.
Tracing
Now that you have these workflows up and running how do you keep track of what they are doing, what state they are in and if the child workflows are healthy? If the child workflows are calling back to the parent what do you do when one dies on you?
I am sure there are a number of ways to tackle this but here is what we did. We used Windows Workflow Foundation tracking we created a custom tracking service so we can track and log the data we need about the workflow to our database. This allows us to track each state of the workflow we care about. So now we can log when a workflow goes into an aborted or unhandledexception state. Keep in mind that if you track everything this will track A LOT of data and write A LOT to the database. So in production it is important to create a custom tracking profile to limit what is logged to only what is required.
Now that you have your workflows logging information to trace you can have your workflows start to monitor that data. In our case we have the parent workflow keep track of what workflows it spins up. Then for each workflow it spins up in goes into a pick activity. One pick branch is triggered when the child calls back. The other pick branch has a heartbeat (a delay followed by a check on the child’s workflow state). If the child workflow state is found to be in an bad state (based on the state logged by the tracker) a flag is set to mark that workflow as unhealthy and the pick branch’s action is triggered. If the child workflow is healthy the heartbeat loops and wait for a configured amount of time before firing off again. If the child workflow stays healthy and calls back the first pick branch’s action is triggered and we go on about our marry way.
Now you have an approach to creating a parent child workflow and monitoring the states of the children. Next up is moving this solution to a load balanced environment (post in progress).
Comments