theburningmonk.com

Series so far:

2. Beyond Hello World

3. Par­al­leliz­ing activities

In my pre­vi­ous post I men­tioned some of the short­com­ings with Ama­zon Sim­ple­Work­flow (SWF) which drove me to cre­ate an exten­sion library on top of the stan­dard .Net SDK to make it eas­ier to model work­flows and busi­ness processes using SWF.

In this series of blog posts I’ll give you more exam­ples of how to use the library to model work­flows to be exe­cuted against the SWF ser­vice to take advan­tage of the reli­able state man­age­ment and task dis­patch it offers, but none of the plumb­ing and boil­er­plate code you would have to deal with using the SDK.

Before we start look­ing at exam­ples, let’s have a quick recap of the SWF terminologies:

• A work­flow is a sequence of steps that are loosely strung together by the deci­sions the decider makes each time the state of the work­flow changes. E.g. step 1 com­plete then sched­ule step 2 to commence.
• A work­flow exe­cu­tion is an instance of a par­tic­u­lar work­flow cur­rently being exe­cuted, many exe­cu­tions of the same work­flow (iden­ti­fied by name and ver­sion) can be in flight at the same time. A work­flow exe­cu­tion can be started with string as input and it can return string as output.
• A deci­sion task is a task that is sched­uled each time a workflow’s state changes.
• A decider is a com­po­nent in your appli­ca­tion which is respon­si­ble for polling SWF for deci­sion tasks and respond with deci­sions. The sequence of steps that need to be per­formed by the work­flow is ulti­mately deter­mined by the decider.
• An activ­ity task is a task that is sched­uled by a decider, it takes a string as input (along with sev­eral other pieces of data which it can be sched­uled with) and returns a string as result.
• An activ­ity worker is a com­po­nent in your appli­ca­tion which is respon­si­ble for polling SWF for activ­ity tasks and respond with com­ple­tion or fail­ure sig­nals, as well as pro­vid­ing reg­u­lar heart­beat sig­nals. If the decider is respon­si­ble for sched­ul­ing work to be done, then the activ­ity worker is respon­si­ble for doing the actual work.
• A child work­flow is a work­flow that is sched­uled by the decider as a step in a work­flow, sim­i­lar to an activity.
• The decider is able to sched­ule both child work­flows and activ­i­ties for a sin­gle step in a work­flow, whilst child work­flows can be rerun as an inde­pen­dent unit of work, activ­i­ties can­not be rerun inde­pen­dently out­side of the con­text of a workflow.
• Both work­flows and activ­i­ties need to be reg­is­tered with the SWF ser­vice before they can be used.

These are the most com­mon concepts/components you’ll see in SWF, but there are also less com­monly used (in my opin­ion at least) fea­tures such as:

• Start­ing a timer to cause a timer event to be fired after some time.
• Sig­nalling an exter­nal work­flow exe­cu­tion to cause an event to be recorded in its exe­cu­tion his­tory and a deci­sion task to be sched­uled. This is a use­ful way to allow inter-workflow com­mu­ni­ca­tion, e.g. one work­flow sus­pends itself, until another work­flow sends it a sig­nal and then it can resume with its execution.
• Record­ing a marker as means to pro­vide addi­tional infor­ma­tion in the exe­cu­tion his­tory of a workflow.

Exam­ple : Hello World

Con­sider a work­flow where there is only one activ­ity, which sim­ply prints the input to the screen and echoes it back out.

If we start a work­flow exe­cu­tion with the input “Hello World!” then we expect to see the input being printed to the con­sole and then the work­flow exe­cu­tion com­pleted with the result “Hello World!”.

In its essence, you can think of an activ­ity as noth­ing more than a func­tion which accepts a string as argu­ment and return a string, i.e. a fun with sig­na­ture string –> string in F#.

With the stan­dard .Net SDK you will need to write a decider for each work­flow in order to pro­vide the orches­tra­tion you need for that work­flow. The decider logic tends to quickly become dif­fi­cult to under­stand and main­tain when the deci­sion logic becomes more com­pli­cated, e.g. when mul­ti­ple activ­i­ties and child work­flows are sched­uled in par­al­lel ‚and you need to retry/fail activities/workflows, etc.

In my view, the decider is largely plumb­ing that devel­op­ers should do with­out, so with the exten­sions library you should not need to write any cus­tom decider code but instead, sim­ply declare what activ­i­ties and/or child work­flows should be sched­uled at each stage of a work­flow and let the library do all the heavy lift­ing for you!

As far as work­flow mod­el­ling is con­cerned, the only thing you need to do is use the cus­tom ++> oper­a­tor (inspired by Dave Thomas’s pipelets project) to attach addi­tional steps to your work­flow. So the above work­flow can be mod­elled as:

and that’s it! No need to reg­is­ter the work­flow and activ­ity and write bespoke decider & activ­ity worker your­self, the library does all of that for you, all you needed to do was to model the work­flow you want.

Notice you haven’t had to pro­vide any ref­er­ence to SWF at all thus far, in fact, you only need to pro­vide an instance of Ama­zon­Sim­ple­Work­flow­Client (from the AWS SDK) when you start the workflow:

This way, it’s pos­si­ble to run the work­flow across mul­ti­ple accounts simul­ta­ne­ously (dev, stag­ing, prod, etc.) by call­ing the Start method with each of the client instances (one for each account), which fits well with the mobile worker model SWF is designed with – SWF holds the state but you can run your work­ers from any­where in and out of the AWS ecosystem.

Once you’ve started the work­flow, the library will auto­mat­i­cally reg­is­ter the domain, work­flow and activ­ity for you if they are not present already. You can ver­ify this by look­ing in the SWF Man­age­ment Con­sole:

Notice that whilst we didn’t spec­ify the “echo” activ­ity with a ver­sion num­ber, it’s reg­is­tered with “echo.0”? I’ll go into more details on the ver­sion­ing scheme in a later post, but for now let’s just be glad that we didn’t have to reg­is­ter these by hand!

Next, you can start a work­flow directly from the man­age­ment con­sole, but tick­ing against the work­flow you want to start and click­ing the “Start New Exe­cu­tion” but­ton:

Let’s fol­low through with the dia­logue box and set the input as Hello World! as below:

Once you start the work­flow exe­cu­tion you will see Hello World! being printed in the console:

This is a sign that our echo func­tion (which is invoked by the gen­er­ated activ­ity worker) had been called.

Back in the SWF Man­age­ment Con­sole, if you look under Work­flow Exe­cu­tions, you should see the exe­cu­tion is closed after hav­ing com­pleted successfully:

Click­ing on the work­flow exe­cu­tion ID allows you to see the sequence of events which had been recorded for this execution:

This is a very gran­u­lar view of what hap­pened dur­ing the work­flow exe­cu­tion, giv­ing you plenty of use­ful infor­ma­tion if you ever need to inves­ti­gate why a work­flow exe­cu­tion failed, for instance.

If you switch to the Activ­i­ties tab, you’ll get a more con­densed view with just the activ­i­ties that were sched­uled, along with their inputs, results, etc.

For now, ignore the JSON string in the Con­trol field and the for­mat of the Activ­ity ID, these are both auto­mat­i­cally gen­er­ated by the library based on a set of con­ven­tions and will be cov­ered by a later post.

So that’s it! I hope you can see that this exten­sion library gives you a pow­er­ful way to express and model a work­flow and focus your devel­op­ment efforts on the things that count (design­ing the process and writ­ing the code that does the actual work) rather than wast­ing pre­cious devel­oper time on get­ting your code to work with SWF!

Part­ing Thoughts

For Java devel­op­ers, there is an exist­ing high-level frame­work (pro­vided by Ama­zon itself) for work­ing with SWF called the Flow Frame­work, which adapts a more object-oriented approach and in my opin­ion requires far more plumb­ing and most impor­tantly does not

In case you’re won­der­ing, this is how a solu­tion to a sim­i­lar Hello World exam­ple looks using the flow frame­work (taken straight from the flow frame­work devel­oper guide) for your comparison: