Announcing AWS Step Functions - December 2016 Monthly Webinar Series
•Descargar como PPT, PDF•
5 recomendaciones•1,561 vistas
AWS Step Functions is a new, fully managed service that makes it easy to coordinate the components of distributed applications and microservices using visual workflows. Step Functions is a reliable way to coordinate components and step through the functions of your application at scale. This session shows how to use Step Functions to create, run, and debug multi-service applications in a matter of minutes. We also share how customers are using Step Functions to build and operate complex applications, such as order processing, report generation, and data transformation – and to innovate faster. Learning Objectives: • Learn about the capabilities and features of AWS Step Functions • Learn about the benefits of AWS Step Functions • Learn about the different use cases • Learn how to get started using AWS Step Functions • Build and coordinate the components of your application as a series of steps • Link AWS Lambda functions or Amazon EC2 instances together to create multi-step apps • Learn to automatically catch and recover from errors • Operate and scale your application in a reliable fashion
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Destacado
Destacado (20)
Similar a Announcing AWS Step Functions - December 2016 Monthly Webinar Series
Similar a Announcing AWS Step Functions - December 2016 Monthly Webinar Series (20)
Más de Amazon Web Services
Más de Amazon Web Services (20)
Último
Último (20)
Announcing AWS Step Functions - December 2016 Monthly Webinar Series
- 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Announcing AWS Step Functions Andy Katz, Senior Product Manager September 2016 Tim Bray, Senior Principal Engineer
- 2. Enterprise Resource Planning Finance Inventory Manu- facturing � � � � Sales From Monoliths to Microservices to Functions
- 5. “I want to sequence functions” “I want to select functions based on data” “I want to retry functions” “I want try/catch/finally” “I have code that runs for hours” “I want to run functions in parallel” Functions into apps
- 6. Scales out Doesn’t lose state Deals with errors/timeouts Easy to build & operate Auditable Coordination must-haves
- 7. AWS Step Functions… …makes it easy to coordinate the components of distributed applications using visual workflows.
- 8. Benefits of AWS Step Functions Diagnose and debug problems faster Adapt to change Easy to connect and coordinate distributed components and microservices to quickly create apps Manages the operations and infrastructure of service coordination to ensure availability at scale, and under failure Productivity Agility Resilience
- 9. How Does AWS Step Functions Work?
- 10. Application Lifecycle in AWS Step Functions Visualize in the Console Define in JSON Monitor Executions Amazon Confidential
- 11. Define in JSON and Then Visualize in the Console { ”Comment”: “Hello World Example", "StartAt” : "HelloWorld”, "States” : { "HelloWorld” : { "Type” : "Task", "Resource” : "arn:aws:lambda:REGION:ACCOUNT_ID:function :FUNCTION_NAME”, "End” : true } }
- 12. Execute One or One Million Start End HelloWorld Start End HelloWorld Start End HelloWorld Start End HelloWorld Start End HelloWorld Start End HelloWorld Start End HelloWorld Start End HelloWorld
- 13. Monitor Executions from the Console
- 14. Monitor Executions from Amazon CloudWatch
- 15. “I want to sequence functions” AWS Step Functions, we can easily change and iterate on the application workflow of our food delivery service in order to optimize operations and continually improve delivery times. AWS Step Functions lets us dynamically scale the steps in our food delivery algorithm so we can manage spikes in customer orders and meet demand. Mathias Nitzsche, CTO, “
- 18. var Twit = require('twit'); var T = new Twit(require('botfiles/config.js')); exports.handler = function myBot(event, context, callback) { var list = event.inputList; var textToTweet = list.shift(); var output = { inputList: list } T.post('statuses/update', { status: textToTweet }, function(err, reply) { if (err) { console.log('error:', err); context.fail(); } else { console.log('tweet:', reply); callback(null, output); } }); };
- 19. Demo
- 20. “I want to select functions based on data” “With AWS Step Functions, it was easy to build a multi-step product updating system to ensure our database and website always have the latest price and availability information. AWS Step Functions let us replace a manual updating process with an automated series of steps, including built-in retry conditions and error handling, so we can reliably scale before a big show, and keep pace with rapidly changing fashions.” Jared Browarnik, CTO, TheTake “
- 22. "ChoiceState": { "Type" : "Choice", "Choices": [ { "Variable": "$.productSource", "StringEquals": "screen-scrape", "Next": "ScreenScrapeState" },{ "Variable": "$.productSource", "StringEquals": "vendor-a", "Next": "VendorA" },{ "Variable": "$.productSource", "StringEquals": "vendor-b", "Next": "VendorB" }, { "Variable": "$.productSource", "StringEquals": "vendor-c", "Next":"VendorC" },{ "Variable": "$.productSource", "StringEquals": "updateProduct", "Next":"UpdateProduct" } ], "Default": "ScreenScrapeState” }
- 23. “I want to retry functions” We get transient errors from a RESTful service we depend on, once every four or five times we call it. But if we keep retrying, it eventually works. “
- 24. { "Comment": "Call out to a RESTful service", "StartAt": "Call out", "States": { "Call out": { "Type": "Task", "Resource": "arn:aws:lambda:eu-central-1:123456789012:function:RestCallout", "Retry": [ { "ErrorEquals": [ "HandledError" ], "MaxAttempts": 10 } ], "End": true } } }
- 25. “I want try/catch/finally” AWS Step Functions makes it simple to coordinate information from many different infrastructure systems using easy to design workflows and create a more intelligent monitoring system for our Platform as a Service (PaaS). With AWS Step Functions, we can reliably automate monitoring decisions and actions in order to reduce human intervention by over 60%, which improves infrastructure operation productivity and customer application availability on our platform. Pedro Pimenta, VP R&D, OutSystems “
- 26. 13 AWS Lambda Task States 6 Choice States 1 Fail State “I want try/catch/finally”
- 27. Try/Catch/Finally for Media Management
- 28. "Access Media": { "Type": "Task", "Resource": "arn:aws:lambda:eu-central-1:123456789012:function:FindMedia", "TimeoutSeconds": 2, "Next": "Graceful Exit", "Retry": [ { "ErrorEquals": [ "States.Timeout" ], "IntervalSeconds": 2, "MaxAttempts": 2, "BackoffRate": 1.5 } ], "Catch": [ { "ErrorEquals": [ "States.ALL" ], "Next": "Clean Up" } ] },
- 29. Demo
- 30. “I have code that runs for hours” We need to gather data from our production line, in units of 8-hour shifts. “
- 31. Task States Poll or Push AWS Lambda…On-premises AWS container, EC2 instance, or… Long poll Push
- 32. Task State to Dispatch Work with Push { "StartAt": "HelloWorld", "States": { "HelloWorld": { "Type": "Task", "Resource":"arn:aws:lambda:REGION: ACCOUNT_ID:function:FUNCTION_NAME", "End": true } } }
- 33. Task State to Dispatch Work by Polling { "StartAt": "HelloWorld", "States": { "HelloWorld": { "Type": "Task", "Resource":"arn:aws:states:REGION: ACCOUNT_ID:activity:FUNCTION_NAME", "End": true } } }
- 34. "NextShift": { "Type": "Wait", "TimestampPath": "$.ShiftStart", "Next": "Gather Plant Data" }, "Gather Plant Data": { "Type": "Task", "Resource": "arn:aws:states:ap-northeast-1:123456789012:activity:PlWatch", "TimeoutSeconds": 30000, "HeartBeatSeconds": 120, "Next": "Clean up" }
- 35. “I want to run functions in parallel” We want to send the captured image to three OCR providers and take the result with the highest confidence value.“
- 37. "Send for OCR": { "Type": "Parallel", "Next": "Pick result", "Branches": [ { "StartAt": "Prep1", "States": { "Prep1": { "Type": "Pass", "Result": { "inputList": [ "OCR Provider 1" ] }, "Next": "Go1" }, "Go1": { "Type": "Task", "Resource": "arn:aws:lambda:eu-central-1:123456789012:function:StatesBot", "End": true } }
- 38. Where does transient application state live? In the machine, in JSON texts passing from state to state.A: Q:
- 39. Input processing { "title": "Numbers to add", "numbers": [ 3, 4 ] } { "Type": "Task", "InputPath": "$.numbers", "Resource": "arn:aws:lambda…" … [ 3, 4 ] Raw input: State spec: Task input:
- 40. Input processing Q: InputPath not provided? A: State gets raw input as-is. Q: InputPath is null? A: State gets an empty JSON object: {} Q: InputPath produces plural output? A: State gets it wrapped in a JSON array.
- 41. Result placement { "title": "Numbers to add", "numbers": [ 3, 4 ] } { "Type": "Task", "InputPath": "$.numbers", "ResultPath": "$.sum”, … { "title": "Numbers to add", "numbers": [ 3, 4 ], ”sum": 7 } Raw input: State spec: Task input:
- 42. Result placement Q: ResultPath is null? A: State input is state output. Q: ResultPath produces plural output? A: Not allowed, validator won’t accept. Q: ResultPath not provided? A: Input discarded, raw output used.
- 43. “I want to sequence functions” “I want to select functions based on data” “I want to retry functions” “I want try/catch/finally” Is this you? “I have code that runs for hours” “I want to run functions in parallel”
- 44. Where is AWS Step Functions Available? Region Region Code US East (N. Virginia) us-east-1 US East (Ohio) us-east-2 US West (Oregon) us-west-2 EU (Dublin) eu-west-1 Asia Pacific (Tokyo) ap-northeast-1
- 46. Getting Started
- 48. Thank you!
Notas del editor
- Many of you have already or perhaps are about to experience this journey? Applications being built on a human scale. From the firm – the monolith--to departments, to individuals, to tasks… And it makes technical sense, too. Greater flexibility, more resilient,
- In 2014, the launch of AWS Lambda expanded our minds, with its core idea of a stateless function in the cloud, reacting to an event. But there aren’t that many apps with only one function, one entry point, one module, one component. So there’ll be more than one function. NEXT In fact, I suspect that it’ll be common to have LOTS of functions, lots of them talking to each other. NEXT And in fact applications, serverless or not, tend to have databases. NEXT And in the cloud, I notice that a lot of them have queues of one kind or another – that gold thing is the iconfor Amazon Simple Queue Service, SQS, which I sometimes work on, and which I’m seeing a lot of in Cloud-Native apps. NEXT And, now I’m going to do something that’s maybe in bad taste. This is the serverless track, but the world still does have servers. And this is a cloud conference, but some of those servers aren’t in the cloud. NEXT Now we’re starting to see a picture of what an actual modern serverless app might really look like.
- You’ll notice that those arrows are in different colors – that’s because there are lots of ways for functions and other serverless components to talk with each other and control each other and orchestrate each other. Before we dive into the “how”, let’s talk about WHAT we want to do with functions when we have more than one.
- These are the things that people are trying to accomplish with those different-colored arrows, and I’m not gonna spend too much time diving in, because each one will get a little love of its own as we go through this story. But I suspect anyone who’s tried to build a nontrivial serverless app around functions in the cloud has dealt with a few of these problems. I know I have. So what are the techniques that people are using today to make these things happen?
- So what I’m saying is that function co-ordination in the serverless world is possible, but it’s too ad-hoc and too difficult. So if we were going to automate all that, what are the must-haves. I suspect most fo these are pretty non-controversial or rather crushingly obvious, but I think it’s the last two that are really important. The things we’re trying to accomplish are easy to describe in words, so they should be easy for a developer to accomplish. And one of the things that *none* of the ad-hoc alternatives have is a good audit trail or log file. So, anyhow, that list is exactly what this announcement and this talk are all about.
- AWS Step Functions makes it easy to coordinate the components of distributed applications using visual workflows.
- ProductivitySpend more time thinking about innovating the business logic that makes your application unique, and your applications are easier to operate and maintain. AgilityAWS Step Functions records a history of each execution, so you can review in one place, all the events in sequence, in one location. Scale instantly from one execution to hundreds of thousands of concurrent executions, especially when used with other serverless AWS resources such as AWS Lambda, Amazon S3, and Amazon DynamoDB. With AWS Step Functions, you pay only for what you use, when you use it. ResilienceAWS Step Functions supports automatic error handling for graceful exits, and operates at scale without you needing to configure or manage its underlying resources.
- With AWS Step Functions, you define your workflow as a series of steps and transitions between each step, also known as a state machine. In Step Functions, we call these “states” and “state transitions”. A simple state machine looks like this, and you define it using simple commands in JSON. When you start a state machine, you pass it input in the form of JSON, and each state changes or adds to this JSON blob as output, which becomes input to the next state. The console provides this visualization and uses it to provide near-real-time information on your state machine execution. Let’s look at each of these more closely.
- This is a classic Hello World example. This code on the left, generates this graph on the right. We specify where we start, we define each state, and we define each state transition. And then we get back to working on the code that makes our apps unque.
- I’ll talk a lot about individual state machines and single executions. The power of Step Functions is that you can define a state machine once, and then run thousands and thousand of concurrent executions. This allows you to break down big tasks into a set of smaller, simpler tasks.
- The console provides all kinds of information. Anything you see in the console, with the exception of the graph, is also accessible through the API. Let’s step through the elements. In the upper right, I can tab between my graph and my JSON code. This graph is color coded to show me successfully executed states, my current state in progress, failed states, or states not passed during exection. In the upper right panel, I get details about my execution. I can see general info, the input I gave to the execution, in the form of JSON key/value pairs, and the final output of the execution. The Step Details toggle at the bottom, allows me to inspect individual states. I can see the input and the output state by state. This is really useful in debugging when something unexpected occurs. Finally, at the bottom is the complete history my execution, step by step with timestamps. Again, this is accessible from the API as well as the console.
- I can also set up Amazon CloudWatch to monitor my executions. Here I am monitoring a different state machine that invoked an AWS Lambda function about 70 times per minute and I can see that the Lambda functions each executed in about 200 – 300 mS.
- FoodPanda is a take-out food service that is focusing on the developing world, and are really making excellent use of Cloud Infrastructure. They have been in the Step Functions beta, and their problem is deceptively simple. They get food orders and have delivery people and get the dinners taken to the hungry. Of course, optimizing this which requires solving the NP-complete traveling salesman problem, but they’re you know, smart. Anyhow, they want to do this with Lambda functions.
- So here’s the actual state machine they built. Most of the states are executed by Lambda tasks. But I want to zero in on one small part of the problem, in the middle there where they run their assignment code and then dispatch a vehicle. Which is an obvious basic thing that many people want to do all the time; run this Lambda function, then run another. So let’s zero on in a small state machine that does just that. Also, if I only look at those two states, I’ll have something small enough to fit all of on one slide.
- I don’t feel the slightest bit guilty about investing time in this moronically simple sequencing problem, because I think it’s actually a strong real-world use case. Often, the first step you take after you deploy a cool Lambda function, is you realize that wait, what I want to do is run THIS function then run THAT function. This sounds easy and it should be easy. So there’s a visual rendition of the machine on the top half, and a JSON expression of it on the bottom. This is an actual screen grab from the Step Functions console. The picture is self-explanatory, so let’s look at the JSON. A state machine’s top level, NEXT along with a comment, has a structure called “States” that contains, well, all the States. And a string field StartAt that says which state to do first. In this case we start with “RunAssignmentAlgorithm”, NEXT which is a Task state that runs a Lambda, then has a Next field saying go to “Compute destinations”, which runs another Lambda, and has an End = true field, so that’s the whole machine. It’s easy enough. Step Functions goes through the states, executes each, and moves along to the next. Obviously it can get fancier,. But let’s use this one to dive into how these things work. I didn’t include foodpanda’s actual Lambda function, I whipped them out and substituted demo-ware. And in fact it’s the same function twice. So let’s look at it.
- Hope most of you can read NodeJS code. Basically, this function does a Tweet to prove it really ran. But I want to focus on the input and output. So, ignore the first couple of lines for now and let’s start here. NEXT The input is an arbitraryJSON blob, and we grab its inputList field, which in this machine is a list of strings. NEXT We assign the first string to the the variable “textToTweet”, and pop that off the front of the list. Then we tweet that text, and assuming everything went OK, NEXT we output whatever was left of the list after we popped off the first string. It turns out this whole notion of input and output is important, so leave a bookmark in your mind, and let’s watch this thing work. Now, everybody knows that demos fail, so I made a screencast of me running the demo, which I could show you, but I noticed that the Internet in here wasn’t too bad, especially if you use the secret speakers’ WiFi. So let’s take a chance and try the live demo before I give up and run the screencast.
- Our customer here is TheTake, who are in the business of movie-based marketing – if you see Matt Damon looking cool in a movie, you can find out what he’s wearing. Anyhow, they work with Amazon, which is nice, but they also go to a few other places, and if all else fails, they resort to screen-scraping. And, they want to do it all with Lambdas. So let’s see how they do this.
- There’s a picture of their state machine. Some of those boxes are Lambda functions. It’s all pretty straightforward, and of course the interesting part is that box at the top labeled “ChoiceState”, where they decide which way they’re going to branch. Let’s dive in and look at the JSON form of that part of their machine, which doesn’t actually involve Lambdas.
- I hope it’s pretty self-explanatory. <pause> So the Choice has four branches. Maybe the most interesting thing about it is that field labeled “Variable”. NEXT Remember how in that last machine, the input to the state was a list of strings? Well, it turns out that the data we pass from state to state is an arbitrary JSON text. So in this case, for their Choice, in the first branch, it looks at the input JSON’s Variable field, and the value has to be a JsonPath that selects one of the fields out of the input data. Then it takes that value and does the String Equals, and if it gets true, takes the “Next” branch. If that doesn’t work it goes to the next choice branch and tries that. It turns out that along with StringEquals there’s a full assortment of logical comparators: Integer, boolean, timestamp, and so on. And of course, there are And/Or/Not connectors so you can build hideous deeply-nested booleans that I would have no chance of fitting on a slide here at re:Invent So I think it’s pretty obvious how the Choice state works. Once again, I want to emphasize this notion passing data along from one state to another in the machine. Generally, each state gets some input, which it can update or replace, and pass along to the next state.
- Now, speaking for myself, my software always works first time and never ever fails. But in the cloud, we have Dependencies. I’ve noticed that when you’re trying to dodge the blame for something, dependencies are very helpful. Seriously, in the real world, functions actually fail. And I’ve noticed that in distributed, asynchronous, cloud-native applications, from time to time, well, uh, “stuff” happens. I got this story from a customer who relies on an external RESTful service that sometimes just blows up. Now that’s a DEPENDENCY. So in this case, let’s just jump straight to the JSON.
- So, to illustrate, this is just a one-state machine. This Task state runs a Lambda called RestCallout, but then NEXT it’s got a “Retry”, that say “hey, if this blows up, keep trying, as many as ten times. I’m not going to dive into the deep details – Notice that the Retry value is actually an array, so you can have a bunch of error handlers for different values that do different things. Let’s see this thing run.
- OutSystems automates their infrastructure using AWS Step Functions to integrate alarm information from many sources. I mentioned already that we have a retry capability, but sometimes, things just break. In most programming languages, we have a try/catch/finally idiom, and I’m pretty convinced that serverless apps are going to need that as well. So let’s dive in and see how to do that.
- So, what we have here is a serious state machine that does some heavy lifting. But it does a lot of try/catch/finally. To spare your eyes, and to make it easy for me to fit things into a re:Invent slide, let’s look at a much smaller macine that shows this off.
- This is a screen grab of a running state machine, an app that accesses a media file over the network and in an ideal world, just makes a graceful exit. But if something goes wrong, it drops into Clean Up mode, and then exits. But sometimes even the Cleanup breaks, and at least we can let the humans know when that happens. So once again, let’s look at the JSON.
- So, here’s your Try/Catch/Finally. It’s a task state that runs a lambda, but it’s got a new thing, a TimeoutSeconds field, that limits how long the state is allowed to run. After two seconds it’ll throw a States.Timeout error. You’ve already seen a Retry state. In this one we catch that States.Timeout error, and will retry twice, once after 2 seconds, and once after 3 seconds. You noticed that in that earlier Retry example, I didn’t have all the backoff parameters. You can leave them out and Step Functions will use reasonable defaults. Anyhow, if we retry twice, and the FetchMedia lambda still didn’t get the job done, eventually the error falls through and hits the Catch clause, which redirects to another state called Clean UP that I’m not showing here; it runs a cleanup function and if *THAT* fails, then it redirects to last gap ”finally” sort of state. OK, let’s look at the last thing we hear about from our customers.
- I want to highlight that readout there at the bottom of the screen. As our applications get more and more heterogeneous and have more and more moving parts, something like Step Functions becomes more and more useful. One reason is that it also provides a global log of everything that went on; when each step started, what its input was, whether it succeeded or not, what its output was – and with everything timestamped. Thoroughly greppable, too. See? State machines are your friends! So, we’ve talked about running machines in sequence, and branching based on data, and now dealing with errors. What else? Well, Lambdas are stateless functions in the cloud, and the moment I hear the word “stateless” I think “I want to run them in parallel!”
- I mentioned already that we have a retry capability, but sometimes, things just break. In most programming languages, we have a try/catch/finally idiom, and I’m pretty convinced that serverless apps are going to need that as well. So let’s dive in and see how to do that.
- AWS Step Functions passes information contained in the input to the stateless Lambda functions and activity workers in order to give them context to do their work. They pass back their results, which are stored in AWS Step Functions, and then “forget about it”, since Step Functions is robustly tracking state. You workers may be EC2 instances or containers, AWS Lambda functions, or servers on premises – anything that can communicate with the Step Functions API may request tasks, complete and return.
- Here’s an example. The JSON is human readable and there are four common commands: “comment”, “type”, “next”, and “end”. This JSON makes this graph, the canonical HelloWorld example.
- Here’s an example. The JSON is human readable and there are four common commands: “comment”, “type”, “next”, and “end”. This JSON makes this graph, the canonical HelloWorld example.
- So here’s an example of a Task state, just like all those others we’ve been looking at that had Lambda functions. But if you look closely at that ARN, you’ll notice that it’s different, it refers to an activity. Also, you can see that aside from what’s inside the state definition, there’s nothing but the ARN details that says what kind of a Task this is. So in the future you can imagine other kinds of tasks, aside from lambda functions and poll-for-work activities, that Step Functions might be able to manage. Anyhow, the other interesting thing here are the time values. In this case, if the activity runs for more than a little over eight hours, we decide it’s broken and time it out. We did that before in the try/catch/finally example. But there’s also HeartBeatSeconds. The idea is that the activity task needs to convince us it’s still alive and working, so we require it to hearbeat, using that API I mentioned on the last slide, regularly, and if it doesn’t for two minutes, we time the task out. This stuff turns out to be super-useful in practice. Raises the question “How long can a State Machine run?” In theory, forever. In practice, one year.
- This customer has an app where one of their customers takes a picture of a reward code with their phone and enters the contest by sending it in. What they wants to do is OCR it, but OCR is a bit of a chancy business, so what they do is send the picture off to multiple OCR services and take the result that’s got the highest confidence value. And obviously, the customer’s waiting, they don’t want to do do these in sequence.
- So, here’s a picture of their state machine, mostlybuilt out of Lambdas. There are three branches, each with a couple of states, and then you get the three OCR results back and the Pick Result state decides which OCR result to go with. Easy enough to understand. Once again, most of these states are Lambdas, but not all. It turns out that the ”Go” states, Go1, Go2, and Go3, all actually use the same Lambda function, which uses a standardized callout interface for the OCR conversion. So the Prep states, Prep1, Prep2, and Prep3, are there to set them up to point at the right OCR service. Let’s dive into the JSON and have a closer look.
- Now, I can’t fit the whole machine on this slide, so let’s zero in on on where it does that parallel branching. So there’s a Parallel state, which has a bunch of Branches, and each one is a mini-state machine, with a bunch of states and a StartAt Pointer. I’ve managed to squeeze Branch 1 onto this slide, so let’s go look into that. This branch has two states. The first one is called Prep1, and it’s not a Task state, it’s a “Pass” state, which basically doesn’t do anything. But, it has this “Result” field, and that becomes the output of the Pass state. So in this case, it passes a JSON structure containing the string to the Go1 state, which actually invokes the Lambda that calls out to the OCR function. People who look closely can spot that the Lambda function I’m actually using is the one that sends tweets, so if I wanted to run a demo of this, you’d get three tweets posted at almost exactly the same time, showing that the OCR Provider 1, Provider 2, and Provider 3, branches got called. We think Parallel is a really big deal for state machines, and that customers will probably test the limits of our system by sending Parallel states that have thousands of branches, nested within other parallel states that have thousands of branches.
- So I’m assuming you’ve noticed a pattern here. Step Functions is all about managing application state in the cloud. Now, a large part of the application state lives in your app’s databases and queues and so on. But Step Functions’ big value-add is that it takes care of all the transient state, and encodes it in that JSON that’s being logically passed from state to state. Now if you look under the covers, it turns out that what we’re doing is stashing that in a database, just exactly the way that people often co-ordinate functions these days by stashing state in databases. Except for, we’re doing it for you and letting you make declarative state machines. So the basic idea is that a state machine execution has an inpu in the form of JSON. Since I am the editor of the IETF’s latest JSON RFC you will allow me to be a bit pedantic. To be exactly correct, the input to a state machine, and also the input to and output from any state, has to be a “JSON Text”. If you are a real JSON weenie, you’ll already know that this means it doesn’t have to be an object or an array – ordinary text containing a number or a quotation-mark delimited string or whatever is just fine. Anyhow, the state machine has an input text, and which becomes the input to the first state, and then by default that state’s output becomes the input to the next state, and the last state’s output from the machine. Only sometimes you don’t want the default, you want to interfere with the JSON coming into or going out of a state. In fact, I wouldn’t be surprised, in complicated state machines, if there were the occasional state containing a Lambda that just rearranges the JSON to prepare it for later processing. But let’s look at some of the built-in stuff we’re providing.
- OK, so let’s imagine a state that adds numbers. And in this particular machine, the data being passed along is that JSON object in the top box. But suppose that state doesn’t know about JSON objects, it just wants ot get an array of numbers to add up. Which is probably good design. NEXT So, we can use the InputPath field, which is a JsonPath, to select the piece of the input that we want to feed to the state. In this case, give it just the array of numbers that it needs.
- Some frequently-asked questions about this stuff. Sometimes you don’t want to fool with the input. In which case, you just don’t use InputPath and nothing happens. Sometimes a state just doesn’t care about its input, it’s all set up with its task in a database or a queue or whatever, so say InputPath: null, and there just won’t be any useful input. Now, if you’re a JsonPath expert, it turns out you can write one that takes a slice of an array, or all the fields in an object – so it returns more than one JSON text. Since our rules say that the input always has to be a JSON text, we just wrap it in square brackets and turn it into an array. This is just bookkeeping but it turns out to hit a pretty good 80/20 point. Now, what about the stuff coming OUT of a state?
- OK, let’s revisit that lambda function that adds numbers. Let’s suppose that it gets an array of numbers as input, and as output it just gives you a sequence of numbers representing the result – in this case, the single digit 7. Remember, this is a valid JSON text. So, if we don’t do anything the output of the state would just be that single digit. But in a lot of cases, we don’t want to discard the input, we want to enrich it. So NEXT we provide a ResultPath field here, that says “keep the input, but put the result of the task here in the input”. So you can see what happens. This turns out to be super-useful – I saw one app where they had JSON that represented a customer and they had a state to normalize the phone number, and another to look up the email address and so on, so the JSON got richer and richer as it worked its way through the machine. Now, some FAQ’s for Result processing
- By default the output of the stae is just whatever its task spits out. So if you have no resultPath, that’s what you’ll get. Sometimes you don’t actually care about the output from the state, it was only there for its side-effects. Just say ResultPath: null, and that’ll pass the input through to the output. Here’s one tricky case – the JsonPath that says where in the input to put the task output? It can’t have array slices or wild-cards, it has to point at one particular place in the input to put the output. So that’s enough input/output bookkeeping help. Oh I should mention that there is also an OutputPath field, which can apply a JsonPath to the JSON on the way out of the state, but it’s most just there for completeness, doesn’t seem to be that widely used. Let’s go back to some customer situations and how they used stae machines to address them.
- From what customers tell us, I bet that there are actually a lot of you. Now, this hasn’t been a complete tour of Step Functions, but I think I’ve hit most of the high points. I’ll be honest the Product Managers have visions of people running huge machines that grind away for days and choreograph hundreds of Lambda invocations. What would actually make me happy is to see Step Functions replace lots and lots of ad-hoc shell scripts that people are using for configuration and control and cleanup and other miscellaneous tasks.
- As of December 1, we are available in five AWS regions across the world: EU (Dublin), US East (N. Virginia), US East (Ohio), US West (Oregon), and Asia Pacific (Tokyo).
- Anyhow, that’s more or less all, folks. It’s here today in five regions, feel free to dive in and use it. We think it’s easy.
- You can get started the classical way. We have a developer guide, and you’ll find the JSON language specification as well as a validation tool called Statelint.
- Or you could go to the console and try building a hello world state machine in a few minutes.