You can become a serverless blackbelt. Enrol in my course Learn you some Lambda best practice for great good! and learn best practices for performance, cost, security, resilience, observability and scalability. By the end of this course, you should be able to make informed decisions on which AWS service to use with Lambda and how to build highly scalable, resilient and cost efficient serverless applications.
Since joining JUST EAT I have been much more active in attending meetups because our office is walking distance to the Code Node. In particular, I have been a regular at Phil Trelford’s F#unctional Londoners group which meets twice a month.
So, combining the inspiration (and code ) from both, I decided to create a twitter bot for generating random arts.
— RandomArtsBot (@randomartsbot) January 23, 2016
I had a few goals in mind when I sat down to write the bot:
- periodically publish randomly generated images (like what the LSystem Bot does for L-Systems)
- support a simple DSL for expressing the maths equations used to generate the images
- allow others to tweet to it in the above DSL and reply with the generated image
Let’s start with the AST, the syntax of the DSL is just how we get to the AST. This is an extended set of the expressions that Phil shared with us in his session:
One approach would be to use a natural maths syntax, e.g.
- x + y
- tan y
- cos x * const
- y + sqr x * sin y
The challenge with this approach is to clearly define and correctly support the precedence rules.
Another approach I considered is to use a LISP syntax, e.g.
- (+ x y)
- (tan y)
- (* (cos x) const)
- (+ y (* (sqr x) (sin y)))
This approach is slightly more verbose, but has the benefit of simplicity and precedence is very obvious. It’s also very easy to write a S-Expression parser, which also factored into my decision to go with this approach.
To go from the AST to the S-Expression is super easy, just override the ToString() method like this:
Now that the easy parts are done, let’s write a S-Expression parser for our DSL.
Broadly speaking, the two common approaches for writing parsers in F# is to use FParsec or vanilla active patterns. Having imagined how I’d write the parser with both approaches I decided the FParsec route is simpler.
The following 25 lines of code would build the foundation for the whole DSL, spend a moment or two to see if you can work out what it’s doing.
OK, going back to the AST, we can break things down into 4 categories:
- primitives : x, y, and cost
- unary functions : sin, cos, tan, well, tent, sqr and sqrt
- binary functions : add, subtract, product, divide, max, min, average and mod
- ternary functions : level and mix
In the code snippet above, we first created parsers for ( and ), then parsers for x, y and const. But this line is interesting…
This allows for a recursive parser for our recursive AST (expression is defined in terms of functions, which are defined in terms of expressions, and so on).
For all the unary functions we have right now and in the future, their structure is the same:
( func_name some_expr )
now see how we parse them:
up until the |>> operator we have captured the Expr object to invoke op with, whatever op is.
Interesting, so op is any function that takes Expr as argument. That’s convenient, because every clause of a union type is a function in its own right. For example, in our AST, Sin is a function that takes an Expr and returns an Expr..
Aha, so we can stay DRY and use the unaryOp, binaryOp and ternaryOp functions above to create corresponding parser for the unary, binary and ternary functions in our AST
To wrap things up on the DSL, we have to define the implementation for our Expr parser:
The use of the attempt function here is significant here, otherwise, each failed attempt to parse an expression would have consumed the leading parenthesis ‘(‘ and cause all subsequent parsers to also fail.
Since I have “borrowed” heavily from both Phil and Mathias’s work, much of the building blocks I needed to write the bot was already there. But I also encountered a few new challenges along the way though.
Random is not threadsafe
When you use an instance of System.Random from multiple threads you can mess up its internal state, it’ll then always return 0 which is hardly random at all…
Since I decided to go async from the start I was able to reproduce this problem regularly. The solution was to simply instantiate a new instance of Random before I need to generate a new formula or draw another image.
Formulae are usually too long to fit inside a tweet
The randomly generated formulae are almost always too long and couldn’t fit into a tweet. To increase the chance of generating formulae that are tweet-sized, I limited the max depth of the expression generation logic to 6. For example:
(- (/ (tan (tan (* (tan const) (+ y const)))) (tan (avg (cos (cos const)) (+ (+ const x) (tan x))))) (cos const)) pic.twitter.com/214gR7NEU4
— RandomArtsBot (@randomartsbot) January 24, 2016
Now, roughly around half the generated formulae can fit into the 140 characters limit.
Many boring images
Another thing that quickly became obvious was that, many of the generated formulae would yield a similar and kinda boring images (mostly a black screen). So, to improve the general quality of the bot’s output, I added another step in the pipeline to inspect the generated bitmap.
For now, this step would filter out images that are mostly a black screen.
Loopy conversation with another bot
It’s easy to get into a loop between two bots, and so far I have some rudimentary measures in place to stop the conversation. However it is just as likely to prematurely stop the conversation with an innocent user who has made an error in his/her tweet.
There’s still work to be done to find the right balance here, if you’ve got some ideas/suggestions, I’d love to hear about them in the comments below.
If someone tweets at the bot then they’re probably showing an interest in it. Let’s auto-follow them, and maybe they’ll follow us back
Supporting direct messages
Unsurprisingly, after a few people became aware of the bot’s existence I found someone sent direct messages to the bot to see if they can get an image back via DM as well as tweets. Sadly it’s not yet supported, but it’s a trivial effort to add so expect to see this in the near future.
So that’s it folks, a quick update on what I did last weekend, if you’d like to give the bot a try then feel free to send tweets to @RandomArtsBot. You can find the documentation for the DSL syntax here, or just tweet “help” at the bot.
— RandomArtsBot (@randomartsbot) January 23, 2016
I specialise in rapidly transitioning teams to serverless and building production-ready services on AWS.
Are you struggling with serverless or need guidance on best practices? Do you want someone to review your architecture and help you avoid costly mistakes down the line? Whatever the case, I’m here to help.
Check out my new podcast Real-World Serverless where I talk with engineers who are building amazing things with serverless technologies and discuss the real-world use cases and challenges they face. If you’re interested in what people are actually doing with serverless and what it’s really like to be working with serverless day-to-day, then this is the podcast for you.
Check out my new course, Learn you some Lambda best practice for great good! In this course, you will learn best practices for working with AWS Lambda in terms of performance, cost, security, scalability, resilience and observability. We will also cover latest features from re:Invent 2019 such as Provisioned Concurrency and Lambda Destinations. Enrol now and start learning!
Check out my video course, Complete Guide to AWS Step Functions. In this course, we’ll cover everything you need to know to use AWS Step Functions service effectively. There is something for everyone from beginners to more advanced users looking for design patterns and best practices. Enrol now and start learning!
Are you working with Serverless and looking for expert training to level-up your skills? Or are you looking for a solid foundation to start from? Look no further, register for my Production-Ready Serverless workshop to learn how to build production-grade Serverless applications!
Here is a complete list of all my posts on serverless and AWS Lambda. In the meantime, here are a few of my most popular blog posts.
- Lambda optimization tip – enable HTTP keep-alive
- You are wrong about serverless and vendor lock-in
- You are thinking about serverless costs all wrong
- Just how expensive is the full AWS SDK?
- Many faced threats to Serverless security
- We can do better than percentile latencies
- Yubl’s road to Serverless
- AWS Lambda – should you have few monolithic functions or many single-purposed functions?
- AWS Lambda – compare coldstart time with different languages, memory and code sizes
- Guys, we’re doing pagination wrong
- Top 10 Serverless framework best practices