No, you don’t need to test every line of your CDK application

Several people have asked me about unit testing CDK applications recently. The questions often go something like this:

“Because it can be in Python or TS, people bring up unit testing for CDK, whereas it’s not even thought about with Serverless Framework, SAM or Terraform. What are your thoughts on this?”

“I have so many tests for my CDK application that they are kind of a pain to maintain. Am I doing this right?”

“Whenever I change my CDK code I also have to change the test, that just doesn’t feel right. That doesn’t feel right… feels like I’m just repeating myself with these tests.”

These are all valid questions, so let’s take a moment to reflect on them.

Testing with purpose

I’m all for testing. Testing is a big part of my serverless workshop and I have dedicated a lot of time to figuring out the best ways to test serverless architectures.

Tests help us reduce defects and build better software, but they also come with costs. They take time to conceive (coming up with good test cases is not always easy), write and maintain.

You should consider the return on investment with everything you do. Because your time is valuable and one of the most scarce resources for your business.

Testing for testing’s sake is pointless and tests that deliver no value have no place in your codebase.

Lack of testing for IaC

Infrastructure management tools come in many flavours.

Tools such as Serverless framework, SAM and Terraform use a declarative approach. They offer limited or no programmability, so you can’t easily introduce business logic into your infrastructure-as-code configurations. This is a blessing for most because what resources are created and how they are configured should be obvious at a glance. I shouldn’t have to spend hours studying a codebase just to figure out if a Lambda function is provisioned, or what IAM permissions it has.

By and large, IaC tools that rely on a declarative don’t require unit testing because:

a) There is no business logic. There are no “behaviours” that need to be validated using different inputs.

b) CloudFormation already ensures your IaC code is syntactically correct.

But being syntactically correct doesn’t mean your IaC is semantically correct. That is, your resources are configured correctly and give you the behaviour that you expect.

Can you use unit tests to ensure a Serverless/SAM/CloudFormation project is semantically correct?

Technically yes. But in practice, probably not. Here’s why.

Imagine you want an S3 bucket so users can upload files directly from the front end. And this is what you wrote:

MyBucket:
  Type: AWS::S3::Bucket
  Properties:
      BucketName: my-bucket

This is semantically incorrect because, amongst other things, you need to configure a CorsConfiguration.

Can a test catch this? Yes. But you probably missed the CorsConfiguration because you didn’t know that’s what you needed. If that’s the case, then how would you know to test for it?

With the declarative approach to IaC, you say what you want. And semantic incorrectness usually stems from you not knowing what you need to say. The same knowledge gap exists when you write tests. After all, you can’t test for what you don’t know.

These are the reasons why unit testing is not high on the list of conversation topics amongst Serverless and SAM users.

But what about the CDK?

Testing for CDK

The ability to use general-purpose language is great for use cases that really need it. I gave one such example in my previous post on CDK, where a client is able to use CDK to build a modulized, white-label solution.

But most applications don’t need this level of customization. In fact, most CDK applications don’t have much business logic. Most of the time, you “declare” what resources you want and how you want them to be configured.

Like this:

new s3.Bucket(scope, 'Bucket', {
  blockPublicAccess: s3.BlockPublicAccess.BLOCK_ALL,
  encryption: s3.BucketEncryption.S3_MANAGED,
  enforceSSL: true,
  versioned: true,
})

Again, is there any value in testing this bit of code?

Where there is no business logic, what exactly would you be testing?

The official CDK documentation includes some unit testing examples that show how you can validate the synthesized CloudFormation template.

But I think these tests are backwards…

If you have to know the synthesised CloudFormation template to trust your CDK code then why not use CloudFormation to begin with? All the typing you saved by using CDK is added back in with the tests.

If you’re copying and pasting the synthesised CloudFormation into your tests (to save on typing…) then the tests would tell you nothing.

With this type of testing, anytime you change your CDK code you will also have to change the corresponding tests. It’s double the work with no meaningful returns.

If you’re using the official CDK constructs and “declaring” (no branching logic or loops, etc.) the resources you want then you likely won’t need to test your CDK code.

On the other hand, if you’re creating custom constructs or embedding business logic into your CDK application then you should be testing them thoroughly!

Governance vs Testing

Finally, it’s worth making a distinction between governance and testing. Most enterprises have to comply with security and regulatory frameworks. These would translate to a set of rules that affect the configuration of AWS resources. For example, DynamoDB tables and S3 buckets need to enable server-side encryption and S3 buckets cannot be publically accessible.

You can include these in your CDK tests. But seeing as these rules affect every team and every CDK application, it creates a lot of duplication. And it’s easy to forget, especially for new joiners (who aren’t aware of these requirements) and when teams are under time pressure to deliver.

It’s much better to implement these compliance requirements as guardrails that are baked into your AWS environments. AWS provides lots of tools to help you in this regard. For example:

• You can use AWS Config to monitor and check AWS resources against your compliance requirements.
• You can use AWS Organizations and account factory to ensure AWS Config and all the relevant rules are enabled in every account.
• You can use Security Hub to monitor non-compliance in a central place and allow your platform team to monitor your entire AWS estate.

And these guardrails are usually owned by platform teams. One downside to doing all these with AWS Organizations is that it involves a lot of clickops… This is where I would turn to org-formation, an open-source tool that gives you infrastructure-as-code for AWS Organizations. It’s a really powerful tool and I strongly recommend that you check it out if you haven’t already.

Wrap up

I hope you have found this article insightful. I hope I’ve managed to challenge you to think about testing CDK applications and where it makes sense, rather than testing purely for the sake of testing.

If you want to learn more about building serverless architecture, then check out my upcoming workshop where I would be covering topics such as testing, security, observability and much more.

Hope to see you there.

Oh, and if you sign up now, you will also get access to my serverless testing course for free. Details are on the workshop page here.