In the previous chapter, we successfully deployed a lambda function that reads from Stack Overflow API and writes to S3. Before we advance the functions of the products, let’s iterate on what we can improve non-functionally.
Before we do that, I think it is important to explain what is the difference between functional and non-functional. These are two things I spent my fair share of time breaking down.
Functionality is available to the end-user of the product to interact with. It is not an internal implementation detail. A single function of a product is indivisible into smaller functions without losing value for the end-user. Currently, there no functions in Stackoverflow Heroes as we are still just building it. There is no interaction with what we have at the moment.
Non-functional (aka technical) is the opposite. It is the internal detail of how exactly a certain function is provided. I.e. a non-functional feature is what database is used, what language, what platform. For instance, the usage of S3 is a non-functional (technical) decision we made about the product. It does not necessarily change how the end-user interacts with Stackoverflow Heroes, but it is important when it comes to the actual implementation.
The misunderstanding of those two terms is very common, especially among engineers. Understandably so — it is sometimes quite difficult to separate the functionality from what you did to achieve that functionality. Especially, when you think long and hard about the implementation. It is what you care about, it is what comes to mind first. Not the end-user, not the business goals.
Good, having that explained (actually, functional vs non-functional deserves a separate article), let’s go back to Stackoverflow Heroes and see what we can improve there. One thing that comes to mind is that we did click quite a lot to deploy the lambda function, setup IAM, S3, and environment variables. Let’s follow the 12 Factor introducing Continuous Deployment to the product!
CI ≠ CD
But we have already done adding GitHub Actions in the previous chapter! Why do we need something more?
Skip this if you know the difference between CI and CD.
So, back then we did the easy part — we simply asked GitHub to run some verifications whenever pushed to master. This is cute, but it is not giving much. Yes, we are continuously testing our code, but, ideally, we want to minimize the human work (and all the related errors) for turning the code into a product. The code becomes a product when it is deployed, so this is why we want continuous deployment.
We will need to deploy the following components:
- Lambda function. Create a new function, upload the code, setup the env variables.
- S3 Bucket — the one that we use for saving the files
- IAM Role and IAM policy to allow the Lambda function to do things with S3 and we also would want logging.
Multiple tools allow you to deploy your components into AWS. There is AWS Cloud Formation, AWS CDK, Serverless framework, Pulumi, and more. I am going with Terraform for Stackoverflow Heroes. What I like about Terraform:
Multicloud support
You can use it successfully be it AWS, GCP, or Azure. It has a vast number of supported providers. Mind you, it does not mean it will be a smooth ride to migrate from one cloud provider to another using Terraform. You do need to rewrite the terraform files, you do need to learn about new resources. Having a cloud-agnostic terraform files is extremely difficult and ever so slightly reasonable.
On a positive side — once you learn Terraform concepts you will be able to reuse them.
Terraform is declarative
In comparison with AWS CDK, Terraform is declarative. You declare a bunch of resources you want and then run Terraform. It will try to reach the state when the real world matches the given description.
AWS CDK takes it up a notch. It uses a whole computer language (Java/Python/C#/JS/TS) for describing the infrastructure. You will have cycles, if statements, and what have you.
I am not so sure it is such a good idea. This reminds me of the difference between Maven and Gradle. In Maven you have XML that is strict and declarative. In Gradle, you are free to code whatever you want. While freedom is good I am not sure it is needed when creating infrastructure. I haven’t seen any good use of cycles or if statements that would apply nicely to the infrastructure code. Moreover, it can be harmful because reading and understanding the code is harder than reading declarations. When I read an infrastructure definition file I want to focus on the resources rather than parsing the code in my head.
Terraform is… big
There are tons of modules and providers available for Terraform. In comparison, the Serverless framework (even though its syntax looks very concise) is very limiting. Terraform’s documentation and samples are extremely helpful, it is not that difficult to learn it.
Let’s see how does Stackoverflow Heroes look in Terraform. We will be using Terraform Modules to keep the infrastructure files along with the code for the resources. That is, we will have a separate Terraform file for the Fetch lambda. This will keep the infra files short and to the point.
Root files
We will have a main.tf file that does global configuration (again, think of Maven and Gradle when you have separate pom.xml/build.gradle files per submodules):
This is written in HCL — language created by Hashicorp. Supposed to be readable by both humans and machines and it kinda is. Here we setup:
- Provider — we will be using aws and we will be using region us-east-1. Change it to the region you like, that’s my default region.
- Module fetch — the location of the module and the variables we need to supply to that module. The variables are the app and secret tokens required by the Stack Overflow API.
- Most importantly, we set up the backend for our Terraform. A couple of words on it (refer to the official doc though). Each Terraform project has a state file that contains the, well, deployment state. What is deployed and what is not. This state then is shared among everyone and everything that attempts to run this Terraform project. In the simplest scenario, when you only run Terraform locally from your computer it is not required to provide a backend. The local file system will be your backend, it will keep the state file pangbian (I mean, “nearby”) the Terraform files. However, since we plan to use Terraform in the continuous deployment process we share the state. You can either run Terraform locally or in the build server (in our case GitHub Actions is the build server). And thus we must share the state to keep things consistent. The easiest way to do it is to keep the state file in some S3 bucket, so anyone with internet access can access the state file. And that’s exactly what we did there — we declared an S3 bucket with a key pointing to some concrete file. If the file is not there the first run will create it. So, create a bucket, pick a name, and let’s have a state in the cloud.
Then, we have a vars.tf file:
We declare two input variables here to pass the secrets to Terraform. As of course we don’t want to hardcode the secrets in a plain text.
Fetch module
In the /fetch folder we also have main.tf and vars.tf files.
You probably noticed that the file structure of Stackoverflow Heroes changed a little bit as well as the Makefile. This is to accommodate the modular structure — we can’t just dump everything into the project’s root folder. The source code itself hasn’t changed though, so no surprises there.
The vars.tf file is not that interesting, so let’s focus on what we have in the fetch/main.tf file:
It is a bunch of stuff here. Let’s go one by one:
Declaration of the lambda resource: configuration of where to get the code for the lambda, parameters, and the environment variables. Notice how we referred to the variables we declared previously here. So, whenever, Terraform is run we can provide secrets and they will be put into lambda. Convenient, I like it a lot, especially in conjunction with GitHub Actions (we will see it later). There is also a tag added for the lambda (and for other resources) — it is just a good practice to have them.
Then we have a resource for the S3 bucket. The only interesting thing here is that we need to be able to shutdown the deployment, so we want to delete all the resources. To delete a non-empty bucket we need to set the force_destroy flag, otherwise, it won’t get destroyed if there are files in there. Let’s have it so that when we take the deployment down we don’t need to care about the files in that bucket.
Finally, we have an IAM role, policy, policy document, and policy document attachment resources. The goal here is to provide permissions to the lambda to work with that S3 bucket we created. The given permissions also enable logging (logs:PutLogEvent) — we want to log from the lambda, it is fine, no big deal.
You probably noticed that the permissions here are a little bit to open (s3:*, logs:*). This is just to simplify things, in the real world, we would be much more restrictive.
That’s it for the Terraform. You can try to run it locally by installing the CLI and then doing terraform init and terraform apply to see the deployment in action from your computer.
Our goal though is not to do it manually, so more GitHub Actions, more automation! Check out the deploy.yml file in /.github/workflows folder of the project:
This flow will be triggered on a push to the master branch (also notice that we changed the just-tests to not run on master push). The first job verify is the same as in just-tests — just verifying tests are okay and that the project is buildable.
The deploy job takes the setup-terraform action to install the Terraform for the job and runs terraform init providing the environment variables required for the deployment. Two important things here:
- The secrets are taken from the GitHub Secrets which is a secrets store in your repository. You can find it in GitHub UI.
- The weird TF_VAR_ prefix is required for all the variables we want to pass to Terraform.
Then, once Terraform is initialized we simply run make deploy that essentially executes terraform apply -auto-approve. The -auto-approve flag is used to not to require user prompt (how would anyone confirm the action when it is run by the build server) on deployment.
Now, whenever we push anything to master we will re-deploy the resources into Terraform! This is neat, it adds some implications, like not too many pushes to master, but still neat.
We have done a lot of work, we improved the project non-functionally. Check it out in GitHub. The question is though — when are we finally going to have something for the user? All of these CI/CDs are cool, but except being proud of putting your code into the Arctic vault — it is useless. Only if the project is used, then we are talking. Let’s focus on that part in the next chapter, see you then!
