Terraform for Cloud State Management: Simplifying Infrastructure as Code

Introduction

Managing cloud infrastructure manually is often complex, inconsistent, and error-prone. Recently, I explored Terraform, an Infrastructure as Code (IaC) tool by HashiCorp, which automates these tasks and makes the process repeatable and trackable.

In this blog series of my project, I'll share what I've learned about using Terraform for cloud state management on AWS. We'll explore its benefits, address common challenges, and provide step-by-step guides to simplify your cloud operations.

Scenario:

As part of a Cloud Management Team, you're responsible for deploying and managing cloud infrastructure. The current manual setup is complex, inconsistent, and hard to track, leading to frequent errors and inefficiencies.

Problem:

• Complex Setup: Numerous steps and configurations.

• Inconsistent: Unique setups causing inconsistencies.

• Tracking Difficulties: No centralized system.

• Human Errors: Prone to mistakes.

• Inefficient: Slow and resource-intensive management.

Architecture and Tools Used

For this project, I have used several tools and resources:

• Terraform: For configuration management and infrastructure as code.

• AWS Services: Including Elastic Beanstalk, VPC, EC2, RDS, Amazon MQ, Elastic Cache, and S3.

• GitHub Repositories:

  • Source code of app

  • Terraform files

Architecture Diagrams:

This diagram shows the VPC setup, subnets, route tables, NAT gateways, EC2 instances, and other services.

This diagram illustrates the integration of AWS services with Terraform.

Step 1: Generating Key Pairs and Setting Up an S3 Bucket for State Management.

For secure access and to manage our infrastructure, we start with generating SSH key pairs and setting up an S3 bucket.

Generating Key Pairs

Generate SSH key pairs using the ssh-keygen command:

This command will create a private key vprofilekey and a public key vprofilekey.pub.

Creating Key Pair in AWS

Create a file named keypairs.tf with the following content to create the key pair in AWS:

resource "aws_key_pair" "vprofilekey" {
  key_name   = "vprofilekey"
  public_key = file(var.PUB_KEY_PATH)
}

Apply the configuration with:

terraform apply

Terraform will confirm the creation of the key pair.

S3 Bucket for State Management

First we manually create a s3 bucket "teraa-app-state" on AWS.

Then create a file named backend.tf with the following content:

terraform {
  backend "s3" {
    bucket = "teraa-app-state"
    key    = "terraform/backend"
    region = "us-east-2"
  }
}

This configuration tells Terraform to use an S3 bucket named teraa-app-state in the us-east-2 region to store its state file.

Initializing Terraform

Run the following command to initialize Terraform:

terraform init

Terraform will confirm the creation of the s3 bucket. Now, you have created the key pair for secure access and set up the S3 bucket for state management.

Step 2: Writing the Providers and Variables Files

In this step, we'll write the providers and variables configuration files for our Terraform setup. These files define the AWS provider settings and the variables we'll use throughout our Terraform configurations.

Providers File

Create a file named providers.tf with the following content:

provider "aws" {
  region                  = var.REGION
  max_retries             = 10
  skip_metadata_api_check = true
}

Explanation:

• provider "aws": Specifies that we're using AWS as the provider.

• region = var.REGION: Sets the AWS region using a variable named REGION.

• max_retries = 10: Configures Terraform to retry API requests up to 10 times in case of failure.

• skip_metadata_api_check = true: Skips the metadata API check, which can help speed up initialization in some environments.

Variables File

Create a file named vars.tf with the following content:

variable "REGION" {
  default = "us-east-2"
}

variable "AMIS" {
  type = map(any)
  default = {
    us-east-2 = "ami-09040d770ffe2224f"
    us-east-1 = "ami-09040d770ffe2224f"
  }
}

variable "PRIV_KEY_PATH" {
  default = "vprofilekey"
}

variable "PUB_KEY_PATH" {
  default = "vprofilekey.pub"
}

variable "USERNAME" {
  default = "ubuntu"
}

variable "MYIP" {
  default = "49.36.200.72/32"
}

variable "rmquser" {
  default = "rabbit"
}

variable "rmqpass" {
  default = "Gr33n@pple123456"
}

variable "dbuser" {
  default = "admin"
}

variable "dbpass" {
  default = "admin123"
}

variable "dbname" {
  default = "accounts"
}

variable "instance_count" {
  default = "1"
}

variable "VPC_NAME" {
  default = "vprofile-VPC"
}

variable "Zone1" {
  default = "us-east-2a"
}

variable "Zone2" {
  default = "us-east-2b"
}

variable "Zone3" {
  default = "us-east-2c"
}

variable "VpcCIDR" {
  default = "172.21.0.0/16"
}

variable "PubSub1CIDR" {
  default = "172.21.1.0/24"
}

variable "PubSub2CIDR" {
  default = "172.21.2.0/24"
}

variable "PubSub3CIDR" {
  default = "172.21.3.0/24"
}

variable "PrivSub1CIDR" {
  default = "172.21.4.0/24"
}

variable "PrivSub2CIDR" {
  default = "172.21.5.0/24"
}

variable "PrivSub3CIDR" {
  default = "172.21.6.0/24"
}

Explanation:

• variable "REGION": Defines the default AWS region.

• variable "AMIS": Specifies a map of AMI IDs for different regions.

• variable "PRIV_KEY_PATH" and variable "PUB_KEY_PATH": Paths to the private and public SSH keys.

• variable "USERNAME": Default username for the instances.

• variable "MYIP": Your IP address for SSH access.

• variable "rmquser" and variable "rmqpass": RabbitMQ username and password.

• variable "dbuser", variable "dbpass", and variable "dbname": Database credentials and name.

• variable "instance_count": Number of instances to create.

• variable "VPC_NAME": Name of the VPC.

• variable "Zone1", variable "Zone2", variable "Zone3": Availability zones.

• variable "VpcCIDR", variable "PubSub1CIDR", variable "PubSub2CIDR", variable "PubSub3CIDR", variable "PrivSub1CIDR", variable "PrivSub2CIDR", variable "PrivSub3CIDR": CIDR blocks for the VPC and subnets.

These files set up the basic configuration needed for our Terraform scripts to interact with AWS and define the variables that will be used throughout the Terraform configurations.

Step 3: Creating a VPC

To set up a Virtual Private Cloud (VPC) on AWS, we use a Terraform module from the Terraform AWS modules registry. This module simplifies the creation of a VPC and associated resources.

VPC Configuration File

Create a file named vpc.tf with the following content:

module "vpc" {
  source = "terraform-aws-modules/vpc/aws"

  name = var.VPC_NAME
  cidr = var.VpcCIDR

  azs             = [var.Zone1, var.Zone2, var.Zone3]
  private_subnets = [var.PrivSub1CIDR, var.PrivSub2CIDR, var.PrivSub3CIDR]
  public_subnets  = [var.PubSub1CIDR, var.PubSub2CIDR, var.PubSub3CIDR]

  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Terraform   = "true"
    Environment = "Prod"
  }

  vpc_tags = {
    Name = var.VPC_NAME
  }
}

Explanation:

• source = "terraform-aws-modules/vpc/aws": Specifies the VPC module from the Terraform AWS modules registry.

• name = var.VPC_NAME: Sets the name of the VPC using a variable.

• cidr = var.VpcCIDR: Defines the CIDR block for the VPC.

• azs, private_subnets, public_subnets: Define the availability zones and the CIDR blocks for the private and public subnets.

• enable_nat_gateway, single_nat_gateway: Enable and configure NAT gateway.

• enable_dns_hostnames, enable_dns_support: Enable DNS hostnames and DNS support.

• tags, vpc_tags: Add tags to the resources for identification and organization.

Running Terraform Apply

After defining the vpc.tf file, apply the configuration with:

terraform apply

Terraform will create the VPC and associated resources as defined in the configuration:

You can verify the creation of the VPC in the AWS Management Console:

By using the Terraform module registry, we efficiently created the entire VPC stack with minimal configuration. This approach simplifies the process and ensures best practices are followed.

Step 4: Creating Security Groups

To secure our AWS infrastructure, we need to set up security groups for various components such as the Beanstalk load balancer, bastion host, EC2 instances, and backend services. These security groups control the inbound and outbound traffic for the associated resources.

Security Groups Configuration File

Create a file named secgrp.tf with the following content:

resource "aws_security_group" "vprofile-bean-elb-sg" {
  name        = "vprofile-bean-elb-sg"
  description = "Security group for bean-elb"
  vpc_id      = module.vpc.vpc_id

  egress {
    from_port   = 0
    protocol    = "-1"
    to_port     = 0
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port   = 80
    protocol    = "tcp"
    to_port     = 80
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_security_group" "vprofile-bastion-sg" {
  name        = "vprofile-bastion-sg"
  description = "Security group for bastion EC2 instance"
  vpc_id      = module.vpc.vpc_id

  egress {
    from_port   = 0
    protocol    = "-1"
    to_port     = 0
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port   = 22
    protocol    = "tcp"
    to_port     = 22
    cidr_blocks = [var.MYIP]
  }
}

resource "aws_security_group" "vprofile-prod-sg" {
  name        = "vprofile-prod-sg"
  description = "Security group for Beanstalk instances"
  vpc_id      = module.vpc.vpc_id

  egress {
    from_port   = 0
    protocol    = "-1"
    to_port     = 0
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port       = 22
    protocol        = "tcp"
    to_port         = 22
    security_groups = [aws_security_group.vprofile-bastion-sg.id]
  }
}

resource "aws_security_group" "vprofile-backend-sg" {
  name        = "vprofile-backend-sg"
  description = "Security group for RDS, ActiveMQ, Elastic Cache"
  vpc_id      = module.vpc.vpc_id

  egress {
    from_port   = 0
    protocol    = "-1"
    to_port     = 0
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port       = 0
    protocol        = "-1"
    to_port         = 0
    security_groups = [aws_security_group.vprofile-prod-sg.id]
  }

  ingress {
    from_port       = 3306
    protocol        = "tcp"
    to_port         = 3306
    security_groups = [aws_security_group.vprofile-bastion-sg.id]
  }
}

resource "aws_security_group_rule" "sec_group_allow_itself" {
  type                     = "ingress"
  from_port                = 0
  to_port                  = 65535
  protocol                 = "tcp"
  security_group_id        = aws_security_group.vprofile-backend-sg.id
  source_security_group_id = aws_security_group.vprofile-backend-sg.id
}

Explanation:

• aws_security_group "vprofile-bean-elb-sg": Security group for the Beanstalk load balancer, allowing HTTP traffic on port 80.

• aws_security_group "vprofile-bastion-sg": Security group for the bastion host, allowing SSH access on port 22 from a specific IP.

• aws_security_group "vprofile-prod-sg": Security group for Beanstalk instances, allowing SSH access from the bastion host.

• aws_security_group "vprofile-backend-sg": Security group for backend services like RDS, ActiveMQ, and Elastic Cache, allowing internal traffic and MySQL access from the bastion host.

• aws_security_group_rule "sec_group_allow_itself": Rule to allow traffic within the backend security group itself.

Applying the Configuration

After writing the secgrp.tf file, apply the configuration with:

terraform apply

Terraform will create the specified security groups and rules. You can verify the creation of the security groups in the AWS Management Console.

By defining these security groups, we ensure that our infrastructure components can securely communicate with each other and are protected from unauthorized access.

Step 5: Initializing Backend Services

In this step, we'll set up the backend services using Amazon RDS, ElastiCache, and Amazon MQ. These services will be initialized and configured to work within our VPC.

Backend Services Configuration File

Create a file named backend-services.tf with the following content:

resource "aws_db_subnet_group" "vprofile-rds-subgrp" {
  name       = "main"
  subnet_ids = [module.vpc.private_subnets[0], module.vpc.private_subnets[1], module.vpc.private_subnets[2]]
  tags = {
    Name = "Subnet group for RDS"
  }
}

resource "aws_elasticache_subnet_group" "vprofile-ecache-subgrp" {
  name       = "vprofile-ecache-subgrp"
  subnet_ids = [module.vpc.private_subnets[0], module.vpc.private_subnets[1], module.vpc.private_subnets[2]]
}

resource "aws_db_instance" "vprofile-rds" {
  allocated_storage      = 20
  storage_type           = "gp2"
  engine                 = "mysql"
  engine_version         = "5.7.44"
  instance_class         = "db.t3.micro"
  username               = var.dbuser
  password               = var.dbpass
  parameter_group_name   = "default.mysql5.7"
  multi_az               = "false"
  publicly_accessible    = "false"
  skip_final_snapshot    = true
  db_subnet_group_name   = aws_db_subnet_group.vprofile-rds-subgrp.name
  vpc_security_group_ids = [aws_security_group.vprofile-backend-sg.id]
}

resource "aws_elasticache_cluster" "vprofile-cache" {
  cluster_id           = "vprofile-cache"
  engine               = "memcached"
  node_type            = "cache.t2.micro"
  num_cache_nodes      = 1
  parameter_group_name = "default.memcached1.6"
  port                 = 11211
  security_group_ids   = [aws_security_group.vprofile-backend-sg.id]
  subnet_group_name    = aws_elasticache_subnet_group.vprofile-ecache-subgrp.name
}

resource "aws_mq_broker" "vprofile-rmq" {
  broker_name        = "vprofile-rmq"
  engine_type        = "ActiveMQ"
  engine_version     = "5.15.16"
  host_instance_type = "mq.t2.micro"
  security_groups    = [aws_security_group.vprofile-backend-sg.id]
  subnet_ids         = [module.vpc.private_subnets[0]]

  user {
    username = var.rmquser
    password = var.rmqpass
  }
}

Explanation:

• aws_db_subnet_group "vprofile-rds-subgrp": Creates a subnet group for RDS within the specified private subnets.

• aws_elasticache_subnet_group "vprofile-ecache-subgrp": Creates a subnet group for ElastiCache within the specified private subnets.

• aws_db_instance "vprofile-rds": Configures an RDS instance with MySQL, specifying storage, engine version, instance type, and credentials.

• aws_elasticache_cluster "vprofile-cache": Sets up an ElastiCache cluster using Memcached, specifying node type, number of nodes, and security settings.

• aws_mq_broker "vprofile-rmq": Configures an Amazon MQ broker using ActiveMQ, specifying instance type, version, security settings, and user credentials.

Applying the Configuration

After writing the backend-services.tf file, apply the configuration with:

terraform apply

Terraform will initialize and configure the backend services as specified. You can verify the creation and configuration of these services in the AWS Management Console.

By setting up these backend services, we ensure our application has access to the necessary database, caching, and messaging infrastructure. This completes the initialization of our backend services using Terraform.

Step 6: Initializing Elastic Beanstalk

To set up AWS Elastic Beanstalk, we need to define the application, its environment, and the necessary IAM roles for permissions. We'll create three Terraform files: bean-app.tf, bean-env.tf, and iamrole.tf.

Application Configuration File

Create a file named bean-app.tf with the following content:

resource "aws_elastic_beanstalk_application" "vprofile-prod" {
  name        = "vprofile-prod"
  description = "vprofile production environment"
}

Explanation:

• aws_elastic_beanstalk_application "vprofile-prod": Defines the Elastic Beanstalk application.

Environment Configuration File

Create a file named bean-env.tf with the following content:

resource "aws_elastic_beanstalk_environment" "vprofile-bean-prod" {
  name                = "vprofile-bean-prod"
  application         = aws_elastic_beanstalk_application.vprofile-prod.name
  solution_stack_name = "64bit Amazon Linux 2 v3.3.6 running Tomcat 8.5 Corretto 11"

  settings {
    namespace = "aws:autoscaling:launchconfiguration"
    name      = "IamInstanceProfile"
    value     = aws_iam_instance_profile.elasticbeanstalk_instance_profile.name
  }

  settings {
    namespace = "aws:autoscaling:launchconfiguration"
    name      = "EC2KeyName"
    value     = var.PRIV_KEY_PATH
  }

  settings {
    namespace = "aws:elasticbeanstalk:environment"
    name      = "ServiceRole"
    value     = aws_iam_role.elasticbeanstalk_service_role.name
  }

  settings {
    namespace = "aws:elasticbeanstalk:environment"
    name      = "EnvironmentType"
    value     = "LoadBalanced"
  }

  settings {
    namespace = "aws:autoscaling:asg"
    name      = "MinSize"
    value     = "1"
  }

  settings {
    namespace = "aws:autoscaling:asg"
    name      = "MaxSize"
    value     = "4"
  }

  settings {
    namespace = "aws:autoscaling:updatepolicy:rollingupdate"
    name      = "RollingUpdateEnabled"
    value     = "true"
  }

  settings {
    namespace = "aws:elb:listener"
    name      = "InstancePort"
    value     = "80"
  }

  settings {
    namespace = "aws:elb:listener"
    name      = "InstanceProtocol"
    value     = "HTTP"
  }

  settings {
    namespace = "aws:elb:listener"
    name      = "LoadBalancerPort"
    value     = "80"
  }

  settings {
    namespace = "aws:elb:listener"
    name      = "LoadBalancerProtocol"
    value     = "HTTP"
  }
}

Explanation:

• aws_elastic_beanstalk_environment "vprofile-bean-prod": Defines the environment for the Elastic Beanstalk application.

• The settings block specifies various configurations for the environment, including instance profile, key name, service role, environment type, scaling settings, rolling update policy, and load balancer settings.

IAM Role Configuration File

Create a file named iamrole.tf with the following content:

resource "aws_iam_role" "elasticbeanstalk_service_role" {
  name = "elasticbeanstalk-service-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "elasticbeanstalk.amazonaws.com"
        }
        Action = "sts:AssumeRole"
      }
    ]
  })

  managed_policy_arns = [
    "arn:aws:iam::aws:policy/service-role/AWSElasticBeanstalkEnhancedHealth",
    "arn:aws:iam::aws:policy/service-role/AWSElasticBeanstalkService"
  ]
}

resource "aws_iam_instance_profile" "elasticbeanstalk_instance_profile" {
  name = "elasticbeanstalk-instance-profile"
  role = aws_iam_role.elasticbeanstalk_service_role.name
}

Explanation:

• aws_iam_role "elasticbeanstalk_service_role": Creates an IAM role with permissions to manage Elastic Beanstalk environments.

• aws_iam_instance_profile "elasticbeanstalk_instance_profile": Creates an instance profile for the IAM role.

Applying the Configuration

After writing the bean-app.tf, bean-env.tf, and iamrole.tf files, apply the configuration with:

terraform apply

Terraform will initialize and configure the Elastic Beanstalk application, its environment, and the necessary IAM roles:

You can verify the creation and status of the Elastic Beanstalk environment in the AWS Management Console:

By setting up Elastic Beanstalk, we simplify the deployment and management of our application, taking advantage of AWS's automated scaling and load balancing features.

Step 7: Initializing Bastion Host and Database

In this step, we will set up a bastion host using Terraform, which will help us initialize the database. We will use the bastion host to provision a template file that sets up the database.

Bastion Host Configuration File

Create a file named bastion-host.tf with the following content:

resource "aws_instance" "vprofile-bastion" {
  ami                    = var.AMIS[var.REGION]
  instance_type          = "t2.micro"
  key_name               = var.PRIV_KEY_PATH
  subnet_id              = module.vpc.public_subnets[0]
  vpc_security_group_ids = [aws_security_group.vprofile-bastion-sg.id]

  tags = {
    Name = "vprofile-bastion"
  }
    provisioner "file" {
    content     = templatefile("templates/db-deploy.tmpl", { rds-endpoint = aws_db_instance.vprofile-rds.address, dbuser = var.dbuser, dbpass = var.dbpass })
    destination = "/tmp/vprofile-dbdeploy.sh"
  }

  provisioner "remote-exec" {
    inline = [
      "chmod +x /tmp/vprofile-dbdeploy.sh",
      "sudo /tmp/vprofile-dbdeploy.sh"
    ]
  }

  connection {
    user        = var.USERNAME
    private_key = file(var.PRIV_KEY_PATH)
    host        = self.public_ip
  }
  depends_on = [aws_db_instance.vprofile-rds]
}

Explanation:

• aws_instance "vprofile-bastion": Configures the bastion host EC2 instance.

• ami, instance_type, key_name, subnet_id, vpc_security_group_ids: Define the instance's AMI, type, key pair, subnet, and security group.

• provisioner "remote-exec": Runs template file on the bastion host to install required software, clone the Git repository, and initialize the database.

Database Deployment Template File

Create a file named db-deploy.tmpl with the following content:

sudo apt update
sudo apt install git mysql-client -y
git clone -b vp-rem https://github.com/huzaifh02/vprofile-app-terraform
mysql -h ${rds-endpoint} -u ${dbuser} --password=${dbpass} -e "CREATE DATABASE IF NOT EXISTS accounts;"
mysql -h ${rds-endpoint} -u ${dbuser} --password=${dbpass} accounts --ssl-mode=DISABLED < /home/ubuntu/vprofile-project/src/main/resources/db_backup.sql

This script is used for provisioning, which updates the instance, installs MySQL, clones the repository, creates the database, and imports the database backup using MySQL commands.

Applying the Configuration

After writing the bastion-host.tf and db-deploy.tmpl files, apply the configuration with:

terraform apply

Terraform will create the bastion host and provision the database initialization script:

You can verify the creation and status of the bastion host in the AWS Management Console:

By setting up the bastion host and provisioning the database, we ensure that our backend database is properly initialized and ready for use. This completes the initialization of the bastion host and database using Terraform.

Step 8: Build and Deploy Artifact

1. Clone the Repository:

    git clone https://github.com/huzaifh02/vprofile-app-terraform
   

2. Build the Artifact Using Maven:

    mvn clean install
   

   This compiles the code, runs tests, and packages the application into vprofile-v2.war.

   

3. Locate the Artifact:

    cd target
    ls
   

   

4. Deploy to Elastic Beanstalk:

   • Navigate to Elastic Beanstalk in the AWS Console.

   • Select vprofile-bean-prod environment.

   • Click "Upload and deploy".

   • Upload vprofile-v2.war.

5. Verify Deployment:

   Ensure the environment status is "Ok" in the AWS Console.

   

We have successfully built and deployed your application using Terraform and AWS Elastic Beanstalk.

Step 9: Validate Services

After deploying all components, the final step is to validate that all services are functioning correctly.

1. Login Page: Ensure the login page is accessible and functioning. Users should be able to log in successfully.

   

2. Profile Page: Check that the user profile page displays the correct information.

   

3. Cache Validation: Verify that data is being cached correctly. For example, when accessing user data, it should indicate whether the data is from the cache or database.

   

4. Database Integration: Ensure the database integration is functioning by confirming data retrieval and storage operations.

   

5. RabbitMQ: Confirm that RabbitMQ is properly set up and handling messages as expected.

   

These validations ensure that all components are working together seamlessly, verifying the integrity and functionality of the deployed infrastructure.

Conclusion

Working through this project has been an enlightening experience. Leveraging Terraform for cloud state management has significantly streamlined the deployment process, making it more efficient and error-free. By automating infrastructure as code, we were able to ensure consistency, repeatability, and ease of management for our AWS resources.

Key takeaways from this project include:

• Efficiency: Automating infrastructure tasks with Terraform reduces the time and effort required for setup and maintenance.

• Consistency: Using Terraform ensures that all environments are set up in a consistent manner, eliminating discrepancies and potential issues.

• Scalability: Infrastructure as code allows for easy scaling and modifications to the infrastructure without manual intervention.

• Collaboration: Storing configurations in version control systems like GitHub enables better collaboration among team members.

Incorporating Terraform into our cloud management practices has proven to be a game-changer, and I highly recommend it for any DevOps toolkit. Whether you're managing a small application or a large-scale enterprise system, Terraform can help you achieve greater control and flexibility over your cloud infrastructure.

For detailed code and configuration files, you can refer to my GitHub repositories:

• Source Code of App

• Terraform Files

I hope this guide has been helpful, and I look forward to exploring more DevOps tools and practices in future projects.