Devops Flashcards
1
Q
What is devops?
A
It’s a practice or culture adopted by your organization, that would increase your organization’s ability to deliver applications(Continuous delivery)
But is devops only about improving delivery? No
1. Improving delivery
2. Automation - Deliver more chips
3. Quality - Customer cares about this
4. Monitoring - How do you make sure automation and quality is being maintained? You have to build in monitoring. So whenever there’s an issue, someone has to report back to us. This is done by monitoring.
5. Testing - Without testing, we cannot ensure that the quality or automation is correct.
So the proper definition is - Devops is a process of improving your application delivery by ensuring that there is a proper automation, the application quality is maintained, ensure there is continuous monitoring and continuous testing in place. So this is devops.
What do we achieve with all these things? - As a devops engineer, your goal is to ensure there should not be any manual process or during your application delivery, you have to fasten your delivery process.
2
Q
Why Devops?
A
To improve the process of delivery. when there were bare metal servers, it used to take many days to deploy and multiple people were involved in the process. It’s a manual process with server admin, system admin, build and release engineer and few other people. To speed up and automate this process, we adopted devops.
As a devops engineer, I’m focused on automating and improving the building, testing and deployment phase. There should not be any manual intervention.
3
Q
How to introduce yourself as a devops engineer?
A
- Im working as a java + devops engineer from 5 yrs.
- Current Roles and responsibilities - I take care of automation, quality, monitoring. I have automated the testing process into the devops lifecycle.
4
Q
What is SDLC?
A
It’s a culture/ methodology to design, develop and test the application. The end goal is to deliver a high quality product.
Stages:
Planning - Requirements. BA does this
Defining - Documenting
Designing - High level and low level design
Building
Testing
Deploying
5
Q
What is hypervisor?
A
A software that can install VM’s on your computer/bare metal servers.
It does logical isolation/separation on the server.
Each VM has its own portion of CPU, hardware and memory
6
Q
You can automate the creation of VM’s or EC2 instances through AWS CLI. What are other ways?
A
- AWS API. In Python, using Boto3 module you can directly make a request to the AWS API.
- AWS CFT - Cloud Formation Template. If you provide this template to AWS, it will return you with 1 to n number of VM’s you’ve requested.
- AWS CDK - Cloud Development Kit
- There’s a great competitor available in market to automate your resource creation - Terraform.
1 terraform - multiple cloud providers
7
Q
How do you create 10 vm’s at once/ what is the automation that you’re using in your organization for infrastructure creation?
A
Terraform or AWS. Look at your organization and determine what you use. If your org is completely focused on AWS, the you don’t have to use terraform. You can either use AWS CLI, API, CFT or CDK
8
Q
When is Terraform used?
A
Orgs these days are using hybrid cloud pattern. So they have their VM’s in one cloud platform, other resource infrastructure in other cloud platform. In this model, terraform is best for you, coz you have to automate the infra across different cloud platforms
9
Q
What is Kernel?
A
Kernel is heart of the OS. Its responsibility is to establish a communication between your hardware and software. It has 4 responsibilities.
1. Device Management
2. Memory Management
3. Process management
4. Handling system calls
It cascades the requests from s/w to h/w and back to h/w.
10
Q
Components in Linux
A
Down to Top
OS - Kernel - System Libraries - compilers| User processes | system software
11
Q
Why Linux is preferred in Prod?
A
Lightweight, so Fast
Free (OSS)
Secure
12
Q
Shell commands
A
! - this is called shebang. This is the first thing you write in a shell script.
Rename or move - mv file file
free -g : see memory of your server
nproc : count of CPU’s
df/df -h : disk size
Top : see all the above info with one command
man ls/ man touch : open manual for any command you want to reference .
chmod- grants permissions to a file
chmod. What are the permissions for admin user, What are the permissions for group, What are the permissions for all users?
chmod 777 - access for everybody
Find command - important command used in devops. It searches entire system.
sudo find / -name pam
Kill Java process - kill -9 processId
Linux uses 421 - 4 for read, 2 for write, 1 for execute.
What’s the purpose of writing a shebang? - to tell the kernel which interpreter should be used to run the commands present in the file.
13
Q
What is shell scripting?
A
A process for automating your daily activities on your Linux computer
14
Q
Command to execute a shell script
A
./file.sh
Or
sh file.sh
15
Q
Where is shell scripting used?
A
As a devops engineer, you maintain all the Infrastructure, code repositories and do a lot of configuration management. For all these activities, on a day to day basis, you use shell scripting.
On a single automation, what a user expects is, you have to login to a specific machine where ansible automation is present and you have to execute the ansible automation.
16
Q
Why are you using shell scripting?
A
To automate node health of my VM’s. We have close to 1000 VM’s and every time it is difficult to monitor the health or status of these VM’s, so I write shell scripts for that.
17
Q
There are many automated tools, so why do you want to write shell scripting?
A
You can say ‘In our org, we’re not using any such tools
Or
These tools are restricted have restricted number of parameters, but the scripts can fetch more parameters that are not provided by these tools.
18
Q
Other Shell scripting use cases
A
Infrastructure automation
Configuration management
Amazon example
Day to day activities monitoring - you want to monitor specific tools and send email notifications - we can use shell scripting
19
Q
I/Q: How do you write script to monitor node health? / How do you monitor node health?
A
I can use the ‘top’ command or I can write custom shell scripts that monitor CPU and Ram usage.
20
Q
Good practices and how to write shell script in a scructured manner.
A
- Start with shebang followed by the executable that we want to use- #!/bin/bash
- Metadata of the file like author name, date etc.
- Set -x #debug mode
- Set -e #exit script when there is error (set -e has a drawback. It doesn’t error out when there’s a pipe. So we have to use the below command when using pipe.
- Set -o #pipefail when there is a pipe
21
Q
Shell command to find processes and process id’s?
A
ps
ps -ef
To filter for Amazon processes only - use grep
ps -ef | grep “Amazon”
22
Q
What does the pipe parameter do?
A
To use it with grep command -
ps -ef | grep “Amazon”
./test.sh | grep 1
Pipe parameter sends the output of first command to second command.
23
Q
I/Q: What will be the output of
date | echo “this” ?
A
It will print “this”, coz date is a system default command. It sends the output to stdin. And pipe won’t receive output from stdin. Pipe can only receive information if the command is not sending information to stdin and if the command is ready to pass the information to the next command .
24
Q
Awk command
A
If you want to retrieve only a specific column from the result, then use awk.
It can filter out information from the output.
cat test.sh | grep Sandeep | awk -F” “ ‘{print $2}’
Difference b/w grep and awk:
Grep command gives you the entire sentence with all columns as a result.
Awk only give you a particular column from the output
25
When an application is failing, the first step a devops engineer does in any company is check the log files.
But when the log files are stored in Aws s3 or Azure blog storage or whatever storage outside your VM, how can you retrieve this information from the Linux terminal?
Using curl command, just provide the url of the log file location.
Curl command retrieves information from internet.
curl url | grep Error
To get response from a website -
curl -X GET website.com
26
I/Q: What’s the diff b/w wget and curl command?
wget will download the file, the we can perform the grep command on the downloaded file. So it’s two command.
curl command does not download. 1 command.
27
Command to login as root user
sudo su -
Meaning substitute user do switch user.
If you want to login to other person’s account - use su command meaning - switch user.
su sandeep
su rasajna
28
I/Q: What is trap command?
Used for trapping any signals that are available on your Linux machine.
When we press Ctrl C, it stops execution. When we trap a signal, even when you do Ctrl C, understand that I have a trap mechanism set on my machine, so don’t do anything. Or if they do Ctrl C, send me an email notification using SMTP server, or print an echo statement saying that you cannot do this operation coz the owner of this Linux machine said he doesn’t want to execute Ctrl C.
trap echo “don’t use Ctrl C” SIGINT
What are signals and what are the different signals available in Linux?
There are a lot of signals in Linux like SIGQUIT, SIGINT. When we use Kill command to kill a process, Linux receives a signal called SIGKILL.
29
Use case of trap command
You’re inserting data into database using a script. Someone presses Ctrl C. Use a trap signal echo statement saying it’s not allowed. Or you can delete the entire data that was inserted until that point using
trap “rm -rf *” SIGINT
30
What is set -u command
The set -u command in Linux is used to set the shell option that causes the shell to exit if a variable is accessed before it is set. This can help catch potential errors in shell scripts where an unset variable is being used.
After running this command, if a script tries to use an undefined variable, the script will exit and an error message will be displayed. This can be helpful for ensuring that all variables are properly initialized in a script, thereby reducing the risk of unexpected behavior due to unset variables.
31
Disadvantages of Shell scripting
1. Slow execution speed, especially for heavy math operations.
2. Not suited for large and complex tasks due to minimal data structure support.
3. Prone to costly errors. A single mistake in a shell script can alter the command.
4. If u dont use set -u to handle undeclared variables, the compiler will not complain.
32
Is bash dynamically typed or statically typed?
Modern day programming languages are statically typed, meaning the type checking is done at compile time - Java, Golang etc.
Shell, Python are dynamically typed, which means that the type checking of variables is done at runtime.
33
Network troubleshooting tool
traceroute.
traceroute Google.com - this can show you the many hops in between your traceroute and Google.com
You can also use tracepath.
tracepath google.com
34
How will you sort list of names in file?
Using sort command.
sort names.txt
sort -r names.txt #in reverse order
sort names.txt > sorted.txt #save sorted output to a new file
35
How will you manage logs of a system that generates huge log files everyday?
There is an efficient way to do this - logrotate
You can say logrotate and define how many days you want to rotate this logs.
Or
You can say, for every 24hrs, just zip this log. You can also define the format - you can say logrotate (zip, gzip), create a compressed version of this zip file and you can say after 30 days, delete this log file
36
Why would somebody move to cloud infrastructure?
1. To reduce maintenance overhead
2. To be Cost effective
37
Suppose we need only instance ID from the AWS resource output of the shell script. What command will you use to parse the json and retrieve only the
jq- json parser
yq- yaml parser
aws ec2 describe-instances | jq ‘.Reservations[].Instances[].InstanceId’
If you want to install it- sudo apt install jq -y
38
I’m maintaining a GitHub for my org. An employee is leaving. His access needs to be revoked. How do I write a given shell script and automate this process?
We will clone the shell script onto an EC2 instance, we will execute this against this particular repository and see if it is working fine or not.
39
Primary concept of version control system - it addresses two problems. What?
1. sharing
2. Versioning
40
I/Q: What is the difference bw Centralized and Distributed version control system? And How is git solving the problem?
Or Diff bw SVN and Git?
You have to push/pull the code to and from the central code repository in Centralized. In Distributed, you create multiple copies of the distributed system code base(fork) and send the changes to that copy. And ask someone else to take the changes from this copy.
If there’s a single point of failure that is offline, when their applications and servers go down, there was no way Developer A and Developer B can communicate - coz CVS or SVN was down. This was the problem with centralized version control systems.
Fork - creating an entire copy of your code
41
I/Q: Difference bw Git and GitHub?
Git is a distributed Version Control system. Open source.
Any org can download git and implement it in their org. Git also comes with a UI.
How do you do it?
Create an EC2 instance, install git in the instance and say every developer should commit their changes to this server.
GitHub is a platform for developers to store and manage their code. It uses git dvcs, can raise issues, bug tracking, code reviewing, commenting, talking to peers, and now GitHub also supports project management.
42
I/Q: What is the git workflow that you use in your org?
git add
Git commit -m “commit message”
Git push
43
I/Q: Difference bw git merge and rebase
If you want to track the commits- which commit came after which commit; and want your commit history will be in a linear way. In such cases always use rebase. whereas if you’re not bothered about it, you can use merge.
git merge is a way of combining changes from one branch (source branch) into another branch (target branch) where as git rebase is a way of moving the changes from one branch onto another branch
Git Rebase: On the other hand, git rebase offers a different approach. It rewrites the commit history by moving your changes on top of the updated remote branch. This creates a linear, cleaner history, which can be super helpful when working on feature branches. It's like repackaging your changes as if you just started with the latest updates.
Rebase is one of two Git utilities that specializes in integrating changes from one branch onto another. The other change integration utility is git merge . Merge is always a forward moving change record. Alternatively, rebase has powerful history rewriting features.
44
When to use Git Rebase or Git Merge
Choose Merge
whenever we want to add changes of a feature branch back into the base branch.
if you want to keep the same history rather than rewrite it.
if you want to revert the changes quickly
Choose Rebase
whenever we want to add changes of a base branch back to a feature branch.
squash multiple commits
reiterate each commit and update the changes
reverting rebase would be very difficult
45
Difference bw git fetch and git pull
Git fetch only downloads the new data from a repository, but it doesn’t integrate the changes into your working files.
But Git pull is a combination of git fetch and git merge. It downloads the new code and also integrates it into our working directory.
In short, git fetch is great for reviewing changes before integrating them, while, git pull can fetch and integrate changes.
46
What is pre commit and post commit hooks?
Hook: If you want to perform an action before or after something, then it’s called a hook.
Pre commit hooks are actions that are taken before you do git commit.
Post commit hooks are actions that are taken after you do git commit.
You have the password files, secret information files, public or private keys etc that you don’t want to accidentally push to git. For that you can configure them in your pre commit hook, and tell git that before every commit, just execute this script, and what git does is that, if you’re accidentally even committing any such files, it executes this pre commit hook that you’ve given to git and says - I can’t commit this coz your pre commit hook is preventing it.
Exactly opposite, if you want to execute an action after a commit, it’s called post commit hook
47
Webhooks (w)
It’s a part of your GitHub or bit bucket or any of your git implementations.
If you want to trigger a pipeline or execute a python script after your git commit is done, or you want GitHub to perform an action, you can simple configure a w - means in GitHub you can configure something called hook, and GitHub does the action that you tell it to do.
If you go to GitHub web hooks, there are a lot of action you can do like perform w after issues, perform w after pull requests… everything. For every action in GitHub, you can configure a w.
48
CLI Command to add remote repository
git remote add “link”
49
While cloning a url, if you use ssh, you authenticate using what?
Public key
If you use Https, you authenticate using your password
50
On the Linux box, most of the times, you have a public key. If you don’t have, you can use a command called what?
ssh-keygen -t rsa
After this, you’ll have a .ssh in your home directory (users/sandeep)
51
Git command to create a branch/feature
git branch -b newFeatureName
Command To check all branches created and to see which branch I am on - git branch
To checkout to new branch - git checkout newBranchName
To checkout master branch - git checkout masterbranchName
52
If you want to get changes from one branch to other branch, how?
Cherry pick
git cherry pick commitId
Cherry pick is done when there are one or two commits. But if Amazon has to implement a new product, it can have 20000 to 50k commits. We can’t cherry pick every commit, so then we merge
53
Command To see all commits in one line format
git log —oneline
54
What is gitops? Difference bw devops and gitops?
Gitops is an operational framework. The concept of Gitops is to continuously monitor the git repository, and whenever there’s a new change, it will pull the application from git repository and deploy it to k8s cluster.
(This is the standard way. But we can modify it accordingly. Let’s say you’re not comfortable with Kubernetes clusters, then remove the k8s part and just say you’re deploying the apps onto ec2 instances as our org has not yet migrated to Kubernetes.
Gitops can be used to manage deployments for any infrastructure. Gitops makes it easy to implement continuous deployment for cloud native apps.
55
What are the 4 principles of gitops?
Declarative Infrastructure,
Git based continuous delivery,
Observability and compliance,
Infrastructure as code
In gitops, git is the only source of truth.
56
What is Configuration Management? (CM). What does CM do?
CM is a way how devops engineers manage the configuration of the infrastructure.
CM will aim to solve the problem of managing configuration of multiple servers.
Most used tool for CM is Ansible.
57
Why Ansible and not Puppet
Puppet is pull mechanism model.
Ansible is push mechanism model.
If there’s any change that needs to happen in the 10ec2 instances, he makes use of the Ansible, write his Ansible playbook on his laptop and push the config to the 10 ec2 instances. All he has to do is execute the Ansible playbook and the configuration is updated.
Puppet uses Master/slave architecture. Creates a master server and this server creates 10 ec2 instances as slaves.
Ansible uses Agentless model. It puts the names of these 10 servers called inventory files. You can just put the IP address or DNS of this server. If you just have the IP address, you can put the IP address and just have the passwordless authentication enabled.
What is Passwordless authentication?
The laptop in which you’re running your Ansible playbook should be able to connect to these ec2 instances without any password.
**If you want to scale up or scale down your servers. Because you’re using agentless approach, even if you’re scaling up your servers 10 or 100 times, all you need to do is get your public ip address and add the public ip address to this inventory file. That’s it. And if you have the passwordless authentication enabled, you are good to go. This is the basic of Ansible.
*Ansible also has Dynamic Inventory. With DI, Ansible will autodetect. Now you don’t even have to go and update your inventory file. Previously you have to go and update your inventory file. But with DI, if you’re creating 100 new servers in your Aws,using the INF file, making some changes, you can ask ansible- whenever an EC2 instance is created on my AWS account in a specific region or az, directly consider it as a server that has to be managed by you. So Ansible will understand… ok xyz person has created a new server in this region and az, so the new server is something that I have to manage along with the existing servers. So the world of configuration management has become extremely simple using Ansible because of this agantless architecture of Ansible.
Ansible is part of red hat and Redhat continuously updates Ansible.
Ansible is very very easy with both windows and Linux. The support is very good compared to puppet and chef.
With puppet, you have to learn puppet language. You need to learn it.
With Ansible it is simple yaml manifest. So Ansible is using a global language that is used by devops engineers already.
Another advantage: You can write your own ansible modules. Ansible is written in Python.
Let’s say your org owns some xyz load balancer. So what you can do is, as part of your organization, you can write some ansible modules let’s say installing the load balancer or deleting the load balancer or removing some configurations from the load balancer, you. An write your own ansible modules, and any other devops engineer in some abc company wants to use these modules, you can share them using Ansible galaxy and these people in ABC can use the modules that you wrote like any other open source tool, and they can configure the load balancer. There’s a bridge in between and that bridge is called Ansible galaxy.
58
Disadvantages of Ansible.
1.CM with Windows is not similar to the CM as Linux.
2. Debugging - Needs improvement in looking at the playbook logs to identify the problem. They’re not that easy for devops engineers to understand the problem. It needs to improve.
3. Performance - Can be improved. Ansible can manage 1000’s of servers, but when it’s managing some 10000’s of servers, we might run into some performance issues. Ansible is trying to address these problems.
59
Does Ansible support every cloud provider?
For Ansible, the cloud does not matter. It will run on any cloud provider. Ansible only needs the public IP address, do you have SSH enabled to this machine, and is the SSH for this machine allowed access from your Ansible machine. Only then you can automate.
It doesn’t matter where the instance is. It only sees if there’s a public ip address, if the ssh is allowed from ansible this machine
60
All that Ansible requires is passwordless authentication. So if ansible can authenticate without any password, it can configure anything on that server. How do we setup passwordless authentication and talk to the target server from the ansible server in a passwordless way?
Click on the aws instance. You will see private Ip address. Always talk to the private IP, coz both of them are in the same VPC and both instances are in AWS. Now SSH into the private IP address, it will ask for the password, coz we haven’t configured any passwordless yet.
1. on the Ubuntu user: ssh-keygen- this command will generate a key pair for you. Cd into that file directory. You will see id_rsa. This is your private key that you use to login to this machine and which you shouldn’t share with anyone. So if someone says let’s communicate with each other, always use the public key.
2. Cat path/public key -> copy this public key
3. Now login to the target server and ssh-keygen here as well.
4. Now open the authorized_keys file that was generated. vim ~/.ssh/authorized_keys
5. Paste the copied public key of the Ansible server into this file and save
6. Now in the Ansible server do ssh ipAddressOfTargetSetver. That’s it. You were able to authenticate without any password. This is the prerequisite for your ansible.
So you copied the public ip address of the ansible server to the authorized keys of the target server. Then ansible host will be able to communicate with the target server in a passwordless authentication way.
61
Use ansible and create some files on a target server
This is a simple task so you don’t have to write an ansible playbook. You can use ansible commands called ansible adhoc commands.
You do this by command: ansible inventoryFileLocation
Inventory file is a file which stores the IP addresses of the target servers.
ansible -i inventory ipaddress
Or
ansible -i inventory all -m “shell” -a “touch devopsclass”
If the output is a yellow line means there’s no errors
How do we know what is this module called “shell” and how do we know what arguments to pass? - There are thousands of modules in Ansible. We can google for Ansible modules. Then look at all modules. Everyday there is so much contribution happening to Ansible since redhat Ansible.
62
shell command to find size of each file
ls -ltr & du -sh
63
Difference bw Ansible adhoc commands and Ansible playbook?
adhoc is for 1 or 2 commands.
Playbook is for multiple commands
64
Run certain number of playbooks only on dbservers and run certain number of playbooks only on webservers. How?
In the inventory file, do grouping of servers.
[dbservers]
172.38.58.00
[webservers]
373.95.27.99
The brackets are very important.
65
Use Ansible playbook to configure your Kubernetes cluster
vi kubernetes.yaml
#Create 3 ec2 instances on aws. (Terraform)
Configure 1 of those ec2 instances as master.(Ansible)
Configure other 2 as worker(Ansible)
66
Ansible roles
Efficient way of writing Ansible playbooks that will improve your efficiency to write complex playbooks.
Example: Let’s say I want to configure kubernetes using Ansible. It will have close to 50-60 tasks and you have lot of variables, lot of parameters, certificates, secrets that I have to configure while creating this Kubernetes cluster.
If we do it with roles, we can segregate everything and we can properly structure your Ansible playbooks. That’s why the concept of roles is introduced.
If we want to start playing with roles, you can use the command called - ansible-galaxy role init kubernetes (this says ‘I want to create a role for Kubernetes’)
But before that - create a folder for k8s and cd into it. Then give the above command. You will create a Kubernetes folder with multiple folders and files. Using them, you can structure your playbooks. This is the concept of roles.
In summary, whenever you want to write some complicated playbooks, you will start using the Ansible galaxy command to create rules.
What happens when you create roles? - you can write structured and efficient Ansible playbooks.
67
ToDo in Ansible - Learn Dynamic Inventory. Ansible dynamic inventory aws demo in YouTube
68
What is terraform?
Terraform solves the problem of multi cloud infrastructure automation. You can write automation for any infrastructure. This concept is called API as code - using which you can automate any provider using their api’s.
So internally terraform talks to the API of Aws, azure, gcp etc.
69
Advantages of using terraform
1. Manage any infrastructure - Any cloud provider, single tool
2. Track your infrastructure - You don’t have to login to your cloud provider to see what infrastructure you’ve created. You can simply login to your terraform machine and look into your state file. If your state file is stored in S3 buckets or any other places, known as Remote Backends, you can go to the s3 bucket and see your entire Aws organization.
3. Automate changes with collaboration - you can put your terraform files *except your State file* in a git repository and collaborate with your peers, ask them to review it.
4. Standardize configurations - with one cloud provider you do things in one way, and another a different way.
70
Terraform lifecycle
Terraform init, terraform plan, terraform apply, terraform destroy
Terraform will take the config files, create an api for it, then submit the api to the target cloud provider.
1. Write tf config files - Go to hashicorp terraform documentation
2. Use terraform dry run - What if you write something wrong? Terraform supports something called as dry run. It’s a way to see what is going to happen when you execute
3. Apply- This step creates the resources on the cloud.
Some also consider terraform destroy as part of the lifecycle which is like deleting the resources that you created with terraform.
71
CI/CD
CI: Process of integrating set of tools and processes before delivering your application to your customer.
CD: process where you deliver your application on a specific platform to your customer.
There are multiple steps that an organization follows to get the application from developers to customer - like unit test your application, functionally test, perform regression testing, even single code commit made by developer would take months. But today we talk about delivering applications in weeks of time or in days of time. To achieve this we automate this entire process.
We use Jenkins for this. Jenkins is called orchestrator coz it integrates these tools by using Jenkins pipeline.
Every org more or less follows these checks before deploying their application:
Unit testing - Maven, junit
Static code analysis
Code quality/ Vulnerability - Sonar
Automation testing(end to end testing)
Generating reports - ALM
Deployment - k8’s/ docker/ec2
With this process we can deliver the application within days or even hours.
We install jenkins on one host, probably on your laptop or ec2 instance, and then you keep adding machines to it. Coz there are 100’s of developers in your company, and 1 jenkins node cannot take all the load. So what you usually do is deploy Jenkins Master on one ec2 instance and then you keep connecting multiple ec2 instances to it. you can set it up saying, for team 1, run it on node 1, team 2 node 2 and so on.
The problem here is that -
So if your team has 10 teams, for each team you might have to setup some additional VM’s for running pipelines on specific nodes. Every time you scale up this Jenkins instances, you know you have to create a lot of compute (Ram, cpu, hardware). When you keep adding compute to your jenkins instances, the setup not only becomes very costly, but the maintenance also becomes very huge.
Instead, we need to look for some setups which can scale up and scale down automatically. If I want to have zero servers when I’m not making any code changes, then jenkins is not the tool to use in this case.
If we see how Kubernetes is maintaining its code with thousands of developers across the world, it’s using GitHub actions for CI/CD. Whenever a code change is made, it will spin up a k8’s pod or docker container, and everything gets executed on the container. If you’re not using it, the server or the worker node which is used to run these docker containers will be used by some other projects - you do a shared resources.
In the concept of shared resources, the advantage is that, if you’re using runner from GitHub actions ann using runner from GitHub or using runner from Microsoft runners (runners are nothing but worker nodes from Microsoft), what Microsoft will do is create containers or Kubernetes pods for you on the Microsoft service itself or Azure service itself. So you don’t even know where the servers are getting created. And if it’s a public project or open source project, you’ll get it for free.
So let’s say you don’t want to use Microsoft, and your code is secure, what you can do is that for the 77 k8’s repositories, create one common server. You can host this server on Azure, you can host this server on AWS , or it can be k8 cluster anywhere in the world, and you can say - whenever any developer makes a commit on any of the 77 repositories, you can create a k8 pod on this on this k8 cluster, and once the execution is done, all the steps are done, just delete the pod so that I’ll be free one more time, and any other project can use that k8 resource. So instead of wasting resource for every project, instead of creating jenkins instances for each and every project, you can create one common GitHub actions which can be used across multiple people/ multiple projects in your org and save your compute/resources. And if no one is using it, then there is no wastage. Coz the server size is shared across everyone, so literally you are using zero compute instances when there is no code changes. This is the modern day CI/CD setup. One more advantage is that it’s extremely scalable.
GitHub actions is event driven CI and CD solution. But jenkins if there is no webhook configuration , then by default it is not event driven.
GitHub actions, you can integrate your pipelines from one project to other projects too.
Gitlab and GitHub actions are very similar. The only difference is syntactical differences and semantic differences.
Jenkins is a Java based application so installing Java is a prerequisite for jenkins
72
Git -> Jenkins <- Argo CD -> K8’s
Take a sample application and implement this architecture. How do I do this? I want to deploy my artifacts onto Kubernetes directly. I want to create a jenkins pipeline from end to end- from checking out the code to deploying the artifacts on Kubernetes.
I saw many people use Ansible instead of Argo CD here. Can this be done?
Ansible is a config management tool. It by luck can deploy your code to a k8 cluster.
But Argo CD is a declarative C delivery application. It can not only deploy but can monitor the state inside your k8 cluster.
Let’s say in your app, you have v3 of your app. So Ansible deploys whatever you’re storing in your helm charts or whatever you’re storing in deployment.yaml files it will just pickup and deploy into your Kubernetes cluster. BUT if you’re using Argo CD or flux or spinnaker - any of the declarative CD tools (or) GitOps tools, they will ensure that the state is always the same. Coz they are k8 controllers. They are actually deployed in your Kubernetes cluster itself.
Create a jenkins pipeline that will watch the GitHub repository. When a commit is made, build the application. Put the Python application artifact I.e, the docker container. I’ll push it to GitHub and also parallel my update the deployment.yaml file inside the GitHub.
If you observe, Argo CD arrow is pointing towards Jenkins. Meaning Argo CD will watch for any change in GitHub. It understands that there’s a new version of todo application, and it will automatically pickup and deploy the application onto Kubernetes.
If you have multiple Kubernetes clusters, you can configure Argo cd to deploy to multiple cluster as well.
73
I/Q: Explain the CI CD process in your org
In our org, to achieve the CI CD process, we’re using Jenkins as orchestrator.
We use bitbucket as repository. When a developer commits the code to the repository, a Jenkins pipeline is automatically triggered. In this step, the pipeline pulls the code from the repository, then builds the code using maven. Then we verify the code quality, code vulnerabilities, static code analysis. Then we use an app called AppScan for security testing. Then we promote this application onto dev environment using ArgoDC and k8s. K8s is our dev environment. we use ArgoCD to continuously monitor the code repository.
ArgoCD would look for k8s manifests in the repository and whenever a new image is updated, ArgoCD will look for this new tag of the image, and using helm chart, it would deploy the new version of the application onto the target Kubernetes clusters.
if you say Argo CD, they’ll probably think you’re using advanced tooling.
74
How do you handle issue in your worker nodes? You have jenkins jobs running on your worker node. How do you handle these things?
Like your worker node is going down, your worker node is not responsive.
I’ll login to this worker node, try to understand the problem, look into the worker node health. I wrote a simple Python application where it will monitor the worker node’s health, and in case the CPU or ram is full, it will send me alerts when the ram reaches 80%. Or you can say we implemented auto scaling on the ec2 instances.
But the best way to say this is what we did in the tutorial.
‘We are using docker agents, and we only spin up containers when there’s a Jenkins job running. So in this way we’re ensuring that our agents or the jenkins is never down.
75
How do you install Jenkins? How do you expose it to outside world?
Install Java. Then install jenkins. Then go to aws ec2 instance and set an inbound rule to expose it to a specific port.
76
What is Incident Management
When there’s an issue in the application or some error is being thrown in the application, the monitoring system shows us the alerts. Now who is receiving these alerts? Or who will act on these alerts?
This entire process is tracked as part of the incident management system.
This system has to create an incident saying that an alert is being thrown by the monitoring system and the specific support guys will look into the issue. They have to fix it right away on the production system. The support guy will update the status of the ticket on the platform until it is resolved.
Service now is a platform where you can create such incidents, or your monitoring system can be directly integrated with service now API’s where your monitoring system directly talks to the service now apis and it creates the incident on its own.
This is how you integrate your monitoring system with service now to automate the incident management process.
Change management starts after the person assigned this incident starts working on this ticket.
77
What is buildah
Buildah is a CLI tool for creating OCI images. You can build containers fast and Buildah’s commands replicate all the commands that are found in a dockerfile.
Redhat created this tool that could be used with existing container host tools to build OCI and Docker compatible container images. These images could then be stored in container registries and used in multiple runtime envs, with or without the use of a container daemon like docker daemon(which requires root access privileges, and the resulting containers of which are then dependent on the functionality of the daemon itself).
1. This allows buildah to build images with or without Dockerfiles while not requiring any root privileges. If you build container images using Buildah, you don’t need a running daemon for the utility to function.
2. Creates container images from scratch or from an existing container image starting point.
3. Doesn’t include build tools within the image itself, reducing the size of the built image, increasing security.
4. Is compatible with Dockerfiles, allowing for easy transition from Docker.
5. Creates user specific images so that images can be sorted by the user who created them.
OCI compatible images - meaning an image that adheres to the OCI Image Format Specification so that an image is compatible across various container tools and runtimes. OCI images are designed to be platform independent and support running containers on different architectures. So the images are compatible with various container runtimes incl. docker and kubernetes.
Buildah has very good integration with tools like skopeo and podman.
Why Buildah instead of Podman?
Buildah provides a wider range of customization options than the generic podman build command. Buildah commands are more detailed than podman’s allowing for finer grained control over image and allowing for the creation of finer image layers. Podman’s ’build’ command uses a subset of Buildah’s functionality.
Buildah is more focused on building temporary containers for image creation. Podman is geared towards managing long lived containers intended for extended use.
78
Several misunderstandings regarding the Linux container architecture persist.
Do containers run on Docker?
Containers do not run on docker. Containers are processes that run on the Linux kernel. Therefore, containers are Linux.
Moreover, docker daemon is only one of the many tools and libraries that communicate with the Linux kernel in order to create containers.
Buildah is an excellent example for this. Buildah can create containers without using docker, meaning users can implement docker and OCI compliant images with Buildah without the need for executing a container runtime daemon.
Buildah provides a simple CLI for creating OCI compliant images. Buildah greatly streamlines the container build pipeline.
Buildah can create a container image using a container image and a docker file or it can be started with an empty image.
Using Buildah to build docker image with the docker command results in
1. The size of the created image is small.
2. Improved security of the image because the software used to create the container is not contained in the image.
79
Difference bw Prometheus & Grafana
Prometheus collects the data. Grafana visualizes it. (Data is visualized and analyzed in Grafana dashboards)
Analogy: Think of Prometheus as a data warehouse, gathering and storing information.
Grafana acts like a BI tool transforming the raw data into insightful visualization that helps users understand what’s happening.
Prometheus - Data collection and storage
Focuses on: scraping metrics(data points) from various sources applications, servers, infra.
Features: Uses query language PromQL. Long term storage of historical data for trend analysis.
Grafana: Data viz and analysis.
Features: Viz like graphs, heatmaps, pie charts and dashboards.
Sharing and collab features for teams to monitor performance together.
80
How to secure cloud infra? Can you speak on cloud security?
1. When we deal with infra, always go with IaC. Do not manually change anything on the cloud. Work on the drift detection strategy.
What is your drift detection strategy?
If you’re doing terraform, tf has some issue with drift detection. If someone goes to cloud UI and changes things, it’s hard for terraform to identify that. You have to add that logic.
If you’re using something like cross plane, cross plane comes with that drift detection and auto healing capability.
If you don’t have a proper drift detection strategy, you have to go with Zero privilege model - you initially give access to people, then you slowly reduce the access until it comes down to zero privilege, so that most of the employees cant modify any infra on the UI, or thru their IAM roles, they should not have access. If you want to perform task, then it should be thru your IaC platform, thru your platform engineering platform that you’re building, or thru your CICD pipelines.
You cannot start with zero privilege model. You have to start with the least privilege model. From there, you can slim down it further until it becomes zero. That means initially you might give IAM access to people. The access need not be strict, then you slim down.
2. Other cloud security is wrt applications - build proper firewalls, have the api gateways, have the web application firewalls, make sure that you don’t have your applications in the private subnet talking to the internet directly, you have a Nat gateway. Only thru the nat gateway, the apps are talking to the internet.
Similarly, any resource you pick up, make sure you don’t Have public access enabled. Then further see if CORS is enabled.
In each resource, you have to think about security, that way you can master cloud security.
81
How/why do you pick a particular tool?
A qualitative and quantitative analysis is done on all the tools like Load test, memory footprint test, CPU Utilization test, latency test…
After that it goes to staging. Then it goes to review committee which assesses our findings. Then that product goes to the Senior Staff or CTO’s. Then after their approval, they choose it. You can’t just choose a tool because it’s the most popular or trending one.
82
What is openstack used for?
Openstack is used to create your own cloud.
You install openstack for on premise infrastructure.
83
Difference bw Jenkins and GitHub actions?
Actions is event driven CICD by default. Even Jenkins is event driven, but you can only configure pull request action using whatever web hooks GitHub is supporting. But if there is no web hook configuration for Jenkins, by default it is not event driven.
Actions - you can integrate your pipelines from one project to the other projects as well
84
Do you prefer GitHub actions or Jenkins?
GitHub actions is platform specific. If your project is public/OSS then GitHub actions is free to use. Or if your project is GitHub and will be on GitHub permanently, then you can use GitHub actions. Otherwise use Jenkins.
If it’s a public project, GA is the solo winner. If you’re a private project, prefer Jenkins
85
How do you secure sensitive information on GitHub?
GitHub settings -> Sectets and variables
86
How do you write a GitHub CI pipeline?
In the git repository, create a .github/workflows folder. In this folder, write the yaml file
Use the ‘on’ keyword in the script followed by ‘push’ or ‘pull’ etc
87
What are the different ways to trigger Jenkins pipeline?
Poll SCM
Build triggers
Web hooks- The efficient way to do this is by using webhooks.
A web hook can be created to notify Jenkins when changes are pushed. Jenkins then automatically builds the updated code.
Whenever developer makes a code change in GitHub, it will send a payload to Jenkins in json format and the pipeline gets started.
88
How to backup Jenkins?
We can take a backup of the .jenkins folder using rsync shell command. If we setup rsync, It will recursively sync the .jenkins folder to any backup system that we configure like EBS volume as a volume or snapshot.
89
How do you handle secrets in Jenkins?
Initially we used Credentials Plugin provided by Jenkins.
Later we moved to Hashicorp vault. We integrated Jenkins with Hashicorp vault.
So whenever there’s a sensitive information we want to use in the pipeline, we invoke the secret from that Hashicorp vault.
90
What is shared modules in Jenkins?
It’s a concept that is similar in different tools and technologies.
It’s nothing but- As a devops engineer I work in dev team x. I wrote pipelines for them. I wrote a pipeline that picks up code from a repository and deploys onto an ec2 instance. There’s Y and Z dev teams who also need the same approach. So instead of they writing the entire pipeline again, they ask me to make this Jenkins pipeline as part of a module and we can share across multiple teams. That’s why this process is called shared modules. So shared modules shared libraries all fall into the same category - where one person writes the code and it can be reused across multiple teams.
91
How to setup auto scaling groups in Jenkins?
Some companies want to have Jenkins with multiple worker nodes. The applications are very huge and they cannot use docker agents so what they decided is they will use a ec2 instance which acts as a Jenkins master and there are 20 Jenkins worker nodes - each worker node for one team. But during certain public holidays they might need extra load, so they require extra worker nodes. But if you create these worker nodes and if you don’t use them again, to solve this problem, what they do is they configure Jenkins with auto scaling groups in ec2 instances or in AWS, auto scaling groups takes care of automatically scaling your ec2 instances using predictive scaling or whatever scaling that you configure.
***See Abhishek day 21 video to see the steps how to setup an auto scaling group for Jenkins in AWS***
92
How to add a new worker node in Jenkins?
Login to Jenkins master, navigate to Manage Jenkins- Manage Nodes - New Node. Enter a name for the new node and select Permanent Agent. Configure ssh and click on Launch.
93
How to add a new plugin in Jenkins?
Using the CLI, we have a command to do it - Java -jar jenkins-cli.jar install-plugin
Using the UI, in Manage Jenkins- Plugins.
94
What is JNLP and why is it used in Jenkins?
JNLP is a way to allow your agents (worker nodes) to talk to your Jenkins master - to be launched remotely by master. This allows Jenkins to distribute build tasks to multiple agents, providing scalability and improved performance. The results of the build are then sent back to the master and displayed in the Jenkins UI.
Install the jnlp jar in jenkins master UI and your Slave nodes will start working as agents.
95
What are some common plugins that you use in jenkins?
Git plugin, Docker plugin, Jira plugin, Maven plugin, Ec2 plugin, Junit plugin, Kubernetes plugin
96
Files and folders that containers use from host OS
The host file system
Networking stack
System calls
Namespaces: Docker containers use Linux namespaces to create isolated environments.
Control groups
97
What is docker?
Docker is a containerization platform that provides easy way to containerize applications, which means, you can build container images, run the images to create containers and also push these containers to container registeries such as docker hub.
Containerization is a concept and Docker implements containerization.
Docker is a centralized platform. It has only one docker daemon.
Are there any other platforms that can implement this containerization? - Podman is a very good competitor. It addresses few problems of docker.
Others are Buildah, Scopio
98
How will docker allow you to push the container docker images to a public or private registry?
It happens through your docker client.
Using the docker client(the CLI), you run CLI commands which is received by the docker daemon and it is executed. When they execute, they create Docker images, docker containers and finally using this docker daemon itself, you can push your docker images to your docker registry.
daemon is a process that we’re going to install on the machine. When I say you install docker, it means install the docker daemon. Docker daemon is the heart of docker. If docker daemon goes down, your docker will stop functioning and your containers will stop working coz these containers are running against this docker daemon and docker daemon is the one that is listening your containers.
99
Docker commands
Check if docker is running - sudo systemctl status docker
List put all networks on this host - docker network ls
Delete network - docker network rm test
Create custom bridge network - docker network create network-name.
Now to create a new secure container that is isolated from the other containers - docker run -d - -name finance - -network= network-name nginx:latest
Stop container - docker stop containerName
100
What is the difference bw CMD and ENTRYPOINT in Dockerfile?
Whenever we run ‘docker run’ command, both cmd and entrypoint can be used to execute as your start command. The only difference is Entrypoint cannot change. So as a user, whenever you’re running a container, let’s say you define python3 in entrypoint (ENTRYPOINT [“python3”]) in the docker file, they cannot overwrite this value in your docker image. Whereas CMD is configurable.
101
What production issues/ problems have you faced with docker or containers? How did you solve them?
Previously, even in the final stage, we were using Ubuntu base images, which were exposed to some kind of vulnerabilities. So we moved to distroless images. Our Python DI only had Python runtime because of which it dint have basic packages like ls, curl etc. So it was providing us highest level of security. After implementing DI, it’s safe to say that our apps are not exposed to any OS related vulnerabilities.
The golang DI have the highest security coz they don’t even need a runtime.
102
Distroless images
A Distroless image (DI) is a very minimalistic image (lightweight) that will only have runtime envs.
If you choose a Python DI, it will only have Python runtime.
Some DI don’t even have runtimes. Go lang apps are statically typed apps. They don’t even require runtimes for executing a go application. So your docker image can be a max of 10-15mb.
By using distroless images, you can improve efficiency or get maximum benefit of multi stage docker builds.
Biggest advantages from using DI
1. Reduce image size drastically (by hundreds of times).
2. Highest security
It is really important to understand the concept of multi stage builds and distroless images as they are the future of containers.
103
What is Nginx application?
Nginx monitors the user information - who’s the logged in user and from which IP address has this user logged into the application.
This information Nginx stores in a log file. This log file is very important- while doing security audit of your company or understanding the details of user information.
104
Docker Volumes: Why volumes?
Problem 1:
Let’s say we installed nginx application in our container. Nginx monitors user login information and stores it into a log file. If the container goes down, the log file is deleted, as containers are ephemeral (short lived). So you cannot perform security audits coz there’s no log file for previous day/week/month.
Problem 2:
We have a frontend and backend each running on a container. The backend writes to a file. The frontend uses this file to render information. If the backend container goes down, and the frontend wants to access the file information from 3 days back, it will not be possible as the data is lost. So now the frontend can serve only today’s records.
Problem 3:
There’s an application running on a container. It wants to read a file on the host system and display to the user. But there’s no way to do that.
To solve all these problems, docker came up with two solutions - bind mounts and volumes
105
Bind mounts and Volumes
It will bind a folder on your container with a folder on your host. This way the data on the container folder is always present on the host system. So if a container is down, we can create a new container and bind the history folder to the folder in the container and the data is not lost.
Volumes also the same solution but volumes offer you a better lifecycle.
Meaning, using the docker CLI and docker commands, you can create a volume, destroy volume, take this volume out of container C1 and later attach it to container C2, or at one single point you can attach the same volume to C1 and C2.
The main advantage of using volumes over Bind mounts is -
1.Using volumes, you don’t provide the directory details. You’re asking docker to create volume on the host. Now this volume is a logical partition on the host. This volume will be mounted to your container.
2. You’re managing the entire thing using docker CLI itself. And this volume has a lifecycle.
3. A bind mount can create a bind on the same host. You’re restricted to only this host. But using volumes, you can create a volume on the same host, on an external ec2 instance, s3… you’re not restricted with options.
4. You can take backups, coz you can also mount it using external storage devices.
5. These volumes are very easy to share from one container to other container.
6. Volumes can have high performance as well.
You can dedicate a specific volume to one container or multiple containers.
So always go for volumes instead of bind mounts.
Commands - docker volume create sandeep, docker volume inspect sandeep, docker volume ls, docker volume rm sandeep
docker run -d - run container in detach mode.
docker run -d - -mount
A volume is a logical partition that you’re creating on your host.
106
How to mount a docker volume?
Use either of the two commands. Both of them do the same thing.
docker -v
docker - -mount source destination permissions
Best practice is to use docker - -mount as its a verbose command if I want to use it in a shell script or share to other members in the team, it helps understand what we’re trying to do.
docker run -d - -mount source=abhishek, target=/app imageName:latest
107
How to delete a volume
We have to stop the container, then delete the container. Only then you’ll be able to delete the volume.
108
Why do you need networking in Docker?
1. Networking allows containers to communicate with each other and the host system.
2. A very good level of isolation between two containers .
Both of them are very very important.
109
Subnet
It’s your networking group.
110
How does docker networking work?
Every host and container has an eth:0 n/w that is available out of the box. If the subnet is different, and you ping the host ip address from the container ip, it gives a n/w error. To solve this problem, docker created a virtual eth - which is basically docker 0. Without this virtual eth, a container cannot talk to your host. So whenever we create a container, by default docker 0 gets created. This is called bridge networking. The default networking in docker is Bridge Networking.
It’s called Bridge Networking coz the container has a different subnet and host has a different subnet, but they can talk to each other using a bridge. If you try to delete or manipulate this network, a container can never talk to its host.
Docker allows us to create our own custom bridge networks to achieve networking isolation from one container to another container.
You create a custom bridge network using the ‘docker network’ command.
When I create a new n/w, for the docker run command, I can simply pass this network as an argument
- -network.
This will connect using the virtual bridge n/w that you created through the host i.e eth 0.
2. Host networking: Containers will directly use the network of your host. When you create a container, docker will directly bind your container with the eth:0 of your host so that both the ip’s are in the same subnet. So pinging from container to host is possible. But in this way, whoever has access to your host has access to your container as well. So this is a very problematic approach.
3. Overlay Networking:
Very popular when dealing with container orchestration engines(COE) because it has multiple hosts and you want to create clusters on multiple hosts. So overlay n/w will create a n/w that is common across this multiple hosts.
If you’re using docker and just want to run containers, then overlay networking is too much.
111
How to run Docker in Detach mode?
docker run -d - -name login nginx:latest
This will download nginx in detach mode.
Here ‘login’ is the name of the container. - -name is the tag under which ‘login’ will be assigned to.
112
How to login to container
docker exec -it login /bin/bash
Here login is the name of container
113
Can we use multiple CMD commands in a docker file?
No you cannot. A CMD command can only be used once.
114
What is Terragrunt?
An open source orchestration tool (for Terraform/OpenTofu) that allows IaC to scale. It’s a wrapper for Terraform to keep your Tf configurations DRY (Don’t Repeat Yourself)
1. DRY configuration- Gives you the ability to keep your code DRY by defining your configuration once and reusing it across multiple envs.
2. Better remote state management - Simplifies the overall management of your Tf state files. you can easily manage remote states and multiple environments. It also keeps your codebase clean and organized.
3. Dependency management- it can automatically manage dependencies between your tf modules, ensuring they are applied in the correct order.
4. Before and after hooks - this allows you to execute custom actions and scripts before the tf commands
115
VPC
Virtual private cloud.
It’s an isolated area of your AWS account that has its own virtual network and IP address space. Every AWS resource deploys into a VPC. If you don’t explicitly specify a VPC, the resource will be deployed into the Default VPC.
If you for some reason deleted the Default VPC in your account, use a different region (each region has its own Default VPC), or create a new Default VPC using the AWS Web Console.
Otherwise you’ll need to update almost every example to include a vpc_id or subnet_id parameter pointing to a custom VPC.
When you do tf apply, if your configuration fails to apply, you may have customized your region or removed your default VPC.
116
What is <<-EOF and EOF in tf?
What is user_data_replace_on_change = true ?
The <<-EOF and EOF are tf’s heredoc syntax. It allows you to write multiline strings without inserting \n characters all over the place.
The user_data_replace_on_change parameter is set to true, so that when you change the user_data parameter and run apply, tf will terminate the original instance and launch a totally new one.
Tf’s default behavior is to update the original instance in place, but since User Data runs on the very first boot and your original instance already went through that boot process, you need to force the creation of a new instance to ensure your new User Data script gets executed.
You need to do one more thing before this web server works. By default, aws does not allow any incoming or outgoing traffic from an EC2 instance. To allow the EC2 Instance to receive traffic on port 8080, you need to create a security group
resource “aws_security_group” and specify the port
117
Tf commands
Tf show - show state file
Tf fmt- format tf file
Tf validate - validate configuration
Tf apply - create configuration
118
What is implicit dependency in Terraform?
When you add a reference from one resource to another, you create an implicit dependency. Terraform parses these dependencies, builds a dependency graph from them, and uses that to automatically determine in which order it should create resources.
For eg. if we have to deploy a ec2 instance. Tf would know to create the security group first before creating the ec2 instance, coz the ec2 instance references the ID of the security group.
You can even get terraform to show you the dependency graph by running the ‘terraform graph’ command.
That’s the beauty of a declarative language: you just specify what you want, and terraform determines the most efficient way to make it happen.
Now if you run the ‘tf apply’ command, you’ll see that terraform wants to create a security group and replace the ec2 instance with a new one that has the new user data. This is an example of the immutable infrastructure paradigm (discussed in server templating tools).
While the web server is being replaced, any users of that web server would experience downtime (so we need to do a zero downtime deployment).
119
How do you write terraform module?
1. Follow a standard structure: Write a main.tf file with all core resources.
Have a readme file for the documentation.
Have an outputs.tf file for the outputs.
Group related resources in logical files(network.tf, instances.tf)
2. Organize modules in a dedicated directory: create a ‘modules’ directory in the project folder. Each module should have its own subdirectory.
3. Design for reusability: Make modules generic and configurable using variables. Avoid hardcoding values specific to one use case.
4. Use clear naming conventions: Give modules descriptive names based on their purpose. Use consistent naming across resources and variables.
5. Expose useful outputs : Provide meaningful descriptions for all outputs. Output values that parent modules need to reference.
6. Limit Complexity: Keep individual modules focused on a specific purpose. Use local values to simplify complex expressions.
7. Use for_each for creating multiple similar resources.
8. Enable deletion protection for stateful resources.
9. Version your modules: Use semantic versioning. Consider publishing reusable modules to a registry.
10. Test modules independently: Run separate pipelines per module. Unit test modules in isolation.
11. Use workspaces for managing different environments.
12. Organize environment-specific configurations: Consider separate directories for dev/staging/prod. Use a structure like
terraform/
modules/
live/
dev/
staging/
prod/
120
Network Security
