1 of 3: CI/CD the Easy Way Using AWS CodePipeline

This article is one of three (3) in a series:

1 of 3: CI/CD the Easy Way Using AWS CodePipeline
In part 1, you will use AWS CodePipeline to automate deployment of application changes.

2 of 3: CI/CD the Easy Way Using AWS CodePipeline
In part 2, you will use AWS CodePipeline to add a Testing stage to your pipeline.

3 of 3: CI/CD the Easy Way Using AWS CodePipeline
In part 3, you will use AWS CodePipeline to add Staging and Manual Approval stages to your pipeline. You will decommission the infrastructure once done, because this is only proof-of-concept.

For background on this series, go here:

CI/CD the Easy Way Using AWS CodePipeline | A Three-Part Series

Prequisite 1 of 2. A Kubernetes cluster named `humangov-cluster` with at least one HumanGov Application state deployed running on Amazon EKS.

If you need instructions for that, check the series on Kubernetes.

If you followed my example from the Kubernetes article, you probably have a few pieces to re-do. I'll list them here, so you have a check-list. Disclaimer: the information below is based on the context of having gone through the prior series. There are assumptions/pre-requisites for the information provided here, and if it doesn't work for you, refer to the prior series. For the sake of brevity, screenshots are not included here. Please see the prior series of articles if you want to see some images.


#1 of 15. eks-user Access keys
AWS Console -/- Identify and Access Management (IAM) -/- Access management -/- Users [eks-user]
[Security credentials]
[Create access key]
Access key best practices & alternatives -/- Other [Next]
Set description tag -optional [Create access key]
Retrieve access keys [Done]

#2 of 15. Disable managed credentials on Cloud9
Preferences -/- AWS Settings -/- Credentials -/- DISABLE 'AWS managed temporary credentails'

#3 of 15. Authenticate with eks-user access key
export AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXXXX
export AWS_SECRET_ACCESS_KEY=YYYYYYYYYYYYYYYYYYYYYYYYY

#4 of 15. Create eks cluster [Warning: this step may take 15 minutes or so]
cd ~/environment/human-gov-infrastructure/terraform
eksctl create cluster --name humangov-cluster --region us-east-1 --nodegroup-name standard-workers --node-type t3.medium --nodes 1

#5 of 15. Update local Kubernetes config
aws eks update-kubeconfig --name humangov-cluster --region us-east-1

#6 of 15. Verify Cluster Connectivity
kubectl get svc
kubectl get nodes

#7 of 15. Load Balancer
cd ~/environment
curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.5.4/docs/install/iam_policy.json

aws iam create-policy \
    --policy-name AWSLoadBalancerControllerIAMPolicy \
    --policy-document file://iam_policy.json


#8 of 15. Associate IAM OIDC provider
eksctl utils associate-iam-oidc-provider --cluster humangov-cluster --approve

#9 of 15. Create service account for load balancer.
eksctl create iamserviceaccount \
  --cluster=humangov-cluster \
  --namespace=kube-system \
  --name=aws-load-balancer-controller \
  --role-name AmazonEKSLoadBalancerControllerRole \
  --attach-policy-arn=arn:aws:iam::502983865814:policy/AWSLoadBalancerControllerIAMPolicy \
  --approve

#10 of 15. Install load balancer controller
# Add eks-charts repository.
helm repo add eks https://aws.github.io/eks-charts


# Install 
helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName=humangov-cluster \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller

#11 of 15. Verify controller installation
kubectl get deployment -n kube-system aws-load-balancer-controller

#12 of 15. Create role and service account for cluster to S3 and DynamoDB tables
eksctl create iamserviceaccount \
  --cluster=humangov-cluster \
  --name=humangov-pod-execution-role \
  --role-name HumanGovPodExecutionRole \
  --attach-policy-arn=arn:aws:iam::aws:policy/AmazonS3FullAccess \
  --attach-policy-arn=arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess \
  --region us-east-1 \
  --approve

#13 of 15. 
cd ~/environment/human-gov-application/src
kubectl get pods
kubectl apply -f humangov-california.yaml
kubectl apply -f humangov-florida.yaml
kubectl get pods
kubectl get svc
kubectl get deployment

#14 of 15. Ingress
kubectl apply -f humangov-ingress-all.yaml
kubectl get ingress

#15 of 15. Double-check Route 53
Make sure the A records for california.humangov-ll3.click and florida.humangov-ll3.click point to the new load balancer

Prerequisite 2 of 2. A pre-existing AWS CodeCommit repository named `human-gov-application` containing the source code of the HumanGov SaaS application.

If you do not have this repository, please create it. Instructions for this are not provided because if you were following this series, you did not delete that repository yet. [I only did the prior steps because I had to do them anyway.]

1 of 13. [Cloud9] Open AWS Cloud9

2 of 13. [Cloud9] Validate connectivity to cluster


kubectl get nodes

3 of 13. [CodeCommit] Validate you have a repository named 'human-gov-application'.

4 of 13. [Cloud9] Push changes to repository.


cd ~/environment/human-gov-application/src
git status
git add -A
git commit -m "added k8s manifests"
git push

5 of 13. [Code Pipeline] Create new pipeline

This works very similarly to Jenkins.


AWS CodePipeline -/- [Create pipeline]

STEP 1 Choose Pipeline settings:
Pipeline name: human-gov-cicd-pipeline
[Next]

STEP 2 Add source stage:
Source provider: AWS CodeCommit
Repository name: human-gov-application
Branch name: master
Change detection options: Amazon CloudWatch Events
Ouput artifact format: CodePipeline default
[Next]

STEP 3 Add build stage
Build provider: AWS CodeBuild
Region: us-east-1 (N. Virginia)
Project name: [Create project]

Project name: HumanGovBuild
Environment image: Managed image
Compute: EC2
Operating System: Amazon Linux
Runtime(s): Standard
Image: aws/codebuild/amazonlinux2-x86_64-standard:4.0
Service role: New service role
Role name: [auto-configured]

[Additional Configuration]
Privileged: Enable this flag if you want to build Docker images or want your builds to get elevated privileges

VPC: choose the VPC running your cluster
Subnets: Choose the PRIVATE subnets for your cluster.
Security Group: eks-cluster-sg [pick the one that starts like that naming convention, as you configured it in earlier articles]
[Validate VPC Settings]

Build specification: Insert build commands -/- [Switch to editor]
Add the contents of the buildspec.yml below.

buildspec.yml


version: 0.2
phases:
  install:
    runtime-versions:
      docker: 20
  pre_build:
    commands:
      - echo Logging in to Amazon ECR...
      - aws --version
      - REPOSITORY_URI=$ECR_REPO
      - aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/i7y0m4q9
  build:
    commands:
      - echo Build started on `date`
      - echo Building the Docker image...
      - cd src
      - docker build -t $REPOSITORY_URI:latest .
      - docker tag $REPOSITORY_URI:latest $REPOSITORY_URI:$CODEBUILD_RESOLVED_SOURCE_VERSION
  post_build:
    commands:
      - echo Build completed on `date`  
      - echo Pushing the Docker image...
      - docker push $REPOSITORY_URI:latest
      - docker push $REPOSITORY_URI:$CODEBUILD_RESOLVED_SOURCE_VERSION
      - export imageTag=$CODEBUILD_RESOLVED_SOURCE_VERSION
      - printf '[{\"name\":\"humangov-app\",\"imageUri\":\"%s\"}]' $REPOSITORY_URI:$imageTag > imagedefinitions.json
      - cat imagedefinitions.json
      - ls -l

env:
  exported-variables: ["imageTag"]
  
artifacts:
  files: 
    - src/imagedefinitions.json
    - src/humangov-california.yaml
    - src/humangov-florida.yaml


continuing STEP 3 ...

Add environment variable: ECR_REPO
Value: public.ecr.aws/i7y0m4q9/humangov-app
# The value corresponds to your humangov-app CodeCommit repository

# Replace the line that starts 'AWS ecr-public get-login-password ...'  (copy from your view push commands)
aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/i7y0m4q9

CloudWatch logs - optional [enabled]

[Continue to CodePipeline]

Project name: HumanGovBuild
[Next]


STEP 4 Add deploy stage
[Skip deploy stage]

STEP 5 Review
[Create pipeline]

6 of 13. [Code Pipeline] Pipeline fails to build

The build in the pipeline will not work initially,. and review of the logs shows a permissions error.

7 of 13. [IAM] Add permissions to Registry

Make sure to choose the policy with 'public' with the name. The one without Public in the name is for private repositories [recall that we are using a Public repository].


AWS IAM -/- Access management -/- [Roles]
[codebuild-HumanGovBuild-service-role]
Permissions -/- [Attach policy]
Select 'AmazonElasticContainerRegistryPublicFullAccess'
[Add permissions]

8 of 13. [CodePipeline] Retry the stage that failed

It should succeed, because the login was successful this time.

9 of 13. [CodePipeline] Add deployment

Note: This is not best practices authentication. The focus of this lab is CI/CD, and not security. If you want an example that leverages authentication in a somewhat better way, go see the yet-to-be-released series of Jenkins articles, that will be published [as time permits -- so maybe never?]


AWS CodePipeline -/- Pipelines -/- [human-gov-cicd-pipeline] -/- [Edit]
[Add stage] #after the build stage

Stage name: HumanGovDeployToProduction
[Add stage]

[Add action group]
Action name: HumanGovDeployToProduction
Action provider: AWS CodeBuild
Region: US East-1 (N. Virginia)

Input artifacts: BuildArtifact

Project name: [Create project]

Project name: HumanGovDeployToProduction
Environment: managed image
Compute: EC2
Operating system: Amazon Linux
Runtime(s): Standard
Image: aws/codebuild/amazonlinux2-x86_64-standard:4.0

Service role: New service role
Role name: auto-generated

[Additional configuration]
Privileged: Enable this flag if you want to build Docker images or want your builds to get elevated privileges
VPC: the one for the cluster
Subnets: Choose the private subnets for the cluster
Security group: eks-cluster-sg-humangov-cluster ...

[Validate VPC Settings]

Environment variables:
Name: AWS_ACCESS_KEY_ID Value: XXXXX
Name: AWS_SECRET_ACCESS_KEY Value: YYYYYYYYYYYYYYYY
# Note: IAM role is the recommmended/more-secure way to do this.

Buildspec:
Build specifications: Insert build commands
Build commands: [Switch to editor]

Add the contents of the below 'buildspec.yml'


version: 0.2

phases:
  install:
    runtime-versions:
      docker: 20
    commands:
      - curl -o kubectl https://amazon-eks.s3.us-west-2.amazonaws.com/1.18.9/2020-11-02/bin/linux/amd64/kubectl
      - chmod +x ./kubectl
      - mv ./kubectl /usr/local/bin
      - kubectl version --short --client
  post_build:
    commands:
      - aws eks update-kubeconfig --region $AWS_DEFAULT_REGION --name humangov-cluster
      - kubectl get nodes
      - ls
      - cd src
      - IMAGE_URI=$(jq -r '.[0].imageUri' imagedefinitions.json)
      - echo $IMAGE_URI
      - sed -i "s|CONTAINER_IMAGE|$IMAGE_URI|g" humangov-california.yaml
      - sed -i "s|CONTAINER_IMAGE|$IMAGE_URI|g" humangov-florida.yaml
      - kubectl apply -f humangov-california.yaml
      - kubectl apply -f humangov-florida.yaml

Continuing ..


CloudWatch Logs [checked]
[Continue to CodePipeline]

Project name: HumanGovDeployToProduction
[Done]

[Save]
# Don't forget to save when done making the modification to the pipeline.

10 of 13. [Cloud9] Update 'humangov-california.yaml' and 'humangov-florida.yaml'

The script will ultimately update the CONTAINER_IMAGE field with the $IMAGE_URI. You need to provide the base files that include CONTAINER_IMAGE, so those files can get updated.


Replace this:
        image: public.ecr.aws/i7y0m4q9/humangov-app:latest

With this:
        image: CONTAINER_IMAGE

11 of 13. [Cloud9] Update home.html


Replace this:
        <h1 class="display-5">HumanGov</h1>
With this:
        <h1 class="display-5">HumanGov SaaS Application</h1>

12 of 13. [Cloud9] Commit and push

Note: I had to run the add/commit/push one more time [not pictured here], BECAUSE I had not saved my Code Pipeline when initially adding the 'DeployToProduction".


cd ~/environment/human-gov-application/src
git status
git add -A
git commit -m "replaced image uri with CONTAINER_IMAGE and updated home.html"
git push