Avi on AWS: Comprehensive Installation Guide

Table of Contents

• Introduction

• Preparation

  • AWS credentials for Avi Setup
  • Setup IAM roles
  • Avi Sizing
  • Network Requirements

• Avi Controller

• Avi Controller install process

  • Initial password setup
  • Initial Avi setup
  • AWS Cloud Setup
  • Configure Avi license
  • Configure Service Engine Group

• Conclusion

Introduction

VMware NSX Advanced Load Balancer (Avi) is a self-service Multi-Cloud App Services Platform that ensures consistent application delivery, with software load balancers, web application firewall (WAF), and container Ingress solutions for app's across data centers and clouds.

Avi can be installed on multiple clouds, but the process differs based on cloud resources and services. In this blog, I have presented a simplified way to install and configure VMware NSX Advanced Load Balancer (Avi) on AWS.

Preparation

Before we start with installation, it's good to make certain checks and perform configurations in AWS.

AWS credentials for Avi Setup

There are 2 ways to connect Avi with AWS -

• access credentials
• IAM roles

Though, access credentials for a user with sufficient privileges can be used to connect Avi with AWS, but it's highly recommended to configure IAM roles.

Setup IAM roles

To configure Avi to use AWS resources, we have to create 2 roles with corresponding policies in AWS.

• vmimport
• AviController-Refined-Role

Avi team have already prepared necessary roles and policies which can be readily applied via AWS CLI or console. These roles and policies can be found in https://github.com/avinetworks/devops.git project at subpath devops/aws/iam-policies. Operator can also refer role install docs for more clarity on the roles.

IAM roles can be easily applied using below AWS CLI commands:

vmimport role :

aws iam create-role --role-name vmimport --assume-role-policy-document file://vmimport-role-trust.json
aws iam put-role-policy --role-name vmimport --policy-name vmimport --policy-document file://vmimport-role-policy.json
aws iam put-role-policy --role-name vmimport --policy-name AviController-vmimport-KMS-Policy --policy-document file://avicontroller-kms-vmimport.json

For AviController-Refined-Role role :

First, we would create the policies for the role.

aws iam create-role --role-name AviController-Refined-Role --assume-role-policy-document file://avicontroller-role-trust.json
aws iam create-policy --policy-name AviController-EC2-Policy --policy-document file://avicontroller-ec2-policy.json
aws iam create-policy --policy-name AviController-S3-Policy --policy-document file://avicontroller-s3-policy.json
aws iam create-policy --policy-name AviController-IAM-Policy --policy-document file://avicontroller-iam-policy.json
aws iam create-policy --policy-name AviController-R53-Policy --policy-document file://avicontroller-r53-policy.json
aws iam create-policy --policy-name AviController-ASG-Policy --policy-document file://avicontroller-asg-policy.json
aws iam create-policy --policy-name AviController-SQS-SNS-Policy --policy-document file://avicontroller-sqs-sns-policy.json
aws iam create-policy --policy-name AviController-KMS-Policy --policy-document file://avicontroller-kms-policy.json

Next step, would be to attach the policies to the AviController-Refined-Role role.

aws iam attach-role-policy --role-name AviController-Refined-Role --policy-arn "arn:aws:iam::<projectid>:policy/AviController-EC2-Policy"
aws iam attach-role-policy --role-name AviController-Refined-Role --policy-arn "arn:aws:iam::<projectid>:policy/AviController-S3-Policy"
aws iam attach-role-policy --role-name AviController-Refined-Role --policy-arn "arn:aws:iam::<projectid>:policy/AviController-R53-Policy"
aws iam attach-role-policy --role-name AviController-Refined-Role --policy-arn "arn:aws:iam::<projectid>:policy/AviController-IAM-Policy"
aws iam attach-role-policy --role-name AviController-Refined-Role --policy-arn "arn:aws:iam::<projectid>:policy/AviController-SQS-SNS-Policy"
aws iam attach-role-policy --role-name AviController-Refined-Role --policy-arn "arn:aws:iam::<projectid>:policy/AviController-ASG-Policy"
aws iam attach-role-policy --role-name AviController-Refined-Role --policy-arn "arn:aws:iam::<projectid>:policy/AviController-KMS-Policy"

NOTE: In the above commands, we need to replace the projectId to corresponding AWS Account.

To complete the role setup, we create the instance profile for AviController-Refined-Role, which will enable role to be attach with EC2 instance.

aws iam create-instance-profile --instance-profile-name AviController-Refined-Role
aws iam add-role-to-instance-profile --instance-profile-name  AviController-Refined-Role --role-name AviController-Refined-Role

Avi Sizing

Detailed sizing requirement for Avi can be referred here.

To summarize, we need following sizing for Avi controller :

• Controller: 8 vCPU cores, 24 GB RAM, and 128 GB of storage
• Service Engine: 1 vCPU cores, 2 GB RAM, and 15 GB of storage

Avi controller sizing on AWS

NSX Advanced Load Balancer recommends general purpose or compute/ memory optimized instances for running Avi Controllers.

Burstable instances are not recommended for running Controller virtual machines. NSX Advanced Load Balancer Controller recommends SE with minimum memory of 2 GB, and 1 vCPU.

Typical HA deployment for Avi has three Controllers. The number of SEs required depends on number of applications being served and the configured level of redundancy.

Network Requirements

NSX Advanced Load Balancer Service Engine data interfaces can be assigned to multiple VRFs (Virtual Routing and Forwarding Context).

Ports and protocol required for Avi (v22.1.4) in a restricted environment can be found here.

In case, we are looking for ports related to a different version of Avi ( in future), we can refer the VMware global ports and protocol page here.

Avi Controller

Avi controller can be developed as a single EC2 instance. It can be configured with cloud setup to connect kubernetes cluster and SEs to provision virtual services.

NSX Advanced Load Balancer Service Engines handle all data plane operations within the NSX Advanced Load Balancer by receiving and executing instructions from the Controller. It performs load balancing and all client and server-facing network interactions. SE collects real-time telemetry data from application traffic flows.

Avi Controller install process

We can get the latest NSX Advanced Load Balancer AMI from AWS Marketplace.

AWS provides the manual launch (EC2 Console) process for the EC2 instance, where we can provide following information during the installation process.

1. Select the AWS region for EC2
2. Instance type can be m5.2xlarge ( for other compatible type, refer avi-sizing)
3. Associate the instance with key-pair
4. Pre-configured VPC and subnet for Avi EC2
5. Auto assign public IP ( if Avi controller to be installed in public subnet).
6. Security group that allows traffic through the firewall, to allow communication between the Controller and the Service Engines (SEs)

• SSH (22)
• http (80)
• https (443)
• custom-tcp (8443)
• udp (123)

7. Storage 128gb or more
8. Select IAM Instance profile - AviController-Refined-Role

With above configuration, Avi controller instance would start provisioning. It might take some time to get configured. Once the instance is running, we can access it's web interface at port 443 with the assigned Public/Private IPv4 address or Public IPv4 domain.

Initial password setup

Log in to the instance with SSH using user as admin and assigned ec2 key-pair and use the sudo /opt/avi/scripts/initialize_admin_user.py script to configure the admin password for the first time login to the Controller

Initial Avi setup

Access Avi GUI and provide the passphrase password ( It's used for Avi instance data backup setup which can be performed in later stage), DNS (optional) and save changes to move for the aws cloud setup in Avi.

AWS Cloud Setup

Operator can navigate in GUI to infrastructure section and create a new cloud of AWS type and configure following details .

1. AWS credentials ( Ensure to choose iam role option to use configured role).
2. Select the AWS region
3. Availability Zone & Service Engine Management Network ( we can select 1 or more AZ with subnets [private/public] which would be used to provision Service Engine instances)
4. Select Use Encryption for SE S3 Bucket and Use Encryption for SE AMI/EBS volumes (Optional)
5. DNS provider ( It's recommended to select Amazon Route 53 )

Access troubleshooting

Some users might get an error while configuring AWS Credentials as Role. The error might look something like below. To resolve the error we need to verify that the AWS role has got all the policies attached.

Configure Avi license

VMware NSX Tanzu customers can subscribe for Avi license based on their requirements. Avi can be configured with license key based on their entitlements. Accordingly Avi would get configured with features and required core counts. These cores would be used by Service Engine and define the usage. The details for licensing can be found here.

Operator can configure Avi's license key for Enterprise , Basic or Essential tier by navigating to Avi console at Administration > Licensing section.

Trial License for Enterprise Tier

Avi with Enterprise Tier can be configured with 1 months expiry for evaluation license with 22 service cores for testing purpose. After 1 month Avi with Enterprise Tier can be degraded to Trial license with 2 service cores for testing purpose in non-prod environments.

Configure Service Engine Group

Operator can navigate to infrastructure > cloud resources and verify the default SE Group for configured cloud. We can either use the default SE Group created for the aws cloud in Avi or we can create a new SE group instance. All the options and fields are pre-populated and it's fine to start Avi SE Group with defaults. These fields can be changed later.

SE in HA configuration

NSX Advanced Load Balancer SE groups support following HA modes:

Elastic HA: It provides fast recovery for individual virtual services following the failure of the SE. Depending on the mode, the virtual service is already running on multiple SEs or is quickly placed on another SE.

The following modes of cluster HA are supported:

• Active/Active
• N + M

Legacy HA: It emulates a 2-device hardware active/standby HA configuration operation. The active SE carries all the traffic for a virtual service placed on it. The other SE in the pair is the standby for the VS, carrying no traffic when the active SE is healthy.

For getting more details to configure SE in Elastic HA mode, we can refer this page.

Once SE Group is configured, revisit cloud section and update the Template Service Engine Group option for the configured cloud in Avi.

NOTE: Default value for Service Engine Name Prefix as Avi should match to the value in S3 IAM policy. It's recommended to leave it unchanged.

The above config would kickoff cloud config in Avi and it would temporarily put the Avi SE AMI in S3 bucket.

Then, the AMI would be finally saved in AWS within EC2 AMI section.

NOTE: At any point, we can verify the events under Operations section, to check for any error or task performed by Avi for configurations.

Now, Avi controller is setup with the Service Engine and can be configured for L4 and L7 use cases.

Conclusion

With the above changes, NSX Advanced Load Balancer (Avi) is configured with license for aws cloud and configurations for SE engine. These configs are sufficient to start with setting up Kubernetes cluster or services to connect with Avi.

Next Step

As the next step, we can configure a Kubernetes cluster (Amazon EKS) with Avi Kubernetes Operator (AKO) to connect with provisioned Avi controller and launch LoadBalancer service or ingress resource for a deployed application. Step by step process is discussed in next post ➡️ Deploying Kubernetes Operator for Avi on Amazon EKS