Network Policy Control On ACK Using The Terway Network Model

1. lab

1.1 Knowledge points

Alibaba Cloud Container Service on Kubernetes (ACK) is used in this lab. It describes how to use the Terway network model of Alibaba Cloud Container Service Kubernetes clusters for network policy control. Terway is an open-source CNI(Container Network Interface) plug-in for Alibaba Cloud’s VPC-based container network interface. It supports the definition of access policies between containers based on Kubernetes standard network policies.

1.2 lab process

• Overview of Alibaba Cloud Kubernetes Network
• Create a Terway network model based cluster
• Connect to the cluster
• Setup network policy

1.3 Scene architecture diagram

1.4 Cloud resources required

• ECS
• CS

1.5 Prerequisites

• If you’re using your own Alibaba Cloud account instead of the account provided by this lab, please note that you’ll need to choose the same Ubuntu 16.04 operating system for your ECS, in order to run the lab smoothly.
• Before starting the lab, please confirm that the previous lab has been closed normally and Alibaba Account has been logged out.

2. Start the lab environment

Click Start Lab in the upper right corner of the page to start the lab.

.

After the lab environment is successfully started, the system has deployed resources needed during the lab in the background. For example, the ECS instance, RDS instance, Server Load Balancer instance, and OSS bucket. An account consisting of the username and password for logging on to the Web console of Alibaba Cloud is also provided.

After the lab environment is started and related resources are properly deployed, the lab starts a countdown. You have an hour to perform labal operations. After the countdown ends, the lab stops, and related resources are released. During the lab, pay attention to the remaining time and arrange your time wisely. Next, use the username and password provided by the system to log on to the Web console of Alibaba Cloud and view related resources:

Go to the logon page of Alibaba Cloud console.

Fill in the sub-user account and click on Next.

Fill in the sub-user password and click on Login.

After successfully log on to the console, you will see the page showing as the following page.

3. Overview of Alibaba Cloud Kubernetes Network

Alibaba Cloud Container Service on Kubernetes Edition integrates Kubernetes network, Alibaba Cloud VPC, and Alibaba Cloud SLB to provide a stable and high-performance container network.

3.1 Container Network CNI

The container network model.

A containerized application deploys multiple services on the same node, and each business needs its own network space. To avoid conflicts with other business networks, pods need to have their own independent network space, and applications in pods need to communicate with other networks. Pods need to be able to access each other with different networks. The container network features are as follows:

• Each pod has its own network space and IP address. Applications of different pods can listen to the same ports without conflicts.
• Pods can access each other through their IP addresses. Pods in a cluster can communicate with other applications through their independent IP addresses:
  • Pods in the same cluster can access each other.
  • Pods can directly accesses the ECS instances in the same VPC.
  • ECS instances in the same VPC can directly access pods.

In ACK, the user can implement the capabilities of the container network through two network models, namely the Flannel network mode and the Terway network mode.

3.2 Terway Network Mode

The Terway network model uses a cloud-native network solution to build a container network based on the ENI resources in the virtualized network of Alibaba Cloud. Pods are directly allocated with IP addresses in the VPC through ENI resources, without specifying the CIDR block of the VPC.

The characteristics of the Terway network model are:

• The container and the virtual machine are in the same network, which facilitates cloud-native migration.

• The network devices assigned to containers can be used for communication without relying on overlay packets or routing tables.

• The node size of a cluster is not limited by quotas such as routing tables or FDB forwarding of packets.

• You do not need to plan the Overlay CIP for containers. Multiple cluster containers can communicate with each other by setting open ports in security groups.

• The container can be directly attached to the SLB backend without using NodePort for forwarding on the node.

• NAT Gateway can perform SNAT for pods directly without the need for SNAT on the container CIDR block. The container accesses resources in the VPC, and the source IP addresses are all container IP addresses, which is convenient for auditing; container access to the external network does not depend on conntrack SNAT, reducing the failure rate.

• Terway network mode allows you to configure network access rules between pods through network policies (NetworkPolicy). A network policy (NetworkPolicy) is a specification of the allowed communication rules between pods and rules between pods and other network endpoints. NetworkPolicy resource using a tag to select pods and define the communication rules allowed by the selected pods.

Terway network mode supports the use of more efficient IPvlan + eBPF links to accelerate container network performance when using an operating system Alibaba Cloud Linux 2 system as a node.

3.3 Flannel network mode

The CIDR block of a pod in the Flannel network mode is independent of the CIDR block of the VPC. Pod CIDR blocks are evenly divided into nodes in each cluster according to the mask. Pods on each node are assigned IP addresses from the CIDR blocks of the nodes. The Flannel network is based on the custom routing capabilities of the Alibaba Cloud VPC to directly access the Pods across nodes to the VPC.

Flannel network mode is characterized:

• The Flannel network based on Alibaba Cloud VPC has no overlay packets, which improves the performance of the default Flannel VXLAN by 20%.
• A pod CIDR block is a virtual CIDR block that is independent of the VPC.
• Each node must correspond to a VPC route table entry. The number of Kubernetes nodes in a VPC is limited by the quota of the VPC routing table.

3.4 Kubernetes Cluster Network Planning

When you create an ACK Kubernetes cluster, you need to specify a VPC, VSwitch, CIDR (CIDR block), and Service CIDR block (CIDR block). Therefore, we recommend that you plan the ECS address, Kubernetes pod address, and Service address in advance. Next, we will introduce the functions of various address types in the ACK Kubernetes cluster in a VPC environment, and how to plan the network segments.

3.4.1 Terway

When configuring the Terway network, you need to set the parameters and parameters for the network segment with following guideline:

• VPC
  When creating a VPC, you must select a CIDR block. You can only select one of 10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16.

• Virtual switch
  The VSwitch used by the ECS instance for network communication between nodes. The CIDR block specified when creating a VSwitch in a VPC must be a subset of the current VPC CIDR block (which can be the same as the VPC CIDR block, but cannot exceed the CIDR block). When configuring the network segment, note the following:

  • A VSwitch is a VPC VSwitch.
  • The address assigned to the ECS instance under the VSwitch is obtained from the CIDR block of the VSwitch.
  • You can create multiple VSwitches in a VPC, but the CIDR blocks of VSwitches cannot overlap.
  • VSwitch and pod VSwitch must be in one zone.

• Pod VSwitch
  Pod addresses are assigned from the VSwitch for network communication. A pod is a concept within a Kubernetes, and each pod has an IP address. The CIDR block specified when creating a VSwitch in a VPC must be a subset of the current VPC CIDR block. When configuring the CIP segment, note the following:

  • A VSwitch is a VPC VSwitch.
  • In Terway network mode, the IP addresses allocated by the pod are obtained from the VSwitch Cage.
  • This CIDR block cannot overlap the Service CIDR block.
  • VSwitch and pod VSwitch must be in one zone.

• Service CIDR
  Service address segment. Service is a concept within the Kubernetes, corresponding to the address Service used when the Service type is ClusterIP, and each Service has its own address. When configuring the service network segment, note the following:

  • Service addresses are used only within the Kubernetes cluster and cannot be used outside the cluster.
  • The Service address segment cannot overlap with the virtual switch address segment.
  • The Service address segment cannot overlap with the pod virtual switch address segment.

Terway configuration example:

3.4.2 Flannel

When configuring the Flannel mode network, the parameters to be set and the parameters to be considered are as follows:

• VPC
  When creating a VPC, you must select a CIDR block. You can only select one of 10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16.

• Virtual switch
  The VSwitch used by the ECS instance for network communication between nodes. The CIDR block specified when creating a VSwitch in a VPC must be a subset of the current VPC CIDR block (which can be the same as the VPC CIDR block, but cannot exceed the CIDR block). When configuring the CIP segment, note the following:

  • A VSwitch is a VPC VSwitch.
  • The address assigned to the ECS instance under the VSwitch is obtained from the CIDR block of the VSwitch.
  • You can create multiple VSwitches in a VPC, but the CIDR blocks of VSwitches cannot overlap.

• Pod network CIDR
  The CIDR block of the pod network. The pod address is assigned from this CIDR block for pod network communication. A pod is a concept within a Kubernetes, and each pod has an IP address. When configuring the CIP segment, note the following:

  • The VSwitch is a virtual CIDR block.
  • The IP address segment cannot overlap with the VSwitch CIP.
  • This CIDR block cannot overlap the Service CIDR block.

• Service CIDR
  Service address segment. Service is a concept within the Kubernetes, corresponding to the address Service used when the Service type is ClusterIP, and each Service has its own address. When configuring the CIP segment, note the following:

  • Service addresses are used only within the Kubernetes cluster and cannot be used outside the cluster.
  • The Service address segment cannot overlap with the virtual switch address segment.
  • The Service address segment cannot overlap with the CIDR address segment of the pod network.

Flannel configuration example:

3.5 Using Terway Network Mode

Terway is an open-source CNI(Container Network Interface) plug-in for Alibaba Cloud’s VPC-based container network interface. It supports the definition of access policies between containers based on Kubernetes standard network policies. Users can use the Terway network plug-in to achieve network communication within the Kubernetes cluster.

Terway network plug-in is a self-developed network plug-in for ACK, which assigns Native ENIs to pods to implement pod networks and supports Kubernetes-based network policies (Network policies). defines the access policies between containers and is compatible with the network policies of Calico.

In the Terway network plug-in, each pod has its own network stack and IP address. The communication between pods in the same ECS instance is directly forwarded through the internal machine, pod communication across ECS instances, and packets are directly forwarded through the ENI of the VPC. Since there is no need to use tunneling technology such as VxLAN to encapsulate messages, the Terway mode network has high communication performance.

Terway network model diagram

Terway versus Flannel

When creating a cluster, ACK provides two network plug-ins, Terway and Flannel:

• Terway: is a self-developed network plug-in for Container Service ACK. Terway assigns Alibaba Cloud ENIs to containers. You can define access policies between containers based on Kubernetes standard network policies and limit bandwidth for a single container. Terway has more flexible IPAM (container address allocation) policies to avoid address waste. If the user does not need to use a Network policy, you can choose Flannel. In other cases, we recommend Terway.

• Flannel: Using a simple and stable community Flannel CNI plug-in. With the high-speed network of Alibaba Cloud VPC, it can provide high-performance and stable container network experience for clusters. Flannel features are simple and supported with few features. For example, network policies based on Kubernetes standards (Network Policy) are not supported

Note:

The maximum number of pods supported by a single node depends on the number of ENIs one node(ECS) can support.

Maximum number of pods supported by shared ENI = (number of ENI supported by ECS -1)