ETSI NFVO Compliant Orchestration in the Kubernetes/Cloud Native World

Publication date: October 25, 2022 (Document revisions)

Abstract and introduction

This whitepaper examines the network functions virtualization (NFV) orchestration requirements and specification as put forth by the European Telecommunication Standard Institute (ETSI), in the context of the Kubernetes way of orchestrating services and applications. This whitepaper explores the interplay between the ETSI components and Kubernetes/Amazon Elastic Kubernetes Service (Amazon EKS) framework by examining requirements that can map from one framework to another, and outlines some of the gaps in mapping others. We also examine evolving business and operational needs, and importance of a true cloud-native orchestration framework to address those needs. We further provide a high-level architecture of a cloud-native orchestrator that can be implemented cost-effectively and reliably by some of the AWS cloud services.

This whitepaper is aimed at Communication Services Providers (CSPs) and Independent Software Vendors (ISVs).

Are you Well-Architected?

The AWS Well-Architected Framework helps you understand the pros and cons of the decisions you make when building systems in the cloud. The six pillars of the Framework allow you to learn architectural best practices for designing and operating reliable, secure, efficient, cost-effective, and sustainable systems. Using the AWS Well-Architected Tool, available at no charge in the AWS Management Console, you can review your workloads against these best practices by answering a set of questions for each pillar.

For more expert guidance and best practices for your cloud architecture—reference architecture deployments, diagrams, and whitepapers—refer to the AWS Architecture Center.

Introduction

In the past, before CSPs push toward virtualization, Independent Software Vendors (ISVs) often created their own proprietary ways of managing hardware and software, sometimes integrated, sometimes separate. While it was difficult to manage a multi-vendor network, methods for each particular physically integrated functions (Physical network functions, or PNFs) worked reasonably well, and CSPs managed to create 2G and 3G networks that are still working in parts of the world.

Running network in such ways was expensive, and operators were often left with huge amounts of hardware that came with dedicated software functions, and became obsolete and required “forklift” upgrades.

When virtualization started being popular in Enterprise space, operators wanted to take advantage of hardware and software separation, and a network function virtualization (NFV) workgroup in the European Telecommunications Standards Institute (ETSI) was created in 2013. This workgroup looked at the concepts and issues of management of network functions and infrastructure in a virtualized world of virtual machines (VMs) and came up with a prescriptive reference called NFV Management and Orchestration (MANO). While the framework was often not implemented as-is, it did a great job in abstracting requirements and providing a unified framework to look at operations and management. As part of this, the group proposed using Topology and Orchestration Specification for Cloud Applications (TOSCA) as a declarative method of indicating network service and function requirements.

In the late 2000s and early 2010s, containers started becoming popular for software development and deployment. Containers are units of application that package code and all dependencies and can be run individually and reliably from one environment to another. Since an application consisted of many of these “container modules”, also called Pods, a way to manage them was needed. Many different container orchestration systems were developed, but the one that became most popular was an open-source project called Kubernetes. Kubernetes ensured declarative interfaces at each level and defined a set of building blocks/intents ("primitives") in terms of API objects. These objects are representation of various resources such as Pods, Secrets, Deployments, Services. Kubernetes ensured that its design was loosely coupled, which made it easy to extend it to support the varying needs of different workloads, while still following intent-based framework.

TOSCA and Kubernetes both use YAML representation. The main difference is that ETSI MANO focused on specifying operations at each interface, whereas Kubernetes focused on the specification of the end state of the operation via APIs. In some ways they are similar. ETSI MANO with TOSCA templates started as a declarative framework in terms of Network Service Descriptor (NSD) and provided an abstraction for Virtual Network Functions (VNFs) independent of specific network functions. However, ETSI MANO with TOSCA then mapped this declarative framework to the imperative style of specification at lower levels. When ETSI NFV was developed, there were multiple competing virtualization frameworks such as OpenStack and VMware. Therefore, ETSI NFV had to create a common abstraction of them to specify architecture and requirements. While the advantages of these abstractions were multiple, creating another layer of abstractions made specifications difficult to apply. Further, mapping declarative specification to imperative operations was also difficult.

Kubernetes kept the philosophy of defining an application independent of its function, but in contrast to ETSI MANO, it kept everything intent/API driven and defined a declarative way to define objects such as deployments. Once an application was defined in terms of its objects, Kubernetes ensured that the application was properly run, and ensured that the various parts of application could communicate easily. Determining and applying application specific configuration is beyond the scope of both ETSI MANO and Kubernetes, though they both provide some supporting constructs such as an element management system (EM), an operator’s framework, or config files.

Network functions have additional requirements than commonly found cloud-applications. ETSI MANO framework was specifically designed to address these specialized requirements in the context of virtualized network functions. As we move to containerization, it becomes important to map the management and orchestration of CNFs to the ETSI MANO framework so that none of the requirements are missed and the containerized orchestration continues to work well without mandating changes in the business and operation support systems. With Kubernetes becoming the dominant common platform for container orchestration, an orchestration specification directly in terms of native Kubernetes terms would provide simplification and enable taking advantage of Kubernetes features and extensions to address MANO requirements. In the absence of such directly mapped specifications, multiple ways of interpretating the MANO requirements in the cloud-native/containerized contexts are possible. There have been multiple projects to address some of the challenges of defining common function orchestration system in containerized and cloud-native contexts. Linux open-source projects, such as ONAP, Nephio, and Anuket point to the need of such work.

In this whitepaper, we examine each component and requirement of NFV MANO architecture, and explore how they are normally addressed in a cloud native world. To keep the discussion closer to implementation, sometimes this whitepaper explores these requirements in the context of AWS services (such as Amazon EKS, Amazon Elastic Container Registry (Amazon ECR, and Amazon Elastic Compute Cloud (Amazon EC2) Auto Scaling groups) to avoid adding another layer of abstraction and ensure easy readability.