Goal

To evaluate options for connecting “external” IP addresses to kubernetes-hosted applications.

Non-Goals

To discuss the kubernetes Service abstraction (mostly, see #1107).

Background

Running Pods and Services within a kubernetes cluster is fairly well defined (or becoming so).  A frequent question, though, is how to publish something to the “outside world” (for which the definition varies depending on the hosting situation).  Today we have a Service.CreateExternalLoadBalancer flag, which tells kubernetes to invoke cloud-provider logic to establish a load balancer across all minions on the service port on the minion’s interface.  This port is answered by the kube-proxy, which does it’s own round-robin “balancing”.

This design was easy to establish as a prototype, but is problematic for the long term because it depends on a flat service-port namespace, which we want to make go away (see #1107), and because it uses two levels of balancing, which is not good for performance or predictability.

We should be able to establish a more performant and direct pattern for externalizing kubernetes-hosted applications.

There are a few ways this could be done.  The remainder of this doc will explore the trade-offs.  I will describe the need for external access as an external port, rather than an external IP.  This leaves the implementations free to optimize IP usage, if needed.

Option 1) Flag some services as “external”, run load balancers inside k8s

Services carry a flag indicating the need for an external port.  When a Service is created with this flag set, the kubernetes master will spin up a new pod (call it a “service pod”) which runs a kubernetes-aware balancer (this might just be kube-proxy or an enlightened HAProxy or ...).  Like all pods, this pod must be managed by a controller (call it a “service controller”) and is given a pod IP address.  The controller for this pod has the responsibility to reach out to the cloud provider and provision an external port which forwards to the service pod’s IP.  Whatever portal solution we decide to use (see #1107) would forward to this service pod.  If the service pod is relocated, the service controller may need to update the cloud provider.

Pros:

• Fairly consistent experience across cloud providers
• Same implementation, minus external, can be used for “real” internal load balancing

Cons:

• Less obvious how to integrate real cloud-provider load-balancing
• Users don’t get to choose their balancing solution
  • could be mitigated by extra fields in Service
• Expensive for singleton pods that want an external port
  • could be mitigated by changing the external flag to indicate N=1 vs N>1

Option 2) Flag some services as “external”, let cloudprovider define the meaning

Similar to option 1, but more abstract.  Services carry a flag indicating the need for an external port.  When a Service is created with this flag set, the kubernetes master will reach out to the cloud provider and provision an external service.  The cloud provider modules would determine what this means.  For the simplest cases, this could be implemented the same as option 1.  For cloud providers that can use native load balancers, this could be implemented in terms of them.

Pros:

• As an abstraction, provides freedom to evolve the implementation
• Sets a pattern for internal load balancing as a function of cloud provider

Cons:

• Less consistent across cloud providers
• Users don’t get to choose their balancing solution
  • could be mitigated by extra fields in Service
• Expensive for singleton pods that want an external port
  • could be mitigated by changing the external flag to indicate N=1 vs N>1

Option 3) Any pod can be flagged as “external”, users set up their own balancing

Similar to option 2, but not tied to Services.  Pods carry a flag indicating the need for an external port.  When a Pod is created with this flag set, the kubernetes master will reach out to the cloud provider and provision an external port.  The cloud provider modules would determine what this means.  The end user is responsible for setting up the pod to be a load balancer, though we can make kube-proxy be suitable for this purpose.  Because this is a plain pod, it needs to be run under a controller, itself - probably a replication controller of N=1.  In order for this pattern to hold for internal load balancing, the user would need another Service with a Portal (see #1107) that forwards to it.

Pros:

• Users can configure anything they want as their balancer
• This is the underlying form of options 1 and 2 anyway
• Conceptually simple
• Very composable
• Works well for singleton pods that want an external IP but don’t need balancing

Cons:

• Forces users to set up balancing themselves
• Less obvious how to integrate real cloud-provider load-balancing
• Not obvious which master module is responsible for syncing external IP to internal IP when the pod is relocated
• Internal use through Portals (See DESIGN: Services v2 #1107) would be an extra hop (kube-proxy + load balancer) since it is not part of the Service abstraction
  • could be mitigated with a new kind of Service, “singleton” or something to make a direct portal