frameworkcontroller/example/run at master · microsoft/frameworkcontroller

We provide various approaches to run FrameworkController:

Notes:

For a single k8s cluster, one instance of FrameworkController orchestrates all Frameworks in all namespaces.
For a single k8s cluster, ensure at most one instance of FrameworkController is running at any point in time.
For the full FrameworkController configuration, see Config Usage and Config Example.

Run By Kubernetes StatefulSet

This approach is better for production, since StatefulSet by itself provides self-healing and can ensure at most one instance of FrameworkController is running at any point in time.
Using official image to demonstrate this example.

Prerequisite

If the k8s cluster enforces Authorization, you need to first create a ServiceAccount with granted permission for FrameworkController. For example, if the cluster enforces RBAC:

kubectl create serviceaccount frameworkcontroller --namespace default
kubectl create clusterrolebinding frameworkcontroller \
  --clusterrole=cluster-admin \
  --user=system:serviceaccount:default:frameworkcontroller

Run

Run FrameworkController with above ServiceAccount and the k8s inClusterConfig:

Run with default config

kubectl create -f frameworkcontroller-with-default-config.yaml

frameworkcontroller-with-default-config.yaml:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: frameworkcontroller
  namespace: default
spec:
  serviceName: frameworkcontroller
  selector:
    matchLabels:
      app: frameworkcontroller
  replicas: 1
  template:
    metadata:
      labels:
        app: frameworkcontroller
    spec:
      # Using the ServiceAccount with granted permission
      # if the k8s cluster enforces authorization.
      serviceAccountName: frameworkcontroller
      containers:
      - name: frameworkcontroller
        image: frameworkcontroller/frameworkcontroller
        # Using k8s inClusterConfig, so usually, no need to specify
        # KUBE_APISERVER_ADDRESS or KUBECONFIG
        #env:
        #- name: KUBE_APISERVER_ADDRESS
        #  value: {http[s]://host:port}
        #- name: KUBECONFIG
        #  value: {Pod Local KubeConfig File Path}

Run with customized config

kubectl create -f frameworkcontroller-customized-config.yaml
kubectl create -f frameworkcontroller-with-customized-config.yaml

frameworkcontroller-customized-config.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: frameworkcontroller-config
  namespace: default
data:
  frameworkcontroller.yaml: |
    kubeClientQps: 200
    kubeClientBurst: 300
    workerNumber: 500
    largeFrameworkCompression: true
    frameworkCompletedRetainSec: 2592000
    #podFailureSpec:
    #- code: 221
    #  phrase: ContainerTensorflowOOMKilled
    #  type:
    #    attributes: [Permanent]
    #  podPatterns:
    #  - containers:
    #    - messageRegex: '(?msi)tensorflow.*ResourceExhaustedError.*OOM.*'
    #      codeRange: {min: 1}
    #      nameRegex: '(?ms).*'
    #- {More customized podFailureSpec, better to also include these in the default config}

frameworkcontroller-with-customized-config.yaml:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: frameworkcontroller
  namespace: default
spec:
  serviceName: frameworkcontroller
  selector:
    matchLabels:
      app: frameworkcontroller
  replicas: 1
  template:
    metadata:
      labels:
        app: frameworkcontroller
    spec:
      # Using the ServiceAccount with granted permission
      # if the k8s cluster enforces authorization.
      serviceAccountName: frameworkcontroller
      containers:
      - name: frameworkcontroller
        image: frameworkcontroller/frameworkcontroller
        # Using k8s inClusterConfig, so usually, no need to specify
        # KUBE_APISERVER_ADDRESS or KUBECONFIG
        #env:
        #- name: KUBE_APISERVER_ADDRESS
        #  value: {http[s]://host:port}
        #- name: KUBECONFIG
        #  value: {Pod Local KubeConfig File Path}
        command: [
          "bash", "-c",
          "cp /frameworkcontroller-config/frameworkcontroller.yaml . &&
          ./start.sh"]
        volumeMounts:
        - name: frameworkcontroller-config
          mountPath: /frameworkcontroller-config
      volumes:
      - name: frameworkcontroller-config
        configMap:
          name: frameworkcontroller-config

Run By Docker Container

This approach may be better for development sometimes.
Using official image to demonstrate this example.

Run

If you have an insecure ApiServer address (can be got from Insecure ApiServer or kubectl proxy) which does not enforce authentication, you only need to provide the address:

docker run -e KUBE_APISERVER_ADDRESS={http[s]://host:port} \
  frameworkcontroller/frameworkcontroller

Otherwise, you need to provide your KubeConfig File which inlines or refers the ApiServer Credential Files with granted permission:

docker run -e KUBECONFIG=/mnt/.kube/config \
  -v {Host Local KubeConfig File Path}:/mnt/.kube/config \
  -v {Host Local ApiServer Credential File Path}:{Container Local ApiServer Credential File Path} \
  frameworkcontroller/frameworkcontroller

For example, if the k8s cluster is created by Minikube:

docker run -e KUBECONFIG=/mnt/.kube/config \
  -v ${HOME}/.kube/config:/mnt/.kube/config \
  -v ${HOME}/.minikube:${HOME}/.minikube \
  frameworkcontroller/frameworkcontroller

Run By OS Process

This approach may be better for development sometimes.
Using local built binary distribution to demonstrate this example.

Prerequisite

Ensure you have installed Golang 1.12.6 or above and the ${GOPATH} is valid.

Then build the FrameworkController binary distribution:

export PROJECT_DIR=${GOPATH}/src/github.com/microsoft/frameworkcontroller
rm -rf ${PROJECT_DIR}
mkdir -p ${PROJECT_DIR}
git clone https://github.com/microsoft/frameworkcontroller.git ${PROJECT_DIR}
cd ${PROJECT_DIR}
./build/frameworkcontroller/go-build.sh

Run

If you have an insecure ApiServer address (can be got from Insecure ApiServer or kubectl proxy) which does not enforce authentication, you only need to provide the address:

KUBE_APISERVER_ADDRESS={http[s]://host:port} \
  ./dist/frameworkcontroller/start.sh

Otherwise, you need to provide your KubeConfig File which inlines or refers the ApiServer Credential Files with granted permission:

KUBECONFIG={Process Local KubeConfig File Path} \
  ./dist/frameworkcontroller/start.sh

For example:

KUBECONFIG=${HOME}/.kube/config \
  ./dist/frameworkcontroller/start.sh

And in above example, ${HOME}/.kube/config is the default value of KUBECONFIG, so you can skip it:

./dist/frameworkcontroller/start.sh

Submit Framework

Run By Kubernetes StatefulSet

Prerequisite

Run

Run with default config

Run with customized config

Run By Docker Container

Run

Run By OS Process

Prerequisite

Run

Next