Multus Example:

In this blog, I am going to create a pod with two interfaces one having flannel network and the other having kube-router network.

Before we begin, note: you can find all the files used in this blog here.

Concept:

When using Multus, we have to do two things to add extra interfaces to our pods:

How it works?

When we tell kubectl to create our pod then kubelet creates network namespace for that pod and then required information is passed to Multus CNI plugin for adding network interface and setting up the network.
Multus checks for annotations: k8s.v1.cni.cncf.io/networks: for the names of interfaces to create if any. If so, then Multus searches for Network-Attachment-Definition for those interfaces and then add those interfaces to the pod.
(Note: Network-Attachment-Definition stores the CNI config for that interface. Multus simply passes this config to CNI with pod details and the new interface is added like a normal interface by calling the required plugin.)

Prerequisite:

I am assuming that you have created your Kubernetes cluster and have configured a default network. In our case, we are going to use flannel as our default network. You can create that with the following command after creating your cluster:
$ kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

Verification:

Run
$ kubectl get pods -A | grep -i flannel
and you should see a flannel pod Running like this: Note: Now, we are good to go with Multus.

Installing Multus

We can install and setup multus either manually or by using multus-deamonset.yml provided by Multus team. If you are doing it for the first time then it is recommended to use multus-daemonset.yml for keeping things simple. I am going to do it by daemonset method.
(If you want to know what daemonset is doing then read this Multus/how-to-use-guide)

For installing Multus by daemonset type the following command:

$ kubectl create -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset.yml

Verification:

Run
$ kubectl get pods -A | grep -i multus
and you should see a multus pod running like this: Note:

What the Multus daemonset file does

Note: we can we can override 00-multus.conf by creating our own 00-multus.conf in /etc/cni/net.d/ before running Multus Daemonset file.
In our case multus config is automatically generated by Daemonset using 00-flannel.conflist as it was alphabetically first file in /etc/cni/net.d.

Exploring 00-multus.conf

file: /etc/cni/net.d/00-multus.conf There are two methods to define 00-multus.conf file: You can read more about "clusterNetwork"+"defaultNetwork" method in Multus/ Configuration Reference. We are going to discuss the Delegate Method and other fields of 00-multus.conf:

Multus uses delegates for clusterNetwork and defaultNetwork. That is, this list of CNI config will always be used for any pod you create.

Here, we have specified only one delegate named "cbr0". So, any pod that we create will have an interface for this network by default. Since cbr0 is first in our delegate list, it will be used for creating "clusterNetwork" of Kubernetes (i.e. eth0 interface on all pod will use flannel). And if we would have specified more interface in delegates list then they would have been used for "defaultNetwork" by Multus. That is all pod that are created will have one interface for each delegate in list. (If we have 3 delegates in list then all pods will have 3 interfaces on creation)

Ok, that means our pod can have multiple interfaces now using delegate method. But all pods will have the same number of interfaces. What if we want to create one pod with 3 interfaces and the other with 4 interfaces?
We will use our Network-Attachment-Definition for that. The interface that we specified in delegates field of 00-multus.conf will remain same for all pods. If we want to add additional interfaces to our pods then we have to specify that in annotations: k8s.v1.cni.cncf.io/networks: with names of Network-Attachment-Definitions we create, delimited by comma(,). We will get to know more about Network-Attachment-Definitions usage in a while but before that few points about Multus:

Using Multus:

Let us create our second interface with kube-router network. As described before we have to create CRD of kind: Network-Attachment-Definition for defining our interfaces. The following is the definition of our required Network-Attachment-Definition for kube-router:

file: mynet.yml Three things to note: Now, let us create this Network-Attachment-Definition by command:
$ kubectl create -f mynet.yml

Verification:

Use command
$ kubectl get network-attachment-definitions
and you should see similar output: Now, we have successfully created our Network-Attachment-Definition.
If you want to see description of mynet Network-Attachment-Definition then use the command:
$ kubectl describe network-attachment-definition mynet

Last but not least: Creating the pod

file: pod.yml This is our simple pod definition with one new field under annotations k8s.v1.cni.cncf.io/networks: mynet telling multus to use Network-Attachment-Definition having name mynet.

Let us create our pod with the command:

$ kubectl create -f pod.yml

Once, the pod is created it will have 3 interfaces-

Verification: Checking for multiple interfaces

Use command:
$ kubectl exec -it nginx ip a

And you will see three interfaces as below: Cheers:)


If you are still stuck then you can ask for help in intel slack or open an issue in intel/multus-cni/issues

Conclusion

We have created a pod with two interfaces using Multus. This is the simplest example that I have covered but there is more that can be done. I have added a few links in the references section for those. If you want to create multiple interfaces without using delegates method then read this blog by Doug

References: