在Kubernetes集群中运行虚拟机

k8s将成为云原生计算的操作系统，那么将虚拟机运行在k8s之上也成为一个实际需求和发展趋势。今天将介绍如何利用KubeVirt运行虚拟机。

| KubeVirt介绍

virt-api

kubevirt API服务，kubevirt是以CRD的方式工作的，virt-api提供了自定义的api请求处理，如vnc console start vm stop vm等。

virt-controller

与k8s api-server通讯监控VMI资源创建删除等状态
根据VMI定义创建virt-launcherpod，该pod中将会运行虚拟机
监控pod状态，并随之更新VMI状态
监控标记为”kubevirt.io/schedulable” node heartbeat

virt-handler

运行在kubelet的node上定期更新heartbeat，并标记”kubevirt.io/schedulable”
监听在k8s apiserver当发现VMI被标记得nodeName与自身node匹配时，负责虚拟机的生命周期管理

virt-launcher

以pod形式运行
根据VMI定义生成虚拟机模板，通过libvirt API创建虚拟机
每个虚拟机会对应对立的libvirtd
与libvirt通讯提供虚拟机生命周期管理

虚拟机创建过程

client 发送创建VMI命令达到 k8s API server.
K8S API 创建VMI 对象
virt-controller监听到VMI创建时，根据VMI spec生成pod spec文件，创建pods
k8s调度创建pods
virt-controller监听到pods创建后，根据pods的调度node，更新VMI 的nodeName
virt-handler监听到VMI nodeName与自身节点匹配后，与pod内的virt-launcher通信，virt-laucher创建虚拟机，并负责虚拟机生命周期管理

| 安装KubeVirt

检查k8s主机节点虚拟化环境，并安装虚拟机组件

zhipu@zps05:~$ sudo apt install libvirt-clients -y

// 检查qemu环境配置
zhipu@zps05:~$ sudo virt-host-validate qemu
  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking if device /dev/kvm exists                                   : PASS
  QEMU: Checking if device /dev/kvm is accessible                            : PASS
  QEMU: Checking if device /dev/vhost-net exists                             : PASS
  QEMU: Checking if device /dev/net/tun exists                               : PASS
  QEMU: Checking for cgroup 'cpu' controller support                         : PASS
  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS
  QEMU: Checking for cgroup 'cpuset' controller support                      : PASS
  QEMU: Checking for cgroup 'memory' controller support                      : PASS
  QEMU: Checking for cgroup 'devices' controller support                     : PASS
  QEMU: Checking for cgroup 'blkio' controller support                       : PASS
  QEMU: Checking for device assignment IOMMU support                         : PASS
  QEMU: Checking if IOMMU is enabled by kernel                               : WARN (IOMMU appears to be disabled in kernel. Add intel_iommu=on to kernel cmdline arguments)
  QEMU: Checking for secure guest support                                    : WARN (Unknown if this platform has Secure Guest support)

// 安装KVM
zhipu@zps05:~$sudo apt install qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils virtinst virt-manager -y

在k8s中部署KubeVirt

// 部署kubevirt
kubectl apply -f kubevirt-operator.yaml
kubectl apply -f kubevirt-cr.yaml

sukai@SuKai:/mnt/g/zhipu/k3s/kubevirt$ kubectl -n kubevirt get pods
NAME                               READY   STATUS              RESTARTS   AGE
virt-operator-976969b94-thlg4      1/1     Running             0          13m
virt-operator-976969b94-rtxpp      1/1     Running             0          13m
virt-api-6bb566bbc7-tbx97          1/1     Running             0          12m
virt-handler-m7pjd                 0/1     ContainerCreating   0          12m
virt-api-6bb566bbc7-nx4qd          1/1     Running             0          12m
virt-controller-5d6975c644-t962z   1/1     Running             0          12m
virt-controller-5d6975c644-f2gtj   1/1     Running             0          12m

| 下载windows工具制作ISO文件

https://www.microsoft.com/en-us/software-download/windows10

| 安装部署CDI(containerized-data-importer)

containerized-data-importer是Kubernetes的管理插件，在PVC上为KubeVirt虚拟机创建磁盘。

sukai@SuKai:/mnt/g/zhipu/k3s/kubevirt$ kubectl apply -f cdi-operator.yaml
namespace/cdi created
customresourcedefinition.apiextensions.k8s.io/cdis.cdi.kubevirt.io created
clusterrole.rbac.authorization.k8s.io/cdi-operator-cluster created
clusterrolebinding.rbac.authorization.k8s.io/cdi-operator created
serviceaccount/cdi-operator created
role.rbac.authorization.k8s.io/cdi-operator created
rolebinding.rbac.authorization.k8s.io/cdi-operator created
deployment.apps/cdi-operator created
configmap/cdi-operator-leader-election-helper created
sukai@SuKai:/mnt/g/zhipu/k3s/kubevirt$ kubectl apply -f cdi-cr.yaml
cdi.cdi.kubevirt.io/cdi created
sukai@SuKai:/mnt/g/zhipu/k3s/kubevirt$ kubectl -n cdi get pods
NAME                              READY   STATUS    RESTARTS   AGE
cdi-operator-6b69998d8b-6w4mg     1/1     Running   0          8m6s
cdi-apiserver-6b6b89954c-4pjps    1/1     Running   0          4m
cdi-uploadproxy-8cf6df88d-46zpw   1/1     Running   0          3m53s
cdi-deployment-7cfd7cd7d6-79wr4   1/1     Running   0          3m55s

通过virtctl上传windows iso文件到PVC，创建一个新的cdi-uploadproxy服务类型为NodePort的服务，方便在k8s集群外上传Windows ISO文件。

sudo cp virtctl-v0.39.0-linux-amd64 /usr/local/bin/virtctl
sukai@SuKai:/mnt/g/zhipu/k3s/iso$ kubectl describe service cdi-uploadproxy -n cdi
Name:              cdi-uploadproxy
Namespace:         cdi
Labels:            cdi.kubevirt.io=cdi-uploadproxy
                   operator.cdi.kubevirt.io/createVersion=v1.33.0
                   operator.cdi.kubevirt.io/updateVersion=v1.33.0
Annotations:       operator.cdi.kubevirt.io/lastAppliedConfiguration:
                     {"kind":"Service","apiVersion":"v1","metadata":{"name":"cdi-uploadproxy","namespace":"cdi","creationTimestamp":null,"labels":{"cdi.kubevir...
Selector:          cdi.kubevirt.io=cdi-uploadproxy
Type:              ClusterIP
IP:                10.43.223.128
Port:              <unset>  443/TCP
TargetPort:        8443/TCP
Endpoints:         10.42.0.88:8443
Session Affinity:  None
Events:            <none>

sukai@SuKai:/mnt/g/zhipu/k3s/iso$ cat <<EOF | kubectl apply -f -
> apiVersion: v1
> kind: Service
> metadata:
>  name: cdi-uploadproxy-nodeport
>  namespace: cdi
>  labels:
>    cdi.kubevirt.io: "cdi-uploadproxy"
> spec:
>  type: NodePort
>  ports:
>    - port: 443
>      targetPort: 8443
>      nodePort: 31001
>      protocol: TCP
>  selector:
>    cdi.kubevirt.io: cdi-uploadproxy
> EOF
service/cdi-uploadproxy-nodeport created
sukai@SuKai:/mnt/g/zhipu/k3s/iso$
sukai@SuKai:/mnt/g/zhipu/k3s/iso$
sukai@SuKai:/mnt/g/zhipu/k3s/iso$
zhipu@zps05:/mnt/data_zhipu/iso$ virtctl image-upload   --image-path='Win10_20H2_v2_Chinese_x64.iso'   --pvc-name=iso-windows10   --pvc-size=7G   --uploadproxy-url=https://192.168.0.53:31001   --insecure
Using existing PVC default/iso-windows10
Uploading data to https://192.168.0.53:31001

 5.72 GiB / 5.72 GiB [===============================================================================================================] 100.00%
 5.72 GiB / 5.72 GiB [===========================================================================================================] 100.00% 22s

Uploading data completed successfully, waiting for processing to complete, you can hit ctrl-c without interrupting the progress
Processing completed successfully
Uploading Win10_20H2_v2_Chinese_x64.iso completed successfully

查看持久卷存储内容

sukai@SuKai:~$ kubectl describe pv pvc-3c596bc3-3031-42b3-8b91-f16f648475b1
Name:              pvc-3c596bc3-3031-42b3-8b91-f16f648475b1
Labels:            <none>
Annotations:       pv.kubernetes.io/provisioned-by: rancher.io/local-path
Finalizers:        [kubernetes.io/pv-protection]
StorageClass:      local-path
Status:            Bound
Claim:             default/iso-windows10
Reclaim Policy:    Delete
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          7G
Node Affinity:
  Required Terms:
    Term 0:        kubernetes.io/hostname in [zps05]
Message:
Source:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rancher/k3s/storage/pvc-3c596bc3-3031-42b3-8b91-f16f648475b1_default_iso-windows10
    HostPathType:  DirectoryOrCreate
Events:            <none>


zhipu@zps05:/mnt/data_zhipu/k8s-local-path-provisioner$ cd /var/lib/rancher/k3s/storage/pvc-3c596bc3-3031-42b3-8b91-f16f648475b1_default_iso-windows10
zhipu@zps05:/var/lib/rancher/k3s/storage/pvc-3c596bc3-3031-42b3-8b91-f16f648475b1_default_iso-windows10$ ls
disk.img
zhipu@zps05:/var/lib/rancher/k3s/storage/pvc-3c596bc3-3031-42b3-8b91-f16f648475b1_default_iso-windows10$ du -sh *
5.8G    disk.img

| 创建Windows虚拟机

sukai@SuKai:/mnt/g/zhipu/k3s/kubevirt$ kubectl apply -f virtualmachine.yaml
persistentvolumeclaim/winhd created
virtualmachine.kubevirt.io/win10 created

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: winhd
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
  storageClassName: local-path
---

---
apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
  name: win10
spec:
  running: false
  template:
    metadata:
      labels:
        kubevirt.io/domain: win10
    spec:
      domain:
        cpu:
          cores: 4
        devices:
          disks:
          - bootOrder: 1
            cdrom:
              bus: sata
            name: cdromiso
          - disk:
              bus: sata
            name: harddrive
          - cdrom:
              bus: sata
            name: virtiocontainerdisk
          interfaces:
          - masquerade: {}
            model: e1000
            name: default
        machine:
          type: q35
        resources:
          requests:
            memory: 8G
      networks:
      - name: default
        pod: {}
      volumes:
      - name: cdromiso
        persistentVolumeClaim:
          claimName: iso-windows10
      - name: harddrive
        persistentVolumeClaim:
          claimName: winhd
      - containerDisk:
          image: kubevirt/virtio-container-disk
        name: virtiocontainerdisk

| 启动Windows虚拟机

sukai@SuKai:/mnt/g/zhipu/k3s/kubevirt$ kubectl apply -f virtualmachine.yaml
persistentvolumeclaim/winhd unchanged
virtualmachine.kubevirt.io/win10 configured
sukai@SuKai:/mnt/g/zhipu/k3s/kubevirt$ virtctl start win10
VM win10 was scheduled to start
sukai@SuKai:/mnt/g/zhipu/k3s/kubevirt$ kubectl get vmi
NAME    AGE   PHASE        IP    NODENAME
win10   15s   Scheduling
sukai@SuKai:~$ kubectl get pods -o wide
NAME                        READY   STATUS    RESTARTS   AGE     IP            NODE    NOMINATED NODE   READINESS GATES
volume-test                 1/1     Running   10         27d     10.42.0.252   zps05   <none>           <none>
virt-launcher-win10-hmfdr   2/2     Running   0          4m46s   10.42.0.13    zps05   <none>           <none>
sukai@SuKai:~$ kubectl get vmi
NAME    AGE     PHASE     IP           NODENAME
win10   5m29s   Running   10.42.0.13   zps05
sukai@SuKai:/mnt/g/zhipu/k3s/iso$ kubectl get vm
NAME    AGE   VOLUME
win10   14h
sukai@SuKai:/mnt/g/zhipu/k3s/iso$ kubectl get vmi
NAME    AGE   PHASE     IP           NODENAME
win10   13h   Running   10.42.0.28   zps05

在k8s物理机上查看运行的VM，virt-launcher负责调度libvirt启动kvm虚拟机

root      571155  0.1  0.0 4241216 68384 ?       Sl   Apr17   1:16 /usr/bin/virt-launcher --qemu-timeout 5m --name win10 --uid 4ff25329-c524-4e41-8e9a-962fc4b249ee --namespace default --kubevirt-share-dir /var/run/kubevirt --ephemeral-disk-dir /var/run/kubevirt-ephemeral-disks --container-disk-dir /var/run/kubevirt/container-disks --grace-period-seconds 45 --hook-sidecars 0 --less-pvc-space-toleration 10 --ovmf-path /usr/share/OVMF --no-fork true
uuidd     571508 60.3  5.9 12963856 7890644 ?    Sl   Apr17 488:12 /usr/libexec/qemu-kvm -name guest=default_win10,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-default_win10/master-key.aes -machine pc-q35-rhel8.3.0,accel=kvm,usb=off,dump-guest-core=off -cpu Cascadelake-Server,ss=on,vmx=on,hypervisor=on,tsc-adjust=on,umip=on,pku=on,md-clear=on,stibp=on,arch-capabilities=on,xsaves=on,ibpb=on,amd-stibp=on,amd-ssbd=on,rdctl-no=on,ibrs-all=on,skip-l1dfl-vmentry=on,mds-no=on,pschange-mc-no=on,hle=off,rtm=off -m 7630 -overcommit mem-lock=off -smp 4,sockets=1,dies=1,cores=4,threads=1 -object iothread,id=iothread1 -uuid a26173eb-9e7e-509b-a27f-e69d1eb1fa81 -smbios type=1,manufacturer=KubeVirt,product=None,uuid=a26173eb-9e7e-509b-a27f-e69d1eb1fa81,family=KubeVirt -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=18,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 -device pcie-pci-bridge,id=pci.2,bus=pci.1,addr=0x0 -device pcie-root-port,port=0x11,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x1 -device pcie-root-port,port=0x12,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x2 -device pcie-root-port,port=0x13,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x3 -device pcie-root-port,port=0x14,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x4 -device virtio-scsi-pci-non-transitional,id=scsi0,bus=pci.3,addr=0x0 -device virtio-serial-pci-non-transitional,id=virtio-serial0,bus=pci.4,addr=0x0 -blockdev {"driver":"file","filename":"/var/run/kubevirt-private/vmi-disks/cdromiso/disk.img","node-name":"libvirt-6-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-6-format","read-only":true,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-6-storage"} -device ide-cd,bus=ide.0,drive=libvirt-6-format,id=ua-cdromiso,bootindex=1,write-cache=on,werror=stop,rerror=stop -blockdev {"driver":"file","filename":"/var/run/kubevirt-private/vmi-disks/hd-1/disk.img","node-name":"libvirt-5-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-5-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-5-storage"} -device ide-hd,bus=ide.1,drive=libvirt-5-format,id=ua-hd-1,write-cache=on,werror=stop,rerror=stop -blockdev {"driver":"file","filename":"/var/run/kubevirt-private/vmi-disks/hd-2/disk.img","node-name":"libvirt-4-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-4-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-4-storage"} -device ide-hd,bus=ide.2,drive=libvirt-4-format,id=ua-hd-2,write-cache=on,werror=stop,rerror=stop -blockdev {"driver":"file","filename":"/var/run/kubevirt-private/vmi-disks/hd-3/disk.img","node-name":"libvirt-3-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-3-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-3-storage"} -device ide-hd,bus=ide.3,drive=libvirt-3-format,id=ua-hd-3,write-cache=on,werror=stop,rerror=stop -blockdev {"driver":"file","filename":"/var/run/kubevirt/container-disks/disk_4.img","node-name":"libvirt-2-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-2-format","read-only":true,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"} -blockdev {"driver":"file","filename":"/var/run/kubevirt-ephemeral-disks/disk-data/virtiocontainerdisk/disk.qcow2","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-1-format","read-only":true,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-1-storage","backing":"libvirt-2-format"} -device ide-cd,bus=ide.4,drive=libvirt-1-format,id=ua-virtiocontainerdisk,write-cache=on,werror=stop,rerror=stop -netdev tap,fd=20,id=hostua-default -device e1000,netdev=hostua-default,id=ua-default,mac=52:54:00:ef:9c:e9,bus=pci.2,addr=0x1,romfile= -chardev socket,id=charserial0,fd=21,server,nowait -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,fd=22,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -vnc vnc=unix:/var/run/kubevirt-private/4ff25329-c524-4e41-8e9a-962fc4b249ee/virt-vnc -device VGA,id=video0,vgamem_mb=16,bus=pcie.0,addr=0x1 -device virtio-balloon-pci-non-transitional,id=balloon0,bus=pci.5,addr=0x0 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on

| VNC连接Windows桌面

下载VNC Viewer并安装，配置PATH环境变量

| 遇到的问题

failed to configure vmi network: Critical network error: could not retrieve pid 2132351 selinux label: operation not supported

sudo apt install policycoreutils selinux-utils selinux-basics

local-path-provisioner路径被rancher恢复配置

Warning  SyncFailed        13s (x4 over 49s)  virt-handler, zps05        (combined from similar events): server error. command SyncVMI failed: "preparing host-disks failed: unable to create /var/run/kubevirt-private/vmi-disks/harddrive/disk.img, not enough space, demanded size 107374182400 B is bigger than available space 83489140736 B, also after taking 10 % toleration into account"

Windows不支持安装的磁盘，将磁盘Bus修改为sata

---

后台回复“加群”，带你进入高手如云交流群

推荐阅读：

微软出品的最新K8S学习指南3.0下载

---

▼

喜欢，就给我一个“在看”

---

10T 技术资源大放送！包括但不限于：云计算、虚拟化、微服务、大数据、网络、Linux、Docker、Kubernetes、Python、Go、C/C++、Shell、PPT 等。在公众号内回复「1024」，即可免费获取！！