安装 kubetest

https://github.com/kubernetes/test-infra/tree/master/kubetest#installation

1

go install k8s.io/test-infra/kubetest

或者

1

GO111MODULE=on go install ./kubetest

构建二进制

1

kubetest --build

启动本地集群

1

./hack/local-up-cluster.sh

如果没有安装 etcd 需要先安装

1

./hack/install-etcd.sh

使用 kubectl 访问

安装 Docker

https://kubernetes.io/docs/setup/production-environment/container-runtimes/#docker

国内加速替换软件源

http://mirrors.ustc.edu.cn/

1
2
3
4
5
6

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key --keyring /etc/apt/trusted.gpg.d/docker.gpg add -

sudo add-apt-repository \
  "deb [arch=amd64] http://mirrors.ustc.edu.cn/docker-ce/linux/ubuntu/ \
  $(lsb_release -cs) \
  stable"

安装 kubeadm

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

Kubernetes

client-go leader election 源码分析

https://zhengyinyong.com/post/kubernetes-pod-leader-election/

https://mathspanda.github.io/2017/05/11/k8s-leader-election/

kubernetes networks

https://draveness.me/kubernetes-service

https://blog.scottlowe.org/2013/09/04/introducing-linux-network-namespaces/

https://www.cyberciti.biz/faq/linux-ip-command-examples-usage-syntax/

kubectl plugin

https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/

https://github.com/fatsheep9146/kubectl-pvc

Kubernetes

deletion timestamp

如果 Pod 的 DeletionGracePeriodSeconds 与 TerminationGracePeriodSeconds 同时存在时，哪个生效？

kubelet 代码里 DeletionGracePeriodSeconds 的优先级高。

Kubernetes

kubernetes dynamic client

typed client 接收固定类型的对象，只能对固定类型对象进行操作。dynamic client 我们只要告诉它 group，version，kind 信息，传入 unstructured object，便可操作“任意类型”的对象。

https://stackoverflow.com/questions/53341727/how-to-submit-generic-runtime-object-to-kubernetes-api-using-client-go

https://soggy.space/namespaced-crds-dynamic-client/

https://www.oreilly.com/library/view/programming-kubernetes/9781492047094/ch04.html

node topology manager

node topology manager 在绑核时可以感知设备拓扑（例如 NUMA Node）：

https://github.com/kubernetes/kubernetes/issues/49964

https://docs.google.com/document/d/1lSwVh2ZfJ2FeLXIeyyiNqN_hKPYpahJiwN5X5cszjOk/edit#

https://upcommons.upc.edu/bitstream/handle/2117/114851/Topology-Aware%20GPU%20Scheduling%20for%20Learning%20Workloads.pdf

https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/0035-20190130-topology-manager.md

Kubernetes

etcd 运维文档

https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/maintenance.md

etcd 压测

不断写入同一个 key，产生大量 revision。
可以通过 compact 和 defrag 解决。

1
2
3
4

while [ 1 ]; do
  dd if=/dev/urandom bs=1024 count=1024 | ETCDCTL_API=3 ./etcdctl --endpoints=https://10.0.2.15:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key put /test || break
done

写入不同的 key。
通过删除 key 解决。

1
2
3

for j in {1..10}; do
  dd if=/dev/urandom bs=1024 count=1024 | ETCDCTL_API=3 ./etcdctl --endpoints=https://10.0.2.15:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key put /test/key/${j} || break
done

Kubernetes

kubernetes 生产环境性能优化

https://caicloud.io/blog/57392eca8241681100000003

容器内信息注入

https://kubernetes.io/docs/concepts/storage/volumes/#using-subpath-with-expanded-environment-variables

https://kubernetes.io/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/#capabilities-of-the-downward-api

kubernetes scheduler

https://ggaaooppeenngg.github.io/zh-CN/2017/09/26/kubernetes-%E6%8C%87%E5%8C%97/

https://caicloud.io/blog/57392eca8241681100000003

https://coreos.com/blog/improving-kubernetes-scheduler-performance.html

Kubernetes

object GroupVersionKind is empty

问题

从 client-go （无论是从 server 还是 cache）获取到的 object 的 TypeMeta 为空

1
2
3

pod := podLister.Pods(namespace).Get(name)
gvk := pod.GetObjectKind().GroupVersionKind()
fmt.Printf("%#v\n", gvk)

1

GroupVersionKind{Group:"", Version:"", Kind:""}

(kubernetes 1.14)

Kubernetes

api conventions

https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md

kube node lease

kube node lease 用来改善 kubelet 定时更新节点状态对 etcd 造成的压力。

https://containers.goffinet.org/k8s/kubectlget.html#what-about-kube-node-lease

https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/0009-node-heartbeat.md

https://kubernetes.io/docs/concepts/architecture/nodes/#node-controller

https://github.com/kubernetes-sigs/kubespray/blob/master/docs/kubernetes-reliability.md

PLEG (pod lifecycle event generator) 是 kubelet 中一个非常重要的模块，它主要完成以下几个目标：

1. 从 runtime 中获取 pod 当前状态，产生 pod lifecycle events
2. 从 runtime 中获取 pod 当前状态，更新 kubelet pod cache

本文我们通过分析 PLEG 模块的源码，来加深对 Kubernetes 的理解，也可以加速在使用过程对一些疑难问题的排查和处理，同时后期可以对一些问题源码进行优化，来解决一些 Kubernetes 本身的坑。