> For the complete documentation index, see [llms.txt](https://marswang.gitbook.io/blog/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://marswang.gitbook.io/blog/k8s/k8s-shi-jian-cilium-ji-qun-ipam-shi-xian.md).
# k8s实践：Cilium集群IPAM实现
## 背景
通过前面遇到的[**Cilium集群下线节点导致某个在线节点容器网络不通问题**](/blog/k8s/k8s-shi-jian-cilium-ji-qun-xia-xian-jie-dian-dao-zhi-qi-ta-jie-dian-rong-qi-wang-luo-bu-tong.md)排查结果，可以确定集群中因为k8s Node资源spec.podCIDRs与CiliumNode资源的spec.ipam.podCIDRs字段不一致，而节点上Pod网段实际以CiliumNode资源的spec.ipam.podCIDRs为准，但是Cilium上报网络设备的BGP路由信息以k8s Node资源spec.podCIDRs为准，这就导致了网络设备上收到的路由信息与节点实际的Pod网段不一致，当下线一个节点时可能会影响另一个节点Pod网络连通性。环境信息如下所示：
```shell
# kubectl get node -o custom-columns=NAME:.metadata.name,PodCIDR:.spec.podCIDRs
NAME PodCIDR
master1 [10.5.112.0/24]
master2 [10.5.114.0/24]
master3 [10.5.113.0/24]
node01 [10.5.115.0/24]
# kubectl get ciliumnode -o custom-columns=NAME:.metadata.name,PodCIDR:.spec.ipam.podCIDRs
NAME PodCIDR
master1 [10.5.113.0/24]
master2 [10.5.115.0/24]
master3 [10.5.114.0/24]
node01 [10.5.112.0/24]
```
但是这个问题难道就是个Cilium项目的Bug么？还是我们配置不恰当导致的？带着问题我们再深入分析一波
## 问题分析
### 环境信息
当前集群的Cilium配置（这里省略掉一些无关的配置）
```shell
# kubectl -n kube-system get cm cilium-config -oyaml
apiVersion: v1
data:
bgp-announce-lb-ip: "true"
bgp-announce-pod-cidr: "true"
cluster-name: default
cluster-pool-ipv4-cidr: 10.5.112.0/21
cluster-pool-ipv4-mask-size: "24"
enable-ipv4: "true"
enable-ipv4-masquerade: "false"
identity-allocation-mode: crd
install-iptables-rules: "true"
install-no-conntrack-iptables-rules: "false"
ipam: cluster-pool
kube-proxy-replacement: strict
...
kind: ConfigMap
metadata:
name: cilium-config
namespace: kube-system
```
这里配置的IPAM模式为`cluster-pool`。
### Cilium IPAM Mode
查看[Cilium文档](https://docs.cilium.io/en/v1.11/concepts/networking/ipam/)我们知道常见IPAM Mode支持以下类型：Cluster Scope (Default)、Kubernetes Host Scope。
#### Kubernetes Host Scope
开启Kubernetes host-scope IPAM模式需要设置`ipam: kubernetes`，这种模式下Cilium直接使用k8s Node的podCIDRs作为当前节点Pod网段并用于分配Pod IP。
**架构**
> 注意：该模式下需要依赖`kube-controller-manager`组件开启`--allocate-node-cidrs`参数来自动为每一个节点分配podCIDRs
#### Cluster Scope (Default)
cluster-scope IPAM模式原理类似于Kubernetes Host Scope模式，它也是每个节点分配一个独立的PodCIDRs。不同之处是它通过Cilium Operator来管理每个节点的PodCIDRs，这些信息维护在CRD `Cilium`资源里面。这个模式的优点是不依赖于k8s为每个节点分配的PodCIDRs。
**架构**
### BGP功能源码分析
#### Cilium 1.11.9
这里我们发现Cilium bgpSpeaker宣告的podCIDRs只支持从k8s Node的podCIDRs获取，更详细的源码分析请参考[源码追踪](/blog/k8s/k8s-shi-jian-cilium-ji-qun-xia-xian-jie-dian-dao-zhi-qi-ta-jie-dian-rong-qi-wang-luo-bu-tong.md#yuan-ma-zhui-zong)
```go
// pkg/bgp/speaker/speaker.go
// OnAddNode notifies the Speaker of a new node.
func (s *MetalLBSpeaker) OnAddNode(node *v1.Node, swg *lock.StoppableWaitGroup) error {
...
// 收到Node Add事件时会构造一个nodeEvent对象添加到bgpSpeaker的队列中，注意这里podCIDRs是从k8s node资源的Spec.PodCIDRs字段获取的
s.queue.Add(nodeEvent{
Meta: meta,
op: Add,
labels: nodeLabels(node.Labels),
podCIDRs: podCIDRs(node),
})
return nil
}
func podCIDRs(node *v1.Node) *[]string {
...
podCIDRs := make([]string, 0, len(node.Spec.PodCIDRs))
if pc := node.Spec.PodCIDR; pc != "" {
if len(node.Spec.PodCIDRs) > 0 && pc != node.Spec.PodCIDRs[0] {
podCIDRs = append(podCIDRs, pc)
}
}
podCIDRs = append(podCIDRs, node.Spec.PodCIDRs...)
return &podCIDRs
}
```
#### Cilium 1.14.2
这里发现Cilium bgpSpeaker宣告的podCIDRs获取方式支持k8s Node和CiliumNode两种方式：
```go
// newDaemon creates and returns a new Daemon with the parameters set in c.
func newDaemon(ctx context.Context, cleaner *daemonCleanup, params *daemonParams) (*Daemon, *endpointRestoreState, error) {
...
// 这里支持根据IPAM模式从不同资源获取podCIDRs
d.k8sWatcher.RegisterNodeSubscriber(d.endpointManager)
if option.Config.BGPAnnounceLBIP || option.Config.BGPAnnouncePodCIDR {
switch option.Config.IPAMMode() {
case ipamOption.IPAMKubernetes:
d.k8sWatcher.RegisterNodeSubscriber(d.bgpSpeaker)
case ipamOption.IPAMClusterPool:
d.k8sWatcher.RegisterCiliumNodeSubscriber(d.bgpSpeaker)
}
}
...
}
```
收到k8s Node事件以后两种不IPAM模式下的处理方法实现，这里只以Add事件为例
```go
// OnAddNode notifies the Speaker of a new node.
func (s *MetalLBSpeaker) OnAddNode(node *slim_corev1.Node, swg *lock.StoppableWaitGroup) error {
return s.notifyNodeEvent(Add, node, nodePodCIDRs(node), false)
}
// OnAddCiliumNode notifies the Speaker of a new CiliumNode.
func (s *MetalLBSpeaker) OnAddCiliumNode(node *ciliumv2.CiliumNode, swg *lock.StoppableWaitGroup) error {
return s.notifyNodeEvent(Add, node, ciliumNodePodCIDRs(node), false)
}
// notifyNodeEvent notifies the speaker of a node (K8s Node or CiliumNode) event
func (s *MetalLBSpeaker) notifyNodeEvent(op Op, nodeMeta metaGetter, podCIDRs *[]string, withDraw bool) error {
...
l.Debug("adding event to queue")
s.queue.Add(nodeEvent{
Meta: meta,
op: op,
labels: nodeLabels(nodeMeta.GetLabels()),
podCIDRs: podCIDRs,
withDraw: withDraw,
})
return nil
}
```
两种获取podCIDRs的方式实现
```go
//从k8s Node获取PodCIDRs
func nodePodCIDRs(node *slim_corev1.Node) *[]string {
if node == nil {
return nil
}
podCIDRs := make([]string, 0, len(node.Spec.PodCIDRs))
// this bit of code extracts the pod cidr block the node will
// use per: https://github.com//cilium/cilium/blob/8cb6ca42179a0e325131a4c95b14291799d22e5c/vendor/k8s.io/api/core/v1/types.go#L4600
// read the above comments to understand this access pattern.
if pc := node.Spec.PodCIDR; pc != "" {
if len(node.Spec.PodCIDRs) > 0 && pc != node.Spec.PodCIDRs[0] {
podCIDRs = append(podCIDRs, pc)
}
}
podCIDRs = append(podCIDRs, node.Spec.PodCIDRs...)
return &podCIDRs
}
//从ciliumNode获取PodCIDRs
func ciliumNodePodCIDRs(node *ciliumv2.CiliumNode) *[]string {
if node == nil {
return nil
}
podCIDRs := make([]string, 0, len(node.Spec.IPAM.PodCIDRs))
podCIDRs = append(podCIDRs, node.Spec.IPAM.PodCIDRs...)
return &podCIDRs
}
```
查看Cilium bgpSpeaker宣告的podCIDRs支持k8s Node和CiliumNode两种方式的[commit链接](https://github.com/cilium/cilium/commit/3f8e54ee315573509aa095d335b9908cfb4d9140)，可以看到1.12以上所有版本都已经包含了该commit。
## 结论
通过以上分析我们有两个结论
* Cilium 1.12以下版本bgpSpeaker只支持从k8s Node中获取podCIDRs，所以这些版本需要开启BGP路由宣告的话，IPAM模式只能使用`Kubernetes Host Scope`。
* Cilium 1.12以上的版本bgpSpeaker支持从k8s Node和CiliumNode中获取podCIDRs，所以这些版本需要开启BGP路由宣告的话，IPAM模式`Cluster Scope (Default)`和`Kubernetes Host Scope`都可以选择。