在kubernetes cluster上部署NGINX服务流程及错误解决
Contents
一创建deployment
[root@master-node ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master-node Ready control-plane,master 18h v1.22.3
node-1 Ready <none> 17h v1.22.3
node-2 Ready <none> 17h v1.22.3
[root@master-node ~]# kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
[root@master-node ~]# kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
nginx 0/1 1 0 9s
[root@master-node ~]#
二创建service
[root@master-node ~]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 19h
[root@master-node ~]# kubectl create service nodeport nginx --tcp=80:80
service/nginx created
[root@master-node ~]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 19h
nginx NodePort 10.109.243.29 <none> 80:32201/TCP 8s
[root@master-node ~]#
三访问报错
[root@master-node ~]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3h50m
nginx NodePort 10.97.11.68 <none> 80:32610/TCP 5s
[root@master-node ~]# curl cluster-node:32610
curl: (6) Could not resolve host: cluster-node; 未知的错误
[root@master-node ~]#
四排查pods信息
kubectl get pods;
kubectl describe pods nginx;
[root@master-node ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-6799fc88d8-qmwcv 0/1 ContainerCreating 0 3h45m
[root@master-node ~]#
[root@master-node ~]# kubectl describe pods nginx
Name: nginx-6799fc88d8-qmwcv
Namespace: default
Priority: 0
Node: node-2/172.16.11.161
Start Time: Thu, 04 Nov 2021 11:44:46 +0800
Labels: app=nginx
pod-template-hash=6799fc88d8
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/nginx-6799fc88d8
Containers:
nginx:
Container ID:
Image: nginx
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2p27l (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-api-access-2p27l:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreatePodSandBox 11m (x12195 over 3h45m) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "f7f19384ea91a37dee5d7314b9899c456685ab8b8c29a7075926c779e60bbd66" network for pod "nginx-6799fc88d8-qmwcv": networkPlugin cni failed to set up pod "nginx-6799fc88d8-qmwcv_default" network: open /run/flannel/subnet.env: no such file or directory
Normal SandboxChanged 6m4s (x12477 over 3h46m) kubelet Pod sandbox changed, it will be killed and re-created.
[root@master-node ~]#
Warning FailedCreatePodSandBox 11m (x12195 over 3h45m) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “f7f19384ea91a37dee5d7314b9899c456685ab8b8c29a7075926c779e60bbd66” network for pod “nginx-6799fc88d8-qmwcv”: networkPlugin cni failed to set up pod “nginx-6799fc88d8-qmwcv_default” network: open /run/flannel/subnet.env: no such file or directory Normal SandboxChanged 6m4s (x12477 over 3h46m) kubelet Pod sandbox changed, it will be killed and re-created.
分析原因，网络哪儿配置错误，导致的。具体本质原因，我还不是很清楚，根据网络上找的资料，重新在2个worker节点，执行：kubeadm reset；重置节点，最后再在master节点上执行该命令。
接下来，通过下述命令：重新初始化master cluster，并把2个worker节点加入：
[root@master-node ~]# kubeadm init --image-repository registry.aliyuncs.com/google_containers --pod-network-cidr=10.244.0.0/16
[init] Using Kubernetes version: v1.22.3
[preflight] Running pre-flight checks
....
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 172.16.11.106:6443 --token lq88wp.ynfo1ulqxelz5r0b \
--discovery-token-ca-cert-hash sha256:7cfb1263ac6e73bf790a300e314b37a5ac7ef006ff7d799ff44e566b3e0d3db8
[root@master-node ~]#
依然报同样的错误！！！
五解决错误
于是，重新再次kubeadm reset；先在2个worker节点执行，以及master节点执行。
并再次，重新初始化master node:
[root@master-node ~]# kubeadm init --image-repository registry.aliyuncs.com/google_containers --pod-network-cidr=10.244.0.0/16
..
...
然后，在worker节点上，执行join操作。
看到节点正常：
[root@master-node ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master-node Ready control-plane,master 76s v1.22.3
node-1 Ready <none> 28s v1.22.3
node-2 Ready <none> 26s v1.22.3
[root@master-node ~]#
但是，没有/run/flannel/subnet.env文件,通过重新执行：kubectl apply -f flannel.yml ，然后该命令会自动在master和所有的worker节点上，创建并生成/run/flannel/subnet.env文件。该文件的创建是自动的，但是需要稍微经过一点儿时间，才可以从文件系统上看到该文件。
[root@master-node ~]# ll /run/flannel/
总用量 0
[root@master-node ~]# kubectl apply -f flannel.yml
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created
[root@master-node ~]# ll /run/flannel/
总用量 0
[root@master-node ~]# ll /run/flannel/
总用量 4
-rw-r--r-- 1 root root 96 11月 4 15:52 subnet.env
[root@master-node ~]#
六重新创建deployment和service
创建deployment，并查看其状态：
[root@master-node ~]# kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
[root@master-node ~]# kubectl get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
nginx 0/1 1 0 10s
[root@master-node ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-6799fc88d8-dg6d9 1/1 Running 0 32s
[root@master-node ~]# kubectl get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
nginx 1/1 1 1 45s
[root@master-node ~]#
可以看到，创建完deployment之后，并不能马上看到其是ready的状态，稍后几秒钟，才能看到其状态变为ready。
创建service，及查看其状态：
[root@master-node ~]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 9m15s
[root@master-node ~]# kubectl create service nodeport nginx --tcp=80:80
service/nginx created
[root@master-node ~]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 9m53s
nginx NodePort 10.108.50.59 <none> 80:30769/TCP 5s
[root@master-node ~]#
七访问服务：
从上，可以看到对应的NGINX服务运行在30769端口上了。
[root@master-node ~]# curl master-node:30769
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@master-node ~]#
同时：
curl master-node:30769
curl node-1:30769
curl node-2:30769
都可以正常访问。再次describe pods的信息，不再报错：
[root@master-node ~]# kubectl describe pods nginx
Name: nginx-6799fc88d8-dg6d9
Namespace: default
Priority: 0
Node: node-2/172.16.11.161
Start Time: Thu, 04 Nov 2021 15:56:49 +0800
Labels: app=nginx
pod-template-hash=6799fc88d8
Annotations: <none>
Status: Running
IP: 10.244.2.2
IPs:
IP: 10.244.2.2
Controlled By: ReplicaSet/nginx-6799fc88d8
Containers:
nginx:
Container ID: docker://0693af01c2b7246e25394f458b12a530f93543b42d22cf4d5984f00fe1661f71
Image: nginx
Image ID: docker-pullable://docker.io/nginx@sha256:644a70516a26004c97d0d85c7fe1d0c3a67ea8ab7ddf4aff193d9f301670cf36
Port: <none>
Host Port: <none>
State: Running
Started: Thu, 04 Nov 2021 15:57:10 +0800
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8hk5t (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kube-api-access-8hk5t:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 20m default-scheduler Successfully assigned default/nginx-6799fc88d8-dg6d9 to node-2
Normal Pulling 20m kubelet Pulling image "nginx"
Normal Pulled 20m kubelet Successfully pulled image "nginx" in 19.846688268s
Normal Created 20m kubelet Created container nginx
Normal Started 20m kubelet Started container nginx
[root@master-node ~]#
当然，通过browser，也都可以正常访问：
http://172.16.11.106:30769/ http://172.16.11.148:30769/ http://172.16.11.161:30769/
八 小结
- 上一章节里安装的kubernetes cluster是存在问题的，因为创建的服务不能正常使用；
- 解决办法是重新初始化并创建kubernetes cluster并带上–pod-network-cidr=10.244.0.0/16选项；
- 然后重新配置网络组件kubectl apply -f flannel.yml；该命令在master节点执行，并且会自动在所有节点创建/run/flannel/subnet.env文件；
- 查看pods和deployment是否可以正常使用，可以通过kubectl get pods|deployments，查看READY列状态数字是不是正常，如果是0/x，则肯定是用不了的；
kubernetes把服务暴露出来的几种方式：
Kubernetes offers several options when exposing your service based on a feature called Kubernetes Service-types and they are:
- ClusterIP – This Service-type generally exposes the service on an internal IP, reachable only within the cluster, and possibly only within the cluster-nodes.
- NodePort – This is the most basic option of exposing your service to be accessible outside of your cluster, on a specific port (called the NodePort) on every node in the cluster. We will illustrate this option shortly.
- LoadBalancer – This option leverages on external Load-Balancing services offered by various providers to allow access to your service. This is a more reliable option when thinking about high availability for your service, and has more feature beyond default access.
- ExternalName – This service does traffic redirect to services outside of the cluster. As such the service is thus mapped to a DNS name that could be hosted out of your cluster. It is important to note that this does not use proxying.
The default Service-type is ClusterIP.
九 参考链接
How to Deploy Nginx on a Kubernetes Cluster
Pods are not starting. NetworkPlugin cni failed to set up pod
一条评论
Pingback: