0x00 前言
最近因為工作需要要從零開始搭建k8s集群（在Openstack上），使用已有的Ceph集群以cephfs形式作為k8s的storageclass存儲，並建好使用let's encrypt作為https證書的wordpress服務。沒接觸過k8s的我踩了一周的坑終於弄好了。。。感動。
0x01 環境
均使用Ubuntu 22.04 lts
Ceph Pacific集群
使用cephadm搭建
k8s 1.25集群
使用kubeadm搭建
Openstack虛擬機上
Master Node三個
Worker Node兩個
0x02 k8s搭建
請參考https://skyao.io/learning-kubernetes/docs/installation/kubeadm/ubuntu.html，使用kubeadm搭建，基本無坑。除了
$ apt-get install -y kubelet kubeadm kubectl之後的步驟會報錯：
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
error execution phase addon/coredns: unable to fetch CoreDNS current installed version and ConfigMap.: rpc error: code = Unknown desc = malformed header: missing HTTP content-type需要使用特定版本：
$ apt-get install kubelet=1.23.5-00 kubeadm=1.23.5-00 kubectl=1.23.5-00還有初始化的時候：
$ sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.100.40 -v=9如果有多個master node的話需要加上 --control-plane-endpoint <vip> 和 --upload-certs，即
$ sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=<api-ip> --control-plane-endpoint <vip> --upload-certs -v=9其中
<api-ip> = 運行命令的master node的ip
<vip> = A/P HA高可用使用的虛擬ip地址，可使用keepalived或pcs創建
pod-network-cidr 不可更改0x03 使用ceph-csi挂載cephfs作為storageclass儲存池
請參考https://dylanyang.top/post/2021/05/15/k8s%E4%BD%BF%E7%94%A8ceph-csi%E6%8C%81%E4%B9%85%E5%8C%96%E5%AD%98%E5%82%A8cephfs/，基本無坑。注意userKey和adminKey是ceph auth get client.admin的輸出，不需要base64編碼。
0x04 搭建wordpress
1. 任一master node安裝helm
$ curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null
$ sudo apt-get install apt-transport-https --yes
$ echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
$ sudo apt-get update
$ sudo apt-get install helm2. 安裝 ingress controller (nginx)
$ helm upgrade --install ingress-nginx ingress-nginx repo https://kubernetes.github.io/ingress-nginx namespace ingress-nginx --create-namespace
#給 ingress controller 配置 ip 以供外部訪問
$ kubectl patch svc ingress-nginx-controller -n ingress-nginx -p '{"spec": {"type": "LoadBalancer", "externalIPs":["<ip>"]}}'<ip> = ip, e.g 192.168.1.153. 安裝 cert-manager
$ helm repo add jetstack https://charts.jetstack.io
"jetstack" has been added to your repositories
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "jetstack" chart repository
Update Complete. ⎈Happy Helming!⎈
$ helm install \
cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--version v1.7.1 \
--set installCRDs=true4. 配置secret
$ nano secret.yaml
secretGenerator:
- name: mysql-password
literals:
- password=<pw1>
- name: mysql-user
literals:
- username=<name1>
- name: mysql-user-password
literals:
- passworduser=<pw2>
- name: mysql-database
literals:
- database=<name2><pw1>, <pw2>, <name1> = 自訂
<name2> = 數據庫名稱，自訂$ kubectl apply -k .
secret/mysql-database-xxxx created
secret/mysql-password-xxxx created
secret/mysql-user-xxxx created
secret/mysql-user-password-xxxx created5. 創建mysql pvc
$ nano mysel-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-pvc
spec:
storageClassName: <storageClassName>
accessModes:
- ReadWriteMany
resources:
requests:
storage: <storage>
$ kubectl apply -f mysel-pvc.yaml<storageClassName> = storageClass 的名稱，e.g. csi-cephfs-sci
<storage> = 分配空間大小，e.g. '50Gi'6. 創建wordpress pvc
$ nano wp-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: wp-pvc
spec:
storageClassName: <storageClassName>
accessModes:
- ReadWriteOnce
resources:
requests:
storage: <storage>
$ kubectl apply -f wp-pvc.yaml<storageClassName> = storageClass 的名稱，e.g. csi-cephfs-sci
<storage> = 分配空間大小，e.g. '50Gi'7. 創建mysql service, deployment配置
$nano mysql-service.yaml
apiVersion: v1
kind: Service
metadata:
name: mysql-wp
spec:
ports:
- port: 3306
selector:
app: wordpress
tier: mysql
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: mysql-wp
spec:
selector:
matchLabels:
app: wordpress
tier: mysql
strategy:
type: Recreate
template:
metadata:
labels:
app: wordpress
tier: mysql
spec:
containers:
- image: mysql:latest
name: mysql
env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: <name1>
key: password
- name: MYSQL_USER
valueFrom:
secretKeyRef:
name: <name2>
key: username
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: <name3>
key: passworduser
- name: MYSQL_DATABASE
valueFrom:
secretKeyRef:
name: <name4>
key: database
ports:
- containerPort: 3306
name: mysql
volumeMounts:
- name: persistent-storage
mountPath: /var/lib/mysql
volumes:
- name: persistent-storage
persistentVolumeClaim:
claimName: mysql-pvc
$ kubectl apply -f mysql-service.yaml
<name*> = 步驟4裏創建的secret
<name1> = mysql-password-xxxx
<name2> = mysql-user-xxxx
<name3> = mysql-user-password-xxxx
<name4> = mysql-database-xxxx8. 創建wordpress service, deployment配置
$ nano wp-service.yaml
apiVersion: v1
kind: Service
metadata:
labels:
app: wordpress
name: wordpress
spec:
selector:
app: wordpress
type: ClusterIP
ports:
- protocol: TCP
port: 80
targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: wordpress
spec:
selector:
matchLabels:
app: wordpress
tier: web
strategy:
type: Recreate
template:
metadata:
labels:
app: wordpress
tier: web
spec:
containers:
- image: wordpress:php8.1-apache
name: wordpress
env:
- name: WORDPRESS_DB_HOST
value: mysql-wp:3306
- name: WORDPRESS_DB_PASSWORD
valueFrom:
secretKeyRef:
name: <name1>
key: passworduser
- name: WORDPRESS_DB_USER
valueFrom:
secretKeyRef:
name: <name2>
key: username
- name: WORDPRESS_DB_NAME
valueFrom:
secretKeyRef:
name: <name3>
key: database
ports:
- containerPort: 80
name: wordpress
volumeMounts:
- name: persistent-storage
mountPath: /var/www/html
volumes:
- name: persistent-storage
persistentVolumeClaim:
claimName: wp-pvc
$ kubectl apply -f wp-service.yaml<name*> = 步驟4裏創建的secret
<name1> = mysql-user-password-xxxx
<name2> = mysql-user-xxxx
<name3> = mysql-database-xxxx9. 解決ubuntu 上coredns 問題
因為coredns容器服務默認使用容器本機的/etc/resolve.conf 裏的nameserver的值作為對外dns查詢使用的nameserver, 但ubuntu上resolve.conf的nameserver值為127.0.0.53，指向本機。而/run/systemd/resolve/resolv.conf裏的nameserver的值才是真正在使用的dns服務器地址，所以需要手動更改文件路徑，否則下一步會因無法解析acme-v02.api.letsencrypt.org而報錯。
$ sudo nano /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
如果沒有KUBELET_EXTRA_ARGS
加上
‘Environment="KUBELET_EXTRA_ARGS=--resolv-conf=/run/systemd/resolve/resolv.conf"’
否則加上
‘--resolv-conf=/run/systemd/resolve/resolv.conf’
到EXTRA_ARGS的最後
$ systemctl daemon-reload
#刪除已有的coredns Pods，系統會自行創建新的， 'kubectl get pods -n kube-system' 查看
$ kubectl delete pods -n kube-system coredns coredns-64897985d-4j5bc
$ kubectl delete pods -n kube-system coredns coredns-64897985d-xkpkr10. 配置ClusterIssuer
$ nano clusterissuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: wp-prod-issuer
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: <email>
privateKeySecretRef:
name: letsencrypt-prod
solvers:
- http01:
ingress:
class: nginx
$ kubectl apply -f clusterissuer.yaml<email> = 你的email地址11. 創建ingress資源
$ nano wp-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: wordpress
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: "wp-prod-issuer"
spec:
rules:
- host: <domain-name>
http:
paths:
- path: "/"
pathType: Prefix
backend:
service:
name: wordpress
port:
number: 80
tls:
- hosts:
- <domain-name>
secretName: wordpress-tls
$ kubectl apply -f wp-ingress.yaml<domain-name> = 你的域名，e.g. 'example.com'12. 配置域名指向
把域名指向ingress controller的external ip
就可以用域名訪問wp了
0x05參考
https://skyao.io/learning-kubernetes/docs/installation/kubeadm/ubuntu.html
Comments | NOTHING