Evan：/* .Install using the repository on debian */

2020-03-03T10:03:06Z

.Install using the repository on debian

新页面

=pre=
搞个离线的吧可以先学习着

[[K8s镜像]]

=info=
<pre>
这次是18.04 master 58; n1 59; n2 60 #Mon May 27 07:44:35 UTC 2019

每台机器最少2GB内存，2CPUs。
集群中所有机器之间网络连接正常。
打开相应的端口，详见： [ Check required ports https://kubernetes.io/docs/setup/independent/install-kubeadm/#check-required-ports]

Kubernetes要求集群中所有机器具有不同的Mac地址、产品uuid、Hostname。可以使用如下命令查看：

# UUID
cat /sys/class/dmi/id/product_uuid

# Mac地址
ip link

# Hostname
cat /etc/hostname

ubuntu 16.04

master 67 allon vbox
node1 66 node2 65
</pre>
https://mirrors.163.com/ubuntu-releases/16.04.6/

[[使用kubeadm离线部署kubernetesv1.9.0]]

=Set Hostname and update hosts file=
<pre>
sudo hostnamectl set-hostname "k8s-master"
sudo hostnamectl set-hostname k8s-node1
sudo hostnamectl set-hostname k8s-node2

#Add the following lines in /etc/hosts file on all three systems,

192.168.88.30 k8s-master
192.168.88.31 k8s-node1
192.168.88.32 k8s-node2

192.168.88.58 k8s-master #k8sumaster1
192.168.88.59 k8s-node1 #k8sun1
192.168.88.60 k8s-node2 #k8sun2

</pre>
[[Ubuntu配置网络和hostname]]

=翻墙=
[[Ubuntu利用shadowsocks和polipo终端翻墙]]
<pre>

cat /etc/profile #最好就是这个写成 polipo那台机器的IP就行了，其它机器就不用再搞 ss polipo了
#这个如何自启动加载呢不然 notready
export http_proxy="http://127.0.0.1:8123/"
export https_proxy=$http_proxy
#export no_proxy="localhost,127.0.0.1,192.168.88.58,10.96.0.0,10.224.0.0"
export no_proxy="localhost,127.0.0.1,192.168.88.58,10.96.0.0,10.224.0.0,10.224.*"
</pre>

如果不想翻墙请参考[https://www.cnblogs.com/RainingNight/p/using-kubeadm-to-create-a-cluster-1-12.html 使用Kubeadm搭建Kubernetes(1.12.2)集群]

=ins docker=
apt-get install docker.io -y #only 4 ubuntu ;docker-compose 直接用官方的二进制包

==debian9 or 10==

===.Install using the repository on debian ===
<pre>

apt install software-properties-common

apt-get remove docker docker-engine docker.io containerd runc

sudo apt-get install \
apt-transport-https \
ca-certificates \
curl \
gnupg2 \
software-properties-common -y

curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -

sudo apt-key fingerprint 0EBFCD88
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian \
$(lsb_release -cs) \
stable"
apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io
</pre>
https://docs.docker.com/install/linux/docker-ce/debian/

===2.install-from-a-package on debian===

Go to https://download.docker.com/linux/debian/dists/, choose your Debian version, browse to pool/stable/, choose either amd64 or armhf, and download the .deb file for the Docker CE version you want to install.

I am stretch so
apt install libltdl7

http://mirrors.aliyun.com/docker-ce/linux/debian/dists/stretch/pool/stable/amd64/

[[Docker入门]]

=docker代理设置=
<pre>

#不要少了开头的service 还要记得check一个代理成功不
mkdir -p /etc/systemd/system/docker.service.d
vi /etc/systemd/system/docker.service.d/http-proxy.conf

[Service]
Environment="HTTPS_PROXY=http://127.0.0.1:8123/" "HTTP_PROXY=http://127.0.0.1:8123/" "NO_PROXY=localhost,127.0.0.1,192.168.88.67,10.96.0.0,10.224.0.0"

#Environment="HTTP_PROXY=http://proxy.example.com:80/" "HTTPS_PROXY=http://proxy.example.com:80/""NO_PROXY=localhost,127.0.0.1,docker-registry.somecorporation.com"

systemctl daemon-reload
systemctl restart docker
systemctl enable docker

systemctl show --property=Environment docker

other
evan@k8s-master:~$ sudo systemctl enable docker
Synchronizing state of docker.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable docker

</pre>
[https://docs.docker.com/config/daemon/systemd/ docker http-proxy]

[https://www.jianshu.com/p/1cb70b8ea2d7 docker 代理设置]

[https://blog.frognew.com/2017/01/docker-http-proxy.html docker代理配置-透过代理服务器pull镜像]

[http://silenceper.com/blog/201809/over-the-wall-pull-docker-mirror/ docker pull 翻墙下载镜像]

[https://blog.csdn.net/northeastsqure/article/details/60143144 docker设置代理]

[https://www.cnblogs.com/atuotuo/p/7298673.html docker - 设置HTTP/HTTPS 代理]

=ins 在所有节点上=
<pre>

swapoff -a; sudo usermod -a -G docker $USER

apt-get update && apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl

#init 之前不要启动
#systemctl start kubelet&& systemctl enable kubelet.service

启动不了
原来是kubelet 的cgroup dirver 与 docker的不一样。docker默认使用cgroupfs,keubelet 默认使用systemd。

https://kubernetes.io/docs/setup/cri/
#这个有改的 18.04上成功了的
vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
[Service]
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"

systemctl daemon-reload && systemctl restart kubelet && systemctl enable kubelet.service
</pre>

=在 Master 节点上配置 kubelet 所需的 cgroup 驱动=
<pre>
使用 Docker 时，kubeadm 会自动为其检测 cgroup 驱动在运行时对 /var/lib/kubelet/kubeadm-flags.env 文件进行配置。
如果您使用了不同的 CRI，您得把 /etc/default/kubelet 文件中的 cgroup-driver 位置改为对应的值，像这样：

KUBELET_EXTRA_ARGS=--cgroup-driver=<value>

这个文件将会被 kubeadm init 和 kubeadm join 用于为 kubelet 获取额外的用户参数。

请注意，您只需要在您的 cgroup driver 不是 cgroupfs 时这么做，因为 cgroupfs 已经是 kubelet 的默认值了。

systemctl daemon-reload; systemctl restart kubelet #需要重启 kubelet：

#me
evan@k8s-master:~$ cat /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS=--cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1 --resolv-conf=/run/systemd/resolve/resolv.conf

</pre>

=初始化master=
<pre> #14:25:52--14:47:55 kubelet 其实是没启动的在init之前
kubeadm init --apiserver-advertise-address=192.168.88.30 --pod-network-cidr=10.224.0.0/16 # --apiserver-advertise-address=masterip

kubeadm join 192.168.88.58:6443 --token fuwhe0.ro0c8u82u4xtmn8q \
--discovery-token-ca-cert-hash sha256:83bd9c19486c44fde674f4ccf0a7382848cd7bfeff8c361d54e7a2955a4dbd60

Alternatively, if you are the root user, you can run:

export KUBECONFIG=/etc/kubernetes/admin.conf

另外有一个小技巧，在init的过程中，另开一个终端，运行

journalctl -f -u kubelet.service

可以查看具体是什么愿意卡住了

</pre>

=配置kubectl认证信息=
<pre>
cat /etc/sudoers.d/evan
echo 'evan ALL=(ALL) NOPASSWD:NOPASSWD:ALL' > /etc/sudoers.d/evan

su - evan
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
echo "source <(kubectl completion bash)" >> ~/.bashrc
exit

# 对于root用户这省不能少不然 # kubectl apply -f kube-flannel.yml The connection to the server localhost:8080 was refused - did you specify the right host or port?

export KUBECONFIG=/etc/kubernetes/admin.conf
#也可以直接放到~/.bash_profile
echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile</pre>

=安装pod网络on master=
#普通用户不要翻墙
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

=添加节点=
不要翻墙了新起个窗口
<pre>
kubeadm join 192.168.88.58:6443 --token fuwhe0.ro0c8u82u4xtmn8q \
--discovery-token-ca-cert-hash sha256:83bd9c19486c44fde674f4ccf0a7382848cd7bfeff8c361d54e7a2955a4dbd60

evan@k8s-master:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s NotReady master 5h12m v1.14.2
u16 NotReady <none> 106m v1.14.2

evan@k8s-master:~$ kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-fb8b8dccf-nprqq 0/1 Terminating 16 5h11m
kube-system coredns-fb8b8dccf-qn85f 0/1 Pending 0 5m4s
kube-system coredns-fb8b8dccf-sgtw4 0/1 Terminating 16 5h11m
kube-system coredns-fb8b8dccf-wsnkg 0/1 Pending 0 5m5s
kube-system etcd-k8s 1/1 Running 0 5h11m
kube-system kube-apiserver-k8s 1/1 Running 0 5h11m
kube-system kube-controller-manager-k8s 1/1 Running 0 5h11m
kube-system kube-flannel-ds-amd64-8vvn6 0/1 Init:0/1 0 107m
kube-system kube-flannel-ds-amd64-q92vz 1/1 Running 0 112m
kube-system kube-proxy-85vkt 0/1 ContainerCreating 0 107m
kube-system kube-proxy-fr7lv 1/1 Running 0 5h11m
kube-system kube-scheduler-k8s 1/1 Running 0 5h11m

evan@k8s-master:~$ kubectl describe pod kube-proxy-85vkt --namespace=kube-system
Name: kube-proxy-85vkt
Namespace: kube-system
Priority: 2000001000
PriorityClassName: system-node-critical
Node: u16/192.168.88.66
****

Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 109m default-scheduler Successfully assigned kube-system/kube-proxy-85vkt to u16
Normal Pulling 108m kubelet, u16 Pulling image "k8s.gcr.io/kube-proxy:v1.14.2"
Normal Pulled 107m kubelet, u16 Successfully pulled image "k8s.gcr.io/kube-proxy:v1.14.2"
Normal Created 107m kubelet, u16 Created container kube-proxy
Normal Started 107m kubelet, u16 Started container kube-proxy
Warning FailedCreatePodSandBox 52m (x119 over 107m) kubelet, u16 Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

放了一个晚上早上还是坏的突然打开已是好的了

evan@ubuntu18:~$ kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-fb8b8dccf-2rbwc 1/1 Running 3 18h
kube-system coredns-fb8b8dccf-67zc2 1/1 Running 3 18h
kube-system etcd-ubuntu18 1/1 Running 10 18h
kube-system kube-apiserver-ubuntu18 1/1 Running 4 18h
kube-system kube-controller-manager-ubuntu18 1/1 Running 5 18h
kube-system kube-flannel-ds-amd64-b6bn8 1/1 Running 45 16h
kube-system kube-flannel-ds-amd64-v9wxm 1/1 Running 46 16h
kube-system kube-flannel-ds-amd64-zn4xd 1/1 Running 3 16h
kube-system kube-proxy-d7pmb 1/1 Running 4 18h
kube-system kube-proxy-gcddr 1/1 Running 0 16h
kube-system kube-proxy-lv8cb 1/1 Running 0 16h
kube-system kube-scheduler-ubuntu18 1/1 Running 5 18h

master 也当作node 这里的master hostname 为 ubuntu18OB
evan@ubuntu18:~$ kubectl taint node ubuntu18 node-role.kubernetes.io/master-
node/ubuntu18 untainted

#master only
kubectl taint node ubuntu18 node-role.kubernetes.io/master="":NoSchedule

</pre>

=master 也当作node =
<pre>
[root@master tomcat]# hostname
master
[root@master tomcat]# kubectl taint node master node-role.kubernetes.io/master-
node/master untainted </pre>

=下面的是不是可以不要翻墙了呢=

=chpater4 k8s architecture=
<pre>
#唯一不是容器形式运行的k8s 组件
evan@k8s-master:~$ sudo systemctl status kubelet.service
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Mon 2019-05-27 07:26:18 UTC; 21min ago
Docs: https://kubernetes.io/docs/home/
Main PID: 817 (kubelet)
Tasks: 19 (limit: 3499)
CGroup: /system.slice/kubelet.service
└─817 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf -

在master节点上发起个创建应用请求
这里我们创建个名为httpd-app的应用，镜像为httpd，有两个副本pod
evan@k8s-master:~$ kubectl run httpd-app --image=httpd --replicas=2
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
deployment.apps/httpd-app created

evan@k8s-master:~$ kubectl get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
httpd-app 0/2 2 0 103s

evan@k8s-master:~$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
httpd-app-6df58645c6-bvg9w 0/1 ContainerCreating 0 2m10s <none> k8s-node1 <none> <none>
httpd-app-6df58645c6-n9xdj 0/1 ContainerCreating 0 2m10s <none> k8s-node2 <none> <none>

evan@k8s-master:~$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
httpd-app-6df58645c6-bvg9w 0/1 ContainerCreating 0 3m58s <none> k8s-node1 <none> <none>
httpd-app-6df58645c6-n9xdj 1/1 Running 0 3m58s 10.224.1.2 k8s-node2 <none> <none>
#OK了
evan@k8s-master:~$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
httpd-app-6df58645c6-bvg9w 1/1 Running 0 6m8s 10.224.2.3 k8s-node1 <none> <none>
httpd-app-6df58645c6-n9xdj 1/1 Running 0 6m8s 10.224.1.2 k8s-node2 <none> <none>

</pre>

=下面关闭ss docker 代理 polipo =

=chapter 5 run apps=
<pre>
evan@k8s-master:~$ kubectl run nginx-deployment --image=nginx:1.7.9 --replicas=2
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
deployment.apps/nginx-deployment created

上面的命令将部署包含两个副本的 Deployment nginx-deployment，容器的 image 为 nginx:1.7.9。

等待一段时间
kubectl get deployment nginx-deployment
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-deployment 2/2 2 2 36m

接下来我们用 kubectl describe deployment 了解更详细的信息

</pre>

=等待=
<pre>
sudo sslocal -c /root/shadowsocks.json -d start
sslocal -c shadowsocks.json -d start
sslocal -c shadowsocks.json -d start

</pre>
=进阶=

[https://blog.csdn.net/shida_csdn/article/details/83176735 K8S 源码探秘之 kubeadm init 执行流程分析]

[https://blog.csdn.net/m0_37556444/article/details/86494791 kubeadm--init]

[https://www.jianshu.com/p/c01ba5bd1359?utm_campaign=maleskine&utm_content=note&utm_medium=seo_notes&utm_source=recommendation 安装k8s Master高可用集群]

=What is new=
在Kubernetes 1.11中，CoreDNS已经实现了基于DNS的服务发现的GA，可作为kube-dns插件的替代品。这意味着CoreDNS将作为各种安装工具未来发布版本中的一个选项来提供。
事实上，kubeadm团队选择将其作为Kubernetes 1.11的默认选项。

[https://blog.csdn.net/k8scaptain/article/details/81033095 CoreDNS正式GA | kube-dns与CoreDNS有何差异？]

[https://juejin.im/post/5b46100de51d4519105d37e3 k8s集群配置使用coredns代替kube-dns]

=trouble=

==Kubenetes服务不启动问题 ==
<pre>
重启系统后，发现kubelet服务没有起来，首先检查：

1.vim /etc/fstab
#注释掉里面的swap一行。

2
/etc/systemd/system/kubelet.service.d/10-kubeadm.conf 文件加入KUBELET_CGROUP_ARGS和KUBELET_EXTRA_ARGS参数，

3.注意在启动参数中也要加入，如下：
[Service]

Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"
Environment="KUBELET_EXTRA_ARGS=--fail-swap-on=false"

ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_CGROUP_ARGS $KUBELET_EXTRA_ARGS

systemctl daemon-reload
systemctl restart kubelet
</pre>
== trouble2 重启一下机器就坏==
<pre>
为什么重启一下机器就坏了呢

systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: activating (auto-restart) (Result: exit-code) since Fri 2019-05-24 20:27:22 CST; 1s ago
Docs: https://kubernetes.io/docs/home/
Process: 1889 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (cod
Main PID: 1889 (code=exited, status=255)

kubelet.service: Main process exited, code=exited, status=255

journalctl -xefu kubelet

原来是kubelet 的cgroup dirver 与 docker的不一样。docker默认使用cgroupfs,keubelet 默认使用systemd。

简单地说就是在kubeadm init 之前kubelet会不断重启。

[kubelet-check] Initial timeout of 40s passed.

Unfortunately, an error has occurred:
timed out waiting for the condition

This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'

在集群初始化遇到问题，可以使用下面的命令进行清理后重新再初始化：

kubeadm reset
ifconfig cni0 down
ip link delete cni0
ifconfig flannel.1 down
ip link delete flannel.1
rm -rf /var/lib/cni/

</pre>

[https://segmentfault.com/q/1010000015988481 K8S 初始化问题，有哪位遇到过，求解！timed out waiting for the condition]

== trouble3 ==
<pre>
evan@k8s-master:~$ docker pull gcr.io/kubernetes-helm/tiller:v2.14.0
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.39/images/create?fromImage=gcr.io%2Fkubernetes-helm%2Ftiller&tag=v2.14.0: dial unix /var/run/docker.sock: connect: permission denied

sudo usermod -a -G docker $USER #普通用户添加天docker 组

</pre>
[https://www.cnblogs.com/informatics/p/8276172.html Docker pull Get Permission Denied]

==trouble 3 ==
docker 223.6.6.6 有时有问题建议用8.8.4.4

=see also=

[https://www.jianshu.com/p/21a39ee86311?utm_campaign=maleskine&utm_content=note&utm_medium=seo_notes&utm_source=recommendation ubuntu 离线搭建Kubenetes1.9.2 集群]

[https://www.cnblogs.com/RainingNight/p/using-kubeadm-to-create-a-cluster-1-12.html 使用Kubeadm搭建Kubernetes(1.12.2)集群]

[https://www.debian.cn/archives/3076 Debian 9 使用kubeadm创建 k8s 集群（上）]

[https://www.debian.cn/archives/3078 Debian 9 使用kubeadm创建 k8s 集群（下）]

[https://www.linuxtechi.com/install-configure-kubernetes-ubuntu-18-04-ubuntu-18-10/ Install and Configure Kubernetes (k8s) 1.13 on Ubuntu 18.04 LTS / Ubuntu 18.10]

[https://www.kubernetes.org.cn/4387.html Ubuntu 18.04 离线安装Kubernetes v1.11.1]

[https://www.cnblogs.com/Leo_wl/p/8511902.html 安装部署 Kubernetes 集群]

[[category:k8s]] [[category:容器]] [[category: container]]

Install and Configure Kubernetes (k8s) on ubuntu - 版本历史

Evan：​/* .Install using the repository on debian */

Evan：/* .Install using the repository on debian */