k3s+kubeedge (2) Deploy the Cloud Core (edgecontroller)

In the previous post, we know how to set up a code-review/debugging environment (some preparations are actually included in this post though). Now, let’s try to bring up the Kubeedge cloudcore (“edgecontroller” as for v1.0.0) on our k3s master. Again, this post assumes you already have a minimal k3s/k8s master up and running at the server, if you don’t know how to do so, feel free to visit my previous posts on guiding you through the k3s setup.

Step 1. Preparation – k3s revisit

In my last post, I forgot to mention about my k3s environment (打脸). Since we are using Docker other than Containerd for k3s, I recommend to change the k3s configuration as following (and disable several services that we are not currently using):

# vi /etc/systemd/system/k3s.service
[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/etc/systemd/system/k3s.service.env
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s \
    server \
    --no-deploy traefik \
    --docker \
    --no-deploy servicelb \

KillMode=process
Delegate=yes
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target

Save the file and reload daemon then restart k3s service:

systemctl daemon-reload
systemctl restart k3s

#Check the k3s is successfully restarted with the modified configurations
systemctl status k3s

Traefik is always restarting automatically with k3s. Thus to truly disable it, one should follow the commands mentioned in here.

Step 2. Build & Deploy cloudimage

Feel free to refresh your memory on how to build the Kubeedge cloudimage in my previous post. However if you only wish to test deploy, you may use mine here r5by/kubeedge_edgecontroller:v1.0.0. After the cloudimage is prepared, the next step is to generate certification files:

cd build/cloud
../tools/certgen.sh buildSecret | tee ./06-secret.yaml

This scripts will generate a bunch of ca/crt files under path /etc/kubeedge/ and write the secrets to 06-secret.yaml. Then next, we should copy and paste the kubeconfig to that location for later usage:

mkdir /etc/kubeedge/cloud
# k3s kubeconfig location
cp /etc/rancher/k3s/k3s.yaml /etc/kubeedge/cloud/kubeconfig.yaml

Step 3. Modify the other yaml files

In the build/cloud path, several yaml files are listed:

cd build/cloud/; ls
# Output:
# 01-namespace.yaml          03-clusterrole.yaml        05-configmap.yaml          07-deployment.yaml         Dockerfile                  README_zh.md
# 02-serviceaccount.yaml     04-clusterrolebinding.yaml 06-secret.yaml             08-service.yaml.example    README.md

Change 03-clusterrole.yaml as following:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: edgecontroller
  labels:
    k8s-app: kubeedge
    kubeedge: edgecontroller
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - nodes/status
  - configmaps
  - pods
  - pods/status
  - secrets
  - services
  - endpoints
  verbs:
  - get
  - list
  - watch
  - update
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - delete

Change 05-configmap.yaml as following:

apiVersion: v1
kind: ConfigMap
metadata:
  name: edgecontroller
  namespace: kubeedge
  labels:
    k8s-app: kubeedge
    kubeedge: edgecontroller
data:
  controller.yaml: |
    controller:
      kube:
        master: https://kubernetes.default.svc.cluster.local:443
        kubeconfig: /etc/kubeedge/cloud/kubeconfig.yaml
        namespace: ""
        content_type: "application/vnd.kubernetes.protobuf"
        qps: 5
        burst: 10
        node_update_frequency: 10
    cloudhub:
      address: 0.0.0.0
      port: 10000
      ca: /etc/kubeedge/certs/rootCA.crt
      cert: /etc/kubeedge/certs/edge.crt
      key: /etc/kubeedge/certs/edge.key
      keepalive-interval: 30
      write-timeout: 30
      node-limit: 10
    devicecontroller:
       kube:
         master: https://kubernetes.default.svc.cluster.local:443
         namespace: ""
         content_type: "application/vnd.kubernetes.protobuf"
         qps: 5
         burst: 10
         kubeconfig: /etc/kubeedge/cloud/kubeconfig.yaml
           logging.yaml: |
    loggerLevel: "INFO"
    enableRsyslog: false
    logFormatText: true
    writers: [stdout]
  modules.yaml: |
    modules:
      enabled: [controller, cloudhub]

Then verify if your 06-secret.yaml is consistent with the ca/cert files in path /etc/kubeedge/, also copy and paste the 07-deployment.yaml and 08-service.yaml files from our last post.

The configurations are now set. Use the shell script we write in last post to initialize start Kubeedge edgecontroller on the cloud.

If everything works as expect, you should be able to verify the edgecontroller service is up:

# `kce` alias `k3s kubectl -n kubeedge`
kce get pod
#Output
#NAME                            READY   STATUS    RESTARTS   AGE
#edgecontroller-5dc9955c-p2c7s   1/1     Running   0          5s

kce get svc
#NAME             TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                          AGE
#edgecontroller   NodePort   10.43.217.231   <none>        10000:30267/TCP,2345:32345/TCP   27s

kce logs edgecontroller-5dc9955c-p2c7s
# No error messages should be found...

Step 4. Trouble Shooting

Okay, I understand life is complicated. Unfortunately if you were caught by some weird error messages, don’t panic (even though the compiler does). Check the issues I mentioned in the previous post, and verify carefully each of these yaml files to make sure you understand what should be there. I also listed here several forum posts that may help with your issues:

• 1) DNS issue, not the fix for mine
• 2) DNS issue, again not the fix for mine
• 3) DNS issue, no luck either
• 4) DNS issue, finally works, fix mentioned in here
• 5) DNS issue, not works for mine
• 6) DNS issue, from k3s forum, not a solution for my case

Finally, if you have other questions/problems and solutions, please leave your comments below to share with others. In the next post, I’ll complete the Kubeedge cluster setup with the edgecore setup on both lovely Raspberry Pi’s.