k3s Dev (1) Environment Setup

Posted on Edited on In
Symbols count in article: 7.6k Reading time ≈ 7 mins.

This post shows in steps how to set up a dev environment for k3s project.

Step 1. Preparation

The following list shows my local development environment setup:

For code review & build purpose:

  • OS: Mac OS
  • Docker: 18.06.1-ce
  • JetBrains GoLand IDE
  • golang: 1.12.7

For testing deploy purpose:

  • VirtualBox
  • Ubuntu 18.04 Server

Alternatively, one could use EC2 instances as testing deploy environment, in which case may directly to go step 2.

1.1 VirtualBox Setup

Download Ubuntu 18.04 iso from official source and install it on VirtualBox. Configure the network interface as following:

1
2
3
4
5
# Check all available network adapters
ifconfig -a

# Then Use netplan to config all interfaces
sudo vi /etc/netplan/50-cloud-init.yaml

Change the content accordingly (192.168.56.1 is my gateway setting, change it according to your own VirtualBox network setting):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
network:
    ethernets:
        enp0s3:
            addresses: []
            dhcp4: true
            dhcp6: no
            optional: true
            nameservers:
               addresses: [8.8.8.8]
        enp0s8:
             addresses: [192.168.56.10/24]
             dhcp4: true
             dhcp6: no
             optional: true
             routes:
                - to: 192.168.56.1/24
                  via: 192.168.56.1
    version: 2

Save and apply the changes:

1
2
3
4
5
6
7
sudo netplan apply

# Test connectivity
ping 192.168.56.101 # another virtual machine

# Test installing k3s
curl -sfL https://get.k3s.io | INSTALL_K3S_BIN_DIR="/home/main/k3s" sh -

1.2 Golang Environment

For Mac OS users, I recommend use homebrew to manage go environment. However, go has its own version manager called GVM, refer here for detailed information. To use earlier version of go, add following to your .bashrc file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# homebrew go version config 
goconfig() {
    case "$1" in
        "1.12")
            brew switch go 1.12.7
            export GOROOT=/usr/local/Cellar/go/1.12.7/libexec
            ;;
        "1.13")
            brew switch go 1.13
            export GOROOT=/usr/local/Cellar/go/1.13/libexec
            ;;
        *)
            brew switch go 1.12.7
            export GOROOT=/usr/local/Cellar/go/1.12.7/libexec
            ;;

    esac

    # Installed Go path
    export GOPATH=$HOME/go
    export PATH=$PATH:GOROOT/bin
    export PATH=$PATH:$(go env GOPATH)/bin
}

To use simply pass the version selected. For example, to change to go version 12:

1
2
# Switch to go version 12
goconfig 1.12

1.3 Source Code Download

With go environment set up, now we go to its workspace to pull the source from github (I have already forked k3s project to my own github account, if that’s not the case for you, simply pull the source from rancher)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
cd /Users/yourname/go/src/github.com/yourgithubacct

# Download my forked project to local
git clone --depth 1 https://github.com/ruby-/k3s.git

# Add rancher remote source
git remote add rancher https://github.com/rancher/k3s.git

# now there should be two remote sources: origin (my own) and rancher's
git remote

# fetch from rancher's master branch and merge to local (shouldn't be any change yet)
git fetch rancher
git merge rancher/master

Step 2. Build & Test Launch

Open the k3s folder in GoLand IDE, on Mac OS, press ⌘ + ⇧ + f to search in path for “k3s is up and running”, open the pkg/cli/server/server.go file and add following statement:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
func run(app *cli.Context, cfg *cmds.Server) error {
    ...

    ctx := signals.SetupSignalHandler(context.Background())
    certs, err := server.StartServer(ctx, &serverConfig)
    if err != nil {
		return err
	}

    logrus.Info("k3s is up and running")
    
    // Add a new logging info here
    logrus.Info("==================> Fog is just clouds that have fell down! <==================")

    if notifySocket != "" {
    	os.Setenv("NOTIFY_SOCKET", notifySocket)
        systemd.SdNotify(true, "READY=1\n")
    }

    ...
}

Now open the terminal within GoLand IDE, then issue make command and wait for it’s pulling required docker images and build the project for us. After the building procedure, there should be hyperkube, k3s produced in the dist/artifacts/ folder. Simply copy k3s to your deploy environment for testing purpose (for me only need to copy to virtualbox ubuntu server that has been setup):

1
2
3
4
5
scp dist/artifacts/k3s main@192.168.56.10:~/k3s/

# Start k3s server and verify the added logged info
k3s server > server.log 2>&1 &
vi server.log

Note that:

  • (1) Remember to chmod 777 k3s_install_path the install path for scp to be able to upload k3s executable;
  • (2) k3s doesn’t support cross compile at current release, refer to issue#530 for detailed discussion regarding this;
  • (3) There are two options for developers to build k3s from source. We are adapting docker method with rancher/dapper wrapper in this post. However for those who are working on Linux, may alternatively download the dependencies and directly build the source with go build commend described here; unfortunately this compile option is not available for Mac OS users at current release, refer to issue#273 for detailed discussion regarding this.
  • (4) For Mac OS users, docker tends to accumulate its data file (store at /Users/yourname/Library/Containers/com.docker.docker/Data/vms/0/Docker.qcow2) which contains its self os image and downloaded containers. Build k3s project may consume ~10GB space on disk and it keeps growing. The way to get around this is to use docker built-in “reset” tab (Also note that docker system prune won’t save you from this pitfall).

Step 3. Breakpoint & debugging

It is essential for us engineers to be able to set up breakpoint and debug the code in software development. In this section we’ll try to connect to remotely deployed k3s executable and use delve tool to navigate through the debugging process.

3.1 Install go and dlv in deploy env

To install and set up go environment and dlv debugger on Ubuntu 18.04 server VM we created earlier, simply follow these two guides:

  • How To Install Go on Ubuntu 18.04
  • dlv - Installation on Linux

To get familiar with dlv basic workflows and combine it with JetBrain GoLand IDE, I recommend to follow these practices:

  • Using the Go Delve Debugger from the command line
  • Debugging with GoLand – Getting Started

Now let us go back to our k3s project, find and modify the following lines in scripts/build.sh:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
...

VERSIONFLAGS="
    -X $PKG/pkg/version.Version=$VERSION
    -X $PKG/pkg/version.GitCommit=${COMMIT:0:8}
    -X $PKG/vendor/$PKG_CONTAINERD/version.Version=$VERSION_CONTAINERD
    -X $PKG/vendor/$PKG_CONTAINERD/version.Package=$PKG_RANCHER_CONTAINERD
    -X $PKG/vendor/$PKG_CRICTL/pkg/version.Version=$VERSION_CRICTL"
# Comment out LDFLAGS (link) for debug: "-w"= wipe out debug info ;; "-s"= remove symbol table
# LDFLAGS="
#    -w -s"
LDFLAGS=""
STATIC="
    -extldflags '-static'
"

...

Go to scripts/package-cli.sh file and change the following:

1
2
3
4
5
6
7
8
9
10
11
12
...

go generate
LDFLAGS="
    -X github.com/rancher/k3s/pkg/version.Version=$VERSION
    -X github.com/rancher/k3s/pkg/version.GitCommit=${COMMIT:0:8}
    -w -s
"
STATIC="-extldflags '-static'"
# CGO_ENABLED=0 go build -ldflags "$LDFLAGS $STATIC" -o ${CMD_NAME} ./cmd/k3s/main.go
# Add gcflags to enable dlv
CGO_ENABLED=0 go build -gcflags "all=-N -l" -ldflags "$STATIC" -o ${CMD_NAME} ./cmd/k3s/main.go

Commit the file changes to our local git repo and rebuild the project, then upload the new k3s executable to our local VM:

1
2
3
4
5
6
7
8
9
10
11
12
# Commit local changes
git add .
git commit -m "enable dlv"

# Rebuild project
make

# Upload to deploy
scp dist/artifacts/k3s main@192.168.56.10:~/k3s/

# Launch with dlv: note `dlv exec xxx -- paras` means passing the parameters after "--" to our executable instead of dlv
dlv --listen=:2345 --headless=true --api-version=2 --accept-multiclient exec /home/main/k3s/k3s -- --debug server

Setup a new go remote debug run/debug configuration in GoLand IDE which looks like this:
image

Toggle some breakpoint at main entry then click debug button to start debugging.
image

Some of my questions:

  • (1) Use the make method to build the whole project takes long time (3-5 min for me). I wonder if there’s some “quick” and “convenient” way to speedup this procedure, e.g. build only parts that have been changed? Please leave comments below if you know the answer to this.
    (Updated) Please refer my following post for a better way to separate source code edit & build/test environment here at k3s Dev (2) Vagrant + VirtualBox Dev.
  • (2) The latest stable version of GoLand(2019.2) leave the channel open after the debugging procedure has been stopped within IDE. Simple kill the dlv by PID to get around this issue.
  • (3) Run k3s command with generate a bunch of code in /var/lib/rancher/k3s/, these run time generated binaries and configurations contain most important components including k3s-server and k3s-agent. Details about these we’ll explore also in next my post.

Summery

Learning k3s from source is important for us to contribute to its community. In my next post, I’ll try to investigate its scheduling mechanism and explain the related major workflow.

ㄟ(●′ω`●)ㄏ
0%