mirror of https://github.com/k3d-io/k3d
[Enhancement] docs: add a project overview (#680)
Thorsten Klein
3 years ago
committed by
GitHub
parent
cb02f6bce1
commit
3b8c877b43
@ -1,4 +1,5 @@ |
||||
title: Internals |
||||
nav: |
||||
- defaults.md |
||||
- networking.md |
||||
- project.md |
||||
- networking.md |
||||
|
||||
@ -0,0 +1,109 @@ |
||||
# Project Overview |
||||
|
||||
## About This Page |
||||
|
||||
On this page we'll try to give an overview of all the moving bits and pieces in k3d to ease contributions to the project. |
||||
|
||||
## Directory Overview |
||||
|
||||
- [`.github/`](https://github.com/rancher/k3d/tree/main/.github) |
||||
- templates for issues and pull requests |
||||
- GitHub Action workflow definitions |
||||
- [`cmd/`](https://github.com/rancher/k3d/tree/main/cmd) |
||||
- everything related to the actual k3d CLI, like the whole command tree, config initialization, argument parsing, etc. |
||||
- [`docgen/`](https://github.com/rancher/k3d/tree/main/docgen) |
||||
- sub-module used to auto-generate the documentation for the CLI commands, which ends up in [`docs/usage/commands/`](https://github.com/rancher/k3d/tree/main/docs/usage/commands) |
||||
- [`docs/`](https://github.com/rancher/k3d/tree/main/docs) |
||||
- all the resources used to build [k3d.io](https://k3d.io) using mkdocs |
||||
- [`pkg/`](<https://github.com/rancher/k3d/tree/main/pkg>) |
||||
- the place where the magic happens.. here you find all the main logic of k3d |
||||
- all function calls within [`cmd/`](https://github.com/rancher/k3d/tree/main/cmd) that do non-trivial things are imported from here |
||||
- this (or rather sub-packages) is what other projects would import as a module to work with k3d without using the CLI |
||||
- [`proxy/`](https://github.com/rancher/k3d/tree/main/proxy) |
||||
- configuration to build the [`rancher/k3d-proxy`](https://hub.docker.com/r/rancher/k3d-proxy/) container image which is used as a loadbalancer/proxy in front of (almost) every k3d cluster |
||||
- this is basically just a combination of NGINX with confd and some k3d-specific configuration details |
||||
- [`tests/`](https://github.com/rancher/k3d/tree/main/tests) |
||||
- a set of bash scripts used for end-to-end (E2E) tests of k3d |
||||
- mostly used for all the functionality of the k3d CLI which cannot be tested using Go unit tests |
||||
- [`tools/`](https://github.com/rancher/k3d/tree/main/tools) |
||||
- sub-module used to build the [`rancher/k3d-tools`](https://hub.docker.com/r/rancher/k3d-tools) container image which supports some k3d functionality like `k3d image import` |
||||
- [`vendor/`](https://github.com/rancher/k3d/tree/main/vendor) |
||||
- result of `go mod vendor`, which contains all dependencies of k3d |
||||
- [`version/`](https://github.com/rancher/k3d/tree/main/version) |
||||
- package used to code k3d/k3s versions into releases |
||||
- this is where `go build` injects the version tags when building k3d |
||||
- that's the output you see when issuing `k3d version` |
||||
|
||||
## Packages Overview |
||||
|
||||
- [`pkg/`](https://github.com/rancher/k3d/tree/main/pkg) |
||||
- [`actions/`](https://github.com/rancher/k3d/tree/main/pkg/actions) |
||||
- hook actions describing actions (commands, etc.) that run at specific stages of the node/cluster lifecycle |
||||
- e.g. writing configuration files to the container filesystem just before the node (container) starts |
||||
- [`client/`](https://github.com/rancher/k3d/tree/main/pkg/client) |
||||
- all the top level functionality to work with k3d primitives |
||||
- create/retrieve/update/delete/start/stop clusters, nodes, registries, etc. managed by k3d |
||||
- [`config/`](https://github.com/rancher/k3d/tree/main/pkg/config) |
||||
- everything related to the k3d configuration (files), like `SimpleConfig` and `ClusterConfig` |
||||
- [`runtimes/`](https://github.com/rancher/k3d/tree/main/pkg/runtimes) |
||||
- interface and implementations of runtimes that power k3d (currently, that's only Docker) |
||||
- functions in [`client/`](https://github.com/rancher/k3d/tree/main/pkg/client) eventually call runtime functions to "materialize" nodes and clusters |
||||
- [`tools/`](https://github.com/rancher/k3d/tree/main/pkg/tools) |
||||
- functions eventually calling the [`k3d-tools`](https://hub.docker.com/r/rancher/k3d-tools) container (see [`tools/`](https://github.com/rancher/k3d/tree/main/tools) in the repo root) |
||||
- [`types/`](https://github.com/rancher/k3d/tree/main/pkg/types) |
||||
- definition of all k3d primitives and many other details and defaults |
||||
- e.g. contains the definition of a `Node` or a `Cluster` in k3d |
||||
- [`util/`](https://github.com/rancher/k3d/tree/main/pkg/util) |
||||
- some helper functions e.g. for string manipulation/generation, regexp or other re-usable usages |
||||
|
||||
## Anatomy of a Cluster |
||||
|
||||
By default, every k3d cluster consists of at least 2 containers (nodes): |
||||
|
||||
1. (optional, but default and strongly recommended) loadbalancer |
||||
|
||||
- image: [`rancher/k3d-proxy`](https://hub.docker.com/r/rancher/k3d-proxy/), built from [`proxy/`](https://github.com/rancher/k3d/tree/main/proxy) |
||||
- purpose: proxy and load balance requests from the outside (i.e. most of the times your local host) to the cluster |
||||
- by default, it e.g. proxies all the traffic for the Kubernetes API to port `6443` (default listening port of K3s) to all the server nodes in the cluster |
||||
- can be used for multiple port-mappings to one or more nodes in your cluster |
||||
- that way, port-mappings can also easily be added/removed after the cluster creation, as we can simply re-create the proxy without affecting cluster state |
||||
|
||||
2. (required, always present) primary server node |
||||
|
||||
- image: [`rancher/k3s`](https://hub.docker.com/r/rancher/k3s/), built from [`github.com/k3s-io/k3s`](https://github.com/k3s-io/k3s) |
||||
- purpose: (initializing) server (formerly: master) node of the cluster |
||||
- runs the K3s executable (which runs containerd, the Kubernetes API Server, etcd/sqlite, etc.): `k3s server` |
||||
- in a multi-server setup, it initializes the cluster with an embedded etcd database (using the K3s `--cluster-init` flag) |
||||
|
||||
3. (optional) secondary server node(s) |
||||
|
||||
- image: [`rancher/k3s`](https://hub.docker.com/r/rancher/k3s/), built from [`github.com/k3s-io/k3s`](https://github.com/k3s-io/k3s) |
||||
|
||||
4. (optional) agent node(s) |
||||
|
||||
- image: [`rancher/k3s`](https://hub.docker.com/r/rancher/k3s/), built from [`github.com/k3s-io/k3s`](https://github.com/k3s-io/k3s) |
||||
- purpose: running the K3s agent process (kubelet, etc.): `k3s agent` |
||||
|
||||
## Automation (CI) |
||||
|
||||
The k3d repository mainly leverages the following two CI systems: |
||||
|
||||
- GitHub Actions |
||||
- 2 workflows in <https://github.com/rancher/k3d/tree/main/.github/workflows> to push the artifact to AUR (Arch Linux User Repository) |
||||
- logs/history can be seen in the Actions tab: <https://github.com/rancher/k3d/actions> |
||||
- DroneCI |
||||
- a set of pipelines in a single file: <https://github.com/rancher/k3d/blob/main/.drone.yml> |
||||
- static code analysis |
||||
- build |
||||
- tests |
||||
- docker builds + pushes |
||||
- render + push docs |
||||
- (pre-) release to GitHub |
||||
- `push` events end up here (also does the releases, when a tag is pushed): <https://drone-publish.rancher.io/rancher/k3d> |
||||
- `pr`s end up here: <https://drone-pr.rancher.io/rancher/k3d> |
||||
|
||||
## Documentation |
||||
|
||||
The website [k3d.io](https://k3d.io) containing all the documentation for k3d is built using [`mkdocs`](https://www.mkdocs.org/), configured via the [`mkdocs.yml`](https://github.com/rancher/k3d/blob/main/mkdocs.yml) config file with all the content residing in the [`docs/`](https://github.com/rancher/k3d/tree/main/docs) directory (Markdown). |
||||
Use `mkdocs serve` in the repository root to build and serve the webpage locally. |
||||
Some parts of the documentation are being auto-generated, like [`docs/usage/commands/`](https://github.com/rancher/k3d/tree/main/docs/usage/commands) is auto-generated using Cobra's command docs generation functionality in [`docgen/`](https://github.com/rancher/k3d/tree/main/docgen). |
||||
@ -1,153 +0,0 @@ |
||||
# Thoughts |
||||
|
||||
## Repository/Package Overview |
||||
|
||||
- `cmd/`: everything around the CLI of k3d = human interface, printed output (e.g. list of clusters) |
||||
- `pkg/`: everything else, can be used as a module from other Go projects |
||||
- `cluster/`: everything around managing cluster components |
||||
- `runtimes/`: translate k3d types (node, cluster, etc.) to container runtime specific types and manage them |
||||
- `types/`: collection of types (structs) and constants used by k3d |
||||
- `util/`: utilities, that could be used for everything, not directly related to the project |
||||
|
||||
## k3d types <-> runtime translation |
||||
|
||||
k3d _should_ work with more than one runtime, if we can implement the Runtime interface for it. |
||||
Here's how k3d types should translate to a runtime type: |
||||
|
||||
- `cluster` = set of _containers_ running in the same _network_, maybe mounting the same _volume(s)_ |
||||
- `node` = _container_ with _exposed ports_ and _volume mounts_ |
||||
|
||||
### Docker |
||||
|
||||
#### Node to Container translation |
||||
|
||||
`container = "github.com/docker/docker/api/types/container"` |
||||
`network = "github.com/docker/docker/api/types/network"` |
||||
|
||||
- Name -> container.Hostname = node.Name |
||||
- Role -> container.Labels["k3d.role"] = node.Role |
||||
- Image -> container.Image = node.Image |
||||
- Volumes -> container.HostConfig.PortBindings |
||||
- Env -> |
||||
- Args -> |
||||
- Ports -> |
||||
- Restart -> |
||||
- Labels -> container.Labels |
||||
|
||||
## Node Configuration |
||||
|
||||
- server node(s) |
||||
- ENV |
||||
- `K3S_CLUSTER_INIT` |
||||
- if num_servers > 1 && no external datastore configured |
||||
- `K3S_KUBECONFIG_OUTPUT` |
||||
- k3d default -> `/output/kubeconfig.yaml` |
||||
- CMD/ARGS |
||||
- `--https-listen-port` |
||||
- can/should be left default (unset = 6443), since we handle it via port mapping |
||||
- `--tls-san=<some-ip-or-hostname>` |
||||
- get from `--api-port` k3d flag and/or from docker machine |
||||
- Runtime Configuration |
||||
- nothing special |
||||
- all nodes |
||||
- ENV |
||||
- `K3S_TOKEN` for node authentication |
||||
- CMD/ARGS |
||||
- nothing special |
||||
- Runtime Configuration |
||||
- Volumes |
||||
- shared image volume |
||||
- cluster-specific (create cluster) or inherit from existing (create node) |
||||
- tmpfs for k3s to work properly |
||||
- `/run` |
||||
- `/var/run` |
||||
- Capabilities/Security Context |
||||
- `privileged` |
||||
- Network |
||||
- cluster network or external/inherited |
||||
- agent nodes |
||||
- ENV |
||||
- `K3S_URL` to connect to server node |
||||
- server hostname + port (6443) |
||||
- cluster-specific or inherited |
||||
- CMD/ARGS |
||||
- nothing special |
||||
- Runtime Configuration |
||||
- nothing special |
||||
|
||||
## Features |
||||
|
||||
## [DONE] Node Filters |
||||
|
||||
- `--port [host:]port[:containerPort][/protocol][@group_identifier[[index] | @node_identifier]` |
||||
- Examples: |
||||
- `--port 0.0.0.0:8080:8081/tcp@agents` -> whole group |
||||
- `--port 80@agents[0]` -> single instance of group by list index |
||||
- `--port 80@agents[0,2-3]` -> multiple instances of a group by index lists and ranges |
||||
- `--port 80@k3d-test-agent-0` -> single instance by specific node identifier |
||||
- `--port 80@k3d-test-server-0@agents[1-5]` -> multiple instances by combination of node and group identifiers |
||||
|
||||
- analogous for volumes |
||||
|
||||
## [WIP] Multi-Server Setup |
||||
|
||||
- to make this possible, we always deploy a load-balancer (nginx) in front of the server nodes as an extra container |
||||
- consider that in the kubeconfig file and `--tls-san` |
||||
|
||||
### Variants |
||||
|
||||
- [x] embedded datastore (dqlite) |
||||
- if `--servers` > 1 deploy a load-balancer in front of them as an extra container |
||||
- [ ] external datastore |
||||
|
||||
## [DONE] Keep State in Docker Labels |
||||
|
||||
- when creating a cluster, usually, you also create a new docker network (and maybe other resources) |
||||
- store a reference to those in the container labels of cluster nodes |
||||
- when deleting the cluster, parse the labels, deduplicate the results and delete the additional resources |
||||
- DONE for network |
||||
- new labels `k3d.cluster.network=<ID>` and `k3d.cluster.network.external=<true|false>` (determine whether to try to delete it when you delete a cluster, since network may have been created manually) |
||||
|
||||
## Bonus Ideas |
||||
|
||||
### Tools |
||||
|
||||
- maybe rename `k3d load` to `k3d tools` and add tool cmds there? |
||||
- e.g. `k3d tools import-images` |
||||
- let's you set tools container version |
||||
- `k3d tools --image k3d-tools:v2 import-images` |
||||
- add `k3d create --image-vol NAME` flag to re-use existing image volume |
||||
- will add `k3d.volumes.imageVolume.external: true` label to nodes |
||||
- should not be deleted with cluster |
||||
- possibly add `k3d create volume` and `k3d create network` to create external volumes/networks? |
||||
|
||||
### Prune Command |
||||
|
||||
- `k3d prune` to prune all dangling resources |
||||
- nodes, volumes, networks |
||||
|
||||
### Use Open Standards (OCI, CRI, ...) |
||||
|
||||
- [https://github.com/opencontainers/runtime-spec/blob/master/specs-go/config.go](https://github.com/opencontainers/runtime-spec/blob/master/specs-go/config.go) |
||||
- move node -> container translation out of runtime |
||||
|
||||
### Private registry |
||||
|
||||
- create a private registry to be used by k3d clusters |
||||
- similar to [https://github.com/rancher/k3d/pull/161](https://github.com/rancher/k3d/pull/161) |
||||
- add `k3d create registry` command to create external registry (maybe instead of flags as in PR #161?) |
||||
|
||||
### Syntactical shortcuts for k3d v1 backwards compatibility |
||||
|
||||
- e.g. `k3d create` -> `k3d create cluster k3s-default` |
||||
|
||||
### Unsorted Ideas |
||||
|
||||
- Integrate build tool (e.g. buildkit, buildah, ...) |
||||
- use `tools.go` to keep tools (like `golangci-lint` and `gox`) dependencies |
||||
- see e.g. [https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module](https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module) |
||||
- see e.g. [https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module](https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module) |
||||
|
||||
### Possible Enhancements |
||||
|
||||
- [!] remove/add nodes -> needs to remove line in `/var/lib/rancher/k3s/server/cred/node-passwd` for the deleted node |
||||
Loading…
Reference in new issue