diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 92383c8c..be237728 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -7,6 +7,8 @@ We welcome everyone who likes to use and improve our software. Before starting to work with and on k3d, please read and understand our [**Code of Conduct**](./CODE_OF_CONDUCT.md). +Get an Overview of the k3d project in the documentation: [k3d.io/internals/project](https://k3d.io/internals/project) + Before opening an issue or a Pull-Request, please use GitHub's search function to check whether something similar is already in process and hook in there instead. ## Get Recognized diff --git a/README.md b/README.md index 5bba5414..10975e91 100644 --- a/README.md +++ b/README.md @@ -112,6 +112,8 @@ k3d is a community-driven project and so we welcome contributions of any form, b Please read our [**Contributing Guidelines**](./CONTRIBUTING.md) and the related [**Code of Conduct**](./CODE_OF_CONDUCT.md). +You can find an overview of the k3d project (e.g. explanations and a repository guide) in the documentation: [k3d.io/internals/project](https://k3d.io/internals/project) + [![Contributor Covenant](https://img.shields.io/badge/Contributor%20Covenant-v2.0%20adopted-ff69b4.svg)](code_of_conduct.md) ## Contributors ✨ diff --git a/docs/internals/.pages b/docs/internals/.pages index b711546b..a6ca7afa 100644 --- a/docs/internals/.pages +++ b/docs/internals/.pages @@ -1,4 +1,5 @@ title: Internals nav: - defaults.md - - networking.md \ No newline at end of file + - project.md + - networking.md diff --git a/docs/internals/project.md b/docs/internals/project.md new file mode 100644 index 00000000..bfd0b3fc --- /dev/null +++ b/docs/internals/project.md @@ -0,0 +1,109 @@ +# Project Overview + +## About This Page + +On this page we'll try to give an overview of all the moving bits and pieces in k3d to ease contributions to the project. + +## Directory Overview + +- [`.github/`](https://github.com/rancher/k3d/tree/main/.github) + - templates for issues and pull requests + - GitHub Action workflow definitions +- [`cmd/`](https://github.com/rancher/k3d/tree/main/cmd) + - everything related to the actual k3d CLI, like the whole command tree, config initialization, argument parsing, etc. +- [`docgen/`](https://github.com/rancher/k3d/tree/main/docgen) + - sub-module used to auto-generate the documentation for the CLI commands, which ends up in [`docs/usage/commands/`](https://github.com/rancher/k3d/tree/main/docs/usage/commands) +- [`docs/`](https://github.com/rancher/k3d/tree/main/docs) + - all the resources used to build [k3d.io](https://k3d.io) using mkdocs +- [`pkg/`]() + - the place where the magic happens.. here you find all the main logic of k3d + - all function calls within [`cmd/`](https://github.com/rancher/k3d/tree/main/cmd) that do non-trivial things are imported from here + - this (or rather sub-packages) is what other projects would import as a module to work with k3d without using the CLI +- [`proxy/`](https://github.com/rancher/k3d/tree/main/proxy) + - configuration to build the [`rancher/k3d-proxy`](https://hub.docker.com/r/rancher/k3d-proxy/) container image which is used as a loadbalancer/proxy in front of (almost) every k3d cluster + - this is basically just a combination of NGINX with confd and some k3d-specific configuration details +- [`tests/`](https://github.com/rancher/k3d/tree/main/tests) + - a set of bash scripts used for end-to-end (E2E) tests of k3d + - mostly used for all the functionality of the k3d CLI which cannot be tested using Go unit tests +- [`tools/`](https://github.com/rancher/k3d/tree/main/tools) + - sub-module used to build the [`rancher/k3d-tools`](https://hub.docker.com/r/rancher/k3d-tools) container image which supports some k3d functionality like `k3d image import` +- [`vendor/`](https://github.com/rancher/k3d/tree/main/vendor) + - result of `go mod vendor`, which contains all dependencies of k3d +- [`version/`](https://github.com/rancher/k3d/tree/main/version) + - package used to code k3d/k3s versions into releases + - this is where `go build` injects the version tags when building k3d + - that's the output you see when issuing `k3d version` + +## Packages Overview + +- [`pkg/`](https://github.com/rancher/k3d/tree/main/pkg) + - [`actions/`](https://github.com/rancher/k3d/tree/main/pkg/actions) + - hook actions describing actions (commands, etc.) that run at specific stages of the node/cluster lifecycle + - e.g. writing configuration files to the container filesystem just before the node (container) starts + - [`client/`](https://github.com/rancher/k3d/tree/main/pkg/client) + - all the top level functionality to work with k3d primitives + - create/retrieve/update/delete/start/stop clusters, nodes, registries, etc. managed by k3d + - [`config/`](https://github.com/rancher/k3d/tree/main/pkg/config) + - everything related to the k3d configuration (files), like `SimpleConfig` and `ClusterConfig` + - [`runtimes/`](https://github.com/rancher/k3d/tree/main/pkg/runtimes) + - interface and implementations of runtimes that power k3d (currently, that's only Docker) + - functions in [`client/`](https://github.com/rancher/k3d/tree/main/pkg/client) eventually call runtime functions to "materialize" nodes and clusters + - [`tools/`](https://github.com/rancher/k3d/tree/main/pkg/tools) + - functions eventually calling the [`k3d-tools`](https://hub.docker.com/r/rancher/k3d-tools) container (see [`tools/`](https://github.com/rancher/k3d/tree/main/tools) in the repo root) + - [`types/`](https://github.com/rancher/k3d/tree/main/pkg/types) + - definition of all k3d primitives and many other details and defaults + - e.g. contains the definition of a `Node` or a `Cluster` in k3d + - [`util/`](https://github.com/rancher/k3d/tree/main/pkg/util) + - some helper functions e.g. for string manipulation/generation, regexp or other re-usable usages + +## Anatomy of a Cluster + +By default, every k3d cluster consists of at least 2 containers (nodes): + +1. (optional, but default and strongly recommended) loadbalancer + + - image: [`rancher/k3d-proxy`](https://hub.docker.com/r/rancher/k3d-proxy/), built from [`proxy/`](https://github.com/rancher/k3d/tree/main/proxy) + - purpose: proxy and load balance requests from the outside (i.e. most of the times your local host) to the cluster + - by default, it e.g. proxies all the traffic for the Kubernetes API to port `6443` (default listening port of K3s) to all the server nodes in the cluster + - can be used for multiple port-mappings to one or more nodes in your cluster + - that way, port-mappings can also easily be added/removed after the cluster creation, as we can simply re-create the proxy without affecting cluster state + +2. (required, always present) primary server node + + - image: [`rancher/k3s`](https://hub.docker.com/r/rancher/k3s/), built from [`github.com/k3s-io/k3s`](https://github.com/k3s-io/k3s) + - purpose: (initializing) server (formerly: master) node of the cluster + - runs the K3s executable (which runs containerd, the Kubernetes API Server, etcd/sqlite, etc.): `k3s server` + - in a multi-server setup, it initializes the cluster with an embedded etcd database (using the K3s `--cluster-init` flag) + +3. (optional) secondary server node(s) + + - image: [`rancher/k3s`](https://hub.docker.com/r/rancher/k3s/), built from [`github.com/k3s-io/k3s`](https://github.com/k3s-io/k3s) + +4. (optional) agent node(s) + + - image: [`rancher/k3s`](https://hub.docker.com/r/rancher/k3s/), built from [`github.com/k3s-io/k3s`](https://github.com/k3s-io/k3s) + - purpose: running the K3s agent process (kubelet, etc.): `k3s agent` + +## Automation (CI) + +The k3d repository mainly leverages the following two CI systems: + +- GitHub Actions + - 2 workflows in to push the artifact to AUR (Arch Linux User Repository) + - logs/history can be seen in the Actions tab: +- DroneCI + - a set of pipelines in a single file: + - static code analysis + - build + - tests + - docker builds + pushes + - render + push docs + - (pre-) release to GitHub + - `push` events end up here (also does the releases, when a tag is pushed): + - `pr`s end up here: + +## Documentation + +The website [k3d.io](https://k3d.io) containing all the documentation for k3d is built using [`mkdocs`](https://www.mkdocs.org/), configured via the [`mkdocs.yml`](https://github.com/rancher/k3d/blob/main/mkdocs.yml) config file with all the content residing in the [`docs/`](https://github.com/rancher/k3d/tree/main/docs) directory (Markdown). +Use `mkdocs serve` in the repository root to build and serve the webpage locally. +Some parts of the documentation are being auto-generated, like [`docs/usage/commands/`](https://github.com/rancher/k3d/tree/main/docs/usage/commands) is auto-generated using Cobra's command docs generation functionality in [`docgen/`](https://github.com/rancher/k3d/tree/main/docgen). diff --git a/thoughts.md b/thoughts.md deleted file mode 100644 index 600467c3..00000000 --- a/thoughts.md +++ /dev/null @@ -1,153 +0,0 @@ -# Thoughts - -## Repository/Package Overview - -- `cmd/`: everything around the CLI of k3d = human interface, printed output (e.g. list of clusters) -- `pkg/`: everything else, can be used as a module from other Go projects - - `cluster/`: everything around managing cluster components - - `runtimes/`: translate k3d types (node, cluster, etc.) to container runtime specific types and manage them - - `types/`: collection of types (structs) and constants used by k3d - - `util/`: utilities, that could be used for everything, not directly related to the project - -## k3d types <-> runtime translation - -k3d _should_ work with more than one runtime, if we can implement the Runtime interface for it. -Here's how k3d types should translate to a runtime type: - -- `cluster` = set of _containers_ running in the same _network_, maybe mounting the same _volume(s)_ -- `node` = _container_ with _exposed ports_ and _volume mounts_ - -### Docker - -#### Node to Container translation - -`container = "github.com/docker/docker/api/types/container"` -`network = "github.com/docker/docker/api/types/network"` - -- Name -> container.Hostname = node.Name -- Role -> container.Labels["k3d.role"] = node.Role -- Image -> container.Image = node.Image -- Volumes -> container.HostConfig.PortBindings -- Env -> -- Args -> -- Ports -> -- Restart -> -- Labels -> container.Labels - -## Node Configuration - -- server node(s) - - ENV - - `K3S_CLUSTER_INIT` - - if num_servers > 1 && no external datastore configured - - `K3S_KUBECONFIG_OUTPUT` - - k3d default -> `/output/kubeconfig.yaml` - - CMD/ARGS - - `--https-listen-port` - - can/should be left default (unset = 6443), since we handle it via port mapping - - `--tls-san=` - - get from `--api-port` k3d flag and/or from docker machine - - Runtime Configuration - - nothing special -- all nodes - - ENV - - `K3S_TOKEN` for node authentication - - CMD/ARGS - - nothing special - - Runtime Configuration - - Volumes - - shared image volume - - cluster-specific (create cluster) or inherit from existing (create node) - - tmpfs for k3s to work properly - - `/run` - - `/var/run` - - Capabilities/Security Context - - `privileged` - - Network - - cluster network or external/inherited -- agent nodes - - ENV - - `K3S_URL` to connect to server node - - server hostname + port (6443) - - cluster-specific or inherited - - CMD/ARGS - - nothing special - - Runtime Configuration - - nothing special - -## Features - -## [DONE] Node Filters - -- `--port [host:]port[:containerPort][/protocol][@group_identifier[[index] | @node_identifier]` - - Examples: - - `--port 0.0.0.0:8080:8081/tcp@agents` -> whole group - - `--port 80@agents[0]` -> single instance of group by list index - - `--port 80@agents[0,2-3]` -> multiple instances of a group by index lists and ranges - - `--port 80@k3d-test-agent-0` -> single instance by specific node identifier - - `--port 80@k3d-test-server-0@agents[1-5]` -> multiple instances by combination of node and group identifiers - -- analogous for volumes - -## [WIP] Multi-Server Setup - -- to make this possible, we always deploy a load-balancer (nginx) in front of the server nodes as an extra container - - consider that in the kubeconfig file and `--tls-san` - -### Variants - -- [x] embedded datastore (dqlite) - - if `--servers` > 1 deploy a load-balancer in front of them as an extra container -- [ ] external datastore - -## [DONE] Keep State in Docker Labels - -- when creating a cluster, usually, you also create a new docker network (and maybe other resources) - - store a reference to those in the container labels of cluster nodes - - when deleting the cluster, parse the labels, deduplicate the results and delete the additional resources - - DONE for network - - new labels `k3d.cluster.network=` and `k3d.cluster.network.external=` (determine whether to try to delete it when you delete a cluster, since network may have been created manually) - -## Bonus Ideas - -### Tools - -- maybe rename `k3d load` to `k3d tools` and add tool cmds there? - - e.g. `k3d tools import-images` - - let's you set tools container version - - `k3d tools --image k3d-tools:v2 import-images` -- add `k3d create --image-vol NAME` flag to re-use existing image volume - - will add `k3d.volumes.imageVolume.external: true` label to nodes - - should not be deleted with cluster - - possibly add `k3d create volume` and `k3d create network` to create external volumes/networks? - -### Prune Command - -- `k3d prune` to prune all dangling resources - - nodes, volumes, networks - -### Use Open Standards (OCI, CRI, ...) - -- [https://github.com/opencontainers/runtime-spec/blob/master/specs-go/config.go](https://github.com/opencontainers/runtime-spec/blob/master/specs-go/config.go) -- move node -> container translation out of runtime - -### Private registry - -- create a private registry to be used by k3d clusters - - similar to [https://github.com/rancher/k3d/pull/161](https://github.com/rancher/k3d/pull/161) -- add `k3d create registry` command to create external registry (maybe instead of flags as in PR #161?) - -### Syntactical shortcuts for k3d v1 backwards compatibility - -- e.g. `k3d create` -> `k3d create cluster k3s-default` - -### Unsorted Ideas - -- Integrate build tool (e.g. buildkit, buildah, ...) -- use `tools.go` to keep tools (like `golangci-lint` and `gox`) dependencies - - see e.g. [https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module](https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module) - - see e.g. [https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module](https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module) - -### Possible Enhancements - -- [!] remove/add nodes -> needs to remove line in `/var/lib/rancher/k3s/server/cred/node-passwd` for the deleted node