From 4cbf88a44311e37738c4779c3233167b3fec30aa Mon Sep 17 00:00:00 2001 From: iwilltry42 Date: Wed, 15 Jan 2020 08:47:45 +0100 Subject: [PATCH] update/structure thoughts --- README.md | 3 +- thoughts.md | 291 +++++++++++++++++++++++++--------------------------- 2 files changed, 144 insertions(+), 150 deletions(-) diff --git a/README.md b/README.md index 06f5afec..77d81317 100644 --- a/README.md +++ b/README.md @@ -56,11 +56,12 @@ Check out the [examples here](docs/examples.md). ## What now? -Find more details under the following Links: +### More Information - [Further documentation](docs/documentation.md) - [Usage examples](docs/examples.md) - [Frequently asked questions and nice-to-know facts](docs/faq.md) +- [Design/Architecture Thoughts, Plans, Unsorted Ideas and more](thoughts.md) ### Connect diff --git a/thoughts.md b/thoughts.md index cf36b2a2..ed47350e 100644 --- a/thoughts.md +++ b/thoughts.md @@ -1,138 +1,6 @@ # Thoughts -## commands - -### `create` - -```shell -k3d - |- create - | |- cluster [NAME ] [flags] - | |- node [NAME ] [flags] - | - |- delete - | |- cluster [NAME ] [flags] - | |- node [NAME ] [flags] - |- get - | |- cluster - | |- node - |- start - | |- cluster - | |- node - |- stop - | |- cluster - | |- node -``` - -## Overview - -- `cmd/`: everything around the CLI of k3d = human interface, printed output (e.g. list of clusters) -- `pkg/`: everything else, can be used as a module from other Go projects - - `cluster/`: everything around managing cluster components - - `runtimes/`: translate k3d types (node, cluster, etc.) to container runtime specific types and manage them - - `types/`: collection of types (structs) and constants used by k3d - - `util/`: utilities, that could be used for everything, not directly related to the project - -## k3d <-> runtime - -k3d _should_ work with more than one runtime, if we can implement the Runtime interface for it. -Here's how k3d types should translate to a runtime type: - -- `cluster` = set of _containers_ running in the same _network_, maybe mounting the same _volume(s)_ -- `node` = _container_ with _exposed ports_ and _volume mounts_ - -### docker - -#### node -> container - -`container = "github.com/docker/docker/api/types/container"` -`network = "github.com/docker/docker/api/types/network"` - -- Name -> container.Hostname = node.Name -- Role -> container.Labels["k3d.role"] = node.Role -- Image -> container.Image = node.Image -- Volumes -> container.HostConfig.PortBindings -- Env -> -- Args -> -- Ports -> -- Restart -> -- Labels -> container.Labels - -## expose ports / volumes => DONE - -- `--port [host:]port[:containerPort][/protocol][@group_identifier[[index] | @node_identifier]` - - Examples: - - `--port 0.0.0.0:8080:8081/tcp@workers` -> whole group - - `--port 80@workers[0]` -> single instance of group by list index - - `--port 80@workers[0,2-3]` -> multiple instances of a group by index lists and ranges - - `--port 80@k3d-test-worker-0` -> single instance by specific node identifier - - `--port 80@k3d-test-master-0@workers[1-5]` -> multiple instances by combination of node and group identifiers - -- analogous for volumes - -## multi master setup => WIP - -- if `--masters` > 1 deploy a load-balancer in front of them as an extra container - - consider that in the kubeconfig file and `--tls-san` - - make this the default, but provide a `--no-lb` flag - -## Store additional created stuff in labels => DONE - -- when creating a cluster, usually, you also create a new docker network (and maybe other resources) - - store a reference to those in the container labels of cluster nodes - - when deleting the cluster, parse the labels, deduplicate the results and delete the additional resources - - DONE for network - - new labels `k3d.cluster.network=` and `k3d.cluster.network.external=` (determine whether to try to delete it when you delete a cluster, since network may have been created manually) - - -# Comparison to k3d v1 - -- k3d - - check-tools - - shell - - --name - - --command - - --shell - - auto, bash, zsh - - create -> `k3d create cluster CLUSTERNAME` - - --name -> y - - --volume -> y - - --port -> y - - --api-port -> y - - --wait - - --image -> y - - --server-arg -> y - - --agent-arg -> y - - --env - - --workers -> y - - --auto-restart - - (add-node) -> `k3d create node NODENAME` - - --role - - --name - - --count - - --image - - --arg - - --env - - --volume - - --k3s - - --k3s-secret - - --k3s-token - - delete -> `k3d delete cluster CLUSTERNAME` - - --name - - --all - - stop -> `k3d stop cluster CLUSTERNAME` - - --name - - --all - - start -> `k3d start cluster CLUSTERNAME` - - --name - - --all - - list - - get-kubeconfig -> `k3d get kubeconfig CLUSTERNAME` - - --name -> y - - --all - - import-images -> `k3d load image [--cluster CLUSTERNAME] [--keep] IMAGES` - - --name -> y - - --no-remove -> y +## Command Tree - k3d - create @@ -180,28 +48,92 @@ Here's how k3d types should translate to a runtime type: - --all - node NAME -## tools +## Feature Comparison to k3d v1 -- maybe rename `k3d load` to `k3d tools` and add tool cmds there? - - e.g. `k3d tools import-images` - - let's you set tools container version - - `k3d tools --image k3d-tools:v2 import-images` -- add `k3d create --image-vol NAME` flag to re-use existing image volume - - will add `k3d.volumes.imagevolume.external: true` label to nodes - - should not be deleted with cluster - - possibly add `k3d create volume` and `k3d create network` to create external networks? +### v1.x feature -> implemented in v3 -## extra commands +- k3d + - check-tools + - shell + - --name + - --command + - --shell + - auto, bash, zsh + - create -> `k3d create cluster CLUSTERNAME` + - --name -> y + - --volume -> y + - --port -> y + - --api-port -> y + - --wait + - --image -> y + - --server-arg -> y + - --agent-arg -> y + - --env + - --workers -> y + - --auto-restart + - (add-node) -> `k3d create node NODENAME` + - --role + - --name + - --count + - --image + - --arg + - --env + - --volume + - --k3s + - --k3s-secret + - --k3s-token + - delete -> `k3d delete cluster CLUSTERNAME` + - --name + - --all + - stop -> `k3d stop cluster CLUSTERNAME` + - --name + - --all + - start -> `k3d start cluster CLUSTERNAME` + - --name + - --all + - list + - get-kubeconfig -> `k3d get kubeconfig CLUSTERNAME` + - --name -> y + - --all + - import-images -> `k3d load image [--cluster CLUSTERNAME] [--keep] IMAGES` + - --name -> y + - --no-remove -> y -- `k3d prune` to prune all dangling resources - - nodes, volumes, networks +## Repository/Package Overview -## use OCI +- `cmd/`: everything around the CLI of k3d = human interface, printed output (e.g. list of clusters) +- `pkg/`: everything else, can be used as a module from other Go projects + - `cluster/`: everything around managing cluster components + - `runtimes/`: translate k3d types (node, cluster, etc.) to container runtime specific types and manage them + - `types/`: collection of types (structs) and constants used by k3d + - `util/`: utilities, that could be used for everything, not directly related to the project -- [https://github.com/opencontainers/runtime-spec/blob/master/specs-go/config.go](https://github.com/opencontainers/runtime-spec/blob/master/specs-go/config.go) -- move node -> container translation out of runtime +## k3d types <-> runtime translation + +k3d _should_ work with more than one runtime, if we can implement the Runtime interface for it. +Here's how k3d types should translate to a runtime type: + +- `cluster` = set of _containers_ running in the same _network_, maybe mounting the same _volume(s)_ +- `node` = _container_ with _exposed ports_ and _volume mounts_ + +### Docker + +#### Node to Container translation + +`container = "github.com/docker/docker/api/types/container"` +`network = "github.com/docker/docker/api/types/network"` + +- Name -> container.Hostname = node.Name +- Role -> container.Labels["k3d.role"] = node.Role +- Image -> container.Image = node.Image +- Volumes -> container.HostConfig.PortBindings +- Env -> +- Args -> +- Ports -> +- Restart -> +- Labels -> container.Labels -## node configuration comparison +## Node Configuration - master node(s) - ENV @@ -244,4 +176,65 @@ Here's how k3d types should translate to a runtime type: ## Features +## [DONE] Node Filters + +- `--port [host:]port[:containerPort][/protocol][@group_identifier[[index] | @node_identifier]` + - Examples: + - `--port 0.0.0.0:8080:8081/tcp@workers` -> whole group + - `--port 80@workers[0]` -> single instance of group by list index + - `--port 80@workers[0,2-3]` -> multiple instances of a group by index lists and ranges + - `--port 80@k3d-test-worker-0` -> single instance by specific node identifier + - `--port 80@k3d-test-master-0@workers[1-5]` -> multiple instances by combination of node and group identifiers + +- analogous for volumes + +## [WIP] Multi-Master Setup + +- if `--masters` > 1 deploy a load-balancer in front of them as an extra container + - consider that in the kubeconfig file and `--tls-san` + - make this the default, but provide a `--no-lb` flag + +## [DONE] Keep State in Docker Labels + +- when creating a cluster, usually, you also create a new docker network (and maybe other resources) + - store a reference to those in the container labels of cluster nodes + - when deleting the cluster, parse the labels, deduplicate the results and delete the additional resources + - DONE for network + - new labels `k3d.cluster.network=` and `k3d.cluster.network.external=` (determine whether to try to delete it when you delete a cluster, since network may have been created manually) + +## Bonus Ideas + +### Tools + +- maybe rename `k3d load` to `k3d tools` and add tool cmds there? + - e.g. `k3d tools import-images` + - let's you set tools container version + - `k3d tools --image k3d-tools:v2 import-images` +- add `k3d create --image-vol NAME` flag to re-use existing image volume + - will add `k3d.volumes.imagevolume.external: true` label to nodes + - should not be deleted with cluster + - possibly add `k3d create volume` and `k3d create network` to create external volumes/networks? + +### Prune Command + +- `k3d prune` to prune all dangling resources + - nodes, volumes, networks + +### Use Open Standards (OCI, CRI, ...) + +- [https://github.com/opencontainers/runtime-spec/blob/master/specs-go/config.go](https://github.com/opencontainers/runtime-spec/blob/master/specs-go/config.go) +- move node -> container translation out of runtime + +### Private registry + +- create a private registry to be used by k3d clusters + - similar to [https://github.com/rancher/k3d/pull/161](https://github.com/rancher/k3d/pull/161) +- add `k3d create registry` command to create external registry (maybe instead of flags as in PR #161?) + +### Unsorted Ideas + +- Integrate build tool (e.g. buildkit, buildah, ...) + +### Required Enhancements + - remove/add nodes -> needs to remove line in `/var/lib/rancher/k3s/server/cred/node-passwd` for the deleted node