mirror of https://github.com/k3d-io/k3d
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
5.8 KiB
5.8 KiB
Thoughts
Repository/Package Overview
cmd/
: everything around the CLI of k3d = human interface, printed output (e.g. list of clusters)pkg/
: everything else, can be used as a module from other Go projectscluster/
: everything around managing cluster componentsruntimes/
: translate k3d types (node, cluster, etc.) to container runtime specific types and manage themtypes/
: collection of types (structs) and constants used by k3dutil/
: utilities, that could be used for everything, not directly related to the project
k3d types <-> runtime translation
k3d should work with more than one runtime, if we can implement the Runtime interface for it. Here's how k3d types should translate to a runtime type:
cluster
= set of containers running in the same network, maybe mounting the same volume(s)node
= container with exposed ports and volume mounts
Docker
Node to Container translation
container = "github.com/docker/docker/api/types/container"
network = "github.com/docker/docker/api/types/network"
- Name -> container.Hostname = node.Name
- Role -> container.Labels["k3d.role"] = node.Role
- Image -> container.Image = node.Image
- Volumes -> container.HostConfig.PortBindings
- Env ->
- Args ->
- Ports ->
- Restart ->
- Labels -> container.Labels
Node Configuration
- server node(s)
- ENV
K3S_CLUSTER_INIT
- if num_servers > 1 && no external datastore configured
K3S_KUBECONFIG_OUTPUT
- k3d default ->
/output/kubeconfig.yaml
- k3d default ->
- CMD/ARGS
--https-listen-port
- can/should be left default (unset = 6443), since we handle it via port mapping
--tls-san=<some-ip-or-hostname>
- get from
--api-port
k3d flag and/or from docker machine
- get from
- Runtime Configuration
- nothing special
- ENV
- all nodes
- ENV
K3S_TOKEN
for node authentication
- CMD/ARGS
- nothing special
- Runtime Configuration
- Volumes
- shared image volume
- cluster-specific (create cluster) or inherit from existing (create node)
- tmpfs for k3s to work properly
/run
/var/run
- shared image volume
- Capabilities/Security Context
privileged
- Network
- cluster network or external/inherited
- Volumes
- ENV
- agent nodes
- ENV
K3S_URL
to connect to server node- server hostname + port (6443)
- cluster-specific or inherited
- CMD/ARGS
- nothing special
- Runtime Configuration
- nothing special
- ENV
Features
[DONE] Node Filters
-
--port [host:]port[:containerPort][/protocol][@group_identifier[[index] | @node_identifier]
- Examples:
--port 0.0.0.0:8080:8081/tcp@agents
-> whole group--port 80@agents[0]
-> single instance of group by list index--port 80@agents[0,2-3]
-> multiple instances of a group by index lists and ranges--port 80@k3d-test-agent-0
-> single instance by specific node identifier--port 80@k3d-test-server-0@agents[1-5]
-> multiple instances by combination of node and group identifiers
- Examples:
-
analogous for volumes
[WIP] Multi-Server Setup
- to make this possible, we always deploy a load-balancer (nginx) in front of the server nodes as an extra container
- consider that in the kubeconfig file and
--tls-san
- consider that in the kubeconfig file and
Variants
- embedded datastore (dqlite)
- if
--servers
> 1 deploy a load-balancer in front of them as an extra container
- if
- external datastore
[DONE] Keep State in Docker Labels
- when creating a cluster, usually, you also create a new docker network (and maybe other resources)
- store a reference to those in the container labels of cluster nodes
- when deleting the cluster, parse the labels, deduplicate the results and delete the additional resources
- DONE for network
- new labels
k3d.cluster.network=<ID>
andk3d.cluster.network.external=<true|false>
(determine whether to try to delete it when you delete a cluster, since network may have been created manually)
- new labels
Bonus Ideas
Tools
- maybe rename
k3d load
tok3d tools
and add tool cmds there?- e.g.
k3d tools import-images
- let's you set tools container version
k3d tools --image k3d-tools:v2 import-images
- e.g.
- add
k3d create --image-vol NAME
flag to re-use existing image volume- will add
k3d.volumes.imageVolume.external: true
label to nodes- should not be deleted with cluster
- possibly add
k3d create volume
andk3d create network
to create external volumes/networks?
- will add
Prune Command
k3d prune
to prune all dangling resources- nodes, volumes, networks
Use Open Standards (OCI, CRI, ...)
- https://github.com/opencontainers/runtime-spec/blob/master/specs-go/config.go
- move node -> container translation out of runtime
Private registry
- create a private registry to be used by k3d clusters
- similar to https://github.com/rancher/k3d/pull/161
- add
k3d create registry
command to create external registry (maybe instead of flags as in PR #161?)
Syntactical shortcuts for k3d v1 backwards compatibility
- e.g.
k3d create
->k3d create cluster k3s-default
Unsorted Ideas
- Integrate build tool (e.g. buildkit, buildah, ...)
- use
tools.go
to keep tools (likegolangci-lint
andgox
) dependencies
Possible Enhancements
- [!] remove/add nodes -> needs to remove line in
/var/lib/rancher/k3s/server/cred/node-passwd
for the deleted node