mirror of https://github.com/k3d-io/k3d
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
9.7 KiB
9.7 KiB
Thoughts
Command Tree
- k3d
- create
- cluster NAME
- --api-port
- --datastore-cafile
- --datastore-certfile
- --datastore-endpoint
- --datastore-keyfile
- --datastore-network
- --image
- --k3s-agent-arg
- --k3s-server-arg
- --lb-port
- --masters
- --network
- --no-lb
- --port
- --secret
- --volume
- --workers
- node NAME
- --cluster
- --image
- --replicas
- --role
- cluster NAME
- delete
- cluster NAME
- --all
- node NAME
- --all
- cluster NAME
- get
- cluster NAME
- --no-headers
- node NAME
- --no-headers
- kubeconfig NAME
- --output
- cluster NAME
- start
- cluster NAME
- --all
- node NAME
- cluster NAME
- stop
- cluster NAME
- --all
- node NAME
- cluster NAME
- create
Feature Comparison to k3d v1
v1.x feature -> implementation in v3
- k3d
- check-tools -> won't do
- shell -> planned:
k3d shell CLUSTER
- --name -> planned: drop (now as arg)
- --command -> planned: keep
- --shell -> planned: keep (or second arg)
- auto, bash, zsh
- create ->
k3d create cluster CLUSTERNAME
- --name -> dropped, implemented via arg
- --volume -> implemented
- --port -> implemented
- --port-auto-offset -> TBD
- --api-port -> implemented
- --wait -> implemented
- --image -> implemented
- --server-arg -> implemented as
--k3s-server-arg
- --agent-arg -> implemented as
--k3s-agent-arg
- --env -> planned
- --label -> planned
- --workers -> implemented
- --auto-restart -> planned
- --enable-registry -> planned (possible consolidation into less registry-related commands?)
- --registry-name -> TBD
- --registry-port -> TBD
- --registry-volume -> TBD
- --registries-file -> TBD
- --enable-registry-cache -> TBD
- (add-node) ->
k3d create node NODENAME
- --role -> implemented
- --name -> dropped, implemented as arg
- --count -> implemented as
--replicas
- --image -> implemented
- --arg -> planned
- --env -> planned
- --volume -> planned
- --k3s -> TBD
- --k3s-secret -> TBD
- --k3s-token -> TBD
- delete ->
k3d delete cluster CLUSTERNAME
- --name -> dropped, implemented as arg
- --all -> implemented
- --prune -> TBD
- --keep-registry-volume -> TBD
- stop ->
k3d stop cluster CLUSTERNAME
- --name -> dropped, implemented as arg
- --all -> implemented
- start ->
k3d start cluster CLUSTERNAME
- --name -> dropped, implemented as arg
- --all -> implemented
- list -> dropped, implemented as
k3d get clusters
- get-kubeconfig ->
k3d get kubeconfig CLUSTERNAME
- --name -> dropped, implemented as arg
- --all -> planned
- --overwrite -> planned
- import-images ->
k3d load image [--cluster CLUSTERNAME] [--keep] IMAGES
- --name -> implemented as
--cluster
- --no-remove -> implemented as
--keep
- --name -> implemented as
Repository/Package Overview
cmd/
: everything around the CLI of k3d = human interface, printed output (e.g. list of clusters)pkg/
: everything else, can be used as a module from other Go projectscluster/
: everything around managing cluster componentsruntimes/
: translate k3d types (node, cluster, etc.) to container runtime specific types and manage themtypes/
: collection of types (structs) and constants used by k3dutil/
: utilities, that could be used for everything, not directly related to the project
k3d types <-> runtime translation
k3d should work with more than one runtime, if we can implement the Runtime interface for it. Here's how k3d types should translate to a runtime type:
cluster
= set of containers running in the same network, maybe mounting the same volume(s)node
= container with exposed ports and volume mounts
Docker
Node to Container translation
container = "github.com/docker/docker/api/types/container"
network = "github.com/docker/docker/api/types/network"
- Name -> container.Hostname = node.Name
- Role -> container.Labels["k3d.role"] = node.Role
- Image -> container.Image = node.Image
- Volumes -> container.HostConfig.PortBindings
- Env ->
- Args ->
- Ports ->
- Restart ->
- Labels -> container.Labels
Node Configuration
- master node(s)
- ENV
K3S_CLUSTER_INIT
- if num_masters > 1 && no external datastore configured
K3S_KUBECONFIG_OUTPUT
- k3d default ->
/output/kubeconfig.yaml
- k3d default ->
- CMD/ARGS
--https-listen-port
- can/should be left default (unset = 6443), since we handle it via port mapping
--tls-san=<some-ip-or-hostname>
- get from
--api-port
k3d flag and/or from docker machine
- get from
- Runtime Configuration
- nothing special
- ENV
- all nodes
- ENV
K3S_TOKEN
for node authentication
- CMD/ARGS
- nothing special
- Runtime Configuration
- Volumes
- shared image volume
- cluster-specific (create cluster) or inherit from existing (create node)
- tmpfs for k3s to work properly
/run
/var/run
- shared image volume
- Capabilities/Security Context
privileged
- Network
- cluster network or external/inherited
- Volumes
- ENV
- worker nodes
- ENV
K3S_URL
to connect to master node- server hostname + port (6443)
- cluster-specific or inherited
- CMD/ARGS
- nothing special
- Runtime Configuration
- nothing special
- ENV
Features
[DONE] Node Filters
-
--port [host:]port[:containerPort][/protocol][@group_identifier[[index] | @node_identifier]
- Examples:
--port 0.0.0.0:8080:8081/tcp@workers
-> whole group--port 80@workers[0]
-> single instance of group by list index--port 80@workers[0,2-3]
-> multiple instances of a group by index lists and ranges--port 80@k3d-test-worker-0
-> single instance by specific node identifier--port 80@k3d-test-master-0@workers[1-5]
-> multiple instances by combination of node and group identifiers
- Examples:
-
analogous for volumes
[WIP] Multi-Master Setup
- if
--masters
> 1 deploy a load-balancer in front of them as an extra container- consider that in the kubeconfig file and
--tls-san
- make this the default, but provide a
--no-lb
flag
- consider that in the kubeconfig file and
[DONE] Keep State in Docker Labels
- when creating a cluster, usually, you also create a new docker network (and maybe other resources)
- store a reference to those in the container labels of cluster nodes
- when deleting the cluster, parse the labels, deduplicate the results and delete the additional resources
- DONE for network
- new labels
k3d.cluster.network=<ID>
andk3d.cluster.network.external=<true|false>
(determine whether to try to delete it when you delete a cluster, since network may have been created manually)
- new labels
Bonus Ideas
Tools
- maybe rename
k3d load
tok3d tools
and add tool cmds there?- e.g.
k3d tools import-images
- let's you set tools container version
k3d tools --image k3d-tools:v2 import-images
- e.g.
- add
k3d create --image-vol NAME
flag to re-use existing image volume- will add
k3d.volumes.imagevolume.external: true
label to nodes- should not be deleted with cluster
- possibly add
k3d create volume
andk3d create network
to create external volumes/networks?
- will add
Prune Command
k3d prune
to prune all dangling resources- nodes, volumes, networks
Use Open Standards (OCI, CRI, ...)
- https://github.com/opencontainers/runtime-spec/blob/master/specs-go/config.go
- move node -> container translation out of runtime
Private registry
- create a private registry to be used by k3d clusters
- similar to https://github.com/rancher/k3d/pull/161
- add
k3d create registry
command to create external registry (maybe instead of flags as in PR #161?)
Syntactical shortcuts for k3d v1 backwards compatibility
- e.g.
k3d create
->k3d create cluster k3s-default
Unsorted Ideas
- Integrate build tool (e.g. buildkit, buildah, ...)
- use
tools.go
to keep tools (likegolangci-lint
andgox
) dependencies
Possible Enhancements
- [!] remove/add nodes -> needs to remove line in
/var/lib/rancher/k3s/server/cred/node-passwd
for the deleted node - directly use cluster struct in
createClusterCmd
filled by flag values