It's time to embark on the dockerless journey

,

It’s time to embark on the dockerless journey

Background Introduction

Docker, as one of the earliest and most widely used container runtimes, has become deeply ingrained in user workflows due to its intuitive interface and extensive features. In Kubernetes’ early days, it used the built-in dockershim component to integrate with Docker. However, as container technologies evolved, Kubernetes introduced the Container Runtime Interface (CRI) to standardize and expand container runtime support. This innovation enhanced Kubernetes’ compatibility with multiple container runtimes, offering users greater flexibility and choice.

The CRI enables Kubernetes system components to interact seamlessly with various container runtimes. This shift expanded Kubernetes’ scope beyond Docker and reduced its dependency on Docker and dockershim. Today, Kubernetes users can confidently adopt other robust runtimes like containerd and CRI-O, simplifying the Kubernetes architecture and improving operational efficiency.

Starting with Kubernetes version 1.24, official support for Docker (via dockershim) was removed. Similarly, Amazon EKS declared containerd as its sole supported runtime. Following this direction, our cluster’s container runtime was transitioned to containerd, replacing Docker CLI with nerdctl.

To better understand this shift, it’s important to introduce key container ecosystem components such as libcontainer, runc, containerd, CRI, and OCI. These components play critical roles in creating an efficient and flexible container runtime environment. By delving into these concepts, we can enhance operational expertise and lay a solid foundation for future application deployment and maintenance. For detailed information on nerdctl, please refer to the corresponding section.

Technical Background

OCI Standards (Open Container Initiative)

The Open Container Initiative (OCI), spearheaded by organizations like Docker and CoreOS (later acquired by Red Hat), establishes unified industry standards for container formats and runtimes. The OCI defines two primary specifications:

  • Image Specification (image-spec): Specifies the format and structure of container images.
  • Runtime Specification (runtime-spec): Governs the execution of container images. The most widely used low-level runtime today is runC, which includes libcontainer and supports operations like namespace and cgroup calls.

Main High-Level Container Runtimes

High-level container runtimes offer advanced functionality such as image management, unpacking, and API integration. The three most popular ones are:

  1. containerd: A CNCF project originally contributed by Docker.
  2. Podman: A Red Hat project, fully supported in RHEL 8.
  3. CRI-O: Another CNCF project designed to integrate seamlessly with Kubernetes.

Container Relationships

The diagram below illustrates the relationships within the container ecosystem, highlighting the interaction between various components.

OCI Container Relationships

Using Docker based on containerd as a container runtime can be categorized as a “high-high-level” runtime.

Containerd

Launched as an independent project in early 2016 and donated to CNCF in March 2017, containerd is a lightweight, robust, and portable container runtime that supports Linux and Windows. It manages complete container lifecycles, including image management, container execution, and networking (via CNI plugins).

Key Components of Containerd

  1. Runtime: Manages container lifecycle tasks, including creation, execution, and monitoring.
  2. Storage: Provides image storage and management capabilities.
  3. gRPC: Facilitates communication with upper-layer applications, offering container management APIs.
  4. Metrics: Offers performance and resource utilization metrics, primarily based on cgroups.
  5. Metadata: Stores essential information about images and containers.
  6. Tasks: Organizes containers as tasks, encompassing runtime states and process data.
  7. Events: Emits state change notifications to help applications monitor and react to container events.

Containerd Architecture

Comparison Between Docker and Containerd

Feature Containerd Docker
Architecture Focused on container lifecycle management. Comprehensive container platform.
Image Management Basic image transfer and storage features. Advanced image building and versioning.
Ecosystem Modular, integrates into larger ecosystems. Integrated tools and services.
Security Optimized for core functionality. Enhanced with image signing and keys.
Extensibility Lightweight, supports diverse scenarios. Plugin mechanism for customization.
Deployment Requires tools like Kubernetes. All-in-one solution for containers.
Community Backed by CNCF and cloud-native ecosystems. Docker-led, vibrant developer base.

CRI (Container Runtime Interface)

The CRI, introduced by Kubernetes, standardizes interactions between Kubernetes and container runtimes. Initially, Kubernetes relied on Docker’s API but later adopted the CRI to reduce dependencies and support emerging runtimes.

CRI defines two gRPC services:

  1. ImageService: Manages operations like pulling, listing, and deleting container images.
  2. RuntimeService: Handles Pod and container lifecycle management.

Evolution of CRI and Dockershim

In early Kubernetes versions, dockershim acted as an adapter, enabling Docker to conform to CRI. Over time, this approach proved suboptimal due to the complexity of Docker’s architecture. By Kubernetes 1.20, dockershim was deprecated, and by 1.24, it was fully removed.

Dockershim Interaction

Switching to containerd offers significant efficiency improvements by simplifying the runtime stack. Kubernetes now interacts directly with CRI-compatible runtimes like containerd, without relying on dockershim. This evolution aligns with Kubernetes’ focus on modularity and scalability.

Command Line Overview

Docker, ctr, nerdctl, and crictl are container management tools, each designed for specific purposes and use cases. Here’s a comparison:

Docker

  • Overview: Docker is a popular container management tool providing a comprehensive ecosystem for building, running, and managing containers.
  • Features: Supports image building, distribution (integrated with Docker Hub), and container lifecycle management (start, stop, delete, etc.).
  • Interface: Offers a powerful and user-friendly CLI and API, making it easy for developers and operators to adopt.
  • Key Characteristics:
    • Rich ecosystem (e.g., Docker Compose).
    • Supports advanced orchestration (e.g., Swarm).
    • Includes an additional daemon (dockerd) to collaborate with container runtimes.

ctr

  • Overview: ctr is the native CLI provided by containerd for direct interaction with it.
  • Features: Enables basic operations like pulling images, running containers, and managing image storage but is simplified for lower-level operations and debugging.
  • Interface: Unlike Docker, ctr targets containerd users without offering advanced APIs or ecosystem support.
  • Key Characteristics:
    • Suitable for developers and system integrators needing direct access to containerd.
    • Lacks advanced features like image building (focused solely on runtime management).

nerdctl

  • Overview: nerdctl is a CLI tool based on containerd that supports Docker-style commands.
  • Features: Provides a Docker-like user experience for running containers, building images, and supporting CNI plugins and volumes.
  • Interface: Closely mirrors Docker CLI, making migration from Docker to nerdctl straightforward.
  • Key Characteristics:
    • Operates without the Docker daemon, relying solely on containerd.
    • Integrates well with Kubernetes (when using containerd as the runtime) and supports Docker Compose-like functionality.

crictl

  • Overview: crictl is a CLI tool for the Container Runtime Interface (CRI), designed specifically for Kubernetes container management.
  • Features: Primarily interacts with CRI-compatible runtimes (e.g., containerd and CRI-O) for managing Pods, images, and containers.
  • Interface: Focuses on Kubernetes cluster debugging and management, with commands tailored to Kubernetes scenarios.
  • Key Characteristics:
    • Highly integrated with Kubernetes, often used to debug or manage Pod and container states.
    • Cannot build images, focusing exclusively on container and Pod lifecycle management.

Horizontal Comparison for Specific Use Cases

1. Image Management

Feature Docker CLI ctr crictl nerdctl
List local images docker images ctr i ls crictl images nerdctl images
Pull image docker pull ctr i pull crictl pull nerdctl pull
Push image docker push ctr i push N/A nerdctl push
Remove local image docker rmi ctr i rm crictl rmi nerdctl rmi
View image details docker inspect N/A crictl inspecti nerdctl inspect
Tag image docker tag ctr i tag N/A nerdctl tag
Export image docker export ctr i export N/A nerdctl save
Import image docker import ctr i import N/A nerdctl load
Build image docker build N/A N/A nerdctl build

2. Container Management

Feature Docker CLI ctr crictl nerdctl
List containers docker ps ctr c ls crictl ps nerdctl ps
Create container docker create ctr c create crictl create nerdctl create
Start container docker start ctr t start crictl start nerdctl start
Stop container docker stop N/A crictl stop nerdctl stop
Remove container docker rm ctr c rm crictl rm nerdctl rm
Inspect container docker inspect ctr c info crictl inspect nerdctl inspect
Attach container docker attach ctr t attach crictl attach nerdctl attach
Execute in container docker exec ctr t exec crictl exec nerdctl exec
View logs docker logs N/A crictl logs nerdctl logs
Monitor stats docker stats ctr t metrics crictl stats nerdctl stats

3. Pod Management

Feature Docker CLI ctr crictl nerdctl
List Pods N/A N/A crictl pods N/A
Inspect Pods N/A N/A crictl inspectp N/A
Run Pod N/A N/A crictl runp N/A
Stop Pod N/A N/A crictl stopp N/A
Remove Pod N/A N/A crictl rmp N/A

Summary

  • Docker: User-friendly and feature-rich, ideal for standalone container management and development environments.
  • ctr: Lightweight, low-level tool for direct interaction with containerd, mainly for debugging and backend management.
  • nerdctl: Docker CLI-compatible, great for users wanting Docker-like commands while directly utilizing containerd.
  • crictl: A Kubernetes-focused tool for managing Pods and containers via CRI; not suitable for general development.

nerdctl is a lightweight alternative to Docker based on containerd, while crictl specializes in Kubernetes CRI-based Pod management.

Guide to Using Containerd

nerdctl is a CLI tool compatible with the Docker CLI style, and it directly supports Docker Compose syntax. This significantly enhances efficiency when using containerd for local development, testing, or single-node container deployment.

1. Installation

You can download the appropriate package from the GitHub Release page and extract it to a directory in your PATH:

1
2
3
4
5
6
# If containerd is not installed, use the nerdctl-full-<VERSION>-linux-amd64.tar.gz package.
➜  ~ wget https://github.com/containerd/nerdctl/releases/download/v1.7.5/nerdctl-1.7.5-linux-amd64.tar.gz

➜  ~ mkdir -p /usr/local/containerd/bin/ && tar -zxvf nerdctl-1.7.5-linux-amd64.tar.gz nerdctl && mv nerdctl /usr/local/containerd/bin/
➜  ~ ln -s /usr/local/containerd/bin/nerdctl /usr/local/bin/nerdctl
➜  ~ nerdctl version

Sample output:

1
2
3
4
5
6
7
8
9
10
11
12
13
WARN[0000] unable to determine buildctl version: exec: "buildctl": executable file not found in $PATH 
Client:
 Version:    v1.7.5
 OS/Arch:    linux/amd64
 Git commit: cffed372371dcbea3dc9a646ce5a913fc1c09513

Server:
 containerd:
  Version: v2.0.0-beta.2-33-g96bf529cb
  GitCommit: 96bf529cbf55940ddb96bb8adc8be51b11922ebb
 runc:
  Version: 1.1.4
  GitCommit: v1.1.4-0-g5fd4c4d

After installation, you can start exploring the nerdctl CLI.

2. Command Overview

Container Management

Run
The nerdctl run command works similarly to docker run. For example:

1
➜  ~ nerdctl run -d -p 80:80 --name=nginx --restart=always nginx:alpine

Available options are consistent with Docker, such as -i, -t, --cpus, and --memory. Use nerdctl run --help for more options.

Exec
Execute commands within a container:

1
➜  ~ nerdctl exec -it <container_id> date

List Containers (ps)
List all containers (active containers by default):

1
➜  ~ nerdctl ps

For Kubernetes containers, use the -n k8s.io namespace flag:

1
➜  ~ nerdctl -n k8s.io ps

Inspect
Display detailed container information:

1
➜  ~ nerdctl -n k8s.io inspect <container_id>

Logs
Fetch container logs:

1
➜  ~ nerdctl -n k8s.io logs <container_id>

Supports options like -f, -t, -n, --since, and --until.

Stop and Remove
Stop and delete containers:

1
2
➜  ~ nerdctl -n k8s.io stop <container_id>
➜  ~ nerdctl -n k8s.io rm <container_id>

Use -f or --force for forced removal.

Image Management

List Images
View available images:

1
➜  ~ nerdctl -n k8s.io images

Pull Images
Pull an image from a registry:

1
➜  ~ nerdctl -n k8s.io pull <image_name>:<tag>

Other Commands
nerdctl supports other image-related commands like build, tag, push, and inspect. Use nerdctl image --help for a complete list.

This guide highlights key nerdctl commands, demonstrating its compatibility with Docker-like workflows while leveraging containerd’s efficiency.

CRICTL Usage Guide

crictl traces its roots to the Container Runtime Interface (CRI) introduced in Kubernetes version 1.5. CRI standardizes the interface between Kubernetes and container runtimes, enabling Kubernetes to interact seamlessly with various runtimes like Docker, containerd, and CRI-O.

As Kubernetes evolved, the demand for more features and better performance from container runtimes grew. CRI addressed these needs by providing a standardized interface for runtime implementation, simplifying the development of custom runtimes.

In this context, crictl was developed as a command-line tool to interact with CRI-compliant container runtimes. It facilitates managing containers and images, querying container states and metadata, and viewing logs. By streamlining runtime management and debugging, crictl promotes CRI adoption.

Below are the primary uses and features of crictl:

  1. Container Management: Start, stop, delete, and query container states.
  2. Image Management: Pull, push, delete, and inspect container images.
  3. Log Viewing: View stdout and stderr logs of containers.
  4. Container Information: Query detailed container information such as ID, name, status, and IP.
  5. Metadata Management: Annotate and tag containers.

crictl is a robust tool for managing containers and images within a runtime, debugging containerized applications, and integrating with orchestration systems like Kubernetes.

1. Installation

First, install crictl by downloading the appropriate binary from the cri-tools release page. Extract the binary to a PATH location:

1
2
3
4
5
6
7
8
➜  ~ VERSION="v1.22.0"
➜  ~ wget https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-$VERSION-linux-amd64.tar.gz
# Use a mirror for faster downloads if needed:
# wget https://download.fastgit.org/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-$VERSION-linux-amd64.tar.gz
➜  ~ tar zxvf crictl-$VERSION-linux-amd64.tar.gz -C /usr/local/bin
➜  ~ rm -f crictl-$VERSION-linux-amd64.tar.gz
➜  ~ crictl -v
crictl version v1.22.0

If the version command runs successfully, the crictl tool is installed.

2. Command Overview

After installation, configure the default settings in /etc/crictl.yaml by specifying the runtime and image endpoint addresses:

1
2
3
4
5
runtime-endpoint: unix:///var/run/containerd/containerd.sock
image-endpoint: unix:///var/run/containerd/containerd.sock
debug: false
pull-image-on-create: false
disable-pull-on-run: false

Once configured, you can start using crictl.

Pod Management

Use the crictl pods command to list the running pods on a node:

1
2
3
➜  ~ crictl pods
POD ID              CREATED             STATE               NAME                       NAMESPACE           ATTEMPT             RUNTIME
1f617ebf0524c       8 weeks ago         Ready               node-exporter-ggjwf        node-exporter       0                   (default)

Filter pods by name or label:

1
2
3
➜  ~ crictl pods --label app=node-exporter
POD ID              CREATED             STATE               NAME                  NAMESPACE           ATTEMPT             RUNTIME
1f617ebf0524c       8 weeks ago         Ready               node-exporter-ggjwf   node-exporter       0                   (default)

Image Management

List all images with crictl images:

1
2
3
➜  ~ crictl images
IMAGE                                      TAG                 IMAGE ID            SIZE
hub.cloud.ctripcorp.com/prom/node-exporter v1.4.0              d3e443c987ef4       11.5MB

View detailed image information with the -v flag:

1
2
3
4
➜  ~ crictl images -v
ID: sha256:d3e443c987ef405e1be101647873d86b5729c9c47bb1dd1ab59ccb24bc9e322c
RepoTags: docker.io/prom/node-exporter:v1.4.0
...

Container Management

List running containers using crictl ps:

1
2
3
➜  ~ crictl ps
CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID
e7f156d31942f       d3e443c987ef4       2 days ago          Running             main                2                   bf1704937991a

Filter containers by status with the -s option:

1
2
3
➜  ~ crictl ps -s Exited
CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID
22a0adacf702f       d3e443c987ef4       2 days ago          Exited              main                1                   bf1704937991a

Execute commands within a container:

1
2
➜  ~ crictl exec -it e7f156d31942f whoami
root

View container logs:

1
2
3
➜  ~ crictl logs e7f156d31942f
ts=2024-11-15T08:37:07.980Z caller=node_exporter.go:182 level=info msg="Starting node_exporter"
...

Follow logs in real time using -f or limit output with --tail N.

Monitor container resource usage with crictl stats:

1
2
3
➜  ~ crictl stats e7f156d31942f
CONTAINER           NAME                CPU %               MEM                 DISK                INODES
e7f156d31942f       main                0.00                37.13MB             0B                  18

For more features, refer to kubernetes-sigs/cri-tools.

  1. Kubernetes Container Runtimes
  2. Zhihu Article
  3. Qikqiak Guide
  4. CRI API Reference
  5. OpenNaru Article
  6. Nerdctl Presentation
  7. Alibaba Cloud Blog