containerd 如何拉取镜像

背景

Dockershim从Kubernetes 1.20版本开始废弃,计划在Kubernetes 1.24版本移除,所以目前1.24+的ACK集群只能containerd作为运行时,但是docker build构建的镜像是可以正常使用的, docker build 创建的镜像适用于任何 CRI 实现, 参考 Dockershim Removal FAQ

本文介绍一下containerd运行的节点上如何拉取公共镜像镜像,私有仓库镜像,便于排查containerd节点的免密或者自建镜像仓库的镜像

工具介绍

提示 很多时候我们都可以通过–help来查看工具都提供了哪些功能,因为版本之间直接可能存在兼容性问题

本文对应的软件版本信息(基于阿里云的ACK)

ACK版本 containerd版本
v1.24.6-aliyun.1 containerd 1.6.20

containerd 支持的客户端工具有很多, 比如 ctr、nerdctl、crictl、能调用cri接口的如kubelet等,其中是ctr、nerdctl是不支持CRI Plugin,比如镜像加速, 但是可以通过--hosts-dir的方式指定

本文就通过介绍ctr和crictl, ACK 中默认安装的。

镜像拉取

提示 很多时候我们都可以通过–help来查看工具都提供了哪些功能,因为版本之间直接可能存在兼容性问题

ctr 工具拉取

ctr 拉取镜像需要加上 镜像的地址/名称空间/仓库名称:[tag] , 比如: docker.io/library/busybox:latest

contained 是区分名称空间的 kubelet拉取的镜像k8s.io名称空间, 如果一个服务安装了docker,那么docker 拉取的镜像保存到moby名称空间, 不加名称空间保存在默认的名称空间default, 所以无论是拉取还是运行服务需要使用–namespace 指定, 不写默认是default

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
$ ctr --namespace k8s.io images pull --help
NAME:
ctr images pull - pull an image from a remote

USAGE:
ctr images pull [command options] [flags] <ref>

OPTIONS:
--skip-verify, -k skip SSL certificate validation
--plain-http allow connections using plain HTTP
--user value, -u value user[:password] Registry user and password
--refresh value refresh token for authorization server
--hosts-dir value Custom hosts configuration directory
--tlscacert value path to TLS root CA
--tlscert value path to TLS client certificate
--tlskey value path to TLS client key
--http-dump dump all HTTP request/responses when interacting with container registry
--http-trace enable HTTP tracing for registry interactions
--snapshotter value snapshotter name. Empty value stands for the default value. [$CONTAINERD_SNAPSHOTTER]
--label value labels to attach to the image
--platform value Pull content from a specific platform
--all-platforms pull content and metadata from all platforms
--all-metadata Pull metadata for all platforms
--print-chainid Print the resulting image's chain ID
--max-concurrent-downloads value Set the max concurrent downloads for each pull (default: 0)

拉取docker官方镜像

1
2
3
4
5
6
7
8
9
$ ctr --namespace k8s.io images pull docker.io/library/busybox:latest
docker.io/library/busybox:latest: resolved |++++++++++++++++++++++++++++++++++++++|
index-sha256:6e494387c901caf429c1bf77bd92fb82b33a68c0e19f6d1aa6a3ac8d27a7049d: done |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:1b0a26bd07a3d17473d8d8468bea84015e27f87124b283b91d781bce13f61370: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:71d064a1ac7d46bdcac82ea768aba4ebbe2a05ccbd3a4a82174c18cf51b67ab7: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:b539af69bc01c6c1c1eae5474a94b0abaab36b93c165c0cf30b7a0ab294135a3: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 4.8 s total: 1.0 Mi (214.2 KiB/s)
unpacking linux/amd64 sha256:6e494387c901caf429c1bf77bd92fb82b33a68c0e19f6d1aa6a3ac8d27a7049d...
done: 85.942512ms

拉取非安全的镜像

http方式
1
2
3
$ ctr --namespace k8s.io images pull  192.168.66.42:5000/busybox:latest
INFO[0000] trying next host error="failed to do request: Head \"https://192.168.66.42:5000/v2/busybox/manifests/latest\": http: server gave HTTP response to HTTPS client" host="192.168.66.42:5000"
ctr: failed to resolve reference "192.168.66.42:5000/busybox:latest": failed to do request: Head "https://192.168.66.42:5000/v2/busybox/manifests/latest": http: server gave HTTP response to HTTPS client

上面的报错可以看到, 默认ctr是不支持http,即明文拉取,可以通过--plain-http 参数声明明文拉取

1
2
3
4
5
6
7
8
$ ctr --namespace k8s.io images pull  --plain-http 192.168.66.42:5000/busybox:latest
192.168.66.42:5000/busybox:latest: resolved |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:62ffc2ed7554e4c6d360bce40bbcf196573dd27c4ce080641a2c59867e732dee: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:5cc84ad355aaa64f46ea9c7bbcc319a9d808ab15088a27209c9e70ef86e5a2aa: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:beae173ccac6ad749f76713cf4440fe3d21d1043fe616dfbe30775815d1d0f6a: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 1.0 s total: 756.6 (756.2 KiB/s)
unpacking linux/amd64 sha256:62ffc2ed7554e4c6d360bce40bbcf196573dd27c4ce080641a2c59867e732dee...
done: 53.354745ms
自签证书
1
2
3
$ ctr --namespace k8s.io images pull harbor.test-cri.com/test-cri/busybox:latest
INFO[0000] trying next host error="failed to do request: Head \"https://harbor.test-cri.com/v2/test-cri/busybox/manifests/latest\": x509: certificate signed by unknown authority" host=harbor.test-cri.com
ctr: failed to resolve reference "harbor.test-cri.com/test-cri/busybox:latest": failed to do request: Head "https://harbor.test-cri.com/v2/test-cri/busybox/manifests/latest": x509: certificate signed by unknown authority

上面的报错可以看到,自签证书会报错,可以通过 --skip-verify 来跳过证书校验

1
2
3
4
5
6
7
8
$ ctr --namespace k8s.io images pull --skip-verify harbor.test-cri.com/test-cri/busybox:latest
harbor.test-cri.com/test-cri/busybox:latest: resolved |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:62ffc2ed7554e4c6d360bce40bbcf196573dd27c4ce080641a2c59867e732dee: exists |++++++++++++++++++++++++++++++++++++++|
config-sha256:beae173ccac6ad749f76713cf4440fe3d21d1043fe616dfbe30775815d1d0f6a: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:5cc84ad355aaa64f46ea9c7bbcc319a9d808ab15088a27209c9e70ef86e5a2aa: exists |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.3 s total: 0.0 B (0.0 B/s)
unpacking linux/amd64 sha256:62ffc2ed7554e4c6d360bce40bbcf196573dd27c4ce080641a2c59867e732dee...
done: 7.565545ms

拉取私有仓库镜像

1
2
3
4
5
$ ctr --namespace k8s.io images pull registry.cn-hangzhou.aliyuncs.com/test-cri/busybox:latest
registry.cn-hangzhou.aliyuncs.com/test-cri/busybox:latest: resolving |--------------------------------------|
elapsed: 0.5 s total: 0.0 B (0.0 B/s)
INFO[0000] trying next host error="pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed" host=registry.cn-hangzhou.aliyuncs.com
ctr: failed to resolve reference "registry.cn-hangzhou.aliyuncs.com/test-cri/busybox:latest": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

上面的报错可以看到,拉私有镜像会报错认证失败,需要通过--user 参数加上账号密码,其中username和password需要替换真实用户名和密码

1
2
3
4
5
6
7
8
$ ctr --namespace k8s.io images pull  --user $username:$password  registry.cn-hangzhou.aliyuncs.com/test-cri/busybox:latest
registry.cn-hangzhou.aliyuncs.com/test-cri/busybox:latest: resolved |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:62ffc2ed7554e4c6d360bce40bbcf196573dd27c4ce080641a2c59867e732dee: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:beae173ccac6ad749f76713cf4440fe3d21d1043fe616dfbe30775815d1d0f6a: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:5cc84ad355aaa64f46ea9c7bbcc319a9d808ab15088a27209c9e70ef86e5a2aa: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.9 s total: 0.0 B (0.0 B/s)
unpacking linux/amd64 sha256:62ffc2ed7554e4c6d360bce40bbcf196573dd27c4ce080641a2c59867e732dee...
done: 12.415462ms

查看镜像

1
$ ctr --namespace k8s.io images ls

删除镜像

1
2
$ ctr --namespace k8s.io images delete registry.cn-hangzhou.aliyuncs.com/test-cri/busybox:latest
registry.cn-hangzhou.aliyuncs.com/test-cri/busybox:latest

crictl 工具拉取镜像

crictl 工具安装官方文档Container Runtime Interface (CRI) CLI

crictl 拉取镜像docker官方镜像的话 可以省略镜像的地址和名称空间,直接使用仓库名称:[tag] , 比如: busybox:latest

crictl 和kubelet 拉取和运行容器默认都是在k8s.io的名称空间, 不需要手动指定

1
2
3
4
5
6
7
8
9
10
11
12
13
14
crictl  pull --help
NAME:
crictl pull - Pull an image from a registry

USAGE:
crictl pull [command options] NAME[:TAG|@DIGEST]

OPTIONS:
--annotation value, -a value [ --annotation value, -a value ] Annotation to be set on the pulled image
--auth AUTH_STRING Use AUTH_STRING for accessing the registry. AUTH_STRING is a base64 encoded 'USERNAME[:PASSWORD]' [$CRICTL_AUTH]
--creds USERNAME[:PASSWORD] Use USERNAME[:PASSWORD] for accessing the registry [$CRICTL_CREDS]
--pod-config pod-config.[json|yaml] Use pod-config.[json|yaml] to override the the pull c
--username USERNAME, -u USERNAME Use USERNAME for accessing the registry. The password will be requested on the command line
--help, -h show help

拉取docker官方镜像

1
2
$ crictl  pull busybox:latest
Image is up to date for sha256:beae173ccac6ad749f76713cf4440fe3d21d1043fe616dfbe30775815d1d0f6a

拉取非安全配置镜像

crictl 命令行默认是不支持的, 需要调整config.toml, 默认是在 /etc/containerd/config.toml调整在下方

拉取私有镜像

1
2
3
$ crictl  pull registry.cn-hangzhou.aliyuncs.com/test-cri/busybox:latest
E0619 16:44:57.810567 1900073 remote_image.go:171] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"registry.cn-hangzhou.aliyuncs.com/test-cri/busybox:latest\": failed to resolve reference \"registry.cn-hangzhou.aliyuncs.com/test-cri/busybox:latest\": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed" image="registry.cn-hangzhou.aliyuncs.com/test-cri/busybox:latest"
FATA[0000] pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "registry.cn-hangzhou.aliyuncs.com/test-cri/busybox:latest": failed to resolve reference "registry.cn-hangzhou.aliyuncs.com/test-cri/busybox:latest": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

上面的报错可以看到,拉私有镜像会报错认证失败,需要通过–creds 参数加上账号密码,其中username和password需要替换真实用户名和密码

1
2
$ crictl  pull --creds  $username:$password registry.cn-hangzhou.aliyuncs.com/test-cri/busybox:latest
Image is up to date for sha256:beae173ccac6ad749f76713cf4440fe3d21d1043fe616dfbe30775815d1d0f6a

查看镜像

1
$ crictl  images

删除镜像

1
2
$ crictl rmi registry.cn-hangzhou.aliyuncs.com/test-cri/busybox:latest
Deleted: registry.cn-hangzhou.aliyuncs.com/test-cri/busybox:latest

containerd的配置文件

可以参考官方文档 full-configuration

注意缩进,一个是通过config_path指定配置文件, 是热更新的 (官方推荐,containerd 1.4之前是不支持的),另一个是通过直接修改config.toml方式添加

配置加速器,拉取docker官方镜像

使用config_path方式

只需确认 /etc/containerd/config.toml是否有配置 config_path和对应的路径, 然后如下创建即可,此种方式是热更新,不用重启containerd

1
2
3
4
5
6
7
8
$ mkdir -p /etc/containerd/cert.d/docker.io

$ cat << EOF > /etc/containerd/cert.d/docker.io/hosts.toml
server = "https://docker.io"

[host."https://xxxxxxxx.mirror.aliyuncs.com"]
capabilities = ["pull", "resolve", "push"]
EOF

配置config.toml的方式

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
$ cat /etc/containerd/config.toml
version = 2
...

[plugins]
...
[plugins."io.containerd.grpc.v1.cri"]
....

[plugins."io.containerd.grpc.v1.cri".registry]
# config_path = "/etc/containerd/cert.d"
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://xxxxxxxx.mirror.aliyuncs.com"]
...
重启containerd
1
systemctl restart containerd

如果有报错 journalctl -xe -u containerd --no-pager 查一下为何报错

配置非安全方式拉取镜像

使用config_path方式

只需确认 /etc/containerd/config.toml是否有配置 config_path和对应的路径, 然后如下创建即可, 此种方式是热更新, 不用重启containerd

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# http方式
$ mkdir -p /etc/containerd/cert.d/192.168.66.42:5000

$ cat << EOF > /etc/containerd/cert.d/192.168.66.42:5000/hosts.toml
server = "http://192.168.66.42:5000"

[host."http://192.168.66.42:5000"]
capabilities = ["pull", "resolve", "push"]
EOF

# 自建证书
$ mkdir -p /etc/containerd/cert.d/harbor.test-cri.com
$ cat << EOF > /etc/containerd/cert.d/harbor.test-cri.com/hosts.toml
server = "https://harbor.test-cri.com"

[host."https://harbor.test-cri.com"]
capabilities = ["pull", "resolve", "push"]
skip_verify = true
# ca = "/opt/ssl/ca.crt" # 或者上传ca证书
EOF

配置config.toml的方式

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
$ cat /etc/containerd/config.toml
version = 2
...
[plugins]
...
[plugins."io.containerd.grpc.v1.cri"]
....

[plugins."io.containerd.grpc.v1.cri".registry]
# config_path = "/etc/containerd/cert.d"
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."192.168.66.42:5000"] # 如果是http方式拉取,只需要申明我们的镜像地址是一个http类型的即可
endpoint = ["http://192.168.66.42:5000"]

[plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.test-cri.com".tls] # 如果是自签证书, 只需要设置不校验证书即可
insecure_skip_verify = true
# ca_file = "/opt/ssl/ca.crt" # 或者上传ca证书
# cert_file =" " # 双向认证cert
# key_file = " " # 双向认证key
[plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.test-cri.com".auth] # 配置认证
username = "admin"
password = "Harbor12345"
...
重启containerd
1
$ systemctl restart containerd

如果有报错 journalctl -xe -u containerd --no-pager 查一下为何报错