Docker

解决中标麒麟OS运行docker容器异常报错的问题一则

一 问题现象

在中标麒麟V10OS上,通过docker来启动一个tomcat容器时,抛出下述错误:

[root@appserver home-tomcat]# docker logs home-tomcat|more
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
# An error report file with more information is saved as:
# /usr/local/tomcat/hs_err_pid1.log
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
# An error report file with more information is saved as:
# /tmp/hs_err_pid1.log
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
# Can not save log file, dump to screen..
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (gcTaskThread.cpp:48), pid=1, tid=0x00007fcafce1b140
#
# JRE version:  (8.0_345-b01) (build )
# Java VM: OpenJDK 64-Bit Server VM (25.345-b01 mixed mode linux-amd64 compressed oops)
# Core dump written. Default location: /usr/local/tomcat/core or core.1
#
​
--More--

二 场景描述

1 服务器环境:

[root@appserver home-tomcat]# cat /etc/os-release 
NAME="Kylin Linux Advanced Server"
VERSION="V10 (Sword)"
ID="kylin"
VERSION_ID="V10"
PRETTY_NAME="Kylin Linux Advanced Server V10 (Sword)"
ANSI_COLOR="0;31"
​
[root@appserver home-tomcat]# uname -rm
4.19.90-25.16.v2101.ky10.x86_64 x86_64
[root@appserver home-tomcat]# 

2 docker版本:

​
[root@appserver home-tomcat]# docker info
Client:
 Context:    default
 Debug Mode: false
​
Server:
 Containers: 11
  Running: 3
  Paused: 0
  Stopped: 8
 Images: 5
 Server Version: 20.10.9
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 5b46e404f6b9f661a205e28d59c982d3634148f8
 runc version: b78641cb69ee91578b4cd7503dbb734d660adaf4-dirty
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.90-25.16.v2101.ky10.x86_64
 Operating System: Kylin Linux Advanced Server V10 (Sword)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 14.41GiB
 Name: appserver
 ID: 2H7O:RNCA:63I7:YNOC:UIXT:AD5Y:3UBN:DZVL:A7MD:PNUG:YN2V:ONLH
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  192.168.29.234
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine
​
[root@appserver home-tomcat]# 

3 tomcat image版本:

[root@appserver home-tomcat]# docker images|grep jre8
tomcat       9.0.68-jre8                 c69f06e08732   5 months ago   241MB
[root@appserver home-tomcat]# 

通过该tomcat:9.0.68-jre8来启动容器时,抛出上述错误。同样的启动脚本,同样的程序包在另外一台CentOS 7.9机器上

[root@ec-tna-27 ~]# cat /etc/redhat-release 
CentOS Linux release 7.9.2009 (Core)
[root@ec-tna-27 ~]# uname -rm
3.10.0-1160.24.1.el7.x86_64 x86_64
[root@ec-tna-27 ~]# 

却可以正常启动运行。

三 分析解决问题

1 调整JVM内存

根据错误日志提示,以为是JVM内存不够,其实不是,当前机器16GB内存,没有运行其它任何应用,内存富余很多。抱着试试看的心态,在容器启动命令行上,添加-e JAVA_OPTS=’-Xmx1g -Xms1g’ 参数,尝试重新启动,依然出现同样的错误。

2 升级docker

尝试把docker从20.10.9升级到20.10.10版本,再次启动运行容器,依然出现同样的错误。

3 设置主机的SWAP

当前机器没有设置SWAP,尝试配置服务器的SWAP,结果依然不奏效。

3 更换tomcat images

在docker hub上,查找其它版本的tomcat image,尝试用一个tomcat:9.0.73-jdk8-corretto-al2的image来启动运行容器。bingo,运行正常。

四 小结

解决该问题,花费了不少时间。也走了不少弯路,一度根据网上的提示,怀疑是docker版本的问题,尝试把docker升级了,问题并没有得到解决。

参考链接:

https://blog.csdn.net/u012809308/article/details/111561850

https://stackoverflow.com/questions/72841549/container-fails-to-start-insufficient-memory-for-the-java-runtime-environment-t

https://hub.docker.com/_/tomcat

留言