解决中标麒麟OS运行docker容器异常报错的问题一则
Contents
一 问题现象
在中标麒麟V10OS上,通过docker来启动一个tomcat容器时,抛出下述错误:
[root@appserver home-tomcat]# docker logs home-tomcat|more # # There is insufficient memory for the Java Runtime Environment to continue. # Cannot create GC thread. Out of system resources. # An error report file with more information is saved as: # /usr/local/tomcat/hs_err_pid1.log # # There is insufficient memory for the Java Runtime Environment to continue. # Cannot create GC thread. Out of system resources. # An error report file with more information is saved as: # /tmp/hs_err_pid1.log # # There is insufficient memory for the Java Runtime Environment to continue. # Cannot create GC thread. Out of system resources. # Can not save log file, dump to screen.. # # There is insufficient memory for the Java Runtime Environment to continue. # Cannot create GC thread. Out of system resources. # Possible reasons: # The system is out of physical RAM or swap space # The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap # Possible solutions: # Reduce memory load on the system # Increase physical memory or swap space # Check if swap backing store is full # Decrease Java heap size (-Xmx/-Xms) # Decrease number of Java threads # Decrease Java thread stack sizes (-Xss) # Set larger code cache with -XX:ReservedCodeCacheSize= # This output file may be truncated or incomplete. # # Out of Memory Error (gcTaskThread.cpp:48), pid=1, tid=0x00007fcafce1b140 # # JRE version: (8.0_345-b01) (build ) # Java VM: OpenJDK 64-Bit Server VM (25.345-b01 mixed mode linux-amd64 compressed oops) # Core dump written. Default location: /usr/local/tomcat/core or core.1 # --More--
二 场景描述
1 服务器环境:
[root@appserver home-tomcat]# cat /etc/os-release NAME="Kylin Linux Advanced Server" VERSION="V10 (Sword)" ID="kylin" VERSION_ID="V10" PRETTY_NAME="Kylin Linux Advanced Server V10 (Sword)" ANSI_COLOR="0;31" [root@appserver home-tomcat]# uname -rm 4.19.90-25.16.v2101.ky10.x86_64 x86_64 [root@appserver home-tomcat]#
2 docker版本:
[root@appserver home-tomcat]# docker info Client: Context: default Debug Mode: false Server: Containers: 11 Running: 3 Paused: 0 Stopped: 8 Images: 5 Server Version: 20.10.9 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 1 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc Default Runtime: runc Init Binary: docker-init containerd version: 5b46e404f6b9f661a205e28d59c982d3634148f8 runc version: b78641cb69ee91578b4cd7503dbb734d660adaf4-dirty init version: de40ad0 Security Options: seccomp Profile: default Kernel Version: 4.19.90-25.16.v2101.ky10.x86_64 Operating System: Kylin Linux Advanced Server V10 (Sword) OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 14.41GiB Name: appserver ID: 2H7O:RNCA:63I7:YNOC:UIXT:AD5Y:3UBN:DZVL:A7MD:PNUG:YN2V:ONLH Docker Root Dir: /var/lib/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 192.168.29.234 127.0.0.0/8 Live Restore Enabled: false Product License: Community Engine [root@appserver home-tomcat]#
3 tomcat image版本:
[root@appserver home-tomcat]# docker images|grep jre8 tomcat 9.0.68-jre8 c69f06e08732 5 months ago 241MB [root@appserver home-tomcat]#
通过该tomcat:9.0.68-jre8来启动容器时,抛出上述错误。同样的启动脚本,同样的程序包在另外一台CentOS 7.9机器上
[root@ec-tna-27 ~]# cat /etc/redhat-release CentOS Linux release 7.9.2009 (Core) [root@ec-tna-27 ~]# uname -rm 3.10.0-1160.24.1.el7.x86_64 x86_64 [root@ec-tna-27 ~]#
却可以正常启动运行。
三 分析解决问题
1 调整JVM内存
根据错误日志提示,以为是JVM内存不够,其实不是,当前机器16GB内存,没有运行其它任何应用,内存富余很多。抱着试试看的心态,在容器启动命令行上,添加-e JAVA_OPTS=’-Xmx1g -Xms1g’ 参数,尝试重新启动,依然出现同样的错误。
2 升级docker
尝试把docker从20.10.9升级到20.10.10版本,再次启动运行容器,依然出现同样的错误。
3 设置主机的SWAP
当前机器没有设置SWAP,尝试配置服务器的SWAP,结果依然不奏效。
3 更换tomcat images
在docker hub上,查找其它版本的tomcat image,尝试用一个tomcat:9.0.73-jdk8-corretto-al2的image来启动运行容器。bingo,运行正常。
四 小结
解决该问题,花费了不少时间。也走了不少弯路,一度根据网上的提示,怀疑是docker版本的问题,尝试把docker升级了,问题并没有得到解决。
参考链接: