ORA-39511|静默安装 Oracle 19c RAC 时遇到的头疼事儿

作者 | JiekeXu

来源 | JiekeXu DBA之路（ID: JiekeXu_IT）

大家好，我是 JiekeXu,很高兴又和大家见面了,今天和大家一起来看看 ORA-39511 问题处理，欢迎点击上方蓝字关注我，标星或置顶，更多干货第一时间到达！

事情是这么个事，看官慢慢听我道来，从上周开始，便陆陆续续的安装操作系统准备安装 RAC，上周五安装好系统，设置好一些参数及基础配置，便解压 GI 安装软件准备安装。

这里也暂时不说详细的配置过程，只说明与本次问题相关的配置，以后再说详细安装过程。本次 IP 及主机名规划如下 hosts 文件所示，公网使用 192.168.75 网段，私网使用 10.10.10 网段。VMware 16 虚拟机使用 VMnet8 和 VMnet1 本地虚拟网卡。

[grid@jiekexu-r1:/home/grid]$ cat /etc/hosts 
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
#public ip
192.168.75.128  jiekexu-r1
192.168.75.129  jiekexu-r2
#private ip
10.10.10.128 jiekexu-r1-priv
10.10.10.129 jiekexu-r2-priv
#vip
192.168.75.130 jiekexu-r1-vip
192.168.75.131 jiekexu-r2-vip
#scanip
192.168.75.132 jiekexu-racscan

而 Linux 虚拟机的两块网卡 ens32 ens33 分别配置如下

 [root@jiekexu-r1 network-scripts]# more ifcfg-ens32 
 TYPE="Ethernet" 
 PROXY_METHOD="none" 
 BROWSER_ONLY="no" 
 BOOTPROTO="dhcp" 
 DEFROUTE="yes" 
 IPV4_FAILURE_FATAL="yes" 
 IPV6INIT="yes" 
 IPV6_AUTOCONF="yes" 
 IPV6_DEFROUTE="yes" 
 IPV6_FAILURE_FATAL="no" 
 IPV6_ADDR_GEN_MODE="stable-privacy" 
 NAME="ens32" 
 UUID="e8332396-7724-410f-9887-1fc0e27fc818" 
 DEVICE="ens32" 
 ONBOOT="yes" 
 GATEWAY="192.168.75.2" 
 IPADDR="192.168.75.128" 
 PREFIX="24" 
 IPV6_PRIVACY="no" 
 [root@jiekexu-r1 network-scripts]# more ifcfg-ens33 
 TYPE=Ethernet 
 PROXY_METHOD=none 
 BROWSER_ONLY=no 
 BOOTPROTO=static 
 DEFROUTE=yes 
 IPV4_FAILURE_FATAL=no 
 IPV6INIT=yes 
 IPV6_AUTOCONF=yes 
 IPV6_DEFROUTE=yes 
 IPV6_FAILURE_FATAL=no 
 IPV6_ADDR_GEN_MODE=stable-privacy 
 NAME=ens33 
 DEVICE=ens33 
 HWADDR="00:0C:29:2C:79:D8" 
 uuid="c96bc909-188e-ec64-3a96-6a90982b08ad" 
 ONBOOT=yes 
 GATEWAY="10.10.10.1" 
 IPADDR="10.10.10.128" 
 PREFIX="8" 

然后解压 GI 软件,解压 RU 补丁 32900083 ，替换 OPatch，配置互信。

分别使用grid和oracle用户调用sshUserSetup.sh脚本配置SSH connectivity，根据提示输入密码。
$ cd $ORACLE_HOME/oui/prov/resources/scripts
<grid>$ ./sshUserSetup.sh -user grid -hosts "jiekexu-r1 jiekexu-r2" -advanced -confirm -noPromptPassphrase
cd /u01/app/19.0.0/grid/oui/prov/resources/scripts
<oracle>$ ./sshUserSetup.sh -user oracle -hosts "jiekexu-r1 jiekexu-r2" -advanced -confirm -noPromptPassphrase

--分别使用grid和oracle用户验证 SSH connectivity：
<grid>$ for h in jiekexu-r1 jiekexu-r1-priv jiekexu-r2 jiekexu-r2-priv;do
ssh -l grid -o StrictHostKeyChecking=no $h date;
done
<oracle>$ for h in jiekexu-r1 jiekexu-r1-priv jiekexu-r2 jiekexu-r2-priv;do
ssh -l oracle -o StrictHostKeyChecking=no $h date;
done

GI 安装配置预检查

$/u01/app/19.0.0/grid/runcluvfy.sh stage -pre crsinst -n jiekexu-r1,jiekexu-r2 -method root

执行补丁和 GI 安装

#### 执行补丁和 GI 安装

$ORACLE_HOME/gridSetup.sh -silent -applyRU /home/soft/32900083/32895426  -force -noconfig  -ignorePrereq  -waitforcompletion \
-responseFile ${ORACLE_HOME}/install/response/gridsetup.rsp \
INVENTORY_LOCATION=/u01/app/oraInventory \
ORACLE_BASE=/u01/app/grid \
ORACLE_HOME=/u01/app/19.0.0/grid \
oracle.install.asm.OSDBA=asmdba \
oracle.install.asm.OSOPER=asmoper \
oracle.install.asm.OSASM=asmadmin \
oracle.install.crs.config.storageOption=ASM \
oracle.install.crs.config.useIPMI=false \
oracle.install.asm.diskGroup.name=OCR \
oracle.install.asm.diskGroup.AUSize=4 \
oracle.install.asm.diskGroup.redundancy=NORMAL \
oracle.install.crs.config.ClusterConfiguration=STANDALONE \
oracle.install.crs.config.configureAsExtendedCluster=false \
oracle.install.crs.config.gpnp.scanPort=1521 \
oracle.install.crs.config.gpnp.configureGNS=false \
oracle.install.option=CRS_CONFIG \
oracle.install.asm.SYSASMPassword=Oracle19C \
oracle.install.asm.monitorPassword=Oracle19C \
oracle.install.asm.diskGroup.diskDiscoveryString=/dev/asm_* \
oracle.install.asm.diskGroup.disks=/dev/asm_ocr01,/dev/asm_ocr02,/dev/asm_ocr03 \
oracle.install.crs.config.gpnp.scanName=jiekexu-racscan \
oracle.install.crs.config.clusterName=jiekexu-racscan \
oracle.install.crs.config.autoConfigureClusterNodeVIP=false \
oracle.install.crs.config.clusterNodes=jiekexu-r1:jiekexu-r1-vip,jiekexu-r2:jiekexu-r2-vip \
oracle.install.crs.config.networkInterfaceList=ens32:192.168.75.0:1,ens33:10.10.10.0:5 \
oracle.install.asm.configureGIMRDataDG=false \
oracle.install.crs.configureRHPS=false \
oracle.install.crs.config.ignoreDownNodes=false \
oracle.install.config.managementOption=NONE \
oracle.install.config.omsPort=0 \
oracle.install.crs.rootconfig.executeRootScript=false 

一、出现问题

完成此步骤后，则需要执行 root.sh ,当在节点一执行 root.sh 时便出现了报错。

查看上图中 asmca 日志 ORA-39511 haip 无法启动，查看 mos 说与网卡有问题，则将网卡删除重建，甚至怀疑节点一主机的问题，弃用节点一主机，使用节点二主机复制新生成主机一，然后重新安装 GI。

ORA-39511: Start of CRS resource for instance ‘215’ failed with error:[CRS-2672: Attempting to start ‘ora.cluster_interconnect.haip’ on ‘jiekexu-r1’

即使这样，当到这一步执行 root.sh 时报错还是一样，删网卡，加网卡，来来回回折腾了不下十次，花费了几天的时间，直到今天早上又重新配置了网卡删除了残余的 GI ，使用 ssh-keygen 配置互信，又来了一遍可还是同样的错误，真让人头疼，欲哭无泪了。。。

这里贴一下删除过程，以备不时之需。

**如果脚本执行错误，重复执行也无法修复时，则需要使用 deinstall 卸载重新安装。**

cd $ORACLE_HOME/deinstall
./deinstall
--根据提示输入和执行脚本  
n   n   y  另开会话执行脚本 ./rootcrs.sh -force  -deconfig -paramfile ……，等待十几分钟

rm -rf /opt/ORCLfmap
rm -rf /etc/oraInst.loc
--检查所有节点的配置文件/etc/oraInst.loc、/etc/oratab是否存在相关配置，若有，则删除之。
--使用 root 用户登录所有节点，执行：
rm -rf /etc/oracle /tmp/.oracle /var/tmp/.oracle /u01/app/oraInventory /u01/app/grid/* /tmp/GridSetupActions*
--卸载后新建 mkdir -p  /u01/app/19.0.0/grid
chown -R grid:oinstall /u01
chmod -R 775 /u01
chown -R oracle:oinstall /u01/app/oracle
另一节点如果还没执行 root.sh 则直接  /u01/app/19.0.0/grid ：
cd $ORACLE_HOME
rm -rf *   rm -rf .opatchauto_storage  rm -rf .patch_storage 
cd /u01/app/grid 
rm -rf *
rm -rf /u01/app/oraInventory

在这期间，修改了 VMware 虚拟网卡 VMnet1 的配置，这是关键的一步.由于之前虚拟机没有使用过 10.10.10 网段的地址，VMnet1 还是原来的 192.168 网段地址。这里将其修改为 10.10.10.0 网段，注意子网掩码改为 255.255.255.0

然后下午排查问题时将 ens32 ens33 两块网卡配置为如下：

cat > /etc/sysconfig/network-scripts/ifcfg-ens32 <<"EOF"
DEVICE=ens32
NAME=ens32
IPADDR=192.168.75.128
NETMASK=255.255.255.0
GATEWAY=192.168.75.2
ONBOOT=yes
USERCTL=no
BOOTPROTO=static
HWADDR=00:0c:29:e4:ce:69
TYPE=Ethernet
IPV6INIT=no
DNS1=114.114.114.114
DNS2=8.8.8.8
NM_CONTROLLED=no
EOF

--注意第二块网卡不配置网关，否则系统默认网关就变成了第二块网卡的网关，从而导致系统不能上外网。

cat > /etc/sysconfig/network-scripts/ifcfg-ens33 <<"EOF"
DEVICE=ens33
NAME=ens33
IPADDR=10.10.10.128
NETMASK=255.0.0.0
ONBOOT=yes
USERCTL=no
BOOTPROTO=static
HWADDR=00:0c:29:e4:ce:73
TYPE=Ethernet
IPV6INIT=no
DNS1=114.114.114.114
DNS2=8.8.8.8
NM_CONTROLLED=no
EOF

顺便使用 kfod 排查了共享磁盘的配置是否正确的问题

[grid@jiekexu-r1:/home/grid]$ kfod disks=all status=true
Disk          Size           Header    Path                                     User     Group   

   1:      10240 MB CANDIDATE /dev/asm_arch01                          grid     asmadmin
   2:      10240 MB CANDIDATE /dev/asm_data01                          grid     asmadmin
   3:      10240 MB CANDIDATE /dev/asm_data02                          grid     asmadmin
   4:       3072 MB CANDIDATE /dev/asm_ocr01                           grid     asmadmin
   5:       3072 MB CANDIDATE /dev/asm_ocr02                           grid     asmadmin
   6:       3072 MB CANDIDATE /dev/asm_ocr03                           grid     asmadmin
KFOD-00313: No ASM instances available. CSS group services were successfully initialized by kgxgncin. 

--直接使用 asmca 静默创建 OCR 也是同样的错误
[grid@jiekexu-r1:/home/grid]$ /u01/app/19.0.0/grid/bin/asmca -silent -diskGroupName OCR -diskList '/dev/asm_ocr03,/dev/asm_ocr01,/dev/asm_ocr02' -redundancy NORMAL -diskString '/dev/asm_*' -configureLocalASM -passwordFileLocation +OCR/orapwASM -au_size 4 
ASM failed to start. Check /u01/app/grid/cfgtoollogs/asmca/asmca-210831PM030816.log for details.
[grid@jiekexu-r1:/home/grid]$ more /u01/app/grid/cfgtoollogs/asmca/asmca-210831PM030816.log
[USMInstance.configureLocalASM:3413]  ORA-39511: Start of CRS resource for instance '215' failed with error:[CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'jiekexu-r1'

二、下午定位问题

既然是 HAIP  问题，那么能不能将其禁用掉，然后再执行 root.sh 呢？死马当活马医吧！！！

cd /u01/app/19.0.0/grid/bin
[root@jiekexu-r1 bin]# ./crsctl modify res ora.cluster_interconnect.haip -attr "ENABLED=0" -init
[root@jiekexu-r1 bin]# ./crsctl modify res ora.asm -attr "START_DEPENDENCIES='hard(ora.cssd,ora.ctssd)pullup(ora.cssd,ora.ctssd)weak(ora.drivers.acfs)',STOP_DEPENDENCIES='hard(intermediate:ora.cssd)'" -init
[root@jiekexu-r1 bin]# 
[root@jiekexu-r1 bin]# /u01/app/19.0.0/grid/root.sh

禁用掉后执行 root.sh 如下报错： ORA-27504

[main] [ 2021-08-31 17:57:27.543 CST ] [UsmcaLogger.logException:187]  ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:if_not_found failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvaddr9
ORA-27303: additional information: requested interface 48.0.0.0 not found. Check output from ifconfig command

看情况还是网卡相关的问题，按照 MOS 上的文章(Doc ID 2356980.1)尝试了一下还是一样的报错：

Error ORA-27504: IPC error creating OSD context - ASM Failed to Start After OS Kernel Patching (Doc ID 2356980.1)

[root@jiekexu-r1 grid]# cd /u01/app/19.0.0/grid/bin
[root@jiekexu-r1 bin]# cd ..
[root@jiekexu-r1 grid]# cd crs/install/
[root@jiekexu-r1 install]#  ./rootcrs.sh -unlock

[grid@jiekexu-r1:/home/grid]$ make -f $ORACLE_HOME/rdbms/lib/ins_rdbms.mk ipc_g ioracle

[root@jiekexu-r1 install]# ./rootcrs.sh -lock

报错一样，则先将禁用的 HAIP 回退掉。

[root@jiekexu-r1 bin]# ./crsctl modify res ora.cluster_interconnect.haip -attr "ENABLED=1" -init
[root@jiekexu-r1 bin]# ./crsctl modify res ora.asm -attr "START_DEPENDENCIES='hard(ora.cssd,ora.ctssd)pullup(ora.cssd,ora.ctssd)weak(ora.cluster_interconnect.haip,ora.drivers.acfs)',STOP_DEPENDENCIES='hard(intermediate:ora.cssd,shutdown:ora.cluster_interconnect.haip)'" -init

思来想去，还是在看看日志吧，除了安装时看的 rootcrs 日志和 asmca-210831PM061542.log 日志外，需要再看看 gipcd.trc日志，由于 19c 日志发生变化，则需要到如下路径下查看。日志发生变化的新特新之前也写过一篇文章，感兴趣的小伙伴可戳此查看。

/u01/app/grid/diag/crs/jiekexu-r1/crs/trace

tail -30f gipcd.trc

2021-08-31 18:06:58.778 :GIPCDCLT:3717416704:  gipcdClientInterfaceQueryToMonitor: enqueue local interface query (2) to worklist
2021-08-31 18:06:58.778 :GIPCDCLT:3717416704:  gipcdClientInterfaceRequest: sent local interface query
2021-08-31 18:06:58.827 : CLSINET:3717416704:  (:CLSINE0018:)WARNING: failed to find interface available for interface definition ens33(:.*)?:10.10.10.0
2021-08-31 18:06:58.829 :GIPCDCLT:3717416704:  gipcdUpdateLocalClient: successfully sent the inf update
2021-08-31 18:06:58.829 :GIPCDCLT:3717416704:  gipcdClientInterfaceRequest: sent local interface list back to client
2021-08-31 18:06:58.830 : CLSINET:3713214208:  (:CLSINE0018:)WARNING: failed to find interface available for interface definition ens33(:.*)?:10.10.10.0
2021-08-31 18:06:59.267 :GIPCDCLT:3717416704:  gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceRequest, endp 0000000000000395
2021-08-31 18:06:59.267 :GIPCDCLT:3717416704:  gipcdClientInterfaceRequest: Received type(gipcdmsgtypeInterfaceRequest), endp(0000000000000395), len(1024), buf(0x7fd2c432c4a8):[hostname(jiekexu-r1), retStatus(gipcretSuccess)]
2021-08-31 18:06:59.267 :GIPCDCLT:3717416704:  gipcdClientInterfaceQueryToMonitor: enqueue local interface query (2) to worklist
2021-08-31 18:06:59.267 :GIPCDCLT:3717416704:  gipcdClientInterfaceRequest: sent local interface query
2021-08-31 18:06:59.315 : CLSINET:3717416704:  (:CLSINE0018:)WARNING: failed to find interface available for interface definition ens33(:.*)?:10.10.10.0
2021-08-31 18:06:59.317 :GIPCDCLT:3717416704:  gipcdUpdateLocalClient: successfully sent the inf update
2021-08-31 18:06:59.317 :GIPCDCLT:3717416704:  gipcdClientInterfaceRequest: sent local interface list back to client
2021-08-31 18:06:59.831 :GIPCDMON:3713214208:  gipcdMonitorCheckXfer: set new infQuery
2021-08-31 18:06:59.831 :GIPCDMON:3713214208:  gipcdMonitorCheckXfer: set new infQuery
2021-08-31 18:06:59.932 : CLSINET:3713214208:  (:CLSINE0018:)WARNING: failed to find interface available for interface definition ens33(:.*)?:10.10.10.

failed to find interface available for interface definition ens33(:.*)?:10.10.10.0

果然报错 ens33 网卡接口的问题，那么到底什么问题呢？网卡都重建过，还有哪里的问题？？？抱着尝试的心态想到好心的网友说的将子网掩码改为 24 试试，那就试试吧。

cd /etc/sysconfig/network-scripts/

--重启网卡后查看 gipcd.trc 文件则已经没有报错了
ifdown ens33
ifup ens33

--执行 root.sh 脚本成功，历时八天终于成功了
[root@jiekexu-r1 network-scripts]# cat /u01/app/19.0.0/grid/install/root_jiekexu-r1_2021-08-31_18-14-46-714402688.log
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/19.0.0/grid
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/19.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/jiekexu-r1/crsconfig/rootcrs_jiekexu-r1_2021-08-31_06-14-47PM.log
2021/08/31 18:14:50 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2021/08/31 18:14:50 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2021/08/31 18:14:50 CLSRSC-363: User ignored prerequisites during installation
2021/08/31 18:14:50 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2021/08/31 18:14:52 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2021/08/31 18:14:53 CLSRSC-594: Executing installation step 5 of 19: 'SetupOSD'.
2021/08/31 18:14:53 CLSRSC-594: Executing installation step 6 of 19: 'CheckCRSConfig'.
2021/08/31 18:14:53 CLSRSC-594: Executing installation step 7 of 19: 'SetupLocalGPNP'.
2021/08/31 18:14:54 CLSRSC-594: Executing installation step 8 of 19: 'CreateRootCert'.
2021/08/31 18:14:57 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2021/08/31 18:14:57 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2021/08/31 18:14:57 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2021/08/31 18:14:58 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2021/08/31 18:15:00 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2021/08/31 18:15:00 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
2021/08/31 18:15:01 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2021/08/31 18:15:04 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.
2021/08/31 18:15:22 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.

ASM has been created and started successfully.

[DBT-30001] Disk groups created successfully. Check /u01/app/grid/cfgtoollogs/asmca/asmca-210831PM061542.log for details.

2021/08/31 18:16:34 CLSRSC-482: Running command: '/u01/app/19.0.0/grid/bin/ocrconfig -upgrade grid oinstall'
CRS-4256: Updating the profile
Successful addition of voting disk 14ed0aa13ffd4f89bfe2d79061f96fbc.
Successful addition of voting disk 27cc8fbc135f4fd3bf574b4d2e62531e.
Successful addition of voting disk c5c806e3a2414f74bf1c70f2add4a821.
Successfully replaced voting disk group with +OCR.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced

STATE    File Universal Id                File Name Disk group

--  -----    -----------------                --------- ---------

  1. ONLINE   14ed0aa13ffd4f89bfe2d79061f96fbc (/dev/asm_ocr03) [OCR]
  2. ONLINE   27cc8fbc135f4fd3bf574b4d2e62531e (/dev/asm_ocr01) [OCR]
  3. ONLINE   c5c806e3a2414f74bf1c70f2add4a821 (/dev/asm_ocr02) [OCR]
     Located 3 voting disk(s).
     2021/08/31 18:17:29 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
     2021/08/31 18:18:28 CLSRSC-343: Successfully started Oracle Clusterware stack
     2021/08/31 18:18:28 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.
     2021/08/31 18:20:11 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.
     2021/08/31 18:20:38 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

节点 2 执行 root.sh 也成功了，兴奋中……

[grid@jiekexu-r2:/home/grid]$ tail -30f /u01/app/19.0.0/grid/install/root_jiekexu-r2_2021-08-31_18-25-18-796337893.log
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/19.0.0/grid
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/19.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/jiekexu-r2/crsconfig/rootcrs_jiekexu-r2_2021-08-31_06-25-49PM.log
2021/08/31 18:25:53 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2021/08/31 18:25:53 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2021/08/31 18:25:53 CLSRSC-363: User ignored prerequisites during installation
2021/08/31 18:25:53 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2021/08/31 18:25:55 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2021/08/31 18:25:55 CLSRSC-594: Executing installation step 5 of 19: 'SetupOSD'.
Redirecting to /bin/systemctl restart rsyslog.service
2021/08/31 18:25:56 CLSRSC-594: Executing installation step 6 of 19: 'CheckCRSConfig'.
2021/08/31 18:25:58 CLSRSC-594: Executing installation step 7 of 19: 'SetupLocalGPNP'.
2021/08/31 18:26:02 CLSRSC-594: Executing installation step 8 of 19: 'CreateRootCert'.
2021/08/31 18:26:02 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2021/08/31 18:26:16 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2021/08/31 18:26:16 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2021/08/31 18:26:20 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2021/08/31 18:26:20 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2021/08/31 18:26:43 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2021/08/31 18:26:43 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
2021/08/31 18:27:30 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2021/08/31 18:27:33 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.
2021/08/31 18:27:42 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
2021/08/31 18:27:56 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2021/08/31 18:28:28 CLSRSC-343: Successfully started Oracle Clusterware stack
2021/08/31 18:28:29 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.
2021/08/31 18:29:13 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.
2021/08/31 18:29:19 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

执行完毕后查看集群状态如下：

配置集群

根据静默安装后的提示内容，执行完 root.sh 脚本后，需要执行 executeConfigTools 配置，更新节点信息。

/u01/app/19.0.0/grid/gridSetup.sh  -silent -executeConfigTools -waitforcompletion \
-responseFile /u01/app/19.0.0/grid/install/response/gridsetup.rsp  \
INVENTORY_LOCATION=/u01/app/oraInventory \
ORACLE_BASE=/u01/app/grid \
ORACLE_HOME=/u01/app/19.0.0/grid \
oracle.install.asm.OSDBA=asmdba \
oracle.install.asm.OSOPER=asmoper \
oracle.install.asm.OSASM=asmadmin \
oracle.install.crs.config.storageOption=ASM \
oracle.install.crs.config.useIPMI=false \
oracle.install.asm.diskGroup.name=OCR \
oracle.install.asm.diskGroup.AUSize=4 \
oracle.install.asm.diskGroup.redundancy=NORMAL \
oracle.install.crs.config.ClusterConfiguration=STANDALONE \
oracle.install.crs.config.configureAsExtendedCluster=false \
oracle.install.crs.config.gpnp.scanPort=1521 \
oracle.install.crs.config.gpnp.configureGNS=false \
oracle.install.option=CRS_CONFIG \
oracle.install.asm.SYSASMPassword=Oracle19C \
oracle.install.asm.monitorPassword=Oracle19C \
oracle.install.asm.diskGroup.diskDiscoveryString=/dev/asm_* \
oracle.install.asm.diskGroup.disks=/dev/asm_ocr01,/dev/asm_ocr02,/dev/asm_ocr03 \
oracle.install.crs.config.gpnp.scanName=jiekexu-racscan \
oracle.install.crs.config.clusterName=jiekexu-racscan \
oracle.install.crs.config.autoConfigureClusterNodeVIP=false \
oracle.install.crs.config.clusterNodes=jiekexu-r1:jiekexu-r1-vip,jiekexu-r2:jiekexu-r2-vip \
oracle.install.crs.config.networkInterfaceList=ens32:192.168.75.0:1,ens33:10.10.10.0:5 \
oracle.install.asm.configureGIMRDataDG=false \
oracle.install.crs.configureRHPS=false \
oracle.install.crs.config.ignoreDownNodes=false \
oracle.install.config.managementOption=NONE \
oracle.install.config.omsPort=0 \
oracle.install.crs.rootconfig.executeRootScript=false 

Launching Oracle Grid Infrastructure Setup Wizard...

You can find the logs of this session at:
/u01/app/oraInventory/logs/GridSetupActions2021-08-31_06-44-55PM

You can find the log of this install session at:
 /u01/app/oraInventory/logs/UpdateNodeList2021-08-31_06-44-55PM.log
Configuration failed.
[WARNING] [INS-43080] Some of the configuration assistants failed, were cancelled or skipped.
   ACTION: Refer to the logs or contact Oracle Support Services.

注意：在 /u01/app/19.0.0/grid/install/response 目录下，新生成了 grid_2021-08-31_02-31-29PM.rsp 响应文件，可以使用新生成的，我这里沿用了前面的配置项也可以。

三、总结归纳

此次问题主要是出在 ens33 私网网卡上，由于在虚拟机外部的网络 VMnet1 中设置了子网掩码为 24，但在内部虚拟机中配置 ens33 时却将子网掩码写成了 8 导致 ens33 网卡不可用，不管新建网卡，还是新建虚拟机都没有想到要改 IP 这块，导致使用了原来的错误配置进行安装，故每次 root.sh 都是同样的问题。

在 RAC 安装配置过程中，每一步都很重要，都需要仔细认真，不可马虎，要做到一丝不苟。利用各种日志排查分析问题，如安装过程中产生的日志，集群启动过程中的各个进程日志以及操作系统日志等等，这些均是分析问题的关键因素，本次问题主要就是忽略了各个进程日志才导致了很多重复工作，以后需要引以为戒。

最后说一句子网掩码 (subnet mask) 又叫网络掩码、地址掩码、子网络遮罩，是一个应用于 TCP/IP 网络的 32 位二进制值。它可以屏蔽掉 IP 地址中的一部分，从而分离出 IP 地址中的网络部分与主机部分，基于子网掩码，管理员可以将网络进一步划分为若干子网。它必须结合 IP 地址一起使用。 子网掩码设置为 24 则只有 254 个可用地址，设置为 8 则有16777214 个地址可用。关于网络问题不是很懂，这里有一篇知乎上的帖子说明子网掩码是个啥？

https://www.zhihu.com/question/56895036/answer/150953183

最大踩坑，真心不易，耐心阅读，此文一定对您有帮助，欢迎点赞、在看与转发，写作不易，坚持写作更不易，您的点赞、转发，举手之劳，便是对我最大的支持，也能让更多的人看到更多的人受益，万分感谢！

———————————————————————————

公众号：JiekeXu DBA之路
墨天轮：https://www.modb.pro/u/4347
CSDN ：https://blog.csdn.net/JiekeXu
腾讯云：https://cloud.tencent.com/developer/user/5645107

———————————————————————————