今天没事重启了一下rac ，本来觉得会没啥问题的，没想到rac2，rac3 节点都正常起来了，但rac1 始终起不来，查看日志，以下:css
[ctssd(3088)]CRS-2402:The Cluster Time Synchronization Service aborted on host rac1. Details at in /u01/app/11.2.0/grid_1/log/rac1/ctssd/octssd.log.
2017-03-09 16:06:40.950:
[ctssd(3106)]CRS-2405:The Cluster Time Synchronization Service on host rac1 is shutdown by user
2017-03-09 16:06:45.255:
[ctssd(3114)]CRS-2401:The Cluster Time Synchronization Service started on host rac1.
2017-03-09 16:06:45.256:
[ctssd(3114)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac2.
2017-03-09 16:06:48.405:
[ctssd(3114)]CRS-2404:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/11.2.0/grid_1/log/rac1/ctssd/octssd.log.
2017-03-09 16:06:48.980:
[ctssd(3114)]CRS-2402:The Cluster Time Synchronization Service aborted on host rac1. Details at in /u01/app/11.2.0/grid_1/log/rac1/ctssd/octssd.log.
2017-03-09 16:06:49.995:
[ohasd(2308)]CRS-2807:Resource 'ora.asm' failed to start automatically.
2017-03-09 16:06:49.995:
[ohasd(2308)]CRS-2807:Resource 'ora.crsd' failed to start automatically.
2017-03-09 16:06:49.995:
[ohasd(2308)]CRS-2807:Resource 'ora.ctssd' failed to start automatically.
2017-03-09 16:06:49.995:
[ohasd(2308)]CRS-2807:Resource 'ora.evmd' failed to start automatically.
node
尝试手动重启crs，发现最后仍是起不来，最好在看一下报错信息，发现The Cluster Time Synchronization Service aborted on host rac1。app
因而根据报错信息，发现rac1的时间和rac2，rac3 已经不一样了，因而赶忙把rac1的时间改回来，日志
[root@rac1 ~]# date -s '2017-03-05 13:50:12'
dns
[root@rac1 ~]# hwclock -w
ip
而后手动重启it
[root@rac1 bin]# ./crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rac1'
CRS-2677: Stop of 'ora.crf' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'
CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'
CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@rac1 bin]# ./crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@rac1 bin]# ps -ef |grep pmon
grid 4130 1 0 13:53 ? 00:00:00 asm_pmon_+ASM1
root 4323 2970 0 13:53 pts/0 00:00:00 grep pmon
[root@rac1 bin]#
io