Oracle Clusterware 11g/12c does not Start when server rebooted

The Oracle Clusterware 11g/12c does not properly function when  Server restarted / rebooted as issue with mounting devices

When the clusterware is manually stopped and started it works fine ; but the issue arises only when the server is rebooted the clusterware does not start.

Seen in CRS log?

2014-01-04 15:23:55.590910 :CLSDYNAM:3642935040: [ora.dgcfs.works01.acfs]{1:1602:2} [start] BaseFSAgent::ACFS-9257: Mounting device '/dev/asm/works01-100' on mount point '/acfs/works01'
2014-01-04 15:23:55.590947 :CLSDYNAM:3642935040: [ora.dgcfs.works01.acfs]{1:1602:2} [start] UsmUtils::execCmd: cmd=/bin/mount -t acfs /dev/asm/works01-100 "/acfs/works01"
2014-01-04 15:23:55.590984 :CLSDYNAM:3642935040: [ora.dgcfs.works01.acfs]{1:1602:2} [start] Utils::getOracleHomeAttrib getEnvVar oracle_home:/u01/app/12.1.0/grid
2014-01-04 15:23:55.591004 :CLSDYNAM:3642935040: [ora.dgcfs.works01.acfs]{1:1602:2} [start] Utils::getOracleHomeAttrib oracle_home:/u01/app/12.1.0/grid
2014-01-04 15:23:55.591046 :CLSDYNAM:3642935040: [ora.dgcfs.works01.acfs]{1:1602:2} [start] Utils:execCmd action = 1 flags = 38 ohome =  cmdname = /bin/mount.
2014-01-04 15:23:55.601540 :CLSDYNAM:2748888832: [ora.dgcfs.oggtrails.acfs]{1:1602:2} [start] (:CLSN00010:)/sbin/mount.acfs.bin: error while loading shared libraries: libclntsh.so.12.1: cannot enable executable stack as shared object requires: Permission denied
2014-01-04 15:23:55.601577 :CLSDYNAM:2748888832: [ora.dgcfs.oggtrails.acfs]{1:1602:2} [start] (:CLSN00010:)
2014-01-04 15:23:55.601667 :CLSDYNAM:2748888832: [ora.dgcfs.oggtrails.acfs]{1:1602:2} [start] (:CLSN00010:)Utils:execCmd scls_process_join() uret 127


[oracle@db001 ~]$ ls -la /u01/app/12.1.0/grid/lib/libclntsh.*
lrwxrwxrwx. 1 oracle oinstall       17 Nov  2 11:50 /u01/app/12.1.0/grid/lib/libclntsh.so -> libclntsh.so.12.1
lrwxrwxrwx. 1 oracle oinstall       12 Nov  2 11:50 /u01/app/12.1.0/grid/lib/libclntsh.so.10.1 -> libclntsh.so
lrwxrwxrwx. 1 oracle oinstall       12 Nov  2 11:50 /u01/app/12.1.0/grid/lib/libclntsh.so.11.1 -> libclntsh.so
-rw-r--r--. 1 root   oinstall 58793846 Nov  2 11:50 /u01/app/12.1.0/grid/lib/libclntsh.so.12.1

[root@db001 ~]# getenforce
Enforcing

[root@db001 ~]# cat /etc/selinux/config

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=enforcing
# SELINUXTYPE= can take one of these two values:
#     targeted - Targeted processes are protected,
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

So basically Linux OS level security kicks in preventing the mount on clusterware.
The option "selinux" should be set to disabled.

To completely disable SELinux, use either of these methods:

1. Edit /etc/selinux/config

• Change the SELINUX value to "SELINUX=disabled".
• Reboot the server.

2. Append kernel boot options

•  Edit the kernel boot line in /boot/grub/grub.conf and append "selinux=0" to the kernel boot options. For example:

title Red Hat Enterprise Linux AS (2.6.9-42.ELsmp)
root (hd0,0)
kernel /vmlinuz-2.6.9-42.ELsmp ro root=LABEL=/ rhgb quiet selinux=0
initrd /initrd-2.6.9-42.ELsmp.img

• Reboot the server.

To set SELinux to Permissive mode, use either of these methods:

1. Changing SELinux running mode without rebooting

The setenforce command is used to change between enforcing and permissive mode.

To change to permissive  mode:

#setenforce 0

Use the getenforce command to view current SELinux mode:

#getenforce
Permissive

2. Setting SELinux to Permissive mode permanently

a. Edit /etc/selinux/config

• Change the SELINUX value to "SELINUX=permissive"

b. Append kernel boot options

• Edit the kernel boot line and append "enforcing=0" to the kernel boot options (Assuming SELinux is not set to disabled as in section above). For example:

title Red Hat Enterprise Linux AS (2.6.9-42.ELsmp)
root (hd0,0)
kernel /vmlinuz-2.6.9-42.ELsmp ro root=LABEL=/ rhgb quiet enforcing=0
initrd /initrd-2.6.9-42.ELsmp.img

• Reboot the server.

To check the status of SELinux, issue:

# /usr/sbin/sestatus