Tuesday, August 25, 2015

Apply the latest PSU5 for 12.1.0.2 RAC Database

                                 

When I was installing it , I have came across few bugs. So thought of sharing it with folks. I have covered both manual and auto install (Method 1 and Method 2). Please see the step by step doc.

1. Create a patch directory as below:

[oracle@Sajidserver01 patches]$ ls -ltr
total 1060264
drwxr-xr-x 7 oracle oinstall       4096 Jul 12 07:04 20996835
-rw-rw-r-- 1 oracle oinstall      84483 Jul 14 05:33 PatchSearch.xml
-rwxr-xr-x 1 oracle oinstall   52853599 Aug  8 19:28 p6880880_121010_Linux-x86-64.zip
-rwxr-xr-x 1 oracle oinstall 1031691784 Aug  8 19:28 p20996835_121020_Linux-x86-64.zip

2. Unzip the p20996835_121020_Linux-x86-64.zip and grant full permissions to  20996835 and make sure if more patches are downloaded they are unzipped in one folder

3. Unzip p6880880_121010_Linux-x86-64.zip in grid home as well as in all the oracle homes

4. Generate response file in both grid home as well as the oracle home. Login as with Supreme Privileges.

[root@Sajidserver02 patches]# cd /usr/app/12.1.0/grid/OPatch
[root@Sajidserver02 OPatch]# ./opatchauto apply /usr/patches/20996835/ -ocmrf /usr/app/oracle/product/12.1.0/OPatch/ocm/bin/ocm.rsp
OPatch Automation Tool
Copyright (c)2014, Oracle Corporation. All rights reserved.

OPatchauto Version : 12.1.0.1.8
OUI Version        : 12.1.0.2.0
Running from       : /usr/app/12.1.0/grid

opatchauto log file: /usr/app/12.1.0/grid/cfgtoollogs/opatchauto/20996835/opatch_gi_2015-08-08_18-05-18_deploy.log

Parameter Validation: Successful

Configuration Validation: Successful

Patch Location: /usr/patches/20996835
Grid Infrastructure Patch(es): 20831110 19872484 20299018 20831113
DB Patch(es): 20831110 20831113

Patch Validation: Successful
Grid Infrastructure home:
/usr/app/12.1.0/grid
DB home(s):
/usr/app/oracle/product/12.1.0


Performing prepatch operations on CRS Home... Successful

Applying patch(es) to "/usr/app/oracle/product/12.1.0" ...
Patch "/usr/patches/20996835/20831110" successfully applied to "/usr/app/oracle/product/12.1.0".
Patch "/usr/patches/20996835/20831113" successfully applied to "/usr/app/oracle/product/12.1.0".

Applying patch(es) to "/usr/app/12.1.0/grid" ...
Command "/usr/app/12.1.0/grid/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_grid_patchList -local  -invPtrLoc /usr/app/12.1.0/grid/oraInst.loc -oh /usr/app/12..0/grid -silent -ocmrf

/usr/app/oracle/product/12.1.0/OPatch/ocm/bin/ocm.rsp" execution failed:
UtilSession failed:
Prerequisite check "CheckApplicable" failed.


################################################################################
[root@Sajidserver01 OPatch]# ./opatchauto apply /usr/patches/20996835/ -ocmrf /usr/app/oracle/product/12.1.0/OPatch/ocm/bin/ocm.rsp
OPatch Automation Tool
Copyright (c)2014, Oracle Corporation. All rights reserved.

OPatchauto Version : 12.1.0.1.8
OUI Version        : 12.1.0.2.0
Running from       : /usr/app/12.1.0/grid

opatchauto log file: /usr/app/12.1.0/grid/cfgtoollogs/opatchauto/20996835/opatch_gi_2015-08-08_19-41-59_deploy.log

Parameter Validation: Successful

Configuration Validation: Successful

Patch Location: /usr/patches/20996835
Grid Infrastructure Patch(es): 20831110 19872484 20299018 20831113
DB Patch(es): 20831110 20831113

Patch Validation: Successful
Grid Infrastructure home:
/usr/app/12.1.0/grid
DB home(s):
/usr/app/oracle/product/12.1.0


Performing prepatch operations on CRS Home... Failed
Command "/usr/app/12.1.0/grid/perl/bin/perl -I/usr/app/12.1.0/grid/perl/lib -I/usr/app/12.1.0/grid/OPatch/opatchautotemp_Sajidserver01/patchwork/crs/install

/usr/app/12.1.0/grid/OPatch/opatchautotemp_Sajidserver01/patchwork/crs/install/rootcrs.pl -prepatch" execution failed:
Died at /usr/app/12.1.0/grid/OPatch/opatchautotemp_Sajidserver01/patchwork/crs/install/crspatch.pm line 773.


For more details, please refer to the log file "/usr/app/12.1.0/grid/cfgtoollogs/opatchauto/20996835/opatch_gi_2015-08-08_19-41-59_deploy.debug.log".

Apply Summary:

Following patch(es) failed to be installed:
GI Home: /usr/app/12.1.0/grid: 20831110,19872484,20299018,20831113
DB Home: /usr/app/oracle/product/12.1.0: 20831110,20831113
Command failure exception

opatchauto failed with error code 2.


Log file Location for the failed command: /usr/app/12.1.0/grid/cfgtoollogs/opatch/opatch2015-08-08_18-13-01PM_1.log

For more details, please refer to the log file "/usr/app/12.1.0/grid/cfgtoollogs/opatchauto/20996835/opatch_gi_2015-08-08_18-05-18_deploy.debug.log".

Apply Summary:
Following patch(es) are successfully installed:
DB Home: /usr/app/oracle/product/12.1.0: 20831110,20831113

Following patch(es) failed to be installed:
GI Home: /usr/app/12.1.0/grid: 20831110,19872484,20299018,20831113
Command failure exception

opatchauto failed with error code 2.

#################################################################################

####
Bug
####
PROC-28: Oracle Cluster Registry already in current version
CRS-1153: There was an error setting Oracle Clusterware to rolling patch mode.
CRS-4000: Command Start failed, or completed with errors.
2015/08/09 12:21:06 CLSRSC-430: Failed to start rolling patch mode

#######
Action
#######

CRS-1153: There was an error setting Oracle Clusterware to rolling patch mode. (Doc ID 1943498.1)

crsctl stop rolling patch


#################################################################################

Applying patch(es) to "/usr/app/12.1.0/grid" ...
Command "/usr/app/12.1.0/grid/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_grid_patchList -local  -invPtrLoc /usr/app/12.1.0/grid/oraInst.loc -oh

/usr/app/12.1.0/grid -silent -ocmrf /usr/app/oracle/product/12.1.0/OPatch/ocm/bin/ocm.rsp" execution failed:
UtilSession failed:
Prerequisite check "CheckApplicable" failed.

Log file Location for the failed command: /usr/app/12.1.0/grid/cfgtoollogs/opatch/opatch2015-08-09_13-55-15PM_1.log

For more details, please refer to the log file "/usr/app/12.1.0/grid/cfgtoollogs/opatchauto/20996835/opatch_gi_2015-08-09_13-44-28_deploy.debug.log".


[Aug 9, 2015 1:55:19 PM]     Patch 20831113:
                             onewaycopyAction : Source File "/usr/patches/20996835/20831113/files/crs/install/dropdb.pl" does not exists or is not readable
                             'oracle.crs, 12.1.0.2.0': Cannot copy file from 'dropdb.pl' to '/usr/app/12.1.0/grid/crs/install/dropdb.pl'
[Aug 9, 2015 1:55:19 PM]     Prerequisite check "CheckApplicable" failed.
                             The details are:

                             Patch 20831113:
                             onewaycopyAction : Source File "/usr/patches/20996835/20831113/files/crs/install/dropdb.pl" does not exists or is not readable
                             'oracle.crs, 12.1.0.2.0': Cannot copy file from 'dropdb.pl' to '/usr/app/12.1.0/grid/crs/install/dropdb.pl'
[Aug 9, 2015 1:55:19 PM]     OUI-67073:UtilSession failed:
                             Prerequisite check "CheckApplicable" failed.
[Aug 9, 2015 1:55:19 PM]     Finishing UtilSession at Sun Aug 09 13:55:19 EDT 2015
[Aug 9, 2015 1:55:19 PM]     Log file location: /usr/app/12.1.0/grid/cfgtoollogs/opatch/opatch2015-08-09_13-55-15PM_1.log
[Aug 9, 2015 1:55:19 PM]     Stack Description: java.lang.RuntimeException:
                             Prerequisite check "CheckApplicable" failed.

Log file Location for the failed command: /usr/app/12.1.0/grid/cfgtoollogs/opatch/opatch2015-08-09_13-55-15PM_1.log

2015-08-09_13-55-25 :
Home Type of the current Oracle Home is : rac
2015-08-09_13-55-37 :
ERROR : PRKH-1010 : Unable to communicate with CRS services.
PRKH-3003 : An attempt to communicate with the CSS daemon failed

2015-08-09_13-55-37 :
Database "SAJIDDB" is not running. PDB instances on the current node will not be checked.
2015-08-09_13-55-49 :
ERROR : PRKH-1010 : Unable to communicate with CRS services.
PRKH-3003 : An attempt to communicate with the CSS daemon failed

2015-08-09_13-55-49 :
#################################################################################

########
Method 1
########

After fixing the cause of failure you have two options shown below:
Run 'opatchauto resume'
  or

[root@Sajidserver01 bin]# pwd
/usr/app/12.1.0/grid/bin
[root@Sajidserver01 bin]# crsctl stop crs
              :
              :=====>Output Truncated
              :

         completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@Sajidserver01 bin]#  su - grid

#############################
Make Sure Nothing is running
############################
[grid@Sajidserver01 ~]$ ps -ef|grep grid
[grid@Sajidserver01 ~]$ ps -ef|grep tns
[grid@Sajidserver01 ~]$ ps -ef|grep d.bin
[grid@Sajidserver01 ~]$ exit

[root@Sajidserver01 bin]#  su - root
[root@Sajidserver01 ~]# cd /usr/app/12.1.0/grid/
[root@Sajidserver01 grid]# cd OPatch/
[root@Sajidserver01 OPatch]# ./opatchauto resume
OPatch Automation Tool
Copyright (c)2014, Oracle Corporation. All rights reserved.

OPatchauto Version : 12.1.0.1.8
OUI Version        : 12.1.0.2.0
Running from       : /usr/app/12.1.0/grid

OPatchauto will attempt to resume the last run session. This might take several minutes...

opatchauto log file: /usr/app/12.1.0/grid/cfgtoollogs/opatchauto/20996835/opatch_gi_2015-08-10_10-14-02_dep                                                                                

        loy.log

Executing command:
/usr/app/12.1.0/grid/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_grid_patchList -local  -invPtrLoc /                                                                                

        usr/app/12.1.0/grid/oraInst.loc -oh /usr/app/12.1.0/grid -silent -ocmrf /usr/app/oracle/product/12.1.0/OPat                                                                        

                ch/ocm/bin/ocm.rsp
 as owner "grid"
... above command successful.

Executing command:
/usr/app/12.1.0/grid/rdbms/install/rootadd_rdbms.sh
 as owner "root"
... above command successful.

Executing command:
/usr/app/12.1.0/grid/perl/bin/perl -I/usr/app/12.1.0/grid/perl/lib -I/usr/app/12.1.0/grid/OPatch/opatchauto                                                                                

        temp_Sajidserver01/patchwork/crs/install /usr/app/12.1.0/grid/OPatch/opatchautotemp_Sajidserver01/patchwork/crs/insta                                                                        

                ll/rootcrs.pl -postpatch
 as owner "root"
... above command successful.

Executing command:
/bin/bash -c 'ORACLE_HOME=/usr/app/oracle/product/12.1.0 ORACLE_SID=Sajid02 /usr/app/oracle/product/12.1.0                                                                                

        /OPatch/datapatch'
 as owner "oracle"
... above command successful.


OPatchauto was able to resume from the previous patching session and complete successfully.

opatchauto succeeded.


#######################################
Manually run the commands listed below
#######################################

##################
Method 2
##################


/usr/app/12.1.0/grid/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_grid_patchList -local  -invPtrLoc /usr/app/12.1.0/grid/oraInst.loc -oh /usr/app/12.1.0/gri
d -silent -ocmrf /usr/app/oracle/product/12.1.0/OPatch/ocm/bin/ocm.rsp (Run as grid) - (TRIED BUT FAILED)

/usr/app/12.1.0/grid/rdbms/install/rootadd_rdbms.sh (Run as root)

/usr/app/12.1.0/grid/perl/bin/perl -I/usr/app/12.1.0/grid/perl/lib -I/usr/app/12.1.0/grid/OPatch/opatchautotemp_Sajidserver03/patchwork/crs/install /usr/app/12.1.0/gri
d/OPatch/opatchautotemp_Sajidserver03/patchwork/crs/install/rootcrs.pl -postpatch (Run as root)

/bin/bash -c 'ORACLE_HOME=/usr/app/oracle/product/12.1.0 ORACLE_SID=Sajid03 /usr/app/oracle/product/12.1.0/OPatch/datapatch' (Run as oracle)

##########################################################

[oracle@Sajidserver03 install]$ chmod 775 dropdb.pl
[oracle@Sajidserver03 install]$ pwd
/usr/patches/20996835/20831113/files/crs/install

-rwx------ 1 oracle oinstall   3541 Jul 12 07:04 dropdb.pl

##################
Check Nothing Runs
##################

[grid@Sajidserver02 ~]$ ps -ef|grep d.bin

[grid@Sajidserver02 ~]$ ps -ef|grep grid

[grid@Sajidserver02 ~]$ ps -ef|grep pmon

[grid@Sajidserver02 ~]$ ps -ef|grep tns

###########################################################################


[grid@Sajidserver03 ~]$ /usr/app/12.1.0/grid/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_grid_patchList -local  -invPtrLoc /usr/app/12.1.0/grid/oraInst.loc -oh /usr/app/12.1.0/grid -

silent -ocmrf /usr/app/oracle/product/12.1.0/OPatch/ocm/bin/ocm.rsp
Oracle Interim Patch Installer version 12.1.0.1.8
Copyright (c) 2015, Oracle Corporation.  All rights reserved.


Oracle Home       : /usr/app/12.1.0/grid
Central Inventory : /usr/app/oraInventory
   from           : /usr/app/12.1.0/grid/oraInst.loc
OPatch version    : 12.1.0.1.8
OUI version       : 12.1.0.2.0
Log file location : /usr/app/12.1.0/grid/cfgtoollogs/opatch/opatch2015-08-09_19-03-01PM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   19872484  20299018  19769480  20299023  20831110  20831113

Do you want to proceed? [y|n]
Y (auto-answered by -silent)
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/usr/app/12.1.0/grid')


Is the local system ready for patching? [y|n]
Y (auto-answered by -silent)
User Responded with: Y
Backing up files...
Applying interim patch '19872484' to OH '/usr/app/12.1.0/grid'

Patching component oracle.wlm.dbwlm, 12.1.0.2.0...

Verifying the update...
Applying interim patch '20299018' to OH '/usr/app/12.1.0/grid'

Patching component oracle.usm, 12.1.0.2.0...

Verifying the update...
Applying sub-patch '19769480' to OH '/usr/app/12.1.0/grid'
ApplySession: Optional component(s) [ oracle.xdk, 12.1.0.2.0 ] , [ oracle.oraolap, 12.1.0.2.0 ]  not present in the Oracle Home or a higher version is found.

              :
              :=====>Output Truncated
              :
Verifying the update...
Composite patch 20831110 successfully applied.
Patches 19872484,20299018,20831113 successfully applied.
Log file location: /usr/app/12.1.0/grid/cfgtoollogs/opatch/opatch2015-08-09_19-03-01PM_1.log

OPatch succeeded.

[Sajid@Sajidserver03 ~]$  su - root
[root@Sajidserver03 ~]# cd /usr/app/12.1.0/grid/rdbms/install
[root@Sajidserver03 install]# ls -ltr
[root@Sajidserver03 install]# ./rootadd_rdbms.sh
[root@Sajidserver03 install]# exit
logout
[Sajid@Sajidserver03 ~]$  su - oracle
[oracle@Sajidserver03 ~]$ exit
logout
[Sajid@Sajidserver03 ~]$  su - root
[root@Sajidserver03 ~]# /usr/app/12.1.0/grid/perl/bin/perl -I/usr/app/12.1.0/grid/perl/lib -I/usr/app/12.1.0/grid/OPatch/opatchautotemp_Sajidserver03/patchwork/crs/install

/usr/app/12.1.0/grid/OPatch/opatchautotemp_Sajidserver03/patchwork/crs/install/rootcrs.pl -postpatch
Using configuration parameter file: /usr/app/12.1.0/grid/OPatch/opatchautotemp_Sajidserver03/patchwork/crs/install/crsconfig_params
2015/08/09 19:08:59 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.

2015/08/09 19:09:22 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.

CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.

              :
              :=====>Output Truncated
              :

Oracle Clusterware active version on the cluster is [12.1.0.2.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [0].

[Sajid@Sajidserver03 ~]$  su - oracle

[oracle@Sajidserver03 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Sun Aug 9 19:13:04 2015

Copyright (c) 1982, 2014, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options

SYS@Sajid03>exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options
[oracle@Sajidserver03 ~]$ /bin/bash -c 'ORACLE_HOME=/usr/app/oracle/product/12.1.0 ORACLE_SID=Sajid03 /usr/app/oracle/product/12.1.0/OPatch/datapatch'
SQL Patching tool version 12.1.0.2.0 on Sun Aug  9 19:13:34 2015
Copyright (c) 2015, Oracle.  All rights reserved.

Connecting to database...OK
Note:  Datapatch will only apply or rollback SQL fixes for PDBs
       that are in an open state, no patches will be applied to closed PDBs.
       Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
       (Doc ID 1585822.1)
Determining current state...done
Adding patches to installation queue and performing prereq checks...done
Installation queue:
  For the following PDBs: CDB$ROOT PDB$SEED
    Nothing to roll back
    The following patches will be applied:
      20831110 (Database Patch Set Update : 12.1.0.2.4 (20831110))

Installing patches...
Patch installation complete.  Total patches installed: 2

Validating logfiles...done
SQL Patching tool complete on Sun Aug  9 19:17:46 2015
[oracle@Sajidserver03 ~]$ exit



No comments:

Post a Comment