Patch db node(linux server node) for ExaCC from oel6 to oel7

Updating the Exadata System Software on database servers is a process that entails patching the entire Oracle Linux operating system, and related Exadata software rather than just updating single packages.

Updating the operating system does not use a traditional “yum update” command but uses a cli tool, patchmgr.

Through patchmgr, you can update the Exadata System Software on database servers with an ISO image staged locally on the database servers or on a separate infrastructure instance.

Go to any database running on the cluster and check for available EPSU patches.

Once you see the available patches, try to do the precheck for the patch you decided for your environment.

Once you run the precheck go to the following location.in /u02

On node 1 and node 2 of the cluster and check, if any file named dbserver.patch.zip_exadata_ol7_19.2.14.0.0.200517_Linux-x86-64.zip is generated there or not.

Here node1 and node 2 are the driving system, hence these folders will only get generated on these nodes.

Patchmgr is run on a “driving system”, which is an Exadata database server. Patchmgr to run from a central server to update multiple Exadata systems, or to update all database servers of a single Exadata system with a single patchmgr invocation.

 Go to /u02 and verify the directory is created or not 


[root@parwxExaCC201vm02 ~]# cd /u02/
[root@parwxExaCC201vm02 u02]#
[root@parwxExaCC201vm02 u02]# ls -ltr
total 32
drwx------ 2 oracle oinstall 16384 Oct 12  2019 lost+found
lrwxrwxrwx 1 root   root        16 Oct 13  2019 app_acfs -> /acfs01/app_acfs
drwxr-xr-x 4 oracle oinstall  4096 Feb 20 08:40 opt
drwxrwxr-x 6 oracle oinstall  4096 May 29 00:17 app
drwxr-xr-x 3 root   root      4096 Jul 11 18:42   dbserver.patch.zip_exadata_ol7_19.2.14.0.0.200517_Linux-x86-64.zip
drwxr----- 3 root   root      4096 Jul 11 19:44 dbnodeupdate

IF you don’t see any file liked above getting generated there, most probably Oracle support hasn’t staged that particular patch in the OSS.

So, raise a SR with Oracle asking them about the issue, that despite after running precheck you don’t see the patch directory is being created under /u02.

If the patch is not there that precheck will fail.

Following reason could make the pre check to fail:
1. DB or ASM home should be at supported minimum patch level  
ERROR: Database home /u02/app/oracle/product/12.1.0/dbhome_2 is at 12.1.0.2.180717 and needs to be at level 12.1.0.2.180831 or later in combination with a certified Grid Infrastructure before proceeding the update. See MOS 888828.1 for requirements


 2. There should be enough free space in /u01 for unzipping the dbnode patch at least make sure we have 7.5G free space in /u01  
/u01/dbnodeupdate.patchmgr/dbnodeupdate.sh][DiaryEntry][] Entering PrintGenError Not enough     free space available  in /u01/dbnodeupdate.patchmgr/iso.stage to unzip exadata_ol7_19.2.14.0.0.200517_Linux-x86-64.zip
 
3.Custom rpm’s needs to be removed before a major os upgrade. 
ERROR: Custom packages found that require removal before updating to another major Oracle Linux release, see ecc106vm01:/var/log/cellos/unkown_packages-rpt.280620130428.txt for more details.

Go to the directory /var/log/cellos on each node and do ls –ltr
        You will see file like
        unkown_packages-rpt.280620130428.txt
        unkown_packages-rpt.280620130428.sh   ( run this .sh file )

Execute ./ unkown_packages-rpt.280620130428.sh from all the nodes, it will remove the  custome rpms

To verify verify if it is removed or nor , run
[root@ecc208vm02 ~]# dbaascli
-bash: dbaascli: command not found (That means the rpm’s are removed successfully)

Once you fix the above issue, please retry the precheck .

Once the Precheck is completed, please try for patching the system with EPSU

Once you initiate the patch from the console and you want to check the progress of the patch, please monitor the patchmgr.log locate in the below directory. 


 [root@parwxExaCC201vm02 patchmgr_log.2020-07-11]# ls –ltr


 -rw-r--r-- 1 root root     178 Jul 11 19:40 patchmgr.stderr
 drwxr-xr-x 2 root root    4096 Jul 11 19:40 tmp
 -rw-r--r-- 1 root root  770639 Jul 11 19:40 patchmgr.trc
 -rw-r--r-- 1 root root  235908 Jul 11 19:40 patchmgr.log
 [root@parwxExaCC201vm02 patchmgr_log.2020-07-11_16.31.56]#




[root@parwxExaCC201vm02 patchmgr_log.2020-07-11_16.31.56]# pwd
/u02/dbserver.patch.zip_exadata_ol7_19.2.14.0.0.200517_Linux-x86-64.zip/dbserver_patch_20.200518/patchmgr_log.2020-07-11_16.31.56


tail -100f /u02/dbserver.patch.zip_exadata_ol7_19.2.14.0.0.200517_Linux-x86-64.zip/dbserver_patch_20.200518/patchmgr_log.2020-07-11_16.31.56

After the patch is applied successfully, in the activity log of console url, you can see the successful patch applied message.

Jul 2, 2020 9:19:19 PM UTC    Phase initialize completed
Jul 2, 2020 9:19:19 PM UTC    APPLY_PATCH job [3567894] initiated by  [PAlam@exaCC.com] started…
Jul 2, 2020 10:13:01 PM UTC    Patching of Database Cloud Service with patch [31205000] completed successfully to version [19.2.14.0.0.200517].
Jul 2, 2020 10:13:02 PM UTC    Phase patch completed
Jul 2, 2020 10:13:03 PM UTC    Completed
Jul 2, 2020 10:13:03 PM UTC    Activity Ended

Re-install Exadata Cloud RPMs


To re-install the Exadata Cloud RPMs, follow the following steps:
Download the zip from patch 29542072 and stage it on your service. 

You can find this patch in MOS. Once found, use the download modal window and download it to your local computer or use the WGET options to download it directly to your ExaCS database server. 

If downloading it locally, then stage it on your ExaCS via a tool such as SCP.
Place the patch zip file in /tmp.
unzip p29542072_191200_Generic.zip
cd 29542072


The following RPMs should be present:

cx_Oracle-5.2.f982f7d-orcl121.el6.x86_64.rpm
dbaastools_exa-1.0-1+18.2.3.1.0_190315.1456.x86_64.rpm
dbcs-agent-2.7OL7-18.2.5.2EXABM_190320.1249.x86_64.rpm
dbcs-agent-update-2.16-18.2.5.2EXABM_190320.1249.x86_64.rpm
dbtools_jdk-1.8.0.192-1.el7.x86_64.rpm
dnsmasq-2.76-7.el7.x86_64.rpm
libxenstore-4.7.0-4.el7.x86_64.rpm
oracle-instantclient12.2-basic-12.2.0.1.0-1.x86_64.rpm
perl-JSON-2.59-2.el7.noarch.rpm
perl-JSON-PP-2.27202-2.el7.noarch.rpm
perl-TermReadKey-2.30-20.el7.x86_64.rpm
perl-Time-HiRes-1.9725-3.el7.x86_64.rpm
python-cx_Oracle-6.4.1-1.el7.x86_64.rpm
wget-1.14-15.el7_4.1.x86_64.rpm


Apply the RPMs to your service starting with perl-JSON-2.59-2.el7.noarch.rpm.


 rpm -ivh perl-JSON-2.59-2.el7.noarch.rpm
 You can now apply the remaining rpms.
 $ rpm -ivh cx_Oracle-5.2.f982f7d-orcl121.el6.x86_64.rpm
 $ rpm -ivh dbtools_jdk-1.8.0.192-1.el7.x86_64.rpm
 $ rpm -ivh libxenstore-4.7.0-4.el7.x86_64.rpm
 $ rpm -ivh dnsmasq-2.76-7.el7.x86_64.rpm
 $ rpm -ivh dbaastools_exa-1.0-1+18.2.3.1.0_190315.1456.x86_64.rpm
 $ rpm -ivh dbcs-agent-2.7OL7-18.2.5.2EXABM_190320.1249.x86_64.rpm
 $ rpm -ivh dbcs-agent-update-2.16-18.2.5.2EXABM_190320.1249.x86_64.rpm
 Referenced doc 
How to update the Exadata System Software (DomU) to 19 from 18 on the Exadata Cloud Service in OCI (Doc ID 2521053.1)
Read me for rpm installations 

 328 total views,  1 views today