Different location of datafile, errors RMAN-06094: datafile 31 must be restored ,RMAN-06571: datafile 31 does not have recoverable copy

I was trying to sync my standby database with incremental backup from production due to huge archive log gap of more than 500 archive.

While trying to catalog the datafile for switching, the switching was failing with

 RMAN> switch database to copy ;
 RMAN-00571: ===========================================================
 RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
 RMAN-00571: ===========================================================
 RMAN-03002: failure of switch to copy command at 03/16/2021 14:02:49
 RMAN-06571: datafile 1 does not have recoverable copy 

When I was trying to catalog the absolute location of datafile , It was failing to catalog saying with no files found  .

I tried to go to the asm location to check if the datafile location are correct

When I logged in via asmcmd , I observer the datafiles for this database were present inside two different directory under the diskgroup , which is why the cataloging of datafiles were failing with no known files found

 ASMCMD> cd standby_diskgrp/
 ASMCMD> ls
 datafile1/
 datafile2/

 ASMCMD> cd datafile1/
 ASMCMD> ls
 CONTROLFILE/
 DATAFILE/
 ONLINELOG/
 PARAMETERFILE/
 TEMPFILE/
 spfile.ora


 ASMCMD> cd datafile2/
 ASMCMD> ls
 CONTROLFILE/
 DATAFILE/
 ONLINELOG/
 dataguardconfig/

When I checked , how many datafiles were present in each of datafile folder , I found only 4 datafiles were created in one directory( here datafile2) and rest were in a separate directory (datafile1)

 ASMCMD> cd datafile2/DATAFILE/
 ASMCMD> ls
 XXX_CSM_TS.317.1066941589
 XXX_MIG_TS.316.1063308261
 DWHINFA_DATA.312.964888803
 DWHINFA_DATA.313.964888805

 ASMCMD> cd datafile1/DATAFILE/ 
  XXXX_TS.286.917853275
  XXXX_TS.287.917853275
  XXXX_TS.288.917853275
  XXXX_TS.289.917853273
  PRICE_IDX_TS.280.917853289
  PRICE_TS.274.917853283
  NOLOGGING_TS.276.917853285
  PROFILE_MANAGER_IDX_TS.282.917853291
  PROFILE_MANAGER_TS.277.917853285
  SYSAUX.260.911995807
  SYSTEM.259.911995793
  UNDOTBS1.261.911995815
  UNDOTBS2.263.911995879
  USERS.264.911995883

This happened because someone didn’t mentioned the correct db_file_name_convert parameter earlier and the datafile got created in a separate directory , when it added in production .

Prior to catalogin if I try to restore or switch the datafile , I was getting the below errors

 RMAN-00571: ===========================================================
 RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
 RMAN-00571: ===========================================================
 RMAN-03002: failure of recover command at 03/16/2021 13:57:30
 RMAN-06094: datafile 31 must be restored

Tried switching the datafile as well

 RMAN> SWITCH DATAFILE 31 to copy ;
 RMAN-00571: ===========================================================
 RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
 RMAN-00571: ===========================================================
 RMAN-03002: failure of switch to copy command at 03/16/2021 14:02:49
 RMAN-06571: datafile 31 does not have recoverable copy

Finally we tried to catalog the whole diskgroup to check which files they are cataloging successfully, It seems to catalog the misplaced 4 datafiles which were in datafile2 folder

  RMAN> catalog start with '+DATA' ;
  searching for all files that match the pattern +DATA
  List of Files Unknown to the Database

 List of Cataloged Files
 File Name: +DATA/datafile2/CONTROLFILE/Backup.315.1000576529
 File Name: +DATA/datafile2/DATAFILE/DWHINFA_DATA.312.964888803
 File Name: +DATA/datafile2/DATAFILE/DWHINFA_DATA.313.964888805
 File Name: +DATA/datafile2/DATAFILE/XXX_MIG_TS.316.1063308261
 File Name: +DATA/datafile2/DATAFILE/XXX_CSM_TS.317.1066941589

Now go to the production database and cross verify the file_id of these 4 datafiles , because we need to switch them to copy

Now check in prod which datafile number these 4 datafile haves
 ASMCMD> cd DATAFILE/
 ASMCMD> ls
 XXX_CSM_TS.317.1066941589
 XXX_MIG_TS.316.1063308261
 DWHINFA_DATA.312.964888803
 DWHINFA_DATA.313.964888805
 these were 
 +DATA/DATA/datafile/dwhinfa_data.300.964888803               32 AVAILABLE
 +DATA/DATA/datafile/dwhinfa_data.299.964888801               31 AVAILABLE
 +DATA/DATA/datafile/system.2XXX_mig_ts01.dbf                33 AVAILABLE
 +DATA/DATA/datafile/system.2XXX_csm_ts01.dbf                34 AVAILABLE

Once matches , now again try to switch these 4 datafiles to copy( for pointing the correct datafile location in the restored control file ) once they are successfully cataloged

RMAN>  switch datafile 31 to copy ;
 datafile 31 switched to datafile copy "+DATA/datafile1/datafile/dwhinfa_data.312.964888803"
 RMAN> switch datafile 32 to copy ;
 datafile 32 switched to datafile copy "+DATA/datafile1/datafile/dwhinfa_data.313.964888805"
 RMAN> switch datafile 33 to copy ;
 using target database control file instead of recovery catalog
 datafile 33 switched to datafile copy "+DATA/datafile1/datafile/XXX_mig_ts.316.1063308261"
 RMAN> switch datafile 34 to copy ;
 datafile 34 switched to datafile copy "+DATA/datafile1/datafile/XXX_csm_ts.317.1066941589"

After performing the above steps , the recover command succeeded

oracle@server1:/home/oracle>rman target /
 Recovery Manager: Release 11.2.0.4.0 - Production on Tue Mar 16 14:11:24 2021
 Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.
 connected to target database: DATA (DBID=XXX, not open)
 RMAN> RUN
 {
 ALLOCATE CHANNEL tape1 DEVICE TYPE disk;
 ALLOCATE CHANNEL tape2 DEVICE TYPE disk;
 ALLOCATE CHANNEL tape3 DEVICE TYPE disk;
 ALLOCATE CHANNEL tape4 DEVICE TYPE disk;
 recover database noredo;
 }
 EXIT;2> 3> 4> 5> 6> 7> 8>
 using target database control file instead of recovery catalog
 allocated channel: tape1
 channel tape1: SID=386 instance=DATA1 device type=DISK
 allocated channel: tape2
 channel tape2: SID=1489 instance=DATA1 device type=DISK
 allocated channel: tape3
 channel tape3: SID=579 instance=DATA1 device type=DISK
 allocated channel: tape4
 channel tape4: SID=629 instance=DATA1 device type=DISK
 Starting recover at 16-MAR-21
 channel tape1: starting incremental datafile backup set restore
 channel tape1: specifying datafile(s) to restore from backup set
 destination for restore of datafile 00003: +DATA/DATA/datafile/undotbs1.261.911995815
 destination for restore of datafile 00005: +DATA/DATA/datafile/users.264.911995883
 destination for restore of datafile 00010: +DATA/DATA/datafile/system.2XXXX_ts05.dbf
 destination for restore of datafile 00015: /software/EXPD/stndby_sync/standby_DATA_tevpsj8m_1_1_1067339030
 channel tape2: starting incremental datafile backup set restore
 channel tape3: reading from backup piece /software/EXPD/stndby_sync/standby_DATA_tfvpsj8n_1_1_1067339031
 channel tape3: piece handle=/software/EXPD/stndby_sync/standby_DATA_tfvpsj8n_1_1_1067339031 tag=FORSTANDBY
 channel tape3: restored backup piece 1
 channel tape3: restore complete, elapsed time: 00:06:36
 channel tape2: piece handle=/software/EXPD/stndby_sync/standby_DATA_tgvpsj8n_1_1_1067339031 tag=FORSTANDBY
 channel tape2: restored backup piece 1
 channel tape2: restore complete, elapsed time: 00:07:16

 194 total views,  5 views today

Leave a Reply

Your email address will not be published. Required fields are marked *