Today I encountered a very weird unexpected issue on our 10GR2 (10203 EE database running on solaris 64 sparc).
I wanted to clean up some archives from another incarnarnation in the archive disk group, so decided to catalog them in RMAN and the plan was to afterwards delete them.
everything went fine until I saw this in the alert.log of the primary
New incarnation branch detected in ArchiveLog, filename +ARCHDG1/DBPRODNL/archivelog/2011_05_26/thread_1_seq_837.8222.752112659
Inspection of file changed rdi from 1 to 4
Setting recovery target incarnation to 4
Mon Aug 22 14:57:11 2011
Setting recovery target incarnation to 4
RMAN list incarnation and v$datababase_incarnation also showed that we went to another incarnation :
DB Key Inc Key DB Name DB ID STATUS Reset SCN Reset Time
——- ——- ——– —————- — ———- ———-
1 1 DBPROD 2789041670 PARENT 1 29-NOV-07
5 5 DBPROD 2789041670 ORPHAN 6238221940 13-MAY-11
4 4 DBPROD 2789041670 CURRENT 6274083013 17-MAY-11
2 2 DBPROD 2789041670 ORPHAN 6468809687 20-JUN-11
3 3 DBPROD 2789041670 ORPHAN 6508613920 29-JUN-11
v$database_incarnation showed similar output.
As soon as the primary found the archives the RFS started to send them to the standby db, which stopped recovery immediately
Mon Aug 22 15:04:38 2011
RFS[96]: Archived Log: ‘+ARCHDG1/DBPRODBE/archivelog/2011_08_22/thread_2_seq_42.11513.759855877’
Mon Aug 22 15:06:41 2011
RFS[94]: Archived Log: ‘+ARCHDG1/DBPRODBE/archivelog/2011_08_22/thread_2_seq_2.11991.759856001’
Mon Aug 22 15:06:43 2011
RFS[96]: Archived Log: ‘+ARCHDG1/DBPRODBE/archivelog/2011_08_22/thread_2_seq_1.11962.759856003’
Mon Aug 22 15:07:56 2011
MRP0: Incarnation has changed! Retry recovery…
Mon Aug 22 15:07:56 2011
Errors in file /u01/oracle/admin/DBPRODBE/bdump/DBPRODn1_mrp0_27062.trc:
ORA-19906: recovery target incarnation changed during recovery
Mon Aug 22 15:07:56 2011
Managed Standby Recovery not using Real Time Apply
Mon Aug 22 15:07:59 2011
Recovery interrupted!
Recovered data files to a consistent state at change 6792290587
oh crap now standby’s are not following anymore ….
what does the incarnation look like on the standby :
RMAN> list incarnation;
using target database control file instead of recovery catalog
List of Database Incarnations
DB Key Inc Key DB Name DB ID STATUS Reset SCN Reset Time
——- ——- ——– —————- — ———- ———-
1 1 DBPROD 2789041670 PARENT 1 29-NOV-07
2 2 DBPROD 2789041670 CURRENT 6274083013 17-MAY-11
after investigation further on the primary I noticed that the reset logs time in v$database
and the reset logs time of the current incarnation indicated by v$database_incarnation , list incarnation.
select RESETLOGS_TIME from v$database;
RESETLOGS
———
29-NOV-07this is the output from list incarnation on the primary
while list incarnation in rman give us following :
Copyright (c) 1982, 2005, Oracle. All rights reserved.
lconnected to target database: DBPROD (DBID=2789041670)
RMAN> list incarnation;
using target database control file instead of recovery catalog
List of Database Incarnations
DB Key Inc Key DB Name DB ID STATUS Reset SCN Reset Time
——- ——- ——– —————- — ———- ———-
1 1 DBPROD 2789041670 PARENT 1 29-NOV-07
5 5 DBPROD 2789041670 ORPHAN 6238221940 13-MAY-11
4 4 DBPROD 2789041670 CURRENT 6274083013 17-MAY-11
2 2 DBPROD 2789041670 ORPHAN 6468809687 20-JUN-11
3 3 DBPROD 2789041670 ORPHAN 6508613920 29-JUN-11RMAN> exit
weird things happening in the end on the standby we decided to change the incarnation to the first one
rman target /
reset database to incarnation 1;
after which we could reenable the recovery, and the application of redo continued.
The primary is still in a somewhat ambiguous state oracle support is working on this to see how we can corret this, it is working correctly for the moment.
keep you posted for the explanation why this exactly happened….
UPDATE :
After lots of updates on the SR Oracle support told me this is basically the expected behaviour, if rman finds archives of another newer incarnation while cataloging it will change the incarnation of the database.
The workaround for my issue with the primary is to reset the incarnation to 1 in Rman.
