Due to capacity issues 2 extra storage cells had to be added to the already 3 licensed high performance X3 cells on the Super Cluster T4. Some details about the installation Cellsrv 12.1.2.1.0 GI 12.1.0.2.4 DB 11.2.0.4 The process was quite straight forwards and is documented here in summary : create cell disks create grid disks on them exactly the same as on the existing cells with the same offsets for each disk group here an example of the first disk group on our cells 


cellcli -e  "LIST GRIDDISK WHERE NAME LIKE 'DATA_DG.*' ATTRIBUTES NAME, SIZE ,STATUS, ASMMODESTATUS, ASMDEACTIVATIONOUTCOME,OFFSET"
         DATA_DG_CD_00_celadm01   423G    active  ONLINE  Yes     32M
         DATA_DG_CD_01_celadm01   423G    active  ONLINE  Yes     32M
         DATA_DG_CD_02_celadm01   423G    active  ONLINE  Yes     32M
         DATA_DG_CD_03_celadm01   423G    active  ONLINE  Yes     32M
         DATA_DG_CD_04_celadm01   423G    active  ONLINE  Yes     32M
         DATA_DG_CD_05_celadm01   423G    active  ONLINE  Yes     32M
         DATA_DG_CD_06_celadm01   423G    active  ONLINE  Yes     32M
         DATA_DG_CD_07_celadm01   423G    active  ONLINE  Yes     32M
         DATA_DG_CD_08_celadm01   423G    active  ONLINE  Yes     32M
         DATA_DG_CD_09_celadm01   423G    active  ONLINE  Yes     32M
         DATA_DG_CD_10_celadm01   423G    active  ONLINE  Yes     32M
         DATA_DG_CD_11_celadm01   423G    active  ONLINE  Yes     32M

ok we did that that way the configuration was exactly the same on the cells. on the database node we added the ip’s of the newly added cells. I was a bit confused by the fact that nothing needed to be added on the storage cells because I recalled that there was a functionality introduced in one of the more recent storage cell versions that made them “talk” to each other on rebalance operations. But adding the ip’s in cellip.ora on the db node was enough and then we were ready to rebalance the disk group 


ALTER DISKGROUP DATA_DG ADD 
DISK 'o/192.168.15.104/DATA_DG*' 
DISK 'o/192.168.25.105/DATA_DG*' 
rebalance power 32 
;

and kaboum 


ERROR at line 1: 
ORA-00600: internal error code, arguments: [kfgpCreate_60], [12], [0], [65535], 
[65535], [65535], [65535], [], [], [], [], []

luckily asm was still running but being already pretty late after a day of migrations this was not funny at all. after chatting with support they suggested to disable appliance.mode on the disk group 


 ALTER DISKGROUP DATA_DG SET ATTRIBUTE 'appliance.mode'='FALSE';

after we did that the addition of the disks worked 


ALTER DISKGROUP DATA_DG ADD 
DISK 'o/192.168.15.104/DATA_DG*' 
DISK 'o/192.168.25.105/DATA_DG*' 
rebalance power 32 
; 

Diskgroup altered.

When the rebalance was finished we put appliance.mode again to TRUE 


ALTER DISKGROUP DATA_DG SET ATTRIBUTE 'appliance.mode'='TRUE';

more info can be found in note EXADATA : What Is Oracle ASM appliance.mode Attribute (Doc ID 1664261.1) So it looks like the appliance.mode is an optimisation I wonder if it is the fact that cells can communicate directly instead of using the host for rebalance operations, however that is not literally stated in the metalink note. hope this helps if you encounter this

On By phil

Leave a Reply

Related Post