The customer I am working for has OVM on Exadata for the dev qualification and testing Exadata we recently added two compute nodes. The main driver was RAM, it was less ( or equally priced) expensive to add two new nodes then extending the RAM ( removing existing DIMM and change them with higher capacity ones)
We now have : 4 X6 nodes and two brand new X7 nodes (we just got them when X8 was released 🙁 ) there are quite some database running here and we since they are using the same cells no need to duplicate them.
My colleague Freek D’hooge pointed me to this document : Moving a User Domain to a Different Database Server
Ok that procedure worked as a charm, at first sight, only infiniband in the vm didn’t come up. lspci didn’t show us the IB card. luckily there was one vm during the installation of the extra nodes on this vm and we could start comparing the configuration : in the vm.cfg of the vm we “copied” from the X6″ we saw following
ib_pfs = ['03:00.0']
ib_pkeys = [{'pf':'03:00.0','port':'1','pkey':['0xffff',]},{'pf':'03:00.0','port':'2','pkey':['0xffff',]},]
on the vm that was working on the X7
ib_pfs = ['3b:00.0']
ib_pkeys = [{'pf':'3b:00.0','port':'1','pkey':['0xffff',]},{'pf':'3b:00.0','port':'2','pkey':['0xffff',]},]
once we put this in the copied vm everything booted afterwards we understood why, my colleague Fred pointed out on the source dom0 (X6)
lspci |grep -i infiniband
03:00.0 InfiniBand: Mellanox Technologies MT27500 Family [ConnectX-3]
03:00.1 InfiniBand: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
03:00.2 InfiniBand: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
03:00.3 InfiniBand: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
03:00.4 InfiniBand: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
03:00.5 InfiniBand: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
03:00.6 InfiniBand: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
03:00.7 InfiniBand: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
03:01.0 InfiniBand: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
....
on target dom0 (X7)
lspci | grep -i 'infiniband'
3b:00.0 InfiniBand: Mellanox Technologies MT27500 Family [ConnectX-3]
3b:00.1 InfiniBand: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
3b:00.2 InfiniBand: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
3b:00.3 InfiniBand: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
3b:00.4 InfiniBand: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
3b:00.5 InfiniBand: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
3b:00.6 InfiniBand: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
3b:00.7 InfiniBand: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
3b:01.0 InfiniBand: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
on an X4 it is even differnt so basically I think this should be amended to the otherwise flawless document mentioned before Hope this helps
