在vmwarevsphereesxi上安裝oracleRAC

1、Oracle Databases on VMwareRAC 安裝手冊(cè)Contents1.Introduction52.Infrastructure Environment62.1Hardware62.2Software Environment62.3VMware Virtual Machines62.4Virtual Disk Layout72.5Oracle RAC Database Environment72.6Network Configuration72.7Physical Architecture102.8Logical Architecture113.Overview of Dep

2、loyment Steps124.Configure NTP Client on All ESX Hosts135.Prepare a Virtual Machine for the First RAC Node155.1Create a Virtual Machine155.2Add Two NICs155.3Add CRS and Voting Disk165.4Install the Guest Operating System195.5Install VMware Tools215.6Timekeeping with the Linux Guest Operating System29

3、6.Prepare Subsequent Virtual Machines/RAC Nodes306.1Clone the First Virtual Machine/RAC Node306.2Configure CSR and Voting Disks326.3Format the Added Disks Using fdisk347.Install Oracle Grid Infrastructure358.Verify the Grid Infrastructure Installation379.Add DATA and REDO Disks to Virtual Machine RA

4、C Nodes3910.Create DATA and REDO ASM Disk Groups as Grid User4311.Install and Create the Oracle Database 11g R2 RAC4412.References451. 介紹本文通過(guò)一個(gè)案例簡(jiǎn)要介紹 Oracle Real Application Clusters (RAC) 在 VMware vSphere® 平臺(tái)上的安裝過(guò)程, 幫助 Oracle 數(shù)據(jù)庫(kù)管理員在VMware 平臺(tái)上規(guī)劃和實(shí)施Oracle RAC環(huán)境. 在VMware環(huán)境下,數(shù)據(jù)庫(kù)管理員通常需要和VMware的管理員

5、以及存儲(chǔ)廠商的管理員合作,以順利部署一個(gè)Oracle RAC環(huán)境.在虛擬化平臺(tái)上部署Oracle RAC在許多方面與純物理環(huán)境上的部署很類(lèi)似,比如:· 在虛擬機(jī)創(chuàng)建完成之后,安裝操作系統(tǒng)和Oracle RAC軟件的過(guò)程與物理環(huán)境下基本一致· 存儲(chǔ)的規(guī)劃,包括用于數(shù)據(jù)存放的LUN數(shù)量的規(guī)劃和多路徑的設(shè)計(jì)與物理環(huán)境也基本一致.差別在于我們可以充分依靠虛擬化軟件, 簡(jiǎn)化許多工作, 從而加快安裝進(jìn)度,減少錯(cuò)誤的發(fā)生,比如:· 利用事先準(zhǔn)備好的模板, 多次安裝同版本的系統(tǒng)和數(shù)據(jù)庫(kù),同時(shí)加強(qiáng)標(biāo)準(zhǔn)化· 利用快照技術(shù), 很方便的進(jìn)行測(cè)試和回退· 利用虛擬機(jī)復(fù)制

6、技術(shù),簡(jiǎn)化多個(gè)RAC節(jié)點(diǎn)的系統(tǒng)安裝、打補(bǔ)丁、配置工作由于安裝過(guò)程的細(xì)節(jié)與軟件版本（尤其是Linux版本和Oracle版本）有較強(qiáng)的關(guān)聯(lián)性，本文只能盡量把安裝步驟描述清楚.讀者在自己進(jìn)行安裝的時(shí)候，還需要參考相應(yīng)版本的官方安裝文檔.先將本次安裝中用到的主要軟件和版本羅列如下：· VMware ESXi 5.1 · VMware vCenter Server 5.1 · vSphere Client 5.1 · Oracle Database 11g . · Oracle Enterprise Linux x86_64 6.3.2.

7、 軟硬件環(huán)境2.1 硬件組件數(shù)量配置ESXi Hosts2Cisco UCS B4404x 10-core CPUs, 256GB RAM, and 4 NICsStorage1NetApp F3240 雙路控制器Ethernet Switches2Cisco FI6248FC Switches1 Cisco Nexus 7000Host Based Adapters4每個(gè)刀片兩個(gè)，與網(wǎng)卡共享流量2.2 軟件環(huán)境組件版本作用vSphere5.1Hypervisor hosting all virtual machinesVMware vCenter5.1管理 vSphereOracle Ent

8、erprise Linux x86_646.3Guest operating system for virtual machines running Oracle Database serverOracle Database 11g (with Oracle RAC and Oracle Grid Infrastructure) Enterprise EditionMicrosoft Windows Server2008 R2用于 vCenter Server2.3 虛擬機(jī)配置虛擬機(jī)用途數(shù)量配置Oracle RAC nodes220 vCPU, 120GB RAM,Oracle

9、 Enterprise Linux x86_64 6.3vCenter12 vCPUs, 8GB RAM, Windows Server 2008 R22.4 共享存儲(chǔ)配置Datastore NameVirtual Disk on ESXLinux路徑虛擬路徑驅(qū)動(dòng)類(lèi)型VMDK Size (GB)用途VMDATASTOREVMDK Hard disk 1/dev/sdaSCSI 0:0LSI Logic150Oracle Enterprise Linux 5.5 OS and Oracle binariesCRS1VMDK Hard disk 2Shared disk/dev/sdb1SCSI

10、1:0Paravirtual20CRS and voting diskREDO1VMDK Hard disk 3Shared disk/dev/sdc1SCSI 1:1Paravirtual900REDO and ArchieveDATA01VMDK Hard disk 4Shared disk/dev/sdd1SCSI 2:0Paravirtual1000RAC database DATADATA02VMDK Hard disk 5Shared disk/dev/sde1SCSI 2:1Paravirtual1000RAC database DATADATA03VMDK Hard disk

11、6Shared disk/dev/sdf1SCSI 3:0Paravirtual1000RAC database DATADATA04VMDK Hard disk 7Shared disk/dev/sdg1SCSI 3:1Paravirtual1000RAC database DATA2.5 Oracle RAC 數(shù)據(jù)庫(kù)環(huán)境Node NameASM InstanceRAC InstanceDatabaseASM StorageRAC1+ASM1RAC1RACASM file systemRAC2+ASM2RAC2RACASM file system2.6 網(wǎng)絡(luò)配置按照Oracle RAC的實(shí)施

12、慣例，每個(gè)RAC節(jié)點(diǎn)都提供一個(gè)Public網(wǎng)口用于承載業(yè)務(wù)，并在其上捆綁虛擬IP地址 (VIP) 用于HA切換. 各RAC節(jié)點(diǎn)之間還需要一個(gè)私有網(wǎng)絡(luò)進(jìn)行節(jié)點(diǎn)之間的通信。在物理環(huán)境中，通常需要考慮鏈路的冗余以消除單點(diǎn)鼓掌。而在虛擬化環(huán)境中，冗余問(wèn)題由底層虛擬化軟件處理，所以對(duì)于每個(gè)RAC節(jié)點(diǎn)，只是簡(jiǎn)單地需要兩個(gè)網(wǎng)口就可以工作，大大簡(jiǎn)化了配置和安裝工作.我們這次測(cè)試的網(wǎng)絡(luò)配置如下三:· UCS底層軟件配置出三個(gè)物理網(wǎng)口（均為10Gb） vmnic0 和 vmnic1用于Oracle Public and Oracle Private網(wǎng)絡(luò), vmnic2用于vSphere® vM

13、otion®工作.· 兩個(gè)分布式虛擬交換機(jī) Oracle Public (VM Net) and Oracle Private (Oracle HB).· 每個(gè)RAC節(jié)點(diǎn)配置3個(gè)IP地址 以節(jié)點(diǎn)1 (RAC1) 為例:o One public static IP address (eth0): 01o One private static IP address (eth1): 0o One virtual static IP address(VIP): 11（由CRS軟件配置和管理）還有三個(gè)虛擬IP用于

14、節(jié)點(diǎn)間負(fù)載均衡(SCAN): 00,01,02（由CRS配置和管理）物理網(wǎng)口 vmnic0 連接到分布式交換機(jī) VM Net,vmnic1連接到分布式交換機(jī)Oracle HB， vMotion 使用標(biāo)準(zhǔn)交換機(jī)vSwitch3. 這樣區(qū)分概念上比較清晰，也可以只建一個(gè)分布式網(wǎng)絡(luò)，用不同的端口組區(qū)分?jǐn)?shù)據(jù)網(wǎng)絡(luò)和心跳網(wǎng)。Figure 1. Network ConfigurationOracle RAC Private2.7 2.8 邏輯結(jié)構(gòu)圖 Figure 2. Logical Architecture of Two-Node Oracle

15、RAC on vSphere3. 部署流程概覽部署流程大體按照下面的流程圖進(jìn)行.Figure 3. Deployment Steps for Four-Node Oracle RAC on vSphere with VMFS4. 為第一個(gè)RAC節(jié)點(diǎn)準(zhǔn)備虛擬機(jī)環(huán)境本節(jié)安裝步驟簡(jiǎn)介如下:· 使用vSphere Client創(chuàng)建第一個(gè)虛擬機(jī)。添加公網(wǎng)和心跳網(wǎng)卡，挑選適當(dāng)?shù)腖UN創(chuàng)建仲裁盤(pán)CRS （VMDK模式）。· 安裝Redhat Linux操作系統(tǒng)，安裝VMware tools，按照Oracle RAC要求安裝相應(yīng)補(bǔ)丁.· 在Linux中配置NTP 客戶(hù)端.4.1

16、創(chuàng)建第一個(gè)虛擬機(jī)1. 使用vSphere Client登錄vCenter.2. 選中一臺(tái)主機(jī),在Getting Started 頁(yè)面單擊 Create a new virtual machine.3. In the Create New Virtual Machine wizard, select Typical for the Configuration, and click Next.4. On the Name and Location page, enter RAC1 for the Name, and click Next.5. On the Datastore page, sele

17、ct VMDATASTORE, and click Next.6. On the Guest Operating System page, select Linux from the Guest Operating System list. For the Version, select Oracle Linux 6 (64-bit) from the drop-down menu. Click Next.7. On the Create a Disk page, set the Virtual Disk Size to 20GB, and click Next.8. On the Ready

18、 to Complete page, select Edit the virtual machine settings before completion. Click Continue.9. On the Virtual Machine Properties page for the RAC1 virtual machine, select 8 CPUs and 128GB Memory.10. Click Finish.4.2 增加兩個(gè)虛擬網(wǎng)卡現(xiàn)在需要增加兩塊網(wǎng)卡. 一塊用于公共數(shù)據(jù)網(wǎng)絡(luò), 另一塊用于心跳網(wǎng)絡(luò). 添加這兩塊網(wǎng)卡的過(guò)程基本相同,差別僅在于添加公網(wǎng)網(wǎng)卡時(shí)選擇標(biāo)簽: VM Net

19、 而添加心跳網(wǎng)卡時(shí)選擇 Oracle HB.添加公網(wǎng)網(wǎng)卡的步驟：1. Select a virtual machine.2. Click Edit settings to display the Virtual Machine Properties pane.3. Click Add.4. Select Ethernet Adaptor and click Next to display the Add Hardware wizard.5. On the Network connection page, select VMXNET 3 from the Adapter Type drop-do

20、wn menu.6. In the Network Connection section of the Network connection page, select Named network with specified label.7. In the Named network with specified label drop-down menu, select VM Net.8. In the Device Status section of the Network connection page, select Connect at power on.9. On the Ready

21、 to Complete page, click Finish, and then click OK.添加心跳網(wǎng)卡的步驟：1. Repeat steps 1 through 6 from the preceding procedure.2. In the Named network with specified label drop-down menu, select Oracle HB.3. Continue with steps 8 and 9 from the preceding procedure.4.3 添加仲裁盤(pán)Oracle RAC需要至少一個(gè)共享的仲裁盤(pán)存放clusterware

22、信息和節(jié)點(diǎn)投票信息。存儲(chǔ)管理員需要事先為仲裁盤(pán)分配好單獨(dú)的LUN，然后我們需要?jiǎng)?chuàng)建單獨(dú)的SCSI控制器并設(shè)為Paravirtual驅(qū)動(dòng). SCSI 總線 sharing policy 設(shè)為 none.下面描述創(chuàng)建共享存儲(chǔ)的步驟. 為虛擬機(jī)創(chuàng)建 20GB datastore：1. In vSphere Client, select a virtual machine.2. Right-click the name of the virtual machine and select Edit Settings to display the Virtual Machine Properties pa

23、ne.3. Click Add to display the Add Hardware wizard.4. In the Add Hardware wizard select Hard Disk for the Device Type, and click Next.5. In the Disk section of the Select a Disk page, select Create a new virtual disk. Set the disk size to 20GB, select the Support clustering features such as Fault To

24、lerance check box, and select CRS1 to specify a datastore. Also, for vSphere 5, in the Disk Provisioning section select Thick Provision Eager Zeroed. Click Next.6. On the Compatibility Mode page, click Next.7. On the Advanced Options page, in the Virtual Device Node section, select SCSI (1:0). In th

25、e Mode section, select Independent. Click Next.8. On the Ready to Complete page, click Finish.9. Click OK.Add configuration parameters for each Oracle RAC virtual machine as described in Disabling simultaneous write protection provided by VMFS using the multi-writer flag (手工修改虛擬機(jī)參數(shù)1. In vSphere Clie

26、nt, select a virtual machine.2. Right-click the name of the virtual machine and select Edit Settings to display the Virtual Machine Properties pane.3. Click the Options tab, click Advanced, and click General.4. Click Configuration Parameters.5. On the Configuration Parameters page, add rows and entr

27、ies for the shared disks in the virtual machine's configuration parameters as shown in the following screenshot.6. Repeat steps 1 through 5 for each of the remaining virtual machines.You can view the virtual machine properties for the RAC1 virtual machine in vCenter as in the following screensho

28、t.web client 顯示如下圖：4.4 安裝操作系統(tǒng)將OEL6.4安裝介質(zhì)ISO文件上傳到datastore，然后進(jìn)行安裝。Note: This document is specific to Linux guest operating system. Oracle RAC and above versions can be deployed and supported on the list of guest operation systems using VMware multi-writer flag (在虛擬機(jī)上安裝Linux操作系統(tǒng)1. In vSphere

29、Client, select a virtual machine.2. Click Edit settings.3. In the Hardware tab of the Virtual Machine Properties pane, select CD/DVD Drive 1.4. In the Device Status section of this pane, select the Connect at power on check box.5. In the Device Type section of this pane, select Datastore ISO File an

30、d enter the path to your Linux ISO image file.6. In the Virtual Device Node section of this pane, select CD/DVD Drive 1 from the drop-down menu.7. Click OK.8. Power on the virtual machine.9. Install Linux with the following settings.Custom partitioning scheme:/dev/sda1 ext3 150GB mounted on /Install

31、ed packages use the default package selection, plus the following:· kernel-uek.x86_64· kmod-oracleasm· oracleasm-2.6.18-194.el5-2.0.5-1.el5.x86_64 (download from · oracleasm-support-2.1.3-1.el5.x86_64· systat-7.0.2-3.el5.x86_6410. Use below script to install remaining packag

32、es:yum install binutils compat-libcap1 compat-libstdc+-33 compat-libstdc+-33.i686 gcc.x86_64 gcc-c+.x86_64 glibc.x86_64 glibc.i686 glibc-devel.x86_64 glibc-devel.i686 unixODBC.i686 unixODBC.x86_64 unixODBC-devel.x86_64 unixODBC-devel.i686 libgcc.x86_64 libgcc.i686 libstdc+.x86_64 libstdc+.i686 libst

33、dc+-devel.x86_64 libstdc+-devel.i686 compat-libstdc+-33.x86_64 compat-libstdc+-33.i686 libaio.x86_64 libaio.i686 libaio-devel.x86_64 libaio-devel.i686 sysstat.x86_64 ksh libXtst-devel.x86_64 libXtst.i686 compat-db.x86_64 xorg-x11-utils.x86_64 xorg-x11-xinit.x86_64 make11. View the installed packages

34、 with this command:run command “rpm q <package name>12. Confirm that the following packages have been installed:· binutils-.2-5.11.el6 (x86_64)· compat-libcap1-1.10-1 (x86_64)· compat-libstdc+-33-3.2.3-69.el6 (x86_64)· compat-libstdc+-33-3.2.3-69.el6.i686· gcc

35、-4.4.4-13.el6 (x86_64)· gcc-c+-4.4.4-13.el6 (x86_64)· glibc-2.12-1.7.el6 (i686)· glibc-2.12-1.7.el6 (x86_64)· glibc-devel-2.12-1.7.el6 (x86_64)· glibc-devel-2.12-1.7.el6.i686· ksh· libgcc-4.4.4-13.el6 (i686)· libgcc-4.4.4-13.el6 (x86_64)· libstdc+-4.4.4-1

36、3.el6 (x86_64)· libstdc+-4.4.4-13.el6.i686· libstdc+-devel-4.4.4-13.el6 (x86_64)· libstdc+-devel-4.4.4-13.el6.i686· libaio-0.3.107-10.el6 (x86_64)· libaio-0.3.107-10.el6.i686· libaio-devel-0.3.107-10.el6 (x86_64)· libaio-devel-0.3.107-10.el6.i686· make-3.81-19

37、.el6·· sysstat-9.0.4-11.el6 (x86_64)· unixODBC-2.2.14-11.el6 (x86_64) or later· unixODBC-2.2.14-11.el6.i686 or later· unixODBC-devel-2.2.14-11.el6 (x86_64) or later· unixODBC-devel-2.2.14-11.el6.i686 or later13. 最后安裝下面兩個(gè)包:· oracle-rdbms-server-11gR2-preinstall (從YU

38、M安裝，這個(gè)包會(huì)配置Oracle RAC所需系統(tǒng)參數(shù)和環(huán)境變量)· cvuqdisk (Oracle安裝介質(zhì)中提供這個(gè)包，用于安裝前檢查)14. 檢查配置文件/etc/sysctl.confkernel.shmmni = 4096 kernel.sem = 250 32000 100 128 fs.file-max = 6815744 (512 x processes for example 6815744 for 13312 processes)net.ipv4.ip_local_port_range = 9000 65500 net.cor

39、e.rmem_default = 262144 net.core.rmem_max = 4194304 net.core.wmem_default = 262144 net.core.wmem_max = 1048576 kernel.shmall = physical RAM size / pagesize For most systems, this will be the value 2097152. See Note: 301830.1 for more information. eg:kernel.shmall=4

40、160749568(62GB)kernel.shnmax=   34359738368(32GB)kernel.shmmax = RAM times 0.5 (or higher at customer's discretion - seeNote:567506.1) /etc/profile ulimit u 16384 ulimit n 6553515. 依照oracle傳統(tǒng)，創(chuàng)建clusterware主用戶(hù) “grid”:rootracnode1 # groupadd -g 1000 oinstall rootracnode1 # groupadd -g 12

41、00 asmadminrootracnode1 # groupadd -g 1201 asmdba rootracnode1 # groupadd -g 1202 asmoper rootracnode1 # useradd -m -u 1100 -g oinstall -G asmadmin,asmdba,asmoper -d /home/grid -s /bin/bash -c "Grid Infrastructure Owner" grid rootracnode1 # id grid uid=1100(grid) gid=1000(oinstall) groups=

42、1000(oinstall), 1200(asmadmin), 1201(asmdba),1202(asmoper) 設(shè)置 grid 帳戶(hù)的口令：rootracnode1 # passwd grid Changing password for user grid. New UNIX password:xxxxxxxxxxx Retype new UNIX password:xxxxxxxxxxx passwd: all authentication tokens updated successfully. 為grid用戶(hù)創(chuàng)建初始化腳本(/home/grid/.bash_pr

43、ofile):# -# .bash_profile# -# OS User: grid# Application: Oracle Grid Infrastructure# Version: Oracle 11g release 2# -# Get the aliases and functionsif -f /.bashrc ; then . /.bashrcfialias ls="ls -FA"# -# ORACLE_SID# -# Specifies the Oracle system identifier (SID)# for the Automatic Storag

44、e Management (ASM)instance# running on this node.# Each RAC node must have a unique ORACLE_SID.# (i.e. +ASM1, +ASM2,.)# -ORACLE_SID=+ASM1; export ORACLE_SID# -# JAVA_HOME# -# Specifies the directory of the Java SDK and Runtime# Environment.# -JAVA_HOME=/usr/local/java; export JAVA_HOME# -# ORACLE_BA

45、SE# -# Specifies the base of the Oracle directory structure# for Optimal Flexible Architecture (OFA) compliant# installations. The Oracle base directory for the# grid installation owner is the location where# diagnostic and administrative logs, and other logs# associated with Oracle ASM and Oracle C

46、lusterware# are stored.# -ORACLE_BASE=/u01/app/grid; export ORACLE_BASE# -# ORACLE_HOME# -# Specifies the directory containing the Oracle# Grid Infrastructure software. For grid# infrastructure for a cluster installations, the Grid# home must not be placed under one of the Oracle base# directories,

47、or under Oracle home directories of# Oracle Database installation owners, or in the home# directory of an installation owner. During # installation, ownership of the path to the Grid # home is changed to root. This change causes # permission errors for other installations.# -ORACLE_HOME=/u01/app/11.

48、2.0/grid; export ORACLE_HOME# -# ORACLE_PATH# -# Specifies the search path for files used by Oracle# applications such as SQL*Plus. If the full path to# the file is not specified, or if the file is not# in the current directory, the Oracle application# uses ORACLE_PATH to locate the file.# This vari

49、able is used by SQL*Plus, Forms and Menu.# -ORACLE_PATH=/u01/app/oracle/common/oracle/sql; export ORACLE_PATH# -# SQLPATH# -# Specifies the directory or list of directories that# SQL*Plus searches for a login.sql file.# -# SQLPATH=/u01/app/common/oracle/sql; export SQLPATH# -# ORACLE_TERM# -# Defines a terminal definition. If not set, it# defaults to the value of your TERM environment# variable. Used by all character mode products. # -ORACLE_TERM=xterm; export ORACLE_TERM# -# NLS_DATE_FORMAT# -# Specifies the default date format to use with the# TO_CHAR and TO_DA