1.集群规划三机器的规划如下表配置项A机器B机器C机器业务IP***内部数据交换网络IP***dmdcr_cfg — CSS — DCR_EP_NAMECSS0CSS1CSS2dmdcr_cfg — CSS — DCR_EP_HOST***dmdcr_cfg — CSS — DCR_EP_PORT112861128611286dmdcr_cfg — ASM — DCR_EP_NAMEASM0ASM1ASM2dmdcr_cfg — ASM — DCR_EP_HOST***dmdcr_cfg — ASM — DCR_EP_PORT112761127611276dmdcr_cfg — DB — DCR_EP_NAMEDSC0DSC1DSC2dmdcr_cfg — DB — DCR_EP_PORT523452345234dmasvrmal — MAL_INST_NAMEASM0ASM1ASM2dmasvrmal — MAL_HOST***dmasvrmal — MAL_PORT112661126611266dminit — 实例名DSC0DSC1DSC2dminit — PORT_NUM523452345234dminit — MAL_HOST***dminit — MAL_PORT112461124611246OGUID210715210715210715监视器IP*——监视器的ip为*2.准备共享存储本次使用虚拟机软件创建共享虚拟磁盘首先创建共享虚拟磁盘文件在C:\Program Files (x86)\VMware\目录下执行如下命令创建一个20GB的数据磁盘vmware-vdiskmanager.exe-c-s20Gb-alsilogic-t2C:\DSC_SharedDisk\dmdata.vmdk创建DCR磁盘vmware-vdiskmanager.exe-c-s2Gb-alsilogic-t2C:\DSC_SharedDisk\dmdcr.vmdk创建Vote磁盘vmware-vdiskmanager.exe-c-s2Gb-alsilogic-t2C:\DSC_SharedDisk\dmvote.vmdk创建LOG盘vmware-vdiskmanager.exe-c-s20Gb-alsilogic-t2C:\DSC_SharedDisk\dmlog.vmdk创建ARCH盘vmware-vdiskmanager.exe-c-s20Gb-alsilogic-t2C:\DSC_SharedDisk\dmarch.vmdk然后为每台虚拟机添加共享磁盘修改每台虚拟机的 .vmx 配置文件加入以下内容确保多个虚拟机可以同时访问到磁盘scsi0.sharedBusvirtualdiskLib.dataCacheMaxSize0diskLib.dataCacheMaxReadAheadSize0diskLib.DataCacheMinReadAheadSize0diskLib.dataCachePageSize4096diskLib.maxUnsyncedWrites0重启虚拟机执行fdisk -l三台机器均可以显示五个磁盘说明共享存储配置成功3.块设备UUID绑定UUID方式在三台机器上/etc/udev/rules.d目录下分别先创建uuid.sh文件内容如下赋权执行该脚本生成绑定配置文件88-dm-asmdevices.rules修改别名SYMLINK与五块磁盘对应:curr$(cd$(dirname$0);pwd)rm-f$curr/88-dm-asmdevices.rulesforiinb c d e f;# 从b开始跳过系统盘sdadoechoKERNEL\sd?\,SUBSYSTEM\block\, PROGRAM\/usr/lib/udev/scsi_id --whitelisted --replace-whitespace --device/dev/\$name\,RESULT\/usr/lib/udev/scsi_id--whitelisted--replace-whitespace--device/dev/sd$i\, SYMLINK\dm/asm-disk$i\,OWNER\dmdba\, GROUP\dinstall\,MODE\0660\$curr/88-dm-asmdevices.rulesdoneudevadm trigger--typedevices--actionchange udevadm control --reload-rules让刷新规则文件使其生效执行chowndmdba:dinstall /dev/dm/asm-*-R授权然后执行命令blockdev--getsize64/dev/dm/asm-dmdcrdmvote/dmdata/dmlog/dmarch验证显示设备UUID绑定成功:4.集群搭建三台服务器上分别在数据库安装目录下/dm/dmdba/dmdbms配置dsc_config/dmdcr_cfg.ini内容如下DCR_N_GRP3##集群环境有多少个 GROUP范围1~16DCR_VTD_PATH/dev/dm/asm-dmvote##规划为 vote 的磁盘DCR_OGUID210715## 消息标识一个组里面只有一个。[GRP]#新建一个 GROUPDCR_GRP_TYPECSS##组类型CSS/ASM/DBDCR_GRP_NAMEGRP_CSS##组名DCR_GRP_N_EP3##组内节点个数DCR_GRP_DSKCHK_CNT65##磁盘心跳容错时间单位:秒[GRP_CSS]DCR_EP_NAMECSS0##CSS 节点名DCR_EP_HOST*##心跳地址DCR_EP_PORT11286##CSS 端口[GRP_CSS]DCR_EP_NAMECSS1 DCR_EP_HOST* DCR_EP_PORT11286[GRP_CSS]DCR_EP_NAMECSS2##CSS 节点名DCR_EP_HOST*##心跳地址DCR_EP_PORT11286##CSS 端口[GRP]DCR_GRP_TYPEASM DCR_GRP_NAMEGRP_ASM DCR_GRP_N_EP3DCR_GRP_DSKCHK_CNT61[GRP_ASM]DCR_EP_NAMEASM0##ASM 节点名和 dmasvrmal 的 MAL_INST_NAME 一致DCR_EP_SHM_KEY42424##共享内存标识DCR_EP_SHM_SIZE1024##共享内存大小DCR_EP_HOST*##心跳地址DCR_EP_PORT11276##ASM 端口DCR_EP_ASM_LOAD_PATH/dev/dm[GRP_ASM]DCR_EP_NAMEASM1 DCR_EP_SHM_KEY42425DCR_EP_SHM_SIZE1024DCR_EP_HOST* DCR_EP_PORT11276DCR_EP_ASM_LOAD_PATH/dev/dm[GRP_ASM]DCR_EP_NAMEASM2 DCR_EP_SHM_KEY42426DCR_EP_SHM_SIZE1024DCR_EP_HOST* DCR_EP_PORT11276DCR_EP_ASM_LOAD_PATH/dev/dm[GRP]DCR_GRP_TYPEDB DCR_GRP_NAMEGRP_DSC DCR_GRP_N_EP3DCR_GRP_DSKCHK_CNT57[GRP_DSC]DCR_EP_NAMEDSC0##实例名和 dm.ini 的 INSTANCE_NAME 一致DCR_EP_SEQNO0##组内序号不能重复DCR_EP_PORT5234##实例端口和 dm.ini 的 PORT_NUM 一致[GRP_DSC]DCR_EP_NAMEDSC1 DCR_EP_SEQNO1DCR_EP_PORT5234[GRP_DSC]DCR_EP_NAMEDSC2 DCR_EP_SEQNO2DCR_EP_PORT5234初始化磁盘在机器A的bin目录下执行./dmasmcmd进入ASM终端执行下述命令进行初始化create dcrdisk/dev/dm/asm-dmdcrdcrcreate votedisk/dev/dm/asm-dmvotevoteinit dcrdisk/dev/dm/asm-dmdcrfrom/dm/dmdba/dmdbms/dsc_config/dmdcr_cfg.iniidentified byZh010922init votedisk/dev/dm/asm-dmvotefrom/dm/dmdba/dmdbms/dsc_config/dmdcr_cfg.inicreate asmdisk/dev/dm/asm-dmdataDATA0create asmdisk/dev/dm/asm-dmlogLOG0create asmdisk/dev/dm/asm-dmarchARCH0在机器A的dsc_config下创建dmasvrmal.ini内容如下并cp到三台机器的dsc_config/DSC0(DSC1/DSC2)目录下[MAL_INST0]MAL_INST_NAMEASM0 MAL_HOST*#心跳地址MAL_PORT11266#MAL 监听端口[MAL_INST1]MAL_INST_NAMEASM1 MAL_HOST* MAL_PORT11266[MAL_INST2]MAL_INST_NAMEASM2 MAL_HOST* MAL_PORT11266在三台机器的dsc_config目录下分别创建dmdcr.ini文件内容如下以机器a为例DMDCR_PATH/dev/dm/asm-dmdcr DMDCR_MAL_PATH/dm/dmdba/dmdbms/dsc_config/DSC0/dmasvrmal.ini DMDCR_SEQNO0#机器b,c分别为12DMDCR_AUTO_OPEN_CHECK111DMDCR_ASM_TRACE_LEVEL2在三个节点启动DMCSS、DMASM服务分别执行命令/dm/dmdba/dmdbms/bin/dmcssDCR_INI/dm/dmdba/dmdbms/dsc_config/dmdcr.ini启动dmcss服务执行命令/dm/dmdba/dmdbms/bin/dmasmsvrDCR_INI/dm/dmdba/dmdbms/dsc_config/dmdcr.ini启动dmasmsvr服务在机器A的bin目录下启动dmasmtool工具执行命令/dm/dmdba/dmdbms/bin/dmasmtoolDCR_INI/dm/dmdba/dmdbms/dsc_config/dmdcr.ini创建DMASM磁盘组执行命令create diskgroupDMLOGasmdisk/dev/dm/asm-dmlog创建日志磁盘组执行命令create diskgroupDMDATAasmdisk/dev/dm/asm-dmdata创建数据磁盘组执行命令create diskgroupDMARCHasmdisk/dev/dm/asm-dmarch创建归档磁盘组:分别在三台机器的dsc_config目录下创建dminit.ini文件以机器A为例内容如下DB_NAMEDSC SYSDBA_PWD* SYSAUDITOR_PWD* SYSTEM_PATHDMDATA/data SYSTEMDMDATA/data/SYSTEM.dbf SYSTEM_SIZE1024ROLLDMDATA/data/ROLL.dbf ROLL_SIZE10000MAINDMDATA/data/MAIN.dbf MAIN_SIZE1024CTL_PATHDMDATA/data/dm.ctl CTL_SIZE8LOG_SIZE2048DCR_PATH/dev/dm/asm-dmdcr DCR_SEQNO0AUTO_OVERWRITE1PAGE_SIZE32EXTENT_SIZE16BLANK_PAD_MODE1[DSC0]CONFIG_PATH/dm/dmdba/dmdbms/dsc_config/DSC0 PORT_NUM5234MAL_HOST* MAL_PORT11246LOG_PATHDMLOG/DSC0_LOG01.log LOG_PATHDMLOG/DSC0_LOG02.log[DSC1]CONFIG_PATH/dm/dmdba/dmdbms/dsc_config/DSC1 PORT_NUM5234MAL_HOST* MAL_PORT11246LOG_PATHDMLOG/DSC1_LOG01.log LOG_PATHDMLOG/DSC1_LOG02.log[DSC2]CONFIG_PATH/dm/dmdba/dmdbms/dsc_config/DSC2 PORT_NUM5234MAL_HOST* MAL_PORT11246LOG_PATHDMLOG/DSC2_LOG01.log LOG_PATHDMLOG/DSC2_LOG02.log在机器A上初始化执行命令/dm/dmdba/dmdbms/bin/dminitcontrol/dm/dmdba/dmdbms/dsc_config/dminit.ini在dsc_config/DSC0目录下生成了实例对应的dm.ini等文件同时还生成了机器B和C对应的配置文件DSC1和DSC2利用共享文件夹分别cp到对应机器。三节点修改各自dm.ini的ARCH_INI为1开启数据库归档功能然后创建各自的归档配置文件dmarch.ini以机器A为例内容如下ARCH_WAIT_APPLY0[ARCHIVE_LOCAL1]ARCH_TYPELOCAL ARCH_DESTDMARCH/ARCH/DSC2/arch ARCH_FILE_SIZE2048ARCH_SPACE_LIMIT102400[ARCHIVE_REMOTE1]ARCH_TYPEREMOTE ARCH_DESTDSC0 ARCH_INCOMING_PATHDMARCH/ARCH/DSC0/arch ARCH_FILE_SIZE2048ARCH_SPACE_LIMIT102400[ARCHIVE_REMOTE2]ARCH_TYPEREMOTE ARCH_DESTDSC1 ARCH_INCOMING_PATHDMARCH/ARCH/DSC1/arch ARCH_FILE_SIZE2048ARCH_SPACE_LIMIT102400启动DMSERVER服务由DMCSS没有配置自动拉起dmserver的功能需要手动启动三个节点的实例服务以机器A为例在bin目录下执行命令./dmserver /dm/dmdba/dmdbms/dsc_config/DSC0/dm.inidcr_ini/dm/dmdba/dmdbms/dsc_config/dmdcr.ini在机器A的 dmcss 终端执行show命令显示CSS组全部openDMCSS集群正常CSS0为控制节点ASM组也全部openDMASM集群正常ASM0为控制节点DB组也全补openDMDSC集群正常DSC0为控制节点为了方便管理分别在三台机器上注册CSS、ASM、DMSERVER后台服务以机器A为例在root下执行命令/dm/dmdba/dmdbms/script/root/dm_service_installer.sh-tdmcss-dcr_ini/dm/dmdba/dmdbms/dsc_config/dmdcr.ini-pCSS /dm/dmdba/dmdbms/script/root/dm_service_installer.sh-tdmasmsvr-dcr_ini/dm/dmdba/dmdbms/dsc_config/dmdcr.ini-yDmCSSServiceCSS.service-pASM /dm/dmdba/dmdbms/script/root/dm_service_installer.sh-tdmserver-dm_ini/dm/dmdba/dmdbms/dsc_config/DSC0/dm.ini-dcr_ini/dm/dmdba/dmdbms/dsc_config/dmdcr.ini-yDmASMSvrServiceASM.service-pDSC5.配置监视器在另外一台机器192.168.138.131的bin目录下创建DSC监视器配置文件dmcssm.ini内容如下CSSM_OGUID210715CSSM_CSS_IP* CSSM_CSS_IP* CSSM_CSS_IP* CSSM_LOG_PATH/dm/dmdba/dmdbms/log CSSM_LOG_FILE_SIZE512CSSM_LOG_SPACE_LIMIT2048为了优化集群在操作系统假死时对节点状态的判断机制给集群设置setcap权限在每台机器的root下分别执行命令setcap cap_net_raw,cap_net_admineip /dm/dmdba/dmdbms/bin/dmserver setcap cap_net_raw,cap_net_admineip /dm/dmdba/dmdbms/bin/dmasmsvr验证授权成功getcap /dm/dmdba/dmdbms/bin/dmserver getcap /dm/dmdba/dmdbms/bin/dmasmsvrbin目录下执行./dmcssmINI_PATH./dmcssm.ini以前台方式启动监视器执行show显示DSC集群状态正常https://eco.dameng.com