RAC - ORACLEAGENT BLOG

Troubleshooting RAC Load Balancer and FAN

Troubleshooting RAC Load Balancer and FAN :-

Below I mentioned simple steps:

Make sure tnsnames.ora has LOAD_BALANCE=ON
Server side Load Balance- Make sure remote_listener is set and pointing to correct entry in server side tnsnames.ora ( SCAN NAME – not the local listener)
If the desire is to get instance based LB make sure CLB_GOAL set to LONG and in cases set prefer_least_loaded_<listener_name>=OFF
If desire is to get LB on node load make sure CLB_GOAL is unset or set to NONE. Check prefer_least_loaded_<listener_name>=ON
If desire to get LB on service goodness then set CLB_GOAL=SHORT and also set Service Goal
Java application not receiving FAN events-check ONS, ICC
.NET and OCI application not receiving FAN events-check if aq_ha_notifications set for service, check system queue.
Java application not doing FCF-check if connection pool has FCF set/enabled
Check ONS, MMON, MMNL on server
Check PMON-enable trace, truss, strace, etc.
Oracle bugs

Notes:

REMOTE_LISTENER: A remote listener is a listener residing on one computer that redirects connections to a database instance on another computer. In a dedicated server RAC environment, we must enable the PMON background process to register with a remote listener. You do this by configuring the REMOTE_LISTENER parameter.

SCAN – Single Client Access Name This simply means client will just have the SCAN Name in the tnsnames.ora in the Client side and no need to know the Name of the nodes involved in the cluster to load balance . The SCAN Name is associated with one or more SCAN VIP’s (max of 3 VIP’s) . Client will get the SCAN Name and the request will be routed to the DNS and GNS and will get resolved by a single SCAN VIP which is associated to the SCAN Name . PMON will be frequently sending the load status on each node to the SCAN Listener and hence the SCAN listener will have a count on the load status on each node . Each Instance will be registered with the SCAN Listener and SCAN Listener will be aware of the load on each instance and will load balance the new requests accordingly .

Thank you for giving your valuable time to read the above information.

If you want to be updated with all our articles send us the Invitation or Follow us:

Ramkumar’s LinkedIn: https://www.linkedin.com/in/ramkumardba/
LinkedIn Group: https://www.linkedin.com/in/ramkumar-m-0061a0204/
Facebook Page: https://www.facebook.com/Oracleagent-344577549964301
Ramkumar’s Twitter : https://twitter.com/ramkuma02877110
Ramkumar’s Telegram: https://t.me/oracleageant
Ramkumar’s Facebook: https://www.facebook.com/ramkumarram8

SRVCTL command in RAC in 19c Database

Oracle RAC Administration and Monitoring Scripts -2

There are lots of options used with SRVCTL command, i mentioned them below.

-d Database Name
-i Instance Name
-s Service Name
-n Node Name
-r Preferred list
-a Available list
-p TAF (Transparent application failover policy)
-v Verbose

Status and Config Database

We can check all instance status of database as follows.

srvctl status database -d DB_NAME

[oracle@RAMdbrac01 ~]$ srvctl status database -d RAM
Instance RAM1 is running on node RAMdbrac01
Instance RAM2 is running on node RAMdbadm02
Instance RAM3 is running on node RAMdbadm03
Instance RAM4 is running on node RAMdbadm04
[oracle@RAMdbrac01 ~]$

We can display the configurations (instances name, nodes and oracle home) of any RAC Database as follows.

srvctl config database -d DB_NAME

srvctl config database -d RAM

Shutdown RAC Database

We can shutdown all instances of any database as follows.

srvctl stop database -d db_name [-o stop_options]
srvctl stop database -d DB_NAME
srvctl stop database -d DB_NAME -o normal
srvctl stop database -d DB_NAME -o immediate
srvctl stop database -d DB_NAME -o transactional
srvctl stop database -d DB_NAME -o abort

[oracle@RAMdbrac01 ~]$ srvctl stop database -d RAM

Startup RAC Database

We can start all instances of any database as follows.

srvctl start database -d db_name [-o start_options]

srvctl start database -d DB_NAME

srvctl start database -d DB_NAME -o nomount
srvctl start database -d DB_NAME -o mount
srvctl start database -d DB_NAME -o open

[oracle@RAMdbrac01 ~]$ srvctl start database -d RAM

Shutdown An Instance

We can shutdown only specific Instance of RAC database as follows.

srvctl stop instance -d db_unique_name [-i "instance_name_list"]} [-o stop_options] [-f]

srvctl stop instance -d DB_NAME -i INSTANCE_NAME

[oracle@RAMdbrac01 ~]$ srvctl stop instance -d RAM -i RAM1

Startup and Status An Instance

We can startup only specific Instance(s) of RAC database as follows.

srvctl start instance -d db_unique_name [-i "instance_name_list"} [-o start_options]

srvctl start instance -d DB_NAME -i INSTANCE_NAME

[oracle@RAMdbrac01 ~]$ srvctl start instance -d RAM -i RAM1

We can display any instance of RAC database as follows.

srvctl status instance -d DB_NAME -i INSTANCE_NAME

[oracle@RAMdbrac01 ~]$ srvctl status instance -d RAM -i RAM1

Add and Remove Database from CRS

We can add new DB to CRS as follows.

srvctl add database -d db_unique_name -o ORACLE_HOME [-p spfile]

srvctl add database -d RAM -o /u01/app/product/12.1.0.2/dbhome_1 -p +DATAC1/RAM/parameterfile/spfileRAM.ora

We can remove existing DB from CRS as follows.

srvctl remove database -d db_unique_name [-f] [-y] [-v]

srvctl remove database -d DB_NAME -f -y

Stop, Start and Check ASM via SRVCTL

We can shutdown an ASM instance as follows.

srvctl stop asm -n NODE_NAME -o immediate

srvctl stop asm -n RAMdbrac01 -o immediate

We can startup an ASM instance as follows.

srvctl start asm -n NODE_NAME

srvctl start asm -n RAMdbrac01

We can display configuration and status an ASM instance as follows.

srvctl status asm -n NODE_NAME
srvctl config asm -n NODE_NAME

[oracle@RAMdbrac01 ~]$ srvctl status asm -n RAMdbrac01
ASM is running on RAMdbrac01
[oracle@RAMdbrac01 ~]$
[oracle@RAMdbrac01 ~]$
[oracle@RAMdbrac01 ~]$ srvctl config asm -n RAMdbrac01
PRKO-2207 : Warning:-node option has been deprecated and will be ignored.
ASM home: <CRS home>
Password file: +DBFS_DG/orapwASM
ASM listener: LISTENER
[oracle@RAMdbrac01 ~]$

Stop, Start and Check Service via SRVCTL

We can start, stop and check any service on Database as follows.

srvctl status service -d db_name
srvctl config service -d db_name
srvctl start service -d dbname -s service_name
srvctl stop service -d dbname -s service_name

srvctl add service -d {DB_NAME} -s {SERVICE_NAME} -r {"preferred_list"} -a {"available_list"} [-P {BASIC | NONE | PRECONNECT}]

srvctl relocate service -d db_name -s service_name -i instance_name -t new_instance_name [-f]

We can remove any service as follows.

srvctl remove service -d {DB_NAME} -s {SERVICE_NAME}

srvctl remove service -d RAM -s RAM_SALES

Enable/disable Autorestart of Instance and Database

We can enable and disable auto restart of the database as follows.

srvctl enable database -d DB_UNIQUE_NAME
srvctl disable database -d DB_UNIQUE_NAME

We can enable and disable auto restart of the instance

srvctl enable instance -d DB_UNIQUE_NAME-i INSTANCE_NAME
srvctl disable instance -d DB_UNIQUE_NAME-i INSTANCE_NAME

Start, Stop and Check status,configurations of NodeApps

srvctl status nodeapps -n nodename
srvctl config nodeapps -n nodename
srvctl start nodeapps -n nodename
srvctl stop nodeapps -n nodename

Thank you for giving your valuable time to read the above information.
If you want to be updated with all our articles send us the Invitation or Follow us:
Ramkumar’s LinkedIn: https://www.linkedin.com/in/ramkumardba/
LinkedIn Group: https://www.linkedin.com/in/ramkumar-m-0061a0204/
Facebook Page: https://www.facebook.com/Oracleagent-344577549964301
Ramkumar’s Twitter : https://twitter.com/ramkuma02877110
Ramkumar’s Telegram: https://t.me/oracleageant
Ramkumar’s Facebook: https://www.facebook.com/ramkumarram8

Oracle RAC Administration and Monitoring Scripts

Oracle RAC Administration and Monitoring Scripts -1

To check clusterware status, execute following command.

[root@racdb1 ~]$ . oraenv
ORACLE_SID = [root] ? +ASM1

[root@racdb ~]$ crsctl check crs

CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

To check cluster status, execute following command.

[root@racdb ~]$ crsctl check cluster 
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@racdb ~]$

To check all cluster resource status, execute following command.

[root@Prodecidbrac01 ~]# crsctl status resource -t
--------------------------------------------------------------------------------
Name Target State Server State details 
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATAC1.dg
ONLINE ONLINE Prodecidbrac01 STABLE
ONLINE ONLINE Prodecidbrac02 STABLE
ora.DBFS_DG.dg
ONLINE ONLINE Prodecidbrac01 STABLE
ONLINE ONLINE Prodecidbrac02 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE Prodecidbrac01 STABLE
ONLINE ONLINE Prodecidbrac02 STABLE
ora.RECOC1.dg
ONLINE ONLINE Prodecidbrac01 STABLE
ONLINE ONLINE Prodecidbrac02 STABLE
ora.asm
ONLINE ONLINE Prodecidbrac01 Started,STABLE
ONLINE ONLINE Prodecidbrac02 Started,STABLE
ora.net1.network
ONLINE ONLINE Prodecidbrac01 STABLE
ONLINE ONLINE Prodecidbrac02 STABLE
ora.ons
ONLINE ONLINE Prodecidbrac01 STABLE
ONLINE ONLINE Prodecidbrac02 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE Prodecidbrac02 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE Prodecidbrac01 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE Prodecidbrac01 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE Prodecidbrac01 169.254.94.159,STABL
E
ora.cvu
1 ONLINE ONLINE Prodecidbrac01 STABLE
ora.dbm01.db
1 ONLINE OFFLINE STABLE
2 ONLINE OFFLINE Instance Shutdown,ST
ABLE
ora.ebys.db
1 ONLINE ONLINE Prodecidbrac01 Open,STABLE
2 ONLINE ONLINE Prodecidbrac02 Open,STABLE
ora.ebytest.db
1 OFFLINE OFFLINE STABLE
2 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
ora.mgmtdb
1 ONLINE ONLINE Prodecidbrac01 Open,STABLE
ora.oc4j
1 ONLINE ONLINE Prodecidbrac01 STABLE
ora.repdb.db
1 ONLINE ONLINE Prodecidbrac01 Open,STABLE
2 ONLINE ONLINE Prodecidbrac02 Open,STABLE
ora.scan1.vip
1 ONLINE ONLINE Prodecidbrac02 STABLE
ora.scan2.vip
1 ONLINE ONLINE Prodecidbrac01 STABLE
ora.scan3.vip
1 ONLINE ONLINE Prodecidbrac01 STABLE
ora.test.db
1 OFFLINE OFFLINE STABLE
2 ONLINE OFFLINE Instance Shutdown,ST
ABLE
ora.test.db
1 OFFLINE OFFLINE STABLE
2 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
ora.Prodeci.db
1 ONLINE ONLINE Prodecidbrac01 Open,STABLE
2 ONLINE ONLINE Prodecidbrac02 Open,STABLE
ora.Prodecidbrac01.vip
1 ONLINE ONLINE Prodecidbrac01 STABLE
ora.Prodecidbrac02.vip
1 ONLINE ONLINE Prodecidbrac02 STABLE
--------------------------------------------------------------------------------
[root@Prodecidbrac01 ~]#

RAC Cluster Command ( Scripts )

To stop Clusterware on specific node, execute following command. Set ASM profile before executing crsctl command.

[root@racdb1 ~]$ . oraenv
ORACLE_SID = [root] ? +ASM1

[root@racdb ~]$ crsctl stop crs

To start Clusterware on specific node, execute following command. Set ASM profile before executing crsctl command.

[root@racdb1 ~]$ . oraenv
ORACLE_SID = [root] ? +ASM1

[root@racdb ~]$ crsctl start crs

To disable Clusterware on specific node, execute following command. Set ASM profile before executing crsctl command.

[root@racdb ~]$ crsctl disable crs

To enable Clusterware on specific node, execute following command. Set ASM profile before executing crsctl command.

[root@racdb ~]$ crsctl enable crs

To Query Voting disk location, execute following command. Set ASM profile before executing crsctl command.

[oracle@racdb ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE bd81ab8e27a64fbebf7b6b326c5f614d (/Prod/mapper/HDD_E0_S03_1418288932p1) [DATA]
2. ONLINE 450684e276874ff9bff52d481befa809 (/Prod/mapper/HDD_E0_S15_1420185788p1) [DATA]
3. ONLINE fd1efd6de5204f6bbfe9793a480c517f (/Prod/mapper/HDD_E0_S06_1423843312p1) [DATA]
Located 3 voting disk(s).

To find OCR files location, execute following command.

[root@racdb ~]# cat /etc/oracle/ocr.loc
#Prodice/file +DATA getting replaced by Prodice +DATA/Prodeci-c/OCRFILE/registry.255.923150247
ocrconfig_loc=+DATA/Prodeci-c/OCRFILE/registry.255.923150247
local_only=false
[root@racdb ~]#

To check the status of the Oracle Cluster registry , execute following command.

[root@racdb ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 4
Total space (kbytes) : 409568
Used space (kbytes) : 2412
Available space (kbytes) : 407156
ID : 2022573354
Prodice/File Name : +DATA
Prodice/File integrity check succeeded
Prodice/File not configured
Prodice/File not configured
Prodice/File not configured
Prodice/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
[root@racdb ~]#

To add voting disk, execute following command. Set ASM profile before executing crsctl command.

crsctl add css votedisk new_vote_disk_path

To Delete voting disk, execute following command. Set ASM profile before executing crsctl command.

crsctl delete css votedisk new_vote_disk_path

To Add voting disk to ASM Disk, execute following command.

crsctl add votedisk asm_disk_group

To Replace or Migrate voting disk, execute following command.

crsctl replace votedisk asm_diskgroup / vote_disk_path

To find OCR backup location, execute following command. Set ASM profile before executing crsctl command.

ocrconfig –showbackup auto

To change OCR Backup location, execute following command.

ocrconfig –backuploc Shared_PATH

To Add OCR files location, execute following command.

ocrconfig –add Location_PATH

To Replace or Change OCR files location, execute following command.

ocrconfig –replace New_PATH

To Repair OCR files location, execute following command. Cluster or Oracle RAC Services should be power off.

ocrconfig –repair –add +DATAC1

There are lots of options used with SRVCTL command, i described them below.

-d Database Name
-i Instance Name
-s Service Name
-n Node Name
-r Preferred list
-a Available list
-p TAF (Transparent application failover policy)
-v Verbose

Click this link for srvctl commands here.

Category Archives: RAC