DOYENSYS Knowledge Portal




We Welcome you to visit our DOYENSYS KNOWLEDGE PORTAL : Doyensys Knowledge Portal




Wednesday, May 31, 2017

Clusterware Configuration issue Resolution


In 3 node rac setup. In the process of root.sh script execution in first 2 nodes were completed successfully.  But when we run the script in the 3'rd  we faced the issue "Failed to start Nodeapps" .


#1 Error Detail:

Using configuration parameter file: /CRS/app/11.2.0/grid_new/crs/install/crsconfig_params
User ignored Prerequisites during installation
PRKO-2190 : VIP exists for node testdb6, VIP name testdb6-vip
/CRS/app/11.2.0/grid_new/bin/srvctl start nodeapps -n testdb6 ... failed
Failed to start Nodeapps at /CRS/app/11.2.0/grid_new/crs/install/crsconfig_lib.pm line 9217.
[NEW root@testdb6 ~]#


Cause analysis:

#1 Check the Logfile:

Logfile : /CRS/app/11.2.0/grid_new/cfgtoollogs/crsconfig/rootcrs_testdb6.log

2017-05-31 00:12:51: starting nodeapps...
2017-05-31 00:12:51: DHCP_flag=0
2017-05-31 00:12:51: nodes_to_start=testdb6
2017-05-31 00:15:05: exit value of start nodeapps/vip is 1
2017-05-31 00:15:05: output for start nodeapps is  PRKO-2421 : Network resource is already started on node(s): testdb6 PRKO-2420 : VIP is already started on node(s): testdb6 PRCR-1013 : Failed to start resource ora.ons PRCR-1064 : Failed to start resource ora.ons on node testdb6 CRS-5014: Agent "/CRS/app/11.2.0/grid_new/bin/oraagent.bin" timed out


#2 Check Cluster Resource Status:

[NEW oracle@testdb6 crsconfig]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------

ora.ons
               ONLINE  ONLINE       testdb4
               ONLINE  ONLINE       testdb5
               OFFLINE  OFFLINE       testdb6        CHECK TIMED OUT



#3 Check the nodeapps status:

srvctl config nodeapps
Network exists: 1/10.80.1.0/255.255.255.0/eth0, type static
VIP exists: /testdb4-vip/10.80.1.72/10.80.1.0/255.255.255.0/eth0, hosting node testdb4
VIP exists: /testdb5-vip/10.80.1.65/10.80.1.0/255.255.255.0/eth0, hosting node testdb5
VIP exists: /testdb6-vip/10.80.1.165/10.80.1.0/255.255.255.0/eth0, hosting node testdb6
GSD exists
ONS exists: Local port 6100, remote port 6200, EM port 2016

#4 Check the connection status:

[NEW oracle@testdb6 ~]$ /CRS/app/11.2.0/grid/bin/onsctli ping

Communication error with the OPMN server local port.
Check the OPMN log files

^Cons is not running ...


Resolve this issue:

#1 Check the ons process:

[NEW oracle@testdb6 ~]$ ps -ef | grep ons
root      2770     1  0 May22 ?        00:00:27 sendmail: accepting connections
root      2909     1  0 May22 hvc0     00:00:00 /sbin/agetty serial_console 9600 vt100-nav
oracle   18960     1  0 May26 ?        00:00:00 /CRS/app/11.2.0/grid/opmn/bin/ons -d
oracle   18961 18960  0 May26 ?        00:05:55 /CRS/app/11.2.0/grid/opmn/bin/ons -d
oracle   20786 20701  0 01:23 pts/0    00:00:00 grep ons


#2 Kill the process manually:

[NEW oracle@testdb6 bin]$ kill -9 18960
[NEW oracle@testdb6 bin]$ kill -9 18961

#3 Check the ons process:

[NEW oracle@testdb6 bin]$ ps -ef | grep ons
root      2770     1  0 May22 ?        00:00:27 sendmail: accepting connections
root      2909     1  0 May22 hvc0     00:00:00 /sbin/agetty serial_console 9600 vt100-nav
oracle   20823 20701  0 01:26 pts/0    00:00:00 grep ons

Note : no ons process is running.

#4 Start the Ons Process:

[NEW oracle@testdb6 bin]$ ./onsctl start
CRS-2672: Attempting to start 'ora.net1.network' on 'testdb6'
CRS-2676: Start of 'ora.net1.network' on 'testdb6' succeeded
CRS-2679: Attempting to clean 'ora.ons' on 'testdb6'
CRS-2681: Clean of 'ora.ons' on 'testdb6' succeeded
CRS-2672: Attempting to start 'ora.ons' on 'testdb6'
CRS-2676: Start of 'ora.ons' on 'testdb6' succeeded

#5 Check the ons status:

[NEW oracle@testdb6 bin]$ ./onsctl ping
ons is running ...
[NEW oracle@testdb6 bin]$ exit


#6 Re-run the root script:

[NEW root@testdb6 crsconfig]# /CRS/app/11.2.0/grid_new/perl/bin/perl -I/CRS/app/11.2.0/grid_new/perl/lib -I/CRS/app/11.2.0/grid_new/crs/install /CRS/app/11.2.0/grid_new/crs/install/rootcrs.pl
Using configuration parameter file: /CRS/app/11.2.0/grid_new/crs/install/crsconfig_params
User ignored Prerequisites during installation
PRKO-2190 : VIP exists for node testdb6, VIP name testdb6-vip
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[NEW root@testdb6 crsconfig]#

#7 Check the resource status:

[NEW oracle@testdb6 crsconfig]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------

ora.ons
               ONLINE  ONLINE       testdb4
               ONLINE  ONLINE       testdb5
               ONLINE  ONLINE       testdb6


Reference:

In 11.2 and 12c, stopping ons fails. The rootupgrade.sh, "crsctl stop crs -f", and "srvctl stop nodeapps" fail because ons fails to stop. (Doc ID 2038830.1)

Monday, May 29, 2017

Steps to resolve Error 1031 received logging on to the standby


One of our DataGuard environment was out of sync , When we checked in the
v$managed_process data dictionary view, we could see that RFS was not starting.

Error :

 "Error 1031 received logging on to the standby".

Findings:

We tried to connect to the standby instance from primary by using the following command and it was unsuccessful.
  sqlplus sys/password@DR.

The time stamp of the password file on the primary and the standby was different. This indicated the possibility of having the wrong password file in the standby.


Solution:

we copied the password file from the primary to the standby and renamed the password file in the following format  ie orapw<sid> . 
we restarted the media recovery process on the standby. 
Once the above steps are completed we could see that redo shipping and redo apply is taking place

Now the Primary and standby is in Sync.

Saturday, May 27, 2017

Configure the Patch fs in R12.2

Scenario:

Assuming the Run and Patch fs are going to be in single node and the Run fs is already configured and the instance is up and running.

Steps to configure the patch File system in R12.2

Now to configure the patch fs.  Do the following

1.   Run the preclone in the Run fs.
[appdoyen@doyserv1 scripts]$ pwd
/test/app/TEST/fs2/inst/apps/TEST_doyserv1/admin/scripts

[appdoyen@doyserv1 scripts]$ adpreclone.pl appsTier
Copyright (c) 2011, 2014 Oracle Corporation
                        Redwood Shores, California, USA
                        Oracle E-Business Suite Rapid Clone
                                 Version 12.2
                      adpreclone Version 120.31.12020000.20
Enter the APPS User Password:
Enter the Weblogic AdminServer password :
 Checking the status of the Oracle WebLogic Administration Server....
 Running perl /test/app/TEST/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/adProvisionEBS.pl ebs-get-serverstatus -contextfile=/test/app/TEST/fs2/inst/apps/TEST_doyserv1/appl/admin/TEST_doyserv1.xml -servername=AdminServer -promptmsg=hide
The Oracle WebLogic Administration Server is up.
Running:
perl /test/app/TEST/fs2/EBSapps/appl/ad/12.0.0/bin/adclone.pl java=/test/app/TEST/fs2/EBSapps/comn/util/jdk64 mode=stage stage=/test/app/TEST/fs2/EBSapps/comn/clone component=appsTier method= appctx=/test/app/TEST/fs2/inst/apps/TEST_doyserv1/appl/admin/TEST_doyserv1.xml showProgress
 Setting the wls environment
Beginning application tier Stage - Mon May 22 18:39:00 2017
/test/app/TEST/fs2/EBSapps/comn/util/jdk64/bin/java -Xmx600M -DCONTEXT_VALIDATED=false -Doracle.installer.oui_loc=/oui -classpath /test/app/TEST/fs2/FMW_Home/webtier/lib/xmlparserv2.jar:/test/app/TEST/fs2/FMW_Home/webtier/jdbc/lib/ojdbc6.jar:/test/app/TEST/fs2/EBSapps/comn/java/classes:/test/app/TEST/fs2/FMW_Home/webtier/oui/jlib/OraInstaller.jar:/test/app/TEST/fs2/FMW_Home/webtier/oui/jlib/ewt3.jar:/test/app/TEST/fs2/FMW_Home/webtier/oui/jlib/share.jar:/test/app/TEST/fs2/FMW_Home/webtier/../Oracle_EBS-app1/oui/jlib/srvm.jar:/test/app/TEST/fs2/FMW_Home/webtier/jlib/ojmisc.jar:/test/app/TEST/fs2/FMW_Home/wlserver_10.3/server/lib/weblogic.jar:/test/app/TEST/fs2/FMW_Home/oracle_common/jlib/obfuscatepassword.jar  oracle.apps.ad.clone.StageAppsTier -e /test/app/TEST/fs2/inst/apps/TEST_doyserv1/appl/admin/TEST_doyserv1.xml -stage /test/app/TEST/fs2/EBSapps/comn/clone -tmp /tmp -method CUSTOM   -showProgress -nopromptmsg
Log file located at /test/app/TEST/fs2/inst/apps/TEST_doyserv1/admin/log/clone/StageAppsTier_05221839.log

  /     20% completed

Completed Stage...

2.     Now create a empty patch fs directory in our case  (fs1).
/test/app/TEST/fs1

3.    Now shut down the runfs (fs2) and copy the EBS directory from run to patch fs
[appdoyen@doyserv1 scripts]$ cd $RUN_BASE
[appdoyen@doyserv1 fs2]$ pwd
/test/app/TEST/fs2

[appdoyen@doyserv1 fs2]$ cp -rp EBSapps /test/app/TEST/fs1/

4.     Unset the env if anything is set already.
5.      Go inside /test/app/TEST/fs1/EBSapps/comn/clone/bin of Patch fs ie) fs1 here , and run adcfgclone.pl
6.      Run perl adcfgclone.pl appsTier
7.      Give the port pool incremented by one on the run fs. for ex. If run fs is running in port pool 36 you can give as 37. Also check if it is already allcated by using 

netstat -an|grep -i 8037
    
[appdoyen@doyserv1 bin]$ perl adcfgclone.pl appsTier

                     Copyright (c) 2002, 2015 Oracle Corporation
                        Redwood Shores, California, USA

                        Oracle E-Business Suite Rapid Clone

                                 Version 12.2

                      adcfgclone Version 120.63.12020000.56

              ***********************************************************
                            In AD-TXK Delta 7, we recommend you clone the run and patch
                            file systems in a single operation using the 'dualfs' option.
                            Separate cloning of the run and patch file systems will be deprecated
              ************************************************************

Enter the APPS password :

Enter the Weblogic AdminServer password :

Do you want to add a node (yes/no) [no] :
 Running: Context clone...
 Log file located at /test/app/TEST/fs1/EBSapps/comn/clone/bin/CloneContext_0522221213.log

Target System File Edition type [run] : patch

Enter the full path of Run File System Context file : /test/app/TEST/fs2/inst/apps/TEST_doyserv1/appl/admin/TEST_doyserv1.xml

Provide the values required for creation of the new APPL_TOP Context file.

Target System Fusion Middleware Home set to /test/app/TEST/fs1/FMW_Home

Target System Web Oracle Home set to /test/app/TEST/fs1/FMW_Home/webtier

Target System Appl TOP set to /test/app/TEST/fs1/EBSapps/appl

Target System COMMON TOP set to /test/app/TEST/fs1/EBSapps/comn

Target System Instance Top set to /test/app/TEST/fs1/inst/apps/TEST_doyserv1

Target System Port Pool [0-99] : 37

Checking the port pool 37
done: Port Pool 37 is free
Report file located at /test/app/TEST/fs1/inst/apps/TEST_doyserv1/admin/out/portpool.lst
The new APPL_TOP context file has been created :
  /test/app/TEST/fs1/inst/apps/TEST_doyserv1/appl/admin/TEST_doyserv1.xml
Check Clone Context logfile /test/app/TEST/fs1/EBSapps/comn/clone/bin/CloneContext_05348.log for details.

Running Rapid Clone with command:

Running:
perl /test/app/TEST/fs1/EBSapps/comn/clone/bin/adclone.pl java=/test/app/TEST/fs1/EBSapps/comn/clone/bin/../jre mode=apply stage=/test/app/TEST/fs1/EBSapps/comn/clone component=appsTier method=CUSTOM appctxtg=/test/app/TEST/fs1/inst/apps/TEST_doyserv1/appl/admin/TEST_doyserv1.xml showProgress contextValidated=true
  
FMW Pre-requisite check log file location : /test/app/TEST/fs1/EBSapps/comn/clone/FMW/logs/prereqcheck.log

Running: /test/app/TEST/fs1/EBSapps/comn/clone/FMW/t2pjdk/bin/java -classpath /test/app/TEST/fs1/EBSapps/comn/clone/prereq/webtier/Scripts/ext/jlib/engine.jar:/test/app/TEST/fs1/EBSapps/comn/clone/prereq/webtier/oui/jlib/OraPrereq.jar:/test/app/TEST/fs1/EBSapps/comn/clone/prereq/webtier/oui/jlib/OraPrereqChecks.jar:/test/app/TEST/fs1/EBSapps/comn/clone/prereq/webtier/oui/jlib/OraInstaller.jar:/test/app/TEST/fs1/EBSapps/comn/clone/prereq/webtier/oui/jlib/OraInstallerNet.jar:/test/app/TEST/fs1/EBSapps/comn/clone/prereq/webtier/oui/jlib/srvm.jar:/test/app/TEST/fs1/EBSapps/comn/clone/prereq/webtier/Scripts/ext/jlib/ojdl.jar:/test/app/TEST/fs1/EBSapps/comn/clone/prereq/webtier/Scripts/ext/jlib/ojdl2.jar:/test/app/TEST/fs1/EBSapps/comn/clone/prereq/webtier/Scripts/ext/jlib/ojdl-log4j.jar:/test/app/TEST/fs1/EBSapps/comn/clone/prereq/webtier/oui/jlib/xmlparserv2.jar:/test/app/TEST/fs1/EBSapps/comn/clone/prereq/webtier/oui/jlib/share.jar:/test/app/TEST/fs1/EBSapps/comn/clone/jlib/java oracle.apps.ad.clone.util.FMWOracleHomePreReqCheck -prereqCheckFMW -e /test/app/TEST/fs1/inst/apps/TEST_doyserv1/appl/admin/TEST_doyserv1.xml -stage /test/app/TEST/fs1/EBSapps/comn/clone -log /test/app/TEST/fs1/EBSapps/comn/clone/FMW/logs/prereqcheck.log

Beginning application tier Apply - Mon Feb 22 22:20:59 2016

/test/app/TEST/fs1/EBSapps/comn/clone/bin/../jre/bin/java -Xmx600M -DCONTEXT_VALIDATED=true -Doracle.installer.oui_loc=/oui -classpath /test/app/TEST/fs1/EBSapps/comn/clone/jlib/xmlparserv2.jar:/test/app/TEST/fs1/EBSapps/comn/clone/jlib/ojdbc6.jar:/test/app/TEST/fs1/EBSapps/comn/clone/jlib/java:/test/app/TEST/fs1/EBSapps/comn/clone/jlib/oui/OraInstaller.jar:/test/app/TEST/fs1/EBSapps/comn/clone/jlib/oui/ewt3.jar:/test/app/TEST/fs1/EBSapps/comn/clone/jlib/oui/share.jar:/test/app/TEST/fs1/FMW_Home/webtier/../Oracle_EBS-app1/oui/jlib/srvm.jar:/test/app/TEST/fs1/EBSapps/comn/clone/jlib/ojmisc.jar:/test/app/TEST/fs1/FMW_Home/wlserver_10.3/server/lib/weblogic.jar:/test/app/TEST/fs1/EBSapps/comn/clone/jlib/obfuscatepassword.jar  oracle.apps.ad.clone.ApplyAppsTier -e /test/app/TEST/fs1/inst/apps/TEST_doyserv1/appl/admin/TEST_doyserv1.xml -stage /test/app/TEST/fs1/EBSapps/comn/clone    -showProgress -nopromptmsg
Log file located at /test/app/TEST/fs1/inst/apps/TEST_doyserv1/admin/log/clone/ApplyAppsTier_05222221.log
  \    100% completed

Completed Apply...
Mon Feb 22 23:10:14 2016

Looking for incomplete CLONE record in ad_adop_session_patches table

The CLONE record status is COMPLETED

8.      Now source the env and startall .


Saturday, May 20, 2017

OTN Yathra-2017-Chennai

                                                                       10-JUNE-2017 (Saturday)


Venue: Loyola-ICAM College of Engineering and Technology (LICET)
Loyola Campus, Nungambakkam, Chennai – 600034, Tamil Nadu, India


About The Event

For Agenda, Location details -  http://otnyathra.in/chennai/ 
For Session Abstracts, Speaker Bio - http://otnyathra.in/downloads/

Thursday, May 4, 2017

Convert Oracle Form to XML using frmf2xml.bat

1. Convert Oracle Form to XML using frmf2xml.bat

Run the frmf2xml.bat by providing your FMB file with its location from the Oracle form’s instance home directory as shown below:

cd C:\Oracle\Middleware\asinst_1\bin

frmf2xml.bat D:\Oracle\Middleware\Oracle_FRHome1\forms\LEAVE_ENTRY.fmb

It will convert and save the XML file into same location of your form.