Monday, December 29, 2014

[Clusterware] Check if storage is shared

If you suspect that your Clusterware is not working properly and you have these conditions:
- Disk mapping is the same.
- Disk major and minor are the same.
- Instead of one cluster you have two different clusters.
- You can't add a new node on the cluster.

Then you have to verify if your storage is shared.

This is the procedure. Please execute it on all the nodes:
1. Stop the Clusterware:
crsctl stop crs -f

2. Record the name of the vfstart and vfend of the Voting Disk:
kfed read /dev/oracleasm/disks/VOTE2|grep vf kfdhdb.vfstart: 320 ; 0x0ec: 0x00000140 kfdhdb.vfend: 352 ; 0x0f0: 0x00000160 

3. Perform a dd dump according to the above numbers: 
dd if=/dev/oracleasm/disks/VOTE2 bs=1048576 skip=320 count=33 |od -c > /tmp/dd<node>.log 

4. Gather the results from previous steps and compare the logs:
diff /tmp/dd<node1>.log /tmp/dd<node2>.log 

If there is any difference between the logs, then the Voting Disk is not shared between the nodes. In that case you have to contact the system administrator to check further.

Friday, December 12, 2014

ORA-19510 and ORA-27045 on RMAN Backup

Situation:
Doing RMAN backup I got this error message:
ORA-19510: failed to set size of 1193456 blocks for file "/backup" (block size=8192)ORA-27045: unable to close the fileLinux-x86_64 Error: 5: Input/output error

Cause:
The backup location is a Windows NFS share. It has a limitation that the backup piece cannot be greater than 2GB. Our backup piece is 9.1GB (1193456*8192/1024/1024/1024).

Solution:
Make the RMAN backup piece respect the file size limit. For this we can use the MAXPIECESIZE option. This is an example of the usage of the command:
RMAN> configure channel device type disk MAXPIECESIZE = 2G;

Wednesday, November 19, 2014

[OEL] Warniung: pci_mmcfg_init marking 256MB space uncacheable

Situation:
The following warning appears in the /var/log/messages file:
Warning: pci_mmcfg_init marking 256MB space uncacheable.
Solution:
Modify the grub.conf file with the kernel parameter "acpi_mcfg_max_pci_bus_num=on" and then reboot the server:

#hiddenmenu
title Enterprise Linux Enterprise Linux Server (2.6.32-100.24.1.el5)
root (hd0,0)
kernel /vmlinuz-2.6.32-100.24.1.el5 ro root=/dev/vg00/lv00 console=tty0 acpi_mcfg_max_pci_bus_num=on
initrd /initrd-2.6.32-100.24.1.el5.img

Tuesday, November 18, 2014

[OEL] Change Hostname

Here are the steps for chaging the server hostname:
1. With a text editor (like vi), open the file /etc/sysconfig/network.
2. Change the value of the entry HOSTNAME=.
3a. Restart the xinetd service:
service xinetd restart
3b. Reboot the machine:
init 6
4. Change any entries on the /etc/hosts file.

Problem solved!

Friday, October 31, 2014

[RDBMS] Redo Log Switch Frequency Map

This query will help you understand the log switching completion frequency on your database:
select to_char(first_time,'YYYY-MON-DD') day,
to_char(sum(decode(to_char(first_time,'HH24'),'00',1,0)),'99') "00",
to_char(sum(decode(to_char(first_time,'HH24'),'01',1,0)),'99') "01",
to_char(sum(decode(to_char(first_time,'HH24'),'02',1,0)),'99') "02",
to_char(sum(decode(to_char(first_time,'HH24'),'03',1,0)),'99') "03",
to_char(sum(decode(to_char(first_time,'HH24'),'04',1,0)),'99') "04",
to_char(sum(decode(to_char(first_time,'HH24'),'05',1,0)),'99') "05",
to_char(sum(decode(to_char(first_time,'HH24'),'06',1,0)),'99') "06",
to_char(sum(decode(to_char(first_time,'HH24'),'07',1,0)),'99') "07",
to_char(sum(decode(to_char(first_time,'HH24'),'08',1,0)),'99') "08",
to_char(sum(decode(to_char(first_time,'HH24'),'09',1,0)),'99') "09",
to_char(sum(decode(to_char(first_time,'HH24'),'10',1,0)),'99') "10",
to_char(sum(decode(to_char(first_time,'HH24'),'11',1,0)),'99') "11",
to_char(sum(decode(to_char(first_time,'HH24'),'12',1,0)),'99') "12",
to_char(sum(decode(to_char(first_time,'HH24'),'13',1,0)),'99') "13",
to_char(sum(decode(to_char(first_time,'HH24'),'14',1,0)),'99') "14",
to_char(sum(decode(to_char(first_time,'HH24'),'15',1,0)),'99') "15",
to_char(sum(decode(to_char(first_time,'HH24'),'16',1,0)),'99') "16",
to_char(sum(decode(to_char(first_time,'HH24'),'17',1,0)),'99') "17",
to_char(sum(decode(to_char(first_time,'HH24'),'18',1,0)),'99') "18",
to_char(sum(decode(to_char(first_time,'HH24'),'19',1,0)),'99') "19",
to_char(sum(decode(to_char(first_time,'HH24'),'20',1,0)),'99') "20",
to_char(sum(decode(to_char(first_time,'HH24'),'21',1,0)),'99') "21",
to_char(sum(decode(to_char(first_time,'HH24'),'22',1,0)),'99') "22",
to_char(sum(decode(to_char(first_time,'HH24'),'23',1,0)),'99') "23"
from v$log_history
group by to_char(first_time,'YYYY-MON-DD') order by day;
Enjoy!

Monday, August 11, 2014

[ODA] How to create and delete an Oracle Cloud FS (a.k.a. ACFS)



This is other post that can help you with ODA Systems.

For upgrade purpose I need to create another filesystem, cause my dmp file is almost 200GB, and I don't have enough space in the other ones.

Enough talking, let's get started.

From your Node 1 (my-oda1) and with grid user execute the next steps:

1. If this is a new ODA you "must" have plenty space in your Diskgroups (+DATA, +REDO, +RECO) so, space is not a topic you should be worry about. But if you have database created in your ODA, you should take a look around how much space you have available in your diskgroups.

Query the space available, in ASM instance :

su - grid
export ORACLE_SID=+ASM1
sqlplus / as sysdba

SQL> select name,total_mb,free_mb from v$asm_diskgroup;

NAME                             TOTAL_MB    FREE_MB
------------------------------ ---------- ----------
DATA                              7372800    2320020
RECO                              9796800    9714460
REDO                               763120     729800

I recommend you, create the volume in +DATA disk group.



Wednesday, August 6, 2014

[ODA] ILOM (Integrated Lights Out Manager) URL



Well, recently, I'm having some experience with Oracle Database Appliance (ODA), so I will be posting some notes about that.

I was researching about ILOM (Integrated Lights Out Manager) which is the interface that you can use to manage and maintain your ODA system remotely.

First experience,  I didn't know anything about the ILOM, so I didn't have a clue what was the URL for this magic console.

Don't panic, the only thing that you need to know is the ILOM IP address in both ODA nodes (each node have their ILOM configure).

[RUEI] Upgrade Fails with Error: "ORA-00600: internal error code, arguments: [kkmmctbf:bad intcoln]"

Situation:
I'm upgrading from my current 12.1.0.X RUEI installation to the version 12.1.0.6 due to a malfunction of the TCP Diagnostic tool.
 
I followed the steps on the "Oracle® Real User Experience Insight Installation Guide 12c Release 5 (12.1.0.6) for Linux x86-64". I reached the step 13 on the section "3.11.1 Upgrading the Reporter System from RUEI 12.1.0.x" and got the following error:
[root@gualaca extra]# ./ruei-upgrade-12.1.0.6.sh rpm_post_install
____________________________________________________________

2014-08-05 14:52:18   rpm_post_install ...
____________________________________________________________

2014-08-05 14:52:18   make_processor_conf ...
2014-08-05 14:52:19   make_processor_conf done
____________________________________________________________
...
2014-08-05 16:29:47   ignore_failed_urls ...
2014-08-05 16:29:48   ignore_failed_urls done
____________________________________________________________

2014-08-05 16:29:48   fix_sesdiag_retention ...
2014-08-05 16:29:48   fix_sesdiag_retention done
____________________________________________________________

2014-08-05 16:29:48   delete_unreferenced_rules ...
    declare
*
ERROR at line 1:
ORA-00600: internal error code, arguments: [kkmmctbf:bad intcoln], [6], [], [],
[], [], [], [], [], [], [], []
ORA-06512: at line 22
ORA-06512: at line 26


2014-08-05 16:31:27   delete_unreferenced_rules ERROR
____________________________________________________________

2014-08-05 16:31:27   convert_psesdiag_ecid ...
2014-08-05 16:31:28   convert_psesdiag_ecid done
____________________________________________________________

2014-08-05 16:31:28   migrate_interfaces ...
2014-08-05 16:31:28   migrate_interfaces done
____________________________________________________________

2014-08-05 16:31:28   drop_unused_userflow_tables ...
2014-08-05 16:31:29   drop_unused_userflow_tables done
2014-08-05 16:31:29   rpm_post_install ERROR

It seems like an error with the RUEI database.
 
Cause:
The error occurs on delete a row from a table that has a child table referenced with an ON DELETE CASCADE foreign key, and the child table contains a compound trigger.
 
Bug:
1435647.1: ORA-600 [kkmmctbf:bad intcoln] 
 
Solution:
Install this patch (13582702) on your RUEI database.
 
Note: The patch I'm sharing is for Linux x86-64. There are similar patches for other environments.

Tuesday, August 5, 2014

[RUEI] Issue with ruei-upgrade-12.1.0.6.sh rpm_pre_install: -bash: line 2: /tmp/ruei-migrate: Permission denied

Situation:
I'm upgrading from my current 12.1.0.X RUEI installation to the version 12.1.0.6 due to a malfunction of the TCP Diagnostic tool.
 
I followed the steps on the "Oracle® Real User Experience Insight Installation Guide 12c Release 5 (12.1.0.6) for Linux x86-64". I reached the step 7 on the section "3.11.1 Upgrading the Reporter System from RUEI 12.1.0.x" and got the following error:
[root@gualaca extra]# ./ruei-upgrade-12.1.0.6.sh rpm_pre_install
____________________________________________________________

2014-08-04 14:29:22 rpm_pre_install ...
____________________________________________________________

2014-08-04 14:29:22 check_markers ...
2014-08-04 14:29:23 check_markers done
____________________________________________________________

2014-08-04 14:29:23 drop_mviews ...
2014-08-04 14:29:23 drop_mviews done
____________________________________________________________

2014-08-04 14:29:23 fix_custom_dims ...
2014-08-04 14:29:23 fix_custom_dims done
____________________________________________________________

2014-08-04 14:29:23 rename_fact_tables ...
2014-08-04 14:29:24 rename_fact_tables done
____________________________________________________________

2014-08-04 14:29:24 rename_table_prefix ...
2014-08-04 14:29:24 rename_table_prefix done
____________________________________________________________

2014-08-04 14:29:24 page_stats ...
2014-08-04 14:29:25 page_stats done
____________________________________________________________

2014-08-04 14:29:25 upgrade_report ...
2014-08-04 14:29:25 upgrade_report done
____________________________________________________________

2014-08-04 14:29:25 prepare_kpi_upgrade ...
-bash: line 2: /tmp/ruei-migrate: Permission denied
2014-08-04 14:29:25 prepare_kpi_upgrade ERROR
2014-08-04 14:29:25 rpm_pre_install ERROR

As it seems like a file permission error, I gave all the permissions (777) to the /tmp/ruei-migrate file to see if it was the problem but after I execute it I got the same permission error:
[root@gualaca tmp]# ls -ld ruei-migrate
-rwxrwxrwx 1 root root 1761546 Aug 4 14:51 ruei-migrate
[root@gualaca ~]# /tmp/ruei-migrate
-bash: /tmp/ruei-migrate: Permission denied
Then I created a script (in the /tmp directory) that echoes the string "Hello World" but when I execute it a permission error appears (like ruei-migrate script). So I pointed the problem to the /tmp filesystem. I review the /etc/fstab file and found that the /tmp filesystem does not have the exec option:
[root@gualaca ~]# cat /etc/fstab
/dev/VolGroup00/LogVol02 /tmp ext3 defaults,nodev,nosuid,noexec  1 2

Cause:
The filesystem for the /tmp directory does not have the exec mount option. This option permits the filesystem to execute its binaries.
 
Solution:
Remount the /tmp filesystem with the proper option and try again the RUEI upgrade script:
[root@gualaca ~]# mount -o remount,exec /tmp
[root@gualaca extra]# ./ruei-upgrade-12.1.0.6.sh rpm_pre_install
____________________________________________________________

2014-08-05 10:14:07   rpm_pre_install ...
____________________________________________________________

2014-08-05 10:14:07   check_markers ...
2014-08-05 10:14:08   check_markers done
____________________________________________________________

2014-08-05 10:14:08   drop_mviews ...
2014-08-05 10:14:08   drop_mviews done
____________________________________________________________

2014-08-05 10:14:08   fix_custom_dims ...
2014-08-05 10:14:09   fix_custom_dims done
____________________________________________________________

2014-08-05 10:14:09   rename_fact_tables ...
2014-08-05 10:14:09   rename_fact_tables done
____________________________________________________________

2014-08-05 10:14:09   rename_table_prefix ...
2014-08-05 10:14:10   rename_table_prefix done
____________________________________________________________

2014-08-05 10:14:10   page_stats ...
2014-08-05 10:14:10   page_stats done
____________________________________________________________

2014-08-05 10:14:10   upgrade_report ...
2014-08-05 10:14:11   upgrade_report done
____________________________________________________________

2014-08-05 10:14:11   prepare_kpi_upgrade ...
2014-08-05 10:14:12   prepare_kpi_upgrade done
2014-08-05 10:14:12   rpm_pre_install done

[RUEI] ./ruei-migrate: error while loading shared libraries: libclntsh.so.11.1

Situation:
Attempting to run the ruei-migrate fails with this errror:
./ruei-migrate: error while loading shared libraries: libclntsh.so.11.1: cannot open shared object file: No such file or directory

Cause:
The problem was caused by the $ORACLE_HOME/lib directory not being exported on the LD_LIBRARY_PATH environment variable. It is also related to the RUEI database environment variables that are not included on the bash_profile file of the $RUEI_USER.
 
Solution:
After appending the RUEI database environment variables to the $RUEI_USER bash_profile file, add $ORACLE_HOME/lib to the LD_LIBRARY_PATH environment variable using the following command:
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH

Monday, August 4, 2014

[RUEI] Upgrade Missing Dependency php-soap RPM

Situation:
You are following the upgrade instructions from the "Oracle® Real User Experience Insight Installation Guide 12c Release 5 (12.1.0.6) for Linux x86-64" and you reach the step where you upgrade the php-soap module but don't have the proper RPM.
 

Solution:
Option A
- Download the RPM from this link.

Option B
- Configure the Oracle Public Yum Server and install the repository. Then, install the package with the yum command.

Monday, July 28, 2014

[RUEI] No Data on Collector

Situation:
The network administrator configure a SPAN Port for one network interface on your RUEI Server, you tested it using the operating system command tcpdump (validating that the traffic from your applications have outputs) and you do not see any results on your RUEI web console:



Solution:
1. Connect as the $RUEI_USER on your Reporter system.
2. Execute the following commands:
execsql config_set_profile_value wg System packetfilter rawfilter replace "eth_all~(tcp)~or~(vlan~and~(tcp))"
execsql config_set_profile_value wg System packetfilter filterformat replace "tcpdump"
3. Restart the RUEI data processing using this commands:
project -stop wg
project -start wg
Note: My RUEI architecture is all deployed on the same server. I only have one Collector (which profile is System). If you have other Collector you should follow these steps on that Collector too but using the correct profile name.

Wednesday, July 2, 2014

[OAS] Installing Oracle Application Server 10gR3 with Oracle Database 11gR2

Situation:
You want to install an Oracle Application Server 10gR3 (10.1.3) that will use an Oracle Database 11gR2 (11.2.0.3). During the installation process you find the following error:

 
Bug Related:
Bug 6265268 - 10.1.3.1SOA INSTALL FAILS AGAINST 11G DATABASE: The installation CD got an old version of the files DBConnectQueries.jar and IP_DBQueries.jar.

Solution:
1. Download the patch p6265268_101310_GENERIC.zip to the server.
2. Extract its content. It should be both jars.
3. Copy the DBConnectQueries.jar to the following location:
Disk1/stage/Queries/DBConnectQueries/8.4/1/
4. Copy the IP_DBQueries.jar to the following location:
Disk1/stage/Queries/IP_DBQueries/3.0/1/
5. Try the installation again. The error should be gone.

Friday, June 27, 2014

[OEM] Can't add Compliance Standard Rule to Compliance Standard

Situation:
You want to add a custom Compliance Standard Rule to a custom Compliance Standard but when you click the 'Save' button the following error appears:




Also when you review your emoms.log you found the following entry:
2014-06-26 08:34:28,675 [[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'] WARN applicationModule.ComplianceLibraryAMImpl logp.251 - SQL Exception
java.sql.SQLException: ORA-01403: no data found
ORA-06512: at "SYSMAN.EM_COMPLIANCE_RULE_OP", line 70
ORA-06512: at "SYSMAN.EM_COMPLIANCE_STANDARD_OP", line 2585
ORA-06512: at line 1

at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:462)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:405)
at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:931)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:481)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:205)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:548)
at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:213)
at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:1111)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1488)
.......... 

Bug Related:
Bug 18380944 - COMPLIANCE STANDARD UPDATE FAIL: When a target is deleted from the Cloud Control, the custom compliance standards associated to it are not deleted.

Solution:
1. Upload the cleanup_stale_data.sql to the repository database server.
2. Login to the repository database as SYSMAN user.
3. Run the cleanup_stale_data.sql script:
SQL>@cleanup_stale_data.sql
4. Execute the following query. It result should be zero:
SQL> select count(*) from sysman.em_cs_tgt_assoc_txf_req where root_target_guid not in (select target_guid from mgmt_targets);

Monday, February 10, 2014

[WLS] BEA-141281 Unable to get file lock, will retry

Scenario:
While starting the Admin/Managed Server, you encounter the following error:
<Feb 10, 2014 10:42:51 AM EST> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>


Solution:
Remove the AdminServer.lok or ManagedServer.lok file from the following directory:
$DOMAIN_HOME/servers/<name_of_the_server>/tmp