Friday, June 12, 2015

[Cloud Control 12c] Agent Unreachable + [UPLOAD_SYSTEM_FATAL_ERROR]

Apparently the agent shows in the Cloud Control that it is unreachable from the OMS. When you try to restart it from the console (as the Oracle best practices says) it succeeded but still no green arrow on the agent.

Sorry but you have to go a little deeper than the console control to start and stop, no worries when you see it is pretty easy.

With the owner of the agent check the status of the agent from the server:

emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------


Agent is Not Running

I started the agent thinking that that was the problem, the agent starts without any problem. I execute the status that showed me the real problem:

emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 12.1.0.3.0
OMS Version       : 12.1.0.3.0
Protocol Version  : 12.1.0.1.0
Agent Home        : /u01/wwwww/ora/Middleware/agent_inst
Agent Binaries    : /u01/wwwww/ora/Middleware/core/12.1.0.3.0
Agent Process ID  : 16985
Parent Process ID : 16967
Agent URL         : https://yyyyy:3839/emd/main/
Repository URL    : https://xxxxx:1139/empbs/upload
Started at        : 2015-06-11 15:09:43
Started by user   : oracle
Last Reload       : (none)
Last successful upload                       : (none)
Last attempted upload                        : (none)
Total Megabytes of XML files uploaded so far : 0
Number of XML files pending upload           : 836
Size of XML files pending upload(MB)         : 0.94
Available disk space on upload filesystem    : 53.77%
Collection Status                            : [UPLOAD_SYSTEM_FATAL_ERROR]
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2015-06-11 15:13:02
Last successful heartbeat to OMS             : 2015-06-11 15:13:02
Next scheduled heartbeat to OMS              : 2015-06-11 15:14:02

---------------------------------------------------------------
Agent is Running and Ready

Check on log gcagent_errors.log:

2015-06-11 15:11:02,182 [38:GC.Upload[0]] ERROR - Disabling the upload system due to protocol internal programming exception
java.lang.NullPointerException
        at oracle.sysman.gcagent.upload.UploadPropertiesFile$RestoredMessageProperties.<init>(UploadPropertiesFile.java:124)
        at oracle.sysman.gcagent.upload.UploadPropertiesFile.load(UploadPropertiesFile.java:164)
        at oracle.sysman.gcagent.upload.UploadStoreForward.loadMessageProperties(UploadStoreForward.java:3787)
        at oracle.sysman.gcagent.upload.UploadStoreForward$dequeuer.xferFile(UploadStoreForward.java:2725)
        at oracle.sysman.gcagent.upload.UploadStoreForward$dequeuer._run(UploadStoreForward.java:3562)
        at oracle.sysman.gcagent.upload.UploadStoreForward$dequeuer.run(UploadStoreForward.java:3452)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)

Look out in My Oracle Support and found this Note: EM 12c : emctl status agent 'Collection Status Shows UPLOAD_SYSTEM_FATAL_ERRORL COLLECTIONS_HALTED DISABLED UPLOAD_SYSTEM_FATAL_ERROR' 'OMS is unreachable' (Doc ID 1996341.1)

The solution as follows:
1. Stop the agent:

emctl stop agent
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Stopping agent ..... stopped.

If there is any agent process remains kill them:

$ps -ef | grep java | grep '<agent_home >'
$ps -ef | grep perl
$kill -9  <Process id>

2. Stop the agent:
Take a backup of the file AGENT_INST/sysman/config/emd.properties and change the next parameter value:

old:
UploadMaxBytesXML = 10240
new:
UploadMaxBytesXML = 50

Now I have it set to 50, but my agent is still "Unreachable"

3. Move all the files from following locations to new location 

$/AGENT_INST/sysman/emd/upload/upload/properties/
$/AGENT_INST/sysman/emd/upload/pending/*.xml
$/AGENT_INST/sysman/emd/upload/upload/sending/*.xml
$/AGENT_INST/sysman/emd/upload/upload/badfile/*.xml

Example:
$ mv /AGENT_INST/sysman/emd/upload/upload/properties/* /u01/agentproperties/ 

The note encourage you to apply a patch, if you are in 12.1.0.2 (I am in 12.1.0.3): 

Note : For 12.1.0.2 agent apply Patch 16421783

4. Secure, start agent and verify the collection status using emctl status agent:

Secure the agent 
If you don't remember the Agent Registration Password you can create a password from the CC Console Setup > Security > Registration Password > Add registration password

emctl secure agent
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Agent is already stopped...   Done.

Securing agent...   Started.
Enter Agent Registration Password : 
EMD gensudoprops completed successfully
Securing agent...   Successful.

Start the agent:

emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Starting agent .................................. started.


Agent status:

emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 12.1.0.3.0
OMS Version       : 12.1.0.3.0
Protocol Version  : 12.1.0.1.0
Agent Home        : /u01/wwwww/ora/Middleware/agent_inst
Agent Binaries    : /u01/wwwww/ora/Middleware/core/12.1.0.3.0
Agent Process ID  : 28039
Parent Process ID : 28015
Agent URL         : https://yyyyy:3839/emd/main/
Repository URL    : https://xxxxx:1139/empbs/upload
Started at        : 2015-06-11 15:39:11
Started by user   : oracle
Last Reload       : (none)
Last successful upload                       : 2015-06-11 15:39:59
Last attempted upload                        : 2015-06-11 15:39:59
Total Megabytes of XML files uploaded so far : 0.09
Number of XML files pending upload           : 3
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 53.76%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2015-06-11 15:39:38
Last successful heartbeat to OMS             : 2015-06-11 15:39:38
Next scheduled heartbeat to OMS              : 2015-06-11 15:40:40

---------------------------------------------------------------
Agent is Running and Ready

and Upload:

emctl upload agent
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully

After that you should see the green arrow on your agent status.