Wednesday, October 2, 2019

OEM Agent Diagnose for Status Pending (Post Blackout)

Sometimes OEM agents will be in status pending status, after ending the blackouts related to maintenance (operating system, Networking, Database, Application). One of my colleagues calls it out as Wonky state. The steps you need to perform to bring it in a green and clean state to start monitoring your databases are as below.



[oracle@sajidserver emd]$ emctl stop agent
[oracle@sajidserver emd]$ cd $AGENT_HOME/sysman/emd
[oracle@sajidserver emd]$ rm -rf state/*
[oracle@sajidserver emd]$ rm -rf upload/*
[oracle@sajidserver emd]$ rm -rf collection/*
[oracle@sajidserver emd]$ rm agntstmp.txt
[oracle@sajidserver emd]$ rm blackouts.xml
[oracle@sajidserver emd]$ emctl clearstate agent
[oracle@sajidserver emd]$ emctl start agent

You can see agents starts normally, but emctl status agents spit out the below error with proper reasoning.


[oracle@sajidserver emd]$ emctl status agent
Blocked Reason: Agent is out-of-sync with repository. This most likely means that the agent was reinstalled or recovered. Please contact an EM administrator to unblock the agent by performing an agent resync from the console.

And even emctl upload agent fails:

[oracle@sajidserver emd]$ emctl upload agent
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload error: full upload has failed: The agent is blocked by the OMS. Agent is out-of-sync with repository. This most likely means that the agent was reinstalled or recovered. Please contact an EM administrator to unblock the agent by performing an agent resync from the console. (AGENT_BLOCKED)

Now drill down into the specific target as below and hit Resynchronization. After that follow the screen with unblocking the agent and submitting a confirmation job as below screens.






Give it a few minutes, then your agent will appear in up and green status as below.




Happy Monitoring :).