EMD pingOMS error: unable to connect to http server

 When you are working on OEM agent upgrade. Sometimes you might face an error EMD pingOMS error: unable to connect to http server at OMShost. [handshake has no peer].

It is a sort of bug, with no proper fix. You can try various workarounds related to OEM agents, but most of them will not be helpful.

Error:

oracle@sajidserver01.com:$ ./emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 4
Copyright (c) 1996, 2018 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version      : 13.4.0.0.0
OMS Version        : (unknown)
Protocol Version   : 12.1.0.1.0
Agent Home         : /u05/app/oracle/agent13c/agent_inst
Agent Log Directory: /u05/app/oracle/agent13c/agent_inst/sysman/log
Agent Binaries     : /u05/app/oracle/agent13c/agent_13.4.0.0.0
Core JAR Location  : /u05/app/oracle/agent13c/agent_13.4.0.0.0/jlib
Agent Process ID       : 1769
Parent Process ID      : 1668
Agent URL              : <Link>
Local Agent URL in NAT : <Link>
Repository URL              :  <Link>
Started at                  : 2022-11-22 10:04:29
Started by user             : oracle
Operating System            : AIX version 7.1 (ppc64)
Number of Targets           : 16
Last Reload                 : (none)
Last successful upload           : 2022-11-22 16:28:12
Last attempted upload            : 2022-11-22 16:28:12
Total Megabytes of XML files uploaded so far   :18.21
Number of XML files pending upload             : 0
Size of XML files pending upload(MB)           : 0.01
Available disk space on upload filesystem      : 52.07%
Collection Status                            : Collections enabled
Heartbeat Status                             : OMS is unreachable
Last attempted heartbeat to OMS              : 2022-11-22 16:28:29
Last successful heartbeat to OMS             : (none)
Next scheduled heartbeat to OMS              : 2022-11-22 16:28:29
---------------------------------------------------------------
Agent is Running and Ready

oracle@sajidserver01.com:$  ./emctl pingOMS
Oracle Enterprise Manager Cloud Control 13c Release 4
Copyright (c) 1996, 2018 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD pingOMS error: unable to connect to http server at OMShost. [handshake has no peer]

oracle@sajidserver01.com:$  ./emctl upload agent
Oracle Enterprise Manager Cloud Control 13c Release 4
Copyright (c) 1996, 2018 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload error:full upload has failed: uploadXMLFiles skipped :: OMS version not checked yet. If this issue persists check trace files for ping to OMS related errors. (OMS_DOWN)

Solution:

Drop the OEM targets related to this server from OEM. Decommission the agent as well. After that try installing the new agent. The beauty of this is you do not need to reconfigure the targets again, all the targets will be configured automatically. It will be an OEM self-healing process as part of automation.

Oracle Server Database components status as Upgraded rather than Valid after Upgrade

A few times the database components will be in upgraded mode in the PDB once you have completed your database upgrade from 12c to version 19c.

column comp_name format a30
column status format a10
select comp_name, version, status from dba_registry;

COMP_NAME                              VERSION              STATUS

--------------------------------   ------------------      ----------

Oracle Database Catalog Views          19.0.0.0.0           UPGRADED

Oracle Database Packages and Types     19.0.0.0.0           UPGRADED

JServer JAVA Virtual Machine           19.0.0.0.0           UPGRADED

Oracle XDK                             19.0.0.0.0           UPGRADED

Oracle Database Java Packages          19.0.0.0.0           UPGRADED

SYS@sajiddb1>alter session set "_oracle_script"=TRUE;
session altered

SYS@sajiddb1>alter pluggable database pdb$seed close immediate instances=all;

Pluggable database altered.

SYS@sajiddb1>alter pluggable database pdb$seed OPEN READ WRITE force;
Pluggable database altered.

SYS@sajiddb1>alter session set container=PDB$SEED;
session altered

To resolve it follow the below steps.

@ORACLE_HOME/rdbms/admin/catalog.sql
@ORACLE_HOME/rdbms/admin/catproc.sql
@ORACLE_HOME/rdbms/admin/utlrp.sql

Check the components, it will be normal now.

SYS@sajiddb1>alter pluggable database pdb$seed close immediate instances=all;
Pluggable database altered.

SYS@sajiddb1>alter pluggable database pdb$seed OPEN READ ONLY force;
Pluggable database altered.

SYS@sajiddb1>show pdbs;
CON_ID  CON_NAME     OPEN MODE   RESTRICTED
------ ------------ ----------- ------------
 2       PDB$SEED     READ ONLY    NO

SYS@sajiddb1>alter session set "_oracle_script"=FALSE;
session altered


column comp_name format a30
column status format a10
select comp_name, version, status from dba_registry;
COMP_NAME                              VERSION               STATUS
--------------------------------    ----------------      ----------
Oracle Database Catalog Views          19.0.0.0.0             VALID
Oracle Database Packages and Types     19.0.0.0.0             VALID
JServer JAVA Virtual Machine           19.0.0.0.0             VALID
Oracle XDK                             19.0.0.0.0             VALID
Oracle Database Java Packages          19.0.0.0.0             VALID

ORA-01586: database must be mounted EXCLUSIVE and not open for this operation

 You can choose among multiple options to drop Oracle databases. 

1. DBCA (Silent Method or GUI)
2. Command Line Interface (startup mount restrict and then issue drop database) using sqlplus or rman
3. Manually cleaning up files

SQL> drop database;
drop database
*
ERROR at line 1:
ORA-01586: database must be mounted EXCLUSIVE and not open for this operation

We are decommissioning the RAC database here. So we need to change the parameter cluster_database=FALSE.

Set your database profile.

srvctl stop database -d <DB_name>
SQL> startup mount;
SQL> alter system enable restricted session;
SQL> alter system set cluster_database=FALSE scope=spfile;
SQL>Drop Database;
Database dropped.

How to find temporary MySQL Root Password after installation

 Sometimes once MySQL installation is done. You will not be able to find the temporary password, here I was working on the latest MySQL version 8.0.30-commercial MySQL Enterprise Server.

[root@Sajidserver01 ~]# grep -i pass /var/log/mysqld.log

Remove the /var/lib/mysql and restart mysqld as below

[root@Sajidserver01 ~]#cd /var/lib
[root@Sajidserver01 ~]#rm -rf mysql 
[root@Sajidserver01 ~]#systemctl stop mysqld
[root@Sajidserver01 ~]#systemctl start mysqld

Once it is done, you will find the new temporary password generated in mysqld.log below.

[root@Sajidserver01 ~]# grep -i pass /var/log/mysqld.log 
2022-09-01T21:32:34.029030Z 6 [Note] [MY-0454] [Server] A temporary password is generated for root@localhost: S2<3!chrde

Once it is done you can reset your password as per your standards using /usr/bin/mysql_secure_installation.

ORA-16525: The Oracle Data Guard broker is not yet available in the dataguard environment

 Sometimes you might be facing an error ORA-16525: The Oracle Data Guard broker is not yet available in the dataguard environment. It might occur because of multiple reasons, but one of the main reasons, is if the particular server is having the issue on which real-time apply is being applied gets rebooted. Then the real-time apply might be running on a different node and goes in hung mode. You can follow the below steps to remediate it.

[oracle@sajidserver01 ~]$ dgmgrl

DGMGRL for Linux: Release 19.0.0.0.0 - Production on Thu Jul 21 12:07:36 2022
Version 19.15.0.0.0
Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.
Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys/<password>;
Connected as SYSDG.
DGMGRL> show configuration;
ORA-16525: The Oracle Data Guard broker is not yet available.
Configuration details cannot be determined by DGMGRL

###
Fix:
###

Bounce the standby database and enable the configuration in dgmgrl.

[oracle@sajidserver01 ~]$ srvctl stop database -d sajid_texas

[oracle@sajidserver01 ~]$ srvctl start database -d sajid_texas

[oracle@sajidserver01 ~]$ dgmgrl

DGMGRL for Linux: Release 19.0.0.0.0 - Production on Thu Jul 21 12:07:36 2022
Version 19.15.0.0.0
Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.
Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys/<password>;
Connected as SYSDG.

DGMGRL> enable configuration;
Enabled.

DGMGRL> show configuration;
Configuration - SAJID_CONF
  Protection Mode: MaxPerformance
  Members:
  sajid_pittsburgh     - Primary database
  sajid_texas - Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS   (status updated 27 seconds ago)

You will see the alert log will be updated with an attempt to start the background Managed Standby Recovery process. MRP0: Background Managed Standby Recovery process started and the Managed Standby Recovery started Real-Time Apply.

ORA-29548:Java system reported: release mismatch

 When you are running the Preupgrade task to upgrade the database from 12c to 19.14.0.0.0 version either using DBUA or the Auto upgrade feature. Sometimes you might face the below error depending on your system. If you happen to run into below error, you can follow the below steps. One of the reasons you may find a similar issue is if an OJVM patch is applied or rolled back.

Error:

Contact Oracle Support for instructions on how to resolve this error. 

Error: ORA-29548 ORA-29548:Java system reported: release mismatch, 

12.2.0.1.180717 1.8 in database (classes.bin) vs 12.2.0.1.0 1.8 in executable

SQL> select dbms_java.lonname('TEST') from dual;

select dbms_java.lonname('TEST') from dual

                        *

ORA-29548: Java system class reported: joxcsys: release mismatch, 

12.2.0.1.180717 1.8 in database (classes.bin) vs 12.2.0.1.0 1.8 in executable

SQL> conn /as sysdba

Connected.

SQL>  @?/javavm/install/update_javavm_db.sql

SQL> SET FEEDBACK 1

SQL> SET NUMWIDTH 10

SQL> SET LINESIZE 80

SQL> SET TRIMSPOOL ON

SQL> SET TAB OFF

SQL> SET PAGESIZE 100

SQL> alter session set "_ORACLE_SCRIPT"=true;

Session altered.

SQL> -- If Java is installed, do CJS.

SQL> -- If CJS can deal with the SROs inconsistent with the new JDK,

SQL> -- the drop_sros() call here can be removed.

SQL> call initjvmaux.drop_sros();

Call completed.

SQL> create or replace java system;

Java created.

SQL> update dependency$

  set p_timestamp=(select stime from obj$ where obj#=p_obj#)

  where (select stime from obj$ where obj#=p_obj#)!=p_timestamp and

  (select type# from obj$ where obj#=p_obj#)=29  and

  (select owner# from obj$ where obj#=p_obj#)=0;

161 rows updated.

SQL> commit;

SQL> alter session set "_ORACLE_SCRIPT"=false;

Session altered.

SQL> select dbms_java.longname('TEST') from dual;

DBMS_JAVA.LONGNAME('TEST')

-------------------------------------------------------------

TEST

1 row selected.

Once the above steps are followed and recompile the invalid objects. Now if you run Preupgrade script, the above error will not pop up. 

Happy Oracle Database Upgrades!!!