Upgrading Enterprise Manager 13.4 to 13.5 - The Silent Way

Just a short description of what I did to upgrade an existing Enterprise Manager 13.4 (RU9) to the latest release 13.5 ...

Environment:

OCI machine, Oracle Linux 7.9, Enterprise Manager 13.4 RU9
OCI machine, Oracle Linux 7.9, Repository Database Release 19.3

Software-Download

EM 13.5 - downloaded from edelivery.oracle.com

Preparation

Creating the new em13500 base directory, the MiddlewareHome and the Instance Home. 

mkdir $ORACLE_BASE/product/em13500
mkdir $ORACLE_BASE/product/em13500/em13500
mkdir $ORACLE_BASE/product/em13500/gc_inst

I prefer to have one separate EM Release Base Directory for each release, instead of creating everything in $ORACLE_BASE/product.

Extracting the response files from the installation media

./em13500_linux64.bin -getResponseFileTemplates -outputLoc /u01/software/em_install

Preparing some init parameters in the repository database (restart after)

alter system reset "_optimizer_nlj_hj_adaptive_join" scope=both sid='*'; 
alter system reset "_optimizer_strans_adaptive_pruning" scope=both sid='*';
alter system reset "_px_adaptive_dist_method" scope=both sid='*';
alter system reset "_sql_plan_directive_mgmt_control" scope=both sid='*';
alter system reset "_optimizer_dsdir_usage_control" scope=both sid='*';
alter system reset "_optimizer_use_feedback" scope=both sid='*';
alter system reset "_optimizer_gather_feedback" scope=both sid='*';
alter system reset "_optimizer_performance_feedback" scope=both sid='*';

Adopting upgrade.rsp to the environment

RESPONSEFILE_VERSION=2.2.1.0.0
UNIX_GROUP_NAME="dba"
INVENTORY_LOCATION=/u01/app/oraInventory
INSTALL_UPDATES_SELECTION=skip
ORACLE_MIDDLEWARE_HOME_LOCATION=/u01/app/oracle/product/em13500/em13500
ORACLE_INSTANCE_HOME_LOCATION=/u01/app/oracle/product/em13500/gc_inst
OLD_BASE_DIR=/u01/app/oracle/product/em13400/em13400
ORACLE_HOSTNAME=oem1.demo.ch
ONE_SYSTEM=true
WLS_ADMIN_SERVER_USERNAME=weblogic
WLS_ADMIN_SERVER_PASSWORD=manager
WLS_ADMIN_SERVER_CONFIRM_PASSWORD=manager
NODE_MANAGER_PASSWORD=manager
NODE_MANAGER_CONFIRM_PASSWORD=manager
WLS_ADMIN_SERVER_PASSWORD=manager
DATABASE_HOSTNAME=oem2.demo.ch
LISTENER_PORT=1521
SERVICENAME_OR_SID=poem1.demo.ch
SYS_PASSWORD=manager
SYSMAN_PASSWORD=manager
EMPREREQ_AUTO_CORRECTION=false
REPOSITORY_BACKUP_DONE=true
PLUGIN_SELECTION={}
b_upgrade=true
EM_INSTALL_TYPE=NOSEED

Stopping the central agent

emctl stop agent

Stopping Enterprise Manager - and only the Enterprise Manager. You'll need a running AdminServer (an an open repository database)

emctl stop oms

Starting the installer

em13500_linux64.bin -silent -responseFile /u01/software/em_install/upgrade.rsp

Waiting, waiting, waiting ... about 30 minutes in my environment. But this can take much longer in real life environments.

To be honest: Because of the missing upgrade instructions, it took two tries to install. In both cases, the installer created a really helpful log file in html format, and left the original installation unchanged.

Finally

Important

An Enterprise Manager Upgrade is - as always - an out-of-place upgrade. Keep in mind that any connection descriptions to LDAP as well as Your certificates are not (!) upgraded / moved to the new installation. So, if You're using LDAP for authentication or having Your own certificates in place: Save the information / the certificate and apply it after the upgrade to the new EM.

Read More

Enterprise Manager 13.3 - After Upgrade to DB Plugin 13.3.2: Missing 'Top Activities' Page and Empty Graphs when using 'Performance Hub'

oops - long title ... sorry for that

Starting with DB Plugin 13.3.2.0, oracle has kicked out the flash plugin and replaced that with JET (JavaScript Extension Toolkit). As a consequence, 'Top Activity' page is not available anymore and is replaced by 'Performance Hub' respectively 'Ash Analytics' and 'SQL Monitoring'. 

That's the good news ... the not that good news is: Both pages will show empty graphs in case the monitored database is of version 12.2 (it works correctly with 18c and 19c)

Solution for that can be found in MOS Note "Usage Of ASH Analytics In EM 13.3 PG Release (Doc ID 2580773.1)". Bad news two (,too ;-)): You have to execute the necessary steps in EVERY pluggable database.

Read More

Enterprise Manager - Advanced Backup Strategies

Oracle's recommendation for an Enterprise Manager backup can be found in "Enterprise Manager Cloud Control Advanced Installation and Configuration Guide", chapter "Backing Up and Recovering Enterprise Manager".

Basically, the recommendation consists of (only) three steps:

·      Software Library Backup

·      Management Repository Backup

·      Oracle Management Service Backup

·      Exportconfig

·      Software Homes

But we have a lot more important stuff within Enterprise Manager which is worth to be backed up, haven't we? Monitoring Templates for example, Metric Extensions or properties which were changed (either possibly or probably), … But, where to get this information from and how to extract it from Enterprise Manager's guts? Repository, emctl and emcli are the keywords – here's how to …:

Software Library Backup

… of course worth a frequent backup. IMHO the best way to get the latest information about the current path is to query the repository database:

select
  base_path
from EM_SWLIB_OMS_LOCATIONS_E
where name ='default_loc';

Put the result in a variable and issue something like this:

tar cvfz software_library.tar $<variableName>/*

Management Repository Backup

… as a DBA You know how to backup an oracle database. I'm sure …

Oracle Management Service Backup

… as oracle's manual states, use

emctl exportconfig oms

And the additional stuff, I was talking about?

OMS Properties

Backup the OMS properties – even if You've never changed them – by issuing an 

emctl list properties

Redirect the result to a file having a timestamp or a sequential number in its filename and You're save.

Monitoring Templates

To backup Monitoring Templates use this SQL, spool the output to a file and execute the file:

select
'emcli export_template -name='''||template_name||''' -target_type='||target_type||' -output_file=$BACKUP_DIR/MT_'||template_name||'.xml'
from sysman.mgmt_templates
where is_public = 0
order by template_name ;

Metric Extensions

Developing Metric Extensions is a time consuming task – so backup the MEs frequently. It is that easy (spool and execute the spool file):

select
'emcli export_metric_extension -file_name=''$BACKUP_DIR/'||name||'_V'||version||'.xml'' -target_type='||target_type||' -name='''||name||''' -version='||version
from SYSMAN.EM_MEXT_VERSIONS_E
order by name, version;

Incident Rulesets

Your Incident Rulesets are candidate to be backed up, too (spool and execute):

select
'emcli export_incident_rule_set -rule_set_name='||ruleset_name||' -rule_set_owner='||owner||' -export_file="$BACKUP_DIR"'
from EM_RULE_SETS
where owner not in ('SYSMAN', '<SYSTEM>')
order by owner, ruleset_name;

Information Publisher Reports

Did You write Information Publisher Reports? Backup them (redirect to a file and execute)!:

emcli get_reports | sed "s/, / /g" | awk 'BEGIN{ FS=","}{x=$1; gsub(/ |-|=|:|\/|\(|\)/, "",x); print "emcli export_report -title="$1" -owner="$2" -output_file=${BACKUP_DIR}/REP_"x".xml" }'

Interested in more possibilities?

emcli offers some more export commands – "emcli help | grep -I export" extracts them from emcli's help text. Perhaps You'll find the one You're searching for.

Questions?

Get in contact ...

Read More

Upgrading Enterprise Manager 13.2 to 13.3 - missing 'Update' Option in GUI

Tried to upgrade an Enterprise Manager Cloud Control 13.2 to the next level of happiness - 13.3 - like I did that for several installations. Unfortunately, the 'Upgrading an existing Enterprise Manager System' option in the 'Installation Types' section of the installation GUI is not available. Usually, this behaviour points to an unavailable or corrupt Oracle Inventory. As additional possibility, it could be a bug as well:
Check MOS Note "EM 13c: Checklist for Upgrading Enterprise Manager Cloud Control from Version 12.1.0.5/13.1/13.2 to 13.3 (Doc ID 2418352.1)", the 'Known Issues' section: "Bug 28350114 : EM 13.3 Upgrade, Installation Types Screen Is Not Showing the Upgrade Option"

Read More

bind csid (#) does not match session csid (#)

Problem

A database's alert log is full of messages like

Wed Jun 28 09:13:09 2017

Errors in file /cloudfs/adr/diag/rdbms/scs12fi/SCS12FI/trace/SCS12FI_ora_16694.trc:

Wed Jun 28 09:13:10 2017

Errors in file /cloudfs/adr/diag/rdbms/scs12fi/SCS12FI/trace/SCS12FI_ora_16694.trc:

Wed Jun 28 09:13:13 2017

Errors in file /cloudfs/adr/diag/rdbms/scs12fi/SCS12FI/trace/SCS12FI_ora_16694.trc:

Wed Jun 28 09:16:25 2017

Errors in file /cloudfs/adr/diag/rdbms/scs12fi/SCS12FI/trace/SCS12FI_ora_16694.trc:

The trace file shows

psdgbt: bind csid (1) does not match session csid (31)

psdgbt: session charset is WE8ISO8859P1

Reason

MOS Note "After restart of database suddenly PLS-00553: character set name is not recognized are seen in the alert.log (Doc ID 1599864.1)" is not exactly a description of what has happend, but points in the correct direction.

Reason was - in my case - that an agent which has connected to a freshly created clone of a production database a little to early was stuck with an Character Set which was current at the time of the first connect. After the database has been fully started, the correct charset is used - which now does not fit anymore to the session charset used by the agent connect.

Solution

in my case: simply restart the agent ...

Read More

EM 13c: TNS-01190 errors when listener owner and agent owner are not the same account

Problem:

When monitoring listener's logfile, lots of TNS-01190 are found:

<msg time='2017-06-26T10:21:47.608+02:00' org_id='oracle' comp_id='tnslsnr'

 type='UNKNOWN' level='16' host_id='edosrvoda03'

 host_addr='192.168.15.70'>

 <txt>26-JUN-2017 10:21:47 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=edosrvoda03)(USER=oracle))(COMMAND=trc_directory)(ARGUMENTS=2)(SERVICE=LISTENER)(VERSION=202375680)) * trc_directory * 1190

 </txt>

</msg>

<msg time='2017-06-26T10:21:47.608+02:00' org_id='oracle' comp_id='tnslsnr'

 type='UNKNOWN' level='16' host_id='edosrvoda03'

 host_addr='192.168.15.70'>

 <txt>TNS-01190: The user is not authorized to execute the requested listener command

 </txt>

</msg>

The owners of the agent process is different from the listener process owner. Example: agent is installed and run under user account 'oracle' - listener is started by user 'grid'. 

Reproducability: Every time, Enterprise Manager tries to discover or rediscover new targets on a system, these TNS-01190 errors will occur in listener's logfile. So, try a rediscover on a certain host - after that, You'll find TNS-01190 errors in the listener log.

'Solution'

MOS Note "How To Disable Oracle Database Listener Alerts TNS-01190 In Enterprise Manager Cloud Control to avoid the error "trc_directory (TNS-1190), log_directory (TNS-1190),. Please check log for details." (Doc ID 1399060.1)" recommends two possible solutions:

1. set a listener password and include that password in 'Monitoring Configuration' of the listener target.
   This recommendation does not work for listeners of version 12.1 and higher, because the password feature is no longer supported.
2. edit the metric and remove the 1190 string from the list of values.
   Remains the only possibility ... change Your monitoring template and/or target setting, edit TNS Errors / TNSMsg and replace 'TNS-[ ]*0*(1169|1189|12508|1190)' with 'TNS-[ ]*0*(1169|1189|12508)' (remove the '|1190') or - and that's imho the best way - do not monitor TNSMsgs ...

Read More