September 3, 2019

Enterprise Manager 13.3 - After Upgrade to DB Plugin 13.3.2: Missing 'Top Activities' Page and Empty Graphs when using 'Performance Hub'

oops - long title ... sorry for that

Starting with DB Plugin, oracle has kicked out the flash plugin and replaced that with JET (JavaScript Extension Toolkit). As a consequence, 'Top Activity' page is not available anymore and is replaced by 'Performance Hub' respectively 'Ash Analytics' and 'SQL Monitoring'. 

That's the good news ... the not that good news is: Both pages will show empty graphs in case the monitored database is of version 12.2 (it works correctly with 18c and 19c)

Solution for that can be found in MOS Note "Usage Of ASH Analytics In EM 13.3 PG Release (Doc ID 2580773.1)". Bad news two (,too ;-)): You have to execute the necessary steps in EVERY pluggable database.


June 28, 2019

Solution: SQL Developer does not show its Startscreen after starting successfully


After starting SQL Developer 18.4 on a Windows 10, the SQL Developer tile is shown in the taskbar, but the Startscreen is not shown on the desktop.

"Quick and Dirty" Solution

  • Stop SQL Developer. 
  • Open Windows Explorer and type "%APPDATA%\SQL Developer" in the Directory/URL bar. You'll see a directory named according the SQL Developer release (for example, something like "system18.4.0.376.1900"). 
  • Rename this directory.
  • Start SQL Developer - the directory is re-created, everything should work fine again. If SQL Developer comes up correctly, delete the renamed directory from "%APPDATA%\SQL Developer"
  • get back to work ... ;-)


June 21, 2019

Enterprise Manager - Advanced Backup Strategies

Oracle's recommendation for an Enterprise Manager backup can be found in "Enterprise Manager Cloud Control Advanced Installation and Configuration Guide", chapter "Backing Up and Recovering Enterprise Manager".
Basically, the recommendation consists of (only) three steps:
·      Software Library Backup
·      Management Repository Backup
·      Oracle Management Service Backup
·      Exportconfig
·      Software Homes
But we have a lot more important stuff within Enterprise Manager which is worth to be backed up, haven't we? Monitoring Templates for example, Metric Extensions or properties which were changed (either possibly or probably), … But, where to get this information from and how to extract it from Enterprise Manager's guts? Repository, emctl and emcli are the keywords – here's how to …:

Software Library Backup

… of course worth a frequent backup. IMHO the best way to get the latest information about the current path is to query the repository database:
where name ='default_loc';
Put the result in a variable and issue something like this:
tar cvfz software_library.tar $<variableName>/*

Management Repository Backup

… as a DBA You know how to backup an oracle database. I'm sure …

Oracle Management Service Backup

… as oracle's manual states, use
emctl exportconfig oms

And the additional stuff, I was talking about?

OMS Properties

Backup the OMS properties – even if You've never changed them – by issuing an 
emctl list properties
Redirect the result to a file having a timestamp or a sequential number in its filename and You're save.

Monitoring Templates

To backup Monitoring Templates use this SQL, spool the output to a file and execute the file:
'emcli export_template -name='''||template_name||''' -target_type='||target_type||' -output_file=$BACKUP_DIR/MT_'||template_name||'.xml'
from sysman.mgmt_templates
where is_public = 0
order by template_name ;

Metric Extensions

Developing Metric Extensions is a time consuming task – so backup the MEs frequently. It is that easy (spool and execute the spool file):
'emcli export_metric_extension -file_name=''$BACKUP_DIR/'||name||'_V'||version||'.xml'' -target_type='||target_type||' -name='''||name||''' -version='||version
order by name, version;

Incident Rulesets

Your Incident Rulesets are candidate to be backed up, too (spool and execute):
'emcli export_incident_rule_set -rule_set_name='||ruleset_name||' -rule_set_owner='||owner||' -export_file="$BACKUP_DIR"'
where owner not in ('SYSMAN', '<SYSTEM>')
order by owner, ruleset_name;

Information Publisher Reports

Did You write Information Publisher Reports? Backup them (redirect to a file and execute)!:
emcli get_reports | sed "s/, / /g" | awk 'BEGIN{ FS=","}{x=$1; gsub(/ |-|=|:|\/|\(|\)/, "",x); print "emcli export_report -title="$1" -owner="$2" -output_file=${BACKUP_DIR}/REP_"x".xml" }'

Interested in more possibilities?

emcli offers some more export commands – "emcli help | grep -I export" extracts them from emcli's help text. Perhaps You'll find the one You're searching for.


Get in contact ...


April 23, 2019

Follow up: Upgrading Enterprise Manager 13.2 to 13.3 - missing 'Update' Option in GUI

First of all, thanks a lot to Martin Decker for his comment and the description how he could solve that problem. Tried that, but unfortunately, these changes alone did not help. I opened a SR and got a recommendation how to solve the problem, exactly designed for my environment. Beside the tasks Martin described, I had to edit some libraries in the directory tree <OMS_HOME>/inventory/featuresets/. 

I asked Oracle Support if I'm allowed to publish the solution, but they said:

"We are still waiting for bug fix for a permanent solution and the workaround may or may not suitable for all situations where upgrade option not available. There are chances like inventory details missing also can cause the similar issue hence its better to log a new request if there are any concerns to verify the install logs/screens and provide solution/workaround"

So, in case You face the 'missing update option, too, please log a Service Request using the keywords "Bug 28350114 : EM 13.3 Upgrade, Installation Types Screen Is Not Showing the Upgrade Option" and supply the install logs.


April 17, 2019

Upgrading Enterprise Manager 13.2 to 13.3 - missing 'Update' Option in GUI

Tried to upgrade an Enterprise Manager Cloud Control 13.2 to the next level of happiness - 13.3 - like I did that for several installations. Unfortunately, the 'Upgrading an existing Enterprise Manager System' option in the 'Installation Types' section of the installation GUI is not available. Usually, this behaviour points to an unavailable or corrupt Oracle Inventory. As additional possibility, it could be a bug as well:
Check MOS Note "EM 13c: Checklist for Upgrading Enterprise Manager Cloud Control from Version to 13.3 (Doc ID 2418352.1)", the 'Known Issues' section: "Bug 28350114 : EM 13.3 Upgrade, Installation Types Screen Is Not Showing the Upgrade Option"

Copy MOS-Note URL's

Perhaps You've had that problem, too ... Something has changed with My Oracle Support URLs. Simply copying the url of an already opened MOS document and sending it to a collegue or pasting it into a doc will not give the expected result, i.e., forwarding to the expected document. But how to get the URL? Can be simply as that: Right click on the document's link, 'Copy link address' and You're done ... *wacko*

January 14, 2019

Enterprise Manager 13c - 'Tablespace Space Used %' metric sends false alarms

Situation is ...

Metric 'Tablespace Space Used %' is used to monitor the free space of the tablespaces. Sometimes You'll get an alarm that the Tablespace has reached it's critical value. But, the tablespace's data files are auto extensible and far away from the max size and the files are located on an ACFS which knows also auto-extensibility - or better said - an auto-resize feature. There is also plenty of space left on the underlying ASM. Nevertheless, FS free space is currently near to 100% and the current free space in the tablespace is lower than the critical value. Obviously a false alarm, isn't it?

Cause ...

for that behaviour is, that Oracle has changed the method to calculate a tablespace's free space for metric 'Tablespace Space Used %' with release 13c. New is that the metric takes the available disk space into account for its calculations. And this is correct , because no more space left on the disk means finally that a tablespace cannot auto-extend anymore. But with ACFS and the auto resize functionality, this shouldn't be a problem at all - ACFS will auto resize the FS if necessary.

Unfortunately, Oracle Support has no fix for this false alarm at this very moment. MOS says that view DBA_TABLESPACE_USAGE_METRICS has a problem to calculate the metric's base information correctly. Well, that might be a possible reason, but IMHO, the values of my particular database / tablespaces / datafiles are fine.

Solution / Workaround

Simply set the value of the auto resize parameter for the ACFS filesystem in question to an appropriate value other than the default of 20G, to avoid the false alarm problem (until the ASM runs full and ACFS cannot auto resize):
"acfsutil size -a <Size>G <FSName>"

acfsutil size -a 100G /u02/oradata/TESTDB
acfsutil size: ACFS-03642: successfully updated auto-resize settings


December 21, 2018

ODA Server Patch 18.3 available - and installed ...

Since today, Oracle Database Appliance software release is available ... I took the chance to download and install the new release - this is the story:

Initial Situation:

I tested the update on an ODA X7-2M with system version, running five databases (12.1). There are three vlans configured on that system.

Documentation used:

Installation Instruction was taken from, Oracle Database Appliance, Release 18.3, X7-2 Deployment and User's Guide for Linux x86-64, Chapter 7 Patching Oracle Database Appliance, subchapter 'Patching Oracle Database Appliance Using the CLI'


- You have to download about 15GB - just for the system update, DB 18.x is around 4GB
- do not copy the update software to a NFS share and do not run the update from this directory
- if You are using NFS shares, be sure that a df -h comes back immediately. With my testsystem I had two NFS Filesystems which were not reachable at all, resulting in a 'never-come-back' df -h - this, in turn, resulted in a stuck Server Patching at step 'Configuring GI'. After I umounted those FS (by using the -f switch), the patching continued
- qosmserver has to run and a crs resource has to be configured
- the patch documentation is missing the hint to first update dcsagent (odacli upgrade-dcsagent -v, but it's a prerequisite
- remove any 'non-default' entries from oracle's .bash_profile prior to the update
- Step 'Setting AUDIT SYSLOG LEVEL' had a status 'Failure' at the end of the update process - overall status of the update was - nevertheless - successful (will check the reason for that failure later)
- Installation documentation says after step 'apply the server update' to check if the update was successful and then to issue an 'odacli update-storage'. ATTENTION! After a successful update, the server will reboot! Wait until the system is up again before You start the update-storage command!
- Server reboot took about 10 Minutes with my X7-2M
- Documentation example for 'Update the storage components' is not correct: use '--rolling' or '-r' instead of '-rolling'
- update-storage using the '--rolling' option is not supported on ODAlite

That's it

The whole process took slightly more than one hour - and remember: the system reboots after an successful 'odacli update-server' (!)


December 7, 2018

ODA - Update to - issue with

Yesterday, I had to update an ODA from Release to . All prerequisite checks including prepatchreport were successful ... but the update itself not!

During the update, at stage 'GI Home Cloning', the update process failed and left the system in an 'unknown' state. A detailled analysis has shown the root cause: crs service qosmserver is missing. So, I tried to setup this missing link to a successful update:

srvctl add qosmserver
srvctl status qosmserver
srvctl start qosmserver

But, qosmserver could not be started - resource '' is missing. Well, that's absolutely true - we do not have such a resource defined for our ODAs. Finally changed the first of our vlan definitions to and could start qosmserver.

Unfortunately and because of the interrupted update, GI is now using its 12.2 home, but odacli describe-component shows, that the current home is the old 12.1 - that finally results in a situation where I cannot run update-server again (GI Home Cloning fails, because the new GI Home already exists)

Outcome: As additional prerequisite, check, wether or not the resource exists on Your ODA and be sure qosmserver is up and running BEFORE starting an 'odacli update-server'.


November 7, 2018

DBID Problem @ ODA - or Count von Count (Sesame Street) calculates DBIDs

I did some more investigations to get an explanation for that problem with the extra long DBIDs and the resulting problems when creating new databases on an ODA ...

And here's the simple result:

Remember? In our environment, oracle's .bash_profile runs an additional script, which prints all - at this very moment existing - oracle databases, listeners, their status and the according ORACLE_HOMEs to STDOUT. Output looks like this:

# Following Databases/Aliases are on the ODA oda-o1234

+ASM1               up      /u01/app/
TEST1               up      /u01/app/oracle/product/
TEST2c              up      /u01/app/oracle/product/
TEST2D              up      /u01/app/oracle/product/
TEST2E              up      /u01/app/oracle/product/

LISTENER            up      /u01/app/

Now, take all numbers from this output, put them in one line, add the 'real' DBID from the database - and You'll have the dbid which is displayed (and used) by odacli in the internal derby database. :-)

The easy test:

Add two 'echo' lines to oracle's .bash_profile that it looks like this:

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc

# User specific environment and startup programs


export PATH
umask 022

echo "1111111111111111111111111111111111111111111111111"
echo "2222222222222222222222222222222222222222222222222"

Create a database using the 'odacli create-database' command:

odacli create-database -m -dh 665b9ce4-3a13-4a04-8ccd-548e66556def -n TEST2F -r ACFS

When the database is created, issue an 'odacli list-databases' and, subsequently, an 'odacli describe-database -i <id of that database>' - check the DBID (concatenated echo "1"s plus the echo "2"s plus the 'real' DBID):

Database details
                     ID: 2a5426bc-8db9-4d07-ba87-b54f06dbbcc3
            Description: TEST2F
                DB Name: TEST2F
             DB Version:
                DB Type: Si
             DB Edition: EE
                   DBID: 111111111111111111111111111111111111111111111111122222222222222222222222222222222222222222222222222498916505
Instance Only Database: false
                    CDB: false
               PDB Name:
    PDB Admin User Name:
                  Class: OLTP
                  Shape: Odb1
                Storage: ACFS
           CharacterSet: AL32UTF8
  National CharacterSet: AL16UTF16
               Language: AMERICAN
              Territory: AMERICA
                Home ID: 665b9ce4-3a13-4a04-8ccd-548e66556def
        Console Enabled: false
     Level 0 Backup Day: Sunday
    AutoBackup Disabled: false
                Created: November 7, 2018 12:58:45 PM CET
         DB Domain Name: world

The database's DBID - which is the last portion in odacli's DBID:

SQL> select dbid from v$database;


Imho, the answer for question 'Bug or Feature?' is quite clear ;-) And, by the way, Oracle Support is still investigating the problem.


Copyright © Robert Crames' Oracle Database and Middleware Blog | Powered by Blogger
Design by SimpleWpThemes | Blogger Theme by