Mac? X11? Oracle Cloud Infrastructure? - How To Get runInstaller To run ...

Summary

Long story told short: Got two OCI machines and tried to install some Oracle stuff - Enterprise Manager, Database Software, Weblogic Server and such 'things'. Problem: How to install that without having a possibility to run a GUI? Well - I'm used to install oracle products using the 'silent' way. Nevertheless - How the heck can I get a GUI to run on a Mac? Here's the How To.

Environment

MacBook Pro (Mid 2014 - yes it's old, but I'm still waiting for the all new MacBook M1X, latest generation, 32GB RAM at least. Possibly, tomorrow I know more :-))

OCI - two machines running Linux, 32GB RAM, nothing special

To Do's

• Get XQUARTZ from the manufacturers website and install it on Your Mac

• connect to Your OCI machine(s) as user opc and login as root (sudo su -)

• check / change / edit '/etc/ssh/sshd_config:
  X11Forwarding yes
  X11UseLocalhost no

• restart sshd
  systemctl restart sshd

• install two more packages - xterm (nice to have to check x11) and xauth. Simply run the two statements. In case xterm and/or xauth are installed already, nothing happens.
  yum -y install xterm
  yum -y install xauth

Let's try ...

• start XQUARTZ

• connect as opc to Your OCI machine (doesn't matter if You use mac's terminal.app or an XQUARTZ window:
  ssh -CX opc@<OCI machine>

• In case You're connecting using a ssh key:
  ssh -CX -i <Path to the public key> opc@<OCI machine>

• run these two commands (You'll need the output right after the login as oracle):
  echo "xauth add $(xauth list | grep `echo $DISPLAY | awk '{ print substr($0,index($0,":"),3) }'` | grep `hostname`)"
  echo export DISPLAY=$DISPLAY

• connect as user oracle
  sudo su - oracle

• execute the result of the two commands (xauth and echo) - which is something like this:
  xauth add <oci machine hostname, full qualified>:<Display #>  MIT-MAGIC-COOKIE-1  <some kind of key>
  export DISPLAY=<IP>:<Display #>

• run Your GUI stuff :-)

Read More

Upgrading Enterprise Manager 13.4 to 13.5 - The Silent Way

Just a short description of what I did to upgrade an existing Enterprise Manager 13.4 (RU9) to the latest release 13.5 ...

Environment:

OCI machine, Oracle Linux 7.9, Enterprise Manager 13.4 RU9
OCI machine, Oracle Linux 7.9, Repository Database Release 19.3

Software-Download

EM 13.5 - downloaded from edelivery.oracle.com

Preparation

Creating the new em13500 base directory, the MiddlewareHome and the Instance Home. 

mkdir $ORACLE_BASE/product/em13500
mkdir $ORACLE_BASE/product/em13500/em13500
mkdir $ORACLE_BASE/product/em13500/gc_inst

I prefer to have one separate EM Release Base Directory for each release, instead of creating everything in $ORACLE_BASE/product.

Extracting the response files from the installation media

./em13500_linux64.bin -getResponseFileTemplates -outputLoc /u01/software/em_install

Preparing some init parameters in the repository database (restart after)

alter system reset "_optimizer_nlj_hj_adaptive_join" scope=both sid='*'; 
alter system reset "_optimizer_strans_adaptive_pruning" scope=both sid='*';
alter system reset "_px_adaptive_dist_method" scope=both sid='*';
alter system reset "_sql_plan_directive_mgmt_control" scope=both sid='*';
alter system reset "_optimizer_dsdir_usage_control" scope=both sid='*';
alter system reset "_optimizer_use_feedback" scope=both sid='*';
alter system reset "_optimizer_gather_feedback" scope=both sid='*';
alter system reset "_optimizer_performance_feedback" scope=both sid='*';

Adopting upgrade.rsp to the environment

RESPONSEFILE_VERSION=2.2.1.0.0
UNIX_GROUP_NAME="dba"
INVENTORY_LOCATION=/u01/app/oraInventory
INSTALL_UPDATES_SELECTION=skip
ORACLE_MIDDLEWARE_HOME_LOCATION=/u01/app/oracle/product/em13500/em13500
ORACLE_INSTANCE_HOME_LOCATION=/u01/app/oracle/product/em13500/gc_inst
OLD_BASE_DIR=/u01/app/oracle/product/em13400/em13400
ORACLE_HOSTNAME=oem1.demo.ch
ONE_SYSTEM=true
WLS_ADMIN_SERVER_USERNAME=weblogic
WLS_ADMIN_SERVER_PASSWORD=manager
WLS_ADMIN_SERVER_CONFIRM_PASSWORD=manager
NODE_MANAGER_PASSWORD=manager
NODE_MANAGER_CONFIRM_PASSWORD=manager
WLS_ADMIN_SERVER_PASSWORD=manager
DATABASE_HOSTNAME=oem2.demo.ch
LISTENER_PORT=1521
SERVICENAME_OR_SID=poem1.demo.ch
SYS_PASSWORD=manager
SYSMAN_PASSWORD=manager
EMPREREQ_AUTO_CORRECTION=false
REPOSITORY_BACKUP_DONE=true
PLUGIN_SELECTION={}
b_upgrade=true
EM_INSTALL_TYPE=NOSEED

Stopping the central agent

emctl stop agent

Stopping Enterprise Manager - and only the Enterprise Manager. You'll need a running AdminServer (an an open repository database)

emctl stop oms

Starting the installer

em13500_linux64.bin -silent -responseFile /u01/software/em_install/upgrade.rsp

Waiting, waiting, waiting ... about 30 minutes in my environment. But this can take much longer in real life environments.

To be honest: Because of the missing upgrade instructions, it took two tries to install. In both cases, the installer created a really helpful log file in html format, and left the original installation unchanged.

Finally

Important

An Enterprise Manager Upgrade is - as always - an out-of-place upgrade. Keep in mind that any connection descriptions to LDAP as well as Your certificates are not (!) upgraded / moved to the new installation. So, if You're using LDAP for authentication or having Your own certificates in place: Save the information / the certificate and apply it after the upgrade to the new EM.

Read More

Enterprise Manager 13.3 - After Upgrade to DB Plugin 13.3.2: Missing 'Top Activities' Page and Empty Graphs when using 'Performance Hub'

oops - long title ... sorry for that

Starting with DB Plugin 13.3.2.0, oracle has kicked out the flash plugin and replaced that with JET (JavaScript Extension Toolkit). As a consequence, 'Top Activity' page is not available anymore and is replaced by 'Performance Hub' respectively 'Ash Analytics' and 'SQL Monitoring'. 

That's the good news ... the not that good news is: Both pages will show empty graphs in case the monitored database is of version 12.2 (it works correctly with 18c and 19c)

Solution for that can be found in MOS Note "Usage Of ASH Analytics In EM 13.3 PG Release (Doc ID 2580773.1)". Bad news two (,too ;-)): You have to execute the necessary steps in EVERY pluggable database.

Read More

Solution: SQL Developer does not show its Startscreen after starting successfully

Problem: 

After starting SQL Developer 18.4 on a Windows 10, the SQL Developer tile is shown in the taskbar, but the Startscreen is not shown on the desktop.

"Quick and Dirty" Solution

• Stop SQL Developer. 
• Open Windows Explorer and type "%APPDATA%\SQL Developer" in the Directory/URL bar. You'll see a directory named according the SQL Developer release (for example, something like "system18.4.0.376.1900"). 
• Rename this directory.
• Start SQL Developer - the directory is re-created, everything should work fine again. If SQL Developer comes up correctly, delete the renamed directory from "%APPDATA%\SQL Developer"
• get back to work ... ;-)

Read More

Enterprise Manager - Advanced Backup Strategies

Oracle's recommendation for an Enterprise Manager backup can be found in "Enterprise Manager Cloud Control Advanced Installation and Configuration Guide", chapter "Backing Up and Recovering Enterprise Manager".

Basically, the recommendation consists of (only) three steps:

·      Software Library Backup

·      Management Repository Backup

·      Oracle Management Service Backup

·      Exportconfig

·      Software Homes

But we have a lot more important stuff within Enterprise Manager which is worth to be backed up, haven't we? Monitoring Templates for example, Metric Extensions or properties which were changed (either possibly or probably), … But, where to get this information from and how to extract it from Enterprise Manager's guts? Repository, emctl and emcli are the keywords – here's how to …:

Software Library Backup

… of course worth a frequent backup. IMHO the best way to get the latest information about the current path is to query the repository database:

select
  base_path
from EM_SWLIB_OMS_LOCATIONS_E
where name ='default_loc';

Put the result in a variable and issue something like this:

tar cvfz software_library.tar $<variableName>/*

Management Repository Backup

… as a DBA You know how to backup an oracle database. I'm sure …

Oracle Management Service Backup

… as oracle's manual states, use

emctl exportconfig oms

And the additional stuff, I was talking about?

OMS Properties

Backup the OMS properties – even if You've never changed them – by issuing an 

emctl list properties

Redirect the result to a file having a timestamp or a sequential number in its filename and You're save.

Monitoring Templates

To backup Monitoring Templates use this SQL, spool the output to a file and execute the file:

select
'emcli export_template -name='''||template_name||''' -target_type='||target_type||' -output_file=$BACKUP_DIR/MT_'||template_name||'.xml'
from sysman.mgmt_templates
where is_public = 0
order by template_name ;

Metric Extensions

Developing Metric Extensions is a time consuming task – so backup the MEs frequently. It is that easy (spool and execute the spool file):

select
'emcli export_metric_extension -file_name=''$BACKUP_DIR/'||name||'_V'||version||'.xml'' -target_type='||target_type||' -name='''||name||''' -version='||version
from SYSMAN.EM_MEXT_VERSIONS_E
order by name, version;

Incident Rulesets

Your Incident Rulesets are candidate to be backed up, too (spool and execute):

select
'emcli export_incident_rule_set -rule_set_name='||ruleset_name||' -rule_set_owner='||owner||' -export_file="$BACKUP_DIR"'
from EM_RULE_SETS
where owner not in ('SYSMAN', '<SYSTEM>')
order by owner, ruleset_name;

Information Publisher Reports

Did You write Information Publisher Reports? Backup them (redirect to a file and execute)!:

emcli get_reports | sed "s/, / /g" | awk 'BEGIN{ FS=","}{x=$1; gsub(/ |-|=|:|\/|\(|\)/, "",x); print "emcli export_report -title="$1" -owner="$2" -output_file=${BACKUP_DIR}/REP_"x".xml" }'

Interested in more possibilities?

emcli offers some more export commands – "emcli help | grep -I export" extracts them from emcli's help text. Perhaps You'll find the one You're searching for.

Questions?

Get in contact ...

Read More

Follow up: Upgrading Enterprise Manager 13.2 to 13.3 - missing 'Update' Option in GUI

First of all, thanks a lot to Martin Decker for his comment and the description how he could solve that problem. Tried that, but unfortunately, these changes alone did not help. I opened a SR and got a recommendation how to solve the problem, exactly designed for my environment. Beside the tasks Martin described, I had to edit some libraries in the directory tree <OMS_HOME>/inventory/featuresets/. 

I asked Oracle Support if I'm allowed to publish the solution, but they said:

"We are still waiting for bug fix for a permanent solution and the workaround may or may not suitable for all situations where upgrade option not available. There are chances like inventory details missing also can cause the similar issue hence its better to log a new request if there are any concerns to verify the install logs/screens and provide solution/workaround"

So, in case You face the 'missing update option, too, please log a Service Request using the keywords "Bug 28350114 : EM 13.3 Upgrade, Installation Types Screen Is Not Showing the Upgrade Option" and supply the install logs.

Read More

Upgrading Enterprise Manager 13.2 to 13.3 - missing 'Update' Option in GUI

Tried to upgrade an Enterprise Manager Cloud Control 13.2 to the next level of happiness - 13.3 - like I did that for several installations. Unfortunately, the 'Upgrading an existing Enterprise Manager System' option in the 'Installation Types' section of the installation GUI is not available. Usually, this behaviour points to an unavailable or corrupt Oracle Inventory. As additional possibility, it could be a bug as well:
Check MOS Note "EM 13c: Checklist for Upgrading Enterprise Manager Cloud Control from Version 12.1.0.5/13.1/13.2 to 13.3 (Doc ID 2418352.1)", the 'Known Issues' section: "Bug 28350114 : EM 13.3 Upgrade, Installation Types Screen Is Not Showing the Upgrade Option"

Read More

Copy MOS-Note URL's

Perhaps You've had that problem, too ... Something has changed with My Oracle Support URLs. Simply copying the url of an already opened MOS document and sending it to a collegue or pasting it into a doc will not give the expected result, i.e., forwarding to the expected document. But how to get the URL? Can be simply as that: Right click on the document's link, 'Copy link address' and You're done ... *wacko*

Read More

Enterprise Manager 13c - 'Tablespace Space Used %' metric sends false alarms

Situation is ...

Metric 'Tablespace Space Used %' is used to monitor the free space of the tablespaces. Sometimes You'll get an alarm that the Tablespace has reached it's critical value. But, the tablespace's data files are auto extensible and far away from the max size and the files are located on an ACFS which knows also auto-extensibility - or better said - an auto-resize feature. There is also plenty of space left on the underlying ASM. Nevertheless, FS free space is currently near to 100% and the current free space in the tablespace is lower than the critical value. Obviously a false alarm, isn't it?

Cause ...

for that behaviour is, that Oracle has changed the method to calculate a tablespace's free space for metric 'Tablespace Space Used %' with release 13c. New is that the metric takes the available disk space into account for its calculations. And this is correct , because no more space left on the disk means finally that a tablespace cannot auto-extend anymore. But with ACFS and the auto resize functionality, this shouldn't be a problem at all - ACFS will auto resize the FS if necessary.

Unfortunately, Oracle Support has no fix for this false alarm at this very moment. MOS says that view DBA_TABLESPACE_USAGE_METRICS has a problem to calculate the metric's base information correctly. Well, that might be a possible reason, but IMHO, the values of my particular database / tablespaces / datafiles are fine.

Solution / Workaround

Simply set the value of the auto resize parameter for the ACFS filesystem in question to an appropriate value other than the default of 20G, to avoid the false alarm problem (until the ASM runs full and ACFS cannot auto resize):
"acfsutil size -a <Size>G <FSName>"

Example:

acfsutil size -a 100G /u02/oradata/TESTDB

acfsutil size: ACFS-03642: successfully updated auto-resize settings

Read More

ODA Server Patch 18.3 available - and installed ...

Since today, Oracle Database Appliance software release 18.3.0.0.0 is available ... I took the chance to download and install the new release - this is the story:

Initial Situation:

I tested the update on an ODA X7-2M with system version 12.2.1.4.0, running five databases (12.1). There are three vlans configured on that system.

Documentation used:

Installation Instruction was taken from docs.oracle.com, Oracle Database Appliance, Release 18.3, X7-2 Deployment and User's Guide for Linux x86-64, Chapter 7 Patching Oracle Database Appliance, subchapter 'Patching Oracle Database Appliance Using the CLI'

Remarks:

- You have to download about 15GB - just for the system update, DB 18.x is around 4GB
- do not copy the update software to a NFS share and do not run the update from this directory
- if You are using NFS shares, be sure that a df -h comes back immediately. With my testsystem I had two NFS Filesystems which were not reachable at all, resulting in a 'never-come-back' df -h - this, in turn, resulted in a stuck Server Patching at step 'Configuring GI'. After I umounted those FS (by using the -f switch), the patching continued
- qosmserver has to run and a crs resource ora.net1.network has to be configured
- the patch documentation is missing the hint to first update dcsagent (odacli upgrade-dcsagent -v 18.3.0.0.0), but it's a prerequisite
- remove any 'non-default' entries from oracle's .bash_profile prior to the update
- Step 'Setting AUDIT SYSLOG LEVEL' had a status 'Failure' at the end of the update process - overall status of the update was - nevertheless - successful (will check the reason for that failure later)
- Installation documentation says after step 'apply the server update' to check if the update was successful and then to issue an 'odacli update-storage'. ATTENTION! After a successful update, the server will reboot! Wait until the system is up again before You start the update-storage command!
- Server reboot took about 10 Minutes with my X7-2M
- Documentation example for 'Update the storage components' is not correct: use '--rolling' or '-r' instead of '-rolling'
- update-storage using the '--rolling' option is not supported on ODAlite

That's it

The whole process took slightly more than one hour - and remember: the system reboots after an successful 'odacli update-server' (!)

Read More

ODA - Update to 12.2.1.4 - issue with ora.net1.network

Yesterday, I had to update an ODA from Release 12.1.2.12 to 12.2.1.4 . All prerequisite checks including prepatchreport were successful ... but the update itself not!

During the update, at stage 'GI Home Cloning', the update process failed and left the system in an 'unknown' state. A detailled analysis has shown the root cause: crs service qosmserver is missing. So, I tried to setup this missing link to a successful update:

srvctl add qosmserver
srvctl status qosmserver
srvctl start qosmserver

But, qosmserver could not be started - resource 'ora.net1.network' is missing. Well, that's absolutely true - we do not have such a resource defined for our ODAs. Finally changed the first of our vlan definitions to ora.net1.network and could start qosmserver.

Unfortunately and because of the interrupted update, GI is now using its 12.2 home, but odacli describe-component shows, that the current home is the old 12.1 - that finally results in a situation where I cannot run update-server again (GI Home Cloning fails, because the new GI Home already exists)

Outcome: As additional prerequisite, check, wether or not the resource ora.net1.network exists on Your ODA and be sure qosmserver is up and running BEFORE starting an 'odacli update-server'.

Read More

DBID Problem @ ODA - or Count von Count (Sesame Street) calculates DBIDs

I did some more investigations to get an explanation for that problem with the extra long DBIDs and the resulting problems when creating new databases on an ODA ...

And here's the simple result:

Remember? In our environment, oracle's .bash_profile runs an additional script, which prints all - at this very moment existing - oracle databases, listeners, their status and the according ORACLE_HOMEs to STDOUT. Output looks like this:

######################################################

# Following Databases/Aliases are on the ODA oda-o1234

######################################################

================================================================================

SID/PROCESS         STATUS  E_ORACLE_HOME

--------------------------------------------------------------------------------

+ASM1               up      /u01/app/12.2.0.1/oracle

TEST1               up      /u01/app/oracle/product/12.1.0.2/dbhome_1

TEST2c              up      /u01/app/oracle/product/12.1.0.2/dbhome_1

TEST2D              up      /u01/app/oracle/product/12.1.0.2/dbhome_1

TEST2E              up      /u01/app/oracle/product/12.1.0.2/dbhome_1

LISTENER            up      /u01/app/12.2.0.1/oracle

Now, take all numbers from this output, put them in one line, add the 'real' DBID from the database - and You'll have the dbid which is displayed (and used) by odacli in the internal derby database. :-)

The easy test:

Add two 'echo' lines to oracle's .bash_profile that it looks like this:

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

        . ~/.bashrc

fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH

umask 022

echo "1111111111111111111111111111111111111111111111111"

echo "2222222222222222222222222222222222222222222222222"

Create a database using the 'odacli create-database' command:

odacli create-database -m -dh 665b9ce4-3a13-4a04-8ccd-548e66556def -n TEST2F -r ACFS

When the database is created, issue an 'odacli list-databases' and, subsequently, an 'odacli describe-database -i <id of that database>' - check the DBID (concatenated echo "1"s plus the echo "2"s plus the 'real' DBID):

Database details

----------------------------------------------------------------

                     ID: 2a5426bc-8db9-4d07-ba87-b54f06dbbcc3

            Description: TEST2F

                DB Name: TEST2F

             DB Version: 12.1.0.2

                DB Type: Si

             DB Edition: EE

                   DBID: 111111111111111111111111111111111111111111111111122222222222222222222222222222222222222222222222222498916505

Instance Only Database: false

                    CDB: false

               PDB Name:

    PDB Admin User Name:

                  Class: OLTP

                  Shape: Odb1

                Storage: ACFS

           CharacterSet: AL32UTF8

  National CharacterSet: AL16UTF16

               Language: AMERICAN

              Territory: AMERICA

                Home ID: 665b9ce4-3a13-4a04-8ccd-548e66556def

        Console Enabled: false

     Level 0 Backup Day: Sunday

    AutoBackup Disabled: false

                Created: November 7, 2018 12:58:45 PM CET

         DB Domain Name: world

The database's DBID - which is the last portion in odacli's DBID:

SQL> select dbid from v$database;

      DBID

----------

2498916505

Imho, the answer for question 'Bug or Feature?' is quite clear ;-) And, by the way, Oracle Support is still investigating the problem.

Read More

Update! - Unexpected Behaviour With V$DIAG_ALERT_EXT

Oops! Errormessages from a Test database in my Production-DB?!?

Story behind:

I wanted to know which "ORA-" Errors have occured in my Production-DB and issued a

select

  to_char(originating_timestamp,'DD.MM.YYYYHH24:MI:SS')

  ,message_text

from v$diag_alert_ext

where message_text like '%ORA-%'

and originating_timestamp > sysdate-31

order by originating_timestamp;

to have an overview about the current - let me say - error situation. So far - so good ...

Yesterday, I've found messages from a TEST database when querying the PRODUCTION db, which made me uncertain:
"Errors in file /u01/app/oracle/diag/rdbms/TEST/TEST/trace/TEST_j000_45562.trc"

Checked that - and discussed it with some collegues ... outcome is:

The View V$DIAG_ALERT_EXT does not only contain information from the current DB but from all databases on that system / from that diagnostic_dest:

select  distinct component_id,filename
from v$diag_alert_ext
order by 1,2;
COMPONENT_ID FILENAME
------------ -----------------------------------------------------------------------
apx          /u01/app/oracle/diag/apx/+apx/+APX1/alert/log.xml
asm          /u01/app/oracle/diag/asm/+asm/+ASM1/alert/log.xml
clients      /u01/app/oracle/diag/clients/user_oracle/host_4163035053_107/alert/log.xml
clients      /u01/app/oracle/diag/clients/user_oracle/host_4163035053_82/alert/log.xml
crs          /u01/app/oracle/diag/crs/host/crs/alert/log.xml
rdbms        /u01/app/oracle/diag/rdbms/dbtest1/DBTEST1/alert/log.xml
rdbms        /u01/app/oracle/diag/rdbms/dbtest10/DBTEST10/alert/log.xml
rdbms        /u01/app/oracle/diag/rdbms/dbtest11/DBTEST11/alert/log.xml
rdbms        /u01/app/oracle/diag/rdbms/dbtest12/DBTEST12/alert/log.xml
rdbms        /u01/app/oracle/diag/rdbms/dbtest13/DBTEST13/alert/log.xml
rdbms        /u01/app/oracle/diag/rdbms/dbtest14/DBTEST14/alert/log.xml
rdbms        /u01/app/oracle/diag/rdbms/dbtest2/DBTEST2/alert/log.xml
rdbms        /u01/app/oracle/diag/rdbms/dbtest3/DBTEST3/alert/log.xml
rdbms        /u01/app/oracle/diag/rdbms/dbtest4/DBTEST4/alert/log.xml
rdbms        /u01/app/oracle/diag/rdbms/dbtest5/DBTEST5/alert/log.xml
rdbms        /u01/app/oracle/diag/rdbms/dbtest6/DBTEST6/alert/log.xml
rdbms        /u01/app/oracle/diag/rdbms/dbtest7/DBTEST7/alert/log.xml
rdbms        /u01/app/oracle/diag/rdbms/dbtest8/DBTEST8/alert/log.xml
rdbms        /u01/app/oracle/diag/rdbms/dbtest9/DBTEST9/alert/log.xml
rdbms        /u01/app/oracle/diag/rdbms/rocrtest/rocrtest/alert/log.xml
tnslsnr      /u01/app/oracle/diag/tnslsnr/host/asmnet1lsnr_asm/alert/log.xml
tnslsnr      /u01/app/oracle/diag/tnslsnr/host/listener/alert/log.xml
tnslsnr      /u01/app/oracle/diag/tnslsnr/host/listener_511/alert/log.xml
tnslsnr      /u01/app/oracle/diag/tnslsnr/host/listener_576/alert/log.xml
tnslsnr      /u01/app/oracle/diag/tnslsnr/host/listener_580/alert/log.xml

Well - that was unexpected ... 

Question is: Bug or feature? I'll open a SR and keep You up to date ...
By the way: Made these tests on different oracle releases: 11.2, 12.1, 12.2.

Update (November 14th, 2018)

Opened a Service Request - and Oracle Support brought an explanation:

"We have verified it. It is an expected scenario. As stated earlier V$DIAG_ALERT_EXT displays trace file and alert file data for the current container (It means all PDB) in a CDB (Even though you connect as ALTER SYSTEM SET CONTAINER - it will show all the database because all database resides on ORACLE HOME). 

Actually, V$DIAG_ALERT_EXT read the logs of all databases and listeners from the ADR Location (ORACLE HOME Directory) So, one connection to a database is enough to see all the database alert files and listener logs registered inside the ADR structure. 

So, this is an expected one. And not a BUG! 
"

Read More

REASON FOUND: DBID Problem @ ODA

Hi There!

in my blogpost "http://robertcrames.blogspot.com/2018/10/oda-bug-and-poor-support-from-oracle.html" I described a problem with the impossibility of creating databases as a result of an incredebly long DBID and the 'poor' oracle support in that specific case.

Well - I contacted Oracle support and had a call with the Support Manager and then ... it worked smoothly 😏. Provided a few more information and Oracle support had an idea:

'Remove any changes from the .bash_profile and try again' ... Unfortunately, we made no changes on root's .bash_profile (which is the 'initiator' of the create database process) :-( - but we made some changes on oracle's .bash_profile. One of these changes was, to set oracle's primary environment to Grid Infrastructure' instance +ASM1. Removed that entry, retried the 'odacli create-database' - and ... it works - DBID is correct and plausible, everything runs fine. :-)

Read More

DBID Problem @ ODA ... Bug (?)

long story told short: 

• Oracle Database Appliance X6-2M and X7-2M, 
• oak 12.1.2.11 up to 12.2.1.4

We were not able to create databases anymore on one of our 15 ODAs. Detailled analysis has shown that the dbid - result from an 'odacli describe-database -i <DB identifier>' - grows and grows and, when it reaches a size of more than 255 characters, any subsequent try to create a database will fail and leave the database in status 'creating'. No more databases can be created, neither can You delete the database which is left in status 'creating'.

This is what a correct DBID (result from a describe-database) looks like: 1653395885 - btw: this is exactly the real dbid found in the database (select dbid from v$database).

An - imho - 'corrupted' DBID looks like this: 2791133100113200112102253200112102127320011210212832001121021533200112102154320011210215632001121021573200112102159320011210217323200112102177232001121021793200112102181320011210218323200112102186320011210215800112102576011210251101121021196170968 - the last 'n' numbers represent the real dbid (select dbid from v$database).

Side effect, when the database creation has failed: 

odacli list-databases

DCS-10001:Internal error encountered: could not execute query.

Would be great if You could do this:

• If You have more than five databases on an ODA, issue an 'odacli list-databases'
• take the id from the last created database and issue an 'odacli describe-database -i <id>'
• Post the result as a comment in this blog

Regarding oracle support:

The Service Request is now open since weeks. Started as a Sev 2, relatively early changed sev to 1. The SR is updated from oracle support very rarely, usually no answer at all when I update the SR. In the meantime I escalated the SR, asked for a recall from a manager - no result and no recall. Today asked for another recall from another manager - will see how long that will take. 

@OracleSupport: that is very poor ... leaving customers helpless and uninformed with such severe Problems. Very disappointing ...

Read More

ODA X7 – no network connect after setup / re-image

Yesterday, I’d prepared an all new X7-2M. During the setup process / network configuration (using VLAN), I ran into a problem: Despite the network configuration was setup absolutely correct, any try to connect to the system from a jumphost was unsuccessful. 

Long story short: If You want to use the SFP28 (Fiberchannel) ports an ODA X7 offers, You have to do some config changes, to make it work. 

First of all: do a firmware upgrade to version 20.08.01.14 or above. 

Next, configure the nic / nics to force the speed to 10000 and turn off autonegotiation. 

The firmware as well as a detailed description of all necessary steps can be found in MOS Note 2373070.1 ‘Using the onboard SFP28 ports on an ODA X7-2 server node’

Read More

How to copy Files to an ODA which is not available via network - by using the network

Sounds a little weird, I know - but works ... *lol*

Problem was: 

The all new ODA has installed old firmware for the SPF28 interfaces. This makes it impossible to connect the system to the network. To upgrade the interfaces You need to have the firmware files on the ODA ... but how can I copy the firmware to the ODA without a network connect?

Solution:

First of all: create an ISO File from the Files You want to have in a virtual cd. With a mac, this is quite easy to do: start disk utility, then 'File / New Image / Image from Folder' and follow the instructions. Disk Utility will create a dmg file which hast to be reamed to iso. That's it, basically.

Next steps:

Open a browser window an connect to the ILOM (https://<ilom's hostname or ip>

Login by using ILOM's root account

Launch console

Open KVMS/Storage 

Add Storage - choose the iso you'd created and klick 'Connect'

Next, in a terminal session, start the console application (start /HOST/console), and mount the cdrom: 

mkdir /media/cdrom

mount /dev/cdrom /media/cdrom

copy the files to whereever You want on the ODA's file system :-)

Read More

Enterprise Manager and the disadvantages of WLS' Dynamic Monitoring Service (lots of metricdump files)

Had an interesting problem this morning with a customers Windows Server where Enterprise Manager 12c is running: Disk C: - where the EM Software is installed - was full. By the way: Imho, oracle software should never ever be installed in C: ...

The Problem:

Reason for the full disk was Weblogic's Dynamic Monitoring Service (or a weak housekeeping tool ;-)).
Explained:
Weblogic has a feature, called DMS, which is checking some domain metrics and saves them to a file in directory (DOMAIN_HOME/server/<ServerName>/logs/metrics) for each managed server. Each file is about 600 to 800k in size, which is not that much - except You have thousands of those files. Filename is, by the way, somewhat like metricdump*. The files are created per default in a 3 hour cycle.

The Solution:

For this customer's system, I decided to stop the Dynamic Monitoring Service for the AdminServer as well as for the EMGC_OMS1 managed server. To achieve this, search for the file dms_config.xml (DOMAIN_HOME/config/fmwconfig/servers/<ServerName>) and change the value of 'enabled' to "false"

<dumpConfiguration>

  <dump intervalSeconds="10800" maxSizeMBytes="75" enabled="false"/>

</dumpConfiguration>




Restart EM to activate the change. 

Another option could be, to change the value of intervalSeconds to a higher value - for example once a day (86'400 secs) or twice a day (43'200 secs). Or, in case you have a proper working housekeeping tool running each day, configure it that it deletes all files older than 'whatever-you-might-think'.

Read More

ODA X6-2M – Expanding Storage can be unsuccessful when impatient

In 'A Brief History of Time' (RIP Mr. Hawking), Chapter 2 is named 'Space and Time' – and this title fits perfect with a current problem, I was faced with, when expanding disk capacity in an ODA X6-2M.

Long story, told short

An ODA X6-2M, equipped with the standard disk configuration, ran out of space and my customer decided to add two NVMe's. So, he ordered two NVMe's – and, by the way - we were waiting for two weeks (!), until delivery. 

Next step - mounting the disks. According documentation, expanding disk storage is a quite easy job:

• Put the disks in
• Set the disks in online state (odaadmcli power disk on pd_02 and odaadmcli power disk on pd_03)
• Expand storage (odaadmcli expand storage)

This results in my environment in a complete disaster:

• Disks are already online when trying a 'power disk on',
• 'expand storage' meant, the disks have already an ASM signature and aborted,
• 'odaadmcli show disk' shows the disks, but als UNKNOWN and with the /dev/nmv* names of the existing NMVes, whilst the old ones had, all of a sudden, new names … (!)
• The command to power off said that the disks are not online, but a subsequent remove of the disks resulted in a system crash …

In short: horror!! Opened a Service Request, tried this, tried that – and finally, after three weeks fighting with Oracle Support, I started a last try: Put the disks in, went to the coffee bar, returned about an hour later. Started to set the disks in online state (result: the disks are already online), and issued an 'odaadmcli expand storage' – expecting the same result as with every try before … but, surprisingly, the command returned no error, ASM started rebalancing and everything went fine J

Solution / Chapter 2 ... 'Space and Time'

You'll get the disk space, when You take Your time ...

Put the disks in, wait at least 15 minutes, power on the disks and issue the 'expand storage' command. Oracle Supports explaination of this behavior was (in my words): The oakd process needs some time to prepare the disks in a way that expand storage works properly …

Read More

ODA Xx – Space Waste When Using ACFS (instead of ASM) For Databases

Disclaimer: I will not discuss the necessity of an ACFS FS to store database files, nor compare ASM and ACFS or it's benefits. What to use / what fits best in Your environment is up to You, folks … ;-)

Pretty sure that this is for lots of people a well-known problem – but for all others:

When creating a database using ACFS as storage option, a FS of size 100GB is created. This 100GB is not a configurable value, by the way. Result is: When having lots of small databases, making use of ACFS wastes a lot of disk space, as every database's file system occupies 100GB.
Oracle Support's answer to that is: "In the future releases you may see the customized option to define the size of ACFS while creating the database". Nevertheless: It is possible to reduce that 100GB FS to an appropriate size. Here's how:  

Examples are based on OAK 12.2.1.2.0, used db release: 12.1.0.2

Create the database:

oracle@eoda03 ~]$ odacli create-database -m -n RCRACFS1 --no-cdb  --dbstorage ACFS -dh cce28d1b-01cc-4917-8237-38683d34f53e

… results in (from a FS perspective)

Filesystem                   Size  Used Avail Use% Mounted on

/dev/asm/datrcracfs1-262     100G  2.5G   98G   3% /u02/app/oracle/oradata/RCRACFS1

Reduce the filesystem size to 10GB:

[oracle@eoda03 ~]$ acfsutil size -90G /u02/app/oracle/oradata/RCRACFS1

acfsutil size: new file system size: 10737418240 (10240MB)

Check the filesystem size

[oracle@oda03 ~]$ df -h /u02/app/oracle/oradata/RCRACFS1

Filesystem                   Size  Used Avail Use% Mounted on

/dev/asm/datrcracfs1-262      20G  2.5G   18G  13% /u02/app/oracle/oradata/RCRACFS1

20G?? 100 minus 90 is 20? Possibly yes (you never know ;-)) – on the other hand and a better explanation: 20GB could be minimum FS for an ACFS file system.

Conclusion:

Yes! It is that easy J

Read More