Oracle Database and Linux Support Articles

OSWbb – How To Install And Configure OSWatcher Black Box For System Diagnostics

System troubleshooting is the major task for admin in day to day activities.

System admins are facing difficulties to identify system performance issue on Linux.

They can not able to figure it out what actually causing the issue. It is very hard to find the root cause.

There are many tools available in Linux to identify system performance issue but each tool (like top, vmstat, ps, iostat, etc) has it own features and we can’t get all in one place to identify the root causes.

Few tools are available in Linux to get all these details in one place. We already covered such kind of topic in past called sosreport.

What Is OSWatcher (OSWbb)

Oracle OSWatcher Black Box (OSWbb) collects and archives operating system and network metrics that you can use to diagnose performance issues. It’s a set of shell scripts OSWbb operates as a set of background processes on the server and gathers data on a regular basis, invoking below Unix utilities.

Make sure the blow packages should be installed on your system as prerequisites. If no, install missing packages using distribution package manager.

§ VMSTAT

§ IOSTAT

§ MPSTAT

§ IFCONFIG

§ NETSTAT

§ TOP

§ TRACEROUTE

oswbb is comprised of two separate components:

§ oswbb: a unix shell script data collector which collects and stores the data.

§ oswbba: a java utility which will analyze the data automatically and provide advice and produce graphs and an html document.

How To Install OSWatcher (OSWbb) In Linux

This is straight forward. Just download the latest available package version from oracle website and extract to descried location.

§ Log on to My Oracle Support (MOS) at http://support.oracle.com

§ Download the file oswbbXXX.tar, which is available at https://support.oracle.com/epmos/main/downloadattachmentprocessor?attachid=301137.1:OSW_file

Move/Copy the file to the directory where you want to install OSWbb and extract the file.

# tar xvf oswbb812.tar

It creates a directory called oswbb, which contains all the directories and files that are associated with OSWbb.

# ls -lh /opt/oswbb

total 516K

drwx------ 2 oracle oinstall 4.0K Sep 6 2017 analysis

drwx------ 13 oracle oinstall 4.0K Mar 1 18:03 archive

drwx------ 13 oracle oinstall 4.0K Oct 20 13:49 archive-old

-rwxr-xr-x 1 oracle oinstall 67 Jan 15 2014 call_du.sh

-rwxr-xr-x 1 oracle oinstall 68 Oct 7 2013 call_sar.sh

-rwxr-xr-x 1 oracle oinstall 71 Jan 7 2014 call_uptime.sh

drwx------ 2 oracle oinstall 4.0K May 2 2017 data

drwx------ 4 oracle oinstall 4.0K Sep 5 2017 docs

-rwxr-xr-x 1 oracle oinstall 665 Feb 6 2017 Example_extras.txt

-rwxr-xr-x 1 oracle oinstall 1.9K Oct 7 2013 Exampleprivate.net

-rwxr-xr-x 1 oracle oinstall 3.9K Apr 17 2017 genprvnet.sh

drwx------ 2 oracle oinstall 4.0K Sep 6 2017 gif

-rwxr-xr-x 1 oracle oinstall 772 May 8 2014 ifconfigsub.sh

-rwxr-xr-x 1 oracle oinstall 743 Oct 7 2013 iosub.sh

drwx------ 2 root root 4.0K Mar 16 19:59 locks

-rwxr-xr-x 1 oracle oinstall 1.5K Jan 8 2014 ltop.sh

-rwxr-xr-x 1 oracle oinstall 542 Oct 7 2013 mpsub.sh

-rwxr-xr-x 1 oracle oinstall 740 Oct 7 2013 nfssub.sh

-rwxr-xr-x 1 oracle oinstall 6.7K Jul 12 2017 OSWatcherFM.sh

-rwxr-xr-x 1 oracle oinstall 47K Aug 14 2017 OSWatcher.sh

-rw-r--r-- 1 oracle oinstall 10K Sep 6 2017 oswbb810.tar

-rwxr-xr-x 1 oracle oinstall 310K Sep 6 2017 oswbba.jar

-rwxr-xr-x 1 oracle oinstall 414 Oct 7 2013 oswib.sh

-rwxr-xr-x 1 oracle oinstall 435 Jan 8 2014 oswnet.sh

-rwxr-xr-x 1 oracle oinstall 825 Oct 7 2013 oswrds.sh

-rwxr-xr-x 1 oracle oinstall 524 Oct 7 2013 oswsub.sh

-rwxr-xr-x 1 oracle oinstall 1.5K Aug 24 2017 psmemsub.sh

-rwxr-xr-x 1 oracle oinstall 557 Jul 12 2017 sarsub.sh

drwx------ 7 oracle oinstall 4.0K Sep 6 2017 src

-rwxr-xr-x 1 oracle oinstall 2.6K Feb 26 2015 startOSWbb.sh

-rwxr-xr-x 1 oracle oinstall 751 Aug 14 2017 stopOSWbb.sh

-rwxr-xr-x 1 oracle oinstall 834 Feb 6 2017 tar_up_full_archive.sh

-rwxr-xr-x 1 oracle oinstall 5.3K Feb 6 2017 tar_up_partial_archive.sh

drwx------ 2 root root 4.0K Mar 16 19:59 tmp

-rwxr-xr-x 1 oracle oinstall 527 Feb 7 2017 topaix.sh

-rwxr-xr-x 1 oracle oinstall 545 Feb 23 2015 vmsub.sh

-rwxr-xr-x 1 oracle oinstall 1.5K Feb 26 2015 xtop.sh

To start OSWbb, run the startOSWbb.sh script from the oswbb directory. This script accepts two parameters, which is Frequency & Duration.

§ Frequency : How frequent (in seconds) data needs to be collected.

§ Duration : How many hours data needs to be archived

If you do not input any arguments, the script runs with default values of 30 and 48. It collects data every 30 seconds and store the last 48 hours of data in archive files.

# ./startOSWbb.sh [Frequency Duration]

The below example collects and stores the last 10 hours of data to archive files at 60 second intervals.

# ./startOSWbb.sh 60 10

Setting the archive log directory to/opt/oswbb/archive

Testing for discovery of OS Utilities...

VMSTAT found on your system.

IOSTAT found on your system.

MPSTAT found on your system.

IFCONFIG found on your system.

NETSTAT found on your system.

TOP found on your system.

TRACEROUTE found on your system.

Discovery of CPU CORE COUNT

CPU CORE COUNT will be used by oswbba to automatically look for cpu problems

CPU CORE COUNT = 6

VCPUS/THREADS = 24

Discovery completed.

Starting OSWatcher v8.1.0 on Tue Feb 27 15:14:54 MST 2018

With SnapshotInterval = 60

With ArchiveInterval = 10

OSWatcher - Written by Carl Davis, Center of Expertise,

Oracle Corporation

For questions on install/usage please go to MOS (Note:301137.1)

If you need further assistance or have comments or enhancement

requests you can email me Carl.Davis@Oracle.com

Data is stored in directory: /opt/oswbb/archive

Starting Data Collection...

oswbb heartbeat:Tue Feb 27 15:14:59 MST 2018

oswbb heartbeat:Tue Feb 27 15:15:59 MST 2018

oswbb heartbeat:Tue Feb 27 15:17:00 MST 2018

oswbb heartbeat:Tue Feb 27 15:18:00 MST 2018

oswbb heartbeat:Tue Feb 27 15:19:00 MST 2018

oswbb heartbeat:Tue Feb 27 15:20:00 MST 2018

oswbb heartbeat:Tue Feb 27 15:21:00 MST 2018

oswbb heartbeat:Tue Feb 27 15:22:00 MST 2018

To stop OSWbb prematurely, run the stopOSWbb.sh script from the oswbb directory.

# ./stopOSWbb.sh

How To View The Collected Data

Collected data will be stored under archive directory, it is created when OSWbb is started for the first time. OSWbb stores data in hourly archive files named system_name_utility_name_timestamp.dat, and each entry in a file is preceded by the characters *** and a timestamp.

# ls -lh /opt/oswbb/archive

total 44K

drwx------ 2 root root 4.0K Mar 1 18:06 oswcpuinfo

drwx------ 2 root root 4.0K Mar 16 19:01 oswifconfig

drwx------ 2 root root 4.0K Mar 16 19:01 oswiostat

drwx------ 2 root root 4.0K Mar 16 19:01 oswmeminfo

drwx------ 2 root root 4.0K Mar 16 19:01 oswmpstat

drwx------ 2 root root 4.0K Mar 16 19:01 oswnetstat

drwx------ 2 root root 4.0K Mar 1 18:03 oswprvtnet

drwx------ 2 root root 4.0K Mar 16 19:01 oswps

drwx------ 2 root root 4.0K Mar 16 19:01 oswslabinfo

drwx------ 2 root root 4.0K Mar 16 19:01 oswtop

drwx------ 2 root root 4.0K Mar 16 19:01 oswvmstat

To check any data, just navigate to corresponding directory and view the corresponding file. For testing purpose, we printed vmstat file output.

# more oracle.2daygeek.com_vmstat_18.02.28.2300.dat

Linux OSWbb v8.1.0 oracle.2daygeek.com

SNAP_INTERVAL 60

CPU_CORES 6

VCPUS 24

OSWBB_ARCHIVE_DEST /opt/oswbb/archive

zzz ***Wed Feb 28 23:00:09 MST 2018