Tyler Muth’s Blog

Technology with a focus on Oracle, Application Express and Linux

Scripted Collection of OS Watcher Files on Exadata

Posted by Tyler Muth on November 2, 2012

I’ve been working a lot with graphing DB and OS metrics in R. I find it especially useful in Exadata POVs (proof of value) to gather and graph the oswatcher vmstat files for the compute nodes and iostat for the cells. For an example, take a look at this graph (PDF, 168 KB) of what happens when you have a connection pooling issue and keep creating connections throughout the test (14,930 to be precise). It’s a nice picture of what happened to the OS as it ran out of memory, started swapping, then fell over. To quote the Real World Performance Team: “When Databases swap, Databases stop”.

Anyway, back to the point of this post. OS Watcher files are automatically collected on each db node and storage cell. It’s not too bad to get them manually on a 1/4 rack as that’s only 5 components, but what about a full X3-2 with 22 components? Or the previous POV I did with 3 X2-8’s which had 42 storage cells and 6 DB nodes? Note that support will often ask for these files when you have an issue and log an SR.

Here’s a script I wrote to collect the files with various find options in the comments at the top. It also optionally takes in 1 parameter of a name to include in the tar.gz files so you can identify them easier. It will create 1 tar.gz file for each component, so in the case of a 1/4 rack it will create 5 files.

if [ $# -gt 0 ]

# A few find examples
# \( -name "*.*" \) \
# \( -name "*vmstat*.*" -o -name "*iostat*.*" \) \
# \( -name "*vmstat*.*" -o -name "*iostat*.*" \) -mmin -60 \
# \( -name "*vmstat*.*" -o -name "*iostat*.*" -o -name "*netstat*.*" \) -mtime -8 \

while read line; do
	(ssh -n -q root@$line 'find /opt/oracle.oswatcher/osw/archive  \
	\( -name "*vmstat*.*" -o -name "*iostat*.*" \) \
	-prune -print0 2>/dev/null | \
	xargs -0 tar --no-recursion -czf - 2>/dev/null ' | cat >  osw${file_name}_${line}.tar.gz
done < /home/oracle/tyler/osw/allnodes

Note the file “allnodes” on the last line is just a file with the names of the db nodes and cells, each on a new line. It’s the same format you use for dcli. This is of course a lot easier if you’ve setup root equivalencies for each component…

10 Responses to “Scripted Collection of OS Watcher Files on Exadata”

  1. Murali said

    very handy script, thanks for sharing, do you have the script which you used to generate the graphs?

    • Tyler Muth said

      That perl and R that generate the graph aren’t really ready for distribution. Sorry.

      • DanyC said


        Would very much appreciate if you could shre the graph script – maybe during xmas time 🙂


      • Yasser said

        Apologize for asking it again after 4 years. Could you please share the scripts you have used to parse and plot oswatcher vmstat files. If scripts are not ready then I can try to develop it for distribution if you can provide me the incomplete scripts. I did check your github repository but they seems to be of different purpose. Thanks in advance.

  2. Jagjeet Singh said

    Thanks for sharing it. Really very useful and handy script. It would be great if you can share perl and R script once ready

  3. Tyler, good useful stuff. You rock! Thanks!

  4. manta.com said

    A motivating discussion is definitely worth comment. There’s no doubt that that you need to write more about this topic, it may not be a taboo subject but usually people do not discuss these topics. To the next! All the best!!

  5. Bhargav Theertham said

    OSWatcher comes with the utility that you can use to generate graphs. Depending on the version of OSwatcher, you will see
    oswbba.jar or a similar jar file. You can run the below command to get to an interactive prompt where you can choose what metrics you want to generate the graphs for

    java -jar oswbba.jar -i

  6. Howdy! I know this is kinda off topic however I’d figured I’d ask.
    Would you be interested in exchanging links or maybe guest writing a blog post or vice-versa?
    My blog goes over a lot of the same subjects as yours and I believe we could greatly benefit
    from each other. If you’re interested feel free to shoot me an email.
    I look forward to hearing from you! Excellent blog by the way!

  7. Excellent blog here! Also your website loads up very fast!

    What host are you using? Can I get your affiliate link to your host?
    I wish my web site loaded up as quickly as yours lol

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: