Tyler Muth’s Blog

Technology with a focus on Oracle, Application Express and Linux

Scripted Collection of OS Watcher Files on Exadata

Posted by Tyler Muth on November 2, 2012

I’ve been working a lot with graphing DB and OS metrics in R. I find it especially useful in Exadata POVs (proof of value) to gather and graph the oswatcher vmstat files for the compute nodes and iostat for the cells. For an example, take a look at this graph (PDF, 168 KB) of what happens when you have a connection pooling issue and keep creating connections throughout the test (14,930 to be precise). It’s a nice picture of what happened to the OS as it ran out of memory, started swapping, then fell over. To quote the Real World Performance Team: “When Databases swap, Databases stop”.

Anyway, back to the point of this post. OS Watcher files are automatically collected on each db node and storage cell. It’s not too bad to get them manually on a 1/4 rack as that’s only 5 components, but what about a full X3-2 with 22 components? Or the previous POV I did with 3 X2-8′s which had 42 storage cells and 6 DB nodes? Note that support will often ask for these files when you have an issue and log an SR.

Here’s a script I wrote to collect the files with various find options in the comments at the top. It also optionally takes in 1 parameter of a name to include in the tar.gz files so you can identify them easier. It will create 1 tar.gz file for each component, so in the case of a 1/4 rack it will create 5 files.

#/bin/bash
file_name=""
if [ $# -gt 0 ]
then
	file_name="_$1"
fi

# A few find examples
# \( -name "*.*" \) \
# \( -name "*vmstat*.*" -o -name "*iostat*.*" \) \
# \( -name "*vmstat*.*" -o -name "*iostat*.*" \) -mmin -60 \
# \( -name "*vmstat*.*" -o -name "*iostat*.*" -o -name "*netstat*.*" \) -mtime -8 \

while read line; do
	(ssh -n -q root@$line 'find /opt/oracle.oswatcher/osw/archive  \
	\( -name "*vmstat*.*" -o -name "*iostat*.*" \) \
	-prune -print0 2>/dev/null | \
	xargs -0 tar --no-recursion -czf - 2>/dev/null ' | cat >  osw${file_name}_${line}.tar.gz
)
done < /home/oracle/tyler/osw/allnodes

Note the file “allnodes” on the last line is just a file with the names of the db nodes and cells, each on a new line. It’s the same format you use for dcli. This is of course a lot easier if you’ve setup root equivalencies for each component…

About these ads

9 Responses to “Scripted Collection of OS Watcher Files on Exadata”

  1. Murali said

    very handy script, thanks for sharing, do you have the script which you used to generate the graphs?

  2. Jagjeet Singh said

    Thanks for sharing it. Really very useful and handy script. It would be great if you can share perl and R script once ready

  3. Tyler, good useful stuff. You rock! Thanks!

  4. manta.com said

    A motivating discussion is definitely worth comment. There’s no doubt that that you need to write more about this topic, it may not be a taboo subject but usually people do not discuss these topics. To the next! All the best!!

  5. Bhargav Theertham said

    OSWatcher comes with the utility that you can use to generate graphs. Depending on the version of OSwatcher, you will see
    oswbba.jar or a similar jar file. You can run the below command to get to an interactive prompt where you can choose what metrics you want to generate the graphs for

    java -jar oswbba.jar -i

  6. Howdy! I know this is kinda off topic however I’d figured I’d ask.
    Would you be interested in exchanging links or maybe guest writing a blog post or vice-versa?
    My blog goes over a lot of the same subjects as yours and I believe we could greatly benefit
    from each other. If you’re interested feel free to shoot me an email.
    I look forward to hearing from you! Excellent blog by the way!

  7. Excellent blog here! Also your website loads up very fast!

    What host are you using? Can I get your affiliate link to your host?
    I wish my web site loaded up as quickly as yours lol

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
Follow

Get every new post delivered to your Inbox.

Join 89 other followers

%d bloggers like this: