Comments on: Tricks to diagnose processes blocked on strong I/O in linux http://blog.notreally.org/2008/02/10/tricks-to-diagnose-processes-blocked-on-strong-io-in-linux/ or is it? Thu, 18 Feb 2010 07:49:57 +0000 http://wordpress.com/ hourly 1 By: Gong Cheng http://blog.notreally.org/2008/02/10/tricks-to-diagnose-processes-blocked-on-strong-io-in-linux/#comment-858 Gong Cheng Wed, 18 Nov 2009 18:01:06 +0000 http://blog.notreally.org/articles/2008/02/10/tricks-to-diagnose-processes-blocked-on-strong-io-in-linux/#comment-858 Thanks for the tip! My system is not really a desktop but I will look to get iotop running on it. As for gdb, normally I could attach gdb to a running process to see what it is doing, but I guess I was asking for too mucWill try that when it happens again.h when it is in D state. Again, thanks! I googled and found a lot of links related to the D state topic, but not many provided me with useful debugging tips. Thanks for the tip! My system is not really a desktop but I will look to get iotop running on it.

As for gdb, normally I could attach gdb to a running process to see what it is doing, but I guess I was asking for too mucWill try that when it happens again.h when it is in D state.

Again, thanks! I googled and found a lot of links related to the D state topic, but not many provided me with useful debugging tips.

]]> By: jroncero http://blog.notreally.org/2008/02/10/tricks-to-diagnose-processes-blocked-on-strong-io-in-linux/#comment-857 jroncero Wed, 18 Nov 2009 00:08:55 +0000 http://blog.notreally.org/articles/2008/02/10/tricks-to-diagnose-processes-blocked-on-strong-io-in-linux/#comment-857 Well, if you have 2.6.31 then you can easily find what's using I/O. Just use iotop to find out which process is using it. If strace (why gdb?) is not showing anything, then it's possibly that the processes are blocked on IO but not doing anything at all apart from waiting from the disks or something and this is easy just by looking at the list of processes which are in <strong>D</strong> state. That's easy. The difficult part is to actually find out which of your many processes in D state is actually causing problems, on linux 2.4. Well, if you have 2.6.31 then you can easily find what’s using I/O. Just use iotop to find out which process is using it.

If strace (why gdb?) is not showing anything, then it’s possibly that the processes are blocked on IO but not doing anything at all apart from waiting from the disks or something and this is easy just by looking at the list of processes which are in D state. That’s easy. The difficult part is to actually find out which of your many processes in D state is actually causing problems, on linux 2.4.

]]> By: Gong Cheng http://blog.notreally.org/2008/02/10/tricks-to-diagnose-processes-blocked-on-strong-io-in-linux/#comment-856 Gong Cheng Tue, 17 Nov 2009 20:09:11 +0000 http://blog.notreally.org/articles/2008/02/10/tricks-to-diagnose-processes-blocked-on-strong-io-in-linux/#comment-856 thanks for the post. unfortunately, when it happened to me on 2.6.31, even strace/gdb won't display anything. thanks for the post. unfortunately, when it happened to me on 2.6.31, even strace/gdb won’t display anything.

]]> By: jroncero http://blog.notreally.org/2008/02/10/tricks-to-diagnose-processes-blocked-on-strong-io-in-linux/#comment-827 jroncero Fri, 06 Nov 2009 09:09:47 +0000 http://blog.notreally.org/articles/2008/02/10/tricks-to-diagnose-processes-blocked-on-strong-io-in-linux/#comment-827 I'm afraid this is difficult to diagnose. Are you using 2.6? 2.4? With 2.6 you have a wider range of IO schedulers you can use (under linux) that you can try. Then you have ionice, the linux utility that let you change the IO priority (doesn't work with all the IO schedulers). Having said that, when a process is in D state, there's not much you can do about it. Not even root can kill it (uninterruptible sleep it is). This usually boils down to the process waiting for the OS to come back from an IO operation, normally disk IO). Shutdown is probably taking a long time because it's writing all the buffers and cache back to disk and finishing everything. This is usually normal (I've seen MySQL instances taking a good few minutes to shut down) and depends on how the filesytems is configured. There is a lot you can do here, depending on what filesystem is, how it's mounted, if it's got delay write, barriers, journaling, etc. I'm afraid I have no idea about the Veritas FS, so I can't give you much advise on this front. I’m afraid this is difficult to diagnose. Are you using 2.6? 2.4? With 2.6 you have a wider range of IO schedulers you can use (under linux) that you can try. Then you have ionice, the linux utility that let you change the IO priority (doesn’t work with all the IO schedulers).

Having said that, when a process is in D state, there’s not much you can do about it. Not even root can kill it (uninterruptible sleep it is). This usually boils down to the process waiting for the OS to come back from an IO operation, normally disk IO).

Shutdown is probably taking a long time because it’s writing all the buffers and cache back to disk and finishing everything. This is usually normal (I’ve seen MySQL instances taking a good few minutes to shut down) and depends on how the filesytems is configured. There is a lot you can do here, depending on what filesystem is, how it’s mounted, if it’s got delay write, barriers, journaling, etc. I’m afraid I have no idea about the Veritas FS, so I can’t give you much advise on this front.

]]> By: Pratheep http://blog.notreally.org/2008/02/10/tricks-to-diagnose-processes-blocked-on-strong-io-in-linux/#comment-826 Pratheep Fri, 06 Nov 2009 08:42:17 +0000 http://blog.notreally.org/articles/2008/02/10/tricks-to-diagnose-processes-blocked-on-strong-io-in-linux/#comment-826 Hi, Very good post, thank you. One of my production Oracle database on Suse linux having the same issue. Some of the oracle process are going in to D state oftenly. So the database shutdown hangs every time. (root also not able to kill the process). I tried strace the pid and that too hangs. File system is Veritas. If you have any idea about to trace this process or point this because of filesystem. Any suggestions appreciated. Thanks in advance Pratheep n.pratheep@gmail.com Hi,

Very good post, thank you.

One of my production Oracle database on Suse linux having the same issue. Some of the oracle process are going in to D state oftenly. So the database shutdown hangs every time. (root also not able to kill the process). I tried strace the pid and that too hangs. File system is Veritas.
If you have any idea about to trace this process or point this because of filesystem. Any suggestions appreciated.
Thanks in advance
Pratheep
n.pratheep@gmail.com

]]> By: Ravi http://blog.notreally.org/2008/02/10/tricks-to-diagnose-processes-blocked-on-strong-io-in-linux/#comment-737 Ravi Thu, 10 Sep 2009 11:25:55 +0000 http://blog.notreally.org/articles/2008/02/10/tricks-to-diagnose-processes-blocked-on-strong-io-in-linux/#comment-737 thanks man, Really good post, it will help to find the process stuck issue and for too increase performance thanks man,
Really good post, it will help to find the process stuck issue and for too increase performance

]]> By: roger http://blog.notreally.org/2008/02/10/tricks-to-diagnose-processes-blocked-on-strong-io-in-linux/#comment-532 roger Mon, 09 Feb 2009 19:59:23 +0000 http://blog.notreally.org/articles/2008/02/10/tricks-to-diagnose-processes-blocked-on-strong-io-in-linux/#comment-532 Finally some real information on finding the IO culprits. Thanks much! Finally some real information on finding the IO culprits. Thanks much!

]]> By: Terminus » Blog Archive » Pregunta de examen http://blog.notreally.org/2008/02/10/tricks-to-diagnose-processes-blocked-on-strong-io-in-linux/#comment-534 Terminus » Blog Archive » Pregunta de examen Mon, 11 Feb 2008 11:48:24 +0000 http://blog.notreally.org/articles/2008/02/10/tricks-to-diagnose-processes-blocked-on-strong-io-in-linux/#comment-534 [...] Actualización 11 febrero: He publicado un artículo en mi otro blog sobre el tema: Trucos para detectar procesos bloqueados por I/O. [...] [...] Actualización 11 febrero: He publicado un artículo en mi otro blog sobre el tema: Trucos para detectar procesos bloqueados por I/O. [...]

]]>