Sherlocking Mock based builds

Jenkins is frozen, waiting for something. One useful link is to look at ${JENKINS_URL}/threadDump. That reveals which thread is doing what.

Mine is stuck like this:

waiting for hudson.remoting.Channel@f7f725

Okay, I logged onto the other machine, and yes, it still has an open ssh connection to the guest. So log on to the guest, and look at the open files:

sudo lsof -i 4

Okay, I now know some PIDs: 775 and 790

Let's see the file descriptors:

sudo ls -l /proc/790/fd/

lrwx------ 1 root root 64 May 16 19:51 0 -> /dev/null
lrwx------ 1 root root 64 May 16 19:51 1 -> /dev/null
lr-x------ 1 root root 64 May 16 19:51 10 -> pipe:[7800]
lrwx------ 1 root root 64 May 16 19:51 2 -> /dev/null
lrwx------ 1 root root 64 May 16 19:51 3 -> socket:[7744]
lrwx------ 1 root root 64 May 16 19:51 4 -> socket:[7781]
lrwx------ 1 root root 64 May 16 19:51 5 -> socket:[7791]
lr-x------ 1 root root 64 May 16 19:51 6 -> pipe:[7798]
l-wx------ 1 root root 64 May 16 19:51 7 -> pipe:[7798]

Nice, I guess that pipe is the bad guy, but how do I know, what is that?

sudo lsof  | grep 7800

sshd        790         ubuntu   10r     FIFO                0,8      0t0       7800 pipe
udevd     30043           root    4w     FIFO                0,8      0t0       7800 pipe

All right, that means that I know a new PID...

sudo ls -l /proc/30043/fd/

Okay, that's nice, but what was the command line?

sudo cat /proc/30043/cmdline
udevd --daemon

Nice, now I need to look at mock and udevd.