unmount problems

[Bushmann]

Hi, I've got a little problem concerning mount/unmount:

I'm using nfs to remotely mount /usr/portage on my computer, but not directly to /usr/portage but to /usr/rportage. In /usr/portage there is still a local version of portage (for emergency/whatever), and when my computer (it's a notebook) is connected to lan I mount the /usr/rportage on /usr/portage via mount --bind (so actually it's doubly mounted... /usr/portage -> /usr/rportage/ -> server:/usr/portage).
/usr/rportage is by the way mounted via autofs so it's all automatic.
Now the problem comes when I get disconnected and I don't unmount _before_. Then I cannot unmount /usr/portage from /usr/rportage, I only get the 'device busy answer'.

But lsof doesn't show anything accessing /usr/portage, so I really don't know how to solve this problem...
Is there a way to force unmounting or to find out _why_ the heck the directory is busy?