Wednesday, August 14, 2013

tail: inotify resources exhausted

I've been facing issue with tail. Once executing:
tail -f <filename_to_tail>
I got error:
tail: die Inotify-Resourcen sind erschöpft
tail: Inotify kann nicht verwendet werden, es wird auf Pollen zurückgegriffen
For those of you having english locale, you'd see:
tail: inotify resources exhausted
tail: inotify cannot be used, reverting to polling
While looking around I've found following to be valuable in understanding the problem and solving it: So the error itself means that system is getting low on inotify watches, that enable programs to monitor file/dirs changes.

To see the currently set limit (including output on my machine), I executed:
cat /proc/sys/fs/inotify/max_user_watches
8192
Well, I had to decide about the limit. So I went for 1048576 (not sure why :) ). To set it I've found 2 solutions.

Temporary solution (not preserved across restarts)

As I'm used to sudo, I went for:
sudo echo 1048576 > /proc/sys/fs/inotify/max_user_watches
-bash: /proc/sys/fs/inotify/max_user_watches: Keine Berechtigung
OK, not working. Complaining about user rights on /proc/sys/fs/inotify/max_user_watches. Then based on: How do I use sudo to redirect output to a location I don't have permission to write to? I went for:
sudo sh -c 'echo 1048576 > /proc/sys/fs/inotify/max_user_watches'
which worked, as the command proofs:
cat /proc/sys/fs/inotify/max_user_watches
1048576
The problem is that the value is reset each time I rebooted.

Permanent solution (preserved across restarts)

Adding line:
fs.inotify.max_user_watches=1048576
to:
/etc/sysctl.conf
fixed the limit value permanently (even between restarts).

Deeper analysis

However if you're interested to investigate problem in deep, it would be good idea to find out who is using the resources. There seem to be an idea at unix.stackexchange: Who's consuming my inotify resources?.
for foo in /proc/*/fd/*; do readlink -f $foo; done | grep inotify | sort | uniq -c | sort -nr
However for me it didn't show any output I could analyse to dig deep.

On the other hand I convinced myself that Dropbox is guilty here. Even though I have no proof for that (just read on some of the sources where someone suspected the same service).

No comments: