HBase Too many open files, OS X 10.6.8

Today I ran into the following error

2011-12-18 13:02:52,918 FATAL org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Replay of hlog required. Forcing server shutdown
org.apache.hadoop.hbase.DroppedSnapshotException: region: .META.,,1
	at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:946)
	at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:839)
	at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:241)
	at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:149)
Caused by: java.io.FileNotFoundException: /Users/maxgarfinkel/Hadoop/hbase-datastore/hbase-maxgarfinkel/hbase/.META./1028785192/info/223659818943414297 (Too many open files)

This is a well documented error and is caused by HBase opening more files than the OS will let it. To be clear this is an OS configuration issue. On the mac OS the default max number of files a process can open is 264, this can however be changed.

The command

launchctl

opens a command line application that lets you view and update this setting and related system settings. Once in launchctl typing the command

 limit

shows the following output:

max-garfinkels-computer:~ maxgarfinkel$ launchctl
launchd% limit
	cpu         unlimited      unlimited      
	filesize    unlimited      unlimited      
	data        unlimited      unlimited      
	stack       8388608        67104768       
	core        0              unlimited      
	rss         unlimited      unlimited      
	memlock     unlimited      unlimited      
	maxproc     266            532            
	maxfiles    20000          50000          
launchd%

When in launchctl you can update the maxfiles property by issuing the command:

limit maxfiles 20000 5000

The first value is the soft limit and the second value is the hard limit. I don’t know what the difference is, however I do know you cannot set either to unlimited. The big problem I found however is that this setting doesn’t seem to stick. To make it stick there is a config file you must change. The file is /etc/launchd.conf, and you will probably have to create it. In nano this is dead easy:

sudo nano /etc/launchd.conf

Then add the following line to the file:

limit maxfiles 20000 50000

save it and reboot. If this fails it seems to fail silently so it is worth verifying by launching launchctl and issuing the limit command.
You can check that HBase has picked up this setting by checking the master log and the regionservers log. The mater log file will have the following line, dumped at start up:

Sun 18 Dec 2011 13:32:52 GMT Starting master on max-garfinkels-computer.local
ulimit -n 20000

and the region log will have :

Sun Dec 18 13:32:53 GMT 2011 Starting regionserver on max-garfinkels-computer.local
ulimit -n 20000

with the ulimit -n xxx being the maxfiles limit we applied.