HBase Too many open files, OS X 10.6.8
Today I ran into the following error
2011-12-18 13:02:52,918 FATAL org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Replay of hlog required. Forcing server shutdown org.apache.hadoop.hbase.DroppedSnapshotException: region: .META.,,1 at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:946) at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:839) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:241) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:149) Caused by: java.io.FileNotFoundException: /Users/maxgarfinkel/Hadoop/hbase-datastore/hbase-maxgarfinkel/hbase/.META./1028785192/info/223659818943414297 (Too many open files)
This is a well documented error and is caused by HBase opening more files than the OS will let it. To be clear this is an OS configuration issue. On the mac OS the default max number of files a process can open is 264, this
Reading files from in a jar
Getting a java.io.FileNotFoundException when trying to load a resource. This didn’t occur when running the code from the IDE but when it was deployed.
The culprit was getClass().getResource(“filename”), which returns a url. I then created a BufferedReader from a FileReader, from a File, from the url, phew.
What I think happens in that the wrong handler get used, java assumes you are trying to read a normal file from the filesystem and uses the file protocol