2011-12-18 13:02:52,918 FATAL org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Replay of hlog required. Forcing server shutdown
org.apache.hadoop.hbase.DroppedSnapshotException: region: .META.,,1
	at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:946)
	at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:839)
	at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:241)
	at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:149)
Caused by: java.io.FileNotFoundException: /Users/maxgarfinkel/Hadoop/hbase-datastore/hbase-maxgarfinkel/hbase/.META./1028785192/info/223659818943414297 (Too many open files)

This is a well documented error and is caused by HBase opening more files than the OS will let it. To be clear this is an OS configuration issue. On the mac OS the default max number of files a process can open is 264, this can however be changed.

The command

launchctl

opens a command line application that lets you view and update this setting and related system settings. Once in launchctl typing the command

 limit

shows the following output:

max-garfinkels-computer:~ maxgarfinkel$ launchctl
launchd% limit
	cpu         unlimited      unlimited      
	filesize    unlimited      unlimited      
	data        unlimited      unlimited      
	stack       8388608        67104768       
	core        0              unlimited      
	rss         unlimited      unlimited      
	memlock     unlimited      unlimited      
	maxproc     266            532            
	maxfiles    20000          50000          
launchd%

When in launchctl you can update the maxfiles property by issuing the command:

limit maxfiles 20000 5000

The first value is the soft limit and the second value is the hard limit. I don’t know what the difference is, however I do know you cannot set either to unlimited. The big problem I found however is that this setting doesn’t seem to stick. To make it stick there is a config file you must change. The file is /etc/launchd.conf, and you will probably have to create it. In nano this is dead easy:

sudo nano /etc/launchd.conf

Then add the following line to the file:

limit maxfiles 20000 50000

save it and reboot. If this fails it seems to fail silently so it is worth verifying by launching launchctl and issuing the limit command.
You can check that HBase has picked up this setting by checking the master log and the regionservers log. The mater log file will have the following line, dumped at start up:

Sun 18 Dec 2011 13:32:52 GMT Starting master on max-garfinkels-computer.local
ulimit -n 20000

and the region log will have :

Sun Dec 18 13:32:53 GMT 2011 Starting regionserver on max-garfinkels-computer.local
ulimit -n 20000

with the ulimit -n xxx being the maxfiles limit we applied.

The culprit was getClass().getResource(“filename”), which returns a url. I then created a BufferedReader from a FileReader, from a File, from the url, phew.

What I think happens in that the wrong handler get used, java assumes you are trying to read a normal file from the filesystem and uses the file protocol handler, when in fact you need the jar protocol handler.

FIX

InputStream inputStream = getClass().getResourceAsStream("fileName");
reader = new BufferedReader(new InputStreamReader(inputStream));

Image image = imageIcon.getImage();

This creates an Image object which is a super class of the BufferedImage. Did a bit of reading and it seemed it should really be a BufferedImage, so all that was really necessary was to call:

ImageIO.write((BufferedImage)image, "png", file);

casting the Image to a BufferedImage. However the system balked at this irresponsible and apparently ill advised abuse of the typing system and delivered me a swift rebuke, in the form of a class cast exception. Well I guess I was asking for it, but what exactly was the problem? was it a volatile image perhaps? The other documented sub class of Image? No such luck, this was the telling off I got: java.lang.ClassCastException: apple.awt.OSXImage cannot be cast to java.awt.image.BufferedImage. Turns out on the mac (oh how I love you and hate you apple), they have their own pretty much undocumented image class OSXImage. So how to deal with this. I did a little googling and came across this. To summarise the advice was essentially:

• Get the Image’s consumer with Image.getSource().
• Use a PixelGrabber to read the image data from the consumer into some
  pixel array
• Create a DataBufferInt from the pixel data
• Create a WritableRaster using the DataBufferInt
• Create a BufferedImage using the WritableRaster
• curse, wait, curse, wait, curse, curse, curse

More than a little daunting, still I hadn’t found any thing else of any use so I gave it a crack. Documentation on this stuff was painfully thin on the ground but I got something that created an image – just not the right image. I had real trouble with understanding the bitmasks required for decoding pixels packed into ints offsets and all sorts of other things I will have to get to grips with at some point. But thankfully not now. After hours of cursing and waiting as advised in the approach quoted, I went back to google. After a while I stumbled across this, thank you dan!

Dan is the man! He out thought the problem and came up with a beautifully simple solution!

Image image = imgIcon.getImage();
int width = imgIcon.getIconWidth();
int height = imgIcon.getIconHeight();
BufferedImage bi = new BufferedImage(width, height, BufferedImage.TYPE_INT_RGB);
Graphics graphics = bi.getGraphics();
graphics.drawImage(image, 0, 0, this);
graphics.dispose();

Basically get the Image out of the icon, create a new BufferedImage. Get its graphics object and just draw our image into it! Now we have a buffered image, with our image in it. This seems like it should be a nice solution that will ensure that this image saving malarkey doesn’t fail on crazy java VM implementations where the image class is unexpectedly extended. Best of all no more, seat of the pants casting!

Thanks Dan!

The story is this:
I created a class called Landmark:

public class Landmark {
    /**
     * Key uniquely identifying landmark.
     */
    private long id;
    /**
     * Textual representation of the landmark.
     */
    private String name;
 
    /**
     * Constructor creates a landmark with id and name.
     * @param id
     * @param name
     */
    public Landmark(long id, String name) {
        this.id = id;
        this.name = name;
    }
 
    /**
     * Creates a landmark object with only an id.
     * @param id
     */
    public Landmark(long id) {
        this.id = id;
    }
....
....

All looks pretty reasonable, however notice the overloaded constructor, you can make a landmark just with the id, which happens to be really handy. Now see if you can spot the problem!

/**
     * Equality is only checked on id
     * @param obj
     * @return
     */
    @Override
    public boolean equals(Object obj) {
        if (obj == null) {
            return false;
        }
        if (getClass() != obj.getClass()) {
            return false;
        }
        final Landmark other = (Landmark) obj;
        if (this.id != other.id) {
            return false;
        }
        /*
        if ((this.name == null) ? (other.name != null) : !this.name.equals(other.name)) {
            return false;
        }*/
        return true;
    }
 
    @Override
    public int hashCode() {
        int hash = 3;
        hash = 53 * hash + (int) (this.id ^ (this.id >>> 32));
        hash = 53 * hash + (this.name != null ? this.name.hashCode() : 0);
        return hash;
    }

Ok so notice that equality was originally checked on name and id, thus is the name was missing they were not the same. You can imagine this became a real pain in the arse, because they should really be the same because they refer to the same landmark. So you can see one day in a fit of rage I simply commented out the line comparing the names and settled on equality based only on id. Spotted the issue yet?

Well being in a fit of rage I didn’t think it through very far, or look a few lines below, at the hashcode implementation. So now equality is defined as simply equality of id but hashcode is calculated from id and name! Guess what happened?

Well I was stashing some Landmarks in a hashmap as a key with some Integer values associated with them. I had a second Set of Landmarks which I needed the Integer for, so I happily try and pull out the Integer using one of my Landmarks. However the Landmark in the hashmap has a title, the Landmark I am using to pull the int doesn’t. Result? can’t retrieve a single value and get null pointer exceptions all over the shot! Why? Because the two landmarks I consider the same and that the .equals(object) method consider the same aren’t gonna be in the same place in the hashtable, so as far as the hash table is concerned, they sure aint the same!

Result? Give thought to equals() and hashcode() and remember what they do. All in all I learnt a good lesson at the expense of some real pain even if it was a programming 101 lesson

class.getResource(path)

and you find yourself banging your head against the wall because it doesn’t seem to work as it should, then you may need to do a clean build of the project as the resource may be missing from the jar. Wish I had known that before giving myself a bloody forehead and my desk a dent!

@Override
public void map(ImmutableBytesWritable key, Result value, Context context)
                                  throws IOException, InterruptedException {
 
    int frequency = Bytes.toInt(value.getValue(HBaseSchemaConstants.FREQUENCY_BYTES));
    String trail = Bytes.toString(value.getValue(HBaseSchemaConstants.TRAILSTRING_BYTES));
 
    String[] landmarks = trail.split(":");
 
    for (int i = 0; i < landmarks.length-1; i++) {
        String pair = landmarks[i] + ":" + landmarks[i+1];
        context.write(new Text(pair), new IntWritable(frequency));
    }
}

So whats the problem here? Well we have a problem with getting input in and we have a problem with verifying the output.

Input

The input is passed through a Result object which has not setters on it, so I can’t pre populate it with dummy data. This is where Mockito comes in. We create a mock of the result object and stub the getValue calls on it.

Result result = mock(Result.class);
byte[] frequencyColumnId = Bytes.toBytes("frequency");
byte[] frequency = Bytes.toBytes(12);
byte[] trailColumnId = Bytes.toBytes("trailstring");
byte[] trail = Bytes.toBytes("1:2:3:4:5:3:2:1");
when(result.getValue(frequencyColumnId)).thenReturn(frequency);
when(result.getValue(trailColumnId)).thenReturn(trail);

The first line creates the actual mock. The next four lines are just me encoding the mock values I will use in the test into byte arrays. The final two lines are the stubs for the getValue() call I will be making in the map method I am testing. These translate as “when getValue is called with an argument equal to frequencyColumnId then return (int)12″ and “when getValue is called with an argument equal to trailColumnId then return (String)1:2:3:4:5:3:2:1″. This ensures I get some mock values into my map.

Output

Now the problem of the output. In my mapper I make multiple calls to context.write(), which I need to verify, once again I can use Mockito, but this time in a slightly different way. The really cool feature of Mockito here is that it records the number of times each method was called with given arguments.

Context context = mock(Context.class);
/**
main body of test
**/
verify(context, times(1)).write(new Text("1:2"), new IntWritable(12));
verify(context, times(1)).write(new Text("2:3"), new IntWritable(12));
verify(context, times(1)).write(new Text("3:4"), new IntWritable(12));
verify(context, times(1)).write(new Text("4:5"), new IntWritable(12));
verify(context, times(1)).write(new Text("3:2"), new IntWritable(12));
verify(context, times(1)).write(new Text("2:1"), new IntWritable(12));

In this block of code we call verify(nameOfMockObject, time(numberOfTimesCalled).methodWeAreInterestedIn(expectedArgs); basically we can repeatedly call verify as many times as we have expected calls on the mock, in this case context.

Ill add more once I have done my reduce method too, but so far its looking like a cinch!

However I have a dilemma – I want experimental results quickly and will learn a lot on the way, however following a rigours test driven approach with refactoring seems to mean I am getting better software but I am also getting further from my results.

In theory (and I do believe the theory) I will end up with much better designed software through using the techniques described in the books. Coupling should be lower making the code more flexible and reusable – important in an ever changing research environment. Also I can make bigger changes with the confidence that I can do them incrementally testing that contracts are kept all along. This is all great, but I am still learning and as such the code base could do with much refactoring simply to enable more through unit testing. As it happens this will also benefit the architecture, but it will cost time.

Also as this progresses I can see that one ends up with a larger structure of code made of smaller bits, so on one hand you are making the software more modular and flexible and generally simplifying and making more readable blocks of code, but is this at the expense of a more complex overall structure?

I can only guess for now, and for now I will still be doing a degree of ugly hacking but it is feeling more and more wrong.

Speaking to other researchers some have told me of there regret for not starting out with unit testing and well engineered code, whilst others have completed plenty of high quality published research on the back of throw-away, proof of concept hacking, and have no real experience of a more engineering oriented approach.

I suspect there isn’t a black and white answer and it depends on the purpose of your code, like most things it is probably generally a nuanced and slightly subjective call based on the details of your requirements.

I will try and update this post with more reflections on this problem as I make progress.

The goal was to move all my local files that used to belong to the previous repository to a new repository hosted on my mediatemple server. Then get netbeans talking to it, and, hopefully have a clue how it all works. This proved to be a complete nightmare and so before I forget all the things I learned I will try and get down as many of them as possible here.

General Points

• Subversion allows for repositories at the top level with multiple projects under that. This is what I have done.
• The svn server stores everything in a folder I named svn – you don’t really want to go looking around it as far as I can tell.
• Items have an address within the repository that looks something like this:

  protocol://host.com/folder/to/the/repository/svn/myRepo

  The path needs to be from the root of the host and the protocol can be several things depending on where you are accessing from. When accessing locally, i.e. you are on the same machine that hosts the repository you should use something akin to this (notice that there is no host, we don’t need it because we are on the machine in question).

  file:///home/max/svn/myRepo

  If however you are accessing a mediatemple hosted subversion repository you need to access it via svn+ssh which is the svn protocol tunnelled through ssh. In this case the path would look like this:

  svn+ssh://myhostname.com/home/1234/users/max/svn/mtSvnRepo1

  or

  svn+ssh://myusername@myhostname.com/home/1234/users/max/svn/mtSvnRepo1

  The only difference being the inclusion of the username that you wish to use with the host (your username for logging into the machine hosting the repository). More about that later. Now this is just the path to the repository, your projects and the files within them are then just bolted on the end thus.

  svn+ssh://myhostname.com/home/1234/users/max/svn/mtSvnRepo1/project1/src/someFile.java

Some Commands

• Create a repository

  svnadmin create --fs-type fsfs

• Import some files into the repository

  svn import /just/a/folder svn+ssh://myhostname.com/home/1234/users/max/svn/mtSvnRepo1 -m "a test of adding a new project remotly"

  the -m flag is the commit message which is stored with the revision number

• List the projects under svn control

  svn list --verbose svn+ssh://myhostname.com/home/1234/users/max/svn/mtSvnRepo1
        2 maxgarfi              Jan 10 15:12 RAVisualisationClient-01/
        4 maxgarfi              Jan 10 15:20 RAVisualisationDataAccess-01/
       18 maxgarfi              Jan 10 18:30 RAVisualisationEngine-01/
        8 maxgarfi              Jan 10 15:32 RAVisualisationGUI-01/
       10 maxgarfi              Jan 10 15:37 RAVisualisationLib-01/
       12 maxgarfi              Jan 10 15:49 RAVisualisationServer-01/
       19 maxgarfi              Jan 10 19:00 librarys/

• Checkout or co. Unlike Team Foundation or Source Safe on windows this doesn’t lock any files it just gets the latest copy from the repository.
• Add – adds a new file to the repository.
• Delete – removes the file from the repository (sort of) and only removes it from the repo when you commit.
• Update – pulls down any changes to the files in the repo and tries to merge them with the local files on your machine.
• Commit  - submits your changed files to the repo and gives them a revision number.

Removing Files From Subversion Control

No this really caught me out. When you associate files with a subversion repository the subversion client on your local machine peppers the folders of your project with hidden .svn folders. This is how it knows what is what and keeps track of stuff. BUT, and this is a big but, if you end up with a dead/no longer accessible repository and you want to move you local files so they are looked after by a new repository, then your in trouble. Its worth noting that at this stage you have lost all the previous history of your files – that went with the previous repository, but hey at least you have your files.

Try just adding these files to a new repository? It aint gonna work – subversion will tell you that they are being looked after by a now non existent repository. Took me a long time to find it but there is a subversion command that “exports” the files in your projects without all the hidden .svn files, and its called export:

 svn export /path/to/old/project/ path/to/new/version/

Great! just one little snag to look out for: if there were files in your project that you hadn’t added to the repo before it all went tits up export wont copy them over – you might need to hunt them down manually and copy them over to your new location.

SSH details

To get netbeans working with svn+ssh there was a fair bit of fafing around, there is a good page on the netbeans site. You also need to set up passwordless ssh access to the subversion server and possible create a .ssh/config file where you can specify the username to use when ssh’ing to that host.

Links

• The SVN book
• Netbeans svn+ssh configuration
• Passwordless ssh
• If you lock yourself out of your server by getting your passwordless ssh set up wrong this post will get you out of trouble.
• mediatemple articles on configuring svn and passwordless ssh