Writing a map reduce program

When I installed Hadoop barely a month ago it was at version 0.20.1, as of writing this, the stable release is 0.20.2.

Unfortunately the official hadoop tutorial at apache is already out of date and will show the use of depreciated calls. Thankfully Yi Wang has updated it for 0.20.1.

Overview of a map reduce program

In Yi and the apache tutorial the entire map reduce program is contained within a single outer class. Within this there are two inner classes, one for handling the map and one for handling the reduce stage. There is also a main in the container class, this main function seems to contain all the job configuration options. These seem to be:

The Mapper

In these examples the mapper class is a public static inner class which extends the hadoop core Mapper object located in org.apache.hadoop.mapreduce.Mapper. This class is generic and so you pass it some type information at the class declaration

public static class Map extends Mapper <Object, Text, Text, IntWritable>{

According to the javadoc the type parameters are: the input keys type, the input values type, the output keys type and the output values type.

The mapper class (Map) doesn’t have a constructor but simply overrides the Mapper method map

@Override
public void map(Object key, Text value, Context context)
        throws IOException, InterruptedException {

The Map.map() method takes a key and a value typed similarly to the Mapper<ik,iv,ok,ov>. It also takes an argument of type Context. The Context type is an inner class of Mappable and it seems to be where you pass back output key-values for passing on to the reduce stage. So in this example we are contracted to return a Text key and a IntWriteable value so the end section of  the map method would roughly look like this:

Text t = new Text();
t.set("some-text-based-key");
IntWritable i = new IntWriteable(1);
context.write(t, i);

So to summarise: the mapper class extends Mapper and implements its own map method. The map method takes a key and a value as well as a context. You process your keys and values in whatever way your mapping phase requires and output the intermediate keys and values via the context object using the write(k,v) method.

The Reducer

Much the same as the mapper, the reducer is a public static inner class and it extends Reducer<keyin, valuein, keyout, valueout>. As you can see Reducer has its types parameterized too. The reducer class (our one) overrides Reducer.reduce(Object, Iterable, Context). Before attempting to explain any further I want to quickly backtrack to what will happen when this is run.

First up, of course, the mapper is called, does all its funky mapping and spits out a bunch of keys and values. So when all of the mappers have finished the mapping and there output is sorted on the key, the reducer phase gets to start. The framework then starts moving the mappers output over to the reducer. The will entail collecting up all the keys that are the same across the whole cluster and merge sorting them into a file for the reducer to ingest.

In this example we have 3 possible key values a, b and c. After each mapper has run all the values keyed on a get copied to the green intermediary file, they are also sorted during this process so the green intermediary file ends up with all the a values in order. The same happens for the b values, which end up in the red file and c values which end up in the blue/green file at the bottom. In this case each reducer then just deals with one intermediary file reducing down all the a values into a final output file. To contextualise it, the word count example would fill the green file with a stack of keys and values that would all be hello 1. The reducer stage would pick up that file, run through it summing all the values for that key. Thus the output file would contain the key, hello, and the sum of the intermediary values i.e. the number of occurrences of  hello.

Given this brief detour we can now say a little more about the arguments for the reduce method. The first argument is the key, its type being the type we specified when extending Reduce<ik, iv, ok, ov>. The next argument is an Iterator type structure of the type we specified for the input values when extending the reduce. If we look back at the detour this should make a little more sense as what we are getting is a bunch of values that have a shared key. Thus the argument for this method gives us a key and the bunch of values associated with it. The final argument, like with the mapper, is a context object and this is what we write are reduced key, value pairs back to, to produce the final output.

Compiling

So I knocked up this code in netbeans and hit clean and build, ran through all the steps I’m about to describe and simply didn’t work. I kept getting

Exception in thread…
Caused by: java.util.zip.ZipException: error in opening zip file
…

So the only lesson I could draw is that I need to compile it manually to get it to work. Before compiling the .java file I added some variables to my bash.rc file to make some of it a little easier:

#hadoop path
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_VERSION=0.20.1
export PATH=$PATH:/usr/local/hadoop/bin

Now to compile the .java file for hadoop

1. Copy the .java file over to my hadoop box (not running it locally yet)
2. In the same folder as .java file create a folder called classes (mkdir classes)
3. Then call the compiler

javac -classpath ${HADOOP_HOME}/hadoop-${HADOOP_VERSION}-core.jar:commons-cli-1.2.jar -d classes UrlCollector.java && jar -cvf urlcollector.jar -C classes/ .

So lets dissect this call bit by bit.

• javac – thats the java compiler.
• -classpath – here we give it a path to the libraries, and here we use some of the variables we configured in the bash.rc file. we are using the variables to basically write out the path to hadoop core jar file, which on my machine is /usr/local/hadoop/hadoop-0.20.1-core.jar.
• :someOtherJar – we separate other libraries with a colon, in this case the apache commons cli library, which seems to be a dependancy, not sure about that yet though.
• -d classes- implies that we are providing a folder for all the class files to go in (remember step two?)
• UrlCollector.java - that was the name of the java file I needed to compile – the one containing the map and reduce functions as well as the main method.
• Not totally sure about the rest except it seems to specify we want a jar and its name (urlcollector.jar). But all importantly DO NOT forget the period (.) at the end. Without that seemingly insignificant little dot it wont work.

Great so now we have a jar file that should work and we have some input files – in the case of the word count tutorial this is a couple of files with some words in them!

Running the job

So we have all our bits and pieces and we can now get ready to run this job. The first thing to note is that we must copy our input files into the hadoop file system but we leave our jar on the host file system. Now that we have set the path to the hadoop bin we can call the hadoop command more easily. Lets copy up the source files

hadoop dfs -copyFromLocal /local/data /hadoop/fs/location/

Now we are ready for the magic, lets run the jar

hadoop jar urlcollector.jar com.maxgarfinkel.mapReduce.UrlCollector /user/hadoop/urlCol/in /user/hadoop/urlCol/out

Lets break this down.

• hadoop jar - We call jar which is basically the run this jar command.
• urlcollector.jar – Thats the name of the jar I want to run (include the path if its not in your working directory)
• com.maxgarfinkel.mapReduce.UrlCollector - That was the package name I had my UrlCollector.java class in and the class name.
• /user/hadoop/urlCol/in - that is the hdfs path to the folder containing the input data we copied up in the previous step.
• /user/hadoop/urlCol/out – this is the hdfs path to the folder where I want the output to be put. I haven’t created this folder, I will let hadoop take care of that.

So this is it written out in a more generic manner

hadoop jar local/path/to/jar full.package.and.class.name hdfs/source/data/location hdfs/output/location

So once our job has run we can pick up our output using the hadoop dfs command copyToLocal.

for(Node n : nodes){
    //do some stuff
    nodes.remove(n);//This will throw the error
}

It makes perfect sense if you think about it. You are stepping over a list, no doubt under the hood somewhere a counter is being used. When you remove a element from a list, all the other elements are shuffled down so now your iterator is going to miss an element and shoot off the end of the list, so instead of that it fails fast and throws an exception. However, more importantly than why it happens, how can we stop it happening? Well we use a list iterator instead.

ListIterator<Node> i = nodes.listIterator();
while(li.hasNext()){
    Node n = i.next;
    li.remove(); //No error if you remove the element through the iterator
}

There is a great addon for firefox called SQLite Manager which allows you to run SQLite queries. Using this you can look at the places.sqlite file, which contains your web history data. I am particulalry interested in looking at the paths I take and the following query will pull out pairs of target->referer pages.

SELECT plc2.url AS 'referer', plc1.url AS 'target'
FROM moz_historyvisits vis, moz_historyvisits vis2
INNER JOIN moz_places plc1
ON plc1.id = vis.place_id
INNER JOIN moz_places plc2
ON plc2.id = vis2.place_id
WHERE vis.from_visit = vis2.id

There is a wealth of information in this little file, the visit_type column, for example gives you an indicator of how the URL was reached.

For more information on this you can view the Mozilla dev pages and the full places system within firefox is documented over here.

A wealth of further information on this DB is available over at firefox forensics

• Installing Hadoop on Ubuntu linux, an excellent tutorial by Michael  Noll.
• Installing Hadoop on Mac OS X , I tried this on snow leopard on a power book and it worked fine with the following caveats – Check it against Michaels tutorial and configure core-site.xml(referred to as hadoop-site.xml), mapred-site.xml, and hdfs-site.xml. I forgot the last two and only hte data node and secondary name node started – remember you can use the jps command from the shell to see what java processes are running.

A list of useful hadoop related commands that I keep forgeting

• Starting core as hadoop user: $<HADOOP INSTALL DIR>/bin/start-all.sh
• Stopping core as hadoop user: $<HADOOP INSTALL DIR>/bin/stop-all.sh
• Starting Zoopkeeper: $<ZOOKEEPER INSTALL DIR>/bin/zkServer.sh start
• Stopping Zoopkeeper: $<ZOOKEEPER INSTALL DIR>/bin/zkServer.sh stop
• Starting HBase: $<HBASE INSTALL DIR>/bin/start-hbase.sh
• Stopping HBase: $<HBASE INSTALL DIR>/bin/stop-hbase.sh
• Useful command to see what is running: $jps
• (Re)format hdfs <HADOOP INSTALL DIR>/bin/hadoop namenode -format

Important paths

• Hadoop install path: /usr/local/hadoop
• web UI for MapReduce job tracker(s)
• web UI for task tracker(s)
• Web UI HDFS Name Node

Hadoop class path

Ran into problems today trying to run a jar in Hadoop that needed to reference the hbase jar and zookeeper jar. To get it to work I had to add some changes to the hadoop-env.sh. The changes were in the line

# Extra Java CLASSPATH elements.  Optional

and the changes were

export HADOOP_CLASSPATH=/usr/local/hbase/hbase-0.20.2.jar:/usr/local/zookeeper/zookeeper-3.2.2.jar

which is basically adding the path to the hbase jar and the zookeeper jar

Building a Hadoop compatible jar from netbeans

Right – couldn’t build a jar that would work with hadoop from netbeans 6.7. After consulting the most knowledgeable java developer I know, we got to the bottom of it, sort of. It turns out the default manifest that netbeans creates seems to cause issues – all sorts of inexplicable errors infact. So the solution was to simply delete the manifest.mf from the root of the project. Phew. Might try and look into it further at some point, still working on a work flow based on netbeans.

Important port numbers

zookeeper client port = 2181

Hbase master port = 60000

Fantastic list of algorithmns and computer science problems. Very usefull