Web Mining 2007-2008

Mauro Brunato, Elisa Cilia

Seventh weekly assignment: MapReduce

The purpose of this assignment is to implement a document indexing feature in a MapReduce framework.

Instructions

• Use the Hadoop framework.
  • If you work on the lab computers, then the framework is installed in /home/brunato. In such case, the following environment variables must be set:

    HADOOP_VERSION = 0.16.3
    HADOOP_HOME = /home/brunato/hadoop-$HADOOP_VERSION
    add $HADOOP_HOME/bin to PATH
    

    If everything is set up properly, the command hadoop should be available.
  • Installing the framework in your computer is easy, however: just download the latest release, expand it in any directory and set the environment variables.
• Follow the tutorial (in doc/index.html within $HADOOP_HOME).
• Write a Hadoop class that solves the following problem:
  Given a set of text documents in the input directory, create an index that associates every term t with the document that contains the highest number of occurrences of t.
  For example, if document doc1 is "alpha alpha beta alpha gamma beta" and document doc2 is "beta gamma beta gamma alpha", the resulting index should look like

  alpha   doc1
  beta    doc2
  gamma   doc2
  

  Any document choice is fine in case of tie (beta, in the example).