Hadoop Configuration in detail

This blog is aimed at describing how to make changes to your hadoop configuration options on gateway/blades from where mapred jobs are submitted. The options that sometimes need to be changed while configuring could be name-node directory, hadoop tmp dir, hdfs data directory, secondary name-node directory etc.
Following steps explain this process in detail. These steps were tried on CDH3b4 single-user set-up:

i)                    Login as root:

# sudo su

ii)                   Stop all the daemons:

# for x in /etc/init.d/hadoop-0.20-*; do $x stop; done

iii)                 if fs.name.dir changed,

a.        create fs.name.dir (say /data/nn1)

                                                               i.      # mkdir /data/nn1

                                                             ii.      # chown hdfs /data/nn1

                                                            iii.      # chgrp hdfs /data/nn1

b.      format namenode:

                                                               i.      # su – hdfs

                                                             ii.      $ hadoop namenode –format

                                                            iii.      $ exit (exit hdfs)

                                                           iv.       

iv)                 if hadoop.tmp.dir changed (say to /data/hadoop-0.20/cache/$(user.name), then do following

a)      # mkdir –p /data/hadoop-0.20/cache

b)      # chmod 777 /data/hadoop-0.20/cache

v)                  if dfs.data.dir is changed (say /data/hdfs), do following:

a.       # mkdir /data/hdfs

b.      # chown hdfs /data/hdfs

c.       # chgrp /data/hdfs

vi)                 Re-start everything:

# for x in /etc/init.d/hadoop-0.20-*; do $x start; done

vii)               Exit root:

# exit

viii)              Try example pi mapred job and it shud work:

$ hadoop jar /usr/lib/hadoop-0.20/hadoop-examples-0.20.2+737.jar pi 4 1000