2013年8月10日星期六

DistributedCache used when the client can not find the file in themap

Use DistributedCache,

job -side configuration is as follows
DistributedCache.createSymlink (conf);
DistributedCache.addCacheFile (new URI ("hdfs :/ / namenode: 9000/hadoop/testFile # testFile"), conf);

end in the map is open

FileReader fr = new FileReader ("testFile");

problems is the file can not be found , java.io.FileNotFoundException testFile (no such file for directory)

experts who urged that where the problem ah ?
------ Solution ---------------------------------------- ----
ding, encountered the same problem
------ Solution --------------------------- -----------------
I have not used symlink, but I have a solution, because new FileReader () is hdfs default should read the document, but distributed cache file is local, need to add file :/ /
See http://hugh-wangp.iteye.com/blog/1468989
------ For reference only ----------------- ----------------------
some people say that a solution should be written in accordance with the following method , but still does not work ,
or have master to a use distributedCache example , not found too depressed

new File ("/ home/ayush/output/part-00000"). toURI ()
------ For reference only ------------ ---------------------------
found the information that is to use the symlink methods often do not immediately produce a symbolic link , often appear fileNotFound problems, so I do not recommend using this approach
------ For reference only ------------------------ ---------------
problem lies in the way Mapper read files using ordinary mode or manner of HDFS

没有评论:

发表评论