java - Error while importing data from MongoDb to Hdfs -

-

i'm trying import documents of collection in mongodb hdfs through mapreduce job. using old api. driver code 

package my.pac;    import org.apache.hadoop.conf.configured;  import org.apache.hadoop.fs.path;  import org.apache.hadoop.io.text;  import org.apache.hadoop.mapred.fileoutputformat;  import org.apache.hadoop.mapred.jobclient;  import org.apache.hadoop.mapred.jobconf;  import org.apache.hadoop.mapred.textoutputformat;  import org.apache.hadoop.util.tool;  import org.apache.hadoop.util.toolrunner;  import com.mongodb.hadoop.mapred.mongoinputformat;  import com.mongodb.hadoop.util.mongoconfigutil;    public class importdriver extends configured implements tool {    	public static void main(string[] args) throws exception {  		int exitcode = toolrunner.run(new importdriver(), args);  		system.exit(exitcode);  	}  	@override  	public int run(string[] args) throws exception {  		jobconf conf = new jobconf();    mongoconfigutil.setinputuri(conf,"mongodb://127.0.0.1:27017/sampledb.shows");          conf.setjarbyclass(importdriver.class);  		conf.addresource(new path("/usr/lib/hadoop/hadoop-1.2.1/conf/core-site.xml"));  		conf.addresource(new path("/usr/lib/hadoop/hadoop-1.2.1/conf/hdfs-site.xml"));  		fileoutputformat.setoutputpath(conf, new path(args[0]));  		conf.setinputformat(mongoinputformat.class);  		conf.setoutputformat(textoutputformat.class);  		conf.setmapperclass(importmapper.class);  		conf.setmapoutputkeyclass(text.class);  		conf.setmapoutputkeyclass(text.class);  		jobclient.runjob(conf);  		return 0;  	}  }

this mapper code:

package my.pac;    import java.io.ioexception;  import org.apache.hadoop.io.text;  import org.apache.hadoop.mapred.mapreducebase;  import org.apache.hadoop.mapred.mapper;  import org.apache.hadoop.mapred.outputcollector;  import org.apache.hadoop.mapred.reporter;  import org.bson.bsonobject;  import com.mongodb.hadoop.io.bsonwritable;    public class importmapper extends mapreducebase implements mapper<bsonwritable, bsonwritable, text, text>{    	@override  	public void map(bsonwritable key, bsonwritable value,  			outputcollector<text, text> o, reporter arg3)  			throws ioexception {  		string val = ((bsonobject) value).get("_id").tostring();  		system.out.println(val);  		  		o.collect( new text(val), new text(val));  		  	}    }

i using

  1. ubuntu-14.0
  2. hadoop-1.2.1
  3. mongodb-3.0.4

i have added following jars:

  1. mongo-2.9.3.jar
  2. mongo-hadoop-core-1.3.0.jar
  3. mongo-java-driver-2.13.2.jar

when run this, getting error :

java.lang.exception: java.lang.classcastexception: com.mongodb.hadoop.io.bsonwritable cannot cast org.bson.bsonobject  	at org.apache.hadoop.mapred.localjobrunner$job.run(localjobrunner.java:354)  caused by: java.lang.classcastexception: com.mongodb.hadoop.io.bsonwritable cannot cast org.bson.bsonobject  	at my.pac.importmapper.map(importmapper.java:18)  	at my.pac.importmapper.map(importmapper.java:1)  	at org.apache.hadoop.mapred.maprunner.run(maprunner.java:50)  	at org.apache.hadoop.mapred.maptask.runoldmapper(maptask.java:430)  	at org.apache.hadoop.mapred.maptask.run(maptask.java:366)  	at org.apache.hadoop.mapred.localjobrunner$job$maptaskrunnable.run(localjobrunner.java:223)  	at java.util.concurrent.executors$runnableadapter.call(executors.java:511)  	at java.util.concurrent.futuretask.run(futuretask.java:266)  	at java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor.java:1142)  	at java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:617)  	at java.lang.thread.run(thread.java:745)

how can rectify this?

you may have outdated driver in classpath that's causing conflict in read preference settings.

see below links similar issues: https://jira.mongodb.org/browse/java-849

https://serverfault.com/questions/268953/mongodb-java-r2-5-3-nosuchmethoderror-on-dbcollection-savedbobject-in-tomca

if doesn't help, https://jira.talendforge.org/browse/tbd-1002 suggests may need re-run mongodb or use separate connection.


Comments

Post a Comment

Popular posts from this blog

Fail to load namespace Spring Security http://www.springframework.org/security/tags -

-
i have problem namespace of spring security on xhtml page. don't understand why can't load uri, , have following issue: whitelabel error page application has no explicit mapping /error, seeing fallback. tue jul 21 16:58:12 utc 2015 there unexpected error (type=not found, status=404). please need help! possible uri has been changed? best regards you using incorrect namespace url. correct url http://www.springframework.org/schema/security the error page seeing spring source website since not recognize url. try visiting http://www.springsource.org/security/tags in browser update : spring security tags used in jsp page correct tag lib declaration is: <%@ taglib prefix="sec" uri="http://www.springframework.org/security/tags" %>
Read more

sql - MySQL query optimization using coalesce -

-
can explain me why second query faster first? (first 1 takes around 40 seconds, second 1 less 0.1) select releases.feature_code, releases.platform_id, version, release_date, general_available, uri, filesize, display_name, feature_name.full_name, if ((select exists(select 1 release_exceptions releases.feature_code = release_exceptions.feature_code , releases.platform_id = release_exceptions.platform_id , releases.version = release_exceptions.version) = 1), 1, 0) release_exceptions releases left outer join platform using(platform_id) left outer join feature_name using(feature_code) order version desc; ~~~ query 2 ------------------ ~~~ select feature_code, version, release_date, general_available, platform_id, uri, filesize, display_name, feature_name.full_name, coalesce(release_exceptions, 0) release_exceptions mus.releases left outer join ( select distinct feature_code, version, platform_id, 1 release_exceptions mus.release_exceptions ) release_exceptions using ...
Read more

unity3d - Unity local avoidance in user created world -

-
i'm using unity3d networked multiplayer online game have large complex 3d terrain scene forest, trees, cliff, hills, mountains, bounders, etc. players can build structures sort of minecraft, , put them anywhere in scene, or move them around anytime. aside human controlled players, there automated ai players , objects animals roaming around scene following path. the problem how make these automated ai players , animals, able navigate around static , dynamic player created structures, because path follow can blocked player created structures, or other players , other ai objects, cliffs etc. have find away around them or on track if tumble down off high cliff example. so made navmesh , used navagents, takes care of static, non moving objects, how make ai players navigate around each other , dynamic structures created players can number in hundreds? i thought of adding navmeshobstacle everything, result in being attached hundreds of objects since user created structures b...
Read more