spark-scala: groupBy throws java.lang.ArrayIndexOutOfBoundsException (null) -

-

i started following rdd has string

((26468ede20e38,239394055),(7665710590658745,-414963169),0,f1,1420276980302) ((26468ede20e38,239394055),(8016905020647641,183812619),1,f4,1420347885727) ((26468ede20e38,239394055),(6633110906332136,294201185),1,f2,1420398323110) ((26468ede20e38,239394055),(6633110906332136,294201185),0,f1,1420451687525) ((26468ede20e38,239394055),(7722056727387069,1396896294),0,f1,1420537469065) ((26468ede20e38,239394055),(7722056727387069,1396896294),1,f1,1420623297340) ((26468ede20e38,239394055),(8045651092287275,-4814845),0,f1,1420720722185) ((26468ede20e38,239394055),(5170029699836178,-1332814297),0,f2,1420750531018) ((26468ede20e38,239394055),(7722056727387069,1396896294),0,f1,1420807545137) ((26468ede20e38,239394055),(4784119468604853,1287554938),0,f1,1421050087824)

just give high level view on description of data. can think first element in main tuple (first tuple) user identification, second tuple product identification, , third element user's preference on product. (for future reference going mark above data set val userdata)

here how construct userdata:

val userdata = data.map(x => {   val userid = x._1.replace("_","")   val programid = x._2   val feature = somemethod(x._5, x._4)   val userhashtuple = (userid, userid.hashcode)   val programhashtuple = (programid, programid.hashcode)   (userhashtuple, programhashtuple, x._3, timefeature, x._4) })

my goal if user has casted both positive (1) , negative (0) preference product take record positive. example:

((26468ede20e38,239394055),(7722056727387069,1396896294),0,f1,1420537469065) ((26468ede20e38,239394055),(7722056727387069,1396896294),1,f1,1420623297340)

i want keep

((26468ede20e38,239394055),(7722056727387069,1396896294),1,f1,1420623297340)

based on provided answer followed following:

val grpdata = userdata.groupby(x => (x._1, x._2)).mapvalues(_.maxby(_._3)).values

but getting error @ stage of groupby. java.lang.arrayindexoutofboundsexception (null) [duplicate 1] , keep repeating same error while increasing duplicate number. exception doesn't point code having tough time figuring out going on.

at end following:

 java.lang.arrayindexoutofboundsexception  driver stacktrace:         @ org.apache.spark.scheduler.dagscheduler.org$apache$spark$scheduler$dagscheduler$$failjobandindependentstages(dagscheduler.scala:1203)         @ org.apache.spark.scheduler.dagscheduler$$anonfun$abortstage$1.apply(dagscheduler.scala:1192)         @ org.apache.spark.scheduler.dagscheduler$$anonfun$abortstage$1.apply(dagscheduler.scala:1191)         @ scala.collection.mutable.resizablearray$class.foreach(resizablearray.scala:59)         @ scala.collection.mutable.arraybuffer.foreach(arraybuffer.scala:47)         @ org.apache.spark.scheduler.dagscheduler.abortstage(dagscheduler.scala:1191)         @ org.apache.spark.scheduler.dagscheduler$$anonfun$handletasksetfailed$1.apply(dagscheduler.scala:693)         @ org.apache.spark.scheduler.dagscheduler$$anonfun$handletasksetfailed$1.apply(dagscheduler.scala:693)         @ scala.option.foreach(option.scala:236)         @ org.apache.spark.scheduler.dagscheduler.handletasksetfailed(dagscheduler.scala:693)         @ org.apache.spark.scheduler.dagschedulereventprocessloop.onreceive(dagscheduler.scala:1393)         @ org.apache.spark.scheduler.dagschedulereventprocessloop.onreceive(dagscheduler.scala:1354)         @ org.apache.spark.util.eventloop$$anon$1.run(eventloop.scala:48)

does mean key empty? if key empty still carry out groupby "". bit confuse.

after digging around found following: https://issues.apache.org/jira/browse/spark-6772

but don't think issue same thats got.


Comments

Post a Comment

Popular posts from this blog

Fail to load namespace Spring Security http://www.springframework.org/security/tags -

-
i have problem namespace of spring security on xhtml page. don't understand why can't load uri, , have following issue: whitelabel error page application has no explicit mapping /error, seeing fallback. tue jul 21 16:58:12 utc 2015 there unexpected error (type=not found, status=404). please need help! possible uri has been changed? best regards you using incorrect namespace url. correct url http://www.springframework.org/schema/security the error page seeing spring source website since not recognize url. try visiting http://www.springsource.org/security/tags in browser update : spring security tags used in jsp page correct tag lib declaration is: <%@ taglib prefix="sec" uri="http://www.springframework.org/security/tags" %>
Read more

sql - MySQL query optimization using coalesce -

-
can explain me why second query faster first? (first 1 takes around 40 seconds, second 1 less 0.1) select releases.feature_code, releases.platform_id, version, release_date, general_available, uri, filesize, display_name, feature_name.full_name, if ((select exists(select 1 release_exceptions releases.feature_code = release_exceptions.feature_code , releases.platform_id = release_exceptions.platform_id , releases.version = release_exceptions.version) = 1), 1, 0) release_exceptions releases left outer join platform using(platform_id) left outer join feature_name using(feature_code) order version desc; ~~~ query 2 ------------------ ~~~ select feature_code, version, release_date, general_available, platform_id, uri, filesize, display_name, feature_name.full_name, coalesce(release_exceptions, 0) release_exceptions mus.releases left outer join ( select distinct feature_code, version, platform_id, 1 release_exceptions mus.release_exceptions ) release_exceptions using ...
Read more

unity3d - Unity local avoidance in user created world -

-
i'm using unity3d networked multiplayer online game have large complex 3d terrain scene forest, trees, cliff, hills, mountains, bounders, etc. players can build structures sort of minecraft, , put them anywhere in scene, or move them around anytime. aside human controlled players, there automated ai players , objects animals roaming around scene following path. the problem how make these automated ai players , animals, able navigate around static , dynamic player created structures, because path follow can blocked player created structures, or other players , other ai objects, cliffs etc. have find away around them or on track if tumble down off high cliff example. so made navmesh , used navagents, takes care of static, non moving objects, how make ai players navigate around each other , dynamic structures created players can number in hundreds? i thought of adding navmeshobstacle everything, result in being attached hundreds of objects since user created structures b...
Read more