Simple calculation using apache spark -

-

i have javapairrdd (string, tuple2) out of join operation. below data detail - [userid, [(name, rating)]]

output: [(user2,[(john,5)]), (user3,[(mac,3), (mac,2)]), (user1,[(phil,3), (phil,4)])]

i want calculate min, max , average each user. not sure transformation/action can me here.

there multiple ways achieve this, easiest use aggregatebykey method.

here example shows how it

public static void main(string[] args) {     sparkconf conf = new sparkconf().setappname("simple application").setmaster("local[4]");     javasparkcontext sc = new javasparkcontext(conf);      string[] values = {"user2:john", "user1:phil", "user3:mac"};     string[] ratingvalues = {"user2:5", "user3:3", "user3:2", "user1:3", "user1:4"};      javardd<string> users = sc.parallelize(arrays.aslist(values));     javardd<string> ratings = sc.parallelize(arrays.aslist(ratingvalues));      javapairrdd<string, string> userspair = users.maptopair(             new pairfunction<string, string, string>() {                 public tuple2<string, string> call(string s) throws exception {                     string[] splits = s.split(":");                     return new tuple2<string, string>(splits[0], splits[1]);                 }             }     );      javapairrdd<string, integer> ratingspair = ratings.maptopair(             new pairfunction<string, string, integer>() {                 public tuple2<string, integer> call(string s) throws exception {                     string[] splits = s.split(":");                     return new tuple2<string, integer>(splits[0], integer.parseint(splits[1]));                 }             }     );      javapairrdd<string, tuple2<string, integer>> joined = userspair.join(ratingspair);      javapairrdd<string, tuple4<integer, integer, integer, integer>> aggregate = joined.aggregatebykey(new tuple4<integer, integer, integer, integer>(integer.min_value, integer.max_value, 0, 0),             new function2<tuple4<integer, integer, integer, integer>, tuple2<string, integer>, tuple4<integer, integer, integer, integer>>() {         public tuple4<integer, integer, integer, integer> call(tuple4<integer, integer, integer, integer> a, tuple2<string, integer> b) throws exception {             return new tuple4<integer, integer, integer, integer>(math.max(a._1(), b._2()), math.min(a._2(), b._2()), a._3() + b._2(), a._4() + 1);         }     }, new function2<tuple4<integer, integer, integer, integer>, tuple4<integer, integer, integer, integer>, tuple4<integer, integer, integer, integer>>() {         public tuple4<integer, integer, integer, integer> call(tuple4<integer, integer, integer, integer> a, tuple4<integer, integer, integer, integer> b) throws exception {             return new tuple4<integer, integer, integer, integer>(math.max(a._1(), b._1()), math.min(a._2(), b._2()), a._3() + b._3(), a._4() + b._4());         }     });      javardd<tuple2<string, tuple3<integer, integer, double>>> aggregatewithmean = aggregate.map(             new function<tuple2<string,tuple4<integer,integer,integer,integer>>, tuple2<string, tuple3<integer, integer, double>>>() {                 public tuple2<string, tuple3<integer, integer, double>> call(tuple2<string, tuple4<integer, integer, integer, integer>> a) throws exception {                     tuple3<integer, integer, double> mean = new tuple3<integer, integer, double>(a._2()._1(), a._2()._2(), a._2()._3().doublevalue()/a._2()._4());                     return new tuple2<string, tuple3<integer, integer, double>>(a._1(), mean);                 }             }     );      aggregatewithmean.foreach(new voidfunction<tuple2<string, tuple3<integer, integer, double>>>() {         public void call(tuple2<string, tuple3<integer, integer, double>> stringtuple3tuple2) throws exception {             system.out.println(stringtuple3tuple2);         }     }); }

with java api becomes messy because of generic types. i'd recommend use scala api instead. looks nicer :-)


Comments

Post a Comment

Popular posts from this blog

Fail to load namespace Spring Security http://www.springframework.org/security/tags -

-
i have problem namespace of spring security on xhtml page. don't understand why can't load uri, , have following issue: whitelabel error page application has no explicit mapping /error, seeing fallback. tue jul 21 16:58:12 utc 2015 there unexpected error (type=not found, status=404). please need help! possible uri has been changed? best regards you using incorrect namespace url. correct url http://www.springframework.org/schema/security the error page seeing spring source website since not recognize url. try visiting http://www.springsource.org/security/tags in browser update : spring security tags used in jsp page correct tag lib declaration is: <%@ taglib prefix="sec" uri="http://www.springframework.org/security/tags" %>
Read more

sql - MySQL query optimization using coalesce -

-
can explain me why second query faster first? (first 1 takes around 40 seconds, second 1 less 0.1) select releases.feature_code, releases.platform_id, version, release_date, general_available, uri, filesize, display_name, feature_name.full_name, if ((select exists(select 1 release_exceptions releases.feature_code = release_exceptions.feature_code , releases.platform_id = release_exceptions.platform_id , releases.version = release_exceptions.version) = 1), 1, 0) release_exceptions releases left outer join platform using(platform_id) left outer join feature_name using(feature_code) order version desc; ~~~ query 2 ------------------ ~~~ select feature_code, version, release_date, general_available, platform_id, uri, filesize, display_name, feature_name.full_name, coalesce(release_exceptions, 0) release_exceptions mus.releases left outer join ( select distinct feature_code, version, platform_id, 1 release_exceptions mus.release_exceptions ) release_exceptions using ...
Read more

unity3d - Unity local avoidance in user created world -

-
i'm using unity3d networked multiplayer online game have large complex 3d terrain scene forest, trees, cliff, hills, mountains, bounders, etc. players can build structures sort of minecraft, , put them anywhere in scene, or move them around anytime. aside human controlled players, there automated ai players , objects animals roaming around scene following path. the problem how make these automated ai players , animals, able navigate around static , dynamic player created structures, because path follow can blocked player created structures, or other players , other ai objects, cliffs etc. have find away around them or on track if tumble down off high cliff example. so made navmesh , used navagents, takes care of static, non moving objects, how make ai players navigate around each other , dynamic structures created players can number in hundreds? i thought of adding navmeshobstacle everything, result in being attached hundreds of objects since user created structures b...
Read more