Scala Spark App submitted to yarn-cluster and unregistered with SUCCEEDED without doing anything -

-

goal

run our scala spark app jar on yarn-cluster mode. works standalone cluster mode , yarn-client, reason not run completion yarn-cluster mode.

details

the last portion of code seems execute on assigning initial value dataframe when reading input file. looks not after that. none of logs abnormal , there no warns or errors either. gets unregistered status succeeded , gets killed. on other deployment mode (eg. yarn-client, standalone cluster mode) runs smoothly completion. 

15/07/22 15:57:00 info yarn.applicationmaster: unregistering applicationmaster succeeded

i have ran job on spark 1.3.x , 1.4.x on vanilla spark/yarn cluster , cdh 5.4.3 cluster well. same results. possibly issue?

job run command below , input file accessible through hdfs.

bin/spark-submit --master yarn-cluster --class assocapp ../associationrulesscala/target/scala-2.10/associationrule_2.10.4-1.0.0.snapshot.jar hdfs://sparkmaster-hk:9000/user/root/breastcancer.csv

code snippets

this code in area dataframe loaded. spits out log message "uploading dataframe..." there nothing else after that. refer driver's logs below

//...   logger.info("uploading dataframe %s".format(filename))   sparkparams.sqlcontext.csvfile(filename)    mdc.put("jobid",jobid.takeright(3))   logger.info("extracting unique vals each of %d columns...".format(frame.columns.length))   private val uniquevals = frame.columns.zipwithindex.map(colname => (colname._2, colname._1, frame.select(colname._1).distinct.cache)). //...

driver logs

slf4j: class path contains multiple slf4j bindings. slf4j: found binding in [jar:file:/tmp/hadoop-root/nm-local-dir/usercache/root/filecache/60/spark-assembly-1.4.0-hadoop2.6.0.jar!/org/slf4j/impl/staticloggerbinder.class] slf4j: found binding in [jar:file:/root/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/staticloggerbinder.class] slf4j: see http://www.slf4j.org/codes.html#multiple_bindings explanation. slf4j: actual binding of type [org.slf4j.impl.log4jloggerfactory] 15/07/22 15:56:52 info yarn.applicationmaster: registered signal handlers [term, hup, int] 15/07/22 15:56:54 info yarn.applicationmaster: applicationattemptid: appattempt_1434116948302_0097_000001 15/07/22 15:56:55 info spark.securitymanager: changing view acls to: root 15/07/22 15:56:55 info spark.securitymanager: changing modify acls to: root 15/07/22 15:56:55 info spark.securitymanager: securitymanager: authentication disabled; ui acls disabled; users view permissions: set(root); users modify permissions: set(root) 15/07/22 15:56:55 info yarn.applicationmaster: starting user application in separate thread 15/07/22 15:56:55 info yarn.applicationmaster: waiting spark context initialization 15/07/22 15:56:55 info yarn.applicationmaster: waiting spark context initialization ...  15/07/22 15:56:56 info assocapp$: starting new association rules calculation. file: hdfs://sparkmaster-hk:9000/user/root/breastcancer.csv 15/07/22 15:56:56 info yarn.applicationmaster: final app status: succeeded, exitcode: 0 15/07/22 15:56:57 info associationrules.primarypackagespark: uploading dataframe hdfs://sparkmaster-hk:9000/user/root/breastcancer.csv  15/07/22 15:56:57 info spark.sparkcontext: running spark version 1.4.0 15/07/22 15:56:57 info spark.securitymanager: changing view acls to: root 15/07/22 15:56:57 info spark.securitymanager: changing modify acls to: root 15/07/22 15:56:57 info spark.securitymanager: securitymanager: authentication disabled; ui acls disabled; users view permissions: set(root); users modify permissions: set(root) 15/07/22 15:56:57 info slf4j.slf4jlogger: slf4jlogger started 15/07/22 15:56:57 info remoting: starting remoting 15/07/22 15:56:57 info remoting: remoting started; listening on addresses :[akka.tcp://sparkdriver@119.81.232.13:41459] 15/07/22 15:56:57 info util.utils: started service 'sparkdriver' on port 41459. 15/07/22 15:56:57 info spark.sparkenv: registering mapoutputtracker 15/07/22 15:56:57 info spark.sparkenv: registering blockmanagermaster 15/07/22 15:56:57 info storage.diskblockmanager: created local directory @ /tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1434116948302_0097/blockmgr-f0e66040-1fdb-4a05-87e1-160194829f84 15/07/22 15:56:57 info storage.memorystore: memorystore started capacity 267.3 mb 15/07/22 15:56:58 info spark.httpfileserver: http file server directory /tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1434116948302_0097/httpd-79b304a1-3cf4-4951-9e22-bbdfac435824 15/07/22 15:56:58 info spark.httpserver: starting http server 15/07/22 15:56:58 info server.server: jetty-8.y.z-snapshot 15/07/22 15:56:58 info server.abstractconnector: started socketconnector@0.0.0.0:36021 15/07/22 15:56:58 info util.utils: started service 'http file server' on port 36021. 15/07/22 15:56:58 info spark.sparkenv: registering outputcommitcoordinator 15/07/22 15:56:58 info ui.jettyutils: adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.amipfilter 15/07/22 15:56:58 info server.server: jetty-8.y.z-snapshot 15/07/22 15:56:58 info server.abstractconnector: started selectchannelconnector@0.0.0.0:53274 15/07/22 15:56:58 info util.utils: started service 'sparkui' on port 53274. 15/07/22 15:56:58 info ui.sparkui: started sparkui @ http://119.xx.xxx.xx:53274 15/07/22 15:56:58 info cluster.yarnclusterscheduler: created yarnclusterscheduler 15/07/22 15:56:59 info util.utils: started service 'org.apache.spark.network.netty.nettyblocktransferservice' on port 34498. 15/07/22 15:56:59 info netty.nettyblocktransferservice: server created on 34498 15/07/22 15:56:59 info storage.blockmanagermaster: trying register blockmanager 15/07/22 15:56:59 info storage.blockmanagermasterendpoint: registering block manager 119.81.232.13:34498 267.3 mb ram, blockmanagerid(driver, 119.81.232.13, 34498) 15/07/22 15:56:59 info storage.blockmanagermaster: registered blockmanager 15/07/22 15:56:59 info cluster.yarnschedulerbackend$yarnschedulerendpoint: applicationmaster registered akkarpcendpointref(actor[akka://sparkdriver/user/yarnam#-819146876]) 15/07/22 15:56:59 info client.rmproxy: connecting resourcemanager @ sparkmaster-hk/119.81.232.24:8030 15/07/22 15:56:59 info yarn.yarnrmclient: registering applicationmaster 15/07/22 15:57:00 info yarn.yarnallocator: request 2 executor containers, each 1 cores , 1408 mb memory including 384 mb overhead 15/07/22 15:57:00 info yarn.yarnallocator: container request (host: any, capability: <memory:1408, vcores:1>) 15/07/22 15:57:00 info yarn.yarnallocator: container request (host: any, capability: <memory:1408, vcores:1>) 15/07/22 15:57:00 info yarn.applicationmaster: started progress reporter thread - sleep time : 5000 15/07/22 15:57:00 info yarn.applicationmaster: unregistering applicationmaster succeeded 15/07/22 15:57:00 info impl.amrmclientimpl: waiting application unregistered. 15/07/22 15:57:00 info yarn.applicationmaster: deleting staging directory .sparkstaging/application_1434116948302_0097 15/07/22 15:57:00 info storage.diskblockmanager: shutdown hook called 15/07/22 15:57:00 info util.utils: shutdown hook called 15/07/22 15:57:00 info util.utils: deleting directory /tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1434116948302_0097/httpd-79b304a1-3cf4-4951-9e22-bbdfac435824 15/07/22 15:57:00 info util.utils: deleting directory /tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1434116948302_0097/userfiles-e01b4dd2-681c-4108-aec6-879774652c7a


Comments

Post a Comment

Popular posts from this blog

Fail to load namespace Spring Security http://www.springframework.org/security/tags -

-
i have problem namespace of spring security on xhtml page. don't understand why can't load uri, , have following issue: whitelabel error page application has no explicit mapping /error, seeing fallback. tue jul 21 16:58:12 utc 2015 there unexpected error (type=not found, status=404). please need help! possible uri has been changed? best regards you using incorrect namespace url. correct url http://www.springframework.org/schema/security the error page seeing spring source website since not recognize url. try visiting http://www.springsource.org/security/tags in browser update : spring security tags used in jsp page correct tag lib declaration is: <%@ taglib prefix="sec" uri="http://www.springframework.org/security/tags" %>
Read more

sql - MySQL query optimization using coalesce -

-
can explain me why second query faster first? (first 1 takes around 40 seconds, second 1 less 0.1) select releases.feature_code, releases.platform_id, version, release_date, general_available, uri, filesize, display_name, feature_name.full_name, if ((select exists(select 1 release_exceptions releases.feature_code = release_exceptions.feature_code , releases.platform_id = release_exceptions.platform_id , releases.version = release_exceptions.version) = 1), 1, 0) release_exceptions releases left outer join platform using(platform_id) left outer join feature_name using(feature_code) order version desc; ~~~ query 2 ------------------ ~~~ select feature_code, version, release_date, general_available, platform_id, uri, filesize, display_name, feature_name.full_name, coalesce(release_exceptions, 0) release_exceptions mus.releases left outer join ( select distinct feature_code, version, platform_id, 1 release_exceptions mus.release_exceptions ) release_exceptions using ...
Read more

unity3d - Unity local avoidance in user created world -

-
i'm using unity3d networked multiplayer online game have large complex 3d terrain scene forest, trees, cliff, hills, mountains, bounders, etc. players can build structures sort of minecraft, , put them anywhere in scene, or move them around anytime. aside human controlled players, there automated ai players , objects animals roaming around scene following path. the problem how make these automated ai players , animals, able navigate around static , dynamic player created structures, because path follow can blocked player created structures, or other players , other ai objects, cliffs etc. have find away around them or on track if tumble down off high cliff example. so made navmesh , used navagents, takes care of static, non moving objects, how make ai players navigate around each other , dynamic structures created players can number in hundreds? i thought of adding navmeshobstacle everything, result in being attached hundreds of objects since user created structures b...
Read more