python - KeyError: SPARK_HOME during SparkConf initialization -

-

i spark newbie , want run python script command line. have tested pyspark interactively , works. error when trying create sc:

file "test.py", line 10, in <module>     conf=(sparkconf().setmaster('local').setappname('a').setsparkhome('/home/dirk/spark-1.4.1-bin-hadoop2.6/bin'))   file "/home/dirk/spark-1.4.1-bin-hadoop2.6/python/pyspark/conf.py", line 104, in __init__     sparkcontext._ensure_initialized()   file "/home/dirk/spark-1.4.1-bin-hadoop2.6/python/pyspark/context.py", line 229, in _ensure_initialized     sparkcontext._gateway = gateway or launch_gateway()   file "/home/dirk/spark-1.4.1-bin-hadoop2.6/python/pyspark/java_gateway.py", line 48, in launch_gateway     spark_home = os.environ["spark_home"]   file "/usr/lib/python2.7/userdict.py", line 23, in __getitem__     raise keyerror(key) keyerror: 'spark_home'

it seems there 2 problems here.

the first 1 path use. spark_home should point root directory of spark installation in case should /home/dirk/spark-1.4.1-bin-hadoop2.6 not /home/dirk/spark-1.4.1-bin-hadoop2.6/bin.

the second problem way how use setsparkhome. if check a docstring goal to

set path spark installed on worker nodes

sparkconf constructor assumes spark_home on master set. it calls pyspark.context.sparkcontext._ensure_initialized which calls pyspark.java_gateway.launch_gateway, which tries acccess spark_home , fails.

to deal should set spark_home before create sparkconf.

import os os.environ["spark_home"] = "/home/dirk/spark-1.4.1-bin-hadoop2.6" conf = (sparkconf().setmaster('local').setappname('a'))

Comments

Post a Comment

Popular posts from this blog

Fail to load namespace Spring Security http://www.springframework.org/security/tags -

-
i have problem namespace of spring security on xhtml page. don't understand why can't load uri, , have following issue: whitelabel error page application has no explicit mapping /error, seeing fallback. tue jul 21 16:58:12 utc 2015 there unexpected error (type=not found, status=404). please need help! possible uri has been changed? best regards you using incorrect namespace url. correct url http://www.springframework.org/schema/security the error page seeing spring source website since not recognize url. try visiting http://www.springsource.org/security/tags in browser update : spring security tags used in jsp page correct tag lib declaration is: <%@ taglib prefix="sec" uri="http://www.springframework.org/security/tags" %>
Read more

sql - MySQL query optimization using coalesce -

-
can explain me why second query faster first? (first 1 takes around 40 seconds, second 1 less 0.1) select releases.feature_code, releases.platform_id, version, release_date, general_available, uri, filesize, display_name, feature_name.full_name, if ((select exists(select 1 release_exceptions releases.feature_code = release_exceptions.feature_code , releases.platform_id = release_exceptions.platform_id , releases.version = release_exceptions.version) = 1), 1, 0) release_exceptions releases left outer join platform using(platform_id) left outer join feature_name using(feature_code) order version desc; ~~~ query 2 ------------------ ~~~ select feature_code, version, release_date, general_available, platform_id, uri, filesize, display_name, feature_name.full_name, coalesce(release_exceptions, 0) release_exceptions mus.releases left outer join ( select distinct feature_code, version, platform_id, 1 release_exceptions mus.release_exceptions ) release_exceptions using ...
Read more

Maven Javadoc 'Cannot find default setter' and fails -

-
i'm trying generate javadocs maven project, , i'm running error every time. unable parse configuration of mojo org.apache.maven.plugins:maven-javadoc-plugin:2.10.3:javadoc parameter #: cannot find default setter in class org.apache.maven.plugin.javadoc.options.group . command i'm using mvn javadoc:javadoc root directory, pom is. i don't have groups configured @ all, or have special configuration @ all. same error whether omit plugin in pom completely, add reporting, or add build plugin. i've tried adding empty groups well, , while message changes somewhat, still appears. i ran across this question , a. don't have testng dependencies, b. i'm not using command line parameters, , c. never resolved. it selenium project, suppose there similar, can't figure out. ideas? i've included maven debug stack trace below. [error] failed execute goal org.apache.maven.plugins:maven-javadoc-plugin:2.10.3:javadoc (default-cli) on project selenium: un...
Read more