python - KeyError: SPARK_HOME during SparkConf initialization -

-

i spark newbie , want run python script command line. have tested pyspark interactively , works. error when trying create sc:

file "test.py", line 10, in <module>     conf=(sparkconf().setmaster('local').setappname('a').setsparkhome('/home/dirk/spark-1.4.1-bin-hadoop2.6/bin'))   file "/home/dirk/spark-1.4.1-bin-hadoop2.6/python/pyspark/conf.py", line 104, in __init__     sparkcontext._ensure_initialized()   file "/home/dirk/spark-1.4.1-bin-hadoop2.6/python/pyspark/context.py", line 229, in _ensure_initialized     sparkcontext._gateway = gateway or launch_gateway()   file "/home/dirk/spark-1.4.1-bin-hadoop2.6/python/pyspark/java_gateway.py", line 48, in launch_gateway     spark_home = os.environ["spark_home"]   file "/usr/lib/python2.7/userdict.py", line 23, in __getitem__     raise keyerror(key) keyerror: 'spark_home'

it seems there 2 problems here.

the first 1 path use. spark_home should point root directory of spark installation in case should /home/dirk/spark-1.4.1-bin-hadoop2.6 not /home/dirk/spark-1.4.1-bin-hadoop2.6/bin.

the second problem way how use setsparkhome. if check a docstring goal to

set path spark installed on worker nodes

sparkconf constructor assumes spark_home on master set. it calls pyspark.context.sparkcontext._ensure_initialized which calls pyspark.java_gateway.launch_gateway, which tries acccess spark_home , fails.

to deal should set spark_home before create sparkconf.

import os os.environ["spark_home"] = "/home/dirk/spark-1.4.1-bin-hadoop2.6" conf = (sparkconf().setmaster('local').setappname('a'))

Comments

Post a Comment

Popular posts from this blog

Fail to load namespace Spring Security http://www.springframework.org/security/tags -

-
i have problem namespace of spring security on xhtml page. don't understand why can't load uri, , have following issue: whitelabel error page application has no explicit mapping /error, seeing fallback. tue jul 21 16:58:12 utc 2015 there unexpected error (type=not found, status=404). please need help! possible uri has been changed? best regards you using incorrect namespace url. correct url http://www.springframework.org/schema/security the error page seeing spring source website since not recognize url. try visiting http://www.springsource.org/security/tags in browser update : spring security tags used in jsp page correct tag lib declaration is: <%@ taglib prefix="sec" uri="http://www.springframework.org/security/tags" %>
Read more

Maven Javadoc 'Cannot find default setter' and fails -

-
i'm trying generate javadocs maven project, , i'm running error every time. unable parse configuration of mojo org.apache.maven.plugins:maven-javadoc-plugin:2.10.3:javadoc parameter #: cannot find default setter in class org.apache.maven.plugin.javadoc.options.group . command i'm using mvn javadoc:javadoc root directory, pom is. i don't have groups configured @ all, or have special configuration @ all. same error whether omit plugin in pom completely, add reporting, or add build plugin. i've tried adding empty groups well, , while message changes somewhat, still appears. i ran across this question , a. don't have testng dependencies, b. i'm not using command line parameters, , c. never resolved. it selenium project, suppose there similar, can't figure out. ideas? i've included maven debug stack trace below. [error] failed execute goal org.apache.maven.plugins:maven-javadoc-plugin:2.10.3:javadoc (default-cli) on project selenium: un...
Read more

lua - nginx string.match non posix -

-
i got string (str1) , want extract after pattern "mycode=", local str1 = "servername/codebase/?mycode=abc123"; local tmp1 = string.match(str1, "mycode=%w+"); local tmp2 = string.gsub(tmp1,"mycode=", ""); from logs, tmp1 => mycode=abc123 tmp2 => abc123 is there better/more efficient way this? belive lua strings not follow posix standard (due size of code base). yes, use capture in pattern control string.match . from lua reference manual (emphasis mine): looks first match of pattern in string s. if finds one, match returns captures pattern ; otherwise returns nil. if pattern specifies no captures, whole match returned. third, optional numerical argument init specifies start search; default value 1 , can negative. it works this: > local str1 = "servername/codebase/?mycode=abc123" > local tmp1 = string.match(str1, "mycode=%w+") > print(tmp1) mycode=abc123 > local tmp2 =...
Read more