Hello,
For some reason javac errors when I import any of the Apache-Spark classes. I've gone through a number of setup guides to see if there's some environment variable I'm missing, but most don't have a step where an environment variable is set. Anyone have a clue what I'm missing? Spark Shell works and I'm able to run example scripts. Do I not compile with javac? Anyone have a clue what I'm missing?
Jump to PostIt would be helpful to know exactly what the error message says.
Jump to PostSounds like the jar is not in your classpath. You should have CLASSPATH environment variable, and one of the entries in it should be where the apache jar is stored.
If not, either add the relevant dir to the classpath, or move the jar to somewhere that's already in the …
It would be helpful to know exactly what the error message says.
It's "package org.apache.spark.api.java does not exist" when I try to import JavaRDD.
Sounds like the jar is not in your classpath. You should have CLASSPATH environment variable, and one of the entries in it should be where the apache jar is stored.
If not, either add the relevant dir to the classpath, or move the jar to somewhere that's already in the classpath
The issue here is I don't know where that jar is or what it would be named. By the way -cp <path of jar file> is how you'd manually do it right?
Yes, you can use -cp in the javac command instead of setting the classpath.
So did you download spark? It's not part of the standard Java distribution.
I've never used it, so I can't offer any more advice, but there must be some kind of user group or WIKI for it.
It's been setup on an unix machine that I ssh to, I can start the Spark Shell and run some example scala code. The issue is I don't know if they did a crappy job setting it up or just a case of my user profile not having the class paths needed.
Problem solved, didn't realize you couln't have a space between -cp and the path. In the end I just set the environment variable to it instead.
We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.