If you’re following along the SparkR Quick Start, you’ll notice that the instructions are not consistent with a more recent build of Spark. Here are instructions that work for SparkR version 1.4.1 on Linux. YMMV on Spark 1.5.
Now that SparkR has been promoted to the core Spark library, it lives in Spark’s bin directory with the other executables. Let’s define
SPARK_HOME as the root of your Spark installation. The only environment variable you need set is JAVA_HOME.
cd $SPARK_HOME ./bin/sparkR
If all goes well, you should see an R console start up followed by Java log messages (prepended with a bunch of
INFOs). Finally, you should see a happy welcome message:
Welcome to SparkR!
Spark context is available as sc, SQL context is available as sqlContext
First, take a look around, using the
ls() function, and you’ll see exactly two objects,
sqlContext, both of which were mentioned in the welcome message.
Now, let’s read in the README.md file and perform some trivial operations. The original documentation says to use the
textFile function, but unfortunately that is no longer exported by default. No worries, since R doesn’t have a strong permission system, so we can use it like this:
> text_file <- SparkR:::textFile(sc, 'README.md')
Again, you’ll see a number of log messages. Don’t worry, as long as you don’t see any
FATALs, there’s no reason to panic. Java is known for being verbose, and this extends to log messages.
> count(text_file) ... Java log messages ...  98
As the name implies, this function is counting the number of lines in this variable. We can confirm this by using the
wc bash command.
$ wc -l README.md 98 README.md
But what does this object look like? Like Hadoop, the data is distributed and abstracted from the end user. This means that simple commands like
head(text_file) won’t work the same as with a native R object. Instead you must use the provided grammar of Spark actions to interact with the RDD. To view a line of the text file thus requires using the
> take(text_file,1) []  "# Apache Spark"
The output indicates that the result is a list of length one. How do we view the whole file? It turns out that
take is synonymous to
head, so to get the whole file means specifying its length, which we obtained above.
> take(text_file, count(text_file))
Now on to some data processing. The first function introduced is a filter operation. Those following my book, Modeling Data With Functional Programming In R, will notice this is one of the canonical higher order functions. Now you see why 🙂 Notice that again we have to specify the function with the
::: notation. These functions should really be exported, and perhaps in 1.5 they are.
> linesWithSpark <- SparkR:::filterRDD(text_file, function(line) grepl("Spark", line))
If you’re eager to see the output of the filter operation, you’ll likely be disappointed. What you’ll see is something like
RRDD at RDD at RRDD.scala:35
Feel free to toss around some bemused expletives. In the meantime, let me explain what’s going on. At this point, Spark has only set up the work to do but hasn’t actually executed it. This is conceptually similar to the computer science notion of a promise. To actually evaluate the job, you need to use the
collect action. This will produce an output similar to the following.
> collect(linesWithSpark) []  "# Apache Spark" []  "Spark is a fast and general cluster computing system for Big Data. It provides" []  "rich set of higher-level tools including Spark SQL for SQL and structured" []  "and Spark Streaming for stream processing."
At this point, we’ve walked through half the quick start. You should have a basic understanding of the Spark action grammar as well as an understanding of how to process simple jobs in Spark. The next step is to run some other types of computations, including map and reduce jobs. We’ll cover that in a subsequent post.
Boris Guarisma said:
Thank you for this post. Do you know if there’s is an alternative for the ‘:::’ operator in release 1.5.0; I’ve tried the ‘%>%’ pipe operator (with magrittr package) as shown in another SparkR post (http://blog.rstudio.org/2015/05/28/sparkr-preview-by-vincent-warmerdam/) but it doesn’t work. Looking forward to reading your book !
Brian Lee Yung Rowe said:
In 1.5 a lot of the interface has changed. The `:::` is a native R operator, so if it doesn’t work, the reason is that the function no longer exists. My assessment is that SparkR is really not ready for prime time.
Excellent post. I’m experiencing a few of these issues as well..