New information sources and spark_apply() capabilities, higher interfaces for sparklyr extensions, and extra!

New information sources and spark_apply() capabilities, higher interfaces for sparklyr extensions, and extra!

Sparklyr 1.7 is now accessible on CRAN!

To put in sparklyr 1.7 from CRAN, run

On this weblog publish, we want to current the next highlights from the sparklyr 1.7 launch:

Picture and binary information sources

As a unified analytics engine for large-scale information processing, Apache Spark
is well-known for its capability to deal with challenges related to the quantity, velocity, and final however
not least, the number of huge information. Due to this fact it’s hardly stunning to see that – in response to current
advances in deep studying frameworks – Apache Spark has launched built-in assist for
image data sources
and binary data sources (in releases 2.4 and three.0, respectively).
The corresponding R interfaces for each information sources, specifically,
spark_read_image() and
spark_read_binary(), had been shipped
lately as a part of sparklyr 1.7.

The usefulness of information supply functionalities equivalent to spark_read_image() is probably greatest illustrated
by a fast demo beneath, the place spark_read_image(), by means of the usual Apache Spark
helps connecting uncooked picture inputs to a classy characteristic extractor and a classifier, forming a strong
Spark software for picture classifications.

The demo

Photograph by Daniel Tuttle on

On this demo, we will assemble a scalable Spark ML pipeline able to classifying photographs of cats and canines
precisely and effectively, utilizing spark_read_image() and a pre-trained convolutional neural community
code-named Inception (Szegedy et al. (2015)).

Step one to constructing such a demo with most portability and repeatability is to create a
sparklyr extension that accomplishes the next:

A reference implementation of such a sparklyr extension could be present in

The second step, in fact, is to utilize the above-mentioned sparklyr extension to carry out some characteristic
engineering. We’ll see very high-level options being extracted intelligently from every cat/canine picture based mostly
on what the pre-built Inception-V3 convolutional neural community has already realized from classifying a a lot
broader assortment of photographs:


# NOTE: the right spark_home path to make use of is dependent upon the configuration of the
# Spark cluster you're working with.
spark_home <- "/usr/lib/spark"
sc <- spark_connect(grasp = "yarn", spark_home = spark_home)

data_dir <- copy_images_to_hdfs()

# extract options from train- and test-data
image_data <- list()
for (x in c("prepare", "take a look at")) {
  # import
  image_data[[x]] <- c("canines", "cats") %>%
      operate(label) {
        numeric_label <- ifelse(identical(label, "canines"), 1L, 0L)
          sc, dir = file.path(data_dir, x, label, fsep = "/")
        ) %>%
          dplyr::mutate(label = numeric_label)
    ) %>%, .)

  dl_featurizer <- invoke_new(
    random_string("dl_featurizer") # uid
  ) %>%
    invoke("setModelName", "InceptionV3") %>%
    invoke("setInputCol", "picture") %>%
    invoke("setOutputCol", "options")
  image_data[[x]] <-
    dl_featurizer %>%
    invoke("remodel", spark_dataframe(image_data[[x]])) %>%

Third step: geared up with options that summarize the content material of every picture properly, we will
construct a Spark ML pipeline that acknowledges cats and canines utilizing solely logistic regression

label_col <- "label"
prediction_col <- "prediction"
pipeline <- ml_pipeline(sc) %>%
    features_col = "options",
    label_col = label_col,
    prediction_col = prediction_col
mannequin <- pipeline %>% ml_fit(image_data$prepare)

Lastly, we will consider the accuracy of this mannequin on the take a look at photographs:

predictions <- mannequin %>%
  ml_transform(image_data$take a look at) %>%

cat("Predictions vs. labels:n")
predictions %>%
  dplyr::select(!!label_col, !!prediction_col) %>%
  print(n = sdf_nrow(predictions))

cat("nAccuracy of predictions:n")
predictions %>%
    label_col = label_col,
    prediction_col = prediction_col,
    metric_name = "accuracy"
  ) %>%
## Predictions vs. labels:
## # Supply: spark<?> [?? x 2]
##    label prediction
##    <int>      <dbl>
##  1     1          1
##  2     1          1
##  3     1          1
##  4     1          1
##  5     1          1
##  6     1          1
##  7     1          1
##  8     1          1
##  9     1          1
## 10     1          1
## 11     0          0
## 12     0          0
## 13     0          0
## 14     0          0
## 15     0          0
## 16     0          0
## 17     0          0
## 18     0          0
## 19     0          0
## 20     0          0
## Accuracy of predictions:
## [1] 1

New spark_apply() capabilities

Optimizations & customized serializers

Many sparklyr customers who’ve tried to run
spark_apply() or
doSpark to
parallelize R computations amongst Spark staff have in all probability encountered some
challenges arising from the serialization of R closures.
In some situations, the
serialized measurement of the R closure can change into too giant, usually because of the measurement
of the enclosing R setting required by the closure. In different
situations, the serialization itself could take an excessive amount of time, partially offsetting
the efficiency acquire from parallelization. Not too long ago, a number of optimizations went
into sparklyr to deal with these challenges. One of many optimizations was to
make good use of the
broadcast variable
assemble in Apache Spark to cut back the overhead of distributing shared and
immutable activity states throughout all Spark staff. In sparklyr 1.7, there’s
additionally assist for customized spark_apply() serializers, which gives extra fine-grained
management over the trade-off between pace and compression stage of serialization
algorithms. For instance, one can specify

options(sparklyr.spark_apply.serializer = "qs")


which can apply the default choices of qs::qserialize() to realize a excessive
compression stage, or

options(sparklyr.spark_apply.serializer = operate(x) qs::qserialize(x, preset = "quick"))
options(sparklyr.spark_apply.deserializer = operate(x) qs::qdeserialize(x))


which can purpose for sooner serialization pace with much less compression.

Inferring dependencies robotically

In sparklyr 1.7, spark_apply() additionally gives the experimental
auto_deps = TRUE choice. With auto_deps enabled, spark_apply() will
study the R closure being utilized, infer the checklist of required R packages,
and solely copy the required R packages and their transitive dependencies
to Spark staff. In lots of situations, the auto_deps = TRUE choice shall be a
considerably higher different in comparison with the default packages = TRUE
habits, which is to ship all the things inside .libPaths() to Spark employee
nodes, or the superior packages = <package deal config> choice, which requires
customers to produce the checklist of required R packages or manually create a
spark_apply() bundle.

Higher integration with sparklyr extensions

Substantial effort went into sparklyr 1.7 to make lives simpler for sparklyr
extension authors. Expertise suggests two areas the place any sparklyr extension
can undergo a frictional and non-straightforward path integrating with
sparklyr are the next:

We’ll elaborate on current progress in each areas within the sub-sections beneath.

Customizing the dbplyr SQL translation setting

sparklyr extensions can now customise sparklyr’s dbplyr SQL translations
by means of the
specification returned from spark_dependencies() callbacks.
One of these flexibility turns into helpful, as an example, in situations the place a
sparklyr extension must insert kind casts for inputs to customized Spark
UDFs. We will discover a concrete instance of this in
a sparklyr extension to facilitate geo-spatial analyses utilizing
Apache Sedona. Geo-spatial UDFs supported by Apache
Sedona equivalent to ST_Point() and ST_PolygonFromEnvelope() require all inputs to be
DECIMAL(24, 20) portions moderately than DOUBLEs. With none customization to
sparklyr’s dbplyr SQL variant, the one approach for a dplyr
question involving ST_Point() to truly work in sparklyr can be to explicitly
implement any kind solid wanted by the question utilizing dplyr::sql(), e.g.,

my_geospatial_sdf <- my_geospatial_sdf %>%
    x = dplyr::sql("CAST(`x` AS DECIMAL(24, 20))"),
    y = dplyr::sql("CAST(`y` AS DECIMAL(24, 20))")
  ) %>%
  dplyr::mutate(pt = ST_Point(x, y))


This may, to some extent, be antithetical to dplyr’s purpose of liberating R customers from
laboriously spelling out SQL queries. Whereas by customizing sparklyr’s dplyr SQL
translations (as applied in
), sparklyr.sedona permits customers to easily write

my_geospatial_sdf <- my_geospatial_sdf %>% dplyr::mutate(pt = ST_Point(x, y))

as an alternative, and the required Spark SQL kind casts are generated robotically.

Improved interface for invoking Java/Scala capabilities

In sparklyr 1.7, the R interface for Java/Scala invocations noticed a lot of

With earlier variations of sparklyr, many sparklyr extension authors would
run into hassle when making an attempt to invoke Java/Scala capabilities accepting an
Array[T] as one among their parameters, the place T is any kind sure extra particular
than java.lang.Object / AnyRef. This was as a result of any array of objects handed
by means of sparklyr’s Java/Scala invocation interface shall be interpreted as merely
an array of java.lang.Objects in absence of extra kind data.
Because of this, a helper operate
jarray() was applied as
a part of sparklyr 1.7 as a solution to overcome the aforementioned drawback.
For instance, executing

sc <- spark_connect(...)

arr <- jarray(
  seq(5) %>% lapply(operate(x) invoke_new(sc, "MyClass", x)),
  element_type = "MyClass"

will assign to arr a reference to an Array[MyClass] of size 5, moderately
than an Array[AnyRef]. Subsequently, arr turns into appropriate to be handed as a
parameter to capabilities accepting solely Array[MyClass]s as inputs. Beforehand,
some potential workarounds of this sparklyr limitation included altering
operate signatures to just accept Array[AnyRef]s as an alternative of Array[MyClass]s, or
implementing a “wrapped” model of every operate accepting Array[AnyRef]
inputs and changing them to Array[MyClass] earlier than the precise invocation.
None of such workarounds was a great answer to the issue.

One other comparable hurdle that was addressed in sparklyr 1.7 as properly includes
operate parameters that should be single-precision floating level numbers or
arrays of single-precision floating level numbers.
For these situations,
jfloat() and
are the helper capabilities that enable numeric portions in R to be handed to
sparklyr’s Java/Scala invocation interface as parameters with desired sorts.

As well as, whereas earlier verisons of sparklyr didn’t serialize
parameters with NaN values accurately, sparklyr 1.7 preserves NaNs as
anticipated in its Java/Scala invocation interface.

Different thrilling information

There are quite a few different new options, enhancements, and bug fixes made to
sparklyr 1.7, all listed within the
file of the sparklyr repo and documented in sparklyr’s
HTML reference pages.
Within the curiosity of brevity, we is not going to describe all of them in nice element
inside this weblog publish.


In chronological order, we want to thank the next people who
have authored or co-authored pull requests that had been a part of the sparklyr 1.7

We’re additionally extraordinarily grateful to everybody who has submitted
characteristic requests or bug reviews, lots of which have been tremendously useful in
shaping sparklyr into what it’s immediately.

Moreover, the creator of this weblog publish is indebted to
@skeydan for her superior editorial ideas.
With out her insights about good writing and story-telling, expositions like this
one would have been much less readable.

In case you want to study extra about sparklyr, we advocate visiting,,
and in addition studying some earlier sparklyr launch posts equivalent to
sparklyr 1.6
sparklyr 1.5.

That’s all. Thanks for studying!

Databricks, Inc. 2019. Deep Studying Pipelines for Apache Spark (model 1.5.0).
Elson, Jeremy, John (JD) Douceur, Jon Howell, and Jared Saul. 2007. “Asirra: A CAPTCHA That Exploits Curiosity-Aligned Handbook Picture Categorization.” In Proceedings of 14th ACM Convention on Pc and Communications Safety (CCS), Proceedings of 14th ACM Convention on Pc and Communications Safety (CCS). Affiliation for Computing Equipment, Inc.
Szegedy, Christian, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. “Going Deeper with Convolutions.” In Pc Imaginative and prescient and Sample Recognition (CVPR).

Leave a Reply

Your email address will not be published. Required fields are marked *