Databrick's Community server for Spark and "notebooks" -- looking at adding threejs in R notebooks?

[Cr6]

Hi guys,

I just wanted to share this link as a possible way to create .r or .py notebooks (.ipython)  that can be run on data sets with SQL, R, Java, Scala or Python.  It's pretty good environment that can be ported to Nvidia's more massive site if needed.  It gives 6 GBs of data storage (database) and files out of the box.

We might be able to build and share notebooks while capitalizing on a lot of free libraries that can be imported into the environment.
It is good for exploring "big data" as well in a managed cloud environment.  All of the cool data munging/discovery with Spark Machine Learning can be added (TensorFlow, Keras, etc.).  It doesn't require a credit-card like Microsoft/AWS. Text mining is available as well.

https://community.cloud.databricks.com

I only throw this up here because the MNIST example for Deep Learning is available on the site and easy to get running. It might be useful at some point to use notebooks to experiment with automating a simulated "bonding" with Mathis's Charge Flows using images from Nevyn's Periodic table. Basically maybe try to feed a classical molecule's chemical notation (CO₂) and get a Mathis style CF molecule generated from the standard notation feed. This could be a big time waster but at least it should provide more resume dressing with "Big Data/ML".  

Keras-MNIST notebook examples:
https://docs.databricks.com/applications/deep-learning/deep-learning-pipelines.html
https://docs.databricks.com/_static/notebooks/keras-single-node.html
https://docs.databricks.com/user-guide/notebooks/index.html#importing-notebooks
https://docs.databricks.com/spark/latest/sparkr/tutorials/index.html
https://github.com/drabastomek/learningPySpark
https://dennyglee.com/2017/08/04/introduction-to-apache-spark-2-0/

[Cr6]

Here's where I'm trying to take this if it is possible. Generating Mathis molecule ouputs like this from standard Chem notation in R on Databricks' R/python style notebooks. This library has come a long way since looking at it back in 2015. There still could be big gaps with the current threejs .js libraries though:

https://bwlewis.github.io/rthreejs/
https://cran.rstudio.com/web/packages/PeriodicTable/


Install from CRAN: install.packages("threejs")
Install the development version of the package directly from GitHub with the devtools package: devtools::install_github("bwlewis/rthreejs")

https://bwlewis.github.io/rthreejs/advanced/cumulative.html
https://dennyglee.com/2016/05/28/on-time-flight-performance-with-graphframes-for-apache-spark/
https://github.com/bwlewis/rthreejs

Summary of changes

The scatterplot3js() function was substantially improved and updated.

   The new pch option supports many point styles with size control.
   Interactive rotation and zooming are greatly improved and panning is now supported: press and hold the right mouse button (or touch equivalent) and move the mouse to pan.
   Mouse over labels are supported in WebGL renderings.
   The points3d() interface has changed to support pipelining.
   Lines are supported too, see lines3d().
   Support for crosstalk selection handles (see demo("crosstalk", package="threejs")).
   Set the experimental use.orbitcontrols=TRUE option for more CPU-efficient (but less fluid) rendering (good for laptops), also applies to graphjs().

The graphjs() function is completely new.

   Greater variety of WebGL vertex rendering ("pch") options, including spheres and much higher-performance options for large graphs.
   Graph layout is now external; for instance use one of the many superb igraph package graph layout options.
   Graph animation is supported, see the examples.
   Interactive (click-able) graph animation is supported, see demo(package="threejs") for examples.
   Limited brushing is available to highlight portions of the graph, see the brush=TRUE option.
   Support for crosstalk selection handles.

Known issues

   RStudio on Windows systems may not be able to render the WebGL graphics emitted by threejs. RStudio users running on Windows systems may need to use the plot "pop out" button to see visualizations in an external browser. We expect this to be a temporary problem until the underlying graphics rendering system used by RStudio is updated later in 2017.
   The fallback Canvas rendering code has diverged too much from the baseline WebGL code and no longer works. We have temporarily disabled Canvas rendering with an error message. See https://github.com/bwlewis/rthreejs/issues/67 for details.
   Crosstalk filter handles are used in a non-standard and experimental way to control graph animation. Don't rely on this experimental feature.

[Cr6]

Just an FYI... looks like Microsoft's Azure DataFactory can also run Databricks' notebooks:

https://docs.microsoft.com/en-us/azure/data-factory/transform-data-using-databricks-notebook

[Sponsored content]