How to run Jupyter and Spark on Mac with minimal settings

Run Spark on iPython Notebook (Jupyter). I've posted many posts on the same theme in the past, but I think this is the easiest method.

environment

Spark installation

Easy to install Homebrew apache-spark. Homebrew installation omitted

brew install apache-spark

Python installation

Python creates a dedicated environment with virtualenv. I named it spark.

mkvirtualenv spark

Install the required modules. numpy comes in at the same time you install pandas. Add scipy as needed.

pip install jupyter pandas matplotlib

Run

Run pyspark with PYSPARK_DRIVER_PYTHON and PYSPARK_DRIVER_PYTHON_OPTS.

PYSPARK_DRIVER_PYTHON=ipython PYSPARK_DRIVER_PYTHON_OPTS=notebook pyspark  --master local[*]

Execution example

Screen Shot 2016-11-26 at 10.30.34 PM.png

Recommended Posts

How to run Jupyter and Spark on Mac with minimal settings
How to install Fast.ai on Alibaba Cloud GPU and run it on Jupyter notebook
How to install Theano on Mac OS X with homebrew
How to install mysql-connector-python on mac
How to install OpenCV on Mac
How to run matplotlib on heroku
How to install OpenCV on Cloud9 and run it in Python
How to install pandas on EC2 (How to deal with MemoryError and PermissionError)
Install Anaconda on Mac and upload Jupyter (IPython) notebook to Anaconda Cloud
How to build Python and Jupyter execution environment with VS Code
How to install drobertadams / toggl-cli on Mac
How to erase Python 2.x on Mac.
How to use jupyter notebook with ABCI
How to run Cython on OSX Memo
Specify the browser to use with Jupyter Notebook. Especially Mac. (And Vivaldi)
How to debug with Jupyter or iPython Notebook
Strategy on how to monetize with Python Java
How to make multiple kernels selectable on Jupyter
Preparing to run ImageMagick + im4java on Mac OS
How to run MeCab on Ubuntu 18.04 LTS Python
How to install OpenGM on OSX with macports
How to switch between Linux and Mac shells
Steps to run TensorFlow 2.1 from Jupyter on supercomputer ITO front end (with GPU)
How to get started with the 2020 Python project (windows wsl and mac standardization)
How to run a Django application on a Docker container (development and production environment)
How to pass args or environment variables with Makefile and make command on #Linux
Install selenium on Mac and try it with python
Settings when you want to run python-mecab with travis
Streaming Python and SensorTag, Kafka, Spark Streaming-Part 5: Connecting from Jupyter to Spark with Apache Toree
How to run tests in bulk with Python unittest
How to install Git GUI and Gitk on CentOS
How to install caffe on OS X with macports
How to share folders with Docker and Windows with tensorflow
How to extract null values and non-null values with pandas
How to run Self bot on Discord.py [Easy vandalism! ]
How to loop and play gif video with openCV
I want to AWS Lambda with Python on Mac!
Autoencoder with Chainer (Notes on how to use + trainer)
Run OpenMVG on Mac
Steps to quickly create a deep learning environment on Mac with TensorFlow and OpenCV
How to run Django on IIS on a Windows server
How to view progress bar on Jupyter Notebook to see progress
[How to!] Learn and play Super Mario with Tensorflow !!
How to operate Firefox with selenium on Windows Memo
How to build Anaconda virtual environment used in Azure Machine Learning and link with Jupyter
[Cyberduck] How to exchange files on Linux (CentOS7) started by VirtualBox with mac using GUI
Detailed explanation How to run the sample code of UNIX programming 3rd edition on Mac
[Raspberry Pi] How to continuously monitor directories and run arbitrary shells with hooks for changes
Word Count with Apache Spark and python (Mac OS X)
How to install Python2.7 python3.5 with pyenv (on RHEL5 CentOS5) (2016 Nov)
Memo to get the value on the html-javascript side with jupyter
How to run a trained transformer model locally on CloudTPU
[TensorFlow 2 / Keras] How to run learning with CTC Loss in Keras
How to share OS and Vim clipboard on Ubuntu 18.04.3 LTS
Building a Python environment on a Mac and using Jupyter lab
Don't lose to Ruby! How to run Python (Django) on Heroku
[Python] How to play with class variables with decorator and metaclass
[Ev3dev] How to display bmp image on LCD with python
How to do Bulk Update with PyMySQL and notes [Python]
How to run an app built with Python + py2app built with Anaconda
Connect to centos6 on virtualbox with ssh connection from Mac