Configure YarnSpawner¶

We now have JupyterHub installed, but it can’t really do anything yet. For JupyterHub to manage user’s notebook servers, we need to configure a Spawner for it to use. YarnSpawner is a Spawner implementation that launches notebook servers on Apache Hadoop/YARN clusters. Here we’ll discuss installation and configuration of this spawner, for more information see the YarnSpawner documentation

Install jupyterhub-yarnspawner
Set the JupyterHub Spawner Class
Configure the Hub Connect IP
Enable Proxy User Permissions
Enable Kerberos Security (Optional)
Specifying Python Environments
- Using a Local Environment
- Using an Archived Environment
Additional Configuration Options
Example

Install jupyterhub-yarnspawner ¶

Yarnpawner should be installed in the same environment that JupyterHub is running in.

$ conda install -c conda-forge jupyterhub-yarnspawner -y

Set the JupyterHub Spawner Class ¶

Tell JupyterHub to use YarnSpawner by adding the following line to your jupyterhub_config.py:

c.JupyterHub.spawner_class = 'yarnspawner.YarnSpawner'

Configure the Hub Connect IP ¶

By default JupyterHub runs its internal communications server on 127.0.0.1, meaning its only accesible from the machine running JupyterHub.

Since the servers started by YarnSpawner are running on other machines in the cluster, we’ll need to update this address to something more accessible:

# Set to '' for all interfaces. Can also set to the hostname of the
# JupyterHub machine.
c.JupyterHub.hub_ip = ''

Enable Proxy User Permissions ¶

YarnSpawner makes full use of Hadoop’s security model, and will start Jupyter notebook server’s in containers with the requesting user’s permissions (e.g. if alice logs in to JupyterHub, their notebook server will be running as user alice). To accomplish this, JupyterHub needs proxy user permissions. This allows the JupyterHub server to perform actions impersonating another user.

For JupyterHub to work properly, you’ll need to enable proxy user permissions for the jupyterhub user account. The users jupyterhub has permission to impersonate can be restricted to certain groups, and requests to impersonate may be restricted to certain hosts. At a minimum, jupyterhub will require permission to impersonate any JupyterHub user, with requests allowed from at least the host running JupyterHub.

<property>
  <name>hadoop.proxyuser.jupyterhub.hosts</name>
  <value>host-where-jupyterhub-is-running</value>
</property>
<property>
  <name>hadoop.proxyuser.jupyterhub.groups</name>
  <value>group1,group2</value>
</property>

If looser restrictions are acceptable, you may also use the wildcard * to allow impersonation of any user or from any host.

<property>
  <name>hadoop.proxyuser.jupyterhub.hosts</name>
  <value>*</value>
</property>
<property>
  <name>hadoop.proxyuser.jupyterhub.groups</name>
  <value>*</value>
</property>

See the proxy user documentation for more information.

Enable Kerberos Security (Optional)¶

If your cluster has Kerberos enabled, you’ll also need to create a principal and keytab for the jupyterhub user.

# Create the jupyterhub principal
$ kadmin -q "addprinc -randkey jupyterhub@YOUR_REALM.COM"

# Create a keytab
$ kadmin -q "xst -norandkey -k /etc/jupyterhub/jupyterhub.keytab

Store the keytab file wherever you see fit (we recommend storing it along with the JupyterHub configuration, as above). You’ll also want to make sure that jupyterhub.keytab is only readable by the jupyterhub user.

$ chown jupyterhub /etc/jupyterhub/jupyterhub.keytab
$ chmod 400 /etc/jupyterhub/jupyterhub.keytab

To configure JupyterHub to use this keytab file, you’ll need to add the following line to your jupyterhub_config.py:

# The principal JupyterHub is running as
c.YarnSpawner.principal = 'jupyterhub'

# Path to the keytab you created
c.YarnSpawner.keytab = '/etc/jupyterhub/jupyterhub.keytab'

Specifying Python Environments ¶

Since the user’s notebook servers will be each running in their own YARN container, you’ll need to provide a way for Python environments to be available to these containers. You have a few options here:

Install identical Python environments on every node
Archive environments to be distributed to the container at runtime (recommended)

In either case, the Python environment requires at minimum:

jupyterhub-yarnspawner
jupyterhub
notebook

Using a Local Environment ¶

If you’ve installed identical Python environments on every node, you only need to configure YarnSpawner to use the provided Python. This could be done a few different ways:

# Use the path to python in the startup command
c.YarnSpawner.cmd = '/path/to/python -m yarnspawner.singleuser'

# OR
# Activate a local conda environment before startup
c.YarnSpawner.prologue = 'conda activate /path/to/your/environment'

# OR
# Activate a virtual environment before startup
c.YarnSpawner.prologue = 'source /path/to/your/environment/bin/activate'

Using an Archived Environment ¶

YARN also provides mechanisms to “localize” files/archives to a container before starting the application. This can be used to distribute Python environments at runtime. This approach is appealing in that it doesn’t require installing anything throughout the cluster, and allows for centrally managing your user’s Python environments.

Packaging environments for distribution is usually accomplished using

conda-pack for conda environments
venv-pack for virtual environments (both venv and virtualenv supported)

Both are tools for taking an environment and creating an archive of it in a way that (most) absolute paths in any libraries or scripts are altered to be relocatable. This archive then can be distributed with your application, and will be automatically extracted during YARN resource localization

Below we demonstrate creating and packaging a Conda environment containing all the required Jupyter packages, as well as pandas and scikit-learn. Additional packages could be added as needed.

Packaging a Conda Environment with Conda-Pack

# Make a folder for storing the conda environments locally
$ mkdir /opt/jupyterhub/envs

# Create a new conda environment
$ conda create -c conda-forge -y -p /opt/jupyterhub/envs/example
...

# Activate the environment
$ conda activate /opt/jupyterhub/envs/example

# Install the needed packages
$ conda install -c conda-forge -y \
conda-pack \
jupyterhub-yarnspawner \
pandas \
scikit-learn
...

# Pip required to avoid hardcoded path in kernelspec (for now)
$ pip install notebook

# Package the environment into example.tar.gz
$ conda pack -o example.tar.gz
Collecting packages...
Packing environment at '/opt/jupyterhub/envs/example' to 'example.tar.gz'
[########################################] | 100% Completed | 17.9s

Using the Packaged Environment

It is recommended to upload the environments to some directory on HDFS beforehand, to avoid repeating the upload cost for every user. This directory should be readable by all users, but writable only by the admin user managing Python environments (here we’ll use the jupyterhub user, and create a /jupyterhub directory).

$ hdfs dfs -mkdir -p /jupyterhub
$ hdfs dfs -chown jupyterhub /jupyterhub
$ hdfs dfs -chmod 755 /jupyterhub

Uploading our already packaged environment to hdfs:

$ hdfs dfs -put /opt/jupyterhub/envs/example.tar.gz /jupyterhub/example.tar.gz

To use the packaged environment with YarnSpawner, you need to include the archive in YarnSpawner.localize_files, and activate the environment in YarnSpawner.prologue.

c.YarnSpawner.localize_files = {
    'environment': {
        'source': 'hdfs:///jupyterhub/example.tar.gz',
        'visibility': 'public'
    }
}
c.YarnSpawner.prologue = 'source environment/bin/activate'

Note that we set visibility to public for the environment, so that multiple users can all share the same localized environment (reducing the cost of moving the environments around).

For more information, see the Skein documentation on distributing files.

Additional Configuration Options ¶

YarnSpawner has several additional configuration fields. See the YarnSpawner documentation for more information on all available options. At a minimum you’ll probably want to configure the memory and cpu limits, as well as which YARN queue to use.

# The memory limit for a notebook instance.
c.YarnSpawner.mem_limit = '2 G'

# The cpu limit for a notebook instance
c.YarnSpawner.cpu_limit = 1

# The YARN queue to use
c.YarnSpawner.queue = '...'

Example ¶

In summary, an example jupyterhub_config.py configuration enabling yarnspawner might look like:

# Make the JupyterHub internal communication accessible from other machines
# in the cluster
c.JupyterHub.hub_ip = ''

# Enable yarnspawner
c.JupyterHub.spawner_class = 'yarnspawner.YarnSpawner'

# Configuration for kerberos security
c.YarnSpawner.principal = 'jupyterhub'
c.YarnSpawner.keytab = '/etc/jupyterhub/jupyterhub.keytab'

# Resource limits per-user
c.YarnSpawner.mem_limit = '2 G'
c.YarnSpawner.cpu_limit = 1

# The YARN queue to use
c.YarnSpawner.queue = 'jupyterhub'

# Specify location of the archived Python environment
c.YarnSpawner.localize_files = {
    'environment': {
        'source': 'hdfs:///jupyterhub/example.tar.gz',
        'visibility': 'public'
    }
}
c.YarnSpawner.prologue = 'source environment/bin/activate'