Page Comparison

...

Due to recent developments in deep learning and related topics, tensorflow and its components have been widely requested by the user community. Due to the nature of the package however, we have decided it is best for the user to manage their own installation to ensure that they can quickly modify or upgrade tensorflow to their own needs without waiting for Research Computing to handle version changes. This page therefore serves to provide basic instructions on how to install tensorflow without elevated privileges, into a local directory that is owned by the user.

Basic Installation

A year agoEarly on, tensorflow was quite difficult to install into a shared computing environment such as O2. However, the developers have since made it far friendlier to set up for the average user that is not computing locally. All that needs to be done is to invoke pip to complete the installation.

Tensorflow is compatible with both Python 2 and 3 as of the writing of this document, and comes in two flavors: tensorflow and tensorflow-gpu. In order to install either of themit, it is strongly recommended to set up a virtual environment.

...

Once you're on a compute node, load the prerequisite modulesmodule:

$ module load gcc/69.2.0

This will expose the available python 3 modules to you. From here, you have two choices.

For Python 2:

$ module load python/2.7.12-ucs4

The other python 2 module will NOT work. This means that if you already have an existing python virtual environment, you may still need to create a new one for Tensorflow.

For Python 3, you can use any version we offer. For It is strongly recommended to use a version that is at least 3.8; we will use 3.9.14 in this example:

$ module load python/3.69.014

If you are planning to install/ use the GPU versionresources, you also need to load the (latest) cuda module. For example:

$ module load cuda/1012.21

Once you have confirmed that gcc and one of the above two python modules , python, and possibly CUDA, are loaded, create a virtual environment (instructions replicated here from Personal Python Packages):

$ virtualenv /path/to/nameofenv --system-site-packages

where /path/to/nameofenv is your chosen name and location of the environment you'd like to install to. (/path/to should already exist.) If you'd like to install fresh components for tensorflow, feel free to omit the --system-site-packages flag. Once the environment is created, you'll want to turn it on.

...

From here, you should be ready to install Tensorflow.As mentioned above, there are two flavors: tensorflow and tensorflow-gpu. tensorflow-gpu can be thought of as a superset of tensorflow: tensorflow will work only on CPU, while tensorflow-gpu will work on both CPU and GPU, but will give GPU priority depending on the implementation of your code. For ease of use (and/or if you have no plans to use GPU), install the regular version. For full functionality (including planned usage of the gpu partition), install the GPU-enabled version, but be aware that if you have installed tensorflow-gpu but are planning to leverage CPU resources (for whatever reason), you will still need to submit to the gpu partition. It may be worth having two separate installations for GPU-enabled and CPU-exclusive Tensorflow depending on your personal requirements so that you can submit to other partitions if you don't need GPU resources for some parts of your work.So, choose one:

(nameofenv)$ pip install tensorflow

for CPU-enabled, or:

(nameofenv)$ pip install tensorflow-gpu

for the GPU- and CPU-enabled version.

If you chose the first option, you're done! You can immediately begin coding with Tensorflow.

If you chose the second option, and Recent builds of tensorflow package both CPU and GPU components.

If you plan to use the gpu (or related GPU-enabled) partition, a couple more steps are required for you to set up your code to leverage GPUs.

...

$ module load gcc/69.2.0
# then either of
$ module load python/2.7.12-ucs4
#
OR
$ module load python/3.6.0
# then9.14
$ source /path/to/nameofenv/bin/activate

...

# for safety, dump your current modules first
$ module purge
$ module load conda2miniconda3

Then, it is highly recommended that you create a new environment for Tensorflow (or your project that happens to use Tensorflow).:

$ conda env create --name tensorflowenvprefix /path/to/nameofenv python=3.6

...

9
# OR
$ conda env create --name nameofenv   # this will put the environment in the default location of $HOME/.conda/envs/nameofenv

Now, we activate the environment:

$ source activate tensorflowenv/path/to/nameofenv

You should see your terminal get modified with the environment name in parentheses just as above using virtualenv. If this command fails, you may need to specify the full path to the environment (especially if you used --prefix instead of --name to create it). From here, you can either use pip as above, or install Tensorflow with conda:

(tensorflowenvnameofenv)$ conda install tensorflow
# OR
(tensorflowenv)$ conda install tensorflow-gpu

Once this completes, you should be ready to use Tensorflow! Keep in mind that all of the above stipulations regarding leverage of hardware resources using the virtualenv process still apply to this method of installation and usage.

...

Depending on when you installed your copy of Tensorflow, you may see something like this when you run your code:

Loaded runtime CuDNN library: 7.0.4 but source was compiled with: 7.2.1. CuDNN ... Segmentation Fault

This is because the version of Tensorflow you installed used a newer version of CuDNN than the one that was found on the cluster. To fix this, you'll need to download a newer version of CuDNN and place it somewhere in a directory you own. You can submit a ticket with us if you'd like assistance with this issue.

Newer CUDA modules will have the associated newest CuDNN libraries included.