User Training

Intermediate O2 is for current O2 users who would like to brush up on their bash skills, learn more advanced file transfer techniques, and unleash some of the powerful features of the SLURM scheduler.

This course introduces Git and GitHub and covers topics including: Getting Started with Git for version control, Using GitHub Desktop effectively, Collaborating with others on GitHub, and Utilizing GitHub Flow for better workflow. No previous exposure is assumed. We hope attendees will leave the class with the knowledge and tools necessary to start integrating Git into their workflows and excited to begin collaborating on GitHub.

Matlab has become the “language of science” in the past few decades. It is simple to use, yet powerful enough to be productive on large computing infrastructures. If you need: 1) Fast prototyping of research ideas; or 2) avoid spending too much time in coding instead of doing real science by taking advantage of Matlab’s built-in functions; 3) User friendly graphical interface and educational documentation; 4) Simplicity of code; 5) Easy access to GPU computing power; 6) Easy plotting and presentation of data; you will find this introduction course useful. This course will introduce the basics of the MATLAB coding language with O2-scalability and data presentation.

Intro to O2 addresses the needs of users who have very little linux experience, and are just getting started with the O2 high performance compute cluster. We will cover linux basics, and the concept of the Slurm scheduler and jobs, as well as data management best practices.

This is a short introduction to Parallel Computing that will include an overview of the basic concepts of parallel programming: from running your job in an embarrassingly parallel way to writing simple shared and distributed memory parallelization codes in different languages. The seminar will cover several examples of actual parallel codes however it will not have any "hands on" components. A basic programming experience (of any language, no parallelization) is preferred in order to better follow the topics presented during the seminar.

Python is a popular scripting language for scientific computing and available across all computer platforms. The course will introduce you to some of the basics of the Python language as well as some of the nuances involved with its use specific to the O2 environment. The goal is to provide users with a foundational level of familiarity. Topics covered include basic data types and declaration, flow control (if/else), loops, a brief introduction to constructing a script, and a briefer introduction to modules. The course will be taught on O2, but general concepts are easily translatable to desktop and local installations.

Intro to using R and Bioconductor. R is a powerful, open-source, highly adaptable statistical language useful for crunching numbers to datasets like those produced by next-gen sequencing. This class covers R basics and learning to think like/understand R. Users will learn how to set up personal R libraries on O2, and use O2 R for its high memory allocations and parallelization. Topics include how to install packages, learn about variables, data types. data manipulation, flow control, and functions, perform simple statistical tests, and create a variety of plots. Laptops are encouraged.

This course is designed to showcase the Globus specific data transfer and sharing functionality that is available to anyone with o2 access. We will be going over not only how to share data with collaborators outside of HMS, but also how to receive data from outside collaborators. The class will delve into the Globus features that are available via the Globus website at first and eventually transition into the functionality provided by the globus cli.

Description:
O2 Portal is web-based access point to O2 powered by Open OnDemand. This short training teaches you the basic functions of O2 Portal, including file transfer, Slurm job submission and monitoring desktop activities and a few popular applications such as Matlab, Jupyter and Rstudio.
Audience:
Anyone interested in using the O2 resources for their projects.
Prerequisites:
Familiarity with the O2 cluster and the Slurm scheduler.

Description:
In this short seminar we cover some simple but essential rules to use the O2 cluster in the most efficient (and inexpensive ) way.
Audience:
Anyone running jobs in the O2 cluster
Prerequisites:
Familiarity with the O2 cluster and the Slurm scheduler.

RCBio is a HPC pipeline runner and workflow tool supported by HMS RC consultants. It was first designed for a Neubiology lab and later opened to all O2 users. RCBio helps users to quickly build and run multiple steps data analysis pipeline on a computer cluster. RCBio automatically keeps logs on all job status and use email to notify user. When re-run the same pipeline, RCBio automatically submit failed jobs and ask user if she/he want to re-run the successful jobs. Here is our wiki page about the RCBio project: https://harvardmed.atlassian.net/wiki/spaces/O2/pages/1600651373

The “Troubleshooting O2 Jobs” class will teach you how to debug problems when running analyses on the O2 High Performance Compute Cluster. We will examine faulty job submissions to give you hands-on problem solving experience. You will learn how to request appropriate resources and set up your environment to successfully execute your jobs.

Prerequisites:

We recommend taking the “Intro to O2” class prior to “Troubleshooting O2 Jobs”.

Description:
As research grows increasingly more complex, arrangements for data storage and transmission of research data require continuous review and modification. Data storage encompasses not only the specific location where data will be kept and managed, but also aspects of data privacy, data security, data organization, and eventual sharing or reuse.

During this short training seminar, we will discuss:

• How to incorporate storage management into your research workflows

• Where to properly store data based on how the data will be generated and utilized

• Highlight the available storage options at Harvard Medical School

• Learn how to protect your data from unauthorized access or data loss

• Understand how to incorporate your storage selections into your NIH Data Management and Sharing Plans for grants 

Audience:

Primarily focused on HMS related tools and materials. Open to members of the Longwood Medical Area community, including researchers, faculty and staff.

Prerequisites:

None

Topic: Dynamically optimize your jobs' RAM and run time requests with SmartSlurm

Description:
SmartSlurm is an automated computational tool designed to estimate and optimize resources for high performance computing jobs run with the Slurm scheduler. The HMS O2 and Longwood clusters use the Slurm scheduler for job resource allocation.

There are two major parts to SmartSlurm:

ssbatch: An sbatch wrapper with a custom function to estimate memory RAM and time based on several factors (i.e., program type, input size, previous job records). Once the memory and time values are estimated, jobs are submitted to the scheduler while keeping a record of the jobs history and sending an optional email notification.

runAsPipeline: It parses bash scripts to find user defined commands and calls ssbatch to submit jobs to Slurm. It handles job dependencies. It also automatically keeps logs on all job statuses and uses email to notify the user. If jobs run out of time or out of memory, jobs will be resubmitted asking for more resources.

The code is available at the GitHub page: https://github.com/ld32/SmartSlurm

During this short training seminar, we will discuss:

• How SmartSlurm works

• How to run simple pipelines with SmartSlurm

• How to use SmartSlurm with your own data

Audience: Open to members of the Longwood Medical Area community, including researchers, faculty and staff.

Prerequisites: Having basic Linux command line skill, know how to login to the O2 cluster and submit Slurm jobs.

Overview

We will introduce parallel and distributed computing with a focus on speeding up application codes.  By working through common scenarios and workflows using hands-on demos, you will gain a detailed understanding of the parallel constructs in MATLAB, their capabilities, and some of the common hurdles that you'll encounter when using them.

Highlights

• Multithreading vs multiprocessing 

• When to use parfor vs parfeval constructs 

• Creating data queues for data transfer 

• Leveraging NVIDIA GPUs 

• Working with large data 

Who Should Attend

Researchers, scientists, and students interested in advancing the pace of their research by using parallel computing strategies.

Overview

Medical images are obtained from various sources including MRI, CT, X-ray, ultrasound, and PET scans. Analyzing these images necessitates a comprehensive environment for data access, visualization, processing, and algorithm development. A key challenge involves extracting quantitative features to generate clinically relevant information using advanced techniques like machine learning algorithms.  This presentation will explore radiomics, which captures characteristics not typically visible. Radiomics features can be employed across various medical imaging modalities and applications, enabling the study of associations between imaging features and patient biology, as well as the prediction of clinical outcomes, making radiomics a versatile technique in medical imaging. These features quantify shape, intensity, and texture characteristics within medical images, reducing reliance on subjective interpretation for clinical workflows.

Highlights

In this session, you will learn how to:

• Import and manage large sets of images without loading them into memory,

• Extract shape, intensity, and texture radiomics features,

• Reduce the number of extracted features for classification,  

• Build a deep learning network to classify images, and

• Review and analyze the results.

Class Materials

Link to class materials

Academic Software Platform (ASP) integrates two major stacks of scientific software: BioGrids stack of ~400 biomedical applications (https://biogrids.org/software/) as well as SBGrid stack of ~500 structural biology applications. Multiple versions of applications are maintained, and users can also easily install the same stack of software on laptops, research workstations or cloud resources, under Linux or Mac OS X operating systems. More information about this resource is available on O2 wiki: https://harvardmed.atlassian.net/wiki/spaces/O2/pages/1630994700/Using+Software+Provided+by+BioGrids

Overview:

 During this hands-on workshop, you will be introduced to parallel and GPU computing in MATLAB for speeding up your applications and offloading computations.  By working through common scenarios and workflows, you will gain an understanding of the parallel constructs in MATLAB, their capabilities, and some of the issues that may arise when using them.

Highlights:

• Speeding up programs with parallel computing

• Working with large data sets

• GPU computing

• Scaling to your HPC cluster

Requirements:

• Basic working knowledge of MATLAB

• A MathWorks Account.  If you do not already have a MathWorks Account, please go to https://www.mathworks.com/login to create one.

• A supported browser https://www.mathworks.com/support/requirements/browser-requirements.html

• As a precursor, please watch the following 30-minute video for an introduction on key MATLAB parallel computing concepts:  Parallel Computing Hands-On Workshop (28:06) 

MATLAB Parallel Server™ lets you scale MATLAB® programs and Simulink® simulations to clusters and clouds. This two-part, hands-on workshop is designed to introduce users to parallel computing constructs in MATLAB and to show them how they can scale their jobs to the HPC clusters on campus.

• Scaling MATLAB to the BigPurple cluster – users will follow along as instructors demonstrate how to configure MATLAB to submit to the   cluster

• Running single- and multi-node MATLAB jobs

• Differences between parpool and batch job submission

• Non-interactive submission via Slurm

In this workshop we invite users of the HMS Research Computing Cluster O2 and members of the Harvard community who are interested in using a compute cluster to join us as we demonstrate some very helpful tips and best practices. We will introduce participants to various commands and approaches to help effectively navigate use the cluster and complete tasks in an efficient manner. We encourage participants to log on to the cluster and follow along interactively, however attendees can also watch the demonstration. This workshop is being held in collaboration with HMS Research Computing, and is an advanced workshop requiring knowledge of the command-line and/or the basic shell skills learned in The Foundation - Basic Shell.