O2 Cluster Status

April-June 2025:

COMPLETED

O2 HPC cluster migration to Red Hat Enterprise Linux (RHEL) 9.x

• Please note: the O2 login process no longer requires lower case for the letters in the username.

• If you encounter any issues using RHEL, please contact Research Computing at rchelp@hms.harvard.edu. 

O2 HPC Cluster

Planned outage for the O2 cluster migration to Red Hat Enterprise Linux (RHEL) 9.x

• Websites hosted on O2 will remain online, but will not be able to submit cluster jobs.

O2 Cluster

June 28: Issue was resolved with the update to Red Hat Enterprise Linux.

June 18: HMS IT is still investigating the root cause, so slurm errors may still occur, but job scheduling is running more smoothly overall.

June 17: Around 10:30 AM this morning, O2 nodes began to close and slurm controllers became unresponsive. 
These issues may cause users to experience errors with job submission, slurm commands failing, and internal errors on O2 portal. 

HMS IT is investigating the cause of these issues.

O2 Cluster software: /n/app

A software freeze is in place, meaning that HMS Research Computing will no longer update any O2 applications accessible as a "module", and HMS IT will no longer update CentOS 7 software.

O2 Cluster

03-18 A brief storage outage around 9-11 PM on 03/13 has put a significant portion of the O2 Cluster in a state where no additional jobs will be accepted by affected nodes. Our Dev-Ops and Research Computing teams are working on restarting these nodes to resume normal operation. 
If your job has landed on one of these nodes it may experience some latency or not work properly. To work around this cancel the impacted job and re-submit the job to land on a different node.
03-19 After some investigation, our teams found an issue with the network file system connection on nodes. Addressing this will require a gradual reboot of most nodes on the cluster. To continue working around this issue, cancel the impacted job and re-submit the job to land on a different node.

We don't currently have a time estimate for when this will be complete. However, reboots may need to be run until next week (03/24-03/28). We will update this timeline as more information becomes available. 

03-27 A batch of high I/O jobs contributed to nodes to becoming slow to respond, causing Slurm to eventually refuse jobs on these nodes. These jobs have been discontinued, and the affected nodes have been restarted. 

O2 performance issues have now been resolved.

/n/files

To improve performance and keep our storage systems updated, HMS IT will migrate data on the research.files.med.harvard.edu server to a new storage array.

Outage window: Wednesday, January 22 (5pm) to Thursday, January 23 (approx. 9am-10am), 2025.

• This will only affect the O2 filesystem: /n/files

which is only accessible from the transfer servers (transfer.rc.hms.harvard.edu) and transfer compute nodes.

Full O2 outage

HMS IT is undertaking a project to relocate our data center within the Markley Data Center to optimize our IT infrastructure. This project will relocate existing services, consolidate servers, reduce power consumption, and decommission outdated hardware to improve efficiency, enhance resiliency, and lower costs. Users can bookmark the web page for this project, which will be updated as more information becomes available.

During this time:

• The O2 Cluster will be completely offline, including O2 Portal.

• All data on O2 will be inaccessible.

• Websites hosted on O2 will be completely offline, including all web content, not just job submissions.

• Other services reliant on O2 will be affected.

filesystem /n/standby

HMS IT is moving to a new Standby storage system designed to efficiently manage large amounts of data on Wednesday December 4^th, 9am – 5pm EST. Standby storage will be inaccessible during this period. 

• In the O2 environment, this maintenance only affects /n/standby on O2's file transfer systems. No other O2 research data or services should be affected. 

Be sure to save any work stored in these affected directories before the migration begins. Do not try to access files in these locations during the migration. The files will be available again after the migration is complete.  

O2 web, filesystem /n/www

As part of the HMS Research Data Migration Project,

The O2 Cluster will be online during this time because this change does not impact the Slurm job scheduler. 

During this outage on July 17 (9:00 AM to 12:00 PM):

• Websites and services hosted on O2 Web Hosting will be unavailable for the duration of the migration.

• The genomebrowser-uploads website will be unavailable

• The /n/www filesystem will be inaccessible

In preparation, on Tuesday July 16, at 4:00 PM EDT (the day before the outage):

• O2 compute nodes will not have access to /n/www. Any O2 cluster job relying on /n/www will fail.

• But until the full outage begins on July 17, websites will be up and /n/www will still remain available on O2 login nodes.

O2

A performance issue affected a number of O2 services, including:

• O2 Portal

• Jupyter Notebooks

• RStudio

• MATLAB

filesystem /n/groups

As part of the HMS Research Data Migration Project,

The O2 Cluster will be online during this time because this change does not impact the Slurm job scheduler. 

However,  /n/groups will be inaccessible during this period.

** Any running jobs which rely on accessing these filesystems will fail once the maintenance begins.

filesystems /n/data2 /n/no_backup2

As part of the HMS Research Data Migration Project,

The O2 Cluster will be online during this time because this change does not impact the Slurm job scheduler. 

However,  /n/data2 and /n/no_backup2 will be inaccessible during this period.

** Any running jobs which rely on accessing these filesystems will fail once the maintenance begins.

filesystems /n/data1 /n/cluster /n/shared_db

As part of the HMS Research Data Migration Project,

The O2 Cluster will be online during this time because this change does not impact the Slurm job scheduler. 

However,  /n/data1 ,  /n/cluster , and /n/shared_db will be inaccessible during this period. Keep in mind:

• /n/data1 contains shared folders for some groups that use O2.

• /n/cluster contains tools including: quota-v2, O2sacct, O2squeue, O2usage, scratch_create_directory.sh

• /n/shared_db contains public databases for research use

** Any running jobs which rely on accessing these filesystems will fail once the maintenance begins.

O2 Cluster

After a successful storage migration, pending jobs on O2 were allowed to dispatch as of 10AM, while login services started coming online as of 10:30 AM.

To provide more robust and reliable storage, 

During this time, the O2 Cluster will be offline. This means: 

• No jobs will run during the outage.

• O2 sign-in services will be unavailable.

• Jobs submitted to O2 through websites will not run until after the outage.

• Globus file transfer will be unavailable.

• Other services reliant on O2 will be affected.

Jobs scheduled to run during the outage will be postponed with ReqNodeNotAvail, Reserved for maintenance; they will start after the upgrade is complete. 

If a job needs to be completed before the upgrade, schedule it as soon as possible. 

O2 Cluster

An issue with the O2 storage environment affected access to use O2:

• O2 logins are unavailable

• O2 Portal is unavailable

• O2 transfer cluster is unavailable

• Currently running jobs may be affected

O2 Cluster

There was a HMS-wide network outage on the morning of Feb 5 which affected access to the O2 cluster as well as most other HMS services.

Please note that it is possible that O2 jobs running during the network outage were affected, depending on the type of job, and also the nature of the network outage, which is still being determined.

O2 scratch storage

To provide more robust and reliable storage, HMS IT has deployed a new storage cluster, designated as /n/scratch, to replace the current /n/scratch3. 

Please update your workflow to use/n/scratch by January 8, and see our documentation about changes in the new Scratch Storage

• HMS IT is not migrating any data from scratch3 to the new /n/scratch.

• Please copy any required data to the new space by January 16, when /n/scratch3 is retired.  

If you have any questions or concerns, contact Research Computing at rchelp@hms.harvard.edu

O2 Cluster

To enhance your experience with our network-based storage and prepare for future growth,

HMS IT will make upgrades during:

• start: Wednesday, December 6, 2023, at 8:00 AM

• end: Thursday, December 7 at 8:00 AM EST (UTC-5).

  • O2 services were restored at approximately 7:00 PM on Wednesday December 6.

During this time, the O2 Cluster will be offline. This means: 

• No jobs will run during the outage 

• O2 login services will be unavailable 

• Jobs submitted to O2 through websites will not run until after the outage 

• Other services reliant on O2 will be affected  

Jobs scheduled to run during the outage will be postponed; they will start after the upgrade is complete.

If a job needs to be completed before the upgrade, schedule it as soon as possible. 

If you have any questions or concerns, contact Research Computing at rchelp@hms.harvard.edu.

Standby Storage

HMS IT performed a gradual storage server upgrade on the HMS Standby storage server.

No impact is expected, but O2 users should avoid doing any large data transfers involving the Standby filesystem ( /n/standby ),

just to allow the upgrade to proceed as smoothly as possible.

O2 Portal, Group and website Storage

A storage outage affected the availability of the following filesystems:

• /n/data1

• /n/data2

• /n/log

• /n/no_backup2

• /n/shared_db

• /n/www

If your O2 jobs access any of these filesystems, they may fail and need to be re-run after the outage is resolved. You may also have problems cd’ing into or seeing data in certain directories. The data is safe; it’s just the access to the data from O2 that is not working. This outage may also affect O2 logins and access to the O2 Portal.

filesystems /n/data1/ /n/data2 /n/www /n/nobackup /n/shared_db /n/standby

Several storage filesystems serving the O2 cluster and related services were not responding. We temporarily suspended all pending and running jobs.

The Storage team investigated and resolved the issue.

/n/groups filesystem unavailable

Start Time: Thursday, July 13 at 7:00 PM

IMPACT:

Scheduled migration of /n/groups storage filesystem to the new hardware is taking longer than expected.

Due to this delay, the filesystem /n/groups is not be available at this time in O2.
Any new job submitted to O2 that requires the /n/groups filesystem will fail. Please do not submit new jobs requiring access to /n/groups

We will notify you once the storage migration is completed and /n/groups is available in O2.

If you have any questions or concerns, contact Research Computing at rchelp@hms.harvard.edu

--

Access to /n/groups has been restored on O2 login, compute, and transfer nodes. All files should be in the same state they were in on Thursday when the outage began. IT is taking steps to improve the type of situation that led to this issue.

PLANNED FULL O2 CLUSTER OUTAGE

To increase the efficiency and security of the O2 cluster, HMS DevOps will upgrade the Slurm job scheduler.

Maintenance Window

• START: Thursday, July 13 at 7:00 PM

• END: Sunday, July 16 at noon. (Completed at 07/17/2023 1PM)

This upgrade will require the O2 cluster to be offline, and as a result, no new jobs will be accepted during the mentioned period. To prevent disruption to your work, ensure all running jobs are complete before the upgrade commencement time.

Certain services related to the O2 cluster will be affected during the upgrade period. In particular:

• the O2 login servers at o2.hms.harvard.edu will be offline

• the O2 Portal will be offline

• you will not be able to submit or execute jobs, including from websites.

• the filesystem /n/groups will also be offline

However, not all services will be unavailable.

• The O2 transfer servers at transfer.rc.hms.harvard.edu will remain operational.

• any services not relying on the O2 job scheduler will continue functioning as usual during the upgrade period.

The upgrade is vital to keep our systems current with necessary security and bug fixes, resulting in enhanced performance for users. The process involves a database schema modification, which is time-consuming, hence the need for downtime.

If you have any questions or concerns, contact Research Computing at rchelp@hms.harvard.edu

O2 job scheduler

The slurm job scheduler is currently experiencing high loads and might fail to accept new jobs, often returning the error:

sbatch: error: slurm_persist_conn_open_without_init: failed to open
persistent connection to host:slurmdb-prod01.rc.hms.harvard.edu

--

The HMS IT DevOps team implemented a fix to remediate this issue

Globus file transfer

We will be performing an upgrade on the Globus File Transfer service, which is used on O2 to share data both internally and with collaborators from outside of HMS. This work will upgrade Globus from version 4 to 5, which will provide improved stability and scalability as well as new user-facing features to be announced. As Globus v4 will be considered end-of-life as of July 31, the upgrade is necessary to maintain support with the vendor.

Maintenance Window: Tuesday, May 2, 9AM – 5PM , (Completed by 3PM)

Impact:

• any collections created in HMSRC will be unavailable both internally to HMS users and to external collaborators

• all data transfers which are still in progress will be canceled

We ask that any necessary long-running transfers be completed prior to May 2. Please plan accordingly, contact your external collaborators, and reach out to us as soon as possible at rchelp@hms.harvard.edu if there are any concerns.

O2 network

HMS IT will be performing emergency maintenance on the O2 network to fix a critical issue. This will temporarily reduce O2's compute capacity.

Maintenance Window:

• SUNDAY, 2023-04-16, 06:00 AM , expected to be completed before 01:00 PM

Impact:

• Long-running jobs on affected compute nodes may be killed

• Job pending times may increase for the "short" and "medium" job partitions in the days before the maintenance

O2 network

There is an ongoing issue with networking gear in the data center that is impacting O2 jobs. In particular, many jobs will be unable to access storage and will either fail immediately or just hang until they time out. Some nodes may need to be rebooted, which will cause more jobs to fail. HMS IT is working to address the issue.

1:45pm update: All pending jobs on O2 (submitted today or before) have been paused for dispatch, except for jobs on the "interactive" partition to allow for some compute access. However, you might still experience problems accessing data from within those running interactive jobs.

5:30pm update: The issue was resolved and pending jobs were allowed to resume.

HMS network

During 6am - 9am on Sunday Mar 5, HMS IT is performing emergency maintenance on the core switches in the data center where O2 is hosted.

Hopefully there will be no impact to O2, but there is a possibility that this work may temporarily affect O2 cluster jobs, logins, or access to data.

Globus

The Globus Server software will be upgraded from v4.0.62 to v4.0.63. This change is necessary to update all security certificates to accommodate a Globus change to their certificate authority (CA) of choice. No other software modifications are made with this update.

Service Window: 12PM - 12:10PM. The process should only take a few minutes.

Impact:

• Access to the Globus web UI is not impacted, however the HMS-RC endpoint will be unavailable while the Globus services are restarted.

• Any Globus transfers started before the upgrade will pause momentarily, then resume once the endpoint is back online.

• This upgrade will not impact O2 jobs or access to O2 login nodes or the transfer cluster.

O2 Portal

The Open OnDemand software that powers the O2 Portal will be upgraded.

Service Window:

• Tuesday, Nov 8, between 8:30 - 9:30 am. The process should only take a few minutes.

Impact:

• The O2 Portal interface will not be available during the upgrade.

• O2 Portal Applications started before the upgrade should remain running and then be accessible again afterwards.

This upgrade will not impact O2 jobs, access to O2 login nodes, or access to the transfer cluster.

O2 cluster

A network outage at HMS affected access to O2 and other HMS services. HMS IT has addressed the core network issues.

Any cluster jobs which were running when the outage occurred may require being resubmitted, depending on the nature of the jobs.

scratch filesystem: /n/scratch3

2022-06-27:

If you still need to recover any data you had on /n/scratch3 on June 6, you MUST contact HMS Research Computing to request a data restore: rchelp@hms.harvard.edu. All we need is your HMS ID (the one you log into O2 with).

The old copy of scratch data will be REMOVED ON JULY 5 to provide space for the new /n/scratch3 to grow.

2022-06-10:

PLEASE CONTACT RCHELP@HMS.HARVARD.EDU IF YOU WISH TO ACCESS THE DATA YOU HAD IN /n/scratch3 AS OF JUNE 6.

• We believe that ALL data that was on the old scratch3 server as of June 6 has been copied to a safe location.

• We have restored data to individual scratch folders for dozens of users who have requested this.

• If your data has been restored, please move ("mv") it out of your _RESTORE directory rather than copying it, as space is currently limited.

• We will not delete old data under /n/scratch3 for at least 30 more days. At some point after this, we will resume the deletion process to recover space.

At some point in the future, we will need to delete the copy of data from June 6 (hundreds of TB) to recover space, so please contact us in the next week if you want that data back.

2022-06-08:

Update on the continuing /n/scratch3 outage:

• The interim scratch solution is up (on a different storage system) at /n/scratch3 and is being actively used by many researchers on O2.

• We are gradually recovering the data from the old /n/scratch3. We do not yet have a timeline, but we hope to have all of it recovered within a few days.

• Our current understanding is that no data were lost, except any new files being created as the outage began.

PLEASE EMAIL rchelp@hms.harvard.edu with your login name (HMS ID, like abc123) if you wish to get a copy of all the data in your /n/scratch3 directory. For logistical reasons, we can not provide self-service retrieval of that data.

• Data will not be deleted for 30 days after being restored, no matter when the files were last accessed.

We ask for your continued patience as the data is restored and HMS IT fulfills your requests.

2022-06-07:

Thank you for your patience while we investigate the sudden unavailability of our O2 scratch storage solution. HMS IT has identified the root cause and have implemented an interim solution for O2 scratch. You should be able to go back to using scratch storage for your O2 jobs now. We are investigating the impact to science that was running when we experienced this sudden outage, and will be reaching out to those affected labs to help address any impacts to research that were created when this incident occurred.

HMS RC has created new scratch folders for all O2 users under the same location, e.g. /n/scratch3/users/a/abc123 , but you will need to recreate directories underneath it.

Slurm job scheduler

O2's job scheduler software (slurm) will be upgraded from version 21.08.4 to 21.08.7

• Maintenance window: 05:00AM - 06:00AM

This is a minor update and we do not expect any significant impact to O2 users during the upgrade process, but at times slurm related commands (e.g. sbatch, srun) may be slow to load or could fail.

O2 login services, general access

An HMS infrastructure issue is affecting access to O2, including login services.

• was resolved by approximately 2:30pm

• some percentage of O2 cluster jobs were affected by the outage and failed

O2 login services

O2 logins were offline this morning due to a wider HMS authentication issue.

• was resolved by approximately 11:00am

GPU scratch filesystem: /n/scratch_gpu

We have decided to repurpose the dedicated GPU scratch space storage /n/scratch_gpu .This is because /n/scratch_gpu was only very lightly used by the community, and its 1 PB of storage can be more efficiently put to use increasing capacity on other O2 group filesystems.

Schedule:

• Dec 15:  /n/scratch_gpu will become read-only across O2. Existing data will remain in place.

• Jan 26:  /n/scratch_gpu will be removed from O2

We're sorry to remove this resource for those who have been making use of it. We encourage everyone to use O2's main scratch space under /n/scratch3 for GPU jobs in addition to regular jobs.

O2 Cluster and Storage

On November 19 and 20, HMS IT will be performing maintenance on both the storage servers and the O2 cluster for improved stability and performance. A full outage of O2 is required during which it will not be possible to run or submit jobs. Slurm will be upgraded, and compute nodes will be patched and rebooted. The storage server software will be updated.

We will configure O2 to not accept any job submissions that overlap into the outage window. For long running jobs, please be sure they complete before November 19.

The Maintenance Window is all day for both days:

• BEGIN: Nov 19, 12:01AM

• END: Nov 20, 11:59PM

Details:

O2/SLURM

• O2 will be unavailable for the duration of this outage.

• No jobs will able to be dispatched nor submitted.

• Login services to o2.hms.harvard.edu will be unavailable.

FILE TRANSFER SERVERS / GLOBUS

To allow access for data transfer on filesystems unaffected by the outage, file transfer servers (transfer.rc.hms.harvard.edu) and Globus will remain online.

The following filesystems will be offline for the duration of this outage:

• /n/standby, /n/no_backup2, /n/data1, /n/data2, /n/shared_db, /n/www

The following filesystems are not affected and will remain online:

• /home , /n/groups , /n/scratch3 , /n/scratch_gpu , /n/files

Web Hosting

Websites hosted by Research Computing (O2 and “Orchestra” hosting) will remain online, except during brief outages when the /n/www filesystem is affected by the storage maintenance:

• Nov 19: There will be a brief outage between 7-8am

• Nov 20: There will be a brief outage, but the exact time depends on the storage maintenance progress and is hard to predict.

• Also, any websites which use O2 to run jobs will be unable to do so throughout the outage.

O2 cluster storage

A storage outage affected availability of the following filesystems:

• /n/data1

• /www

• /n/no_backup2

• /n/shared_db

If your O2 jobs access any of these filesystems, they may die and need to be re-run after the outage is resolved. You may also have problems cd'ing into or seeing data in certain directories. The data is safe; it's just the access to the data from O2 that is not working.

O2 interactive logins and network

HMS IT will be updating and restarting network switches which serve the O2 cluster network from 9:00 PM EDT to 01:00 AM EDT.

IMPACT:

• Interactive logins to O2 login and transfer servers will be disabled, as those systems will require a restart. Any old screen/tmux sessions on these servers will be lost.

• Network storage mounts on O2 ( /home , lab and scratch storage) may potentially be unstable on individual systems. Any affected mounts will be restored and verified during the maintenance window.

• New cluster jobs should be prevented from running though the maintenance window. Any jobs which are still running and cannot tolerate periods hung network storage may be impacted.

Orchestra production Database Servers

There will be an emergency maintenance of Orchestra production database services from 9:00 PM EDT to 11:59 PM EDT. This maintenance is for security remediation on the database servers. The database services that will be offline during the maintenance period are listed here:

• mysql.orchestra

• pgsql.orchestra

• pgsql96.orchestra

Orchestra development and staging Database Servers

There will be an emergency maintenance of Orchestra development and staging database services from 9:00 PM EDT to 11:59 PM EDT. This maintenance is for security remediation on the database servers. The database services that will be offline during the maintenance period are listed here:

• dev.mysql.orchestra

• dev.pgsql.orchestra

• dev.pgsql96.orchestra

• stage.pgsql.orchestra

• stage.mysql.orchestra

Weekly jobs report / Job priority rewards

The weekly O2 report email and the extra priority reward QoS are currently unavailable due to a problem with the Slurm database queries.

O2 cluster

HMS IT will be performing a storage server upgrade that will impact the O2 Cluster. The new storage, which has all flash drives and more current hardware, will both improve the performance and reliability of O2’s storage and replace aging infrastructure. This is the first stage of a two-phase upgrade to improve O2’s storage.

A full outage of the O2 job scheduler is required, as some of the migrating data are used by the Slurm scheduler itself. While it will not be possible to run jobs during the outage, unaffected data will remain accessible via O2’s file transfer servers. We will be configuring O2 to not accept any job submissions that overlap into the outage window. For long running jobs, please be sure they complete before March 3.

Maintenance Schedule:

• Wed. March 3, 9:00am – 5:00pm

The following services will be offline:

• Access to all O2 login servers at: o2.hms.harvard.edu

• O2 job submission and execution, including from websites which are otherwise not affected.

• The following filesystems:

•  

  • /n/data1

  • /n/shared_db

The following services will remain online:

• Login access to the transfer servers: transfer.rc.hms.harvard.edu

• Access to all unaffected filesystems, including:

•  

  • /home

  • /n/groups

  • /n/data2

  • /n/scratch3

  • /n/no_backup2

  • /n/www

Slurm scheduler

The Slurm job scheduler on O2 experienced an outage overnight. Resolved ~ 11:45am with an upgrade to Slurm to fix a bug in the scheduler.

Impact:

Job submissions and other Slurm commands (e.g. sbatch , srun, squeue) have not been functioning for several hours. Many jobs did continue to run, although some may have failed.

O2 database services

In order to perform an urgent and critical infrastructure migration, the virtual machines hosting several legacy databases will be must be shut down for a period of time overnight.  During the window, the virtual machines will be migrated to a different storage backend. 

The outage window will be from 8:00 PM on Thursday, 2/18 until 8:00 AM on Friday, 2/19.  There will be no other changes to the database servers at this time.

The following database servers will be affected:

mysql.orchestra
stage.mysql.orchestra
dev.mysql.orchestra
crhealth.mysql.orchestra
pgsql96.orchestra
dev.pgsql96.orchestra
pgsql.orchestra
dev.pgsql.orchestra
stage.pgsql.orchestra

Slurm scheduler

1:15pm - 3:15pm (resolved)

There is a currently a problem with the O2 cluster Slurm job scheduler. Jobs already running on compute nodes are not affected, but you may get errors when trying to submit new jobs.

O2 cluster network

HMS IT will be performing priority network maintenance to correct a bug in affected network switches at the Markley Data Center where O2 is hosted.

MAINTENANCE WINDOW:

• Sunday, Jan 17, 5:00 am– 10:00 am

IMPACT:

• Some sets of O2 compute nodes will lose network connectivity for about 30 minutes during the window, when the switches get restarted. Affected nodes have names beginning with either of:

•  

  • compute-e

  • compute-p

• All remaining compute nodes should not be affected, as they have redundant network hardware which should mitigate the impact.

We have configured O2 to not submit any new jobs to the affected nodes, so the maintenance should only affect longer jobs which are already running on compute-e and compute-p nodes. 

/www and websites hosted by Research Computing

/n/no_backup2

A storage issue is affecting the availability of the following filesystems on O2:

• /www

• /n/no_backup2

The /www outage is resulting in most RC-hosted websites being offline.

Slurm scheduler

The Slurm job scheduler on the O2 HPC cluster is currently having a performance issue and Slurm commands (e.g. sbatch, srun, squeue) may be unavailable.

Impact:

• New job submissions may not work

• Slurm commands may be unresponsive

• Jobs already running should continue without problems.

/n/files

To improve performance and keep our storage systems updated, HMS IT will migrate data on the research.files.med.harvard.edu server to a new storage array.

Outage window: Saturday, November 14, 2020, from 8:00 AM to 8:00 PM

• This will only affect the O2 filesystem: /n/files

which is only accessible from the transfer servers (transfer.rc.hms.harvard.edu) and transfer compute nodes.

O2 cluster

On Saturday, September 26, 2020, from 6 AM to 1 PM EDT, HMS IT will be completing a strategic network upgrade which will increase the HMS campus internet connectivity from 40 to 100 gigabits per second. This upgrade improves support for data-intensive science, online education, and remote work.

The O2 cluster will remain fully operational. However, there is the potential for issues related to O2’s authentication service during the maintenance. This could result in any of the following issues:

• Difficulty logging into O2

• Authentication timeouts

• New job submissions could be slow or may fail

Jobs which are already running are expected to continue without any problems.

O2 authentication

Intermittent problems with authentication for O2 login, transfer, and compute nodes.
This issue can potentially result in: Slow or failed logins to O2, Missing group membership, Failed job submissions

O2 cluster

HMS IT will be performing minor maintenance on the O2 cluster which is expected to improve the responsiveness of the SLURM job scheduler (see outage notes for 8/9/2020)

MAINTENANCE WINDOW:

• Wednesday, Aug 26, 8:00 am– 9:00 am

IMPACT: 

• You will not be able to submit new jobs to O2 during this time.

• SLURM commands (squeue, sbatch, srun, etc. ) will likely fail with a timeout error.

• Already running jobs will continue to run normally.

O2 cluster

New jobs are intermittently not starting on the cluster (or the sbatch command has errors) due to an issue with cluster-storage communication. We believe that currently running jobs are still executing normally. Disk read/writes may be slower than usual, which can cause other commands to be slow. We will provide details as we get them.

Full O2 cluster

Unplanned SLURM outage, due to unbalanced file system allocations on a primary storage cluster. Service restored 3pm

/n/no_backup2

Scheduled Maintenance window: 2020-07-29 5:00 PM to 2020-07-30 5:00 PM

HMS IT will be migrating data from /n/no_backup2 to a newer filesystem.