Scratch3 Storage

Owned by William (Gaby) Rodriguez
Last updated: Jan 19, 2024 by Andy Bergman

To provide more robust and reliable storage, HMS IT has deployed a new storage cluster, designated as /n/scratch, to replace /n/scratch3. 

The /n/scratch3 filesystem was retired on Jan 16, 2024

The timeline for this update was:  

  1. November 13 – Beta access to new /n/scratch for preliminary testing. 

  2. December 8 – Full access to /n/scratch for all users. The existing /n/scratch3 will temporarily remain available. 

  3. January 8 – /n/scratch3 becomes read-only. 

  4. January 16 – /n/scratch3 is retired. The path /n/scratch3 will no longer exist on O2 and no data will be recoverable from the old /n/scratch3

Please update your workflow to use/n/scratch see our documentation about changes in the new https://harvardmed.atlassian.net/wiki/spaces/O2/pages/2652045313

If you have any questions or concerns, contact Research Computing at rchelp@hms.harvard.edu

Scratch3 Overview

Scratch3 Configuration

Scratch3 is a storage location on O2 for temporary data. In summary:

  • THE SCRATCH SPACE IS NOT BACKED UP!

  • Files are AUTOMATICALLY DELETED 30 days after they are last accessed.

  • There are NO SNAPSHOTS available for scratch3 folders.

  • O2 users can have up to 10 TiB and 1 million files/directories in their scratch folder.

  • Use this space for intermediate files in a longer workflow, or for extra output that doesn't matter much.

To reiterate, if you delete a file on purpose or by accident, or just leave it sitting for 30 days, it is GONE. You can't get it back, and Research Computing can't get it back for you. Using this model lets us provide cost-effective, high-end storage to many more users than would be possible with permanent and backed-up files. But it is important that you understand the risks of using it.

Important - Artificially modifying file access times is against policy and may result in losing your access to scratch3.

Scratch3 Directories: Users

On scratch3, user directories are found under:

  • /n/scratch3/users/<first_hms_id_char>/<HMSID>

Where "<first_hms_id_char>" is the first character of your HMS ID (formerly eCommons ID) and  "<HMSID>" is your HMS ID.

 

For example, the scratch3 directory for a user with the username "abc123" would be located at:

  • /n/scratch3/users/a/abc123

 

For an HMS ID of zz999 the scratch3 directory will be at:

  • /n/scratch3/users/z/zz999

Scratch3 Storage Usage

The storage usage is on a user basis rather than a group. To check your personal and group storage utilization and limits, please run the following command:

quota-v2

More information about the O2 quota-v2 command can be found on this wiki page.

HMS Research Computing retired the old tool for reporting storage utilization and limits called quota on August 8, 2023.

The quota-v2 tool retrieves more comprehensive information than the previous quota tool, executes faster, and runs on all O2 nodes (login, compute, and transfer cluster).

Digital Accessibility