Page Comparison

In order to keep filesystems from filling up and disrupting work, we use filesystem quotas to limit usage in certain areas by user or group. This also helps us observe growth in disk usage over time so we can plan future expansion.

By default, the filesystem quotas are as follows:

* scratch_gpu is only available for labs whose PI has a primary or secondary appointment in a pre-clinical HMS department.

Checking UsageChecking Usage

You can use the quota-v2 and du commands to check filesystem usage.

Usage by User and Group

The quota-v2 command on O2 will show your usage and usage by groups of which you are a member for directories (accessible on O2) that have quotas imposed.

Type quota-v2 at the command prompt on any O2 system. The output will look something like:

  /home [i]                                      Usage for mfk8 (As usageof 2022-10-04 16:28:47 EDT-0400)
warning┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
     limit
┃ path  =====                                     ┃ type  ┃ username     -----    -------   ┃ usage  -----    user mfk8┃ storage limit ┃ last update                  ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
(uid 5005)      75GiB      95GiB     100GiB *
  
  /n/groups [f]│ /home                                      │ usageuser  │ mfk8 warning      limit   =========      │ 44.76 GiB   │ 100 GiB                             -----    -------      -----
  group smith│ 2022-10-04 16:28:59 EDT-0400 │
├────────────────────────────────────────────┼───────┼─────────────────────┼─────────────┼───────────────┼──────────────────────────────┤
│ /n/groups/smith                            │ user (gid 3204)│ mfk8    142GiB    user mfk8       │ 18.67 GiB   │             (uid 5005) │ 2022-10-04 16:28:47 EDT-0400 │
17GiB├────────────────────────────────────────────┼───────┼─────────────────────┼─────────────┼───────────────┼──────────────────────────────┤
│ 
  /n/groups/smith [f]                           │ group │   usage    warning      limit   ===================     │ 13.02 TiB   │ 50 TiB                    -----    -------      -----│ 2022-10-04 16:28:47 EDT-0400 │
├────────────────────────────────────────────┼───────┼─────────────────────┼─────────────┼───────────────┼──────────────────────────────┤
│ /n/scratch/users      directory                     │ user  │ mfk8               142GiB │ 56.43 GiB   │ 25 TiB        200GiB

  clusters
  ========
  [ f - itisimdcp05 as of 2021-05-10 13:00:01 ]    based on default quota -- *
  [ i - itisimdcp10 as of 2021-05-10 13:00:00 ]       usage exceeds limit --- !│ 2022-10-04 16:28:59 EDT-0400 │
└────────────────────────────────────────────┴───────┴─────────────────────┴─────────────┴───────────────┴──────────────────────────────┘

You will see a number representing your individual storage usage in your /home directory on the "user" line above with the 100GiB quota. If you’re using /n/scratch, you’ll see a line for your personal usage in your sub-directory within /n/scratch/users.

Note: It is against RC policy to artificially refresh last access time of any file located under /n/scratch.

For more information on the "directory" line. Also, if scratch, please refer to the dedicated scratch wiki page.

If you are a member of a group directory, you'll have a line for your “user” usage above the "directorygroup" line showing all group members' usage.

If you have reached a quota limit, you will see an exclamation mark or ! at the end of the line for the directory that has exceeded the quotaextra “exceeded” table which denotes which paths have exceeded their quotas.

The quota data is updated on an hourly basis. You can tell how recent the data you're viewing is from the timestamp at in the bottom last column of the quota-v2 output. It is possible to hit a quota limit (e.g. in your home directory or your group directory) but not have the output from the quota-v2 command reflect this problem for about an hour due to the information being periodically updated.

scratch3 quotas

For data on /n/scratch3, you need to use the scratch3_quota.sh command:

$ /n/cluster/bin/scratch3_quota.sh
Directory: /n/scratch3/users/m/mfk8
Space used: 0TiB used of 10TiB
Files/directories: 1256 of 1000000

Note that "limit" is the total amount of space you are allowed to use. ("quota" is actually describing a "soft quota" where you get a warning but can still write. The /n/scratch3 filesystem doesn't use these.)

Note: It is against RC policy to artificially refresh last access time of any file located under /n/scratch3.

For more information on scratch3, please refer to the dedicated scratch3 wiki page.

scratch_gpu quotas

Quota utilization for /n/scratch_gpu is reported by the command quota if used from a node where the /n/scratch_gpu filesystem is available (login, transfer or compute-g nodes)

Usage by Directory

Another way to check usage is to total the size of files in a directory using the du command. For example, you might want to see how much space your sub-directory in your group's shared directory is consuming:

• To check the size of a directory (e.g. /n/groups/smith/mydirectory ):
  

  • Run the command: du --apparent-size -hs /n/groups/smith/mydirectory

  • The output returned is the total size.

• Note that du can take quite some time for directories containing large numbers (tens of thousands or more) of files, because it must check the size of every file to compute the total. In general, it is better to use quota-v2 to find usage information, when possible, or at least to run du on sub-directories instead of top-level directories.

• The --apparent-size option is required to find files' actual sizes. Without this option, the reported size will include data protection overhead (redundant copies of data on the O2 file server, which protects against hard drive failures).

• Please do not run du from a login node. Long running and computationally intensive processes will be killed on login nodes. To ensure that your command for checking directory usage is not interrupted, please run du from a compute node instead. You can use the srun --pty command to start an interactive job, and then run du once you have been allocated resources. More information on running SLURM jobs can be found on the Using Slurm Basic wiki page.

...

You can verify that you are over quota by running the quota-v2 command. If you see an ! a red exceeded table at the end of a line of the output, then it means you have hit or exceeded a limit.

...

Use the commands above to confirm that you are above your quota, and delete data as needed to let you write new files again.

Note that the quota-v2 command results are only updated hourly. If you were writing files very rapidly, the quota-v2 command might not show a completely full quota. Also, deleting files won't immediately change the results from that command. If you delete 5 GiB of files, you should be able to write 5 GiB of new files in that location immediately, even if quota-v2 hasn't caught up yet.

You can delete a whole directory with a command like rm -rf dir. Please be careful when using a command like this: you could delete all of your files!

...