NOTICE: FULL O2 Cluster Outage, January 3 - January 10th
O2 will be completely offline for a planned HMS IT data center relocation from Friday, Jan 3, 6:00 PM, through Friday, Jan 10
- on Jan 3 (5:30-6:00 PM): O2 login access will be turned off.
- on Jan 3 (6:00 PM): O2 systems will start being powered off.
This project will relocate existing services, consolidate servers, reduce power consumption, and decommission outdated hardware to improve efficiency, enhance resiliency, and lower costs.
Specifically:
- The O2 Cluster will be completely offline, including O2 Portal.
- All data on O2 will be inaccessible.
- Any jobs still pending when the outage begins will need to be resubmitted after O2 is back online.
- Websites on O2 will be completely offline, including all web content.
More details at: https://harvardmed.atlassian.net/l/cp/1BVpyGqm & https://it.hms.harvard.edu/news/upcoming-data-center-relocation
Querying O2 Filesystem Quota Usage
- Kathleen Chappell
- Alex Truong
We have released a new tool on O2, called quota-v2, to check O2 filesystem usage and quotas. As of August 8, 2023 we have decommissioned the previous quota tool as it ran on older code and did not report all of the quota information that is relevant to the O2 user community.
Checking usage (quota-v2)
The quota-v2 command works similarly to the older quota tool.
In its default usage, quota-v2 will report your usage and the usage of groups that you are a member of that have O2-accessible directories and that have quotas imposed.
The output will look similar to this:
$ quota-v2
Usage for mfk8 (As of 2022-10-04 16:28:47 EDT-0400)
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ path ┃ type ┃ username ┃ usage ┃ storage limit ┃ last update ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ /home │ user │ mfk8 │ 44.76 GiB │ 100 GiB │ 2022-10-04 16:28:59 EDT-0400 │
├────────────────────────────────────────────┼───────┼─────────────────────┼─────────────┼───────────────┼──────────────────────────────┤
│ /n/groups/smith │ user │ mfk8 │ 18.67 GiB │ │ 2022-10-04 16:28:47 EDT-0400 │
├────────────────────────────────────────────┼───────┼─────────────────────┼─────────────┼───────────────┼──────────────────────────────┤
│ /n/groups/smith │ group │ │ 13.02 TiB │ 50 TiB │ 2022-10-04 16:28:47 EDT-0400 │
├────────────────────────────────────────────┼───────┼─────────────────────┼─────────────┼───────────────┼──────────────────────────────┤
│ /n/scratch3/users │ user │ mfk8 │ 56.43 GiB │ 10 TiB │ 2022-10-04 16:28:59 EDT-0400 │
└────────────────────────────────────────────┴───────┴─────────────────────┴─────────────┴───────────────┴──────────────────────────────┘
Each of the fields in the above output from quota-v2 are explained below:
Column | Meaning |
|---|---|
path | The path of the associated storage usage and/or limit |
type | Type of storage limit, either “user” quota or “group” quota |
username | The O2 username which corresponds to the usage and/or storage limit |
usage | The amount of data being stored in this path |
storage limit | The total maximum amount of data that could be stored at this path. If the usage reaches the storage limit, you may be unable add more data or modify data. |
last update | The timestamp representing when this quota information was last refreshed. The information from |
In the previous example, user mfk8 's usage was below the storage limit for each path they have access to. The quota-v2 report will show an additional “exceeded” table if you (or groups you are a member of) have reached or surpassed a storage limit. Here is an example:
Exceeded quotas
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ path ┃ type ┃ username ┃ usage ┃ storage limit ┃ last update ┃ grace remaining ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ /n/groups/smith │ group │ │ 100 GiB │ 100 GiB │ 2022-10-04 16:28:47 EDT-0400 │ None │
└─────────────────────────┴───────┴──────────┴─────────┴───────────────┴──────────────────────────────┴─────────────────┘
Please clear up the space in this path or you will lose write access!Note that the “exceeded” table has many of the same columns as in the standard “usage” table. One additional piece of information is reported: the “grace remaining”.
Column | Meaning |
|---|---|
grace remaining | The amount of time you have to clean up and reduce your usage to below the storage limit. You can delete files or move files to another storage location, as you see fit. If you (or members of your group) do not reduce the usage within the remaining grace period, the directory will “lock” and you will be unable to add/modify data under that path. Not all O2 directories have an associated grace period. If you hit/exceed a storage limit and you do not have a grace period, your directory will immediately be unable to hold additional data. |
Checking usage by path (quota-v2 path)
The quota-v2 tool also allows users to check information for a group directory that they are a member of. The previous quota tool had this functionality, but it required manually granting permission to individual cluster users. Here is an example for user mfk8's group directory:
$ quota-v2 path /n/groups/smith
Usage for /n/groups/smith (As of 2022-10-04 17:29:05 EDT-0400)
┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ type ┃ username ┃ name ┃ id ┃ usage ┃ storage limit ┃ last update ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ user │ bes123 │ Bob E. Smith │ 12345 │ 6.09 TiB │ │ 2022-10-04 17:29:05 EDT-0400 │
├───────┼─────────────────────┼───────────────────────────┼────────┼────────────┼───────────────┼──────────────────────────────┤
│ user │ djf987 │ Debra J. Fakename │ 9876 │ 48.65 GiB │ │ 2022-10-04 17:29:05 EDT-0400 │
├───────┼─────────────────────┼───────────────────────────┼────────┼────────────┼───────────────┼──────────────────────────────┤
│ group │ smith │ │ 1256 │ 6.75 TiB │ 50 TiB │ 2022-10-04 17:29:05 EDT-0400 │
└───────┴─────────────────────┴───────────────────────────┴────────┴────────────┴───────────────┴──────────────────────────────┘
With the quota-v2 path output, you can see how much storage space each member of your lab or group is utilizing within the directory. You can use this information within your group to prioritize who needs to clean up their data (deleting or moving to Standby or Cold storage), or to determine if you’d need to soon request additional storage space.
In the above output, there are a few additional fields when using the path subcommand:
Column | Meaning |
|---|---|
id | Numeric user ID or group ID |
name | The real first name and last name corresponding to an O2 cluster user |
Taking action on exceeded quotas
Because quota information only refreshes once per hour, if you write a lot of data or delete a lot of data within that window, the new usage will not be reflected in the table output immediately, but the actual, real-time storage occupied will change immediately. This means it is possible to exceed a quota even if the tool currently reports that usage is below that quota. If you are receiving errors while writing to that directory such as no space left on device, that is an indication that the target directory has exceeded its quota (and/or grace period), regardless of what quota-v2 is reporting at that given moment. In order to continue writing data to that directory, it will be necessary to free up space in the directory.
Options to do so include:
downloading to local storage
deleting temporary/unneeded files
moving data that does not need to be accessed in the medium term to Standby
requesting a storage increase for the directory
Feedback
Please reach out to us at rchelp@hms.harvard.edu with any bugs you may find, or questions about using quota-v2!