HMS RC is currently investigating an interaction between vendor software and the O2 filesystems that presently does not allow the removal of files via rm from a directory that has exceeded its quota.
This interaction is present on all O2-accessible filesystems, including /home, /n/data1, /n/data2, /n/groups, and /n/scratch3.
Symptoms
If you have observed a No space left on device error when trying to rm files from a directory that has exceeded its quota, that is due to this interaction.
Resolution
HMS RC is in communication with the vendor to resolve this problem. This page will be updated as status updates become available. We apologize for the inconvenience.
As of August 31, 2022, this interaction still exists.
Workaround
While HMS RC investigates this interaction, the following workaround is recommended to drop storage utilization below its prescribed directory quota to re-enable the usage of rm.
Identify large files/directories that you would otherwise have deleted to free up space
truncate them via the following commands:
for a single file:
echo > target_filename
for all files in a single directory:
find target_foldername -type f | xargs -l bash -c 'cat /dev/null > $0'
Once the above has been executed such that the directory has fallen back under its quota, you can use rm to delete the truncated files (and anything else identified to be cleaned up).
Note that the output of quota updates hourly; file truncations will therefore not immediately be reflected in the output of quota unless you are truncating at approximately the top of the hour. However, the storage will be freed immediately.