Deleting Files from a Directory on O2 After Hitting the Storage Limit

AS OF NOVEMBER 23, 2022: this interaction should no longer occur and should be resolved on all O2 hosts. Please inform HMS Research Computing if you observe this interaction after the aforementioned date (see Resolution below).

HMS RC is currently investigating an interaction between vendor software and the O2 filesystems that presently does not allow the removal of files via rm from a directory that has exceeded its quota.

This interaction is present on all O2-accessible filesystems, including /home, /n/data1, /n/data2, /n/groups, and /n/scratch3.

Symptoms

If you have observed a No space left on device error when trying to rm files from a directory that has exceeded its quota, that is due to this interaction.

Resolution

HMS RC is in communication with the vendor to resolve this problem. This page will be updated as status updates become available. We apologize for the inconvenience.

As of November 23, 2022, this interaction should no longer occur. If you observe this interaction anywhere after the aforementioned date, please contact rchelp@hms.harvard.edu with details.

Workaround

While HMS RC investigates this interaction, the following workaround is recommended to drop storage utilization below its prescribed directory quota to re-enable the usage of rm.

1. Identify large files/directories that you would otherwise have deleted to free up space

2. truncate them via the following commands:

for a single file:

for all files in a single directory:

Once the above has been executed such that the directory has fallen back under its quota, you can use rm to delete the truncated files (and anything else identified to be cleaned up).

Note that the output of quota updates hourly; file truncations will therefore not immediately be reflected in the output of quota unless you are truncating at approximately the top of the hour. However, the storage will be freed immediately.