NOTICE: FULL O2 Cluster Outage, January 3 - January 10th

O2 will be completely offline for a planned HMS IT data center relocation from Friday, Jan 3, 6:00 PM, through Friday, Jan 10

  • on Jan 3 (5:30-6:00 PM): O2 login access will be turned off.
  • on Jan 3 (6:00 PM): O2 systems will start being powered off.

This project will relocate existing services, consolidate servers, reduce power consumption, and decommission outdated hardware to improve efficiency, enhance resiliency, and lower costs.

Specifically:

  • The O2 Cluster will be completely offline, including O2 Portal.
  • All data on O2 will be inaccessible.
  • Any jobs still pending when the outage begins will need to be resubmitted after O2 is back online.
  • Websites on O2 will be completely offline, including all web content.

More details at: https://harvardmed.atlassian.net/l/cp/1BVpyGqm & https://it.hms.harvard.edu/news/upcoming-data-center-relocation

Submitting data to GEO

Owned by Kathleen Chappell
Last updated: May 12, 2023 by Gaby Rodriguez
2 min read

Often, researchers will need to submit their experimental data to the NCBI database Gene Expression Omnibus (GEO) prior to a publication or at the completion of a study. GEO accepts data from such methods as ChIP-seq, RNA-seq, and bisulfite sequencing, among many other types.

This page is a guide to transferring your data, that is stored on the O2 cluster, to NCBI's FTP server. Please note that this page is not a replacement for the submission instructions provided by GEO; it is a supplement that clarifies how to perform the transfer process using HMS RC resources.

Here is a description of the transfer and submission preparation process:

  1. Ensure your experiment and data type is accepted by the GEO database. See here and here for more details.

  2. Create an NCBI account if you do not already have one.

  3. Collect the files necessary for the GEO submission process - metadata spreadsheet, raw data, and processed data files. These files should all be on the O2 cluster.

    1. To aid in transfer times, you can compress your raw data files with either bzip2 or gzip.

    2. You should include md5 checksums for your raw data files in your metadata spreadsheet; these will be used to identify if any files were corrupted or transferred incompletely. You can calculate these checksums with the md5sum command on the O2 cluster.

  4. Create a folder named with your NCBI username, and move all of the files you want to submit there.

  5. If you will transfer 1TB or more of data, contact GEO before you do so. See here.

  6. Transfer the data to GEO using the transfer cluster: transfer.rc.hms.harvard.edu 

    1. Your username and password for the transfer cluster are the same credentials you use to log in to O2 (HMS ID and password). Your username must be in lowercase.

    2. You can use lftp to connect to the NCBI FTP server. To obtain the appropriate lftp command, log into GEO and navigate to this page. The command will be listed under "Uploading your submission" > "FTP instructions" > "Linux/Unix" > "Here is a typical 'lftp' session". The command will be in the form of: lftp ftp://username:password@server

    3. Once you're connected to the server, you can transfer your data like so: mirror -R NCBIaccount_directory

  7. Send an email to NCBI when your transfer has completed. More information can be found here.



If you have any difficulty following these instructions, feel free to contact us!



Digital Accessibility