Page Comparison

...

• scp, sftp, rsync - these are automatically installed on Mac and Linux

• pscp, psftp - Windows-only. These can be installed with the PuTTY ssh program.

• ftp - available on O2 for downloading from external sites which only accept FTP logins. But, O2 does not accept incoming FTP logins.

• aspera - a data transport and streaming technology, now owned by IBM.

• awscli - Amazon AWS command line interface

• basemount - an Illumina tool to mount BaseSpace Sequence Hub data.

• bbcp - a point-to-point network file copy application from NERSC
  

• lftp - can transfer files via FTP, FTPS, HTTP, HTTPS, FISH, SFTP, BitTorrent, and FTP over HTTP proxy.

• gcloud - Google Cloud command line interface, including the gsutil command

• NBIA Data Retriever - a tool for downloading data from the TCIA Data Portal , installed under /opt/NBIADataRetriever

• rclone - rsync for cloud storage

• Globus - If the other side support Globus

For graphical tools, see the documentation that came with the program. Also, see our instructions on how to use these tools with two-factor auth. Many tools will by default copy somewhere in your /home directory, which has a small 100GB 100GiB storage quota. Make sure to explicitly specify whether you want to copy there or to a different location like: /n/scratch3scratch/users/m/mfk8/

If you just have a single file to copy and you're on a Mac, you can also run a command like the following from the Terminal application:

me@mydesktop:~$ scp myfile my_o2_id@transfer.rc.hms.harvard.edu:/n/scratch3scratch/users/m/mfk8/

By default, scp will copy to/from your home directory on the remote computer. You need to give the full path, starting with a /, in order to copy to other filesystems.

...

You can connect to the transfer nodes using ssh at the hostname: transfer.rc.hms.harvard.edu . If you're on Linux or Mac, you can use the native terminal application to connect to the transfer nodes. If you're on Windows, you will need to install a program to connect to the transfer servers; we recommend MobaXterm. In either terminal or MobaXterm, type the following command:

...

The O2 login nodes and most compute nodes do not currently mount /n/files. There are 2 ways to access this filesystem from O2:

1. Use O2's dedicated file transfer servers: 

   1. SSH login to the hostname: transfer.rc.hms.harvard.edu . You will be connected to a system which has access to /n/files .
      

   2. Once logged in, just run your commands (e.g. rsync, scp, cp) normally without using sbatch.

   3. Transfer servers can not submit jobs to the cluster, and research applications (modules) are not available from those systems.

2. If you have a batch job workflow that must use /n/files , you can request access to be able to use the "transfer" job partition. This partition has access to a few lower performance compute nodes which mount /n/files . They are only recommended when using the transfer servers is not an option, as these nodes are slower and generally less available.

Using the transfer job partition

...

# If you are transferring large data set, you can start a screen session, so that you can go back to the session in case you lose connection: 
# For details: https://wikiharvardmed.rc.hms.harvard.edu/pages/viewpage.action?pageId=20676715atlassian.net/wiki/spaces/O2/pages/1601699946/screen+Keep+Linux+Sessions+Alive+so+you+can+go+back+to+the+same+terminal+window+from+anywhere+anytime
# copy screen default setting file. Only need run this once. Does not hurt to run more than once though.
mfk8@login02:~$ cp /n/shared_db/misc/rcbio/data/screenrc.template.txt ~/.screenrc 

# Start a new screen session. If you already have one screen session before, you also attach that one. See above link how to attach a screen session. 
mfk8@login02:~$ screen

# login to transfer cluster:
mfk8@login02:~$ ssh transfer

mfk8@transfer01:~$ rsync -av --remove-source-files /n/groups/lab/tier2 /n/standby/hms/dept/lab/

# In case the transfer stops somehow, rerun the rsync command. rsync will resume from the breakpoint:
mfk8@transfer01:~$ rsync -av --remove-source-files /n/groups/lab/tier2 /n/standby/hms/dept/lab/

...