Versions Compared

Old Version 60

changes.mady.by.user lingsheng_dong

Saved on

compared with

New Version 61

changes.mady.by.user Kathleen Chappell

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Table of Contents

...

  • scp, sftp, rsync - these are automatically installed on Mac and Linux

  • pscp, psftp - Windows-only. These can be installed with the PuTTY ssh program.

  • ftp - available on O2 for downloading from external sites which only accept FTP logins. But, O2 does not accept incoming FTP logins.

  • aspera - a data transport and streaming technology, now owned by IBM.

  • awscli - Amazon AWS command line interface

  • basemount - an Illumina tool to mount BaseSpace Sequence Hub data.

  • bbcp - a point-to-point network file copy application from NERSC

  • lftp - can transfer files via FTP, FTPS, HTTP, HTTPS, FISH, SFTP, BitTorrent, and FTP over HTTP proxy.

  • gcloud - Google Cloud command line interface, including the gsutil command

  • NBIA Data Retriever - a tool for downloading data from the TCIA Data Portal , installed under /opt/NBIADataRetriever

  • rclone - rsync for cloud storage

  • Globus - If the other side support Globus

...

For graphical tools, see the documentation that came with the program. Also, see our instructions on how to use these tools with two-factor auth. Many tools will by default copy somewhere in your /home directory, which has a small 100GB 100GiB storage quota. Make sure to explicitly specify whether you want to copy there or to a different location like: /n/scratch3/users/m/mfk8/

...

The O2 login nodes and most compute nodes do not currently mount /n/files. There are 2 ways to access this filesystem from O2:

  1. Use O2's dedicated file transfer servers: 

    1. SSH login to the hostname: transfer.rc.hms.harvard.edu . You will be connected to a system which has access to /n/files .

    2. Once logged in, just run your commands (e.g. rsync, scp, cp) normally without using sbatch.

    3. Transfer servers can not submit jobs to the cluster, and research applications (modules) are not available from those systems.

  2. If you have a batch job workflow that must use /n/files , you can request access to be able to use the "transfer" job partition. This partition has access to a few lower performance compute nodes which mount /n/files . They are only recommended when using the transfer servers is not an option, as these nodes are slower and generally less available.

Using the transfer job partition

...