Versions Compared

Old Version 61

changes.mady.by.user Kathleen Chappell

Saved on

compared with

New Version 62

changes.mady.by.user Cox, Calvin D.

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: The link to the screen info page actually just went to the overview page, updating so that it goes to the info page about screen
Table of Contents

...

  • scp, sftp, rsync - these are automatically installed on Mac and Linux

  • pscp, psftp - Windows-only. These can be installed with the PuTTY ssh program.

  • ftp - available on O2 for downloading from external sites which only accept FTP logins. But, O2 does not accept incoming FTP logins.

  • aspera - a data transport and streaming technology, now owned by IBM.

  • awscli - Amazon AWS command line interface

  • basemount - an Illumina tool to mount BaseSpace Sequence Hub data.

  • bbcp - a point-to-point network file copy application from NERSC

  • lftp - can transfer files via FTP, FTPS, HTTP, HTTPS, FISH, SFTP, BitTorrent, and FTP over HTTP proxy.

  • gcloud - Google Cloud command line interface, including the gsutil command

  • NBIA Data Retriever - a tool for downloading data from the TCIA Data Portal , installed under /opt/NBIADataRetriever

  • rclone - rsync for cloud storage

  • Globus - If the other side support Globus

...

Code Block
# If you are transferring large data set, you can start a screen session, so that you can go back to the session in case you lose connection: 
# For details: https://wikiharvardmed.rc.hms.harvard.edu/pages/viewpage.action?pageId=20676715atlassian.net/wiki/spaces/O2/pages/1601699946/screen+Keep+Linux+Sessions+Alive+so+you+can+go+back+to+the+same+terminal+window+from+anywhere+anytime
# copy screen default setting file. Only need run this once. Does not hurt to run more than once though.
mfk8@login02:~$ cp /n/shared_db/misc/rcbio/data/screenrc.template.txt ~/.screenrc 

# Start a new screen session. If you already have one screen session before, you also attach that one. See above link how to attach a screen session. 
mfk8@login02:~$ screen

# login to transfer cluster:
mfk8@login02:~$ ssh transfer

mfk8@transfer01:~$ rsync -av --remove-source-files /n/groups/lab/tier2 /n/standby/hms/dept/lab/

# In case the transfer stops somehow, rerun the rsync command. rsync will resume from the breakpoint:
mfk8@transfer01:~$ rsync -av --remove-source-files /n/groups/lab/tier2 /n/standby/hms/dept/lab/

...