Training @ Lu Lab
Lu Lab Docs
  • Home
    • Training @ Lu Lab
  • Drylab Training
    • Genomics
      • RNA Types in Genome
  • Wetlab Training
    • Wetlab Safety Guide
    • Wetlab FAQ
  • Archive
    • Archive 2021
      • cfDNA Methylation
      • Genomic Annotation
    • Archive 2019 - Wetlab Training
      • Class I. Basics
        • 1. Wet Lab Safety
        • 2. Wet Lab Regulation
        • 3. Wet Lab Protocols
        • 4. How to design sample cohort
        • 5. How to collect and manage samples
        • 6. How to purify RNA from blood
        • 7. How to check the quantity and quality of RNA
        • 8. RNA storage
        • 9. How to remove DNA contanimation
        • 10. What is Spike-in
      • Class II. NGS - I
        • 1. How to do RNA-seq
        • 2. How to check the quantity and quality of RNA-seq library
        • 3. What is SMART-seq2 and Multiplex
    • Archive 2019 - Drylab Training
      • Getting Startted
      • Part I. Programming Skills
        • Introduction of PART I
        • 1.Setup
        • 2.Linux
        • 3.Bash and Github
        • 4.R
        • 5.Python
        • 6.Perl
        • Conclusion of PART I
      • Part II. Machine Learning Skills
        • 1.Machine Learning
        • 2.Feature Selection
        • 3.Machine Learning Practice
        • 4.Deep Learning
      • Part III. Case studies
        • Case Study 1. exRNA-seq
          • 1.1 Mapping, Annotation and QC
          • 1.2 Expression Matrix
          • 1.3.Differential Expression
          • 1.4 Normalization Issues
        • Case Study 2. exSEEK
          • 2.1 Plot Utilities
          • 2.2 Matrix Processing
          • 2.3 Feature Selection
        • Case Study 3. DeepSHAPE
          • 3.1 Background
          • 3.2 Resources
          • 3.3 Literature
      • Part IV. Appendix
        • Appendix I. Keep Learning
        • Appendix II. Public Data
        • Appendix III. Mapping Protocol of RNA-seq
        • Appendix IV. Useful tools for bioinformatics
      • Part V. Software
        • I. Docker Manual
        • II. Local Gitbook Builder
        • III. Teaching Materials
  • Archive 2018
Powered by GitBook
On this page
  • 1) Github
  • (1) Desktop Version for Mac or Windows
  • (2) Terminal Version for Linux or Mac
  • 2) Bash Examples
  • Example I
  • Example II
  • 3) Tips
  • 4)More Readings and Practices
  • 5)Homework
  • 6) Video
  • 6a) Bash and Github
Edit on GitHub
  1. Archive
  2. Archive 2019 - Drylab Training
  3. Part I. Programming Skills

3.Bash and Github

Last updated 3 years ago

How to set up multiple jobs as a pipeline

1) Github

(1) Desktop Version for Mac or Windows

I downloaded the on mac.

(2) Terminal Version for Linux or Mac

Setup:

1) add a setting file:

~/.gitconfig:

[user] 
email =[NNN@NN.com]
name = Shared

2) Copy your ssh key to github website () 3) Add a repository in github website

Clone a repository:

git clone git@github.com:lulab/RNAfinder_Server.git

Add:

git add exmaples/
git commit -a
git push origin

Change:

git commit -a
git push origin

Remove:

git rm *.file
git commit -a
git push origin
git pull?

Pull (update):

git pull origin master
git log -n 2 # look at the last two log entries.

Case study: how to use google to solve question

  • Q1: How to install terminal version of git in mac?

  • Q2: After I installed git, I got such error:

xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun"

2) Bash Examples

Example I

install software and tools in a brand-new OS

(1) 安装vim

yum -y install vim

(2) 打开一个空shell文件

vim myinstall.sh

(3) 复制如下shell脚本到smyinstall.sh内。此shell文件会自动安装包括:R-2.15.3,perl-5.26.1等

yum -y install wget
yum -y install readline-devel
yum -y install libXt-devel
yum -y install gcc gcc-c++
yum -y install gcc-gfortran
yum -y install libg2c.so.0
yum -y install less
yum -y install bzip2-devel
yum -y install zlib-devel
yum -y install xz-devel.x86_64
yum -y install make
yum -y install zip unzip

mkdir /download
cd /download
wget -c http://www.cpan.org/src/5.0/perl-5.26.1.tar.gz
tar -xvzf perl-5.26.1.tar.gz
cd /download/perl-5.26.1
sh Configure -de
make
make test
make install

(4) 执行shell脚本

bash myinstall.sh > myinstall.log    # Standard output to a log file
bash myinstall.sh >& myinstall.log   # Log both Standard output and Standard error
bash myinstall.sh > myinstall.log  2> myinstall.err # Log STDOUT and STDERR separately

Example II

backup files using rsync and crontab

(0) install rsync and crontab

yum -y install rsync
yum -y install crontabs
mkdir /mac/backup   # prepare the backup dir

(1) Prepare a backup script, for example, "~/backup.sh"

#!/bin/bash

#1. Local backup  
RSYNC="rsync --stats  --compress --recursive --times --perms --links --delete --max-size=100M --exclude-from=/home/john/.rsync/exclude"

echo "1. Backup of /home/john start at:"
date
$RSYNC /home/john/data/  /mac/backup/
echo "Backup end at:"
date



#2. Remote backup 
RSYNC="rsync --stats  --compress --recursive --times --perms --links --delete --max-size=100M"

echo "2. Backup 172.22.220.20:/data/ to /mac/backup2/ start at:"
date
$RSYNC john@172.22.220.20:/home/john/data/ /mac/backup2/
echo "Backup end at:"
date

(2) Using "crontab" command to execute the backup script routinely, and record in a log file, for example,

execute the command "~/backup.sh > ~/backup.log" in 5:10am everyday:

chmod +x ~/backup.sh
1) open crontab and edit it by the following command: 

crontab -e  or crontab ~/cronjob

2) type in the following lines or write the following in a file (i.e. ~/crontab): 

# minute hour day_in_month month day_in_week command
     10   5  * * *   ~/backup.sh > ~/backup.log 

3) exit and save (like in VIM)

3) Tips

  • awk

    gawk '{print $2 $1}' FILE | head -100
    cat FILE | gawk '(NR>2) {print }'
    
    gawk -f FILE # take command from file
    
      Begin { a=1; ORS = " "}  # ORS: output record seperator
      ...
  • input STANDIN in your script

rnatotal.o <<EOF
trna.lis
trna.out
trna
EOF
  • grep non ATGC nucleotides in fasta file

grep -v "^>" test.fasta | \    # ^ means start with here
grep --color -i "[^ATCG]"      # ^ means non- here
  • A tee in your pipe

program1 input.txt | tee intermediate-file.txt | program2 > results.txt
  • Exit Status

program1 input.txt > inter.txt && \   # run program 2 only if program1 is success (i.e. exit status is 0: $? = 0)
program2 inter.txt> results.txt

program1 input.txt > inter.txt || \
echo "warning: an error occurred
  • Name dir with today's date

mkdir results-$(date +F%)  # +F% format is 2018-03-18

alias today="date +F%"  # you can put this in ~/.bashrc
mkdir results-$(today)

4)More Readings and Practices

  • for Beginners

阅读和练习《鸟哥的Linux私房菜-基础学习篇》如下章节:

第11章 认识与学习bash 第12章 正则表达式与文件格式化处理 第13章 学习shell script

  • for Advanced Readers

    • <<Bioinformatics Data Skills>>

      • 5) Git for Scientists

      • 6) Bioinformatics Data

      • 12) Bioinformatics Shell Scripting, Writing Pipelines, and Parallelizing Tasks

5)Homework

  1. 利用google等工具找到如何让awk读输入文件时只按照tab来做separator?

6) Video

6a) Bash and Github

A1: google "" --> I got

A2: google this whole sentence --> I got

:Week II. 3. Bash

: Bash

学习和完成这里面的作业: : Bash (:Week II. Bash and R -- Shared by Li)

desktop version of github
set up ssh key
more
install git _on _mac
the answer
the answer
More Bash Examples at Github
@Youtube
@Bilibili
Teaching Video
Additional Tutorial
Additional Tutorial
Teaching Video