3.Bash and Github
How to set up multiple jobs as a pipeline

1) Github

(1) Desktop Version for Mac or Windows

I downloaded the desktop version of github on mac.

(2) Terminal Version for Linux or Mac

Setup:
1) add a setting file:
~/.gitconfig:
1
[user]
3
name = Shared
Copied!
2) Copy your ssh key to github website (set up ssh key) 3) Add a repository in github website
Clone a repository:
git clone [email protected]:lulab/RNAfinder_Server.git
Add:
1
git add exmaples/
2
git commit -a
3
git push origin
Copied!
Change:
1
git commit -a
2
git push origin
Copied!
Remove:
1
git rm *.file
2
git commit -a
3
git push origin
4
git pull?
Copied!
Pull (update):
1
git pull origin master
2
git log -n 2 # look at the last two log entries.
Copied!
more
Case study: how to use google to solve question
    Q1: How to install terminal version of git in mac?
    A1: google "install git _on _mac" --> I got the answer
    Q2: After I installed git, I got such error:
xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun"
    A2: google this whole sentence --> I got the answer

2) Bash Examples

Example I

install software and tools in a brand-new OS
(1) 安装vim
1
yum -y install vim
Copied!
(2) 打开一个空shell文件
1
vim myinstall.sh
Copied!
(3) 复制如下shell脚本到smyinstall.sh内。此shell文件会自动安装包括:R-2.15.3,perl-5.26.1等
1
yum -y install wget
2
yum -y install readline-devel
3
yum -y install libXt-devel
4
yum -y install gcc gcc-c++
5
yum -y install gcc-gfortran
6
yum -y install libg2c.so.0
7
yum -y install less
8
yum -y install bzip2-devel
9
yum -y install zlib-devel
10
yum -y install xz-devel.x86_64
11
yum -y install make
12
yum -y install zip unzip
13
14
mkdir /download
15
cd /download
16
wget -c http://www.cpan.org/src/5.0/perl-5.26.1.tar.gz
17
tar -xvzf perl-5.26.1.tar.gz
18
cd /download/perl-5.26.1
19
sh Configure -de
20
make
21
make test
22
make install
Copied!
(4) 执行shell脚本
1
bash myinstall.sh > myinstall.log # Standard output to a log file
2
bash myinstall.sh >& myinstall.log # Log both Standard output and Standard error
3
bash myinstall.sh > myinstall.log 2> myinstall.err # Log STDOUT and STDERR separately
Copied!

Example II

backup files using rsync and crontab
(0) install rsync and crontab
1
yum -y install rsync
2
yum -y install crontabs
Copied!
1
mkdir /mac/backup # prepare the backup dir
Copied!
(1) Prepare a backup script, for example, "~/backup.sh"
1
#!/bin/bash
2
3
#1. Local backup
4
RSYNC="rsync --stats --compress --recursive --times --perms --links --delete --max-size=100M --exclude-from=/home/john/.rsync/exclude"
5
6
echo "1. Backup of /home/john start at:"
7
date
8
$RSYNC /home/john/data/ /mac/backup/
9
echo "Backup end at:"
10
date
11
12
13
14
#2. Remote backup
15
RSYNC="rsync --stats --compress --recursive --times --perms --links --delete --max-size=100M"
16
17
echo "2. Backup 172.22.220.20:/data/ to /mac/backup2/ start at:"
18
date
19
$RSYNC [email protected]:/home/john/data/ /mac/backup2/
20
echo "Backup end at:"
21
date
Copied!
(2) Using "crontab" command to execute the backup script routinely, and record in a log file, for example,
execute the command "~/backup.sh > ~/backup.log" in 5:10am everyday:
1
chmod +x ~/backup.sh
Copied!
1
1) open crontab and edit it by the following command:
2
3
crontab -e or crontab ~/cronjob
4
5
2) type in the following lines or write the following in a file (i.e. ~/crontab):
6
7
# minute hour day_in_month month day_in_week command
8
10 5 * * * ~/backup.sh > ~/backup.log
9
10
3) exit and save (like in VIM)
Copied!

3) Tips

    awk
    1
    gawk '{print $2 $1}' FILE | head -100
    2
    cat FILE | gawk '(NR>2) {print }'
    3
    4
    gawk -f FILE # take command from file
    5
    6
    Begin { a=1; ORS = " "} # ORS: output record seperator
    7
    ...
    Copied!
    input STANDIN in your script
1
rnatotal.o <<EOF
2
trna.lis
3
trna.out
4
trna
5
EOF
Copied!
    grep non ATGC nucleotides in fasta file
1
grep -v "^>" test.fasta | \ # ^ means start with here
2
grep --color -i "[^ATCG]" # ^ means non- here
Copied!
    A tee in your pipe
1
program1 input.txt | tee intermediate-file.txt | program2 > results.txt
Copied!
    Exit Status
1
program1 input.txt > inter.txt && \ # run program 2 only if program1 is success (i.e. exit status is 0: $? = 0)
2
program2 inter.txt> results.txt
3
4
program1 input.txt > inter.txt || \
5
echo "warning: an error occurred
Copied!
    Name dir with today's date
1
mkdir results-$(date +F%) # +F% format is 2018-03-18
2
3
alias today="date +F%" # you can put this in ~/.bashrc
4
mkdir results-$(today)
Copied!

4)More Readings and Practices

阅读和练习《鸟哥的Linux私房菜-基础学习篇》如下章节:
第11章 认识与学习bash 第12章 正则表达式与文件格式化处理 第13章 学习shell script

5)Homework

    1.
    利用google等工具找到如何让awk读输入文件时只按照tab来做separator?
    2.
    学习和完成这里面的作业:Additional Tutorial : Bash (Teaching Video:Week II. Bash and R -- Shared by Li)

6) Video

6a) Bash and Github

@Youtube
@Bilibili
Last modified 3yr ago