2.Linux

1) Examples

    grep
1
grep -v -c pattern FILES #(-v exclude -c count lines)
2
grep -l pattern FILES #(-l: list of the files)
Copied!
    head, tail V.S. cut
    cat V.S. paste
1
head -100 FILE
2
tail -100 FILE
3
cut -f 2 FILE #(-f: field)
4
cut -d ; -f 2 FILE #(-d change delimiter)
5
6
cat FILES* > NEW_FILE
7
paste FILES* > NEW_FILE
Copied!
    rev and tac
1
echo john temp | rev #(output pmet nhoj)
2
tac #(reverse by lines)
Copied!
    wc
1
ls * |wc -w or wc -l #(count the number of files)
Copied!
    comm and vimdiff, diff
1
comm -2 file1 file2
2
diff file1 file2
Copied!
    sort
1
sort at position 1, delimeter is ','; -r reverse sort
2
sort -t ',' -k 1 sort -r -t ',' -k 1
Copied!
    uniq
1
cat human.gtf | cut –f 2,3 | sort | uniq –c
Copied!
    seq
1
for i in `seq 1 2 10`; do echo "$i"; done
2
for i in `ls *`;do head "$i" | cut -f 3 ; done
Copied!
    sed
1
cat 1.gtf | head |sed 's/1802/12/'
Copied!
    awk
1
cat 1-6.gtf |head | awk '{if($4>1000 && $5>2900)print}' | cut –f 3 | sort | uniq –c
Copied!
    history
1
Ctrl - r : search previous command
2
!V (im) : last command (be cautious !)
Copied!
    find and locate
1
find . -name "*.gtf" # find . -name "pattern"
2
locate
Copied!
    xargs
grep for pattern in all files on the system:
1
find / | xargs grep pattern > out &
Copied!
Move files in olddir to newdir, showing each command:
1
ls olddir | xargs -i mv olddir/{} newdir/{}
2
find . -type f -name "*.txt" | xargs -i mv {} newdir/{}
Copied!

2) Tips

(1) bashrc and .bash_profile

Example:
1
# .bashrc
2
# Source global definitions
3
if [ -f /etc/bashrc ]; then
4
. /etc/bashrc
5
fi
6
7
8
# User specific environment and startup programs
9
if [ -f $HOME/shortcuts ]; then
10
source $HOME/shortcuts
11
fi
12
PATH=$HOME/bin:$PATH
13
export PATH
14
15
16
# User specific aliases and functions
17
alias qstat="qstat -u '*'"
18
#alias screen="/usr/bin/screen -D -R"
19
#alias rm="$HOME/bin/del.sh"
20
#alias undel="$HOME/bin/del.sh -u"
21
#alias ls="ls --color"
22
alias ld="ls -d"
23
alias c="clear"
24
alias l="ls -alh"
25
alias lf="ls -F|grep /"
26
alias lt="ls -tlr"
27
alias mv="mv -i"
28
alias cp="cp -pi"
29
alias diff="diff -b"
30
31
32
#PERL5LIB=$MYHOME/perllib:$MYHOME/perllib/lib64/perl5/site_perl/5.8.5:$MYHOME/perlib/lib/perl5/site_perl/5.8.5
33
#export PERL5LIB
34
#export R_LIBS_USER=~/R:/data/apps/R_library
Copied!

(2) nohup, screen and qsub

    run something at background
./run.bat >& run.log &
fg
    nohup:
nohup nice -19 run.bat >& run.log&
    tmux:
      start a new session: tmux or tumx new-session -s session-name
      detach: ctrl-a, d
      re-attach: tmux attach-session -d -t session-name #detach it first
    screen: # a popular alternative of tmux
      start a new session: screen
      detach: ctrl-a, d
      re-attach: screen -R -D # detach it first
    qsub:
      qlogin not allowed in some servers
      Check qstat -u '\*'

(3) secure your files

    make your files Read-only
      permission for a executable bash script is usually 755
      using chmod -R a-w for raw data and input files
    777, rwxrwxrwx is forbidden (using chmod -R o-w)
    Change user group and permission
1
usermod –G admin john # add sudo
2
3
usermod –G lulab liyang # add to lulab group
4
chgrp –R lulab yourdir/
5
chmod –R g+w yourdir/
6
7
chmod a+x yourdir/
8
chmod o+x yourdir/
9
chmod o-w yourdir/
Copied!
    Blocking the root:
vim /etc/ssh/sshd_config #PermitRootLogin yes —>no

(4) Setup ssh key

    ssh-keygen (authorized_keys id_rsa.pub 权限设置为 600)
1
ssh-keygen -t rsa
2
copy authorized key in ~/.ssh/id_rsa.pub to remote_machine:~/.ssh/athorized_keys
Copied!
This will be very useful later when we work on remote machine as a local one, especially for jobs like backup and script editing.

(5) System

    kernel version
1
uname -r
2
uname -a
Copied!
    mount dir/ to local machine using NFS or sshfs (then you can Edit text and view figures remotely)
I recommend using "transmit" app in your mac instead of mounting NFS or sshfs.
1
$ yum install fuse-sshfs
2
$ usermod -a -G fuse john
3
sshfs [email protected]:/Users/john /mnt/nyefs fusermount -u mountpoint
4
automounting: in /etc/fstab
5
sshfs#[email protected]: /mnt/nyefs fuse uid=500,gid=100,allow_other 0 0
6
mount /mnt/nyefs; umnout /mnt/nyefs
Copied!
    forward the log file to an email address
1
vim /home/.forward or /root/.forward
Copied!
    Kill batch job
1
top
2
ps -edalf | grep username
3
kill -9 PID
Copied!

3) More Readings and Practices

    Teaching Video:Week I. 2. Linux
    for Beginners
      阅读和练习《鸟哥的Linux私房菜-基础学习篇》如下章节:
      《“笨办法”学python》附录“命令行快速入门”
第5章 5.3.1 man page 第6章 6.1用户与用户组 6.2 LINUX文件权限概念 6.3 LINUX目录配置 第7章Linux文件与目录管理 7.1目录与路径 7.2文件与目录管理 7.3文件内容查阅 7.5命令与文件的查询 7.6权限与命令间的关系 第8章 8.2文件系统的简单操作 第9章 9.1压缩文件的用途与技术 9.2 Linux系统常见的压缩命令 9.3打包命令:tar 第10章vim程序编辑器 (或者其他编辑器文档) 第11章 认识与学习bash 第25章 LINUX备份策略 25.2.2完整备份的差异备份 25.3鸟哥的备份策略 25.4灾难恢复的考虑 25.5重点回顾
第11章 认识与学习bash 第12章 正则表达式与文件格式化处理 第13章 学习shell script
    for Advanced Readers
《Bioinformatics Data Skills》
3) Remedial Unix Shell
7) Unix Data Tools

4) Homework

    1.
    解释gtf/gff文件中第4、5列($4,$5)代表什么,exon长度应该是$5-$4+1还是$5-$4
    2.
    从gtf/gff文件中寻找3个最长的exon:
    grep exon *.gtf | awk '{print $5-$4+1}' | sort -n | tail -3这个方法有什么bug?
    有新的方法加分,但必须注释清楚每个语句和参数的意义和结果。
    3.
    从gtf/gff文件中寻找并计算每一个transcript的长度,注意不能重复计算,不能包含intron。

5) Video

5a) Linux

@youtube
@Bilibili
Last modified 3yr ago