2.Linux

1) Examples

  • grep

grep -v -c pattern FILES #(-v exclude -c count lines)
grep -l pattern FILES #(-l: list of the files)
  • head, tail V.S. cut

  • cat V.S. paste

head -100 FILE
tail -100 FILE
cut -f 2 FILE #(-f: field)
cut -d ; -f 2 FILE #(-d change delimiter)
cat FILES* > NEW_FILE
paste FILES* > NEW_FILE
  • rev and tac

echo john temp | rev #(output pmet nhoj)
tac #(reverse by lines)
  • wc

ls * |wc -w or wc -l #(count the number of files)
  • comm and vimdiff, diff

comm -2 file1 file2
diff file1 file2
  • sort

sort at position 1, delimeter is ','; -r reverse sort
sort -t ',' -k 1 sort -r -t ',' -k 1
  • uniq

cat human.gtf | cut –f 2,3 | sort | uniq –c
  • seq

for i in `seq 1 2 10`; do echo "$i"; done
for i in `ls *`;do head "$i" | cut -f 3 ; done
  • sed

cat 1.gtf | head |sed 's/1802/12/'
  • awk

cat 1-6.gtf |head | awk '{if($4>1000 && $5>2900)print}' | cut –f 3 | sort | uniq –c
  • history

Ctrl - r : search previous command
!V (im) : last command (be cautious !)
  • find and locate

find . -name "*.gtf" # find . -name "pattern"
locate
  • xargs

grep for pattern in all files on the system:

find / | xargs grep pattern > out &

Move files in olddir to newdir, showing each command:

ls olddir | xargs -i mv olddir/{} newdir/{}
find . -type f -name "*.txt" | xargs -i mv {} newdir/{}

2) Tips

(1) bashrc and .bash_profile

Example:

# .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
# User specific environment and startup programs
if [ -f $HOME/shortcuts ]; then
source $HOME/shortcuts
fi
PATH=$HOME/bin:$PATH
export PATH
# User specific aliases and functions
alias qstat="qstat -u '*'"
#alias screen="/usr/bin/screen -D -R"
#alias rm="$HOME/bin/del.sh"
#alias undel="$HOME/bin/del.sh -u"
#alias ls="ls --color"
alias ld="ls -d"
alias c="clear"
alias l="ls -alh"
alias lf="ls -F|grep /"
alias lt="ls -tlr"
alias mv="mv -i"
alias cp="cp -pi"
alias diff="diff -b"
#PERL5LIB=$MYHOME/perllib:$MYHOME/perllib/lib64/perl5/site_perl/5.8.5:$MYHOME/perlib/lib/perl5/site_perl/5.8.5
#export PERL5LIB
#export R_LIBS_USER=~/R:/data/apps/R_library

more examples

(2) nohup, screen and qsub

  • run something at background

./run.bat >& run.log &

fg

  • nohup:

nohup nice -19 run.bat >& run.log&

  • tmux:

    • start a new session: tmux or tumx new-session -s session-name

    • detach: ctrl-a, d

    • re-attach: tmux attach-session -d -t session-name #detach it first

  • screen: # a popular alternative of tmux

    • start a new session: screen

    • detach: ctrl-a, d

    • re-attach: screen -R -D # detach it first

  • qsub:

    • qlogin not allowed in some servers

    • Check qstat -u '\*'

(3) secure your files

  • make your files Read-only

    • permission for a executable bash script is usually 755

    • using chmod -R a-w for raw data and input files

  • 777, rwxrwxrwx is forbidden (using chmod -R o-w)

  • Change user group and permission

usermod –G admin john # add sudo
usermod –G lulab liyang # add to lulab group
chgrp –R lulab yourdir/
chmod –R g+w yourdir/
chmod a+x yourdir/
chmod o+x yourdir/
chmod o-w yourdir/
  • Blocking the root:

vim /etc/ssh/sshd_config #PermitRootLogin yes —>no

(4) Setup ssh key

  • ssh-keygen (authorized_keys id_rsa.pub 权限设置为 600)

ssh-keygen -t rsa
copy authorized key in ~/.ssh/id_rsa.pub to remote_machine:~/.ssh/athorized_keys

This will be very useful later when we work on remote machine as a local one, especially for jobs like backup and script editing.

(5) System

  • kernel version

uname -r
uname -a
  • mount dir/ to local machine using NFS or sshfs (then you can Edit text and view figures remotely)

I recommend using "transmit" app in your mac instead of mounting NFS or sshfs.

$ yum install fuse-sshfs
$ usermod -a -G fuse john
sshfs john@nye:/Users/john /mnt/nyefs fusermount -u mountpoint
automounting: in /etc/fstab
sshfs#john@nye: /mnt/nyefs fuse uid=500,gid=100,allow_other 0 0
mount /mnt/nyefs; umnout /mnt/nyefs
  • forward the log file to an email address

vim /home/.forward or /root/.forward
zhi_lu@tsinghua.edu
  • Kill batch job

top
ps -edalf | grep username
kill -9 PID

3) More Readings and Practices

  • Teaching Video:Week I. 2. Linux

  • for Beginners

    • 阅读和练习《鸟哥的Linux私房菜-基础学习篇》如下章节:

    • 《“笨办法”学python》附录“命令行快速入门”

第5章 5.3.1 man page 第6章 6.1用户与用户组 6.2 LINUX文件权限概念 6.3 LINUX目录配置 第7章Linux文件与目录管理 7.1目录与路径 7.2文件与目录管理 7.3文件内容查阅 7.5命令与文件的查询 7.6权限与命令间的关系 第8章 8.2文件系统的简单操作 第9章 9.1压缩文件的用途与技术 9.2 Linux系统常见的压缩命令 9.3打包命令:tar 第10章vim程序编辑器 (或者其他编辑器文档) 第11章 认识与学习bash 第25章 LINUX备份策略 25.2.2完整备份的差异备份 25.3鸟哥的备份策略 25.4灾难恢复的考虑 25.5重点回顾

第11章 认识与学习bash 第12章 正则表达式与文件格式化处理 第13章 学习shell script

  • for Advanced Readers

《Bioinformatics Data Skills》

3) Remedial Unix Shell

7) Unix Data Tools

4) Homework

  1. 解释gtf/gff文件中第4、5列($4,$5)代表什么,exon长度应该是$5-$4+1还是$5-$4

  2. 从gtf/gff文件中寻找3个最长的exon:

    grep exon *.gtf | awk '{print $5-$4+1}' | sort -n | tail -3这个方法有什么bug?

    有新的方法加分,但必须注释清楚每个语句和参数的意义和结果。

  3. 从gtf/gff文件中寻找并计算每一个transcript的长度,注意不能重复计算,不能包含intron。

5) Video

5a) Linux

@youtube

@Bilibili