A Short Introduction to Unix for Bioinformatics

Bioinformatics: Beyond Using Websites

You can do a lot of sophisticated bioinformatics using public websites But at some point you may be faced with a LOT of data The only solution is to have your own bioinformatics computer, database, and custom programs. Needs more processor power and more hard drive space than a typical desktop personal computer

Bioinformatics Requires Powerful Computers

One definition of bioinformatics is "the use of computers to analyze biological problems. As biological data sets have grown larger and biological problems have become more complex, the requirements for computing power have also grown. Computers that can provide this power generally use the Unix operating system - so you must learn Unix

@HWI-EAS305:1:1:1:991#0/1 GCTGGAGGTTCAGGCTGGCCGGATTTAAACGTAT +HWI-EAS305:1:1:1:991#0/1 MVXUWVRKTWWULRQQMMWWBBBBBBBBBBBBBB @HWI-EAS305:1:1:1:201#0/1 AAGACAAAGATGTGCTTTCTAAATCTGCACTAAT +HWI-EAS305:1:1:1:201#0/1 PXX[[[[XTXYXTTWYYY[XXWWW[TMTVXWBBB @HWI-EAS305:1:1:1:70#0/1 ATGATAATAATACCTCTTGCAGTTTGCATCATGT +HWI-EAS305:1:1:1:70#0/1 OYYYY[[Z[YZ[Y[[WYYP[YYYWZTZ[ZYBBBB @HWI-EAS305:1:1:1:983#0/1 ACCCAATACCGGTACAGGAATTGCAGCAATCAAA +HWI-EAS305:1:1:1:983#0/1 OYYXVVYYYYUVUYXVRSUYYUTUTPVYYYYVVT @HWI-EAS305:1:1:1:1671#0/1 AATTACACAACAAAAGGAGATCAAAGGGATACAA +HWI-EAS305:1:1:1:1671#0/1 OY[[[[YXXX[[[ZXUWXZZ[Y[[VTTUU[[[YX @HWI-EAS305:1:1:1:1699#0/1 GTTGGCTCTACGATCACGTTGCTCACCATGTGGG +HWI-EAS305:1:1:1:1699#0/1 JSUSSUWTWWQUUUVVUUUTTTUTUWBBBBBBBB @HWI-EAS305:1:1:1:1616#0/1 ATTGGCGACGATATTCAGGTCCATGTTTCTTGCG +HWI-EAS305:1:1:1:1616#0/1 NYYYVRVVVSVYYYYWUNNPWVYBBBBBBBBBBB @HWI-EAS305:1:1:1:1755#0/1 CAGTCACGAATCGGTGCGTCTTTCACCTGACACA +HWI-EAS305:1:1:1:1755#0/1 PWVWWVTKTWWWPMTRWQWUWWUUUSSODORSUU @HWI-EAS305:1:1:1:1046#0/1 CCCTGACAGGTAACAGGAGGTGGCTGGGGCTGAG +HWI-EAS305:1:1:1:1046#0/1 BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB

[browns02@sanger]$ more s_1_sequence.txt . [browns02@sanger]$ wc s_1_sequence.txt 32806276 32806276 1062955212 s_1_sequence.txt [browns02@sanger]$ grep "@" s_1_sequence.txt | wc 8201569 8201569 244422691

Unix Runs the Internet

Unix is a command line interface, used by most large, powerful computers. In fact, Unix is the underlying structure for most of the Internet and most large scale bioinformatics operations. A knowledge of Unix is likely to be helpful in your future career, regardless of where you pursue it.

CHIBI operates UNIX servers

mendel.med.nyu.edu
The student computer is a Sun V20Z
this is the machine that runs EMBOSS it has 2 x 2 GHz 64-bit processors (SMP) over 2 TB of hard disk space (RAID)

Unix Advantages
It is very popular, so it is easy to find information and get help
pick up books at the local bookstore (or street vendor) plenty of helpful websites USENET discussions and e-mail lists most Comp. Sci. students know Unix

Unix can run on virtually any computer

(IBM, Sun, Compaq, Macintosh, etc)

Unix is free or nearly free

Linux/open source software movement RedHat, FreeBSD, MKLinux, LinuxPPC, etc.

Stable and Efficient

Unix is very stable - computers running Unix almost never crash Unix is very efficient
it gets maximum number crunching power out of your processor (and multiple processors) it can smoothly manage extremely huge amounts of data it can give a new life to otherwise obsolete Macs and PCs

Most new bioinformatics software is created for Unix - its easy for the programmers

Unix has some Drawbacks

Unix computers are controlled by a command line interface
NOT user-friendly difficult to learn, even more difficult to truly master

Hackers love Unix

there are lots of security holes most computers on the Internet run Unix , so hackers can apply the same tricks to many different computers

There are many different versions of Unix with subtle (or not so subtle) differences

Open Source Bioinformatics

Almost all of the bioinformatics software that you need to do complex analyses is free for UNIX computers The Open Source software ethic is very strong among biologists
Bioinformatics.org Bioperl.org Open-bio.org

New algorithms generally appear first as free software (a publication requirement)

Free Software
Linux operating system, mySY QL database Perl - programming language Blast and Fasta - similarity search Clustal - multiple alignment Phylip - phylogenetics Phred/Phrap/Consed - sequence assembly and SNP detection EMBOSS - a complete sequence analysis package created by the EMBL (like GCG)

Simple Programs
You can use the Unix shell to run programs right from the command line, or save them as shell scripts. Simple loops can run a program (such as Blast or FASTA) on many sequence files. Then you can check the output files for specific results, and use if statements to sort or take other actions More about this next week.

General Unix Tips

UNIX is case sensitive!!
myfile.txt and MyFile.txt do not mean the same thing
I like to use capital letters for directory names - it puts them at the top of an alphabetical listing

Every program is independent

the core operating system (known as the kernel) manages each program as a distinct process with its own little chunk of dedicated memory. If one program runs into trouble, it dies, but does not affect the affect the kernel or the other programs running on the computer.

The Unix Shell

You communicate with a Unix computer through a command program known as a shell. The shell interprets the commands that you type on the keyboard. There are actually many different shells available for Unix computers, and on some systems you can choose the shell in which you wish to work. You can use shell commands to write simple programs (scripts) to automate many tasks

Unix Commands
Unix commands are short and cryptic like vi or rm.
Computer geeks like it that way; you will get used to it.

Every command has a host of modifiers which are generally single letters preceded by a hyphen: ls -l or mv -R
Capital letters have different functions than small letters, often completely unrelated. A command also generally requires an argument, meaning some file on which it will act:

cat -n mygene.seq

Wildcards
You can substitute the * as a wildcard symbol for any number of characters in any filename. If you type just * after a command, it stands for all files in the current directory:
lpr * will print all files

You can mix the * with other characters to form a search pattern:
ls a*.txt will list all files that start with a and end in .txt will copy draft1.doc, draft2.doc, draftb.doc, etc.

The ? wildcard stands for any single character:

cp draft?.doc

Typing Mistakes
Unix is remarkably unforgiving of typing mistakes
You can do a lot with just a few keystrokes, but it can be hard or impossible to undo

If you have not yet hit return

The delete key removes the characters that you just typed Which key on your keyboard will actually function as delete will vary depending on the type of computer that you are using, the Telnet program and the Unix shell that you are using, or if you are running a specific Unix program

Control Characters
You type Control characters by holding down the control key while also pressing the specified character. While you are typing a command:
ctrl-W erases the previous word ctrl-U erases the whole command line

Control commands that work (almost) any time

ctrl-S suspends (halts) output scrolling up on your terminal screen ctrl-Q resumes the display of output on your screen ctrl-C will abort any program

Getting Help in Unix

Unix is not a user-friendly computer system.
While not actively user-hostile, it is perfectly happy to sit there and taunt you with a blank screen and a blinking > cursor.

There is a rudimentary Help system which consists of a set of "manual pages for every Unix command. The man pages tell you which options a particular command can take, and how each option modifies the behavior of the command. Type man and the name of a command to read the manual page for that command.

> man ls Reformatting page. Please wait... completed

ls(1)
NAME ls - Lists and generates statistics for files SYNOPSIS

ls(1)

ls [-aAbcCdfFgilLmnopqrRstux1] [file...|directory...] STANDARDS Interfaces documented on this reference page conform to industry standards as follows: ls: XPG4, XPG4-UNIX Refer to the standards(5) reference page for more information about industry standards and associated tags. OPTIONS -a Lists all entries in the directory, including the entries that begin with a . (dot). Entries that begin with a . are not displayed unless you refer to them specifically, or you specify the -a option. -A [Compaq] Lists all entries, except . (dot) and .. (dot-dot). If you issue the ls command as the superuser, it behaves as if you specified this option. -b [Compaq] Displays nonprintable characters in octal notation. -c Uses the time of last inode modification (file created, mode changed, and so on) for sorting when used with the -t option. Displays the time of last inode modification (instead of the time at which the file's contents were last modified) when used with the -l option. This option has effect only when used with either -t or -l or both.

manaacsba (10%)

More Help (?)

The man pages, such as they are, give information about specific commands So what if you dont know what command you need? There is a command called apropos that will give you a list of commands that contain a given keyword in their man page header:
apropos password The man command with the -k modifier gives a similar result to apropos

Get yourself a good "Intro to Unix" book

Unix Help on the Web

Here is a list of a few online Unix tutorials: Unix for Beginners
http://www.ee.surrey.ac.uk/Teaching/Unix/

Introduction to Unix (OSU)

http://8help.osu.edu/wks/unix_course/intro-1.html

Unix Guru Universe

http://www.ugu.com/sui/ugu/show?help.beginners

Getting Started With The Unix Operating System

http://www.nbcs.rutgers.edu/~edseries/U1-ES.html

Logging in to the Sanger Server

Open a telnet/ssh program from your computer
Terminal on Mac OS X Putty, Win32, or QVT/Net for Windows Connect to: mendel.med.nyu.edu

Type your username and password

You cant backspace/delete while typing username and password Notice that when you type a password, nothing shows up on the screen, this is for your security

Customize Your Login

We have a standard set of customizations for all users, but you can change your settings.
Automatically detects and sets your terminal type Sets the T-shell as your default (tsch) Sets rm and mv to require confirmation before moving or deleting files

You can change your password with the passwd command. You can create a .login file in your home directory that executes any set of Unix commands every time that you login.

Connecting from Outside NYU

NYU School of Medicine has a firewall, so it is not possible to connect by ssh to sanger directly from the Internet. The Med school also operates a VPN, so you can connect from your home computer.
Once you are connected to Onsite Health, your computer becomes part of our local network - inside the firewall You must obtain a separate account and password to use the VPN.
Go to the IT Help Desk in Greenberg Hall SC2-97, with your Medical Center ID card. Instructions: http://www.med.nyu.edu/it/

Unix Filenames
Unix is cAsE sEnsItiVe UNIX filenames contain only letters, numbers, and the _ (underscore), . (dot), and (dash) characters. Unix does not allow two files to exist in the same directory with the same name.
Whenever a situation occurs where a file is about to be created or copied into a directory where another file has that exact same name, the new file will overwrite (and delete) the older file. Unix will generally alert you when this is about to happen, but it is easy to ignore the warning.

Filename Extensions
Most UNIX filenames start with a lower case letter and end with a dot followed by one, two, or three letters: myfile.txt
However, this is just a common convention and is not required. It is also possible to have additional dots in the filename.

The part of the name following the dot is called the extension. The extension is often used to designate the type of file.

Some Common Extensions

By convention:
files that end in .txt are text files files that end in .c are source code in the "C language files that end in .html are HTML files for the Web Compressed files have the .zip or .gz extension

Unix does not require these extensions (unlike Windows), but it is a sensible idea and one that you should follow

Working with Directories

Directories are a means of organizing your files on a Unix computer.
They are equivalent to folders on Windows and Macintosh computers

Directories contain files, executable programs, and sub-directories Understanding how to use directories is crucial to manipulating your files on a Unix system.

Your Home Directory

When you login to the server, you always start in your Home directory. Create sub-directories to store specific projects or groups of information, just as you would place folders in a filing cabinet. Do not accumulate thousands of files with cryptic names in your Home directory

File & Directory Commands

This is a minimal list of Unix commands that you must know for file management:
ls (list) mkdir (make directory) rmdir (remove directory) pwd (present working directory) more (view by page) cat (view entire file on screen) cd (change directory) cp (copy) mv (move) rm (remove)

All of these commands can be modified with many options. Learn to use Unix man pages for more information.

Navigation
pwd (present working directory) shows the name and location of the directory where you are currently working: > pwd
/u/browns02

This is a pathname, the slashes indicate sub-directories The initial slash is the root of the whole filesytem

ls (list) gives you a list of the files in the current directory: > ls
assembin4.fasta Misc test2.txt bin temp testfile

Use the ls -l (long) option to get more information about each file
> ls -l
total 1768 drwxr-x--- 2 browns02 users 8192 Aug 28 18:26 Opioid -rw-r----- 1 browns02 users 6205 May 30 2000 af124329.gb_in2 -rw-r----- 1 browns02 users 131944 May 31 2000 af151074.fasta

Sub-directories
cd (change directory) moves you to another directory
>cd Misc > pwd /u/browns02/Misc

mkdir (make directory) creates a new sub-directory inside of the current directory
> ls assembler phrap > mkdir subdir > ls assembler phrap space space subdir

rmdir (remove directory) deletes a subdirectory, but the sub-directory must be empty
> rmdir subdir

> ls assembler phrap

space

Shortcuts
There are some important shortcuts in Unix for specifying directories . (dot) means "the current directory"
.. means "the parent directory" - the directory one level above the current directory, so cd .. will move you up one level ~ (tilde) means your Home directory, so cd ~ will move you back to your Home.
Just typing a plain cd will also bring you back to your home directory

Unix File Protections

File protection (also known as permissions) enables the user to set up a file so that only specific people can read (r), write/delete (w), and execute (x) it.

Write and delete privilege are the same on a Unix system since write privilege allows someone to overwrite a file with a different one.

File Owners and Groups

Unix file permissions are defined according to ownership. The person who creates a file is its owner.
You are the owner of files in your Home directory and all its sub-directories

In addition, there is a concept known as a Group.

Members of a group have privileges to see each other's files. We create groups as the members of a single lab - the students, technicians, postdocs, visitors, etc. who work for a given PI.

View File Permissions

Use the ls -l command to see the permissions for all files in a directory:
> ls -l drwxr-x--- 2 browns02 users 8192 Aug 28 18:26 Opioid -rw-r----- 1 browns02 users 6205 May 30 2000 af124329.gb_in2 -rw-r----- 1 browns02 users 131944 May 31 2000 af151074.fasta

The username of the owner is shown in the third column. (The owner of the files listed above is browns02) The owner belongs to the group users

The access rights for these files is shown in the first column. This column consists of 10 characters known as the attributes of the file: r, w, x, and r w x indicates read permission indicates write (and delete) permission indicates execute (run) permission indicates no permission for that operation

> ls -l
drwxr-x--- 2 browns02 users 8192 Aug 28 18:26 Opioid -rw-r----- 1 browns02 users 6205 May 30 2000 af124329.gb_in2 -rw-r----- 1 browns02 users 131944 May 31 2000 af151074.fasta

The first character in the attribute string indicates if a file is a directory (d) or a regular file (-). The next 3 characters (rwx) give the file permissions for the owner of the file. The middle 3 characters give the permissions for other members of the owner's group. The last 3 characters give the permissions for everyone else (others) The default protections assigned to new files on our system is: -rw-r----- (owner=read and write, group =read, others=nothing)

Change Protections
Only the owner of a file can change its protections To change the protections on a file use the chmod (change mode) command.
[Beware, this is a confusing command.] First you have to decide for whom you will change the access permissions:
the file owner (u) the members of your group (g) others (o) (ie. anyone with an RCR account)

Next you have to decide if you are adding (+), removing (-), or setting (=) permissions.

Taken all together, it looks like this:

> chmod u=rwx g+r o-x myfile.txt
This will set the owner to have read, write, and execute permission; add the permission for the group to read; and remove the permission for others to execute the file named myfile.txt.

Commands for Files

Files are used to store information, for example, data or the results of some analysis.
You will mostly deal with text files Files on the RCR Alpha are automatically backed up to tape every night.

cat dumps the entire contents of a file onto the screen.

For a long file this can be annoying, but it can also be helpful if you want to copy and paste (use the buffer of your telnet program)

more
Use the command more to view at the contents of a file one screen at a time:
> more t27054_cel.pep !!AA_SEQUENCE 1.0 P1;T27054 - hypothetical protein Y49E10.20 - Caenorhabditis elegans Length: 534 May 30, 2000 13:49 Type: P Check: 1278 .. 1 MLKKAPCLFG SAIILGLLLA AAGVLLLIGI PIDRIVNRQV IDQDFLGYTR 51 DENGTEVPNA MTKSWLKPLY AMQLNIWMFN VTNVDGILKR HEKPNLHEIG 101 PFVFDEVQEK VYHRFADNDT RVFYKNQKLY HFNKNASCPT CHLDMKVTIP

t27054_cel.pep (87%)

Hit the spacebar to page down through the file

Ctrl-U moves back up a page At the bottom of the screen, more shows how much of the file has been displayed

More sophisticated options for viewing text files are available in a text editor (next week).

Copy & Move

cp lets you copy a file from any directory to any other directory, or create a copy of a file with a new name in one directory
cp filename.ext newfilename.ext cp filename.ext subdir/newname.ext cp /u/jdoe01/filename.ext ./subdir/newfilename.ext

mv allows you to move files to other directories, but it is also used to rename files.
Filename and directory syntax for mv is exactly the same as for the cp command.
mv filename.ext subdir/newfilename.ext

NOTE: When you use mv to move a file into another directory, the current file is deleted.

Delete
Use the command rm (remove) to delete files There is no way to undo this command!!!
We have set the server to ask if you really want to remove each file before it is deleted. You must answer Y or else the file is not deleted.
> ls af151074.gb_pr5 test.seq > rm test.seq rm: remove test.seq? y > ls af151074.gb_pr5

Moving Files between Computers

You will often need to move files between computers - desktop to server and back There are several options
Sneaker net (floppy, zip, writeable CD)
(not an option for the mendel machine)

E-mail Network filesharing FTP

FTP is Simple
File Transfer Protocol is standard for all computers on any network. The best way to move lots of data to and from remote machines:
put raw data onto the server for analysis get results back to the desktop for use in papers and grants

Graphical FTP applications for desktop PCs

On a Mac, use Fetch, CyberDuck (!) On a Windows PC, use WS_FTP, FileZilla

FTP Login
When you open an FTP program, you connect to sanger just as you would with a terminal We now use sFTP (secure FTP) Your username and password are the same.

 You will automatically end up in your home directory. Put files from you PC to the server, Get files from the server to your desktop machine.

Some More Advanced UNIX Commands

grep: searches a file for a specific text pattern cut: copies one or more columns from a tab-delimited text file wc: word count | : the pipe sends output of one command as input to the next > : redirect output to a file sed : stream editor change text inside a

@HWI-EAS305:1:1:1:991#0/1 GCTGGAGGTTCAGGCTGGCCGGATTTAAACGTAT +HWI-EAS305:1:1:1:991#0/1 MVXUWVRKTWWULRQQMMWWBBBBBBBBBBBBBB @HWI-EAS305:1:1:1:201#0/1 AAGACAAAGATGTGCTTTCTAAATCTGCACTAAT +HWI-EAS305:1:1:1:201#0/1 PXX[[[[XTXYXTTWYYY[XXWWW[TMTVXWBBB @HWI-EAS305:1:1:1:70#0/1 ATGATAATAATACCTCTTGCAGTTTGCATCATGT +HWI-EAS305:1:1:1:70#0/1 OYYYY[[Z[YZ[Y[[WYYP[YYYWZTZ[ZYBBBB @HWI-EAS305:1:1:1:983#0/1 ACCCAATACCGGTACAGGAATTGCAGCAATCAAA +HWI-EAS305:1:1:1:983#0/1 OYYXVVYYYYUVUYXVRSUYYUTUTPVYYYYVVT @HWI-EAS305:1:1:1:1671#0/1 AATTACACAACAAAAGGAGATCAAAGGGATACAA +HWI-EAS305:1:1:1:1671#0/1 OY[[[[YXXX[[[ZXUWXZZ[Y[[VTTUU[[[YX @HWI-EAS305:1:1:1:1699#0/1 GTTGGCTCTACGATCACGTTGCTCACCATGTGGG +HWI-EAS305:1:1:1:1699#0/1 JSUSSUWTWWQUUUVVUUUTTTUTUWBBBBBBBB @HWI-EAS305:1:1:1:1616#0/1 ATTGGCGACGATATTCAGGTCCATGTTTCTTGCG +HWI-EAS305:1:1:1:1616#0/1 NYYYVRVVVSVYYYYWUNNPWVYBBBBBBBBBBB @HWI-EAS305:1:1:1:1755#0/1 CAGTCACGAATCGGTGCGTCTTTCACCTGACACA +HWI-EAS305:1:1:1:1755#0/1 PWVWWVTKTWWWPMTRWQWUWWUUUSSODORSUU @HWI-EAS305:1:1:1:1046#0/1 CCCTGACAGGTAACAGGAGGTGGCTGGGGCTGAG +HWI-EAS305:1:1:1:1046#0/1 BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB

[browns02@sanger]$ more s_1_sequence.txt [browns02@sanger]$ wc s_1_sequence.txt 32806276 32806276 1062955212 s_1_sequence.txt [browns02@sanger]$ grep "@" s_1_sequence.txt | wc 8201569 8201569 244422691 [browns02@sanger]$ grep A 1 "@ s_1_sequence.txt > s1.seq

[browns02@sanger]$ sed -i 's/@/>/g' s1.seq

@HWI-EAS305:1:1:1:991#0/1 GCTGGAGGTTCAGGCTGGCCGGATTTAAACGTAT +HWI-EAS305:1:1:1:991#0/1 MVXUWVRKTWWULRQQMMWWBBBBBBBBBBBBBB @HWI-EAS305:1:1:1:201#0/1 AAGACAAAGATGTGCTTTCTAAATCTGCACTAAT +HWI-EAS305:1:1:1:201#0/1 PXX[[[[XTXYXTTWYYY[XXWWW[TMTVXWBBB @HWI-EAS305:1:1:1:70#0/1 ATGATAATAATACCTCTTGCAGTTTGCATCATGT +HWI-EAS305:1:1:1:70#0/1 OYYYY[[Z[YZ[Y[[WYYP[YYYWZTZ[ZYBBBB @HWI-EAS305:1:1:1:983#0/1 ACCCAATACCGGTACAGGAATTGCAGCAATCAAA +HWI-EAS305:1:1:1:983#0/1 OYYXVVYYYYUVUYXVRSUYYUTUTPVYYYYVVT @HWI-EAS305:1:1:1:1671#0/1 AATTACACAACAAAAGGAGATCAAAGGGATACAA +HWI-EAS305:1:1:1:1671#0/1 OY[[[[YXXX[[[ZXUWXZZ[Y[[VTTUU[[[YX @HWI-EAS305:1:1:1:1699#0/1 GTTGGCTCTACGATCACGTTGCTCACCATGTGGG +HWI-EAS305:1:1:1:1699#0/1 JSUSSUWTWWQUUUVVUUUTTTUTUWBBBBBBBB @HWI-EAS305:1:1:1:1616#0/1 ATTGGCGACGATATTCAGGTCCATGTTTCTTGCG +HWI-EAS305:1:1:1:1616#0/1 NYYYVRVVVSVYYYYWUNNPWVYBBBBBBBBBBB @HWI-EAS305:1:1:1:1755#0/1 CAGTCACGAATCGGTGCGTCTTTCACCTGACACA +HWI-EAS305:1:1:1:1755#0/1 PWVWWVTKTWWWPMTRWQWUWWUUUSSODORSUU @HWI-EAS305:1:1:1:1046#0/1 CCCTGACAGGTAACAGGAGGTGGCTGGGGCTGAG +HWI-EAS305:1:1:1:1046#0/1 BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB

[browns02@sanger]$ more s_1_sequence.txt [browns02@sanger]$ wc s_1_sequence.txt 32806276 32806276 1062955212 s_1_sequence.txt [browns02@sanger]$ grep "@" s_1_sequence.txt | wc 8201569 8201569 244422691 [browns02@sanger]$ grep A 1 "@ s_1_sequence.txt > s1.seq

[browns02@sanger]$ sed -i 's/@/>/g' s1.seq