Lab 1: UNIX Tutorial

Objectives

Become proficient at navigating and operating in a UNIX environment
Understand the tree-like structure of UNIX sytems and the directory structure of the server
Learn to log in to the remote server
Execute basic UNIX skills such as viewing, creating, moving, renaming, and editing files (cd, ls, less, mv, pwd, mkdir/rmdir, nano)
Learn to use UNIX manuals with man

Note: This UNIX primer is heavily borrowed from a lab originally designed by two excellent bioinformaticians at UC Davis, Keith Bradnam and Ian Korf. You can find their version of the UNIX primer here: UNIX and Perl Primer for Biologists

Why UNIX?

UNIX systems allow us to use a keyboard to issue commands, without the need for a graphical interface or mouse navigation. Importantly, this feature enables us to automate keyboard tasks—a necessity when working with genome-scale data.

Increasingly, the raw output of biological research is usually in the format of large text files that must be analyzed in silico. UNIX is particularly well-suited for working with such files and has several powerful and flexible commands that allow for easy data processing.

The terminal

We will be accessing the UNIX system through the terminal. The terminal provides a command line interface through which we can control the UNIX-based operating system.

To access the terminal, use the Spotlight search tool (the little magnifying glass in the top right of the menu bar), and then launch Apple’s terminal application. You can also access the terminal through the Applications menu.

Your terminal should look something like:

Accessing the server

Now, it’s time to try and log in to the server. You will need to access it remotely using a command called ssh. We will first log on to Cornell’s login server (cbsulogin). Then, we will log on to the class server (cbsufindley).

Substitute my_lab_id with your lab id and you will be prompted to enter your password after each command.

$ ssh my_lab_id@cbsulogin.tc.cornell.edu   ## substitute your lab id for my_lab_id; this allows your computer access to the class server
    <enter your password>
$ ssh cbsufindley                   ## this is our class server, where computational tasks can be performed
    <enter your password>

Note: If you are having trouble logging in to Cornell’s login server (cbsulogin), it may be because it’s too busy. If this happens, try accessing a different login server (cbsulogin2, cbsulogin3).

Navigating in UNIX

Your first UNIX command

UNIX keeps files arranged in a hierarchical structure. From the ‘top-level’ of the computer, there will be a number of directories, each of which contain files and subdirectories. Each of those can contain files and directories, ad infinitum. It’s important that you recognize that you will always be ‘in’ a particular directory when using the terminal. The default behavior is that when you open a new terminal you start in your own ‘home’ directory (containing files and directories only you can modify).

To see what files are in the home directory, us the ls command to list the directories and files.

$ ls  
Applications    Shortcuts    Documents    Library    Desktop

There are four things you should note about this:

You will probably see a different output to what is shown here. It depends on your particular directory.
The $ is a command prompt. Yours will contain a username (my_lab_id@cbsufindley) and sometimes the name of the working directory (more on that later). Throughout this text, we will simply use $ to indicate the entire command prompt.
The output of the ls above lists five things. In this case, they are all directories, but they could be files.
After the ls command finishes, it produces a new command prompt $, ready for you to type the next command.

The ls command is used to list the contents of any directory, not necessarily the one you are currently in. Try this:

$ ls /workdir/genomics
data ffinseth lab1  ## other files will likely show up as people set up their own folders

Here, you have listed the contents of a different directory, the course directory within the workdir directory of the server. This is where you will perform most of your work.

The ls command (like most UNIX commands) has a set of options that can be added to the command to change the results. Command-line options in UNIX are specified by using a dash (‘-’) after the command name, followed by various letters. If you add -l to the ls command, it will give you a longer output compared to the default.

$ ls -l /workdir/genomics
total 0
drwxrwxr-x 2 ffinseth ffinseth 46 Jan 27 18:18 data
drwxrwxr-x 2 ffinseth ffinseth 85 Jan 27 21:34 ffinseth
drwxrwxr-x 2 ffinseth ffinseth 10 Jan 27 18:03 lab1

For each directory or file, we now see more information (including ownership and modification times). Directories have a ‘d’ at the beginning of the line, files are missing this ‘d’.

There are many, many different options for the ls command. Try some of these out and see how the output changes.

$ ls -R /workdir/genomics  
$ ls -h /workdir/genomics  
$ ls -lRh /workdir/genomics

Note that the last example combines multiple commands but only uses one dash. This is a very common way of specifying command-line options.

Finding out where you are

There may be hundreds of directories an any UNIX machine. How do you know which one you are in? The command pwd will print the working directory.

$ pwd  
/home/ffinseth

When you log in to a UNIX computer, you are typically placed into your home directory. In this example, after I logged in, I was placed into a directory called ‘ffinseth’, which itself is a subdirectory of ‘home’. Conversely, ‘home’ is the parent directory of the ‘ffinseth’. The first forward slash that appears in a list of directory names is always the top level directory of the file system, commonly known as the root directory. The remaining forward slash delimits the various parts of the directory hierarchy.

As you learn UNIX, you will frequently type commands that don’t seem to work. Many times this is because you are in the wrong directory, so it’s a good habit to run pwd a lot.

** TASK: What is your working directory? **

Getting from ‘A’ to ‘B’

You are most likely in your home directory, but we will want to perform most of our work in the server’s computational powerhouse directory, somewhat confusingly named workdir. To change directories in UNIX, we use the command cd.

$ cd /workdir/ 
$ ls  
BAMS  Droso.fa  GENOME_FILES  genomics2018  krabbani3  PTCBac        Raw        root
Data  ffinseth  genomics      jmodlis       old        Python-3.6.1  rawSummer
$ pwd  
/workdir

The first command reads as ‘change the directory to the /workdir directory that is in the root directory (denoted by a /)’. Did you notice that the command prompt changed after you ran the cd command? This is a useful feature of the command prompt, as it lets you know where you are.

Note: if you get an error, it may be because you have not included the preceding slash (/), which is a necessary part of the directory address.

Up, up, and away

Frequently, you will want to go ‘upwards’ one level in the directory hierarchy. Two dots (..) are used in UNIX to refer to the parent directory of wherever you are. Every directory has a parent except the root level of the computer.

First, move into the lab1 directory of the genomics directory. Then move up one level.

$ cd genomics/lab1
$ pwd
/workdir/genomics/lab1
$ cd ..
$ pwd
/workdir/genomics/lab1

What if you wanted to navigate up two levels in the file system? Just use two sets of the .. operator, separated by a slash.

$ cd /workdir/genomics/lab1
$ pwd
/workdir/genomics/lab1
$ cd ../..
$ pwd
/workdir

Time to go home

Remember that the command prompt shows you the name of the directory that you are currently in? When you are in your home directory it shows you a tilde character (~) instead. This is because UNIX uses the tilde character as a shorthand way of specifying the home directory.

See what happens when you try the following commands (use the pwd command after each one to confirm results).

$ cd ~
$ cd

Hopefully, you should see that cd and cd ~ both take you back to the home directory.

Man your battle stations!

Recall that we previously used multiple options with the ls command.

$ ls -lrh /workdir/

But what do ‘l’, ‘r’, and ‘h’ even do? Well, thankfully every UNIX command has an associated manual that you can access by using the man command.

$ man ls

TASK: Take a look at the manual page for ls. What do the following options for ls do?

-r:

-h:

-l:

Make directories, not war

If we want to make a new directory (e.g., to store work related data), we can use the mkdir command.

$ cd /workdir/genomics
$ mkdir your_name      ## substitute your lab id for your_name
$ ls                   ## do you see your new directory listed?
$ cd your_name         ## substitute your lab id for your_name 
$ mkdir Temp
$ cd Temp
$ pwd
/workdir/genomics/your_name/Temp

TASK: Now, try to move back to /workdir only using the .. notation? How did it go? Practice a bit until you feel comfortable making and moving around in directories.

The art of typing less to do more

Saving keystrokes may not seem important, but the longer you spend typing in a terminal window, the happier you will be if you can reduce the time you spend at the keyboard. Plus, it can avoid typos. So, one of the best UNIX tricks is to tab complete the names of files and programs on most UNIX systems. Type the first few names of a file, program, or directory and press tab. UNIX will do the rest.

Let’s try it out. Later on, we will be using the command nano. Type nan and press tab. What happened? nan provided enough letters for the system to recognize that we wanted to access nano.

Now, type na and press tab. Now press tab again. While na alone was not enough information to allow for a unique match to a command, pressing tab twice will give you all the options.

TASK: This tab completion feature also works with directories. Navigate to your home directory using cd. Now, navigate to /workdir/genomics/my_lab_id using tab completion.

Working with files

Moving files

The following sections will deal with moving, renaming, removing, and looking at files. First, make a temporary file named tmp1.txt using the command touch. Then, move this file to the /workdir/genomics/my_lab_id/Temp. We will do this using the mv command.

For the mv command, we always have to specify a source file (or directory) that we want to move, and then specify a target location.

$ cd /workdir/genomics/my_lab_id
$ touch tmp1.txt
$ mv tmp1.txt /workdir/genomics/my_lab_id/Temp
$ ls
Temp
## You have moved the tmp1.txt file, so you should not see it listed here.
$ ls Temp/
## Is tmp1.txt there?  It should be.

Renaming files

In the earlier example, the destination for the mv command was a directory name (Temp). So we moved a file from its source location to a target location (source and target are important concepts for many UNIX commands). But note that the target could have been a (different) file name, rather than a directory. Try it out.

$ cd /workdir/genomics/my_lab_id/Temp
$ ls
tmp1.txt
$ mv tmp1.txt tmp2.txt
$ ls
tmp2.txt

In this example, we created used mv to move the file called tmp1.txt to a file called tmp2.txt. In UNIX, mv is essentially how we rename files, as UNIX does not have a separate rename command.

You can also use mv to move and rename directories, just as you did with files.

TASK: Try making a new directory (Temp2) in your /workdir/genomics/your_name/ folder. Move the Temp2 directory inside the Temp/ directory using mv.

The most dangerous UNIX command you will ever learn!

How can we remove files from directories? In order to do this, we must use the rm command.

Potentially, rm is a very dangerous command. UNIX does not have a trash or recycling bin. Therefore, if you delete something with rm, you will not get it back! Indeed, it is possible to delete everything in your home directory (all directories and subdirectories) with rm.

Let me repeat that. It is possible to delete EVERY file you have ever created with one simple rm command. Are you scared yet? It is also possible for you to delete everything in the course! Now that’s extra scary. Luckily, there is a way of making rm slightly less dangerous. Adding the -i makes the prompt ask for confirmation prior to deleting.

$ cd /workdir/genomics/your_name/Temp
$ ls
Temp2 tmp2.txt
$ rm -i tmp2.txt
rm: remove regular empty file ‘tmp2.txt’?
## type a 'y' and enter to confirm that you want the file removed

Isn’t the -i option helpful? From here on out, you should always add -i when you use the rm command.

Go forth and multiply

Copying files with the cp (copy) command is very similar to moving them. Remember to always specify a source and a target location. Let’s create a new file and make a copy of it.

$ cd /workdir/genomics/my_lab_id/
$ touch tmp3
$ cp tmp3 tmp4
$ ls
tmp3 tmp4

There are two files, Droso.fa and rna.fa in the /workdir/genomics/data folder. Copy these into your folder, using the cp command.

$ cd /workdir/genomics/my_lab_id
$ cp /workdir/genomics/data/Droso.fa /workdir/genomics/my_lab_id
$ cp /workdir/genomics/data/rna.fa .
$ ls
Droso.fa  rna.fa  Temp  tmp3 tmp4

The last step introduces a new concept. In UNIX, the current directory can be represented by a . (dot). You wil mostly use this for copying files to the current directory you are in.

Time to tidy up

We now have an empty directory that we should remove. To do this, use the rmdir command. This will only remove empty directories, so is quite safe to use.

$ cd /workdir/genomics/my_lab_id/Temp     
$ rmdir Temp2
$ cd ..
$ rmdir Temp
$ rm -i tmp*

In the last step, we have introduced the concept of wildcards characters. The asterisk (*) acts as a wilcard, essentially meaning ‘match anything’. Because both tmp3 and tmp4 begin with tmp, tmp* will refer to both files.

Viewing and editing files

Less is more

So far, we have covered listing the contents of directories and moving/copying/deleting either files and/or directories. Now we will briefly cover how you can look at files. In UNIX, the less command lets you view (but not edit) text files.

Let’s take a look at a file of nucleotide sequences that encode rna. Each sequence is represented by a header line that starts with a ‘>’. These sequences include information about species and gene names in the header lines (“Migut” stands for Mimulus guttatus) and have a gene ID associated with them after the period (e.g., “E00076” is the 76th gene on the fifth or “E” chromosome).

$ less rna.fa

TASK: Explore the file using by scrolling around with a mouse. How many headers are there in the file?

When you are done looking at the file, get out of less by typing q for quit.

Editing files with nano

In addition to looking at files, you may also want to create files you can manipulate or edit files that already exist. Fortunately, UNIX has a built-in text editor that is pretty handy. You can access the text editor with the command nano.

$ nano test

Your screen should look like:

Look at all those handy functions detailed across the bottom!

Type ‘this is a test’ in the file and close it by pressing ‘control’ and ‘X’ at the same time. You should say ‘y’ when asked if you want to save it. You should now have a new file called ‘test’ in your directory. If you use less to look at it, it should read ‘this is a test’.