# Bash, reading a file lineby line and extracting substrings

1. Feb 12, 2017

### Arnoldjavs3

1. The problem statement, all variables and given/known data
so I have this text file that I'm reading line by line
and here is an example of a line:

root pts/1 01-19 13:41 (10.0.0.48)

2. Relevant equations

3. The attempt at a solution
How can I extract the terminal and timeframe only?

I tried
echo $line | cut -d : -f2, 4 and it just gives me "1 10.0.0.48" is it not just delimited by spaces? even echo$line | cut -d ' ' -f2,4
only gives me "pts/1" and they're poorly formatted.

How can I get it to show me the terminal and timeframe only?

2. Feb 12, 2017

### Staff: Mentor

You specified ":" as delimiter. I'm surprised you get a meaningful result with -f2,4, as there are just two parts separated by ":", and I get them with -f1 and -f2:

> echo "root pts/1 01-19 13:41 (10.0.0.48)" | cut -d : -f 1
root pts/1 01-19 13
> echo "root pts/1 01-19 13:41 (10.0.0.48)" | cut -d : -f 2
41 (10.0.0.48)

What you want is a separation by spaces:

> echo "root pts/1 01-19 13:41 (10.0.0.48)" | cut -d " " -f4
13:41
> echo "root pts/1 01-19 13:41 (10.0.0.48)" | cut -d " " -f3,4
01-19 13:41
> echo "root pts/1 01-19 13:41 (10.0.0.48)" | cut -d " " -f2,4
pts/1 13:41
> echo "root pts/1 01-19 13:41 (10.0.0.48)" | cut -d " " -f2,3,4
pts/1 01-19 13:41

If your input line uses a mixture of space and tabs or other whitespace characters, then things get more complicated.

3. Feb 12, 2017

### Arnoldjavs3

in my second command, i changed ' ' to " " and it worked. What gives? Why does it differ in output

4. Feb 12, 2017

### Staff: Mentor

Doesn't make a difference in my shell, so I cannot reproduce it.

5. Feb 12, 2017

### Arnoldjavs3

Very odd... although thank you for guiding me down this realization

6. Feb 13, 2017

### Staff: Mentor

If you are processing a large file, it is inefficient to set up a pipe and call a utility for every line you process; it is better to keep everything within bash.

You can make use of bash's built-in pattern matching to select the desired text, e.g., here's a start
x=${line#* } x=${x% *}
echo "$x" I've shown it verbosely, you can condense it to a single statement. 7. Feb 13, 2017 ### jim mcnamara ### Staff: Mentor This is just a comment: UNIX has really very good tools designed for dealing with files specifically. bash is great on a general level, but as @NascentOxygen points out, you can wind up introducing inefficient code which bogs down to a crawl on big files. There is a tool meant to do what you want: parsing lines into fields. It is done by default with no action on your part. The tool is awk. You specify fields with$[field number]: $0 is the whole line,$1 is the first field, $2 is the second ...$(NF) is the last fields where NF is a variable that provides the number of fields:
Assume you want pts/1 (second field) and (10.0.0.48) is the last field. One line of awk and you're done. You can write a new file as a side effect.
Code (Text):

awk '{ print $2,$(NF) }'  myinputfilename  > myoutputfilename

The point is not that you have to use awk, bash may be okay, but you should always consider the whole toolset to save yourself grief. Frequently homework makes you use a less friendly tool, so that's certainly okay - which is the case here.

FWIW: in bash, if you must use it, consider arrays
Code (Text):

do
arr=( $line ) # create an array named arr, index 0 is the first array element, we use$IFS
#      variable as the delimiter [i.e., spaces and tabs == default]
# you can specify different values for the IFS bash internal variable is you want.
echo "${arr[1]}${arr[4]}"    # 1 is really the second element in the array
done < myinputfilename  > myoutputfilename

8. Feb 13, 2017

### Arnoldjavs3

Thanks for the clarity. Bash feels alot different to wrap my head around than with today's systems.

Just curious as well, say I have a textfile(list of medications and their info for example) that contains some information. If I do this:

medDoses=$(cut -f3 <medFile.txt>) Would medDoses be considered a temporary file or no? And how do you clean/remove temporary files in bash? 9. Feb 13, 2017 ### jim mcnamara ### Staff: Mentor Oops I was distracted and had to edit things.... I cannot type well when distracted. It is stored as a variable,$medDoses, in memory. Not a file at all. Try
Code (Text):
echo "$MedDoses" It is interesting. Not what you would expect at all. Temporary files are another 'thing'. Some uses in shell that did create temporary files in the past, e.g., pipes, are now totally in memory as shared memory segments -- read [ and sometimes write] access memory shared between different processes. True temporary files on disk are easily made with the mkstemp command. There are standards about how commands work (FYI POSIX is the standards name), so what I'm saying is the standard. Your system may have some extra tweaks. Or if it uses a really old OS, this may vary. Read man mkstemp to see. mkstemp tries to create a file with a unique filename in the directory, so that you cannot accidentally open somebody else's file, for example. Two or more processes can have one file open simultaneously, which causes problems when you do not know that happened. Nothing beats an example: Code (Text): #!/bin/bash export tmpfile=$(mkstemp /tmp/myscript.XXXXXX)  # creates an empty file, when process exits
# the function will delete the file
finish()  # cleanup function
{
rm $tmpfile } trap finish EXIT ls -l$tmpfile   # to see what you have
[ use more commands on the file if you want ]
exit 0

The file is deleted just before the process running the bash script is destroyed by the OS. That is what exit means basically.

mkostemp allows the use of flags to provide finer control over file creation parameters.

Last edited: Feb 13, 2017
10. Feb 13, 2017

### Staff: Mentor

medDoses is a shell variable. It's not a file.

11. Feb 13, 2017

### Staff: Mentor

When using a variable, it is often easy to break its value into pieces using read like this:

IFS=" " read -r f1 f2 f3 f4 f5 <<<"$line" echo$f2 $f4$f1 $f3$f5