1. Not finding help here? Sign up for a free 30min tutor trial with Chegg Tutors
    Dismiss Notice
Dismiss Notice
Join Physics Forums Today!
The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

Bash, reading a file lineby line and extracting substrings

  1. Feb 12, 2017 #1
    1. The problem statement, all variables and given/known data
    so I have this text file that I'm reading line by line
    and here is an example of a line:

    root pts/1 01-19 13:41 (10.0.0.48)

    2. Relevant equations


    3. The attempt at a solution
    How can I extract the terminal and timeframe only?

    I tried
    echo $line | cut -d : -f2, 4

    and it just gives me "1 10.0.0.48"
    is it not just delimited by spaces?

    even
    echo $line | cut -d ' ' -f2,4
    only gives me "pts/1" and they're poorly formatted.


    How can I get it to show me the terminal and timeframe only?
     
  2. jcsd
  3. Feb 12, 2017 #2

    mfb

    User Avatar
    2016 Award

    Staff: Mentor

    You specified ":" as delimiter. I'm surprised you get a meaningful result with -f2,4, as there are just two parts separated by ":", and I get them with -f1 and -f2:

    > echo "root pts/1 01-19 13:41 (10.0.0.48)" | cut -d : -f 1
    root pts/1 01-19 13
    > echo "root pts/1 01-19 13:41 (10.0.0.48)" | cut -d : -f 2
    41 (10.0.0.48)

    What you want is a separation by spaces:

    > echo "root pts/1 01-19 13:41 (10.0.0.48)" | cut -d " " -f4
    13:41
    > echo "root pts/1 01-19 13:41 (10.0.0.48)" | cut -d " " -f3,4
    01-19 13:41
    > echo "root pts/1 01-19 13:41 (10.0.0.48)" | cut -d " " -f2,4
    pts/1 13:41
    > echo "root pts/1 01-19 13:41 (10.0.0.48)" | cut -d " " -f2,3,4
    pts/1 01-19 13:41

    If your input line uses a mixture of space and tabs or other whitespace characters, then things get more complicated.
     
  4. Feb 12, 2017 #3
    in my second command, i changed ' ' to " " and it worked. What gives? Why does it differ in output
     
  5. Feb 12, 2017 #4

    mfb

    User Avatar
    2016 Award

    Staff: Mentor

    Doesn't make a difference in my shell, so I cannot reproduce it.
     
  6. Feb 12, 2017 #5
    Very odd... although thank you for guiding me down this realization
     
  7. Feb 13, 2017 #6

    NascentOxygen

    User Avatar

    Staff: Mentor

    If you are processing a large file, it is inefficient to set up a pipe and call a utility for every line you process; it is better to keep everything within bash.

    You can make use of bash's built-in pattern matching to select the desired text, e.g., here's a start
    x=${line#* }
    x=${x% *}
    echo "$x"

    I've shown it verbosely, you can condense it to a single statement.
     
  8. Feb 13, 2017 #7

    jim mcnamara

    User Avatar

    Staff: Mentor

    This is just a comment: UNIX has really very good tools designed for dealing with files specifically. bash is great on a general level, but as @NascentOxygen points out, you can wind up introducing inefficient code which bogs down to a crawl on big files. There is a tool meant to do what you want: parsing lines into fields. It is done by default with no action on your part.

    The tool is awk. You specify fields with $[field number]: $0 is the whole line, $1 is the first field, $2 is the second ... $(NF) is the last fields where NF is a variable that provides the number of fields:
    Assume you want pts/1 (second field) and (10.0.0.48) is the last field. One line of awk and you're done. You can write a new file as a side effect.
    Code (Text):

    awk '{ print $2, $(NF) }'  myinputfilename  > myoutputfilename
     
    The point is not that you have to use awk, bash may be okay, but you should always consider the whole toolset to save yourself grief. Frequently homework makes you use a less friendly tool, so that's certainly okay - which is the case here.

    FWIW: in bash, if you must use it, consider arrays
    Code (Text):

    while read line
    do
       arr=( $line )  # create an array named arr, index 0 is the first array element, we use $IFS
                           #      variable as the delimiter [i.e., spaces and tabs == default]
                           # you can specify different values for the IFS bash internal variable is you want.
       echo "${arr[1]}  ${arr[4]}"    # 1 is really the second element in the array
    done < myinputfilename  > myoutputfilename
     
     
  9. Feb 13, 2017 #8
    Thanks for the clarity. Bash feels alot different to wrap my head around than with today's systems.

    Just curious as well, say I have a textfile(list of medications and their info for example) that contains some information. If I do this:

    medDoses=$(cut -f3 <medFile.txt>)

    Would medDoses be considered a temporary file or no? And how do you clean/remove temporary files in bash?
     
  10. Feb 13, 2017 #9

    jim mcnamara

    User Avatar

    Staff: Mentor

    Oops I was distracted and had to edit things.... I cannot type well when distracted.

    It is stored as a variable, $medDoses, in memory. Not a file at all. Try
    Code (Text):
     echo "$MedDoses"
    It is interesting. Not what you would expect at all.

    Temporary files are another 'thing'. Some uses in shell that did create temporary files in the past, e.g., pipes, are now totally in memory as shared memory segments -- read [ and sometimes write] access memory shared between different processes.

    True temporary files on disk are easily made with the mkstemp command. There are standards about how commands work (FYI POSIX is the standards name), so what I'm saying is the standard. Your system may have some extra tweaks. Or if it uses a really old OS, this may vary. Read man mkstemp to see. mkstemp tries to create a file with a unique filename in the directory, so that you cannot accidentally open somebody else's file, for example. Two or more processes can have one file open simultaneously, which causes problems when you do not know that happened.

    Nothing beats an example:
    Code (Text):

    #!/bin/bash
    export tmpfile=$(mkstemp /tmp/myscript.XXXXXX)  # creates an empty file, when process exits
                                         # the function will delete the file
    finish()  # cleanup function
    {
        rm $tmpfile
    }
    trap finish EXIT
    ls -l $tmpfile   # to see what you have
    [ use more commands on the file if you want ]
    exit 0
     
    The file is deleted just before the process running the bash script is destroyed by the OS. That is what exit means basically.

    mkostemp allows the use of flags to provide finer control over file creation parameters.
     
    Last edited: Feb 13, 2017
  11. Feb 13, 2017 #10

    NascentOxygen

    User Avatar

    Staff: Mentor

    medDoses is a shell variable. It's not a file.
     
  12. Feb 13, 2017 #11

    NascentOxygen

    User Avatar

    Staff: Mentor

    When using a variable, it is often easy to break its value into pieces using read like this:

    IFS=" " read -r f1 f2 f3 f4 f5 <<<"$line"
    echo $f2 $f4 $f1 $f3 $f5
     
Know someone interested in this topic? Share this thread via Reddit, Google+, Twitter, or Facebook

Have something to add?
Draft saved Draft deleted



Similar Discussions: Bash, reading a file lineby line and extracting substrings
  1. Reading a File (Replies: 2)

Loading...