How to change a line in a file

[member 428835]

Hi PF!

I'm trying to edit two lines from files in different directories, U and controlDict. I want to change the lines with numbers from two other .dat files, VEL.dat (vector of numbers) and VEL_t.dat (single scalar number). Below is my attempt, but I know at least two things that are wrong.

# LOAD FILES TO WORK WITH
filenameVEL     = "/home/josh/Documents/NASA/PSI_DATA/Double_Drain/ICF1-9/VEL.dat"
filenameVEL_t     = "/home/josh/Documents/NASA/PSI_DATA/Double_Drain/ICF1-9/VEL_t.dat"
filenameU         = "./0/U"
filenameCD         = "./system/controlDict"

# Read filenameVEL
fileVEL         = open(filenameVEL,'r')
linesVEL         = fileVEL.readlines()
fileVEL.close

# Read filenameVEL_t
fileVEL_t         = open(filenameVEL_t,'r')
linesVEL_t         = fileVEL_t.readlines()
fileVEL_t.close

# Read filenameU
fileU             = open(filenameU,'r')
linesU             = fileU.readlines()
fileU.close

# Read filenameCD
fileCD            = open(filenameCD,'r')
linesCD         = fileCD.readlines()
fileCD.close

# CHANGE LINES OF BOTH FILES
for j in range(0,length(VEL):

    # Replace line 33 in U with RHS of equation below
    linesU[33]     = sample line here linesVEL[j]

    # Replace line 26 in controlDict with RHS of equation below
    linesCD[26]    = endTime j*linesVEL_t[1]

    os.system("mpirun -np 16 interFoam -parallel")                        # RUN interFoam IN PARALLEL

1) in line 28, I use length(VEL) to signify the length of the row number from the VEL.dat file. This is incorrect, but I can't figure out how to do it.

2) the RHS of line 34 should paste "endTime NUMBER" where "NUMBER" is a real number. Example: if VEL_t is a .dat file with lone entry 12 and j = 2, then line 26 in the controlDict file should be replaced with "endTime 24".

Really appreciate any help you have to offer. I've googled as much as I can to get this far (and with the help of some members here too!).

 

[PeterDonis]

in line 28, I use length(VEL) to signify the length of the row number from the VEL.dat file. This is incorrect, but I can't figure out how to do it.

In Python the function that gives the number of elements in an iterable (e.g., items in a list or characters in a string) is called len. Another of those curve balls. Python was originally designed in the 1990s, and characters in function and variable names were still considered a scarce resource then.

However, I'm not sure why you are trying to take the length of VEL, since that variable doesn't appear anywhere in the code you gave.

the RHS of line 34 should paste "endTime NUMBER" where "NUMBER" is a real number.

The simplest way to do this in Python is to just concatenate strings; the addition operator does this, so you would have "endTime " + str(NUMBER), where the str function gives you a string representation of its argument. You could also do it with string formatting, but if you're just learning Python that's probably overkill.

Note also that "endTime " is in quotes with a space at the end (since it looks like there is supposed to be a space before the number). In Python string literals have to always be quoted.

I think you also need something similar in line 31.

 

[PeterDonis]

A couple of other items:

(1) It's good practice to use the with statement when opening files, since that ensures that the file gets closed even if there is an error; for example:

with open(filename, 'r') as f:
    lines = f.readlines()

When the with block ends the file is automatically closed and the lines variable has the lines that were read.

(2) I don't see where any file data is written back out anywhere. So the code you've shown won't actually change any data on disk, which means each run of the external program will see the same data, which is not what you want. So it looks to me like before the os.system call in each run of the loop, you need something like:

with open(filenameU, 'w') as f:
    f.writelines(linesU)

with open(filenameCD, 'w') as f:
    f.writelines(linesCD)

This writes out the changed data to disk so the external program can see it.

 

[member 428835]

However, I'm not sure why you are trying to take the length of VEL, since that variable doesn't appear anywhere in the code you gave.

I'm trying to read each component of VEL.dat, and so I need the for loop to loop over all inputs in VEL: does the code below do this?

# Read filenameVEL
with open(filenameVEL, 'r') as f:
    linesVEL     = f.readlines()

with open(filenameVEL_t, 'r') as f:
    linesVEL_t     = f.readlines()

with open(filenameU, 'r') as f:
    linesU         = f.readlines()

with open(filenameCD, 'r') as f:
    linesCD     = f.readlines()

# CHANGE LINES OF FILES
for j in range(0,len(linesVEL)):

    # Replace line 33 in U with RHS of equation below
    linesU[33]     = "sample line here linesVEL[j]"

    # Replace line 26 in controlDict with RHS of equation below
    linesCD[26]    = "endTime " + str(j*linesVEL_t[1]) +";"

    # write file to disk
    with open(filenameU, 'w') as f:
        f.writelines(linesU)

    # write file to disk
    with open(filenameCD, 'w') as f:
        f.writelines(linesCD)

    os.system("mpirun -np 16 interFoam -parallel")                    # RUN interFoam IN PARALLEL

Also, can you explain what the

    with open(filenameU, 'w') as f:
        f.writelines(linesU)

does exactly? It's unclear why I need it.

Your posts are VERY helpful!

EDIT: in line 18, would I actually have to write

    linesU[33]     = "sample line here " + str(linesVEL[j])

 

[DaveC426913]

Also, can you explain what the

with open(filenameU, 'w') as f:
f.writelines(linesU)

does exactly? It's unclear why I need it.

This is defining where exactly it's writing to.

f is a pointer to the file called filenameU.
'w' means for write access
Take f, which pints to the output file, and write all the lines to it.

 

[PeterDonis]

I'm trying to read each component of VEL.dat, and so I need the for loop to loop over all inputs in VEL: does the code below do this?

Yes.

It's unclear why I need it.

I'm not sure what's unclear. You are reading in data from two files (the "U" and "CD" files), and then making changes to that data; each time you change the data (i.e., on each iteration of the for loop), you have to write it back out to the file. Otherwise the external program you are running won't see the data you changed. The external program can't see any data inside your Python program.

in line 18, would I actually have to write

Yes.

 

[member 428835]

I'm not sure what's unclear. You are reading in data from two files (the "U" and "CD" files), and then making changes to that data; each time you change the data (i.e., on each iteration of the for loop), you have to write it back out to the file. Otherwise the external program you are running won't see the data you changed. The external program can't see any data inside your Python program.

Got it! I was unclear what the script did but DaveC426913 explained. What you say makes complete sense, and you implied it earlier too: I appreciate it!

I do have two errors that come up. The first: I switched line 18 to look this this

    linesU[32]  = "        value           uniform (0 "+str(linesVEL[j])+"0);"

The U file initially looks like this

    outlet
    {
    type            fixedValue;
        value           uniform (0   -1 0);
    }

but after running the script it looks like this

    outlet
    {
    type            fixedValue;
        value           uniform (0   -3.8871994e-03
0);    }

which makes me think entry linesVEL[j] has a new line built into it. Do you know how I can suppress this?

Second: after running the line

    linesCD[25] = "endTime "+str(j*linesVEL_t[1]*0.01)+";"

the terminal outputs

Traceback (most recent call last):
  File "runSim.py", line 45, in <module>
    linesCD[25] = "endTime "+str(j*linesVEL_t[1]*0.01)+";"
IndexError: list index out of range
[3]+  Done                    gedit /home/josh/Documents/NASA/PSI_DATA/Double_Drain/ICF1-9/VEL_t.dat

I checked and VEL_t has 1 input and controlDict has 80 lines of code. You can view these https://github.com/joshmccraney/python-script-to-change-lines. Any ideas what's going wrong?

 

[PeterDonis]

which makes me think entry linesVEL[j] has a new line built into it

It probably does, since there is one in the file (there has to be to mark the end of a line), and Python's readlines method doesn't filter those out. You can strip whitespace from the ends of a string in Python (which should take care of this) with the strip method of string objects, like this: linesVEL[j].strip().

If the string "-1" is always going to be in the original file in that location, you could also just do a string replace, like this: linesU[32] = linesU[32].replace("-1", linesVEL[j].strip()). That way you don't have to put all the extra stuff that's in the line in your code. This technique might also work for the other data changes you're doing.

VEL_t has 1 input

That means it will be at line $0$, not line $1$. Python indexes into sequences are zero-based. So you want linesVEL_t[0].

 

[member 428835]

It probably does, since there is one in the file (there has to be to mark the end of a line), and Python's readlines method doesn't filter those out. You can strip whitespace from the ends of a string in Python (which should take care of this) with the strip method of string objects, like this: linesVEL[j].strip().

Thanks! The final solution looks like this

    linesU[32]  = "        value           uniform (0 "+str(linesVEL[j].strip())+" 0); \n"

If the string "-1" is always going to be in the original file in that location, you could also just do a string replace, like this: linesU[32] = linesU[32].replace("-1", linesVEL[j].strip()). That way you don't have to put all the extra stuff that's in the line in your code. This technique might also work for the other data changes you're doing.
That means it will be at line $0$, not line $1$. Python indexes into sequences are zero-based. So you want linesVEL_t[0].

Unfortunately the number will likely never be -1, but this is good to know for future use! And ugh, that indexing is going to take some getting used to... Thanks a ton for your help: the script does exactly what I want it to do!

 

[PeterDonis]

Thanks a ton for your help: the script does exactly what I want it to do!

You're welcome! I'm glad it's working.

 

[member 428835]

You've no idea how much time this will help me save (or perhaps you do?)

 

[PeterDonis]

You've no idea how much time this will help me save (or perhaps you do?)

Perhaps in a general way.

 

[member 428835]

Last (I hope) issue with this script (finalized below). Line 53 is executed by the terminal before lines 39 and 47 are completed. I know this because if I change lines 32 and 26 in the separate files to "blah", I get an error message regarding "blah". Evidently there is a method to have the program wait to execute the next line until the previous is completed, though I'm not too sure. Any recommendations?

#!/usr/bin/python3

#-----------------------SCRIPT DESCRITION-----------------------#
# THIS SCRIPT CHANGES endTime IN system/controlDict AND ALSO
# CHANGES THE VELOCITY OUTPUT IN 0/U TO MATCH ISS VALUES, IMPORTED
# FROM A DIFFERENT DIRECTORY
#---------------------------------------------------------------#

import subprocess
import os

from subprocess import call

subprocess.call(['./preProc.sh'])    # RUN preProc.sh

# LOAD FILES TO WORK WITH
filenameVEL      = "/home/josh/Documents/NASA/PSI_DATA/Double_Drain/ICF1-9/VEL.dat"
filenameVEL_t    = "/home/josh/Documents/NASA/PSI_DATA/Double_Drain/ICF1-9/VEL_t.dat"
filenameU        = "./0/U"
filenameCD       = "./system/controlDict"

# Read filenameVEL
with open(filenameVEL, 'r') as f:
    linesVEL    = f.readlines()

with open(filenameVEL_t, 'r') as f:
    linesVEL_t  = f.readlines()

with open(filenameU, 'r') as f:
    linesU      = f.readlines()

with open(filenameCD, 'r') as f:
    linesCD     = f.readlines()

# CHANGE LINES OF FILES
for j in range(0,len(linesVEL)):

    # Replace line 33 in U with RHS of equation below
    linesU[32]  = "        value           uniform (0 "+str(linesVEL[j].strip())+" 0); \n"

    # write file to disk
    with open(filenameU, 'w') as f:
        f.writelines(linesU)

    # Replace line 26 in controlDict with RHS of equation below
    timeStep    = int(round(float(linesVEL_t[0])))                    # MAKE VEL_t AN INTEGER TO AVOID ROUNDOFF ERROR
    linesCD[25] = "endTime         "+str((j+1)*timeStep)+"; \n"

    # write file to disk
    with open(filenameCD, 'w') as f:
        f.writelines(linesCD)

    os.system("mpirun -np 16 interFoam -parallel")    # RUN interFoam IN PARALLEL

 

[PeterDonis]

Line 53 is run before lines 39 and 47. I know this because if I change these lines in the file to "blah", I get an error message regarding "blah".

Lines 39 and 47 don't affect anything in the external files, so I don't understand your reasoning here. Do you perhaps mean lines 43 and 51?

I'm also not sure exactly what is going wrong. What is happening when the external interFoam command is run that isn't supposed to happen?

Also, it would make your code more readable if you either put spaces around the "+" signs when concatenating strings, or used Python's string formatting instead.

 

[PeterDonis]

Evidently there is a method to have the command wait

Do you mean the issue is that the new data isn't written to the files before the interFoam command is run?

If that's the case, you can force the files to be flushed to disk by calling f.flush() after each writelines. I'm not entirely sure why that should be necessary; AFAIK when a file is closed (which should happen when the with block is exited), it is flushed to disk. But the flush might possibly be delayed by the filesystem driver if flush is not called explicitly.

 

[member 428835]

Lines 39 and 47 don't affect anything in the external files, so I don't understand your reasoning here. Do you perhaps mean lines 43 and 51?

Yes, thank you! :)

I'm also not sure exactly what is going wrong. What is happening when the external interFoam command is run that isn't supposed to happen?

Lines 43 and 51 are being run in Python before line 53, but the computer has not made these changes to the external files before line 53 is executed by the computer. I know this because line 33 in the external file we change titled U should be rewritten by the python script, so anything I write in line 33 shouldn't matter. But when I run this script, it chokes, and the terminal gives me an error about exactly what I wrote.

Also, it would make your code more readable if you either put spaces around the "+" signs when concatenating strings, or used Python's string formatting instead.

Done!

 

[member 428835]

Never mind, someone else answered this question here. Thanks for everything though!

 

[PeterDonis]

someone else answered this question here.

I think you're being a little premature in considering the problem solved. I'll respond further in the other thread.

 

[member 428835]

I think you're being a little premature in considering the problem solved. I'll respond further in the other thread.

Cool, thanks!