Running CFD with python and bash: not in agreement

[member 428835]

Hi PF!

I'm running a CFD (computational fluid dynamics) program OpenFOAM. Liquid is sucked from a tank, where the velocity at the tank outlet (suction point) is controlled by a python script. The boundary condition for velocity at the outlet is below, line 4 being the velocity prescribed at the outlet.

    outlet
    {
    type            fixedValue;
        value           uniform (0 -2.6254593e-03 0); 
    }

The python script changes the entire line 4 to a different input (same form, different float). When I run the python script, the volumetric flow rate is off by a lot. However, when I hard code the outlet BC and run it through bash, it runs fine.

THIS IS THE WEIRD PART: after running through bash, when I re-run through python, the volumetric flow rate is perfect. WHAT'S EVEN WEIRDER is after it works fine, if I copy the entire work file and run it again, the same error occurs.

Anyone have any experience with this? Very confusing for me.

 

[phyzguy]

Are you sure the Python script is really changing the file before the CFD program is running it? Maybe the file is changed, but doesn't get written before the CFD program reads it. Try having the Python program read back the changed file and print it out. Or add a wait statement to make sure the file is finished writing. We could help more if you showed us the Python script.

 

[member 428835]

Are you sure the Python script is really changing the file before the CFD program is running it? Maybe the file is changed, but doesn't get written before the CFD program reads it. Try having the Python program read back the changed file and print it out. Or add a wait statement to make sure the file is finished writing. We could help more if you showed us the Python script.

Yea, I think the Python script changes the file: let me explain. When I open the file with the BC in post 1, and then run the python script, I get an alert at the top of the file saying "The file “/home/josh/OpenFOAM/josh…9/quarter_roof_3/case/0/U” changed on disk." When I select "Reload" I can view the file as it is currently. And it's perfect: identical to running it through bash with me hardcoding the BC.

Here's the python script

#!/usr/bin/python3

#-----------------------SCRIPT DESCRITION-----------------------#
# THIS SCRIPT CHANGES endTime IN system/controlDict AND ALSO
# CHANGES THE VELOCITY OUTPUT IN 0/U TO MATCH ISS VALUES, IMPORTED
# FROM A DIFFERENT DIRECTORY
#---------------------------------------------------------------#

import subprocess
import os

from subprocess import call

subprocess.call(['./preProc.sh'])    # RUN preProc.sh

# LOAD FILES TO WORK WITH
filenameVEL      = "/home/josh/Documents/NASA/PSI_DATA/Double_Drain/ICF1-9/VEL.dat"
filenameVEL_t    = "/home/josh/Documents/NASA/PSI_DATA/Double_Drain/ICF1-9/VEL_t.dat"
filenameU        = "./0/U"
filenameCD       = "./system/controlDict"

# Read filenameVEL
with open(filenameVEL, 'r') as f:
    linesVEL    = f.readlines()

with open(filenameVEL_t, 'r') as f:
    linesVEL_t  = f.readlines()

with open(filenameU, 'r') as f:
    linesU      = f.readlines()

with open(filenameCD, 'r') as f:
    linesCD     = f.readlines()

# CHANGE LINES OF FILES
for j in range(0,len(linesVEL)):

    # Replace line 33 in U with RHS of equation below
    linesU[32]  = "        value           uniform (0 "+str(linesVEL[j].strip())+" 0); \n"

    # write file to disk
    with open(filenameU, 'w') as f:
        f.writelines(linesU)

    # Replace line 26 in controlDict with RHS of equation below
    timeStep    = int(round(float(linesVEL_t[0])))                    # MAKE VEL_t AN INTEGER TO AVOID ROUNDOFF ERROR
    linesCD[25] = "endTime         "+str((j+1)*timeStep)+"; \n"

    # write file to disk
    with open(filenameCD, 'w') as f:
        f.writelines(linesCD)

    os.system("mpirun -np 16 interFoam -parallel")    # RUN interFoam IN PARALLEL
    # os.system("mpirun -np 16 interFoam -parallel > log.interFoam &")# 'tail -f log.interFoam' TO PRINT CURRENT LOG TO SCREEN

 

[phyzguy]

Yes, I see, but probably it has not finished writing the file before the mpirun launches. You either need to include the Popen.wait commands, like I posted in another thread, or you need to put in a long wait command, or you need to read back the file and verify that the line is actually changed before you launch the mpirun command.

 

[member 428835]

Yes, I see, but probably it has not finished writing the file before the mpirun launches. You either need to include the Popen.wait commands, like I posted in another thread, or you need to put in a long wait command, or you need to read back the file and verify that the line is actually changed before you launch the mpirun command.

Okay, thanks! I'll check this out.

As a possible dummy-check to see if the code is written before mpirun launches, would I be able to simply write nonsense on line 4, so that there would be an error if it ran without the Python script rewriting the line? Then if mpirun launches error free, doesn't this imply it has definitely finished writing the file before mpirun launches?

 

[phyzguy]

Okay, thanks! I'll check this out.

As a possible dummy-check to see if the code is written before mpirun launches, would I be able to simply write nonsense on line 4, so that there would be an error if it ran without the Python script rewriting the line? Then if mpirun launches error free, doesn't this imply it has definitely finished writing the file before mpirun launches?

No, that won't tell you if it has finished. The old file could still be intact. These disc operations like reading and writing can take a very long time compared to the operation of the program, and they run independently of the thread of your program. If you want a check, I would suggest adding code at line 52 that reads the file you just wrote again, and prints out the line you changed.

Edit: Maybe I see what you mean. If you manually wrote nonsense in line 4 before you run the Python script, then I think this would be a check to see if the file has been changed by the program before you launch the mpirun.

 

[pbuk]

Edit: see if this works - I have to refresh my memory on how file operations in Python work so I'm not sure.

    # write file to disk
    with open(filenameU, 'w') as f:
        f.writelines(linesU)
        # Complete all file operations on f and write it synchronously.
        f.flush()
        os.fsync(f)

 

[member 428835]

No, that won't tell you if it has finished. The old file could still be intact. These disc operations like reading and writing can take a very long time compared to the operation of the program, and they run independently of the thread of your program. If you want a check, I would suggest adding code at line 52 that reads the file you just wrote again, and prints out the line you changed.

Edit: Maybe I see what you mean. If you manually wrote nonsense in line 4 before you run the Python script, then I think this would be a check to see if the file has been changed by the program before you launch the mpirun.

Wow, sure enough when I write "blah" I get an error output complaining about "blah". Had no idea this could even be an issue: thank you!

Okay, so I am now checking out the Popen command, but I'm unsure how to implement it using the "with" command. Online seems to suggest using the following

process = subprocess.Popen(["your_cmd"]...)
process.wait()

but I'm using

with open(filenameVEL, 'r') as f:
    linesVEL    = f.readlines()

Any help to further this is much appreciated!

pbuk, thanks for your response! It seems like your suggestion applies to troubleshooting the previous issue, but I think it's been diagnosed.

 

[phyzguy]

Well, I'm not necessarily proud of this, since I'm sure there are more elegant ways to do this, but this is what I have done in the past, and this works:

import os, time
from subprocess import Popen

filename = 'temp.txt'
# Read filename
file = open(filename,'r')
lines = file.readlines()
file.close()

# Delete the original file
delFile = Popen('rm %s'%filename, shell=True)
Popen.wait(delFile)

# Replace line 1
lines[1] = 'DEF\n'

# Write out the new filename
newfile = open(filename,'w')
for line in lines:
        newfile.write(line)
newfile.close()

# Verify that the new file has been written:

numTries = 10
waitTime = 0.1
count = 0
while count < numTries and not os.path.exists(filename):
        count += 1
        time.sleep(waitTime)

if count >= numTries:
        print("File writing failed!")
        sys.exit()
else:
        print("Success after %d waits"%count)

 

[member 428835]

Well, I'm not necessarily proud of this, since I'm sure there are more elegant ways to do this, but this is what I have done in the past, and this works:

import os, time
from subprocess import Popen

filename = 'temp.txt'
# Read filename
file = open(filename,'r')
lines = file.readlines()
file.close()

# Delete the original file
delFile = Popen('rm %s'%filename, shell=True)
Popen.wait(delFile)

# Replace line 1
lines[1] = 'DEF\n'

# Write out the new filename
newfile = open(filename,'w')
for line in lines:
        newfile.write(line)
newfile.close()

# Verify that the new file has been written:

numTries = 10
waitTime = 0.1
count = 0
while count < numTries and not os.path.exists(filename):
        count += 1
        time.sleep(waitTime)

if count >= numTries:
        print("File writing failed!")
        sys.exit()
else:
        print("Success after %d waits"%count)

I'm not too great at Python, but it looks like what you are doing is using the time.sleep() function to pause everything for a fraction of a second? Is that all, or am I missing something?

 

[PeterDonis]

I'm sure there are more elegant ways to do this

There are indeed.

The basic issue is, we write data to a file, and we want to make sure that data is actually written to disk before we run an external program.

Deleting the old file via an external shell script (btw, Python's os module has functions to do this, you don't need to shell out to do it), writing a new file (which can be done in a single writelines call, you don't need an explicit loop), then testing to see if the filename exists, won't necessarily solve this problem; if the filename exists, you don't know for sure whether it's the old file, not yet deleted (since waiting on the rm external process to exit does not guarantee that the file is actually deleted, since the OS could still buffer the write) or the new file you wrote out.

The obvious solution to the problem is, after you write the data to the file (and, as I suggested in another thread, call flush on the file before closing it--or exiting the with block, which should be used to ensure the file gets closed even if an exception is thrown), then open the file again for reading, read in the data, and compare it with the data you just wrote. If they're the same, you're good.

it looks like what you are doing is using the time.sleep() function to pause everything for a fraction of a second?

That's what time.sleep does, yes.

Btw, I have mentioned the Python documentation several times now in some threads of yours. Have you looked at it? A lot of the questions you are asking can be answered by reading the documentation. Python's docs, I find, are pretty good in that respect (which is not true of all programming languages or libraries).

 

[phyzguy]

@joshmccraney: @PeterDonis' solution is clearly better than mine. Do you understand how to implement it? Can you share some code which does this?

 

[member 428835]

The obvious solution to the problem is, after you write the data to the file (and, as I suggested in another thread, call flush on the file before closing it--or exiting the with block, which should be used to ensure the file gets closed even if an exception is thrown), then open the file again for reading, read in the data, and compare it with the data you just wrote. If they're the same, you're good.

@joshmccraney: @PeterDonis' solution is clearly better than mine. Do you understand how to implement it? Can you share some code which does this?

Unclear to me if @phyzguy is asking me or @PeterDonis but I'll take a stab at it (lines 45-51):

#!/usr/bin/python3

#-----------------------SCRIPT DESCRITION-----------------------#
# THIS SCRIPT CHANGES endTime IN system/controlDict AND ALSO
# CHANGES THE VELOCITY OUTPUT IN 0/U TO MATCH ISS VALUES, IMPORTED
# FROM A DIFFERENT DIRECTORY
#---------------------------------------------------------------#

import os, subprocess

from subprocess import call

subprocess.call(['./preProc.sh'])    # RUN preProc.sh

# LOAD FILES TO WORK WITH
filenameVEL      = "/home/josh/Documents/NASA/PSI_DATA/Double_Drain/ICF1-9/VEL.dat"
filenameVEL_t    = "/home/josh/Documents/NASA/PSI_DATA/Double_Drain/ICF1-9/VEL_t.dat"
filenameU        = "./0/U"
filenameCD       = "./system/controlDict"

# READ FILES
with open(filenameVEL, 'r') as f:
    linesVEL    = f.readlines()

with open(filenameVEL_t, 'r') as f:
    linesVEL_t  = f.readlines()

with open(filenameU, 'r') as f:
    linesU      = f.readlines()

with open(filenameCD, 'r') as f:
    linesCD     = f.readlines()

# CHANGE LINES OF FILES
for j in range(0,len(linesVEL)):

    # REPLACE LINE 33 IN U WITH RHS OF EQUATION BELOW
    line32_replace = "        value           uniform (0 " + str(linesVEL[j].strip()) + " 0); \n"
    linesU[32]  = line32_replace

    # WRITE FILE TO DISK
    with open(filenameU, 'w') as f:
        f.writelines(linesU)

     # REREAD DATA
    with open(filenameU, 'r') as f:
        linesU      = f.readlines()

    if linesU[32] != line32_replace
        print("File writing failed!")
        sys.exit()

     os.system("mpirun -np 16 interFoam -parallel")    # RUN interFoam IN PARALLEL

Btw, I have mentioned the Python documentation several times now in some threads of yours. Have you looked at it? A lot of the questions you are asking can be answered by reading the documentation. Python's docs, I find, are pretty good in that respect (which is not true of all programming languages or libraries).

Yes, I've been reading their documentation as well as a lot of forums. I always put in a lot of time before asking on here how to do something. I apologize for appearing lazy (which shows how tough this is for me). But I'll learn it: just take some time.

 

[phyzguy]

@joshmccraney I meant for you to try it. Thanks. This looks OK, but rather than have it quit if the line doesn't match, I would have it loop several times with a wait statement. If it just hasn't finished writing, you would rather the program waited until it finished instead of just quitting.

 

[PeterDonis]

I'll take a stab at it (lines 45-51)

Yes, what you've coded is what I was suggesting. The only thing I would add is f.flush() after the call to writelines.

rather than have it quit if the line doesn't match, I would have it loop several times with a wait statement

I assume you mean a call to time.sleep. You're not waiting on a subprocess at that point.

I would also limit the number of times it can loop.

 

[phyzguy]

I assume you mean a call to time.sleep. You're not waiting on a subprocess at that point.

Yes, that's what I meant.

I would also limit the number of times it can loop.

Agreed. Like I did in Post #10.

 

[PeterDonis]

I would have it loop several times

Btw, unlike the case of deleting a file, then writing a new one, there is actually no need to loop for the case of writing the file, then reading back the same file. The filesystem guarantees that if you write to a file, close it, then open it for reading and read it, you must get back the same data you just wrote. (At least, AFAIK every filesystem that is actually on a production computer does this.) So the only reason for the data to not match would be a problem with the disk, and that won't get fixed by trying again. Looping won't do any harm as long as the number of loops is limited, but it won't help any either.

The key difference in the two cases is that deleting a file and then writing a new file with the same name is operating on two different files. The new one you write, even though it has the same name, is not "the same file" as the one you just deleted. They are two different files as far as the filesystem is concerned, and the filesystem makes no guarantees about the order of operations on two different files. To get a little more technical: the first operation, deleting a file, is actually an operation on the directory the file is in--it deletes the file's name from the directory. The second operation, writing the new file, then puts back the same file name into the directory, but now pointing to a totally different file that, as far as the filesystem is concerned, has no connection with the old file that got deleted and happened to have the same name. Even the directory entries will be different.

When you write to a file and then read back from the same file, however, you aren't changing any associations between file names and files--you aren't changing directory entries at all (technically you are updating the file modification and access times in the directory entry, but that doesn't change which entry it is)--so the file you read from is guaranteed to be the same file you wrote to, which means the filesystem now does guarantee order of operations--the write has to finish before the read starts, since that's the order in which you called those filesystem functions on the same file.

 

[pbuk]

Hmmm, I'm not sure about these answers.

There is no nead to call f.flush() or os.sync(f) when you use with, this is done for you (in f.__exit__ which with calls before completing). And there is no need to test the writes are successful - if they are not then with will throw an error and the code will terminate without moving on.

The problem I think you have with your code is not that the file operations are asychronous, because using with makes them synchronous.

No, I think the problem is THIS asynchronous call:

     os.system("mpirun -np 16 interFoam -parallel")    # RUN interFoam IN PARALLEL

Every time you go through this loop:

for j in range(0,len(linesVEL)):

you are modifying an input file for interFoam. You then start an instance of interFoam running and then immediately go back around the loop, modifying the input file but you have no way of knowing if that last instance of interFoam has finished reading it yet.

If you want to execute multiple instances of interFoam in parallel then you need to set up separate input files (or separate lines in the same file) for each instance. Have you read and understood https://cfd.direct/openfoam/user-guide/v6-running-applications-parallel/ ?

 

[PeterDonis]

There is no nead to call f.flush() or os.sync(f) when you use with, this is done for you (in f.__exit__ which with calls before completing).

Is it? AFAIK the __exit__ method just calls the file object's close method, which does not automatically flush or sync the file, it just closes the file descriptor. That leaves it up to the OS and the filesystem driver when to actually write the data to the physical disk.

You then start an instance of interFoam running and then immediately go back around the loop, modifying the input file but you have no way of knowing if that last instance of interFoam has finished reading it yet.

This looks like a valid point to me. The subprocess module would be better for this case since it allows you to wait for the subprocess to complete (in fact the most common convenience methods in that module do this automatically).

 

[Tom.G]

How about comparing the file times before and after writing? When it changes, at least the OS recognizes that a change has occured.
I'm not familiar with Linux so have no idea if this would actually work.

 

[Svein]

I had (almost) the same issue (as usual: way back). One computer was writing data files to a directory and my program was supposed to analyze the content of those files. But the files would show up in the directory before the writer was done and the file was closed. The solution: Before doing anything I tried to move the files to my own directory. Since this was an OS operation, it would not move files that were open (and it would give an error message which I ignored). Then I would analyze the files that existed in my own directory (and delete them afterwards).

 

[pbuk]

Is it? AFAIK the __exit__ method just calls the file object's close method, which does not automatically flush or sync the file, it just closes the file descriptor. That leaves it up to the OS and the filesystem driver when to actually write the data to the physical disk.

According to the Python3 docs, close does flush (and by inference sync) before releasing the handle:

close()
Flush and close this stream...

This is just as well because none of the with tutorials I have seen use flush, sync or close.

This looks like a valid point to me. The subprocess module would be better for this case since it allows you to wait for the subprocess to complete (in fact the most common convenience methods in that module do this automatically).

Yes - apart from anything else there's not a lot of sense in launching concurrently more than one set of threads each using all of your cores in parallel.