Python Running a for loop in parallel via multiprocessing

AI Thread Summary
The discussion centers on a Python script utilizing the multiprocessing module to perform numerical integration in parallel. The user encounters an issue where the results stored in list K appear as zeros when printed from the main process. The confusion arises because each subprocess has its own instance of K, which does not affect the K in the master process. The results of the computations are actually stored in a separate variable called "result," which is not printed in the main function. To resolve the issue, it is suggested to print the "result" variable in the main function to view the computed values. Additionally, the conversation highlights the challenges of debugging multiprocessing code, emphasizing the need for a mental shift in understanding how data is handled across different processes. The user is advised to refer to the Python documentation for best practices with the multiprocessing module.
member 428835
Hi PF!

I'm wanting to run a function in parallel, which I've denoted integrate below, on line 33.
Python:
from math import *
from random import *
import math, scipy.special
import statistics
import multiprocessing as mp
# READ MATHEAMTICA FUNCTIONS
with open("funcL.txt") as fileL:
    fL = fileL.readlines()
with open("funcR.txt") as fileR:
    fR = fileR.readlines()
# DEFINE MATH REQUIRED
Pi = math.pi
def Sqrt(x):
    return math.sqrt(x)
def Power(a,b):
    return a**b
def BesselJ(z,v):
    return scipy.special.jv(z,v)
def Cosh(x):
    return math.cosh(x)
def Cos(x):
    return math.cos(x)
def Csc(x):
    return 1/math.sin(x)
def Cot(x):
    return 1/math.tan(x)
# DEFINE ALPHA
with open("alpha.txt") as alpha:
    alpha1 = alpha.readlines()
    alpha = eval(alpha1[0])
# MCI INTEGRATION
K = [0] * len(fL)
def integrate(i):
    def funcL(x,y):
        # RETURN INTEGRAND AS FUNCTION
        return (eval(fL[i]))
    def funcR(x,y):
        # RETURN INTEGRAND AS FUNCTION
        return (eval(fR[i]))
    #  DEFINE DOMAINS
    def testRegion(pt):
        return (pt[0] > pt[1])
    def genpoint():
        # GENERATE COORDINATES IN A SQUARE
        x = (1 - math.sin(alpha))*random() + math.sin(alpha)
        y = (1 - math.sin(alpha))*random() + math.sin(alpha)
        return (x,y)
   
    # INITIALIZE
    SumL = 0.0
    SumR = 0.0
    Area = (1 - math.sin(alpha))**2
    NL = 0
    NR = 0
    # PARAMETERS OF MCI
    samp_pts = 100
    int_dist = []
    iterations = 100
    # INTEGRATION
    for _ in range(iterations):
        for _ in range(samp_pts):
            pt = genpoint()
            if testRegion(pt):
                SumR += funcR(pt[0],pt[1])
                NR += 1
            else:
                SumL += funcL(pt[0],pt[1])
                NL += 1
        solL = SumL*Area/NL
        solR = SumR*Area/NR
        sol = solL + solR
        int_dist.append(sol)
   
    K[i] = statistics.mean(int_dist)
    return K[i]
def main():
  pool = mp.Pool(mp.cpu_count())
  result = pool.map(integrate, range(len(fL)) )
if __name__ == "__main__":
    main()
    print(K)
I'd like to run the integrate function over a given range, in this case range(len(fL)) = [0,1,2,3,4,5,6,7,8]. Notice I'm trying to store the results of integrate(i) as K. But on my output, K is seemingly a list of zeros. However, if I add print(K) immediately before line 75 I get the correct value. Something seems to be overwriting K, Any ideas?

I've attached the necessary .txt files so you can run it too.

Thanks so much!
 

Attachments

Technology news on Phys.org
joshmccraney said:
Something seems to be overwriting K,
Nothing is overwriting K. The results of the overall computation are being stored in "result", and you're not printing that anywhere. (You would have to either print it in "main" or return it from "main" to print where you are printing now.)

Note also that, because you are using multiprocessing, there is not just one "K". There is one "K" for each subprocess you run, that is separate from the K in the master process. So storing anything in the K in each subprocess doesn't do anything at all to the K in the master process, which is the only K you can print from the master process. That's why printing K from the master process is showing you all zeros: that's what you initialized K to in the master process, and nothing else in the master process changes it. When you print K inside the integrate function you are printing it inside the subprocesses, which is why that shows you the actual results.
 
  • Like
Likes bikashdaga, Baluncore and member 428835
PeterDonis said:
Nothing is overwriting K. The results of the overall computation are being stored in "result", and you're not printing that anywhere. (You would have to either print it in "main" or return it from "main" to print where you are printing now.)

Note also that, because you are using multiprocessing, there is not just one "K". There is one "K" for each subprocess you run, that is separate from the K in the master process. So storing anything in the K in each subprocess doesn't do anything at all to the K in the master process, which is the only K you can print from the master process. That's why printing K from the master process is showing you all zeros: that's what you initialized K to in the master process, and nothing else in the master process changes it. When you print K inside the integrate function you are printing it inside the subprocesses, which is why that shows you the actual results.
Got it, makes sense! Thanks so much, I was beating my head against a wall here.:headbang:
 
joshmccraney said:
I was beating my head against a wall here.:headbang:
You're not the first to have that experience when trying to debug programs that use multiprocessing. :wink: It can be a very useful tool, but it takes a certain mental shift to get in sync with what it's doing, and (unfortunately IMO) the drive to make it so easy to use also makes it harder to make the mental shift, because code that uses it looks just like code that doesn't, so one is led to think that both types of code should work the same. But they don't.
 
  • Like
Likes member 428835
PeterDonis said:
You're not the first to have that experience when trying to debug programs that use multiprocessing. :wink: It can be a very useful tool, but it takes a certain mental shift to get in sync with what it's doing, and (unfortunately IMO) the drive to make it so easy to use also makes it harder to make the mental shift, because code that uses it looks just like code that doesn't, so one is led to think that both types of code should work the same. But they don't.
Good advice for me moving forward, and for others who may read this. I appreciate your insight.
 
try changing line 32 to:

K = [0 for x in range(len(fL))]
 
emacstheviking said:
try changing line 32 to:

K = [0 for x in range(len(fL))]
That will (a) give the same K initialization that's already there, and (b) not fix the problem described in the OP.
 
emacstheviking said:
try changing line 32 to:

K = [0 for x in range(len(fL))]
Thanks for the interest. As @PeterDonis alludes to earlier, the solution ##K## is in the main() function, and stored as result. I don't think any code needs to be changed.
 
Back
Top