Running a for loop in parallel via multiprocessing

[joshmccraney]

Hi PF!

I'm wanting to run a function in parallel, which I've denoted integrate below, on line 33.

from math import *
from random import *
import math, scipy.special
import statistics
import multiprocessing as mp
# READ MATHEAMTICA FUNCTIONS
with open("funcL.txt") as fileL:
    fL = fileL.readlines()
with open("funcR.txt") as fileR:
    fR = fileR.readlines()
# DEFINE MATH REQUIRED
Pi = math.pi
def Sqrt(x):
    return math.sqrt(x)
def Power(a,b):
    return a**b
def BesselJ(z,v):
    return scipy.special.jv(z,v)
def Cosh(x):
    return math.cosh(x)
def Cos(x):
    return math.cos(x)
def Csc(x):
    return 1/math.sin(x)
def Cot(x):
    return 1/math.tan(x)
# DEFINE ALPHA
with open("alpha.txt") as alpha:
    alpha1 = alpha.readlines()
    alpha = eval(alpha1[0])
# MCI INTEGRATION
K = [0] * len(fL)
def integrate(i):
    def funcL(x,y):
        # RETURN INTEGRAND AS FUNCTION
        return (eval(fL[i]))
    def funcR(x,y):
        # RETURN INTEGRAND AS FUNCTION
        return (eval(fR[i]))
    #  DEFINE DOMAINS
    def testRegion(pt):
        return (pt[0] > pt[1])
    def genpoint():
        # GENERATE COORDINATES IN A SQUARE
        x = (1 - math.sin(alpha))*random() + math.sin(alpha)
        y = (1 - math.sin(alpha))*random() + math.sin(alpha)
        return (x,y)
   
    # INITIALIZE
    SumL = 0.0
    SumR = 0.0
    Area = (1 - math.sin(alpha))**2
    NL = 0
    NR = 0
    # PARAMETERS OF MCI
    samp_pts = 100
    int_dist = []
    iterations = 100
    # INTEGRATION
    for _ in range(iterations):
        for _ in range(samp_pts):
            pt = genpoint()
            if testRegion(pt):
                SumR += funcR(pt[0],pt[1])
                NR += 1
            else:
                SumL += funcL(pt[0],pt[1])
                NL += 1
        solL = SumL*Area/NL
        solR = SumR*Area/NR
        sol = solL + solR
        int_dist.append(sol)
   
    K[i] = statistics.mean(int_dist)
    return K[i]
def main():
  pool = mp.Pool(mp.cpu_count())
  result = pool.map(integrate, range(len(fL)) )
if __name__ == "__main__":
    main()
    print(K)

I'd like to run the integrate function over a given range, in this case range(len(fL)) = [0,1,2,3,4,5,6,7,8]. Notice I'm trying to store the results of integrate(i) as K. But on my output, K is seemingly a list of zeros. However, if I add print(K) immediately before line 75 I get the correct value. Something seems to be overwriting K, Any ideas?

I've attached the necessary .txt files so you can run it too.

Thanks so much!

 

[PeterDonis]

Something seems to be overwriting K,

Nothing is overwriting K. The results of the overall computation are being stored in "result", and you're not printing that anywhere. (You would have to either print it in "main" or return it from "main" to print where you are printing now.)

Note also that, because you are using multiprocessing, there is not just one "K". There is one "K" for each subprocess you run, that is separate from the K in the master process. So storing anything in the K in each subprocess doesn't do anything at all to the K in the master process, which is the only K you can print from the master process. That's why printing K from the master process is showing you all zeros: that's what you initialized K to in the master process, and nothing else in the master process changes it. When you print K inside the integrate function you are printing it inside the subprocesses, which is why that shows you the actual results.

 

[joshmccraney]

Nothing is overwriting K. The results of the overall computation are being stored in "result", and you're not printing that anywhere. (You would have to either print it in "main" or return it from "main" to print where you are printing now.)

Note also that, because you are using multiprocessing, there is not just one "K". There is one "K" for each subprocess you run, that is separate from the K in the master process. So storing anything in the K in each subprocess doesn't do anything at all to the K in the master process, which is the only K you can print from the master process. That's why printing K from the master process is showing you all zeros: that's what you initialized K to in the master process, and nothing else in the master process changes it. When you print K inside the integrate function you are printing it inside the subprocesses, which is why that shows you the actual results.

Got it, makes sense! Thanks so much, I was beating my head against a wall here.

 

[PeterDonis]

I was beating my head against a wall here.

You're not the first to have that experience when trying to debug programs that use multiprocessing. It can be a very useful tool, but it takes a certain mental shift to get in sync with what it's doing, and (unfortunately IMO) the drive to make it so easy to use also makes it harder to make the mental shift, because code that uses it looks just like code that doesn't, so one is led to think that both types of code should work the same. But they don't.

 

[joshmccraney]

You're not the first to have that experience when trying to debug programs that use multiprocessing. It can be a very useful tool, but it takes a certain mental shift to get in sync with what it's doing, and (unfortunately IMO) the drive to make it so easy to use also makes it harder to make the mental shift, because code that uses it looks just like code that doesn't, so one is led to think that both types of code should work the same. But they don't.

Good advice for me moving forward, and for others who may read this. I appreciate your insight.

 

[PeterDonis]

The programming guidelines in the Python documentation for the multiprocessing module are worth a look:

https://docs.python.org/3/library/multiprocessing.html#multiprocessing-programming

 

[emacstheviking]

try changing line 32 to:

K = [0 for x in range(len(fL))]

 

[PeterDonis]

try changing line 32 to:

K = [0 for x in range(len(fL))]

That will (a) give the same K initialization that's already there, and (b) not fix the problem described in the OP.

 

[joshmccraney]

try changing line 32 to:

K = [0 for x in range(len(fL))]

Thanks for the interest. As @PeterDonis alludes to earlier, the solution $K$ is in the main() function, and stored as result. I don't think any code needs to be changed.