How can I efficiently vary and run a program with multiple input parameters?

[mar2194]

Hey everyone,

I'm currently running a program which accepts an input file. The input file has various parameters, 3 of which I need to change, run again, etc.
The range of those 3 parameters are 15 values for one, 12 values for another and 10 for another, so I have 15*12*10 (=1800) possible input files I need to make for the program. The output filename will need to match the parameters for that run as well. Obviously creating 3375 input files by hand would be impractical, not only because it would be time consuming but the risk of human error would be great. I need help with the following:

1. Varying the parameters of my input file. I know this can be done with AWK, or perhaps using shell script. The input file is plain text (ascii)

2. recursively running the script each time a parameter is changed. Suppose my input file looks like:

#input file mockup
#Value for A
1
#Value for B
2
#Value for C
4
#Value for D
12

Suppose the three parameters I want to change are A, B and D. Suppose each value of A is spaced 5 units apart, and I want to run over 3 values. My possible values of A would then be {1, 6, 11}
My possible values for B are spaced 3 units apart and there are 4 values {2, 5, 8, 11}
and my Values for D are spaced 1 apart and there are two of them {12, 13}
Allowing for repeated values, the sets I would want process are:

{1, 2, 12}
{1, 5, 12}
{1, 8, 12}
{1, 11, 12}

{1, 2, 13}
{1, 5, 13}
{1, 8, 13}
{1, 11, 13}

{6, 2, 12}
{6, 5, 12}
{6, 8, 12}
{6, 11, 12}

{6, 2, 13}
{6, 5, 13}
{6, 8, 13}
{6, 11, 13}

{11, 2, 12}
{11, 5, 12}
{11, 8, 12}
{11, 11, 12}

{11, 2, 13}
{11, 5, 13}
{11, 8, 13}
{11, 11, 13}

where each place in the brackets corresponds to the values for {A, B, D}

So my two problems are essentially:
1 Make a wrapper to run the program recursively with the changed input file
2 Change the three variables so I get every combination and pass the corresponding input file to the program.

here's what I think it should look like in pseudo-code (I know this is wrong need some help):

A = 1
B = 2
D = 12

for B<11,
for A<11,
for D<13,
B+3, A+5, D++, program input file

#input file mockup
#Value for A
Some Expression for {A}
#Value for B
Some Expression for {B}
#Value for C
4
#Value for D
Some Expression for {D}

outputdirectory: mkdir Run$A$$B$$D$

or something like this...


NOTE: I've posted this question on a different site to see if anyone else could answer it there. I am not trolling or crowd-sourcing; looking for genuine help.
Also, if anyone's interested, I'm running a co-seismic displacement model for which I need to vary the depth, length and displacement each run through. I'm assuming linear slip and uniform viscosity throughout the medium. The program is fine and ready to go I just need to figure out a way to alter my input file with variable parameters.

 

[mar2194]

Actually I realized (I think...) that my logic loop should look like:
A=1
B=2
D=12
While A<11
While B<11
While D<13
D++
B+3
A+5
??

 

[billiards]

Hi,

you don't need to use logic.

you can simply list the numbers you want to loop over.

for example, in BASH (a shell) you might have something like

for A in 1 2 3; do
for B in 6,8,10; do
for C in 4,7,11; do

# write the input file to your binary exec
echo $A $B $C > file

# run the exec
./exec

done
done
done

 

[mar2194]

Hi,

you don't need to use logic.

you can simply list the numbers you want to loop over.

for example, in BASH (a shell) you might have something like

for A in 1 2 3; do
for B in 6,8,10; do
for C in 4,7,11; do

# write the input file to your binary exec
echo $A $B $C > file

# run the exec
./exec

done
done
done

What do you mean write the input file to my binary executable? Where do I write the for loops?

 

[billiards]

You write the for loops in a shell script. The example I gave was in BASH. If you want to write a shell script in BASH I recommend putting the line:

#!/bin/bash

as the very first line in your script. That way whenever you run the script it will automatically know to use the bash shell.

The input file needs to change before each call to your exec. SO you change the input file inside the loop before calling the exec. Exactly what form your input file needs to take only you know. But you can use the echo command to write lines and the > symbol to write that line to a file. If you need to add more lines you can use the >> syntax to add to a preexisting file.

 

[jim mcnamara]

There is a lot of confusion here. Please show us EXACTLY what one "run" with a single set of parameters look like. This would be the awful case of you typing the command by hand:

We think is it like:

some_program_name 11 3 9

where some_program_name is some kind of compiled program.
And 11 3 9 are parameters you feed the program. ( I made up the numbers so that does not confuse you)

Please show us if you want useful help.

 

[mar2194]

Here is the input file (run1.input):

# grid size (sx1, sx2, sx3)
256 256 256
# sampling size, smoothing & nyquist (dx1,dx2,dx3,beta,nq)
0.5 0.5 0.5 0.5 2
# origin position & rotation
0 0 0
# geographic origin (longitude and latitude), UTM zone and real length unit (usually m or km)
# displacements and stress are converted to lon/lat geographic coordinates
# unit corresponds to a scaling from dx1,dx2,dx3 to real unit
# use unit = 1   if dimensions are described in units of m
# use unit = 1e3 if dimensions are described in units of km
120 22 51 1e3
# observation depth (displacements and stress) (stress is only exported in GRD)
0 5
# output directory (all output files are stored in the following directory)
output1
# the elastic parameters (Lambda and mu) are the Lame parameter and the shear modulus
# gamma=(1-nu)*rho*g/mu is a wavelength relevant to buoyancy.
# elastic parameters and gamma = (1-nu) rho g / mu = 8.33e-7 /m = 8.33e-4 /km
4.26E+04 2.6E+04 8.48e-4
# integration time, time step and scaling 
0 -1 1
# number of viscous observation slice 
0 
# number of observation points 
0 
# number of Coulomb patches
0
# number of prestress interfaces 
0 
# number of linear viscous interfaces 
0 
# number of power-law viscous interfaces 
0 
# number of friction faults 
0
# number of interseismic loading strike-slip and opening
0
0
# number of coseismic events (when slip distribution is prescribed) 
1
# number of shear dislocations (strike-slip and dip-slip faults) 
1
# no   slip xs ys zs length  width  strike dip rake
  1	2   0  0  40   20    10	    0      60	0
# number of tensile cracks 
0 
# number of dilatation sources 
0 
# number of surface traction 
0

Here is the shell-script I currently use to run over that input file using the program(run1.sh):

#!/bin/sh
WDIR=./output1

if [ ! -e $WDIR ]; then
	echo adding directory $WDIR
	mkdir $WDIR
fi

OMP_NUM_THREADS=4 relax $* < run1.input | tee $WDIR/in.param

As you can see, it copies the output of the program to an in.param file in the directory output1 after running the program relax on 4 threads using run1.input

I need to change 3 parameters from my input file and create a new directory for every run of new variable combination.

 

[justsomeguy]

#!/bin/sh

WDIR=./output1

if [ ! -d $WDIR ]; then
  if [ -e $WDIR ]; then
    echo ${WDIR} exists and is not a directory.
    exit
  fi
	echo adding directory $WDIR
	mkdir $WDIR
fi

for I in 1 6 11
do
  for J in 12 13
  do
    for K in 2 5 8 11
    do
#      OMP_NUM_THREADS=4 relax ${I} ${K} ${J} < run1.input | tee $WDIR/in.param
      echo ${I} ${K} ${J}
    done
  done
done

Will do what you want with your example. I kept the actual relax bit commented out and just have it echoing instead. If you want to change the loop arguments, it should be obvious where to look.

Also changed the -e to a -d, so it will exit out if 'output1' exists but is not a directory.

 

[jhae2.718]

I would not use shell scripts or awk for what I understand you are trying to do.

 

[billiards]

Looks like your code is always going to write to a directory called output1. You could change that in the input file. Or more simply, you can rename the folder output1 to a new unique name after each call to the exec.

e.g.

start loop

# some code

mv outfile1 unique_name

end loopwhere the command mv stands for move, and is effectively a rename.
note: it might be wise to name the directories by the actual parameters they contain, so if you have a run with parameters 1,6,13, you could name the folder "run_1_16_13", rather than something obscure like "run_54". Just a suggestion.