Compiled with user input vs scripting language

[geouke]

I am going to use Fortran and Python as examples of a compiled and a scripting language in this question.

If I write a Fortran program and compile it, it is going to process data more quickly than a python script that does the same thing. However, what if I write a Fortran program that allows user-supplied data without re-compiling. For example, 'program_name data.dat' at the console would feed a data file into a Fortran program that would process it and spit out an answer. An even more general example would be if I write a Fortran console program that allows a user to define variables, do special functions, plot easily, etc. In other words, I have created a scripting language with Fortran. These are my questions:

(1) If a data file is read in using 'program_name data.dat' rather than the data being compiled into an executable and ran, have I lost some execution speed other than the time it takes the program to read the data? Why?

(2) It takes program_name.f03 time 't1' to compile and run and it takes 'program_name data.dat' time 't2' to read data and execute, is t1 <. ==. or > t2? Why?

I am in a graduate program where I need to do potentially time-consuming models repetitively with different parameters. The question of whether to use Fortran or Python for small tasks is obvious, but I am wondering what I have gained by using Fortran. More importantly, I would like to know why I have gained.

Thanks! You guys are always great.

 

[rcgldr]

So the question is can you create a scripting language run from a Fortran compiled program that is more effecient that the Python interpreter on the same machine? Seems one of the issues would be the efficiency in parsing scripts in Fortran, which could depend on library functions to assist parsing, possibly written in assembly, but callable from Fortran.

Another alternative would be a scripting language that tokenizes user input as it's entered or editted, similar to how functions are defined in some versions of Basic.

Then there's the issue that even if the end result is more efficient, will it make up for the time you spent developing it.

 

[geouke]

That is part of what I am asking. Maybe it would be more clear if I remove Python and say that I am comparing a compiled Fortran program which has data compiled with it to one that acts on a data file. Let's say my problem involves a large matrix in several dimensions and could take days or weeks to run. Furthermore, it depends on data that is fed into it.

Scenario 1: All of the data is read into the program when it is compiled.
Scenario 2: Only the processes that act on the data are compiled and a data file is read in.

Scenario 2 is basically like a scripting language. Let's say my program opens with a prompt that I type, "set datafile 'data.dat'". It is behaving like MATLAB or Python now.

In scenario 1, time is taken to read in the data during compilation and some set of instructions is carried out on it.
In scenario 2, time is taken AFTER compilation to read in data and the same instructions are carried out on it.

So, factoring in the extra time it takes to compile in scenario 1, is the overall time still less? If so, why?

 

[NemoReally]

In general, the FORTRAN program is going to run faster than an interpreted language equivalent and people have been writing FORTRAN programs to read in data files and process them since it first appeared - I don't remember ever writing a 'serious' program that had the data compiled with it. I suspect that the compile and link time in a modern system is going to be in the order of seconds and, if you intend to run the program many times without further modification, then you can probably ignore it once you've got it working. It may be slower to develop a working program in FORTRAN because of the compile-link overhead but the end performance gains may be worth (particularly if you have a program that will take a long time to run).

The reason it's quicker in general is that the compiler and linked libraries produce native executables for the machine you are running the program on and the code is not re-evaluated every time you run it, unlike a standard interpreter (although some interpreters effectively compile on the fly, look up Just In Time interpreters)

 

[geouke]

So, let me make sure this statement is correct:

Let's say I wrote a program in Fortran that can process the equivalent of a Python or MATLAB script and distributed as .f source code, it would run more quickly than MATLAB or Python (assuming that same math underlied the routines being done) because it was compiled on each system rather than being installed from an executable? If that is the case (that is, assuming I understand correctly), why is there no open source equivalent to what I am talking about?

 

[geouke]

does it take longer for a MATLAB script to execute because the computer has to convert the script into a language that it understands? Also, does that take longer than actually executing the code in some cases?

x = ones(1000,1000,1000)
x = rand(x)
x = erf(x)

Now, let's say I have a Fortran program that does the same thing. Is the speed terribly different even though MATLAB does not have much to interpret? Is the interpreting process more involved than I am imagining?

 

[rcgldr]

If the code is fixed (although it may have conditionally executed sections) and only the data changes, then it would be faster to compile the code one time, and use it on many sets of data files, as oppposed to including the data sets, as header files, and recompiling for each dataset.

If the data set includes crude script like features, sort of like a programmable calculator, then the issue would be the significance of the parsing / scripting overhead versus the actual operations to be performed.

Another alternative would be the equivalent of "compiling" the scripting part of the datasets into some form of tokens that would reduce the scripting overhead from the Fortran generated program.

 

[NemoReally]

does it take longer for a MATLAB script to execute because the computer has to convert the script into a language that it understands? Also, does that take longer than actually executing the code in some cases?

x = ones(1000,1000,1000)
x = rand(x)
x = erf(x)

Now, let's say I have a Fortran program that does the same thing. Is the speed terribly different even though MATLAB does not have much to interpret? Is the interpreting process more involved than I am imagining?

I'm not a Matlab user, but as far as I'm aware, Matlab is specialized for matrix and calls C or FORTRAN libraries to handle many of the built-in array operations. It is possible that the execution time for such a simple program will be comparable to a compiled program, as the interpreter does indeed have very little to do. J (derived from APL) is another interpreted array language and that is claimed to have very good performance for such problems.

You'll need the 64-bit version of Matlab running on 64-bit Windows or Linux to handle that size array.