Call program written in language B from program written in language A

[CGandC]

Let's say I have an algorithm $ALG_A$ written in language $A$, and I wish to call algorithm $ALG_B$ written in language $B$ through $ALG_A$. What are some ways of achieving this in implementation?

For example, say I want to calculate stuff in C language and provide this calculated data as an input to an algorithm written in Matlab or Python, which then receive the data and plot it - how is this invocation from one language to another is achievable? perhaps some script needs to be written that is called from C? ( which receives that data, and invokes the algorithm in Matlab/Python ).

 

[CGandC]

One option could be to write the program in C so that it writes to an output file, and then I can go into Matlab/Python, read that file and produce the graph - but this approach is too exhausting ( since I have to alternate between the languages on each run ), I prefer to run the code once in C and immediately have the graph results shown to me.

 

[PeterDonis]

how is this invocation from one language to another is achievable?

It depends on the language. For Python, there are several different options:

(1) Use the ctypes module to load a C shared library and call functions and manipulate structures from it.

(2) Use the Python/C API to write an extension module that wraps the C code in such a way that the module can be imported in Python and used like any other importable module.

(3) Use one of a number of third-party tools to automatically generate Python wrappers for the C code.

 

[PeterDonis]

One option could be to write the program in C so that it writes to an output file

You could also write it so that it outputs its results to standard output, and then you can just pipe the standard output to a program written in another language. That can be done either from a shell script, or in Python you can use the subprocess module to call a program and read its standard output.

 

[Wrichik Basu]

One option could be to write the program in C so that it writes to an output file, and then I can go into Matlab/Python, read that file and produce the graph - but this approach is too exhausting ( since I have to alternate between the languages on each run ), I prefer to run the code once in C and immediately have the graph results shown to me.

You can write a bash script that will first execute the C program, and then the Python program (which, in turn, utilizes output from the C program).

Edit: @PeterDonis already suggested that. I need to increase my typing speed.

 

[CGandC]

It depends on the language. For Python, there are several different options:

(1) Use the ctypes module to load a C shared library and call functions and manipulate structures from it.

(2) Use the Python/C API to write an extension module that wraps the C code in such a way that the module can be imported in Python and used like any other importable module.

(3) Use one of a number of third-party tools to automatically generate Python wrappers for the C code.

Regarding method (2), what is an extension module? and generally how does one write a C code wrapper for Python?

You can write a bash script that will first execute the C program, and then the Python program (which, in turn, utilizes output from the C program).

Edit: @PeterDonis already suggested that. I need to increase my typing speed.

Is it possible to write such script in any scripting language? ( such as LISP ). And why can't a C program do this? ( first executing its code and then calling the Python program )

 

[PeterDonis]

what is an extension module? and generally how does one write a C code wrapper for Python?

The Python documentation gives plenty of detail:

https://docs.python.org/3/extending/extending.html

https://docs.python.org/3/c-api/index.html

 

[PeterDonis]

Is it possible to write such script in any scripting language? ( such as LISP ). And why can't a C program do this? ( first executing its code and then calling the Python program )

How much programming experience have you had? I ask because these are the sorts of questions whose answers one normally finds by trying to do them, not by asking other people.

What any programming language can do depends on the language. The C language does have facilities for calling other processes (basically using thin wrappers in the C library around the corresponding system calls), so that would be an option as well. Generally speaking, though, it's going to be much more cumbersome to have a C program be the "master" program that orchestrates everything, while only particular things get done in a higher level language like Python, than to have things the other way around: the higher level language has the "master" program that orchestrates everything, and a language like C is only used for particular things (usually algorithms that require more speed than the higher level language can provide).

 

[FactChecker]

There are several ways to pass information to and from the called program.
Most often the called program is compiled into a library and the calling program link must include that library and follow the exact method that the called program uses to get and return information. The fact that the languages were different does not matter other than that.
Alternatively, as several posts have said, you can read and write information in a standard text format and send it to and from the called program in a file or in STDIN/STDOUT.

 

[CGandC]

Thanks, I know some languages at different levels of expertise ( I'm not a beginner ) and I was curious as to what other people might shed the subject and how they might invocate programs between different languages.
From what I've searched on google, each language is a different case in its own right ( for example, C, Python, Java,... all have different ways of calling .exe files ), the case is similar to writing wrappers for different languages to different languages ( for example, it turns out that in order to write a Java wrapper for C++ you can use Java Native Interface ).

 

[pbuk]

Say I want to calculate stuff in C language and provide this calculated data as an input to an algorithm written in Matlab or Python.

The easiest way to do this in Python is to call your C program from Python and capture the output.

import os
import csv
import matplotlib.pyplot as plt

# Capture comma-separated data (one point per line) written to stdout by my_executable
outputStream = os.popen("my_executable")
reader = csv.reader(outputStream, quoting=csv.QUOTE_NONNUMERIC)

xValues = []
yValues = []
for line in reader:
    xValues.append(line[0])
    yValues.append(line[1])

fig, ax = plt.subplots()
ax.plot(xValues, yValues)

Let's say I have an algorithm $ALG_A$ written in language $A$, and I wish to call algorithm $ALG_B$ written in language $B$ through $ALG_A$. What are some ways of achieving this in implementation?

This is a completely different question to which you've been given some good answers, and I'll add
Consider implementing $ALG_B$ in $A$.
Consider implementing $ALG_A$ in $B$.
Consider using a third language to control the whole process.

 

[FactChecker]

If the programs are not linked together as an upper-level program calling a subprogram, then you need the main program to invoke the operating system to run the sub-program. Each language has its own method for invoking the operating system to run an executable secondary program. Also, there are scripting languages that are very well designed to use the operating system to call both programs in order. My favorite is Perl, but there is also Python, Windows DOS (whatever it is called now), and Unix/Linix shell scripts.

 

[whartung]

Calling foreign code can be a broad subject.

Typically, when folks talk about calling foreign code it's almost always from the X language to the C language, as C is the ubiquitous systems programming language across most platforms. So, a big draw for X->C is to call system runtimes, such as Unix file I/O or Windows GUI functions, most all of which are published as C libraries.

Since many languages are themselves written in C, it makes even more sense to be able to invoke C routines from the language runtime.

What you typically won't find is something like a Lua -> Haskell, or Python -> Ruby.

When a language calls a C function, that function needs to be linked into the running image. This linking is typically done through a dynamic library (DLL in Windows, Shared Library in Unix). Dynamic modules have the advantage of being, well, dynamic! Meaning the language runtime doesn't necessarily need specific information about library when the runtime is originally created. It can do everything later, after it's built.

But this is why you see many times some Python library needing not just Python, but some other project. This is so that you can be sure that the shared library from the project is available when the Python code goes to look for it.

For languages other than C, the mechanics are typically different. If you want to call Python from Ruby, you're probably better off firing off a Python sub process, and then talking to it via a pipe or a socket. But that can be quite expensive.

Other things you can do is you can embed one language into another. For example, you can embed Python in C, that is the C program can invoke Python code. Well, since you can invoke Python from C, and can call C from Ruby, well then Ruby -> C -> Python. The C code acts as a bridge.

There are a lot of reasons to not do this, it's expensive, it can be problematic, having two conflicting high level language runtimes fighting each other. But, it's possible.

Other compiled languages, like Pascal or Fortran, can more easily call or be called from C. This is because their compiled machine language routines tend to have little overhead. But there are other problems that need to be addressed.

For example Pascal and C historically have different calling conventions. At a minimum, they pass arguments to stack in a different order. But also, things like strings are represented differently, and who knows how other internal things are represented. All manageable, and many modern implementations of cognizant of living in a mixed language environment. In Pascal, you can add an option that a particular routine is a C routine so the compiler can juggle the parameters properly. Similarly, some C compilers can do the same thing with Pascal. This was much more prevalent in the past when Pascal was more popular. The original Macintosh libraries were all written to work with Pascal, so Mac C compilers had this capability.

 

[CGandC]

Calling foreign code can be a broad subject.

Typically, when folks talk about calling foreign code it's almost always from the X language to the C language, as C is the ubiquitous systems programming language across most platforms. So, a big draw for X->C is to call system runtimes, such as Unix file I/O or Windows GUI functions, most all of which are published as C libraries.

Since many languages are themselves written in C, it makes even more sense to be able to invoke C routines from the language runtime.

What you typically won't find is something like a Lua -> Haskell, or Python -> Ruby.

When a language calls a C function, that function needs to be linked into the running image. This linking is typically done through a dynamic library (DLL in Windows, Shared Library in Unix). Dynamic modules have the advantage of being, well, dynamic! Meaning the language runtime doesn't necessarily need specific information about library when the runtime is originally created. It can do everything later, after it's built.

But this is why you see many times some Python library needing not just Python, but some other project. This is so that you can be sure that the shared library from the project is available when the Python code goes to look for it.

For languages other than C, the mechanics are typically different. If you want to call Python from Ruby, you're probably better off firing off a Python sub process, and then talking to it via a pipe or a socket. But that can be quite expensive.

Other things you can do is you can embed one language into another. For example, you can embed Python in C, that is the C program can invoke Python code. Well, since you can invoke Python from C, and can call C from Ruby, well then Ruby -> C -> Python. The C code acts as a bridge.

There are a lot of reasons to not do this, it's expensive, it can be problematic, having two conflicting high level language runtimes fighting each other. But, it's possible.

Other compiled languages, like Pascal or Fortran, can more easily call or be called from C. This is because their compiled machine language routines tend to have little overhead. But there are other problems that need to be addressed.

For example Pascal and C historically have different calling conventions. At a minimum, they pass arguments to stack in a different order. But also, things like strings are represented differently, and who knows how other internal things are represented. All manageable, and many modern implementations of cognizant of living in a mixed language environment. In Pascal, you can add an option that a particular routine is a C routine so the compiler can juggle the parameters properly. Similarly, some C compilers can do the same thing with Pascal. This was much more prevalent in the past when Pascal was more popular. The original Macintosh libraries were all written to work with Pascal, so Mac C compilers had this capability.

Thanks a lot for the detailed answer!

 

[Mark44]

Let's say I have an algorithm $ALG_A$ written in language $A$, and I wish to call algorithm $ALG_B$ written in language $B$ through $ALG_A$. What are some ways of achieving this in implementation?

One possibility that isn't mentioned in this thread is implementing a function or set of functions in assembly language, and calling them from a program written in C or C++. This is something I've done many times over the past many years. What has to happen is that both parts of the overall program have to be in sync on how parameters to the assembly functions will be passed, and how return values, if any, will be returned.

To build such a program, the C/C++ portion has to be compiled, and the assembly portion has to be assembled. The resulting object code is then joined by a linker or link editor to produce an executable program.

Something similar occurs when, say, a Fortran main program calls routines written in C, where the C code is compiled into library code. When the Fortran program is built, the Fortran code is compiled and then linked to the library code. Again, there has to be agreement between the caller (Fortran code) and the library code as to how parameters are passed and how return values are returned to the caller.

 

[CGandC]

To build such a program, the C/C++ portion has to be compiled, and the assembly portion has to be assembled. The resulting object code is then joined by a linker or link editor to produce an executable program.

Are you able to call steps of the C compilation process individually?

What I mean is,
I know that C compilation process is made up of 4 parts:
1. Preprocessing - which puts the header declarations into my .c program and creates .i file that contains the original code + header declarations.
2. Compiling - which creates from my .i file a .s file which contains the C code in assembly language specific to my computer.
3. Assembler - takes the .s file and converts it into a .o file containing machine level instructions.
4. Linker - handles the merging of the function declarations in the header files from their actual implementation in some external .o file to the currently created .o file. After this merging is done an .exe file is created, ready to be executed.

Is there a possibility to do only steps (1) + (2) in the above parts ( without steps 3 + 4 ) ? I'm not aware of executing different steps independently since this compilation process is done as a whole.

 

[Mark44]

It's possible to do what you're calling steps 1 and 2 only for separate files that make up the project., and then use the linker to compile the various chunks of object code.

Your step 3, assembly, is not always present. The compiler converts your C source code to object code (machine code). Some compilers can present this object code in the form of assembly language, but as far as I know, this is something that doesn't always occur.

If your source code uses library routines for I/O, operating system functionality, or whatever, the compiler just inserts placeholders for these calls. The linker/link editor fixes up these placeholders, substituting in the locations of the library that is being called.

In addition, if your goal is to produce a library of functions (either a static library or a dynamic link library (DLL) rather than a standalone executable), your step 4 is not used.

 

[rcgldr]

A historical and current legacy example is Cobol programs calling assembly functions on an IBM mainframe. This was originally done because the database access functions were implemented as macros in IBM assembly, which Cobol programs would call. This in turn led to having some optimized functions being written in IBM assembly. This legacy combination of Cobol and assembly exists to this day due to no one having the time or wanting to risk re-writing large amounts of working legacy code.

 

[whartung]

A historical and current legacy example is Cobol programs calling assembly functions on an IBM mainframe.

But this should be kept in the context that a LOT of code on the IBM was written in assembly. BAL was quite commonplace back in the day, so it was a more natural mechanic of extension. It was also done for specific use case (nominally for performance).

Today, the world is quite different. Comparatively little assembly is written, deferring to C. The desire to run code across languages is much more feature oriented than specifically performance related. "I want to use the tested, vetted, and proven crypto tech in OpenSSL in my Haskel program", not necessarily for performance, but simply that crypto is Hard, and best to use proven libraries.

C is much more the lower common denominator implementation choice because of its lightweight runtime needs. Calling C++ from other languages is not as popular, because C++ has a heavier footprint. Thus many common utility libraries (though certainly not all), such as OpenSSL, choose C in order to be more broadly portable and applicable across environments. Some folks refer to C as "portable assembly language" (which isn't really accurate, C semantics are much different than assembly), but it is the lingua franca of common, portable utility functionality.

 

[Melbourne Guy]

Let's say I have an algorithm ALGA written in language A, and I wish to call algorithm ALGB written in language B through ALGA.

I assume your algos are all local to the PC, @CGandC, but you can also execute code over the network, and this venerable primer for Linux describes that at the high level. Many++ years ago, a colleague and I used RPC to prank each other by doing silly things on the other's Sun #/60 workstations, typically by messing with the display frame buffer.

https://www.linuxjournal.com/article/2204

 

[rcgldr]

little assembly is written, deferring to C. The desire to run code across languages is much more feature oriented than specifically performance related. "I want to use the tested, vetted, and proven crypto tech in OpenSSL in my Haskel program", not necessarily for performance, but simply that crypto is Hard, and best to use proven libraries.

Some of the libraries that involve finite field math include assembly code for carryless multiply, such as PCLMULQDQ on X86 processors. Erasure code, including cloud storage, also uses carryless multiply and|or parallel table lookup such as PSHUFB on X86, if it doesn't infringe on some patent that some group managed to get passed in spite of if being obvious.

 

[Jarvis323]

Let's say I have an algorithm $ALG_A$ written in language $A$, and I wish to call algorithm $ALG_B$ written in language $B$ through $ALG_A$. What are some ways of achieving this in implementation?

The general term for this is language binding.

In programming and software design, binding is an application programming interface (API) that provides glue code specifically made to allow a programming language to use a foreign library or operating system service (one that is not native to that language).

https://en.wikipedia.org/wiki/Language_binding

 

[harborsparrow]

In my opinion, a question like this cannot be generalized and depends on the exact languages and other details. If there exists a formal API, its documentation should tell you everything you need. If not, you have to personally be aware of a wide variety of issues on how the two languages are specified, implemented and behave. An example of how things could go wrong is whether arrays are stored row-wise or column-wise. Without a formal, supported API, it probably isn't a good idea to do it in terms of maintainability.