What is the best way to add new calculations to a running list in Linux?

  • Thread starter Thread starter brydustin
  • Start date Start date
  • Tags Tags
    Calculations Linux
Click For Summary

Discussion Overview

The discussion centers around managing and scheduling multiple background calculations in a Linux environment. Participants explore methods for appending new jobs to a running list of calculations and executing them sequentially without manual intervention.

Discussion Character

  • Technical explanation
  • Exploratory
  • Debate/contested

Main Points Raised

  • One participant inquires about the possibility of appending new jobs to a currently running list of calculations or scheduling them to run next without manual input.
  • Another participant asserts that it is not possible to insert new lines into a running shell process and suggests using a combination of commands to manage job execution.
  • Alternative methods for job scheduling are proposed, including the use of job schedulers that can handle load balancing on a single computer.
  • Clarification is sought regarding the behavior of the "wait" command, with a participant expressing concern about ensuring that calculations are not interrupted.
  • A later reply confirms that "wait" does indeed make the new job wait for the completion of the current job, but emphasizes that it must be executed in the same shell session.
  • Further suggestions are made for using a loop to monitor the process ID of the first job and execute the second job once the first completes, with options for running everything in the background.

Areas of Agreement / Disagreement

Participants generally agree that direct insertion into a running process is not feasible. However, there are multiple competing views on the best methods for scheduling and managing the execution of jobs, and the discussion remains unresolved regarding the optimal approach.

Contextual Notes

Limitations include the need for specific shell behavior and the dependency on process IDs, which may vary based on the user's environment and the commands used.

brydustin
Messages
201
Reaction score
0
hi,
I'm running some calculations in the background of my Linux work computer, and I now have some more jobs for it to do, but the current calculations are not yet finished. For example, my new file, RUN.in has the following:
./cal1
./cal2
./cal3
./etc

and this is similar to what I have running right now in my first list, except that I want to "append the list".
Can I just go into the first List of calculations and literally add these new jobs, whilst the current ones are running (by adding the list of RUN.in to the end of the first list which looks like RUN.in)... or is it possible to run the new list of calculations under the list "RUN.in" but not do it manually after the first is finished (i.e. give the computer instructions to do it next even if I'm not present)
... or is it possible to do either (i.e. append to list OR tell it to do the RUN.in list next)?
Help much appreciate thanx
 
Computer science news on Phys.org
You can't insert the lines AFTER the shell is already running.

What about

user@host:/tmp$ RUN1.in &
[1] 1234
user@host:/tmp$ wait %1 ; RUN2.in &

Or maybe:

user@host:/tmp$ ( RUN1.in ; RUN2.in) &
[1] 2768
 
If you want to do this sort of thing regularly, look at
http://en.wikipedia.org/wiki/Job_scheduler#Batch_queuing_for_HPC_clusters

Not all those apps are free software (and the ones I have used personally are not free) but it I bet somebody in the Linux community has written a freeware solution.

Don't be put off by the "high performance computing clusters" description. They are also useful for load-balancing on a single computer, which is what you are trying to do.
 
Rogerio said:
You can't insert the lines AFTER the shell is already running.

What about

user@host:/tmp$ RUN1.in &
[1] 1234
user@host:/tmp$ wait %1 ; RUN2.in &

Or maybe:

user@host:/tmp$ ( RUN1.in ; RUN2.in) &
[1] 2768

Specifically what will
user@host:/tmp$ wait %1 ; RUN2.in &
do?? I just want to be very careful not to make a mistake (i.e. the calculations are going now, and don't want to waste any time). Does "wait" mean that the new job is "waiting" for the first (current) one to finish? If so... that's exactly what I want.
 
brydustin said:
Specifically what will
user@host:/tmp$ wait %1 ; RUN2.in &
do?? I just want to be very careful not to make a mistake (i.e. the calculations are going now, and don't want to waste any time). Does "wait" mean that the new job is "waiting" for the first (current) one to finish? If so... that's exactly what I want.

Yes, it does. But... the "wait" has to be in foreground (or it won't know who is the "%1" job) at the SAME shell session where the "Job %1" is running.

Being so, maybe you prefer to use:
user@host:/tmp$ RUN1.in &
[1] 1234

And then, at ANY shell session (or any terminal):
user@host:/tmp$ (while kill -0 1234; do sleep 60; done); RUN2.in &

The "while" keeps testing the existence of the process 1234 (the first job PID) every 60 seconds, and, when it is gone, the "RUN2.in" command will be called.
The while will run on foreground, and the RUN2 at background.

If you prefer everything at background:
user@host:/tmp$ ((while kill -0 1234; do sleep 60; done); RUN2.in) &
 

Similar threads

  • · Replies 12 ·
Replies
12
Views
5K
Replies
0
Views
1K
  • · Replies 24 ·
Replies
24
Views
11K
Replies
10
Views
5K
  • · Replies 12 ·
Replies
12
Views
6K
  • · Replies 4 ·
Replies
4
Views
3K
  • · Replies 4 ·
Replies
4
Views
7K
Replies
9
Views
3K
  • · Replies 65 ·
3
Replies
65
Views
9K
  • · Replies 2 ·
Replies
2
Views
4K