Programming style in R - to array-process or not?

  • Thread starter andrewkirk
  • Start date
  • #1
andrewkirk
Science Advisor
Homework Helper
Insights Author
Gold Member
4,035
1,608
TL;DR Summary
Can array-processing capability sometimes be more of a burden than a boon? Do you feel moral pressure to avoid loops when using an array-processing-capable language? Do you give in to the pressure?
R enables processing arrays by single statements rather than via loops, by having almost all commands able to apply to arrays as well as to scalars. I seem to recall, when I first learned it (a long while ago now) feeling that the intro texts, and user comments online, encouraged the practice of avoiding loops where possible.

I find the ability to process an array without looping very powerful, and it is dead-easy when one wants to do something like add two arrays componentwise, so we just write c <- a + b rather than having to write a nest of loops with one nesting level for each dimension of the arrays. But it becomes difficult when one has to perform operations that are more abstract, and may have several stages. I have become good at this over the years, and enjoy the elegance of a piece of code that does something really complex and multi-dimensional in a handful of lines. But I am often nagged by the thought that I could still do it much faster if I just wrote out the loops and performed the operations componentwise. Some purist (puritanical?) streak in me usually stops me from doing that, even when I feel sure I could write the code for that in two minutes, whereas it takes me half an hour, and much searching to find specialist helper functions, to do it without loops.

I have been learning Python recently and get the feeling that, while Python can do some array processing, there is no philosophy of avoiding loops. I find that rather liberating.

I'd love to hear the opinions of other R users as to whether they have ever felt a similar pressure to avoid loops, the extent to which their code avoids loops as a result, ad whether it takes them a long time to find a non-looping solution where a looping solution would be quicker to write, despite taking up more lines.

There may be other languages for which similar issues arise. I used APL thirty years ago and seem to recall that as having array-processing capabilities and a philosophy of avoiding loops where possible. I'd be interested in comments on this issue for other languages too.

Another reason I try to avoid loops is a belief that code that uses array processing capabilities will generally run faster than code with explicit loops, since R is an interpreted language. I don't know if there's anything in that.
 

Answers and Replies

  • #2
14,171
8,149
Sometimes implicit array processing can be parallelized in ways hand written loops cannot And that’s the primary reason for using these constructs.

However array processing can be more memory intensive with intermediate results storage.

one example, I saw was with APL programming Where a very compact program computed the prime numbers from 1 to 100000. It did it by making 1000000x1000000 array with each row representing a number and each column of each row containing a 0 or 1 to indicate that the column number was a factor of the row number. Then is summed over all columns to make a single row of counts.

a filter was applied over the single row to create a list of column numbers where the row sum was 2 Meaning the number had two factors 1 and itself. That list was the primes numbers from 1 to 1000000.

anyway, you can see in this example the heavy use of memory initially.
 
  • #3
anorlunda
Staff Emeritus
Insights Author
11,184
8,577
In power systems analysis, the matrices are very sparse, with 98% of off-diagonal elements zero. We use special sparse matrix libraries and methods, and of course we really don't allocate or store those 98% zero elements.

I've been retired for a long time, so my info is dated. Can modern compilers and parallel processing features deal efficiently with sparse matrices?
 

Suggested for: Programming style in R - to array-process or not?

  • Last Post
Replies
2
Views
488
Replies
9
Views
951
Replies
21
Views
2K
  • Last Post
Replies
24
Views
1K
Replies
18
Views
726
Replies
10
Views
1K
  • Last Post
Replies
1
Views
903
Replies
18
Views
899
  • Last Post
Replies
6
Views
885
Replies
20
Views
1K
Top