Declaring/filling arrays of arbitrary size in Mathematica

[yourgoldteeth]

edit: Forgot to add Mathematica in the title. Sorry!

Hi everyone.

I mined a lot of data and want to analyze it in Mathematica.

The data is of the form

{{x11, y11, x12, y11, x13, y13},{x21, y21, x22, y22}...}

meaning that each set of x,y pairs is not necessarily the same size.

Maybe I'm just dense, but I'm having a tough time writing it to my array correctly.

For[i = 1, i <= Length[rangeData], i++,
  {
   For[j = 1, j <= Length[rangeData[[i]]], j++,
     {
      If[rangeData[[i, j]] >= 0,
        {
         index = Mod[j, 2] + 1;
         cleanData[[i, j, (Mod[j, 2] + 1)]] = rangeData[[i, j]];
         }];
      }];
   }];

I'm getting the 'Part specification does not exist' error. I don't want to declare cleanData as a vector because I end up with empty spaces.

Any suggestions?

Thanks

 

[Xtensity]

If I'm understanding what you're saying, you're having trouble writing to the array because you don't know how many variables are going to be in the array?I had to do something similar to this in a delphi code I was working on a few weeks ago... I searched all online for answers.

The only answer I found was that, you have to loop whatever generates the variable. For each variable that is generated, then an integer is increased. Then you reloop the function and store them in the array because after the first loop through you will know how many variables will be in each section depending on how many times the integer was increased.

If you have multiple arrays which you do not know the value of then you may need to use an Array of Integers to account for the increase in each specific part.

Hope this was clear enough and related to your issue... The only thing is, its still a very inefficient way of doing it as if your original function takes a long time to execute, then it can bring timing problems.

 

[yourgoldteeth]

Found a way to do it. I realized that the longest vector in my array was 52 long. So I did this

(*specify an empty array with x as a placeholder*)
cleanData = ConstantArray["x", {Length[rangeData], 26, 2}];
(*write to the new array *)
For[i = 1, i <= Length[rangeData], i++,
  {
   For[j = 1, j <= Length[rangeData[[i]]], j++,
     {
      If[rangeData[[i, j]] >= 0,
        {
         index = 2 - (Mod[j, 2] + 1);
         temp = IntegerPart[(j + 1)/2];
         cleanData[[i, temp, (Mod[j, 2] + 1)]] = rangeData[[i, j]];
         }];
      }];
   }];

(*remove the x's, and have a clean array! *)
cleanData = DeleteCases[cleanData, {"x", "x"}, Infinity];

I use x's as a placeholder and delete them after I'm done. It works without flattening out my data.

 

[airborne18]

My Mathematica machine took a vacation on Friday, so I do not have any sample code. But I have found that Mathematica really does better with arrays using Map and Thread instead of procedural constructs.

I went through this paradigm realignment as I worked with large amounts of commodity pricing data. It takes some time to rethink how you approach the implmentation, but it is worth it.

I found significant performance improvements by eliminating recursive loops and using the built-in functionality. Most of the operations on data sets can be done with less code and even take advantage of multi-processing.

Just a thought. I am using real-time pricing and complex calculations across very large datasets that included text and numeric data and I hit a wall very quickly. Most of my issues were solved by playing the Mathematica way.

 

[yourgoldteeth]

airborne18: thanks for the reply!. When you get your Mathematica machine back, I would love to see examples of your method.

 

[airborne18]

airborne18: thanks for the reply!. When you get your Mathematica machine back, I would love to see examples of your method.

It will probably be a week or so. I run it on a laptop, and the power jack on the laptop broke. So I went to radio shack and took it apart and put my own jack in, and now the soldlering job I did came loose. I have to take it apart and break out the soldering iron.

I do not look forward to it, and I have other computers, just only one with Mathematica. Also it is back to college week for my daughter, so that is my focus. Plus my own coursework is due. ( I am a 43 year old college student ).

So give me a bit.

Just keep in mind, Mathematica is not Matlab. Mathematica supports procedural, but there are usually more elegant methods of doing things. I am a disabled vet and have free time to play with it, and it took a while for the C/C++ programmer in me to rethink my methods.

 

[yourgoldteeth]

I found a faster, more elegant solution. Since the empty parts of my array were of type $String, I did this:

masterData = DeleteCases[rawData[[1]], _String, Infinity];

Which got rid of all the empty parts without flattening my vector.

 

[airborne18]

Looking at your original code I thought you were also ordering the sets. But great job. That is life as Mathematica loves it. You should do the timing of each one.. just to see the difference.

One thing to read up on in Mathematica is the pattern and transformation feature, it is very powerful but takes effort to learn it. _String is a basic pattern, but you can get very complex and do all the manipulation of list data you want.

There are good sections in the online documentation on iterating lists ( everything is really a list )..

Also, if all else fails, you can create a function that takes the list elements and does the manipulation, then use the Map or Apply to iterate through the list. ( this is my usual technique ).

One more piece of advice.. if you have pairs of data.. x1,y1 x2,y2 and you want to do operations on pairs,, then split them into two lists.. listx { x1, x2} listy {y1,y2}
It will speed calculations and it is easier to code in most cases. You can always join the lists if you need. It really all depends on your need. ( patterns are better implemented on one list }.

 

[yourgoldteeth]

One more piece of advice.. if you have pairs of data.. x1,y1 x2,y2 and you want to do operations on pairs,, then split them into two lists.. listx { x1, x2} listy {y1,y2}
It will speed calculations and it is easier to code in most cases. You can always join the lists if you need. It really all depends on your need. ( patterns are better implemented on one list }.

Thanks for your help.

I will try splitting the list and see if it helps.

The big issue is this: the bottleneck of my program is a likelihood function that takes 2000 numerical integrals. And then I have to optimize a three-parameter model with that likelihood function (that takes about 100 iterations). And then I have to resample 1000 times and reoptimize for each one. So we are talking about 200 million numerical integrals. I haven't even figured out how long it's going to take yet.

 

[airborne18]

You should spend some time and look at the parallel processing features of Mathematica. I have had mixed results with it, but with integration it should benefit your effort. Assuming you have a true mulicore processor you can get all kernels running full blast.

Again, it will require some thoughtful design to maximize it. ( it is something I am still wrangling on with my own code. I have fairly simple integrals, but the data sets are large. )