Register to reply

C++ help- my code reads from small files but not large

by davidsmith315
Tags: c++, csv file, large
Share this thread:
davidsmith315
#1
Apr23-12, 04:00 AM
P: 3
Hello, I'm trying to make part of a code that reads a csv file with data containing the date and time, and the wind speed at that time, then does various output things.
When I copy and paste 1000 entriesinto a csv file and ru it, it works great, but if I use the file thats about 6,000kb it misses every second date and time part out, code anyone help me with this?
I'm a complete novice to programming- this is part of my Electrical Engineering course.

heres my code:
#include <iostream>
#include <sstream>
#include <fstream>
#include <vector>
#include <string>
#include <ctime>
#include <FileHC.h>

using namespace std;

int main ()
{


	  ofstream outFile("C:\\WindSpeedAnswer.csv");
  if (!outFile) {
	  cout<< "error opening file"<<endl;
	  return -1;
  }
  
  
    ifstream inFile ("C:\\Copy of Kirkwall_23_hm_proc15.csv");
    string line;
    int linenum = 0;
   if (inFile.is_open())
  {
	  double randomMax;
	  double randomMin;
	  cout<<"would you like to add a random varience, enter 'yes' if you do "<<endl;
	 string Question;
	cin >> Question;
	string answerYes= "yes";
	if (Question==answerYes){
	cout<<"enter the max value of random varience "<<endl;
	cin>>randomMax;
	randomMax=randomMax-1;

	
	cout<<"enter the min value of random varience "<<endl;
	cin>>randomMin;
	randomMin=randomMin-1;}




	while (getline (inFile, line))
    {
        linenum++;
       // cout << "\nLine #" << linenum << ":" << endl;
        istringstream linestream(line);
        string item;
        int itemnum = 0;
		
        while (getline (linestream, item, ','))
        {
            itemnum++;
			double WindSpeed=0;
		string DateAndTime;
	
  


            //cout << "Item #" << itemnum << ": " << item << endl;
			if (itemnum%2 == 1) {
		
			DateAndTime=item;

				
		string replace= "-99999";


	while(DateAndTime==replace){
			DateAndTime= "Data Missing";}

	cout<<"at "<<DateAndTime<<endl;
		outFile<< DateAndTime<<","	;}
			
		
	if (itemnum%2 == 0) {
		
	WindSpeed=atof(item.c_str());
	if(WindSpeed==-99999){
			
	WindSpeed=9;
			}
	if (Question==answerYes){
	for(int index=0; index<itemnum; index++){
	double X = randomMax+ rand() * (randomMax - randomMin) / RAND_MAX;
		if (randomMin<X<randomMax){
		WindSpeed=X;}
			}}



cout<<"the wind speed is "<<WindSpeed<<","<<endl;
		
outFile<<WindSpeed<<","<<endl;
//outFile.write(WindSpeed); 
			}
		}
			}
		
	 inFile.close();
 }

  else cout << "Unable to open file"; 
  

  
	  outFile.close();
  
  char stopchar;
  cin>>stopchar;
  return 0;
}
Phys.Org News Partner Science news on Phys.org
Physical constant is constant even in strong gravitational fields
Montreal VR headset team turns to crowdfunding for Totem
Researchers study vital 'on/off switches' that control when bacteria turn deadly
Hobin
#2
Apr23-12, 04:04 AM
P: 194
Here's something to try: See if it still doesn't work when you only take the first 100 lines or so of the 6,000kb file. If it still doesn't work, check the file for errors. It's much easier to check a 100-line file, and it's quite possible that there are a few thingies that make your program go weird.
davidsmith315
#3
Apr23-12, 04:16 AM
P: 3
Re Hobin:
Thanks fo reply
It work great if i take the firs 100 lines, or 1000 lines- its only when I run the 6,oookb file it doesntwork

Hobin
#4
Apr23-12, 06:08 AM
P: 194
C++ help- my code reads from small files but not large

Quote Quote by davidsmith315 View Post
It work great if i take the firs 100 lines, or 1000 lines- its only when I run the 6,oookb file it doesntwork
That's odd. To be perfectly honest, I can't find anything in your code that would make it stop working at some arbitrary amount of data - have you tried running your code on a different machine? Try splitting the file in the middle (if that's possible), if it works, take 3/4 of the file, if it doesn't, take 1/4 - repeat until you find the boundary where it stops working.
M Quack
#5
Apr23-12, 06:10 AM
P: 673
Sounds like you have a memory leak somewhere, for example objects are being created inside the loop, but never deleted.

For better readability, I would put the treatment of each complete line (everything inside the while(getline) loop) into a subroutine.
davidsmith315
#6
Apr23-12, 06:28 AM
P: 3
Thanks for replies, I tried using half the input file and it failed, but using 1/4 f it worked perfectly.

Re M Quck- "For better readability, I would put the treatment of each complete line (everything inside the while(getline) loop) into a subroutine"
Sorry to ask but how would I go about doing this?
I tried this:
if (itemnum%2 == 1) {

DateAndTime=item;
string replace= "-99999";
while(DateAndTime==replace){
DateAndTime= "Data Missing";}
cout<<"at "<<DateAndTime<<endl;
}
if (itemnum%2 == 0) {
WindSpeed=atof(item.c_str());
if(WindSpeed==-99999){
WindSpeed=9;
}
if (Question==answerYes){
for(int index=0; index<itemnum; index++){
double X = randomMax+ rand() * (randomMax - randomMin) / RAND_MAX;
if (randomMin<X<randomMax){
WindSpeed=X;}
}}

cout<<"the wind speed is "<<WindSpeed<<","<<endl;

outFile<<DateAndTime<<","<<WindSpeed<<","<<endl;



It nearly worked- except it left out the time and date part- the time and date row was blank, but the WindSpeed row was output perfectly
Hobin
#7
Apr23-12, 12:41 PM
P: 194
Quote Quote by davidsmith315 View Post
Thanks for replies, I tried using half the input file and it failed, but using 1/4 f it worked perfectly.
Very good, now try (3/4)/2 part of the file.

I hope you see where I'm going with this. By cutting what's left in half every time, you could theoretically very quickly go to the point where you come to the specific line where your file fails to work properly. Going to that extreme is probably unnecessary, but as you 'zoom in' on the part where it fails to work, take a very good look at possible anomalies in the file.
rcgldr
#8
Apr23-12, 01:21 PM
HW Helper
P: 7,182
Couldn't you use a source level debugger to see where the failure occurs? If you have windows, you can get microsoft visual c++ express for free, and it includes a source level debugger.
Hobin
#9
Apr23-12, 01:35 PM
P: 194
^That, too. Alternatively, if you use GCC, you can obviously download and use the GDB.
and9
#10
May1-12, 10:34 AM
P: 27
Whenever this happens for me, I just use GDB and run the code with the proper arguments and print the stack trace to see where the fault occurred.

Doing what the previous user suggested above in terms of essentially binary searching for the line where the code stops at is helpful but won't solve your problem if the issue is a memory leak. That style of debugging will only be helpful if thats the line of input error.

GDB will tell you if the input is faulty or if you have a memory leak. Also consider running your executable with memcheck to see if you have memory leaks.


Register to reply

Related Discussions
Any good short reads on optimisation of code? (math-specific) Math & Science Software 2
Problem loading large files (IOSTAT=100) Programming & Computer Science 0
MATLAB: Average a large number of matricies from .mat files Math & Science Software 2
Small or large faculty Academic Guidance 8
Small or Large Undergrad Uni? Academic Guidance 3