Problem with appending a dataframe after a loop

  • #1
53
6
I am iterating over 2 variables below and after the calculation are done, i'd like to append the dataframe to add the rows after each iteration, but what is happening now is that the row is getting replaced instead of getting added.

Python:
pre = [10, 9, 8, 7, 6, 5, 4, 3, 2, 1]
post  = [0, 1, 2, 3, 4, 5]

for i in pre:
    for j in post:
        results = pd.DataFrame(index=None)
        row = pd.DataFrame({'pre':i, 'post:j})
        results = results.append(row, ignore_index=True)

how do i ensure that at every iteration a new row will be added instead of replacing the existing one? Thanks
 

Answers and Replies

  • #3
53
6
What does that line do?
it creates a new dataframe called results and i am appending this dataframe with the values from row
 
  • #4
Ibix
Science Advisor
Insights Author
9,042
8,730
it creates a new dataframe called results and i am appending this dataframe with the values from row
So what does it do the second time round the loop?
 
  • #5
53
6
So what does it do the second time round the loop?
so for the first row it should add 10, 0 and when it goes through the loop again, there should be a new row in with values 10, 1 but whats happening now is the 10,0 is getting replaced with 10,1 instead of getting added.
 
  • #6
Ibix
Science Advisor
Insights Author
9,042
8,730
Not what I wanted to know. What does that line I quoted do the second time around the loop?
 
  • #7
53
6
Not what I wanted to know. What does that line I quoted do the second time around the loop?
yes, i get what you mean now. it creates an empty dataframe again. didn't occur to me until now! thanks. i will move it to before the loop begins.
 
  • Like
Likes rbelli1 and Ibix
  • #8
53
6
changed the code to below:

Python:
import pandas as pd
pre = [10, 9, 8, 7, 6, 5, 4, 3, 2, 1]
post  = [0, 1, 2, 3, 4, 5]
index = 0

results = pd.DataFrame(index=None)
for i in pre:
    for j in post:
        row = pd.DataFrame({'pre':i, 'post':j})
        results = results.append(row, ignore_index=True)
        print('The new data frame is: \n{}'.format(results))

but its giving me this error now

ValueError: If using all scalar values, you must pass an index --- at the row line.. i am not sure what index should i place in there.
 
  • #9
53
6
changed the code to below:

Python:
import pandas as pd
pre = [10, 9, 8, 7, 6, 5, 4, 3, 2, 1]
post  = [0, 1, 2, 3, 4, 5]
index = 0

results = pd.DataFrame(index=None)
for i in pre:
    for j in post:
        row = pd.DataFrame({'pre':i, 'post':j}, index[0])
        results = results.append(row, ignore_index=True)
        print('The new data frame is: \n{}'.format(results))
solved with adding index[0]
 
  • #10
1
0
Iterating through large pandas dataFrame objects is generally slow. Pandas iteration beats the whole purpose of using DataFrame. It is an anti-pattern and is something you should only do when you have exhausted every other option. It is better look for a List Comprehensions , vectorized solution or DataFrame.apply() method.

Pandas DataFrame loop using list comprehension example

Code:
result = [(x, y,z) for x, y,z in zip(df['column_1'], df['column_2'],df['column_3'])]
 
Last edited by a moderator:

Related Threads on Problem with appending a dataframe after a loop

Replies
5
Views
763
Replies
1
Views
944
Replies
4
Views
426
Replies
3
Views
872
Replies
62
Views
10K
Replies
8
Views
6K
O
Replies
5
Views
5K
  • Last Post
Replies
3
Views
2K
Replies
3
Views
3K
Top