Problem with appending a dataframe after a loop

[msn009]

I am iterating over 2 variables below and after the calculation are done, i'd like to append the dataframe to add the rows after each iteration, but what is happening now is that the row is getting replaced instead of getting added.

pre = [10, 9, 8, 7, 6, 5, 4, 3, 2, 1]
post  = [0, 1, 2, 3, 4, 5]

for i in pre:
    for j in post:
        results = pd.DataFrame(index=None)
        row = pd.DataFrame({'pre':i, 'post:j})
        results = results.append(row, ignore_index=True)

how do i ensure that at every iteration a new row will be added instead of replacing the existing one? Thanks

 

[Ibix]

        results = pd.DataFrame(index=None)

What does that line do?

 

[msn009]

What does that line do?

it creates a new dataframe called results and i am appending this dataframe with the values from row

 

[Ibix]

it creates a new dataframe called results and i am appending this dataframe with the values from row

So what does it do the second time round the loop?

 

[msn009]

So what does it do the second time round the loop?

so for the first row it should add 10, 0 and when it goes through the loop again, there should be a new row in with values 10, 1 but what's happening now is the 10,0 is getting replaced with 10,1 instead of getting added.

 

[Ibix]

Not what I wanted to know. What does that line I quoted do the second time around the loop?

 

[msn009]

Not what I wanted to know. What does that line I quoted do the second time around the loop?

yes, i get what you mean now. it creates an empty dataframe again. didn't occur to me until now! thanks. i will move it to before the loop begins.

 

[msn009]

changed the code to below:

import pandas as pd
pre = [10, 9, 8, 7, 6, 5, 4, 3, 2, 1]
post  = [0, 1, 2, 3, 4, 5]
index = 0

results = pd.DataFrame(index=None)
for i in pre:
    for j in post:
        row = pd.DataFrame({'pre':i, 'post':j})
        results = results.append(row, ignore_index=True)
        print('The new data frame is: \n{}'.format(results))

but its giving me this error now

ValueError: If using all scalar values, you must pass an index --- at the row line.. i am not sure what index should i place in there.

 

[msn009]

changed the code to below:

import pandas as pd
pre = [10, 9, 8, 7, 6, 5, 4, 3, 2, 1]
post  = [0, 1, 2, 3, 4, 5]
index = 0

results = pd.DataFrame(index=None)
for i in pre:
    for j in post:
        row = pd.DataFrame({'pre':i, 'post':j}, index[0])
        results = results.append(row, ignore_index=True)
        print('The new data frame is: \n{}'.format(results))

solved with adding index[0]

 

[carlhyde]

Iterating through large pandas dataFrame objects is generally slow. Pandas iteration beats the whole purpose of using DataFrame. It is an anti-pattern and is something you should only do when you have exhausted every other option. It is better look for a List Comprehensions , vectorized solution or DataFrame.apply() method.

Pandas DataFrame loop using list comprehension example

result = [(x, y,z) for x, y,z in zip(df['column_1'], df['column_2'],df['column_3'])]