Questions about the for loop, iterable, iterator in Python

[fog37]

Hello,
I understand that a $for$ loop is intended to repeat the same code a finite and specified number of time. The loop syntax is

for i in iterable:

      statement(s)

for = keyword
i = counter variable
iterable = any iterable (list, tuple, dictionary, range())
statements = body of the loop
in = membership operator
i in iterable = Boolean condition. If true, the loop body is executed. If false, the loop body is not run

My understanding is that the for loop works in the following way under the hood:

1. a counter variable $i$ is automatically created
2. the iterable is converted to an iterator via the $iter()$ function
3. the $next()$ function is applied to the iterator and each value that $next()$ sifts from the iterator is assigned to the variable $i$

Is that correct?

Another way to understand what for loop is that:

1. A variable $i$ is automatically created and fist assigned as value the first element of the iterable. Because of that, the condition i in iterable is True so the body of the loop is executed
2. the for statement is run again and this time the i variable has been updated and assigned the 2nd value inside the iterable, the loop body is executed and so on until the variable $i$ has assumed the last value of the iterable...

Which one is the correct way to interpret what is happening inside a $for$ loop?

I understand a little better the difference between iterables and iterators. I am still unsure of why an iterator is better than a list in the sense that we can selecting the elements of a list using indexing, for example, instead of converting an iterable into a iterator and using the $next()$ function...

Thank you!

 

[hmmm27]

From w3schools Python For Loops
"

A for loop is used for iterating over a sequence (that is either a list, a tuple, a dictionary, a set, or a string).​

This is less like the for keyword in other programming languages, and works more like an iterator method as found in other object-orientated programming languages.​

With the for loop we can execute a set of statements, once for each item in a list, tuple, set etc.​

"

Bearing in mind that I'm not a Python programmer, it looks like it simply takes a list, string, table, whatever - ie: something that's defined as a bunch of iterations - and runs through it from beginning to end, element by element.

Your i in that regard is not an integer (unless it is), nor a counter ; just, a placeholder for the group's element. The linked page explains other stuff about the statement. Try the range() parameter to get a regular for loop.

As far as "why" is concerned, I can only postulate that the language's creators wanted you to be able to cycle through a list without bothering with an otherwise unnecessary explicit index.

 

[FactChecker]

My understanding is that the for loop works in the following way under the hood:

1. a counter variable $i$ is automatically created

1. No. If 'i' is in the iterables, it is not a counter variable, it is one of the iterable things [EDIT]in the iterable. It may be a completely different type of thing than a counter variable.

   [*]the iterable is converted to an iterator via the $iter()$ function

   An iterable is one of the members of the list [EDIT]conains an ordered set of items. The iterator is the thing that progresses [EDIT] through the iterable from one iterable item to the next. It progresses through the iterables in the list.

   [*]the $next()$ function is applied to the iterator and each value that $next()$ sifts from the iterator is assigned to the variable $i$

Is that correct?

Another way to understand what for loop is that:

1. A variable $i$ is automatically created and fist assigned as value the first element of the iterable. Because of that, the condition i in iterable is True so the body of the loop is executed

1. This sounds wrong. There is no True/False test applied here.

   [*]the for statement is run again and this time the i variable has been updated and assigned the 2nd value inside the iterable, the loop body is executed and so on until the variable $i$ has assumed the last value of the iterable...

Which one is the correct way to interpret what is happening inside a $for$ loop?

I understand a little better the difference between iterables and iterators. I am still unsure of why an iterator is better than a list

They do not play the same role. A list iterable contains iterables items, not an iterator. An iterator progresses through the list of iterable, taking the value of each iterable item in turn. It is not the entire list.

in the sense that we can selecting the elements of a list using indexing, for example, instead of converting an iterable into a iterator and using the $next()$ function...

Introducing an index just makes an often unnecessary variable that has to be incremented at the correct time. There are times when you really need to know the index (for use to get things out of other lists) and this would be appropriate. If that is not the case, the index just gets in the way.

 

[PeterDonis]

My understanding is that the for loop works in the following way under the hood:

1. a counter variable $i$ is automatically created
2. the iterable is converted to an iterator via the $iter()$ function
3. the $next()$ function is applied to the iterator and each value that $next()$ sifts from the iterator is assigned to the variable $i$

Is that correct?

Not quite. The variable $i$ is not a "counter" variable; it doesn't tell you the index of the item you're working with in the body of the for loop. It just gives you the item itself. If you want to have the index available as well as the item, you have to use the enumerate built-in function, like so:

for i, item in enumerate(iterable):
    statement(s)

In this code, inside the for loop body, the variable i gives the index you are at in the iterable, and the variable item gives the item that is at that index in the iterable.

Another way to understand what for loop is that:

1. A variable $i$ is automatically created and fist assigned as value the first element of the iterable. Because of that, the condition i in iterable is True so the body of the loop is executed
2. the for statement is run again and this time the i variable has been updated and assigned the 2nd value inside the iterable, the loop body is executed and so on until the variable $i$ has assumed the last value of the iterable...

No. The i in iterable is not a condition being tested. The in keyword has a different meaning here than it does in an if statement.

I understand a little better the difference between iterables and iterators. I am still unsure of why an iterator is better than a list in the sense that we can selecting the elements of a list using indexing, for example, instead of converting an iterable into a iterator and using the $next()$ function...

You should never have to manually convert an iterable into an iterator; just use the Python constructs like for loops in the natural way and Python will create iterators behind the scenes whenever it needs to.

Note that an "iterable" is much more general than just a list. You can iterate over dicts, sets, and instances of custom classes that have the appropriate methods defined. You can also use generators as iterables. And again, if you just use constructs like for loops in the natural way, Python will convert all of these to iterators behind the scenes as needed; you don't have to worry about that part of it at all.

 

[PeterDonis]

the language's creators wanted you to be able to cycle through a list without bothering with an otherwise unnecessary explicit index

That's part of it; another part is, as I mentioned in my previous post just now, that you can iterate over lots of things, not just lists, and many of them don't even have an "index" the way a list does.

 

[PeterDonis]

A list contains iterables

No. A list contains items. The list itself is the iterable.

An iterator progresses through the list of iterables

No. An iterator progresses through a list of items--or through the items in whatever container is being iterated over, which need not be a list. The overall container it is iterating over is the iterable.

 

[FactChecker]

No. A list contains items. The list itself is the iterable.
No. An iterator progresses through a list of items--or through the items in whatever container is being iterated over, which need not be a list. The overall container it is iterating over is the iterable.

I stand corrected on the terminology and will try to edit my post.

 

[fog37]

Not quite. The variable $i$ is not a "counter" variable; it doesn't tell you the index of the item you're working with in the body of the for loop. It just gives you the item itself. If you want to have the index available as well as the item, you have to use the enumerate built-in function, like so:

for i, item in enumerate(iterable):
    statement(s)

In this code, inside the for loop body, the variable i gives the index you are at in the iterable, and the variable item gives the item that is at that index in the iterable.
No. The i in iterable is not a condition being tested. The in keyword has a different meaning here than it does in an if statement.
You should never have to manually convert an iterable into an iterator; just use the Python constructs like for loops in the natural way and Python will create iterators behind the scenes whenever it needs to.

Note that an "iterable" is much more general than just a list. You can iterate over dicts, sets, and instances of custom classes that have the appropriate methods defined. You can also use generators as iterables. And again, if you just use constructs like for loops in the natural way, Python will convert all of these to iterators behind the scenes as needed; you don't have to worry about that part of it at all.

 

[jtbell]

This is from wikipedia describing how a for loop has an initialization as well as a condition

Which Wikipedia article is this? I can't find the text in your images in the article about the "for loop"

https://en.wikipedia.org/wiki/For_loop

 

[fog37]

Which Wikipedia article is this? I can't find the text in your images in the article about the "for loop"

https://en.wikipedia.org/wiki/For_loop

My mistake. It is from Wikipedia but at https://en.wikipedia.org/wiki/Conditional_loop
I will remove it so others don't get confused.

I guess, as PeterDonis I believe suggested, no for loop has a condition being tested...

 

[fog37]

As far as the meaning of the membership operator in, the operator in performs membership testing in different ways for if and for structures.
https://stackoverflow.com/questions/54124354/python-in-operator-with-for-loop-and-with-if-statement

In the case of the for loop, both next() and the method __contain__ are called.

In the case of an if statement, only the method __contain__ iscalled when in is implemented.

 

[wle]

As far as the meaning of the membership operator in, the operator in performs membership testing in different ways for if and for structures.
https://stackoverflow.com/questions/54124354/python-in-operator-with-for-loop-and-with-if-statement

In the case of the for loop, both next() and the method __contain__ are called.

No, you're over thinking this. The for loop as a whole executes its body repeatedly with the variable bound to successive elements of the iterable. The "in" in "for var in iterable:" is not an operator that "does" anything in this context. It is just part of the Python syntax of a for loop.

Elsewhere, in expressions, in is a binary operator that returns True if the first operand is contained in the second and False if it isn't. It is not limited to being used only in if statements. For example,

print(3 in [1,2,4])

prints "False".

 

[PeterDonis]

no for loop has a condition being tested

That's correct. Python is different in this respect from languages like C.

the membership operator in,

As I have already pointed out, the Python keyword in has a different meaning (i.e., is a different operator) in a for loop than in an if statement. In an if statement, "in" is the membership operator, testing whether an object is contained in a container. In a for loop, "in" is just a keyword that designates the container that is being iterated over. So anything about "in" as a membership operator is irrelevant to for loops in Python.

In the case of the for loop, both next() and the method __contain__ are called.

Wrong. I have no idea where you are getting this from, but it certainly isn't from any knowledge about how the Python interpreter actually works.

Fortunately, Python makes it easy to test how the interpreter actually works. Consider the following:

>>> class TestContainer(object):
...     def __init__(self, items):
...         self.items = items
...         self.iterator = None
...     def __iter__(self):
...         self.iterator = iter(self.items)
...         return self
...     def __next__(self):
...         try:
...             return next(self.iterator)
...         except TypeError:
...             raise StopIteration
...         except StopIteration:
...             self.iterator = None
...             raise
...
>>> test = TestContainer([1, 2, 3])
>>> for item in test:
...     print(item)
...
1
2
3
>>> 1 in test
True
>>> 0 in test
False
>>> class TestContainer1(TestContainer):
...     def __contains__(self, item):
...         raise NotImplementedError
...
>>> test1 = TestContainer1([1, 2, 3])
>>> for item in test1:
...     print(item)
...
1
2
3
>>> 1 in test1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in __contains__
NotImplementedError
>>> 0 in test1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in __contains__
NotImplementedError

This code shows several things:

(1) A for loop never calls the __contains__ method on an object; that method doesn't exist at all for the first test container, and raises NotImplementedError for the second, and the for loop works fine in both cases.

(2) The "in" keyword in an if statement does call the __contains__ method on an object, if that method exists; that's why NotImplementedError gets raised for both if statements on the second container.

(3) If the __contains__ method does not exist on an object, the "in" keyword in an if statement still works and returns proper results, as shown in the if statements on the first container. How does that work? Well, under the hood, the Python interpreter, if it can't find a __contains__ method on an object, sees if it can iterate over the object; if it can, it does so, and looks to see if any of the values produced are the one it is looking for. That's what is happening with the if statements on the first container.

Exercise for the reader: could you add print statements to the above code to explicitly test my claim in #3 above? That is, to explicitly test that the __iter__ and __next__ methods on the container get called when an if statement with the "in" operator is used and no __contains__ method is present?

 

[FactChecker]

In Python, the "for" loop is actually a "for ... in" statement. The "for" and the "in" go together and their meaning is specific to that use. Any other meaning of "in" does not apply. (See https://docs.python.org/3/reference/compound_stmts.html#the-for-statement )

 

[fog37]

In Python, the "for" loop is actually a "for ... in" statement. The "for" and the "in" go together and their meaning is specific to that use. Any other meaning of "in" does not apply. (See https://docs.python.org/3/reference/compound_stmts.html#the-for-statement )

Thanks.
I believe that the format "for...in" is surely one way to go but I think I have seen a construct without the "in".

"for...something :" and if that "something" is true the loop gets executed.

 

[FactChecker]

Thanks.
I believe that the format "for...in" is surely one way to go but I think I have seen a construct without the "in".

"for...something :" and if that "something" is true the loop gets executed.

The link I gave is to Python.org's documentation of Python 3.9. I do not see the "for" without "in" there. (There is "in" without "for", but that is not the same thing.)

 

[fog37]

The link I gave is to Python.org's documentation of Python 3.9. I do not see the "for" without "in" there. (There is "in" without "for", but that is not the same thing.)

Sorry, you are correct. I am making mental mistakes these days.

I was thinking about a while loop which can have either a Boolean condition in the form of an expression to be tested (for example, while ...i<5: etc.) or can just have a simpler Boolean case, for example "while string: " and if "string" is True the while loop is executed...

But the for loop always must have the "for...in" structure.

Case closed :)