Iterators and generators and the data they generate

AI Thread Summary
Iterators in Python are objects that implement the __iter__ and __next__ methods, allowing data to be generated one item at a time without consuming large amounts of RAM. They can pull data from existing sources, like text files, without loading everything into memory, and can also generate data dynamically, such as an infinite sequence of numbers. Generators, which are functions that use the yield statement, create custom iterators and return generator objects that can be iterated over. While iterables like range() and map() are not iterators themselves, they can be converted into iterators using the iter() method. Overall, iterators and generators provide efficient ways to handle data generation and retrieval in Python.
fog37
Messages
1,566
Reaction score
108
TL;DR Summary
Iterators and generators and the data they generate
Hello,
I have been focusing on iterators and generators and I understood a lot but still have some subtle questions...

An iterator, which is a Python object with both the __iter__ and the __next__ methods, saves its current state. Applying the next() method to an iterator gives us one item of data at a time. On the other hand, when a regular Python list is created, all the data in the list is generated at once taking a lot of RAM. But when an iterator is created, I believe we are essentially saving the "recipe" on how to create the data but the data is generated a piece at a time and only upon our request. As we ask the iterator for data, step by step using the next() method, we are not creating and storing the data in RAM: the iterator does not save (unless we explicitly code for it) the data ahead of time and the data that is generates, correct?

Example: the data the iterator is using may already exist and be saved in the permanent memory. For example, there may be a huge text file saved on the computer. The iterator may pick a line at time from the text file without loading the entire file in RAM.
The iterator may also generate its data dynamically. For example, when we use an iterator to generate an infinite set of numbers: we don't really create those numbers in memory ahead of time or even save them after they are generated...I believe..

A generator is a special type of function with the return statement replaced by the yield statement. Is a generator just a function whose outcome/return is an iterator? Is a generator essentially a way to create a custom iterator? Python has iterator objects like range(), map(), etc. We can also convert certain iterable data structures, like lists, dictionaries, tuples, etc. into iterators using the iter() method... Are generators a way to create flexible iterators?
 
Technology news on Phys.org
fog37 said:
the iterator does not save (unless we explicitly code for it) the data ahead of time and the data that is generates, correct?
Correct. It generates the data only as it is needed.

fog37 said:
Example: the data the iterator is using may already exist and be saved in the permanent memory. For example, there may be a huge text file saved on the computer. The iterator may pick a line at time from the text file without loading the entire file in RAM.
Yes.

fog37 said:
The iterator may also generate its data dynamically. For example, when we use an iterator to generate an infinite set of numbers: we don't really create those numbers in memory ahead of time or even save them after they are generated...I believe..
Yes. For example, look at the count function in the itertools module.

fog37 said:
A generator is a special type of function with the return statement replaced by the yield statement.
More precisely, a generator is any function that has one or more yield statements in its body. It can also have return statements in its body, although this is very rarely done. (A return inside a generator causes it to stop iteration immediately.)

fog37 said:
Is a generator just a function whose outcome/return is an iterator?
Not quite. Calling the generator function returns a generator object, which can be iterated over like any other iterable, for example in a for loop. Calling iter on an iterable (whether it's a generator or any other iterable) returns an iterator (although usually you don't need to do this explicitly, it gets done implicitly inside the interpreter when you use something like a for loop).

fog37 said:
Is a generator essentially a way to create a custom iterator?
Essentially, yes. But see above for some important details.

fog37 said:
Python has iterator objects like range(), map(), etc.
Actually, those aren't iterators, they are iterables. An iterable has an __iter__ method, but not a __next__ method. The built-in iter function implicitly calls the __iter__ method of an iterable to return an iterator over that iterable. The Python documentation goes into a fair bit of detail about all this.

fog37 said:
We can also convert certain iterable data structures, like lists, dictionaries, tuples, etc. into iterators using the iter() method... Are generators a way to create flexible iterators?
I'm not sure what you mean by "flexible".
 
Dear Peeps I have posted a few questions about programing on this sectio of the PF forum. I want to ask you veterans how you folks learn program in assembly and about computer architecture for the x86 family. In addition to finish learning C, I am also reading the book From bits to Gates to C and Beyond. In the book, it uses the mini LC3 assembly language. I also have books on assembly programming and computer architecture. The few famous ones i have are Computer Organization and...
I have a quick questions. I am going through a book on C programming on my own. Afterwards, I plan to go through something call data structures and algorithms on my own also in C. I also need to learn C++, Matlab and for personal interest Haskell. For the two topic of data structures and algorithms, I understand there are standard ones across all programming languages. After learning it through C, what would be the biggest issue when trying to implement the same data...

Similar threads

Back
Top