This is the question which always hit my mind, How this works ? How this can be paused and resumed ?
Generally, functions just do their work by start executing its code from the first line till the return statement occurs. Singly entry point and multiple exit points, in the form of return statements.
General function are called by some other caller function and once the execution of code is done the control is returned back to the caller func( ). The function’s stack of local variable is cleared and their associated memory is also freed up by the system.
Generator functions however don’t behave this way, they have some extra powers than general functions they can be paused and then resumed at any time from any function and the state of the function is also intact nothing is lost.
Generator func () have multiple entry and exit points, each yield statement in a func( ) defines the Entry and Exit Point. Function’s execution continues until the yield statement is encountered. Till that point, the state of the function is preserved and the flow of control is yielded to the caller function.
When the next is called, the generator func ( ) is resumed, it’s last state is recalled and the execution starts from the line following the yield statement at which the func ( ) was last paused.
How the function works in CPython Interpreter? Is their some kind of memory allocation taken care by the CPython Interpreter internally for the functions?
Expected behavior of normal function will be like, all the variables used by the function are pushed to stack frame and it remains there, till the execution of the function doesn’t complete. And it is expected that at the time of function exits it should clear up the stack frame and the corresponding memory allocation.
But that is not the case here in Python, Python stack frame are not allocated on Stack memory, it allocates at Heap Memory. Because of this python stack frames can outlive their respective function calls and Generator func ( ) leverage this behavior to do its work, which makes them different from the normal func ( ).
When the CPython compiler encounters the yield statement in a function, it sets a flag on the compiled object to tell the CPython interpreter that the function is a generator function.
When the CPython interpreter sees the flag on the object associated with a function, it does not execute the function but instead returns a generator object. The generator object is an iterator. Which means that we can iterate through it using the next keyword or via for loop(see below).
Sample Generator Function:
# A simple generator function
def sample_generator():
n = 1
print('First')
# Generator function contains yield statements
yield n
n += 1
print('Second')
yield n
n += 1
print('Third')
yield n
# Using for loop
for item in sample_generator():
print(item)
When you run the program, the output will be:
First 1 Second 2 Third 3
As we iterate through the generator function, execution continues until a yield statement is encountered. At that point, the stack frame of the function is frozen and control is returned back to the caller of the generator function.
As we continue advancing through the generator func ( ) by calling the next keyword or via loop, execution always begin from the point where it last left off. Stack frame object is storing all these information which is associated with the generator instance being executed.
pointer is an index into the bytecode string of the code object associated with the body of the generator function and points to the last bytecode instruction that was run within the context of a stack frame. When an instance of a generator function is resumed, the CPython interpreter uses the pointer on the associated stack frame to determine where to begin executing the code object of the generator function .
When a generator func ( ) is called it creates and returns a generator object, no code in the function’s body get executed during this call. Pointer at that time is initialized and it advances from one yield statement to another, pausing and then resuming the same point, until the function is exhausted and a StopIteration exception is thrown by the compiler.
Read more about the memory management – Click Here ( Stack Overflow )
