Manjusaka

Source annotated-py-asyncio

Supplementary Reading:#

1. Basic Concepts:#

1.1 Coroutines:#

• "A coroutine is a computer program component that produces subroutines for non-preemptive multitasking, coroutines allow different entry points to pause or start executing the program at different positions".
• Technically, "a coroutine is a function that can be paused and resumed".
• If you think of it as "just like a generator", then you're right.

1.2 Event Loop:#

• An event loop "is a programming architecture that waits for a program to allocate events or messages".
• Basically, an event loop is "when A happens, do B".
• Perhaps the simplest example to explain this concept is the JavaScript event loop that exists in every browser.
• When you click on something ("when A happens"), this click action is sent to the JavaScript event loop, which checks if there is a registered onclick callback to handle this click ("do B").
• As long as there are registered callback functions, they will be executed along with the details of the click action.
• The event loop is considered a loop because it continuously collects events and responds to them through a loop.

1.3 Python's Event Loop:#

• For Python, asyncio is added to the standard library to provide an event loop.
• asyncio focuses on solving problems in network services, where the event loop treats I/O from sockets that are ready to read and/or write as "when A happens" (through the selectors module).
• In addition to GUI and I/O, event loops are also often used to execute code in other threads or subprocesses, and the event loop serves as a coordination mechanism (e.g., cooperative multitasking).
• If you happen to understand Python's GIL, the event loop is useful in places where you need to release the GIL.
• The event loop provides a looping mechanism that allows you to "when A happens, do B".
• Basically, the event loop listens for what happens, and the event loop also cares about it and executes the corresponding code.
• Python 3.4 and later have the ability to obtain event loop features through the standard library asyncio.

1.4 async, await:#

• Consider async/await as the API for asynchronous programming.
• Basically, async and await produce magical generators, which we call coroutines.
• Additional support is needed, such as awaitable objects and converting ordinary generators into coroutines.
• All of these are combined to support concurrency, making Python better suited for asynchronous programming.
• Compared to similar features in threads, this is a more wonderful and simpler approach.

In Python 3.4, the function for asynchronous programming and marked as a coroutine looks like this:

    # This also works in Python 3.5.
    @asyncio.coroutine
    def py34_coro():
        yield from stuff()

Python 3.5 added the types.coroutine decorator, which can also mark generators as coroutines, just like asyncio.coroutine.
You can define a coroutine function using async def, although this function cannot contain any form of yield statement; only return and await can return values from the coroutine.

    async def py35_coro():
        await stuff()

You will find that not only async, but Python 3.5 also introduced the await expression (which can only be used in async def).
Although the use of await is similar to yield from, the objects that await can accept are different.
Of course, await can accept coroutines, as the concept of coroutines is the basis of all this.
However, when you use await, the object it accepts must be an awaitable object: it must define the await() method and this method must return an iterator that is not a coroutine.
Coroutines themselves are also considered awaitable objects (which is why collections.abc.Coroutine inherits from collections.abc.Awaitable).
This definition follows Python's tradition of translating most syntax structures into method calls at the lower level, just like a + b is actually a.add(b) or b.radd(a).

Why do async-based coroutines and generator-based coroutines differ in the corresponding suspension expressions?
The main reason is for optimizing Python performance, to ensure that you don't confuse different objects with the same API.
Since generators implement the coroutine API by default, it is very likely that you will mistakenly use a generator when you want to use a coroutine.
And because not all generators can be used in coroutine-based control flow, you need to avoid using generators incorrectly.

Coroutines can be defined using async def.
Another way to define coroutines is through the types.coroutine decorator

• Technically, it adds the CO_ITERABLE_COROUTINE flag
• Or it is a subclass of collections.abc.Coroutine.
  You can only achieve coroutine suspension through generator-based definitions.

An awaitable object is either a coroutine or an object that defines the await() method

• It is also a collections.abc.Awaitable
• And await() must return an iterator that is not a coroutine.

The await expression is basically the same as yield from, but it can only accept awaitable objects (ordinary iterators cannot).
An async-defined function either contains a return statement

• Including the default return None for all Python functions
• And/or an await expression (yield expressions are not allowed).
  The restrictions on async functions ensure that you do not mix generator-based coroutines with regular generators, as the expectations for these two types of generators are very different.

1.5 Comparison of Python Coroutines and Golang:#

• From Python to Go and Back Again
  • PPT
  • Discussion on this PPT: Go is not much faster than PyPy, but the complexity and difficulty of debugging increase significantly
  • The conclusion promotes Rust.
• Reference to asynchronous libraries:
  • hyper
  • curio
    • Considers asyncio as a framework for asynchronous programming using the async/await API
    • David considers async/await as the API for asynchronous programming and created the curio project to implement his own event loop.
    • Allows projects like curio to have different ways of operating at a lower level
    • (e.g., asyncio uses future objects as the API for communication with the event loop, while curio uses tuples)

2. Source Code Module:#

2.1 futures.py#

2.1.1 References:#

• How the heck does async/await work in Python 3.5?

  • Translated: How the heck does async/await work in Python 3.5?
  • Translated: GitHub
  • Translated: Blog

• Python Coroutine: From yield/send to async/await

  • Analysis of the future source code

• concurrent.futures Source Code Reading Notes (Python)

  • concurrent.futures is an asynchronous library
  • concurrent.futures — Asynchronous computation

2.1.2 Generator VS Iterator:#

• Improve Your Python: Yield and Generators Explained

  • Translated: 提高你的 Python: 解释‘yield’和‘Generators（生成器）’
  • Outside of Python, the simplest generator should be something called coroutines.
  • A generator is used to produce a series of values.
  • yield is like the return result of a generator function.
  • Another thing that yield does is save the state of a generator function.
  • A generator is a special type of iterator.
  • Similar to an iterator, we can use next() to get the next value from a generator.
  • We can also ignore some values by implicitly calling next().

• Relationship between Generators and Iterators

  • A generator is a special type of iterator, and its implementation is simpler and more elegant. yield is the key to implementing the next() method of a generator.

• Python Black Magic - Iterators

• Generators