1
Q
“What features does Python support because it’s an object-oriented programming language?”
A
“Python supports a full range of features (like inheritance
2
Q
“What do you have to do to get things done in Python?”
A
“To get things done in Python
3
Q
“What do classes and inheritance make easy to do?”
A
“The classes and inheritance make it easy to express a program’s desired behavior through objects.”
4
Q
“What else do classes and inheritance let you do?”
A
“They also let you to improve and expand functionality over time.”
5
Q
“What does knowing how to write good classes enable?”
A
“Knowing how to write good classes enables you to write maintainable code.”
6
Q
“What is Python’s built-in dictionary type useful for?”
A
“Python’s built-in dictionary type is useful for maintaining dynamic (situations where you need to do booking for an unexpected numbers of identifiers) internal state over the lifetime of an object.”
7
Q
“What risk exists with dictionaries and related built-in types?”
A
“Dictionaries and their related built-in types are so easy to use that there’s a risk of overextending them and creating brittle code.”
8
Q
“What is one solution to the dictionary overuse problem?”
A
“One solution to the above problem is to use a defaultdict instance to replace an inner dictionary and handle missing subjects.”
9
Q
“What should you do if your classes have a complex array of built-in types?”
A
“If your classes have a complex array of built-in types like dictionaries
10
Q
“What did Python’s built-in dictionary and tuple types make easy?”
A
“Python’s built-in dictionary and tuple types made it easy to keep adding layers to the internal bookkeeping.”
11
Q
“How much nesting should you avoid with dictionaries?”
A
“However
12
Q
“What should you do when internal bookkeeping gets complicated?”
A
“As soon as your internal bookkeeping gets complicated
13
Q
“What can you then provide after breaking into classes?”
A
“You can then provide well-defined interfaces that encapsulate all your data.”
14
Q
“What other benefit does this approach provide?”
A
“This approach also lets you create a layer of abstraction between your interfaces and your concrete implementations.”
15
Q
“How many approaches to refactoring exist?”
A
“There are many approaches to refactoring.”
16
Q
“What is extending tuples similar to?”
A
“Extending tuples longer and longer is similar to deepening layers of dictionaries.”
17
Q
“When should you use another approach instead of tuples?”
A
“As soon as your tuples get bigger than a two-tuple
18
Q
“What does the namedtuple type solve?”
A
“The namedtuple type in the collections module solves the problem mentioned above: it lets you define tiny
19
Q
“How can namedtuple classes be created?”
A
“The classes mentioned above can be created with positional or keyword arguments.”
20
Q
“How are namedtuple fields accessible?”
A
“The fields are accessible with named attributes.”
21
Q
“What does having named attributes make easy?”
A
“Having named attributes makes it easy to move from a namedtuple to a class later if the requirements change (like if you need to support mutability or behaviors in the simple data containers).”
22
Q
“What trade-off exists when breaking built-in types into classes?”
A
“Breaking built-in types into classes greatly increases the implementation size of a project
23
Q
“Are there situations where namedtuple can do more harm than good?”
A
“Although there are good things about namedtuple
24
Q
“Can you specify default argument values in namedtuple?”
A
“You can’t specify default argument values.”
25
“What does this limitation make namedtuple?”
“This makes them unwieldy when your data may have optional properties.”
26
“What might be a better choice if you’re using more than a handful of attributes?”
“If you’re using more than a handful of attributes
27
“How are namedtuple attribute values still accessible?”
“The attribute values of namedtuple instances are still accessible using numerical indexes and iteration.”
28
“What can this numerical access lead to?”
“This can lead to unintentional usage changes that make it harder to move to real classes later.”
29
“When is it better to explicitly define a new class instead of using namedtuple?”
“If you’re not in control of all the usages of your namedtuple instances
30
“What do many of Python’s built-in APIs let you do?”
“Many of Python’s built-in APIs let you customize behavior by passing in a function (these are called hooks).”
31
“What are hooks used for?”
“These hooks are used by APIs to call back your code when they execute.”
32
“How might hooks be defined in other languages?”
“In other languages
33
“What are many hooks in Python?”
“In Python
34
“Why are functions good for hooks?”
“Functions are good for hooks because they’re easier to describe and simpler to design than classes.”
35
“Why do functions work as hooks in Python?”
“Functions work as hooks because Python has first-class functions (functions and methods that can be referenced and passed around like any other value in the language).”
36
“What does supplying functions to functions do for APIs?”
“Supplying functions to functions makes APIs easy to build and test because it separates side effects from deterministic behavior.”
37
“What is another benefit of accepting simple functions for interfaces?”
“Another benefit of accepting simple functions for interfaces is that it’s easy to add functionality later by hiding a state closure.”
38
“What is the problem with defining a closure for stateful hooks?”
“The problem with defining a closure for stateful hooks is that it’s harder to read than stateless functions.”
39
“What is another approach for stateful hooks?”
“Another approach is to define a small class that encapsulates the state you want to track.”
40
“What can you pass to defaultdict as a default value hook?”
“In Python
41
“Is it easy to have an object instance’s method satisfy a function interface?”
“It’s easy to have an object instance’s method satisfy a function interface.”
42
“What is the problem with using a helper class for stateful closures?”
“Using a helper class to provide the behavior of a stateful closure is clearer than a function
43
“What special method does Python allow classes to define?”
“To fix the above the problem
44
“What does the **call** method cause the callable function to do?”
“It also causes the callable function to return True for an instance just like a method.”
45
“What are objects that can be executed like functions called?”
“Objects that can be executed in this manner are called callables.”
46
“How does defining **call** compare to passing an instance’s method?”
“Defining the **call** method is much clearer than passing an instance’s method to defaultdict.”
47
“What does **call** indicate about a class’s instances?”
“It also indicates that a class’s instances will be used somewhere where a function argument would be helpful (like API hooks).”
48
“What does **call** direct new readers to?”
“It directs new readers of the code to the entry point that’s responsible for the code’s behavior and provides a strong indication that the goal of the class is to act as a stateful closure.”
49
“Does defaultdict know what’s going on when you use **call**?”
“defaultdict has no view what’s going on when you use **call**.”
50
“What does defaultdict require?”
“The only thing that defaultdict requires is a function for the default value hook.”
51
“How many ways does Python provide to satisfy a simple function interface?”
“Python provides a lot of ways to satisfy a simple function interface
52
“What do classes support in Python in addition to objects?”
“In Python
53
“What does polymorphism enable?”
“Polymorphism enables multiple classes in a hierarchy to implement their own versions of a method.”
54
“What does this mean for classes?”
“This means that several classes can fulfill the same interface or abstract base class while providing different functionality.”
55
“What do you need when creating classes that represent app components?”
“When you create a bunch of classes that represent the components of an app
56
“What is the simplest approach to connecting objects?”
“The simplest approach to solve the problem mentioned above is to build and connect the objects with some helper functions.”
57
“What should you use when creating a base class with many subclasses?”
“If you want to create a base class and then make a bunch of subclasses based off of it
58
“What do you need for constructing objects generically?”
“You need a generic way to construct objects.”
59
“How would other languages solve generic object construction?”
“In other languages
60
“What is the problem with constructor polymorphism in Python?”
“The problem is that Python only allows for the single constructor method **init**
61
“What is the best way to solve generic object construction in Python?”
“The best way to solve the problem mentioned above is to use class method polymorphism.”
62
“What is class method polymorphism?”
“This is exactly like instance method polymorphism
63
“What can you use to create objects instead of **init**?”
“You can use the cls function to create objects instead of **init**.”
64
“What is the old way to initialize a parent class from a child class?”
“The old
65
“When does the direct call approach work and when does it break?”
“The above approach works fine for basic class hierarchies but breaks in many cases.”
66
“What can happen with multiple inheritance and direct **init** calls?”
“If a class is affected by multiple inheritance
67
“What is one problem with direct **init** calls in multiple inheritance?”
“One problem is that the **init** call order isn’t specified across all sub-classes.”
68
“What is diamond inheritance?”
“Another problem is diamond inheritance (when a subclass inherits from separate classes that have the same superclass somewhere in the hierarchy).”
69
“What does diamond inheritance cause?”
“Diamond inheritance causes the common superclass’s **init** method to run multiple times
70
“What does Python have to solve multiple inheritance problems?”
“To solve the problems mentioned above
71
“What does super ensure?”
“super ensures that common superclasses in diamond hierarchies are only run once.”
72
“What does the MRO define?”
“The MRO defines the ordering in which superclasses are initialized
73
“How does super handle diamond inheritance?”
“When you use super to deal with diamond inheritance
74
“What other benefit does super().**init** provide?”
“Besides making multiple inheritance robust
75
“How many parameters can the super function be called with?”
“The super function can also be called with 2 parameters: the first one is the type of the class whose MRO parent view you’re trying to access
76
“Are super parameters required for object instance initialization?”
“However
77
“What does Python’s compiler automatically provide for super?”
“Python’s compiler automatically provides the correct parameters (**class** and self) for you when super is called with zero arguments within a class definition.”
78
“When should you provide parameters to super?”
“The only time you should provide parameters to super is in situations where you need to access the specific functionality of the superclasses’s implementation from a child class (e.g. to wrap or use functionality).”
79
“What facilities does Python have for multiple inheritance?”
“Python is an object-oriented language with built-in facilities for making multiple inheritance traceable.”
80
“Should you use multiple inheritance?”
“However
81
“What should you think about writing instead of using multiple inheritance?”
“If you want the convenience and encapsulation that comes with multiple inheritance
82
“What don’t mix-in classes define or require?”
“Mix-in classes don’t define their own instance attributes nor require their **init** constructors to be called.”
83
“Why is writing mix-ins easy?”
“Writing mix-ins is easy because Python makes it easy to inspect the current state of any object
84
“What does dynamic inspection mean for mix-ins?”
“Dynamic inspection means you can write generic functionality just once
85
“What can mix-ins be composed and layered to do?”
“Mix-ins can be composed and layered to minimize repetitive code and maximize reuse.”
86
“What is one good feature of mix-ins?”
“One good feature of mix-ins is you can make their generic functions pluggable so behaviors can be overridden when required.”
87
“Can mix-ins be composed together?”
“Mix-ins can also be composed together.”
88
“What happens if a class inherits a mix-in higher up in its class structure?”
“If a class you use a mix-in on inherits that mix-in higher up in its class structure
89
“How will the resulting class behave?”
“The resulting class will behave the same way because of super.”
90
“How many types of visibility exist for a class’s attributes in Python?”
“In Python
91
“How can public attributes be accessed?”
“Public attributes can be accessed by anyone using the dot operator on an object.”
92
“How are private fields specified?”
“Private fields are specified by prefixing an attributes name with a double underscore.”
93
“How can private fields be accessed?”
“They can be accessed directly by methods of the containing class.”
94
“What happens when accessing private fields from outside the class?”
“However
95
“Do class methods have access to private attributes?”
“Class methods have access to private attributes too because they are declared in the surrounding class block.”
96
“Can a subclass access its parent’s private fields?”
“A subclass can’t access its parent’s private fields.”
97
“How is private attribute behavior implemented?”
“Private attribute behavior is implemented with a simple transformation of the attribute name.”
98
“What does the Python compiler do with private attribute access?”
“When the Python compiler sees private attribute access in methods like MyChildObject.get_private_field
99
“Can you access private attributes if you know the renaming scheme?”
“If you know the private renaming scheme
100
“Why doesn’t the syntax for private attributes enforce strict visibility?”
“The syntax for private attributes doesn’t enforce strict visibility because of the popular Python motto ‘We are all consenting adults here.’”
101
“What does the Python motto mean?”
“This means that people don’t need language to prevent them from doing what they want to do.”
102
“What choice do people have according to Python philosophy?”
“It’s peoples’ choice to extend functionality as they wish and take responsibility for the consequences of that risk.”
103
“What do Python programmers believe about being open?”
“Python programmers believe that the benefits of being open (permitting unplanned extensions of classes by default) outweighs the downsides.”
104
“What ability does hooking language features enable?”
“Beyond the above
105
“What question does this raise about private attribute access?”
“If you can do that
106
“How do Python programmers minimize damage from accessing internals?”
“To minimize damage from accessing internals unknowingly
107
“What are fields protected by a single underscore?”
“Fields protected by a single underscore (like _protected_field) are protected by convention
108
“What do many new Python programmers use private fields for?”
“A lot of programmers who are new to Python use private fields to indicate an internal API that shouldn’t be accessed by subclasses or externally.”
109
“Is using private fields for internal APIs the right approach?”
“The above approach is the wrong one.”
110
“What will inevitably happen with your class?”
“Inevitably someone will want to subclass your class to add new behavior or to work around deficiencies in existing methods.”
111
“What are you doing by choosing private attributes?”
“By choosing private attributes
112
“What will potential subclassers do?”
“Your potential subclassers will still access the private fields when they absolutely need to do so.”
113
“What will happen if the class hierarchy changes?”
“But if developers use the above strategy and the class hierarchy changes
114
“What is better in general for subclasses?”
“In general
115
“What should you document about protected fields?”
“Document each protected field and explain which fields are internal APIs available to subclasses and which should be left alone.”
116
“When should you seriously consider using private attributes?”
“The only time to seriously consider using private attributes is when you’re worried about naming conflicts with subclasses.”
117
“When does the naming conflict problem occur?”
“This problem occurs when a child class unwittingly defines an attribute that was already defined by its parent class.”
118
“What type of classes is this problem primarily a concern for?”
“The above problem is primarily a concern of classes that are part of a public API; the subclasses are out of your control
119
“When is such a conflict especially likely?”
“Such a conflict is especially likely with attribute names that are very common (like value).”
120
“How can you reduce the risk of naming conflicts?”
“To reduce the risk of this issue occurring
121
“What is a lot of programming in Python about?”
“A lot of programming in Python is defining classes that contain data and describing how such objects relate to each other.”
122
“Is every Python class a container?”
“Every Python class is a container of some kind
123
“What else does Python provide?”
“Python also provides built-in container types for managing data.”
124
“What is natural when designing classes for sequences?”
“When you’re designing classes for simple use cases like sequences
125
“What do you have to implement to make a class act like a sequence?”
“To make a Python class act like a sequence type
126
“What can you provide to make a class act like a sequence?”
“To make a class act like a sequence
127
“How can you access instances that implement **getitem**?”
“You can access instances of classes that implement **getitem** like you would a list.”
128
“What is the problem with just implementing **getitem**?”
“The problem with just implementing **getitem** is that it isn’t enough to provide all the sequence schematics you’d expect from a list instance.”
129
“What else do you have to implement along with **getitem**?”
“You have to implement the **len** function as well so that len will work on the sequence type you’re trying to create.”
130
“What other methods do you have to implement?”
“You also have to implement methods that provide functionality for the count and index methods.”
131
“Is defining custom container types easy?”
“Defining custom container types is fairly hard.”
132
“How can you avoid the challenges of custom containers?”
“To avoid all these challenges
133
“What happens when you forget to implement required methods?”
“When you subclass these abstract base classes and forget to implement these methods
134
“What does collections.abc provide when you implement required methods?”
“When you implement all the methods required by an abstract base class from collections.abc
135
“When are the benefits of abstract base classes even greater?”
“The benefits of using abstract base classes is even greater from complex container types like Set or MutableMapping
136
“What else does Python use special methods for?”
“Beyond the collections.abc module
137
“Are metaclasses commonly understood in practice?”
“Metaclasses are mentioned a lot
138
“What does the name metaclass imply?”
“The name metaclass implies a concept above and beyond a class.”
139
“What do metaclasses let you do?”
“Metaclasses let you intercept Python’s class statement and provide special behavior each time a class is defined.”
140
“What are similarly unknown Python features?”
“Similarly unknown are Python’s features for dynamically customizing attribute access.”
141
“What do these facilities provide?”
“Along with Python’s object-oriented constructs
142
“What is the problem with metaclasses and dynamic attribute access?”
“The problem with these tools is they come with a lot of negative consequences.”
143
“What can dynamic attribute access do?”
“Dynamic attribute access can override objects and cause unexpected side effects.”
144
“What can metaclasses cause?”
“Metaclasses can cause extremely strange behavior that is difficult for newcomers to work with.”
145
“What rule should you follow with metaclasses?”
“You need to follow the rule of least surprise and only use them to implement common idioms.”
146
“What might new Python programmers try to implement?”
“New Python programmers might try to implement explicit getter and setter methods in their classes.”
147
“How are getters and setters described?”
“Using getters and setters is simple
148
“What are they particularly clumsy for?”
“They’re also particularly clumsy for operations like incrementing in place.”
149
“What do these utility methods make easier?”
“However
150
“What does this in turn make easier?”
“This
151
“Why are these important goals?”
“These are important goals for designing a class to ensure that you don’t break callers as the class evolves over time.”
152
“Do you need to implement explicit getter or setter methods in Python?”
“In Python
153
“How should you start your implementations?”
“You should start your implementations with simple public attributes.”
154
“What can you do if you need special behavior when an attribute is set later?”
“If you need special behavior when an attribute is set later
155
“What must match for @property to work properly?”
“In order for code like this to work properly
156
“What does specifying a setter on a property enable?”
“Specifying a setter on a property enables you to perform type checking and validation on values passed to the class.”
157
“What can you use @property for regarding parent classes?”
“You can use @property to make attributes from parent classes immutable.”
158
“What should you be sure about when using @property methods?”
“When you use @property methods to implement getters and setters
159
“What can setting other attributes in getter property methods lead to?”
“Setting other attributes in getter property methods can lead to very strange behavior.”
160
“What is the best policy for @property.setter methods?”
“The best policy is to modify only related object state in @property.setter methods.”
161
“What side effects should you avoid in @property methods?”
“Be sure to avoid other side effects that the caller might not expect beyond the object (e.g. importing modules dynamically
162
“What will users of a class expect about its attributes?”
“Users of a class will expect its attributes to be like any other Python object: quick and easy.”
163
“When should you use normal methods?”
“Use normal methods to do anything more complex or slow.”
164
“What is the biggest shortcoming of @property?”
“The biggest shortcoming of @property is that the methods for an attribute can only be shared by subclasses.”
165
“Can unrelated classes share the same @property implementation?”
“Unrelated classes can’t share the same implementation.”
166
“What does Python support for reusable property logic?”
“Python also supports descriptors that enable re-usable property logic and many other use cases.”
167
“What does the @property decorator make easy?”
“The @property decorator makes it easy for simple access of an instance’s attribute to act smarter.”
168
“What is one advanced but common use of @property?”
“One advanced but common use of @property is a former simple numerical attribute into an on-the-fly calculation.”
169
“Why is this very helpful?”
“This is very helpful because it lets you migrate all existing usage of a class to have new behaviors without requiring any of the call sites to be rewritten (which is especially important if there’s calling code you don’t control).”
170
“What does @property provide for interfaces?”
”@property also provides an important stopgap for improving interfaces over time.”
171
“What does @property let you do regarding data models?”
“One good thing about @property is it lets you make incremental progress towards a better data model over time.”
172
“What example is given of poor initial implementation?”
“For example
173
“What kind of tool is @property?”
”@property is a tool to address problems you’ll come across in real-world code.”
174
“Should you overuse @property?”
“Don’t overuse it.”
175
“When should you refactor your classes instead of using @property?”
“If you find yourself repeatedly extending @property methods
176
“Can @property methods be reused for unrelated classes?”
“They also can’t be reused for unrelated classes.”
177
“What is the big problem with @property?”
“The big problem with @property is reuse.”
178
“Can @property methods be reused for multiple attributes of the same class?”
“The methods it decorates can’t be reused for multiple attributes of the same class.”
179
“Can they be reused by unrelated classes?”
“They also can’t be reused by unrelated classes.”
180
“What do you have to do if you want to reuse @property methods?”
“If you want to reuse @property methods
181
“What is a better way to solve the reuse problem?”
“A better way to solve the problem mentioned above is to use a descriptor.”
182
“What does the descriptor protocol define?”
“The descriptor protocol defines how attribute access is interpreted by the language.”
183
“What can a descriptor class provide?”
“A descriptor class can provide **get** and **set** methods that let you reuse @property methods without boilerplate.”
184
“Why are descriptors better than mix-ins for this purpose?”
“For this purpose
185
“How is setting a property on a descriptor class interpreted?”
“When you set a property on a class that implements the descriptor protocol
186
“How is retrieving a property of a descriptor class interpreted?”
“Retrieving a property of a class that implements the descriptor protocol is interpreted as Class.**dict**[‘var_name’].**get**(var_name
187
“What method drives descriptor behavior?”
“The **getattribute** method of the object drives this behavior.”
188
“What does Python do when an attribute can’t be found?”
“In short
189
“What does Python assume about class attributes with **get** and **set**?”
“If this class attribute is an object that has **get** and **set** methods
190
“What is the wrong approach for implementing descriptors?”
“Creating a class that implements the descriptor protocol and implements **get** and **set** is the wrong approach and it results in broken behavior.”
191
“When does the naive descriptor approach cause problems?”
“When you access multiple attributes on an object instance that uses this approach
192
“Why does this happen?”
“This happens because a single object instance is shared across object instances of classes that use that kind of object.”
193
“When is the descriptor object instance constructed?”
“The object instance is constructed once in the program’s lifetime
194
“How do you solve the shared instance problem?”
“To solve the problem that’s mentioned above
195
“How can you do this?”
“You can do this by saving the per-instance state in a dictionary.”
196
“Does the dictionary solution work well?”
“The solution mentioned above is simple and it works well
197
“Why does the dictionary solution leak memory?”
“The dictionary you use to keep track of a state holds a reference to every class that implements the descriptor protocol ever passed to **set** over the lifetime of the program.”
198
“What does this cause?”
“This causes instances to never have their reference count go to zero
199
“How can you fix the memory leak?”
“To fix this
200
“What does the weakref module provide?”
“It provides a special class called WeakKeyDictionary that can take the place of the dictionary that’s used to hold all the values.”
201
“What does WeakKeyDictionary do?”
“WeakKeyDictionary removes object instances from its set of items when it knows it’s holding the instance’s last remaining reference in the program.”
202
“What bookkeeping does Python do?”
“Python does the bookkeeping and makes sure the values dictionary will be empty when the object instances aren’t in use.”
203
“What do Python’s object hooks make easy?”
“Python’s object hooks make it easy to write generic code for gluing systems together.”
204
“Do you need to specify object structure in Python?”
“You don’t need to specify the structure of certain objects in Python (e.g. you don’t need to specify the schema in your Python classes); you can do this generically.”
205
“What makes the dynamic behavior possible?”
“The dynamic behavior mentioned above is possible because of the **getattr** special method.”
206
“When is **getattr** called?”
“If a class defines **getattr**
207
“Can plain instance attributes do what **getattr** does?”
“Plain instance attributes
208
“What do you need to use to avoid infinite recursion with **getattr**?”
“You need to use super().**getattr**() to use the superclass’s implementation of **getattr** when you define a subclass’s implementation of **getattr** in order to fetch the real property value and avoid infinite recursion.”
209
“What is **getattr** really helpful for?”
“The above behavior is really helpful for use cases like lazily accessing schema-less data.”
210
“How many times does **getattr** run?”
”**getattr** runs once to do the hard work of loading a property; all subsequent accesses retrieve the existing results.”
211
“What won’t **getattr** let you do?”
“The **getattr** hook won’t let you check to see if a record is valid (or a transaction-type operation is valid) because it uses the object’s instance dictionary as the fast path for existing attributes.”
212
“What can you use for more advanced use cases?”
“To enable the more advanced use case mentioned above
213
“When is **getattribute** called?”
“This special method is called every time an attribute is accessed on an object
214
“What does **getattribute** let you do?”
“This lets you do things like check global transaction state on every property access.”
215
“What can this operation incur?”
“That operation can incur significant overhead and negatively impact performance
216
“What should you raise if a dynamically accessed attribute shouldn’t exist?”
“If you implement **getattribute** or **getattr** and a dynamically accessed attribute shouldn’t exist
217
“What does Python code implementing generic functionality often rely on?”
“Python code implementing generic functionality often relies on the hasattr built-in function to determine when properties exist
218
“Where do hasattr and getattr look first?”
“These functions all look in the instance dictionary for an attribute name before calling **getattr**.”
219
“When is **getattribute** called for hasattr or getattr?”
“Classes that implement **getattribute** have that method called each time hasattr or getattr is used for an instance.”
220
“What does **setattr** let you intercept?”
“The **setattr** object hook lets you intercept arbitrary attribute assignments.”
221
“Why is there no need for two separate methods with **setattr**?”
“Unlike retrieving an attribute with **getattr** or **getattribute**
222
“What is the problem with **getattribute** and **setattr**?”
“The problem with **getattribute** and **setattr** is that they’re called on every attribute access for an object
223
“What happens if you access the instance dictionary in **getattribute**?”
“If you try to access an object’s instance dictionary when you implement the **getattribute** method
224
“Why does this recursion happen?”
“This happens because **getattribute** accessed the dictionary
225
“How do you avoid the recursion problem?”
“The solution to the problem mentioned above is to use super().**getattribute** method to fetch values from the instance attribute dictionary.”
226
“What does this avoid?”
“This avoids the recursion.”
227
“What do **setattr** methods need to use?”
”**setattr** methods that modify attributes on an object also need to use super().**setattr** accordingly.”
228
“What is one of the simplest applications of metaclasses?”
“One of the simplest applications of metaclasses is verifying that a class was defined correctly.”
229
“When might you want to enforce style or require overriding methods?”
“When you’re building a complex class hierarchy
230
“What do metaclasses enable for validation?”
“Metaclasses enable these use cases by providing a reliable way to run your validation code each time a new sub-class is defined.”
231
“When does a class’s validation code often run?”
“Often a class’s validation code runs in the **init** method
232
“When can metaclasses for validation raise errors?”
“Using metaclasses for validation can raise errors much earlier
233
“What is important to understand about metaclass action?”
“It’s important to understand the metaclass action for standard objects.”
234
“How is a metaclass defined?”
“A metaclass is defined by inheriting from type.”
235
“What does a metaclass receive in the default case?”
“In the default case
236
“What can you do in the **new** method?”
“There
237
“What does the metaclass have access to?”
“The metaclass has access to the name of the class
238
“Do all classes inherit from object?”
“All classes inherit from object
239
“How can you add validation to a metaclass?”
“You can add functionality to the Meta.**new** method in order to validate all of the parameters of an associated class before it’s defined.”
240
“Should you apply validation to the base class?”
“It’s important not to apply the same validation discussed above to the base class.”
241
“What happens if a class violates its metaclass’s validation methods?”
“If someone defines a class that violates its metaclass’s validation methods
242
“What does this mean for program execution?”
“This means that the program won’t be able to start running when they define a class (unless it’s a dynamically imported module).”
243
“Does metaclass machinery seem like a lot for a simple task?”
“This seems like quite a lot of machinery to get Python to accomplish a simple class.”
244
“What did Python 3.6 introduce?”
“Luckily
245
“What does **init_subclass** let you achieve?”
“This method lets you achieve the same behavior without using metaclasses.”
246
“What is another problem with standard Python metaclass machinery?”
“Another problem with the standard Python metaclass machinery is that you can only specify a single metaclass per class definition.”
247
“What is one solution to the single metaclass problem?”
“One solution to the problem mentioned above is to create a complex hierarchy of metaclass type definitions to layer validation.”
248
“What is the problem with that approach?”
“The problem with the approach mentioned above is it ruins composability
249
“What happens if you want to apply metaclass validation logic to another hierarchy?”
“If you want to apply class validation logic you create with metaclasses to another class heirarchy
250
“Can **init_subclass** solve the composability problem?”
“The **init_subclass** special class method can also be used to solve this problem.”
251
“How can **init_subclass** be defined?”
“It can be defined by multiple levels of class hierarchy as long as super is used to call any parent or sibling **init_subclass** definitions.”
252
“Is **init_subclass** compatible?”
“It’s even compatible.”
253
“Can you inherit from multiple classes to define a new class?”
“You can inherit from multiple classes to define a new class.”
254
“What should both classes call?”
“Both classes should call super().**init_subclass**()
255
“Can you use **init_subclass** in complex cases?”
“You can even use **init_subclass** in complex cases like diamond inheritance.”
256
“What is another common use of metaclasses?”
“Another common use of metaclasses is to automatically register types in a program.”
257
“What is registration useful for?”
“Registration is useful for doing reverse lookups
258
“Is creating a registry of classes easy?”
“Creating a registry of classes to make sure you’re performing operations on the right class is error-prone and challenging for beginners.”
259
“Can the same omission happen with class decorators?”
“The same omission can happen with class decorators.”
260
“How can you create a successful class registry?”
“You can create a successful class registry with metaclasses because they enable you to intercept the class statement when subclasses are defined.”
261
“What is an even better approach than metaclasses for registration?”
“An even better approach is to use the **init_subclass** special class method.”
262
“What does this simplified syntax do?”
“This simplified syntax
263
“What does it make more approachable?”
“It also makes it more approachable to beginners who may be confused by the metaclass syntax.”
264
“What can you ensure by using **init_subclass** for registration?”
“By using **init_subclass** (or metaclasses) for class registration
265
“What does this work well for?”
“This works well for serialization
266
“What useful feature do metaclasses enable for properties?”
“One useful feature enabled by metaclasses is the ability to modify or annotate properties after a class is defined but before the class is actually used.”
267
“What is this approach commonly used with?”
“This approach is commonly used with descriptors to give them more introspection into how they’re being used within their containing class.”
268
“What might seem convenient to build?”
“It might seem convenient to build a class hierarchy that symbolizes a single thing (e.g. a database row
269
“What is one problem with this approach?”
“One problem with this approach is that sometimes the classes representing the thing is vague and they won’t know specifically what it’s assigned to.”
270
“How can you eliminate the redundancy?”
“To eliminate the redundancy above
271
“What do metaclasses let you hook?”
“Metaclasses let you hook the class statement directly and take action as soon as the class body is finished.”
272
“What is the problem with the metaclass approach for this?”
“The problem with the above approach is that you can’t use more specific base classes unless you inherit from the metaclass and the first
273
“What is the solution to this problem?”
“The solution to the problem mentioned above is to use the **set_name** special method for descriptors.”
274
“When was **set_name** introduced?”
“This method
275
“What parameters does **set_name** receive?”
“It receives as parameters the owning class that contains the descriptor instance and the attribute name to which the descriptor instance was assigned.”
276
“Do metaclasses allow you to customize class creation in multiple ways?”
“Although metaclasses allow you to customize class creation in multiple ways
277
“What is the problem with applying decorator to a subclass?”
“The problem with applying decorator to a subclass is that you have to rewrite all the methods that you want to decorate.”
278
“What else is problematic about this?”
“It’s also redundant boilerplate thats hard to read and it’s error prone.”
279
“What happens if a new method is added to the superclass?”
“Also
280
“What is one way to solve this problem?”
“One way to solve the problem mentioned above is to use a metaclass to automatically decorate all the methods of a class.”
281
“What is the problem with using a metaclass for auto-decoration?”
“The problem with the solution mentioned above is you can’t use the metaclass you create with superclasses if they already have a metaclass.”
282
“Why does it fail?”
“It fails because your metaclass doesn’t inherit from the superclass’s metaclass.”
283
“Can you solve this with metaclass inheritance?”
“In theory
284
“Will this approach work in all cases?”
“However
285
“What does the metaclass approach put on the class?”
“The metaclass approach puts too many constraints on the class that’s being modified.”
286
“What is the best solution to this problem?”
“The best solution to this problem is to use class decorators.”
287
“How do class decorators work?”
“Class decorators work like normal function decorators: you apply them with the @ symbol prefixing a function before the class decoration.”
288
“What is the function expected to do?”
“The function is expected to modify or recreate the class accordingly and return it.”
289
“Do class decorators work when the class already has a metaclass?”
“Class decorators also work when the class being decorated already has a metaclass.”
290
“When are class decorators the best tool?”
“When you’re looking for composable ways to extend classes
291
“What does concurrency let a computer do?”
“Concurrency lets a computer do many things seemingly at the same time.”
292
“What does parallelism involve?”
“Parallelism involves actually doing many different things at the same.”
293
“What can a computer with multiple CPU cores do?”
“A computer with multiple CPU cores can execute multiple programs simultaneously.”
294
“What does each CPU core do?”
“Each CPU core runs the instructions of a separate program
295
“What is concurrency within a single program?”
“Within a single program
296
“What do concurrent programs enable?”
“Concurrent programs enable many distinct paths of execution (including separate streams of I/O) to make forward progress in a way that seems to be simultaneous and independent.”
297
“What is the key difference between parallelism and concurrency?”
“The key difference between parallelism and concurrency is speedup.”
298
“What happens when two distinct paths execute in parallel?”
“When two distinct paths of execution in a program make forward progress in parallel
299
“How do concurrent programs compare to parallel programs regarding speedup?”
“In contrast
300
“Does Python make concurrent programming easy?”
“Python makes it easy to write concurrent programs in a variety of the styles.”
301
“How much concurrency do threads vs coroutines support?”
“Threads support a relatively small amount of concurrency
302
“How can Python be used for parallel work?”
“Python can also be used to do parallel work through system calls
303
“Is it easy to make concurrent Python code truly run in parallel?”
“But it can be very difficult to make concurrent Python code truly run in parallel.”
304
“Why is it important to understand Python’s concurrency options?”
“It’s important to understand how to best utilize Python in these different situations.”
305
“What libraries does Python have for child processes?”
“Python has battle-hardened libraries for running and managing child processes.”
306
“What does this make Python great for?”
“These libraries make Python a great language for gluing together other tools
307
“When is rewriting shell scripts in Python a natural choice?”
“When existing shell scripts get complicated
308
“Can child processes started by Python run in parallel?”
“Child processes started by Python are able to run in parallel
309
“How can Python be used even if it’s CPU bound?”
“Although Python may be CPU bound
310
“What is the best choice for managing child processes?”
“Python has many ways to run subprocesses (e.g. os.popen
311
“Is running a child process with subprocess simple?”
“Running a child process with subprocess is simple.”
312
“How do child processes run relative to their parent?”
“Child process run independently from their parent process (the Python interpreter).”
313
“What can you do if you use Popen instead of run?”
“If you use the Popen class to create a subprocess instead of the run function
314
“What does decoupling child processes do?”
“Decoupling the child process from the parent frees up the parent process to run many child processes.”
315
“What can the communicate method do?”
“The communicate method can let processes execute and then terminate them.”
316
“Can you pipe data into subprocesses?”
“You can also pipe data from a Python program into a subprocess and retrieve its output.”
317
“What does piping data allow you to do?”
“This allows you to utilize many other programs to do work in parallel.”
318
“Do multiple child processes run in parallel?”
“Multiple child processes run in parallel.”
319
“What can you create with child processes?”
“You can create chains of parallel processes (just like UNIX pipelines) connecting the output of one child process to another.”
320
“What should you be careful about with process chains?”
“If you create a chain of parallel processes
321
“When does I/O happen in process chains?”
“The I/O between multiple child processes in a process chain happens automatically once they are started.”
322
“How can you handle child processes that never finish?”
“If you’re worried about child processes never finishing or blocking other input or output pipes
323
“What does the timeout parameter do?”
“This causes an exception to be raised if the child process hasn’t finished within the time period.”
324
“What is the standard implementation of Python called?”
“The standard implementation of Python is called CPython.”
325
“How does CPython run a Python program?”
“CPython run a Python program in 2 steps: first
326
“What must be maintained while a Python program executes?”
“The bytecode interpreter has state and that must be maintained and coherent while the program executes.”
327
“What mechanism does CPython use to enforce coherence?”
“CPython enforced coherence with a mechanism called the global interpreter lock.”
328
“What is the GIL?”
“The GIL is essentially a mutual-exclusion lock (mutex) that prevents CPython from being executed by preemptive multithreading (where one thread take control of a program by interrupting another thread.”
329
“What could thread interruption corrupt?”
“Such an interruption could corrupt the interpreter state (e.g. garbage collection
330
“What does the GIL prevent and ensure?”
“The GIL prevents these interruptions and ensures that every bytecode instruction works correctly with the CPython implementation and its C-extension modules.”
331
“What is an important negative side effect of the GIL?”
“The GIL has an important negative side effect.”
332
“How do C++ or Java programs utilize multiple CPU cores?”
“With programs written in languages like C++ or Java
333
“Does Python support multiple threads of execution?”
“Python supports multiple threads of execution
334
“What happens when you try to use threads for parallel computation?”
“When you try to use threads to do parallel computation and speed up your Python programs
335
“Why does using multiple threads make sense in other languages?”
“Using multiple threads in other languages makes sense because you can take advantage of multiple CPU cores.”
336
“How do threads perform compared to serial execution in Python?”
“Using threads takes even longer to run a function than running it in serial.”
337
“What speedup might you expect in other languages with threads?”
“With one thread per number
338
“What speedup might you expect on a dual-core machine?”
“You might only expect a 2x speedup on a dualcore machine.”
339
“Can threads result in worse performance in Python?”
“It might result in worse performance when there are multiple CPUs to utilize.”
340
“What does this demonstrate about the GIL?”
“This demonstrates the effect of the GIL (e.g. locking contention and scheduling overhead) on programs running in the standard CPython interpreter.”
341
“Are there ways to get CPython to utilize multiple cores?”
“There are ways to get CPython to utilize multiple cores
342
“What is one reason Python supports threads?”
“One reason Python supports threads is that using multiple threads makes it easy for a program to seem like it’s doing multiple things at the same time.”
343
“Is managing multiple tasks easy to implement yourself?”
“Managing multiple tasks is difficult to implement yourself.”
344
“How can you use threads for concurrency in Python?”
“You can use threads to make Python run your threads concurrently.”
345
“Why does this work despite the GIL?”
“This works because Python ensures a level of fairness between threads
346
“What is another reason Python supports threads?”
“Another reason Python supports threads is to deal with blocking I/O
347
“What does a Python program use system calls for?”
“A Python program uses system calls to ask the computer’s operating system to interact with the external environment on its behalf.”
348
“What does blocking I/O include?”
“Blocking I/O includes things like reading and writing files
349
“How do threads help with blocking I/O?”
“Threads help to handle blocking I/O by insulating a program from the time it takes for the operating system to respond to requests.”
350
“How is using threads in Python in practice?”
“Using threads in Python is awful in practice.”
351
“When should you consider moving system calls to threads?”
“When you find yourself needing to do blocking I/O and computation simultaneously
352
“Will system calls in threads run in parallel?”
“System calls that use Python threads will run in parallel even though they’re limited by the GIL.”
353
“Does the GIL affect system calls?”
“The GIL prevents Python code from running in parallel
354
“How does this work with the GIL and system calls?”
“This works because Python threads release the GIL just before they make system calls
355
“Are there alternatives to threads for blocking I/O?”
“There are many other ways to deal with blocking I/O besides using threads (e.g. the asyncio module)
356
“What might using alternatives to threads require?”
“But using these options might require extra work in refactoring your code to fit a different model of execution.”
357
“What is the simplest way to do blocking I/O in parallel?”
“Using threads is the simplest way to do blocking I/O in parallel with minimal changes in your code.”
358
“What do many new Python programmers assume about the GIL?”
“After many new Python programmers learn about the global interpreter lock (GIL)
359
“What do they think the GIL does for data structures?”
“They think that since the GIL is preventing threads from running in parallel
360
“Does the GIL prevent data structure corruption?”
“The GIL doesn’t prevent corruption of a program’s data structures.”
361
“When can thread operations be interrupted?”
“A thread’s operations on data structures can be interrupted between any two bytecode instructions in the Python interpreter.”
362
“What does this make dangerous?”
“This makes it dangerous to access the same objects from multiple threads simultaneously and corrupt your program.”
363
“How does the Python interpreter enforce fairness between threads?”
“The Python interpreter enforces fairness between all of the threads that are executing to ensure they get equal processing time.”
364
“How does Python accomplish this fairness?”
“To accomplish this
365
“Do you know when Python will suspend a thread?”
“Unfortunately
366
“What does Python include to prevent data races?”
“To prevent data races and other forms of data structure corruption
367
“What is the simplest and most useful threading tool?”
“The simplest and most useful of them is the Lock class
368
“What does the Lock class do?”
“This makes it so only one thread can acquire the lock at a time.”
369
“What do concurrent Python programs often need?”
“Python programs that do a lot of things concurrently often need to coordinate their work.”
370
“What is one of the most useful arrangements for coordination?”
“One of the most useful arrangements to do this is a pipeline of functions.”
371
“How does a pipeline work?”
“A pipeline works like an assembly line.”
372
“What do pipelines have?”
“Pipelines have many phases in serial
373
“What is constantly added to pipelines?”
“New pieces of work are constantly added to the beginning of the pipeline.”
374
“Do the functions in a pipeline operate concurrently?”
“The functions operate concurrently.”
375
“How does work move forward in a pipeline?”
“The work moves forward as each function completes until there are no phases remaining.”
376
“What type of work is the pipeline process good for?”
“This process is really good for work that includes blocking I/O or subprocesses (activities that are easy to parallelize using Python).”
377
“Are pipelines bad to implement?”
“Pipelines aren’t bad; it’s hard to build a good producer-consumer queue yourself.”
378
“What does the Queue class provide?”
“The Queue class from the queue module provides all the functionally you need to create a producer-consumer queue.”
379
“What does Queue’s get method do?”
“Queue’s get method blocks until new data is available.”
380
“What does Queue let you specify?”
“Queue lets you specify the maximum amount of pending work to allow between two phases.”
381
“What does the buffer size do?”
“This buffer size calls put to block when the queue is already full.”
382
“How can you track progress with Queue?”
“You can use the task_done method of the Queue class to track the progress of what you’re doing.”
383
“What does task_done let you do?”
“This lets you wait for a queue to be empty and gets rid of the need to poll the last phase of the pipeline (you can use done_queue to do this).”
384
“What is a sentinel item?”
“A sentinel item is an item of a queue that indicates that there is no items after it.”
385
“Can you extend the Queue approach?”
“You can extend this approach to use multiple worker threads per phase
386
“Does Queue work well for all situations?”
“Queue works well in the case of a linear pipeline
387
“What happens as the scope of a program grows?”
“As the scope of a program grows
388
“What is one of the most difficult parts of programming?”
“Dealing with the expand requirements in a way that maintains clarity
389
“What is one of the hardest changes to handle?”
“One of the hardest parts of change to handle is moving from a single-threaded program to one that needs multiple concurrent lines of execution.”
390
“How can you speed up something done in serial?”
“A way to speed up something done in serial is to do the I/O in parallel.”
391
“What is fanning-out?”
“Fanning-out is the process of spawning a concurrent line of execution for each unit of work.”
392
“What is fanning-in?”
“Fanning-in is the process of waiting for all those concurrent units to finish before moving on to the next phase of the coordinated process.”
393
“What does Python provide for fan-outs and fan-ins?”
“Python provides a lot of tools for fan-outs and fan-ins with various trade-offs.”
394
“What should you understand about concurrency approaches?”
“You should understand the pros and cons of each approach and choose the best approach for the job
395
“What is the first natural tool for parallel I/O?”
“Threads are the first natural tool to use in order to do parallel I/O in Python.”
396
“What downsides do threads have for fan-out?”
“However
397
“What is the first drawback of thread fan-out?”
“Thread instances require special tools to coordinate with each other safely. This makes the code that uses threads harder to reason about than the procedural
398
“What is the second drawback of thread fan-out?”
“Threads require a lot of memory (about 8 MB per executing thread). On many computers
399
“What is the third drawback of thread fan-out?”
“Starting a thread is costly
400
“What is the fourth drawback of thread fan-out?”
“It would be difficult to debug if something went wrong in some cases.”
401
“How does the Thread class handle exceptions?”
“Code that creates a Thread (and possibly calls join) can raise an exception and be unaffected because the Thread class will independently catch any exceptions that are raised by the target function and then write their traceback to sys.stderr.”
402
“Are exceptions re-raised to the caller that started the thread?”
“Such exceptions are never re-raised to the caller that started the thread in the first place.”
403
“Are threads the solution for constantly creating concurrent functions?”
“Because of all these issues
404
“Does Python provide other solutions besides threads?”
“Python provides other solutions that are a better fit.”
405
“What is the next approach for parallel I/O after threads?”
“The next approach for using threads to do parallel I/O is to implement a threaded pipeline using Queue from the queue module.”
406
“What is the general approach with Queue for parallelism?”
“The general approach is to create a fixed number of worker threads and have them do parallelized I/O as needed.”
407
“What does using fixed worker threads keep under control?”
“This keeps your research usage under control and eliminates the overhead of frequently starting new threads.”
408
“What is one Queue-based approach for I/O?”
“One approach is to consume items from an input queue
409
“What can threads in the Queue design do?”
“The threads used by this design can run concurrently
410
“How do queues interact in the Queue design?”
“The next step in the design discussed above can interact with these queues to request state transition decisions and receive corresponding responses.”
411
“What causes fan-out and fan-in with queues?”
“Adding items to the input queue causes fan-out
412
“What can be single-threaded in the Queue design?”
“You can make the individual objects in your program that represent important pieces of it can be single-threaded instead of requiring Lock instances for synchronization if you include them in the main processing/coordination part of your program.”
413
“What is the first remaining problem with Queue?”
“You might have to add extra support classes to make code easier to read
414
“What is the second remaining problem with Queue?”
“You have to specify the amount of potential parallelism (the number of running threads) upfront based on your expectations of the workload instead of having the system scale up parallelism automatically as needed.”
415
“What is the third remaining problem with Queue?”
“In order to enable debugging
416
“What is the biggest problem with Queue?”
“The biggest problem with the code is refactoring it if the requirements change again.”
417
“Is Queue better than Thread instances alone?”
“Using Queue is a better approach than using Thread instances on their own to resolve fan-out and fan-in problems
418
“Are there better tools than Queue in Python?”
“There are better tools in Python.”
419
“What module includes ThreadPoolExecutor?”
“Python includes the concurrent.futures module
420
“What does ThreadPoolExecutor combine?”
“It combines the best of the Thread and Queue approaches to solving the parallel I/O problems.”
421
“How do you use ThreadPoolExecutor for fan-out?”
“Another approach to using threads is to fan-out by submitting a function to an executor that will be run in a separate thread.”
422
“How can you wait on ThreadPoolExecutor results?”
“You can wait on the result of all the tasks to fan-in.”
423
“Can threads be allocated in advance with ThreadPoolExecutor?”
“The threads that are used for the executor can be allocated in advance
424
“How can you prevent memory blow-up with ThreadPoolExecutor?”
“You can also specify the maximum number of threads to use for the pool using max_workers to prevent memory blow-up issues.”
425
“What is one of the best features of ThreadPoolExecutor?”
“One of the best features of ThreadPoolExecutor is that it automatically propagates exceptions back the the caller when the result method is called on the Future instance by using the submit method.”
426
“What is the big problem with ThreadPoolExecutor?”
“The big problem with ThreadPoolExecutor is that it provides a limited amount of I/O parallelism.”
427
“Can ThreadPoolExecutor have scaling problems?”
“Even if you use a large max_workers parameter
428
“When is ThreadPoolExecutor a good choice?”
“ThreadPoolExecutor is a good choice for situations where there is no asynchronous solution (e.g. file I/O)
429
“What do all threading approaches fall short of?”
“All of the other previous threading approaches fall short of their ability to handle thousands of simultaneously concurrent functions.”
430
“How does Python address highly concurrent I/O needs?”
“Python addresses the need for highly concurrent I/O with coroutines.”
431
“What do coroutines let you have?”
“Coroutines let you have a very large number of seemingly simultaneous functions in your Python program.”
432
“How are coroutines implemented?”
“They’re implemented using the async and await keywords along with the same infrastructure that powers generators.”
433
“What is the cost of starting a coroutine?”
“The cost of starting a coroutine is a function call.”
434
“How much memory does an active coroutine use?”
“Once a coroutine is active
435
“How are coroutines like threads?”
“Like threads
436
“What is the difference between coroutines and threads?”
“The difference is that coroutines pause at each await expression and resume executing an async function after the pending awaitable is resolved (similar to how yield behaves in generators).”
437
“How do async functions seem to run?”
“Many separate async functions advanced in lockstep all seem to run simultaneously
438
“What advantages do coroutines have over threads?”
“However
439
“What powers coroutines?”
“The magical mechanism powering coroutines is the event loop
440
“What does calling an async function return?”
“Calling an async function doesn’t immediately return that function. It returns a coroutine instance that can be used with an await expression at a later time.”
441
“What is this similar to with generators?”
“This is similar to generator functions that use yield return a generator instance when they’re called instead of executing immediately.”
442
“What mechanism causes fan-out with coroutines?”
“When used with threading
443
“What causes fan-in in asyncio?”
“The gather function from the asyncio library causes fan-in.”
444
“Do you need locks if all code runs in a single thread with asyncio?”
“If you convert code to use asyncio
445
“How does I/O become parallelized with asyncio?”
“The I/O becomes parallelized as part of the event loop that’s provided by asyncio.”
446
“Can you use the debugger with coroutines?”
“You can actually use the debugger with coroutines.”
447
“How can you port code to asyncio?”
“You can port your code to asyncio by adding async and await keywords to all the existing call sites.”
448
“What is one of the plusses of coroutines?”
“One of the plusses of coroutines is that they decouple your code’s instructions for the external environment (i.e. I/O) from the implementation that caries out your wishes (i.e. the event loop).”
449
“What do coroutines let you focus on?”
“They let you focus on the logic of what you’re trying to do instead of wasting time trying to figure out how you’re going to accomplish your goals concurrently.”
450
“Does it seem hard to port an existing codebase to coroutines?”
“Once you understand the advantage of coroutines
451
“Is Python’s async support well integrated?”
“Luckily
452
“Is it straightforward to move to coroutines and async I/O?”
“This makes it straightforward to move code that does threaded
453
“What is a common way to build client/server systems?”
“A common way to build a client/server system is to use blocking I/O and threads.”
454
“Do all Python features have async equivalents?”
“A lot of features in Python don’t have an asynchronous equivalent.”
455
“What async versions are currently missing?”
“There are currently not any asynchronous versions of next and iter; you have to await on the **anext** and **aiter** methods directly.”
456
“Is there an async version of yield from?”
“There also isn’t an asynchronous version of yield from
457
“Will these async features be available in the future?”
“These features will likely be available in the future.”
458
“What does the asyncio module provide?”
“The asyncio module provides several helper functions and shortens the amount of boilerplate required to write a server like this.”
459
“What features does asyncio have for adoption?”
“The asyncio module has a lot of I/O
460
“Is it feasible to modify large programs to use async functions?”
“It’s rarely feasible to modify large programs to use asynchronous functions.”
461
“What do you need to do for large programs instead?”
“Instead
462
“What does your codebase need to be able to do during migration?”
“In order to do that
463
“What does this mean practically for threads and coroutines?”
“Practically
464
“Does asyncio include facilities for interoperability?”
“Luckily
465
“How many approaches exist for converting threaded code to asyncio?”
“There are 2 approaches for incrementally converting threaded code to use asyncio and coroutines: top-down and bottom-up.”
466
“What does top-down conversion mean?”
“Top-down means starting at the high point of the codebase (like the main entry points) and working down to individual functions and classes.”
467
“When can the top-down approach be useful?”
“This approach can be useful when you maintain a lot of common modules that you use across many different programs.”
468
“What can you do by porting entry points first?”
“By porting the entry points first
469
“What is step 1 of the top-down approach?”
“Change the top function to use async def instead of def.”
470
“What is step 2 of the top-down approach?”
“Wrap all of its calls that do I/O (potentially blocking the event loop) to use asyncio.run_in_executor instead.”
471
“What is step 3 of the top-down approach?”
“Ensure the resources or callbacks used by run_in_executor invocations are properly synchronized (i.e. using Lock or async.run_coroutine_threadsafe).”
472
“What is step 4 of the top-down approach?”
“Try to eliminate get_event_loop and run_in_executor calls by moving downward through the call hierarchy and converting intermediate functions and methods to coroutines (following the first 3 steps).”
473
“What does the run_in_executor method do?”
“The run_in_executor method instructs the event loop to run a function using a specified ThreadPoolExecutor or the default executor instance when the first parameter is None.”
474
“How can a coroutine fan out and fan in with run_in_executor?”
“One way to have a coroutine fan out and fan in is to make multiple calls to the run_in_executor method without any corresponding await expressions
475
“What does asyncio.run_coroutine_threadsafe do?”
“The asyncio.run_coroutine_threadsafe function allows regular worker threads to call a coroutine and have it execute in the event loop from the main thread (or from any other thread
476
“What is the bottom-up approach to adopting coroutines?”
“The bottom-up approach to adopting coroutines has four steps that are similar to the steps of the top-down style
477
“What is step 1 of the bottom-up approach?”
“Create a new asynchronous coroutine version of each leaf function that you’re trying to port.”
478
“What is step 2 of the bottom-up approach?”
“Change the existing synchronous functions so they call the coroutine versions and then run the event loop instead of implementing any real behavior.”
479
“What is step 3 of the bottom-up approach?”
“Move up a level of the call hierarchy
480
“What is step 4 of the bottom-up approach?”
“Delete synchronous wrappers around coroutines created in step 2 as you stop requiring them to glue the pieces together.”
481
“Is this all you could do with asyncio adoption?”
“This is a good start for adopting asyncio
482
“What might block the asyncio event loop?”
“When you have an asynchronous
483
“What could event loop blocking hurt?”
“This could hurt overall responsiveness and increase latency
484
“How can you detect when event loop blocking is happening?”
“You can detect when this problem is happening by passing debug=True to the asyncio.run function.”
485
“What do you need to minimize for maximum responsiveness?”
“If you want the most responsive program possible
486
“How can you minimize system calls in the event loop?”
“One way to do this is to write a subclass that does everything you need it to do.”
487
“Can coroutines in other threads call thread-safe wrapper methods?”
“Coroutines in other threads can directly call and await thread-safe wrapper methods that wrap other asynchronous methods in classes that use the strategy mentioned above.”
488
“What does using thread-safe wrapper methods eliminate?”
“It also eliminates the need for Lock.”
489
“What methods can you define for with statements with async classes?”
“You can also define the **aenter** and **aexit** methods so the kind of class mentioned above can be used with with statements.”
490
“What wall might you hit in Python programs?”
“At some point in writing Python programs
491
“What might happen even after optimizing your code?”
“Even after optimizing your code
492
“What is a reasonable assumption for modern computers?”
“On modern computers that have an increasing number of CPU cores
493
“Does the GIL allow true parallelism in threads?”
“Unfortunately
494
“What is another common suggestion for performance?”
“Another common suggestion is to re-write your performance-critical code as an extension module
495
“What can C do for Python programs?”
“C gets you closer to the bare metal and can run faster than Python
496
“Can C extensions start native threads?”
“C extensions can also start native threads independent of the Python interpreter than run in parallel and utilize multiple CPU cores with no concern for the GIL.”
497
“Is Python’s C extension API well documented?”
“Python’s API for C extensions is well documented and a good choice for an escape hatch.”
498
“What tools can help with C extensions?”
“It’s also worth checking out tools like SWIG and CLIF.”
499
“What is the cost of rewriting code in C?”
“However
500
“How does Python code compare to C code in terms of complexity?”
“Code that is short and understandable in Python can become verbose and complicated in C.”
501
“What does porting to C require?”
“Such a port requires extensive testing to ensure that the functionality is equivalent to the original Python code and that no bugs have been introduced.”
502
“Is it sometimes worth porting to C?”
“Sometimes it’s worth it
503
“What tools ease the transition to C?”
“There are even open-source tools such as Cython and Numba that ease the transition to C.”
504
“Is moving one piece of your program to C sufficient?”
“The problem is that moving one piece of your program to C isn’t sufficient most of the time.”
505
“What do optimized Python programs usually have as sources of slowness?”
“Optimized Python programs usually don’t have one major source of slowness; rather
506
“What would you need to port to get C’s benefits?”
“To get the benefits of C’s bare metals and threads
507
“Is there a better way than porting to C?”
“There has to be a better way to preserve your investment in Python to solve difficult computation problems.”
508
“What module may be what you need for parallelism?”
“The multiprocessing module
509
“What does multiprocessing enable?”
“It enables Python to use multiple CPU cores in parallel by running additional interpreters as child processes.”
510
“Are child processes separate from the main interpreter?”
“These child processes are separate from the main interpreter
511
“What can each child fully utilize?”
“Each child can fully utilize one CPU core.”
512
“What does each child have?”
“Each child has a link to the main process where it receives instructions to do computation and return results.”
513
“Can you change ThreadPoolExecutor to ProcessPoolExecutor?”
“In some cases
514
“What is step 1 of how ProcessPoolExecutor works?”
“It takes each item from the numbers input data to map.”
515
“What is step 2 of how ProcessPoolExecutor works?”
“It serializes items into binary data by using pickle.”
516
“What is step 3 of how ProcessPoolExecutor works?”
“It copies the serialized data from the main interpreter process to a child interpreter process over a local socket.”
517
“What is step 4 of how ProcessPoolExecutor works?”
“It deserializes the data back into Python objects
518
“What is step 5 of how ProcessPoolExecutor works?”
“It imports the Python module using the gcd function.”
519
“What is step 6 of how ProcessPoolExecutor works?”
“It runs the function on the input data in parallel with other child processes.”
520
“What is step 7 of how ProcessPoolExecutor works?”
“It serializes the result back into binary data.”
521
“What is step 8 of how ProcessPoolExecutor works?”
“It copies that binary data back through the socket.”
522
“What is step 9 of how ProcessPoolExecutor works?”
“It deserializes the binary data back into Python objects in the parent process.”
523
“What is step 10 of how ProcessPoolExecutor works?”
“It merges the results from multiple children into a single list to return.”
524
“Does multiprocessing look simple to programmers?”
“Though it looks simple to a programmer
525
“What do you need in most other languages to coordinate threads?”
“In most other languages
526
“Why is the overhead of multiprocessing high?”
“The overhead of using multiprocessing via ProcessPoolExecutor is high because of all the serialization and deserialization that must that must happen between parent and child processes.”
527
“What types of functions is multiprocessing well-suited for?”
“The above scheme is well-suited to certain types of isolated
528
“What does isolated mean?”
“Isolated means functions that don’t need to share state with other parts of the program.”
529
“What does high-leveraged mean?”
“High-leveraged means situations where only a small amount of data must be transmitted between the parent and child processes to enable a large amount of computation.”
530
“What is an example of an algorithm that works well with multiprocessing?”
“The greatest common divisor algorithm is one example of this
531
“What happens if your computation doesn’t have these characteristics?”
“If your computation doesn’t have these characteristics
532
“What does multiprocessing provide when ProcessPoolExecutor isn’t enough?”
“When that happens
533
“Are these advanced multiprocessing features complex?”
“But all these features are very complex.”
534
“Why is it hard to reason about these tools?”
“It’s hard enough to reason about tools in the memory space of a single process shared between Python threads.”
535
“What makes this approach much more difficult?”
“Extending that complexity to other processes and involving sockets makes this approach much more difficult to understand.”
536
“What should you initially avoid?”
“You should initially avoid all parts of the multiprocessing module.”
537
“What can you start with for parallelism?”
“You can start with the ThreadPoolExecutor class to run isolated
538
“What can you move towards later?”
“Later you can move towards theProcessPoolExecutor class to get a speedup.”
539
“When should you consider using multiprocessing directly?”
“Finally
Python
flashcards
Decks in class (7)
# Cards
