As part of a lecture on exploring large-scale software, I decided to explore python, which I had been interested in for some time. Since the task of reviving the python2 print statement has been achieved, this time I decided to tackle the task of introducing a private variable into python.
There is no ** private variable in the Python class **. Instead, it is customary to treat a variable as a private variable by prefixing the variable name with "_". It's up to the programmer's conscience. This is a very different idea from object-oriented languages such as Java. The background is the following stack overflow post http://stackoverflow.com/questions/1641219/does-python-have-private-variables-in-classes Please refer to. However, I thought that there might be a new programmer who does not know the custom of python and rewrites variables that should not be rewritten from outside the class (without permission), so this time I do not care about the idea of python, and it is a private variable. I wanted to implement the convention in a more rigorous way.
The original private variable is designed to throw an error and stop processing when it is accessed from the outside (probably). However, as you can see by trying it, it seems that there are various processes to access private variables from the outside in python's internal processing, so if you try to fail with an error, python will fall without permission (or rather, it will not compile) Hmm). Therefore, this time I decided to implement it in the form of issuing a warning when accessing the private variable. I'm wondering if I've introduced a private variable, but I thought this was enough to help the target of users who didn't know python conventions, so I compromised.
First, if you subdivide the problem you are working on this time
Continuing from the last time, I basically focused on gdb to find out.
At first I was groping for the compiler because I was dragged by playing with the compiler last time. But if you think about it, this method shouldn't work. This is because python classes are just dictionaries, and you don't always know what key (member variable) they have at compile time. Therefore, at the time of compilation, you do not access the information of the member variables of the instance, and you can only know if the member variables exist by accessing the variables. In other words, what you really need to find out is how to handle the timing when a member variable is actually accessed based on the compiled opcode. So you're groping for a python Virtual Machine, not a compiler. After a little research, I found that a file called ceval.c evaluates the opcode. The processing around it is described in detail on the following site. http://www.nasuinfo.or.jp/FreeSpace/kenji/sf/python/virtualMachine/PyVM.html
Next, I used dis, a handy built-in module of python, to figure out which opcode to look at. dis converts the passed python code to the opcode used inside the python interpreter, so I actually generated the opcode for the following code.
Before conversion
class Foo:
def __init__(self, foo):
self._foo = foo
def mymethod(self, foo):
self._foo = foo
f = Foo("constructor")
f._foo = "external access"
f.mymethod("internal access")
After conversion
2 0 LOAD_BUILD_CLASS
1 LOAD_CONST 1 (<code object Foo at 0x118a0bdb0,line 2>)
4 LOAD_CONST 2 ('Foo')
7 MAKE_FUNCTION 0
10 LOAD_CONST 2 ('Foo')
13 CALL_FUNCTION 2 (2 positional, 0 keyword pair)
16 STORE_FAST 0 (Foo)
8 19 LOAD_FAST 0 (Foo)
22 LOAD_CONST 3 ('constructor')
25 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
28 STORE_FAST 1 (f)
9 31 LOAD_CONST 4 ('external access')
34 LOAD_FAST 1 (f)
37 STORE_ATTR 0 (_foo)
10 40 LOAD_FAST 1 (f)
43 LOAD_ATTR 1 (mymethod)
46 LOAD_CONST 5 ('internal access')
49 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
52 POP_TOP
53 LOAD_CONST 0 (None)
56 RETURN_VALUE
As I saw it, I found that the opcode STORE_ATTR was suspicious. Now we have all the ingredients. From here, we will explore ceval.c while utilizing gdb.
First, when I honestly searched for the character string STORE_ATTR in ceval.c, I found the following part.
TARGET(STORE_ATTR) {
PyObject *name = GETITEM(names, oparg);
PyObject *owner = TOP();
PyObject *v = SECOND();
int err;
STACKADJ(-2);
err = PyObject_SetAttr(owner, name, v);
Py_DECREF(v);
Py_DECREF(owner);
if (err != 0)
goto error;
DISPATCH();
}
It seems that we are processing the STORE_ATTR opcode here. And there were some things I could understand by looking at this code. First, it is inferred from the variable name that name is ** the name of the variable you want to access ** and owner is the instance ** you should set **. And it seems that the function that actually sets the variable is PyObject_SetAttr. Since the arguments of the PyObject_SetAttr function are name, owner, v, it is presumed that v is probably the value you want to set. First, to confirm these hypotheses, I set a breakpoint here for the time being and ran the following code test.py with gdb.
test.py
class Foo:
def __init__(self, foo):
self._foo = foo
def mymethod(self, foo):
self._foo = foo
f = Foo("constructor")
f._foo = "external access"
f.mymethod("internal access")
However, when I actually try it, it seems that python makes heavy use of STORE_ATTR even when loading modules, and if this is the case, too much processing will be caught and the work will not proceed. If you don't stop it only when the attribute "_foo" is called, you can't touch it. The problem is that name is of type PyObject, a Python object (this time unicode), so you simply can't access the variable name. So, I looked into the python unicode type implementation described in unicodeobject.h and unicodeobject.c to see if it could be converted to a C string (char array). Then, in unicodeobject.h, there was a function like that.
#ifndef Py_LIMITED_API
PyAPI_FUNC(char *) PyUnicode_AsUTF8(PyObject *unicode);
#define _PyUnicode_AsString PyUnicode_AsUTF8
#endif
In fact, when I applied this function to the name object, I was able to get the string pointed to by name as a char * type. Using this, I set a breakpoint inside the if statement that holds on the condition that name and "_foo" match so that gdb stops only when name is "_foo", and I tried running gdb. ,It went well. Here we can certainly substantiate the hypothesis that name is the name of the variable we are actually trying to access.
Next, we looked at the processing of the PyObject_SetAttr function and tested the hypothesis that the instance that owner is actually trying to set and v is the value that they are trying to set. After diving for a while, I found that the following function was called with owner as the first argument.
int
_PyObject_GenericSetAttrWithDict(PyObject *obj, PyObject *name,
PyObject *value, PyObject *dict)
{
PyTypeObject *tp = Py_TYPE(obj);
PyObject *descr;
descrsetfunc f;
PyObject **dictptr;
int res = -1;
if (!PyUnicode_Check(name)){
PyErr_Format(PyExc_TypeError,
"attribute name must be string, not '%.200s'",
name->ob_type->tp_name);
return -1;
}
if (tp->tp_dict == NULL && PyType_Ready(tp) < 0)
return -1;
Py_INCREF(name);
descr = _PyType_Lookup(tp, name);
Py_XINCREF(descr);
f = NULL;
if (descr != NULL) {
f = descr->ob_type->tp_descr_set;
if (f != NULL && PyDescr_IsData(descr)) {
res = f(descr, obj, value);
goto done;
}
}
if (dict == NULL) {
dictptr = _PyObject_GetDictPtr(obj);
if (dictptr != NULL) {
res = _PyObjectDict_SetItem(Py_TYPE(obj), dictptr, name, value);
if (res < 0 && PyErr_ExceptionMatches(PyExc_KeyError))
PyErr_SetObject(PyExc_AttributeError, name);
goto done;
}
}
if (dict != NULL) {
Py_INCREF(dict);
if (value == NULL)
res = PyDict_DelItem(dict, name);
else
res = PyDict_SetItem(dict, name, value);
Py_DECREF(dict);
if (res < 0 && PyErr_ExceptionMatches(PyExc_KeyError))
PyErr_SetObject(PyExc_AttributeError, name);
goto done;
}
if (f != NULL) {
res = f(descr, obj, value);
goto done;
}
if (descr == NULL) {
PyErr_Format(PyExc_AttributeError,
"'%.100s' object has no attribute '%U'",
tp->tp_name, name);
goto done;
}
PyErr_Format(PyExc_AttributeError,
"'%.50s' object attribute '%U' is read-only",
tp->tp_name, name);
done:
Py_XDECREF(descr);
Py_DECREF(name);
return res;
}
The important thing is to get the dictionary of the first argument (obj) here. Since a python class instance is basically a dictionary, this means that you are accessing obj itself. In other words, we were able to confirm that the owner was actually the instance to be accessed. And since I know that v is set in the dictionary with the key named name with PyDict_SetItem, I was able to confirm that v is the value to be set.
So what you have to do from here is to find out the information of the ** calling instance ** and match it with the owner (compare the pointer). So I started looking for information about the calling instance.
In Python methods, the instance itself is always set as the first argument. In other words, you should be able to calculate back the instance information from the method. The method referred to the official documentation to find out how to find out which instance it belongs to. https://docs.python.org/3/c-api/method.html Apparently, a function called PyMethod_GET_SELF is getting the instance to which it belongs. And one more thing I found out is that a method is basically just a structure that has information about the function to execute and the instance to which it belongs. This will become important later. Based on this information, I ran test.py again with gdb to search for instance information. In conclusion, I couldn't find any information about the instance calling the method, no matter how down the stack. And when I climbed the stack trace, I found out why.
classobject.c
static PyObject *
method_call(PyObject *func, PyObject *arg, PyObject *kw)
{
PyObject *self = PyMethod_GET_SELF(func);
PyObject *result;
func = PyMethod_GET_FUNCTION(func);
if (self == NULL) {
PyErr_BadInternalCall();
return NULL;
}
else {
Py_ssize_t argcount = PyTuple_Size(arg);
PyObject *newarg = PyTuple_New(argcount + 1);
int i;
if (newarg == NULL)
return NULL;
Py_INCREF(self);
PyTuple_SET_ITEM(newarg, 0, self);
for (i = 0; i < argcount; i++) {
PyObject *v = PyTuple_GET_ITEM(arg, i);
Py_XINCREF(v);
PyTuple_SET_ITEM(newarg, i+1, v);
}
arg = newarg;
}
result = PyObject_Call((PyObject *)func, arg, kw);
if(PyDict_Contains(funcdict, PyUnicode_FromString("__parent_instance__"))){
PyDict_DelItem(funcdict, PyUnicode_FromString("__parent_instance__"));
}
Py_DECREF(arg);
return result;
}
You can see that the method call seems to be executed by the above function. And if you take a closer look at this process, you can see that the method invocation is roughly done in the following steps:
Think twice. ** Instances are just function arguments. ** This means that by the time the method is executed by method_call, the information about the instance to which the method belongs is already lost. And the processing of STORE_ATTR is executed after method_call is called. In other words, if this is left as it is, it will not be possible to distinguish between access by external functions and access by internal methods at the timing of accessing the member variables of the instance.
class Foo:
def inner_method(self, foo):
self._foo = foo
def outer_function(hoge, foo):
hoge._foo = foo
f = Foo()
f.inner_method(100)
outer_function(f, 100)
This is a problem. Somehow, we need to pass the caller instance information to the process that executes STORE_ATTR. If this happens, you have no choice but to forcibly hand it over.
The method I took this time is to forcibly register the instance in the global variable with the variable name "\ _ \ _ parent_instance \ _ \ _" at the time of method_call. Actually, I should define it with a new scope, but it is troublesome, so I decided to break through with such a brute force method this time. Global variables are one of the properties of a function and are implemented as a python dictionary. While checking the dictionary type implementation in dictobject.h and dictobject.c, I registered it in the global variable as follows.
Modified class object.c
static PyObject *
method_call(PyObject *func, PyObject *arg, PyObject *kw)
{
PyObject *self = PyMethod_GET_SELF(func);
PyObject *result;
#if PRIVATE_ATTRIBUTE
PyFunctionObject *temp;
PyObject *funcdict;
#endif
func = PyMethod_GET_FUNCTION(func);
if (self == NULL) {
PyErr_BadInternalCall();
return NULL;
}
else {
Py_ssize_t argcount = PyTuple_Size(arg);
PyObject *newarg = PyTuple_New(argcount + 1);
int i;
if (newarg == NULL)
return NULL;
Py_INCREF(self);
PyTuple_SET_ITEM(newarg, 0, self);
for (i = 0; i < argcount; i++) {
PyObject *v = PyTuple_GET_ITEM(arg, i);
Py_XINCREF(v);
PyTuple_SET_ITEM(newarg, i+1, v);
}
arg = newarg;
}
#if PRIVATE_ATTRIBUTE
temp = (PyFunctionObject *)func;
funcdict = temp->func_globals;
if(funcdict == NULL){
funcdict = PyDict_New();
}
PyDict_SetItem(funcdict, PyUnicode_FromString("__parent_instance__"), self);
#endif
result = PyObject_Call((PyObject *)func, arg, kw);
if(PyDict_Contains(funcdict, PyUnicode_FromString("__parent_instance__"))){
PyDict_DelItem(funcdict, PyUnicode_FromString("__parent_instance__"));
}
Py_DECREF(arg);
return result;
}
The part enclosed by #if PRIVATE_ATTRIBUTE and #endif is the added code. From the function, you can see that the dictionary in which global variables are registered is obtained and the variables are forcibly registered there.
One point to note is that if you do not delete \ _ \ _ parent_instance \ _ \ _ from the global variable when the method execution is finished, you can access it once the private variable is accessed from the internal method. It has become. Therefore, when the method finishes executing, PyDict_DelItem finally deletes \ _ \ _ parent_instance \ _ \ _.
Next, when STORE_ATTR was executed, the following processing was added and corrected.
Modified ceval.c
TARGET(STORE_ATTR) {
PyObject *name = GETITEM(names, oparg);
PyObject *owner = TOP();
PyObject *v = SECOND();
int err;
#if PRIVATE_ATTRIBUTE
char *name_as_cstr;
PyObject *parent_instance;
if(PyDict_Contains(f->f_globals, PyUnicode_FromString("__parent_instance__"))){
parent_instance = PyDict_GetItem(f->f_globals, PyUnicode_FromString("__parent_instance__"));
}else{
parent_instance = NULL;
}
#endif
STACKADJ(-2);
#if PRIVATE_ATTRIBUTE
name_as_cstr = _PyUnicode_AsString(name);
if(name_as_cstr[0] == '_'){
if(!parent_instance || (parent_instance - owner) != 0){
printf("Warning: Illegal access to a private attribute!\n");
}
}
#endif
err = PyObject_SetAttr(owner, name, v);
Py_DECREF(v);
Py_DECREF(owner);
if (err != 0)
goto error;
DISPATCH();
}
If the first character of name is \ _, it is determined that the access target is a private variable. And, if there is \ _ \ _ parent_instance \ _ \ _ in the global variable and the owner and \ _ \ _ parent_instance \ _ \ _ match, it is considered as a legitimate access, otherwise Warning is printedf. I added a simple process of displaying. And when I try to compile with PRIVATE_ATTRIBUTE set to 1 in config ...
python
>>> class Foo:
... def __init__(self, foo):
... self._foo = foo
... def mymethod(self, foo):
... self._foo = foo
...
>>>
>>>
>>> f = Foo(1)
>>> # no warnings!
>>>
>>> f._foo = 2
Warning: Attempting to access private attribute illegally
The constructor is called on the line f = Foo (1). At first glance, it seems that the warning does not appear here and the warning occurs when access from the outside occurs. But here's the problem.
python
>>> f.mymethod(3)
Warning: Attempting to access private attribute illegally
For some reason, I was angry even though it was an access from the inside. When I run test.py with gdb to find out the cause, \ _ \ _ parent_instance \ _ \ _ is not registered in the global variable. Somewhere the process isn't working. Looking back at the stack trace, it turns out that ** method_call is not called ** in f.mymethod (3).
Instead, a mysterious function called fast_function was called.
When I looked up what fast_function is, there was a useful explanation on the following site. http://eli.thegreenplace.net/2012/03/23/python-internals-how-callables-work
To understand what fast_function does, it's important to first consider what happens when a Python function is executed. Simply put, its code object is evaluated (with PyEval_EvalCodeEx itself). This code expects its arguments to be on the stack. Therefore, in most cases there's no point packing the arguments into containers and unpacking them again. With some care, they can just be left on the stack and a lot of precious CPU cycles can be spared.
Apparently, it is wasteful to create tuples of arguments one by one, so it seems that python is trying to speed up by fetching variables from the stack. The function called fast_function is in charge of that. In other words, it is faster by not going through method_call. However, the method should still be called at some point and the instance should have been acquired. If you register it in a global variable at that timing, you should be able to achieve the same thing as an extension of the implementation so far. So, when I searched for the part that is accessing the instance to which the method belongs, that is, the part where PyMethod_GET_SELF is called, the following part of the call_function function of ceval.c was hit.
call_inside the function function
if (PyMethod_Check(func) && PyMethod_GET_SELF(func) != NULL) {
/* optimize access to bound methods */
PyObject *self = PyMethod_GET_SELF(func);
PCALL(PCALL_METHOD);
PCALL(PCALL_BOUND_METHOD);
Py_INCREF(self);
func = PyMethod_GET_FUNCTION(func);
Py_INCREF(func);
Py_SETREF(*pfunc, self);
na++;
n++;
}
So, with a simple idea, I rewrote it as follows.
Modified call_function function
if (PyMethod_Check(func) && PyMethod_GET_SELF(func) != NULL) {
/* optimize access to bound methods */
PyObject *self = PyMethod_GET_SELF(func);
PCALL(PCALL_METHOD);
PCALL(PCALL_BOUND_METHOD);
Py_INCREF(self);
func = PyMethod_GET_FUNCTION(func);
Py_INCREF(func);
Py_SETREF(*pfunc, self);
na++;
n++;
#if PRIVATE_ATTRIBUTE
temp = (PyFunctionObject *)func;
funcdict = temp->func_globals;
if(funcdict == NULL){
funcdict = PyDict_New();
}
PyDict_SetItem(funcdict, PyUnicode_FromString("__parent_instance__"), self);
#endif
}
Note that temp and funcdict are the same as when classobject.c was modified, including the type and meaning. If you don't define it at the beginning of the function, the compiler will get angry, so the definition is just not in this code. The deletion was done as follows at the end of the function.
assert((x != NULL) ^ (PyErr_Occurred() != NULL));
#if PRIVATE_ATTRIBUTE
temp = (PyFunctionObject *)func;
funcdict = temp->func_globals;
if(funcdict == NULL){
funcdict = PyDict_New();
}
if(PyDict_Contains(funcdict, PyUnicode_FromString("__parent_instance__"))){
PyDict_DelItem(funcdict, PyUnicode_FromString("__parent_instance__"));
}
#endif
return x;
}
Now try compiling and running it. Then, on Mac OS, it worked as follows.
>>> class Foo:
... def __init__(self, foo):
... self._foo = foo
... def mymethod(self, foo):
... self._foo = foo
...
>>> f = Foo(10)
>>> f._foo = 100
Warning: Illegal access to a private attribute!
>>> f.mymethod(200)
>>>
>>> def outerfunc(f, foo):
... f._foo = foo
...
>>> outerfunc(f, 100)
Warning: Illegal access to a private attribute!
You can see that mymethod is not throwing an error. It was surprisingly easy. However, when I tried to compile the source with the same modification on Ubuntu, I was angry that I shouldn't mess with the Global variable, and the compilation failed with an error. Originally, it should be modified so that it works on Ubuntu, but this time I didn't have that much time, so I made this a compromise for the time being.
Through this experiment, I learned some important lessons for playing with large-scale software, so I will verbally summarize it for myself.
You should rely on the official documentation This time, there were many scenes where understanding progressed at once just by reading the official document. Of course, that's not the case unless the official documentation is in place, but I think it's a good idea to read the official documentation as soon as possible. It's a matter of course. However, documents related to the internal processing of many software, not just python, can be difficult to find by simply searching. The best way to find the official documentation is to use the keywords contributor, developer. If you are looking for large-scale software, search for "software name contributor", "software name developer guide", etc., and find a guide for the part you are trying to contribute (grammar for grammar, performance for performance, etc.). I think it's a good idea to start by looking for it. Therefore, the efficiency is completely different between following the code after making a star and simply following it.
The importance of verbalizing the process I think that by clearly verbalizing what you want to do, it becomes clear what you need to do for the first time. For example, the policy of implementing private variables,
"Make the variables of an instance accessible only from within that instance"
I mean
"At the timing of accessing an instance variable, determine whether the variable name is prefixed with" _ "and check whether the target trying to access the variable is an object that has that variable as an instance variable."
That's a completely different possible policy. With the former explanation, it is unclear where to start. On the other hand, in the latter explanation, processing is performed at the timing of accessing the variable, so it naturally develops into the idea of searching for that timing. In fact, I spent a day exploring the compiler because I didn't explicitly verbalize what I had to do first. When you think about it later, it's natural that you should play with Virtual Machine, but it's easy to overlook such a natural thing unless you verbalize it and organize your mind. As is often said, I think it's a good idea to start the work by writing down what you are trying to do in an easy-to-understand manner with the feeling of explaining it to others.
"The key to understanding a program is to understand its data structures. With that in hand, the algorithms usually become obvious."
There were many scenes where I was keenly aware of this. Especially around the compiler (the moment I understood the data structures such as AST and DFA, the whole flow became visible at once), but even in the implementation of private variables, for example, in the data structure where the method contains functions and instances. Knowing that something is happening and that global variables are dictionaries was quite important in understanding the overall process flow. A more practical lesson from this lesson is that header files (.h files) should not be underestimated. It's also easy to see more than just looking at the working code by looking at the struct definition in the header file.
Official guide for Python developers https://docs.python.org/devguide/
A blog with lots of easy-to-understand and detailed articles about Python's internal processing http://eli.thegreenplace.net/tag/python-internals
Recommended Posts