Dot Notation in Python

[fog37]

TL;DR Summary

Understanding dot notation in general

Hello (again).

I have become quite familiar with the dot notation in Python. The dot "operator", assuming operator is the right term, seem to have many uses.
For example:

1) After an instance name, it can be used to access instance attributes and apply instance methods to the object/instance itself.

 car1=Car()    
 car1.speed      #speed is the attributed
 car1.turn   #turn is the method

2) After importing a module, it can be used be used to access functions inside the module, access classes contained inside the module to create objects, access variables contained inside the module, what else?
Example:

import module1  #import a user-defined module called module1
module1.function1 #access a particular function called function1 inside the module

object1=module1.Class1() #create an object called object 2 from Class 1 contained inside the module

module1.object2.attribute1 #access a specific attribute of object2 contained inside the module
module1.object2.method1  #access a specific method of object2 contained inside the module

Is that correct? I feel like it should be called the "access" operator since it allows us to access things.

What are the exact inner working and the more general definition the dot notation/dot operator?

Everything in Python is an object, as we repeatedly mentioned. So also a module, which can contain functions, classes, variables, objects, lines of code, should be considered an object in some sense...What class would a module belong to?

Thanks!

 

[PeterDonis]

The dot "operator", assuming operator is the right term, seem to have many uses.

Yes, "operator" is a reasonable term. What appear to you to be "many" uses are actually all the same thing: the code A.B tells the Python interpreter to look for an attribute named B on the object A; if the object itself doesn't have the attribute, the interpreter looks at its type, and any other types that that type inherits from.

Instance variables like your speed are attributes on the instance; methods like your turn are attributes on the class, which is the type of the instance (and that class might inherit from other classes, and the method might be on one of those).

Variables, functions, and classes defined in a module are attributes on the module object.

What class would a module belong to?

Try it in the interpreter and see!

>>> import sys
>>> type(sys)
<class 'module'>

In other words, 'module' is a type, of which each individual module is an instance. (This type is built into the Python interpreter, like other built-in types such as list or int.)

 

[wle]

The dot "operator", assuming operator is the right term, seem to have many uses.

If you consider it an "operator" then it's a rather odd one (or one treated in a very special way by the Python interpreter) in that it takes as a second operand something that is not a Python object. This means for example you can't do

x = turn
car.x()

as an alternative way of doing car.turn(). It doesn't make sense in Python because the method name turn in the source code is not a Python object^*.

On the other hand, you can get at a method (or attribute, or something in a module) named by a given string, but not using the dot:

x = 'turn'
car.__getattribute__(x)()

----------

^*This is by contrast with some so-called homoiconic programming languages, like the Lisp family of languages, which have a built-in symbol type used to represent variable names and keywords appearing in their own source code, among other uses.

 

[pbuk]

I have become quite familiar with the dot notation in Python. The dot "operator", assuming operator is the right term, seem to have many uses.

The correct term is "delimiter". The linked text notes that "the period can also occur in floating-point and imaginary literals" and "a sequence of three periods has a special meaning as an ellipsis literal".

What are the exact inner working and the more general definition the dot notation/dot operator?

Note that as you have observed, everything in Python is an object; similarly every object is implemented as a dictionary consisting of its attributes. The function of the . delimiter is to separate an attribute reference (the name following the ".") from the name of the dictionary which precedes the ..

 

[PeterDonis]

If you consider it an "operator" then it's a rather odd one (or one treated in a very special way by the Python interpreter) in that it takes as a second operand something that is not a Python object.

Sort of. As you note, you can't do this:

x = turn
car.x()

But you can do this:

x = "turn"
getattr(car, x)()

In this respect getattr works similarly to the various functions in the operator module that give functional versions of other operators; for example, compare:

x = 1
y = 2
z = x + y

with:

x = 1
y = 2
z = operator.add(x, y)

Note, btw, that the "function call" notation () can also be considered an operator in Python. The general rule is, if you can overload it in a user-defined class by defining a double underscore method, it can be considered an operator. You can overload the function call operator by defining a __call__ method, and you can overload the attribute access operator by defining __getattr__ or __getattribute__, depending on what kind of overloading you want to do.

 

[PeterDonis]

The correct term is "delimiter".

As far as lexical analysis of the source code, yes, the dot is a delimiter, not an operator.

However, as far as what the code actually does, many of the things called "delimiters" in lexical analysis function like operators--they can be overloaded by defining double underscore methods, and they have corresponding functions in the operator module. Other examples besides the dot include the parentheses (function call), the square brackets (get item by either index or key), and the various in-place operators (e.g., += for in-place addition).

 

[pbuk]

As far as lexical analysis of the source code, yes, the dot is a delimiter, not an operator.

However, as far as what the code actually does, many of the things called "delimiters" in lexical analysis function like operators--they can be overloaded by defining double underscore methods, and they have corresponding functions in the operator module. Other examples besides the dot include the parentheses, the square brackets,

Not in Python - see the link I gave to the documentation and also the documentation for the operator module.

and the various in-place operators (e.g., += for in-place addition).

Yes, these are operators in Python: from the linked documentation "the second half of the list [of delimiters], the augmented assignment operators, serve lexically as delimiters, but also perform an operation."

 

[PeterDonis]

Not in Python

It is true that the operator module does not have functions corresponding to the attribute access operator (dot), the function call operator (parentheses) and the item access operator (square brackets). However, all of those can be overloaded by defining double underscore methods; in that respect they are like operators. Attribute access also has a builtin function, getattr, so there would be no point in having a function defined for it in the operator module.

(There actually used to be a built-in function for function call as well, apply, but it was removed in one of the later Python 2 versions IIRC. It would make sense to have a built-in function called getitem for item access, but Python has never had one. Which just means that terms like "operator" are fuzzy in Python; some things have some of the features of operators but not others.)

 

[fog37]

Great! Thanks.

The dot notation/operator can be used twice in the same statement to access something inside something else. For example:

import matplotlib.pyplot
import numpy as np

xpoints = np.array([0, 6])
ypoints = np.array([0, 250])

# both plot() and show() are functions inside the sub-module pyplot contained inside the module matplotlib
matplotlib.pyplot.plot(xpoints, ypoints)
matplotlib.pyplot.show()

I guess a module that contains other modules should be called a "library", correct? So matplotlib is actually a library instead of a module.

It is easy to create a module: it is just a .py file containing variables, functions, objects, classes, etc.
I remember that, in general, there should be something (is it main() or some other function?) inside every module but it is possible to create modules without it...

In essence, once again, everything is an object that derives from a class and has attributes. The term attributes comprises both the instance variables and instance methods. In some lectures, I notices that the term attribute is used as synonym for instance variables...

Thanks as always!

 

[pbuk]

It is true that the operator module does not have functions corresponding to the attribute access operator (dot), the function call operator (parentheses) and the item access operator (square brackets). However, all of those can be overloaded by defining double underscore methods;

No, you cannot override the attribute access operator, obj.attr will always access the attr attribute of the obj object if it exists. What you can do is provide a fallback __getattr__ special method which will be called if the attribute does not exist. Similarly obj.attr() will always call the obj.attr method if it exists, but if not it will fall back to any user-defined obj.__call__) special method. The difference is subtle, but it is a difference.

 

[PeterDonis]

you cannot override the attribute access operator

Yes, you can. Go look up the documentation for the __getattribute__ special method.

 

[PeterDonis]

Similarly obj.attr() will always call the obj.attr method if it exists, but if not it will fall back to any user-defined obj.__call__) special method.

No, it won't. obj.attr() retrieves the attribute "attr" from the object "obj", and then applies the function call operation to whatever object is retrieved. That object does not have to be a method; it can be any object that is callable (for example, an instance attribute of "obj" that is an instance of a user-defined class with a obj.__call__ special method defined). It does not apply the function call operation to the object "obj" itself, which is what obj.__call__ would overload.

 

[pbuk]

I guess a module that contains other modules should be called a "library", correct? So matplotlib is actually a library instead of a module.

I wouldn't get too hung up on this, you will see the terms 'library', 'module' and 'package' used pretty much interchangeably.

The term attributes comprises both the instance variables and instance methods.

Yes.

In some lectures, I notices that the term attribute is used as synonym for instance variables...

Yes, although this contradicts the above the meaning should be clear from the context.

 

[PeterDonis]

The dot notation/operator can be used twice in the same statement to access something inside something else.

Yes. That just means that you are looking up an attribute twice. First you look up the "pyplot" attribute on the "matplotlib" object (which gives you a module); then you look up the "plot" attribute on the "pyplot" object (which gives you a function defined in that module).

I guess a module that contains other modules should be called a "library", correct?

The strictly correct Python term for this is "package". In your example, matplotlib is a package and matplotlib.pyplot is a module inside that package.

 

[pbuk]

Yes, you can. Go look up the documentation for the __getattribute__ special method.

You are absolutely right, I stand corrected.

 

[pbuk]

No, it won't. obj.attr() retrieves the attribute "attr" from the object "obj" (which can, and then applies the function call operation to whatever object is retrieved. That object does not have to be a method; it can be any object that is callable (for example, an instance attribute of "obj" that is an instance of a user-defined class with a obj.__call__ special method defined). It does not apply the function call operation to the object "obj" itself, which is what obj.__call__ would overload.

Oh dear, I seem to have been thinking about a different language.

 

[PeterDonis]

I remember that, in general, there should be something (is it main() or some other function?) inside every module

What you might be thinking of is the fact that, if a package has a module in it named __main__, then that package can be run as if it were a script; in other words, if I have a package mypackage with a module in it named __main__, then the command line

python -m mypackage

will load the module mypackage.__main__ and run it as if it were a script.

You might also be thinking of the stanza that appears in many Python files to detect when they are being run as scripts, instead of being imported as modules; something like this:

# code here will run whether the file is imported as a module
# or run as a script

if __name__ == '__main__':
    # code here will only run if the file is run as a script,
    # not if it is imported as a module

 

[PeterDonis]

The term attributes comprises both the instance variables and instance methods.

And variables defined on a class (which can, for example, be used to provide default values for instance variables), and possibly other weird things if you have overridden the __getattr__ or __getattribute__ special methods.

In some lectures, I notices that the term attribute is used as synonym for instance variables...

Yes, sometimes "attributes" is used in this limited way, and "methods" is used for methods (and other weirder possibilities are ignored). However, as far as the interpreter is concerned, "attribute access" (the dot) is just one thing; the interpreter does the same stuff under the hood whether what it ends up retrieving is an instance variable, a method, or something else.

 

[fog37]

Yes, I am thinking about __main__. Let me see if I got it.

First all all, __main__ is a special variable, not a special method (I originally thought that dunder objects where always and only special methods).

• Let's say a create 4 Python scripts, name them __main__, script01, script02, script03.
• The four scripts are all modules: each one of these four scripts contain functions tailored to a particular purpose (math, graphing, etc.)
• I then save the four files as .py, and place them into a regular Windows folder which I call Module_Folder.
• The folder called Module_Folder is a package and the four Python scripts are the modules inside of it.
• If I type Module_Folder, the name of the folder/package at the interpreter prompt, then the script __main__ runs (only that script).
  
  Is that correct?

If none of the four modules had been named __main__, then typing Module_Folder would have produced no result. So calling one if the scripts __main__ makes one of those scripts run when I type the package name.

I can certainly run each one of the four modules, regardless of their name by typing the module's name at the interpreter prompt. However, if a module contains only functions, nothing would really run, I believe.

So what scenarios would I make us name a module in a package __main__?
Modules are imported into other files using import to use their functionalities.

 

[PeterDonis]

First all all, __main__ is a special variable, not a special method

Actually, the special variable is __name__; the value of that variable when something is being run as a script is the string "__main__".

(I originally thought that dunder objects where always and only special methods).

No, not all dunder object names refer to special methods. If you run the python interpreter in interactive mode, you will see a number of dunder variables defined, none of which refer to methods:

>>> dir()
['__builtins__', '__cached__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', '__warningregistry__', 'pythonrc_py']

The folder called Module_Folder is a package and the four Python scripts are the modules inside of it.

Not as you've described it, no.

First, to make the folder a package, it needs to contain a file called "__init__.py". The file can be empty, but it must be there.

Second, for the Python interpreter to access the package, the parent folder to "Module_Folder" needs to be included in sys.path. If you run the interpreter from the parent folder to "Module_Folder", that will happen automatically. However, most Python packages are installed by installers that put their folders inside a parent folder that is already set up to be on sys.path.

If I type Module_Folder, the name of the folder/package at the interpreter prompt, then the script __main__ runs (only that script).

No. You don't type the names of commands at the Python interpreter prompt. If you run the interpreter in interactive mode from the parent folder of "Module_Folder", and that folder were set up as a package, then you could type import Module_Folder at the interpreter prompt and that would import the package. By itself, though, that would only run whatever code was in the "__init__.py" file, which might be nothing. To access the particular modules in the package, you would type things like import Module_Folder.script01.

To run the package as a script, you would type python -m Module_Folder at the Windows command prompt; that would work as long as either the current directory at the command prompt was the parent folder to "Module_Folder", or that parent folder was set up to be on sys.path.

I can certainly run each one of the four modules, regardless of their name by typing the module's name at the interpreter prompt.

Have you actually tried it? (Hint: it won't work. See above.)

What will work is if you open the Windows command prompt, make "Module_Folder" the current directory, and then type the name of one of the modules at that prompt. Which I suspect is what you are actually doing; but that is not the same as typing the module name at the Python interpreter prompt.

However, if a module contains only functions, nothing would really run, I believe.

No, code would run. In Python, function and class definitions, assignments to global module variables, etc., are all executable code. The interpreter is running code when it loads a module and does all those things. This is very different from a language like C, where all those things are really instructions to the compiler to tell it what executable code to compile. (Note that the results of some executable code in Python, like function and class definitions, can be other executable code: the function and class objects. This is also very different from a language like C; while it is not impossible to write code in C that generates other code--after all, that's what a compiler program does--it is not something that's inherent in the structure of the language itself.)

What is true is that none of those things have visible output. So if you ran a module that only had those things in them, yes, it would look like nothing at all happened. But the Python interpreter would still have run code; it would just be code that had no output, and all of the stuff that got defined would just be thrown away without doing anything with it.

So what scenarios would I make us name a module in a package __main__?

Convenience and ease of maintenance are two common reasons. Many times, a Python package will contain code that makes sense to run as a script, or requires such a thin wrapper around it to run as a script that it makes sense to include the thin wrapper along with the package. The Python package pip, which installs other Python packages, is a good example; most people know it as a Python program, but all that the program called "pip" actually does is load a module from the package "pip" and run it as a script. And you can do exactly the same thing by typing the command python -m pip at the Windows command prompt; the same code ends up running in either case.

 

[fog37]

Actually, the special variable is __name__; the value of that variable when something is being run as a script is the string "__main__".

No, not all dunder object names refer to special methods. If you run the python interpreter in interactive mode, you will see a number of dunder variables defined, none of which refer to methods:

>>> dir()
['__builtins__', '__cached__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', '__warningregistry__', 'pythonrc_py']

Thanks PeterDonis.

A few remarks to conclude:

1) Special Methods
Python is an "object-based" programming language where everything (almost. what is not an object?) is an object with attributes (attributes comprise both object variables and object methods). Some objects like strings, integers, floats, etc. are built-in, have their own data type (i.e. come from a specific class) and have their own specific methods that only apply to them. However, we can create instances from a class we define, which are a new kind of user-created object.
These new objects don't naturally work well with various built-in operators (like +, -, ==, etc.) and built-in functions (like print(), str(), etc.). We can apply these operators and functions to the instantiated objects but the result is generally not what we expect. However, we can "adapt" these functions and operators using their corresponding dunder methods inside our class definitions. The instantiated objects will then be able to use the operators and methods with the results that we normally expect. For example, the built-in operators +, - ,*, ==, have the corresponding methods __add__, __sub__, __mul__, __eq__ that can be included inside a class when we want to expand its functionality so we can use them on the class objects. Special methods are essentially about operator overloading and method overriding.

2) Variables for Built-in Objects
In the case of simple data types like strings and integers, I see how each different data type has its own associated methods. For example, in the case of list objects, there are list method like list_name.append(). Strings have string methods, dictionaries have dict methods.
If a list or an integer are objects, what are the variables associated to these objects since an object has attributes that are both variables and methods?

3) Wrapper Function
I have read about decorators as being functions that take another function and extend their behavior without explicitly modifying it. As a practical example from the topic of ML, is the library Keras a wrapper to the library tensorflow? Keras is often called an API for tensorflow but I wonder if it is actually a wrapper...

4) Different installed Python versions
When typing where python at the prompt, I see that there are two Python interpreters in different locations:

Which one of the two interpreters does the IDLE use? I think it uses the one in the Python37 folder. How can I check that?
When I install modules, those modules will not work with both Python interpreters, correct? Which version will they use?
What if I removed the version of Python in the Anaconda3 folder? Would it create an issue when I run Jupiter notebooks or other things inside the distribution?

Gratefully!

 

[pbuk]

If a list or an integer are objects, what are the variables associated to these objects since an object has attributes that are both variables and methods?

Objects don't have to have non-functional attributes. You can see the API for built-in types here.

Keras is often called an API for tensorflow but I wonder if it is actually a wrapper...

You could describe Keras as a wrapper for Tensorflow, but it is definitely not a decorator because there is not a 1:1 correspondence between Tensorflow methods and Keras methods.

I think it uses the one in the Python37 folder. How can I check that?

You could try renaming it and seeing if it still works.

When I install modules, those modules will not work with both Python interpreters, correct? Which version will they use?

It's hard to say with Windows - it is all down to the somewhat mysterious %PATH% environment variable, however it is likely that any packages you install through Anaconda will only be visible to Anaconda's Python interpreter.

What if I removed the version of Python in the Anaconda3 folder? Would it create an issue when I run Jupiter notebooks or other things inside the distribution?

Probably. Why would you want to do that? If you really want to use IDLE then why not use it from Anaconda?

 

[PeterDonis]

what is not an object?

While there are plenty of internal implementation details in the CPython interpreter that don't involve objects, I can't think of anything that's accessible at the Python level that isn't an object.

attributes comprise both object variables and object methods

Note that methods are attributes of the class, whereas "object variables" are attributes of the instance.

Some objects like strings, integers, floats, etc. are built-in, have their own data type (i.e. come from a specific class) and have their own specific methods that only apply to them. However, we can create instances from a class we define, which are a new kind of user-created object.

Actually, all objects in Python belong to a single object hierarchy. The base of the hierarchy is object, and many common methods that all objects have are methods of object. That means there are default implementations of all these methods, so subclasses only have to define new ones if the defaults are not suitable. For example, if you define a user-defined class and don't explicitly define a __repr__ or __str__ method, you can still print the object. Try it!

The various built-in types are then subclasses of object that add particular methods to the ones that object has, and sometimes override the defaults inherited from object. User-defined classes are also subclasses of object (if you don't explicitly declare a base class, it will be object--if you do define a different base class, object will still be the ultimate base class once you follow the inheritance chain all the way down). So the object hierarchy is completely unified. You can even define a user-defined class that inherits from a built-in type other than object, such as int or list. Try it!

These new objects don't naturally work well with various built-in operators (like +, -, ==, etc.) and built-in functions (like print(), str(), etc.).

They do with some, such as print() and str(), because of the defaults inherited from object (see above). But it's true that they won't with others if you just derive a class from object, because many of the built-in operators are only meant to work with some built-in types, not all. But if you inherit a user-defined class from int, for example, it will work with all the operators you'd expect an int to work with. Try it!

we can "adapt" these functions and operators using their corresponding dunder methods inside our class definitions

Yes. And the built-in types that work with those operators do it the same way. For example:

>>> int.__add__(2, 3)
5

If a list or an integer are objects, what are the variables associated to these objects since an object has attributes that are both variables and methods?

Remember, methods are attributes on the class, not the instance. An instance of int or list will not have any useful "variables" (attributes on the instance), because it doesn't need any. Everything useful that you can do with these objects is done using methods or built-in operators (which get translated into method calls).

Note, though, that if you define a user-defined class that inherits from int or list, you can put whatever variables you want on an instance of that class. Try it!

A final note: you'll notice that I said "Try it!" in a number of places above. One of the neatest things about Python is that you can answer pretty much any question you have about how the interpreter works by just firing up an interactive session and typing in commands to the interpreter, the way I did in the code snippet above where I showed how int objects have an __add__ method that is where addition is implemented for them. Making a habit of doing this will lead you to a much better understanding of how Python works than any amount of discussion in a forum could do.

 

[pbuk]

Note that methods are attributes of the class, whereas "object variables" are attributes of the instance.

As the OP seems to be aiming at completeness it is probably worth mentioning that methods can be bound to instances. The Python documentation on this is somewhat opaque; one way is using the descriptor protocol like this:

myObject.myBoundMethod = myUnboundFunction.__get__(myObject)

Whilst this is a common pattern in other languages it is seems to be generally discouraged in Python.

A final note: you'll notice that I said "Try it!" in a number of places above... Making a habit of doing this will lead you to a much better understanding of how Python works than any amount of discussion in a forum could do.

This advice is absolutely spot on, and does not apply just to Python. @fog37 you have posted threads on many different topics from which I infer that your preferred style of learning is to read a lot about a subject and then write a summary so that other people can check your understanding. May I suggest that it would be much more effective if you checked your understanding by trying things out: if it works, great, you have understood it well enough to use it (and the best way to reinforce and enhance your understanding is practice); if it doesn't work then you need to revise what you have learned and try it again; if it still doesn't work and you can't work out why, then is the time to start a thread on PF.

 

[PeterDonis]

it is probably worth mentioning that methods can be bound to instances

More precisely, what you appear to be describing here is a way to make a bound method on an instance from a function that is not an attribute of the instance's class. There is actually a less tricky way of doing it, using the types module:

>>> import types
>>> class Test:
...     pass
...
>>> def f(self):
...     return type(self).__name__
...
>>> test = Test()
>>> test.f = types.MethodType(f, test)
>>> test.f()
'Test'

You can also dynamically add the function to the class as an attribute, which makes it a method accessible to all instances of the class:

>>> class Test:
...     pass
...
>>> def f(self):
...     return type(self).__name__
...
>>> test = Test()
>>> Test.f = f
>>> test.f()
'Test'

Whilst this is a common pattern in other languages it is seems to be generally discouraged in Python.

Explicitly invoking dunder methods is generally discouraged in Python, yes. But often there are other ways of doing the same thing that don't require that, as shown in the above examples.

 

[fog37]

As the OP seems to be aiming at completeness it is probably worth mentioning that methods can be bound to instances. The Python documentation on this is somewhat opaque; one way is using the descriptor protocol like this:

myObject.myBoundMethod = myUnboundFunction.__get__(myObject)

Whilst this is a common pattern in other languages it is seems to be generally discouraged in Python.


This advice is absolutely spot on, and does not apply just to Python. @fog37 you have posted threads on many different topics from which I infer that your preferred style of learning is to read a lot about a subject and then write a summary so that other people can check your understanding. May I suggest that it would be much more effective if you checked your understanding by trying things out: if it works, great, you have understood it well enough to use it (and the best way to reinforce and enhance your understanding is practice); if it doesn't work then you need to revise what you have learned and try it again; if it still doesn't work and you can't work out why, then is the time to start a thread on PF.

Thank you. I really appreciate all your help and advice and will surely follow it more. I often try things but I then remain perplexed with interpreting the results (sometimes I get lost in the myriad details). The discussion on the forum really brings things home. Generally, after reading all your comments, reading the Python documentation becomes much less disorienting. It is also more fun to discuss things these days even if virtually. Hopefully all these threads can be useful to others who have similar dilemmas. Thanks again.