Is there an interesting way to define a continuous composition of functions?

People have found ways to extend the definition of some operations that are ostensibly discrete (such as differentiation - e.g. 1st, 2nd, 3rd derivatives) to operations that are defined for fractions ( e.g. fractional derivatives). Is there an interesting way to extend the operation of composing two functions to define such a fractional operation?

There are trivial ways. For example, the 0-th composition of f with g could be f(x). The 1st order composition could be f(g(x)). The 1/3 rd order could be (1/3) f(x) + (2/3) f(g(x)). But one could make a similar definition for almost any discrete operation and that sort of thing isn't very compelling.

In a recent thread on nested functions, it has been pointed out that there are many ways to define sequences of functions recursively such that the sequences converge to the identity function. If you visualize the sequences in reverse (so to speak) they go from x to some function g(x). Apply f() to such a sequence would go from f(x) to f(g(x)). However, I don't understand how to apply these methods to find a sequence that begins at an arbitrary g(x) and tends toward the identity function.

 

https://www.physicsforums.com/showthread.php?t=539396&highlight=Nested

 

Those links pursue the rather natural idea that the k-th iterate of a function should be analogous to the kth power of a variable. So, for example, we define [itex] f^{[1/2]}(x) [/itex] to be a function [itex] r(x) [/itex] such that [itex] r(r(x)) = f(x) [/itex].

Now: how to apply that to the composition of two different function?

Suppose I want the 0th order composition of f with g to be [itex] f \circ_{[0]} g = f [/itex] and the 1-th order composition of f with g to be [itex] f \circ_{[1]} g = f(g(x)) [/itex].

One possibility is to define [itex] f \circ_{[k]} g = f( g^{[k]}(x)) [/itex].