Let \(f:\Delta\to\Delta\) be analytic and let \(f^n=f\circ\cdots\circ f\) denote the \(n\)-fold iterate. Show that either \(f\) has a fixed point in \(\Delta\), or \[ |f^n(z)|\longrightarrow1 \] for every \(z\in\Delta\).
Solution
1. If \(f\) is conjugate to a rotation, then \(f\) has a fixed point in\(\Delta\).
Suppose \[ f=A^{-1}\circ R_\theta\circ A, \qquad R_\theta(z)=e^{i\theta}z, \] for some \(A\in\operatorname{Aut}(\Delta)\). Since \(R_\theta(0)=0\), the point \[ p=A^{-1}(0)\in\Delta \] satisfies \(f(p)=p\).
2. Suppose \(f\) is not conjugate to a rotation. Then there is\(p\in\overline\Delta\) such that\[ f^n(z)\longrightarrow p \]for every \(z\in\Delta\).
This is the Denjoy–Wolff theorem.
3. If the point \(p\) from 2 lies in \(\Delta\), then it is a fixed pointof \(f\).
For any \(z\in\Delta\), \[ f^{n+1}(z)=f(f^n(z)). \] Letting \(n\to\infty\) and using continuity of \(f\) gives \[ p=f(p). \]
4. If \(f\) has no fixed point in \(\Delta\), then\[ |f^n(z)|\longrightarrow1 \]for every \(z\in\Delta\).
By 1, such an \(f\) cannot be conjugate to a rotation, so 2 applies. By 3, its Denjoy–Wolff point cannot lie in \(\Delta\). Hence \[ p\in\partial\Delta, \qquad |p|=1. \] Therefore \[ |f^n(z)|\longrightarrow|p|=1 \] for every \(z\in\Delta\), as required.