- Keras Reinforcement Learning Projects
- Giuseppe Ciaburro
- 291字
- 2025-04-04 16:10:17
Transition diagram
A very intuitive alternative to the description of a Markov chain through a transition matrix is associating an oriented graph (transition diagram) to a Markov chain to which the following two statements apply:
- Vertices are labeled by the S1, S2,…, Sn states (or, briefly, from the indices 1, 2, …, n of the states)
- There is a directed edge that connects the Si vertex to the Sj vertex if and only if the probability of transition from Si to Sj is positive (a probability that is in turn used as a label of the edge itself)
It is clear that the transition matrix and transition diagram provide the same information regarding the same Markov chain. To understand this duality, we can look at a simple example. Say that we have a Markov chain with three possible states—1, 2, and 3—and the following transition matrix:
The following diagram shows the transition for the preceding Markov chain. In this diagram, there are three possible states—1, 2, and 3—and the directed edge from each state to other states shows the transition probabilities pij. When there is no arrow from state i to state j, it means that pij=0:
In the previous diagram, we can see that the arrows that come out of a state always sum up exactly at 1, just as the values of every row in the transition matrix must add up exactly to 1, which represents the probability distribution. From the comparison between the transition matrix and transition diagram, it is possible to understand the duality between the two resources. As always, a diagram is much more illustrative.