The degree to which the characteristic polynomial fails to charcterise similarity can be illustrated, for the case where $R$ is a field, by the rational canonical form, which in this case does characterise similarity. The rational canonical form is determined by a uniquely defined sequence of monic polynomials (invariant factors), each one a multiple of any one of polynomials that follow (I prefer this to the more common requirement of dividing what follows); the sequence can have any length, but it can be thought of ending with an indefinite repetition of the constant polynomial $1$ (just like partitions of an integer are weakly decreasing sequences of integers that can be thought of as ending with endless occurrences of $0$). The first invariant factor (in the order I chose) is the minimal polynomial, and the product of all invariant factors is the characteristic polynomial.
Now if you fix the characteristic polynomial, its irreducible factors in $R[X]$ are determined; all of them must occur as a factor of the minimal polynomial. Indeed the multiplicity of a fixed irreducible polynomial $P$ in the invariant factors must be weakly decreasing, so these multiplicities form a partition of the multiplicity of $P$ in the characteristic polynomial. Choosing a partition for each occurring irreducible factor $P$ is exactly the freedom one has for choosing a rational canonical form, and therefore determines the number of similarity classes of transformations with a given characteristic polynomial.
The "regular" case is where the minimal polynomial is equal to the characteristic polynomial; this is forced when the characteristic polynomial is squarefree. In this case the centraliser of $\alpha$ (in the endomorpism algebra) is equal to the set of polynomials in $\alpha$, which is of dimension $n$ (the same as that of the vector space on which $\alpha$ acts). I think that in all other cases the dimension of the centraliser is strictly larger than $n$, whence the similarity class is smaller than in the regular case; this intuitively explains why there aren't any polynomial functions that can separate the "singular" cases from the regular case with the same characteristic polynomial.