Hello,
Let's assume a polygonal of n vertices.
Starting from of vertice we can reach n-1 vertices by 2 must be excluded.
So there are n(n-1-2) possibilities but each possibility is counted twice
Number of diagonals = n(n-3)/2
Examples:
n=4==>#=4(4-3)/2=41/2=2
n=5==>#=5(5-3)/2=52/2=5
n=6==>#=6(6-3)/2=63/2=9
n=17==>#=17*(17-3)/2=119.