It is the shape of the incisal edges of individual anterior teeth that helps to determine the Smile Style.
Specifically two areas are involved, the shape of the incisal edge and also the contour where that edge meets the mesial or distal to form the embrasure.
Details are illustrated below.

Masculine Incisal Contour Characteristics
The above illustration displays the extreme of the masculine style smile, called the Vigorous Style. It is characterized by flat or sharp (canine) incisal contours. Anterior embrasures are angular and quite sharply defined.

The overall effect is a broad smile with relatively flat incisors. Teeth are not individualized.

*as an extreme smile characterization although definitely masculine, this is not a particularly commonly used smile style.

Feminine Incisal Contour Characteristics
The above illustration displays the extreme of the feminine style smile, called the Oval Style. It is characterized by rounded incisal contours. Anterior embrasures are open, softly contoured and individualize each anterior tooth.

The overall effect is a complex smile where attention is drawn from tooth to tooth. This is also a interesting and youthful smile.

*although this is an extreme characterization, this is a commonly used style for females.