We feel that our primary dilemma involved the concept of ‘spectral leakage’. Because a finite signal is essentially a continuous function multiplied by a square pulse, its corresponding frequency representation is actually the CTFT/DTFT convolved with a sync function. When the signal content is located at integer multiples of the sampling frequency / N, this is not a problem – we see the peak as an impulse, and zeros elsewhere, as shown below in the picture from Lyons [2]. However, when content is located at non-integer multiples of the bin size, we see ‘leakage’ of the spectral energy into adjacent bins. While using a windowing function can minimize the sidelobes of the sync, and improve our ability to distinguish separate frequencies, it does not allow us to recover the ‘true peak’ at the frequency of interest. Example windows are shown below, also from Lyons [2].

This can be particularly troubling with narrow band signals such as ours. Because the data was sampled at 97.656KHz, which has few factors, there was no good sub-window size that would enable us to have a bin at an integer multiple that landed on 6.4KHz. When taking into account the fact that typical magnitudes of the distortion product are barely above the noise floor, and the observed amplitude decay typically spans only ~5 dB, it is necessary to sample at a rate that allows the user to hit the ‘peak’ at an integer multiple of bins to ensure accurate results.