Some data after 12 hours of no fan. Temp: avg 62 C (~ 37 C with fan), Offset avg 2.9 ms (0.09 ms with fan), drift avg 8.46 ppm (4.4 ppm with fan) From that very limited data sample, it looks like NTP offset and drift get worse with CPU temp. Again, very little data and lots of possible unknown causation. Also, the cyclical pattern in the drift graph was still there with the absolute values of the max and min for each cycle being greater than that of the system with a fan.
I think this is another case of the clock drift not being associated with the CPU temperature, but rather the board temperature. The problem is that the board temperature is not directly measurable.
From the drift graph that was posted earlier, a +/- 2ppm drift in response to ambient temperatures changing seems about right.
From the material I have read, your average crystal oscillator will drift about 1ppm per degree Celsius.
My old PCs running NTPD used to run at an average drift of about 20-30ppm, which would certainly correspond with that rule of thumb.
So I reckon that even with the ambient temperature changing by about 10 degrees between morning and night, this probably only translates into a change of board temperature of a couple of degrees, which leads to a corresponding fluctuation in the drift.
Adding the fan to the router may drop the CPU temperature by many tens of degrees, but likely is not dropping the board temperature a whole lot, as the board will already be far closer to ambient, than the CPU is.