I'm using Chrony and my drift has a very discernible pattern, both daily and over the long run. The graph shows that each day the drift reaches a low point in the afternoon and climbs back higher in the evening and overnight hours. This pattern repeats every single day. It is also clear to see the upward trend over the long term. I'm dubious that this is due to hardware, but I can't figure what would give such a clear pattern to the drift value.