At least for my purposes everything is fine now! It turned out that one of Cloudflare's DNS servers is in the same location as my ISP.
I noticed this after switching from Cloudflare to Quad9.
@eibgrad
Thanks for your input on this, I'll keep an eye on it. I have already noticed some of these...