I 'll try to explain to the best of my powers what's wrong. This is a very strange situation. The site to site vpn works as expected. No complaints there. I just need to fine tune it due to my special circumstances.
If you are not familiar with Apple's ecosystem in regards to smart home, Apple TV can work as a homekit hub. That means, it speaks with compatible smart devices in local network, passes the status of these devices to my rest iOS devices and controls these smart devices when my iOS device is not present in LAN.
Homekit hubs may be either in primary or stand by mode. Someone may have multiple Apple TVs, Homepods, iPads at home. All of these can operate as hubs for the smart home, but only one can be the active and the rest will be on stand by mode.
Returning back to my case, it looks my Apple TV of Location-A has found through the site-to-site VPN that another homekit hub exists in Location-B. In that respect, both Apple TVs are now set us as hubs for both Smart Homes.
This situation leads to all kind of problems. For example, Apple TV physically located in Location-A went into standby status and the Apple TV of Location-B took over as a primary hub for Location-A. If both hubs where under the same WAN, I think there shouldn't be any issues, but they are not and when that switch happens, I cannot control my devices.
I can avoid all this, if I could simply restrict some IPs from exchanging packets with the VPNs...
Sorry for my long message.