Depends on the router, as long as you use a configurable one its much easier. I think your network is confused, it just doesnt have specific paths.
Looking at your setup it is seems really bad. Is your entire network in the same building? If it is you are using the wrong router. Although many here call me a mikrotik fan theres a reason why routers like mikrotik are very useful and this is one of them but they arent necessary. just much easier to do this with.
Im going to assume its all in the same building. You have 2 WANs and 2 LANs. The consumer router you have cannot handle 2 LANs, it doesnt have the configurative ability for it (i hope
@RMerlin can add LAN port VLANs and layer 3 segmentation to his firmware, if it is already there please correct me, i suspect his firmware can do it but by modifying files via SSH ). Essentially you want to connect both WANs to the router with CPU connected ports, have both modems only as bridge and have the router handle the layer 3 segmentation and routing so LAN 1 and 2 can communicate without port forwarding but at the same time using VLAN to prevent layer 2 between LAN 1 and 2 so that the popular DLNA will not work between LAN 1 and 2.
Because WAN 1 and 2 are different you have to do load balancing, there are many types and you shouldnt balance based on bandwidth. Balance WANs by using their latency and a bit of selective routing. If WAN 1 has lower latency to a particular server make a rule to route traffic meant for it through that WAN. If WAN 2 has lower internet latency, (your ping to ISP, google and core local datacenters and servers are lower) than make that the priority WAN. Essentially this is the purpose of BGP so you can have multiple WANs but everyone updates each other on what is the fastest route between places but ISPs do not simply offer that, you'd need some real expensive internet for it (still feel free to ask your ISP for BGP but your router must support it).
Some routers can learn, if given 2 routes with equal weightage they will use the faster one (the one with lower latency).
I know the concept of networking is that you can wire things about in the ways you want but there are optimal ways to do things and suboptimal ways to do things.
with RMerlin's firmware you can use IPTables to replace port forwarding if you only want LAN 1 and 2 to communicate only via the allowed ports/protocols. But this is only with the setup i suggested.
Edit: Just for a bit of humour ( you will need to be a networking guy to understand it) you can use your network setup. So if using a configurable router heres how you do it.
LAN 1 has your PCs, LAN 2 has your pi server and other modem that is doing NAT. So your router has WAN defined like normal for the modem connected to it. Now for the other network your router needs to be a part of it so give it an IP like a client, it should have a weightage of 0 just like LAN 1. Define the gateway of that LAN in the route to being your modem that is acting as a router.
Now with your modem that is also being a router define it with LAN 1, it requires an IP and set the router as gateway. Once this is done you have inter routing between LAN 1 and 2 but no load balancing yet. Now for the NAT rules, you have to define 2 NAT rules on both routers, 1 for each WAN to work for both LANs. If on a configurable router like mikrotik you can actually set the speed limit too if you want them to automatically use both in the sense of if NAT rule 1 is full than NAT rule 2 is used. This introduces a basic load balancing. Next you define your QoS and NAT hijacking rules. What this is for is after you've done the math in QoS you mark packets, and in hijacking simply change their destination. This is done for traffic within the router so instead of traffic going out at WAN 1 it can go out at WAN 2. Using QoS you can balance using priority by setting a buffer size or speed cap so extra traffic ignores that rule.
