Thanks for the ideas!
So I poked around, running tcpdump on both the router and the nginx box at the same time, the router is seeing requests, but the nginx box does not most of the time.
Example tcpdump output:
23:41:18.512894 IP <remote ip>.55093 > <local ip>.www: Flags [S], seq 1511498285...