That is what happens.
ssh client started on "my client system" to remote system with -X and -Y. You wind up at a shell. If you echo $DISPLAY you wind up with something like "localhost:10.0". Keep in mind that is on the REMOTE system. Now on that remote system you start an application, that does X. Remote system, X-application runs, discovers $DISPLAY and sends X stuff to $DISPLAY. $DISPLAY on the remote system for this ssh session sends things to one end of an ssh tunnel. Data flow over ssh tunnel and is displayed on the system the ssh client session originated from "my client system" in this example.
So yeah, if you ssh client with -X -Y don't try to muck with the value of DISPLAY on the remote or local system unless you know what you are doing. ssh magic does the right things.