If you want BPM windowed so that you can adjust the size then go to Menu -> Settings -> Display -> "Big Picture Mode" -> "Start Big Picture Mode Windowed." Turning the option on should allow BPM to appear in a resizable window. Deactivated, BPM should show full-screen.

I was trying to replicate your problem with BPM set to full-screen in the options by starting a game that was set to a resolution that was lower than that of my Desktop. Alt-tabbing to the Steam client made it appear in a fixed window like the one you you linked. I then pressed Alt-Enter to return Steam to desktop mode and then clicked the "Enter Big Picture Mode" button on the top right. This returned the client to full-screen BPM. Thereafter I tried to repeat everything I had done previously to replicate the problem but Steam would not appear as before in a window of fixed size. Don't know if Alt-Enter/"Enter BPM" fixed the problem, but if it did it is admittedly not an ideal solution.

Ahh I didn't try toggling windowed mode. Did that just now but still having the same issue. Did a reinstall of the steam client and still having the same issue as well. Thanks for the info!

Following up on this:

Microsoft pushed an update for Windows 11 late May which started making HDR + Dolby Vision act wonky on my LG C2 OLED TV (details: https://www.reddit.com/r/OLED_Gaming/comments/1lgfmml/a_fix_for_dolby_vision_in_windows_11_being_forced/)

I haven't been able to land a solution that doesn't involve me having to manually toggle that Dolby checkbox in my display settings every time I wake or startup my machine, so I just turned off HDR completely.

The tie into my OG post is that the overlay in big picture mode is once again taking up the entire screen as intended, so switching off HDR seemed to do the trick.

win+alt+b is a combo you can use to toggle HDR w/o having to dive into settings (if you have Game Bar enabled in Windows 11).