This is 90s thinking. Why must terminal emulators only be text and only do things that a physical terminal could?
I keep trying to imagine what abandoning TTY interfaces in Linux would look like and I can’t comprehend the rework that would be required. It’s so fundamentally different.
For example, how would the SSH protocol work? How would that be compatible? Would we have to abandon SSH or always X forward?
There is definitely a pressure to extend beyond standard TTY. Tmux captures mouse action and has a window management system. fish shell has autocomplete. But both of these still use the same medium of text.
I may simply lack imagination.