Why? Aren't its config files under ~/.config/?
~/.cache/ is just that, cache, and not configuration, like ~/*, ~/.config/* and ~/.local/*.
Bar some very specific cases, like cracauer@'s LLM example or gstreamer as you mentioned, things like fontconfig and mesa_shader_cache also don't have to be...