It doesn't really have anything to do with Qt, and we don't really show Qt widgets per se.
We inject into the systems DLL loading and on loading DLLs check whether they are 3D accelerated applications (using DirectX, OpenGL...), and whether we want to show an overlay. If so, we inject our own functions into the original DirectX functions, replacing the original code with a jump to our own code, preserving what we replaced.
Then we use a Pipe for communication between the Mumble client and the overlay that was injected into other applications.
Refer to the overlay folder
for the overlay specific code and src/mumble/overlay.h
for the client specific stuff.
* Checking whether to inject
You may want to check out the "minhook" library for code injection.
If you were talking about the interactive mode where we actually draw the Mumble client itself in the overlay, I'm not too sure about that, and I think we removed it in the current codebase because it apparently had some open issues and no maintainer. You'd have to check for the "interactive mode" flag and code paths.