Mumble Client & Server SDK

[HeroCC]

Hello World,

I am a new user here so I am likely not as informed as all of you, but I just had a question. Does anyone know of a good C++ Mumble Client (and Server) library / SDK? I am working with a team on an IRL research project and we were evaluating different communications technologies, but each has it's disadvantages; Radio (which we are using now) can't be used programmatically, TS3's SDK requires extensive licensing, and Discord doesn't allow you to host your own server. After evaluating all of these we settled on Mumble, as it is open source, has positional audio, and , and began looking to build an application to link our software to it. However, after scavenging around the internet, I was unable to find a comprehensive C++ SDK for Mumble. The ones I did find were feature incomplete, undocumented, or hadn't been touched in years:

• https://github.com/mumble-voip/libmumble: Official library, but strange build system (that won't compile), and 4+ years untouched
  
• https://github.com/pcgod/libmumbleclient: 8+ years old, no documentation
  
• https://github.com/HeroCC/mumlib: Right now this one looks the best, but is 3rd party and not documented, untouched for 2+ years
  

 

I realize a good SDK is hard to come by in any project, but I would like any advice you could give me. Even a headless mumble client with good RPC / APIs could work, but I can't find those either.

Thank you for your help Mumble Community, have a nice one!

[kissaki]

Hello HeroCC

The Mumble protocol has been stable for years, so no changes on libmumble does not necessarily indicate it is outdated or not working.

Sadly our main codebase is not cleanly split into a lib of underlying code and head.

We do not have an SDK as development on the source itself can be done just like that, on the source itself.

We do have protocol documentation. The protocol uses protobuf, so implementing clients can generate networking code just like that.

* https://mumble-protocol.readthedocs.io/

* https://wiki.mumble.info/wiki/Protocol

* https://github.com/mumble-voip/mumble/blob/master/src/Mumble.proto

Mumbles source code uses the Qt framework, so you can use Qt to build Mumbles sources. https://github.com/mumble-voip/mumble

Depending on your system/OS you may find our mumble-releng repository useful, which can prepare a build environment for you. Specifically for Windows. It will download the depenencies and compile them. See https://github.com/mumble-voip/mumble-releng

For a server it may be viable to just use an existing one.

If you want to implement a client, I would suggest trying out libmumble or implementing your own client (generating networking code from protobuf, implementing audio and control handling yourself).

I don't really have an overview of what options are out there; implementations you may refer to or libraries you could use. But our list at 3rd Party Applications may help you https://wiki.mumble.info/wiki/3rd_Party_Applications

[HeroCC]

Hi Kissaki, thanks for the reply!

I was unable to compile libmumble in my testing (it may be my computer) but I did find a simple library (mumlib) that works with some tweaks. I did more research and yeah the Protobuf is quite useful, I didn't realize at first that the protocol was so open.

I think I am going to implement the client with a library, and run `murmurd` controlled by gRPC (I made a PR to enable this but I am having trouble navigating the Mumble Build Systems). Installing Ice from pacports is broken so I can't use that for RPC unfortunately.

One quick question, is the positional audio handled on the server or the client? Would I need to do anything special to implement it?

Thanks for writing this great piece of software!

[kissaki]

Positional Audio is handled entirely client side. The server just forwards the packets, in the case of PA including some additional positional metadata.

[Pawel]

Hi,

 

 I would like to integrate mumble voice client into my C++ application. I was able to build libmumble, but attached examples of client and server are quite simplified. Am I right that example client in examples/ExampleClient directory just connects to server and receives version ? Is there some working example of custom client ? Or should it be modeled after real client (https://github.com/mumble-voip/mumble/tree/master/src/mumble ?) ?

 

Documentation is not very extensive, thanks in advance for any help and for great product.

 

Pawel

[Krzmbrzl]

Yes the examples in libmumble are very limited at the moment. However, libmumble itself is not yet API stable anyway, so creating a lot of detailed examples wouldn't make sense at this point. That's also why there isn't a lot of documentation yet - it's currently more in a Beta state.

 

On 4/23/2023 at 9:36 PM, Pawel said:

Or should it be modeled after real client

That doesn't matter. Both use the same principles and eventually we'll use libmumble inside the real client as well.

[Pawel]

8 hours ago, Krzmbrzl said:

That doesn't matter. Both use the same principles and eventually we'll use libmumble inside the real client as well.

Thanks a lot for answer, but:

- does it mean that real client does not use libmumble now ?

- am I right that libmumble example client (examples/ExampleClient directory) is very limited and it really only makes network connection to the server ? According to messages, it receives version and I think nothing more happens. Looking at the protocol documentation (https://mumble-protocol.readthedocs.io/en/latest/establishing_connection.html) this seems to be the first phase of handshake.

- I am digging into the real client, there is a lot of code there, including Qt/GUI-related stuff. Could you just point me, very briefly and perhaps just in several points, what is required to run working client ? As I said my custom client will be a part of the application, so no own GUI related stuff is needed. Now it seems to me that I should pay attention to the following classses: ServerHandler, AudioInput, AudioOutput, (Global seems to be configuration container). What is the basic flow/architecture ?

- I would like to be able to connect, disconnect, join and quit some defined room/channel - it shoulf be possible, looking at the real client functionality.

 

Thanks a lot in advance for any help and sorry for this questions, but it is not easy to navigate in this and sometimes a few hints from someone knowledgeable may have dramatic impact on tasks like this.

 

Regards,

Pawel

[Krzmbrzl]

2 hours ago, Pawel said:

does it mean that real client does not use libmumble now

yes

 

2 hours ago, Pawel said:

am I right that libmumble example client (examples/ExampleClient directory) is very limited and it really only makes network connection to the server

yep, that's all it does at the moment (plus keeping the connection open)

 

2 hours ago, Pawel said:

I am digging into the real client, there is a lot of code there, including Qt/GUI-related stuff. Could you just point me, very briefly and perhaps just in several points, what is required to run working client ? As I said my custom client will be a part of the application, so no own GUI related stuff is needed. Now it seems to me that I should pay attention to the following classses: ServerHandler, AudioInput, AudioOutput, (Global seems to be configuration container). What is the basic flow/architecture ?

Have a look at https://github.com/mumble-voip/mumble/blob/master/docs/dev/TheMumbleSourceCode.md - that should be a good starting point. So in a nutshell: yes the classes that you listed are probably the most important ones for you.

As a side-note: I think it'll probably be less work to adapt libmumble and then refactor your code when the API changes instead of rewriting the entire protocol- and network-level communication and handling yourself. So if you have to read source code anyway, I would suggest looking into libmumble for how to use it. And feel free to create an issue in its repository if you have questions about it.

 

2 hours ago, Pawel said:

I would like to be able to connect, disconnect, join and quit some defined room/channel - it shoulf be possible, looking at the real client functionality.

It sure is. For connection you have to perform the respective handshake, for disconnecting you only have to close the connection and moving the user around can be achieved through UserState messages (see the respective functions inside the ServerHandler class).

 

2 hours ago, Pawel said:

Thanks a lot in advance for any help and sorry for this questions, but it is not easy to navigate in this and sometimes a few hints from someone knowledgeable may have dramatic impact on tasks like this.

Nothing to be sorry for. The main Mumble source code unfortunately still is quite messy at times (we are working on it though) and since libmumble isn't documented at all yet, it is clear that it is not obvious how it should be used (and what it is capable of).

Happy to help!

[Pawel]

OK, I focused on libmumble example client. Example client seems to finish on receiving version from server. According to protocol description (https://mumble-protocol.readthedocs.io/en/latest/establishing_connection.html) this should be followed by user authentication. I created Authenticate structure and sent it through connection (once only) - it seems the user was authenticated and it appeared in remote client as connected and joined channel.

Examining Message::type(pack) I see that later my client gets the following messages from server:

CryptSetup
CodecVersion
ChannelState
PermissionQuery
UserState (x 2)

ServerSync
ServerConfig
 

which more or less conforms to protocol documentation.

 

Then, when I talk to remote client, I see that my client gets UDPTunnel messages, presumably with voice data ?

This is big step forward, I think.

How should I handle incoming messages ?

Should I decode somehow messages from Pack to proper Message derivatives ? How should I do it ?

Thanks for help so far.

[Krzmbrzl]

On 5/1/2023 at 5:09 PM, Pawel said:

Then, when I talk to remote client, I see that my client gets UDPTunnel messages, presumably with voice data ?

yeah. that'd indicate that your UDP connection to the server is currently unreliable so the server decides to instead send you audio through TCP

 

On 5/1/2023 at 5:09 PM, Pawel said:

How should I handle incoming messages ?

That depends on what you want to do. And what messages in particular you are talking about

 

On 5/1/2023 at 5:09 PM, Pawel said:

Should I decode somehow messages from Pack to proper Message derivatives ? How should I do it ?

This would be a question for @davidebeatrici

[davidebeatrici]

From https://github.com/mumble-voip/libmumble/blob/4c1b92064017cd3b7c2f0e13d0fcdce79479bcfd/examples/ExampleServer/Node.cpp#L177-L180:

 

Message::Authenticate auth;
if (!pack(auth)) {
	break;
}

[Pawel]

> That depends on what you want to do. And what messages in particular you are talking about

Sorry for not being precise enough. I meant these UDPTunnel messages. They are clearly correlated with my voice talking to 2nd client, do I guess this is voice data. How should I handle it ? I take a look at ServerHandler.cpp in main distribution.

There is a method there

 

void ServerHandler::message(Mumble::Protocol::TCPMessageType type, const QByteArray &qbaMsg)

 

it seems to be called with type and byte array. If type is Mumble::Protocol::TCPMessageType::UDPTunnel, then it is decoded by m_tcpTunnelDecoder and

passed to

 

void ServerHandler::handleVoicePacket(const Mumble::Protocol::AudioData &audioData)

 

where it is directed to AudioOutput.

Am I right that this is handling of incomming voice data ?

 

How shoud I obtain this byte array with voice data ? I have Pack &pack and Message::type(pack) is Type::UDPTunnel.

In libmumble, I see Audio structure in udp::Message namespace, but pack(msg) with it fails to compile. What should be the target message type ?

 

Also, m_tcpTunnelDecoder is of type Mumble::Protocol::UDPDecoder< Mumble::Protocol::Role::Client >. These decoders are in main source distribution but I don't see something similar in libmumble.

 

Sorry for stupid questions, but it seems that regular client and libmumble differ a bit.

[Krzmbrzl]

On 5/8/2023 at 7:09 PM, Pawel said:

Am I right that this is handling of incomming voice data ?

yes

 

On 5/8/2023 at 7:09 PM, Pawel said:

How shoud I obtain this byte array with voice data ?

the way Davide sowed, I suppose - only with the UDPTunnel message type.

 

On 5/8/2023 at 7:09 PM, Pawel said:

In libmumble, I see Audio structure in udp::Message namespace, but pack(msg) with it fails to compile. What should be the target message type ?

the tunnel message is sent via tcp and thus has to be parsed as such

 

On 5/8/2023 at 7:09 PM, Pawel said:

Also, m_tcpTunnelDecoder is of type Mumble::Protocol::UDPDecoder< Mumble::Protocol::Role::Client >. These decoders are in main source distribution but I don't see something similar in libmumble.

That's the implementation of the new audio format (to-be introduced with Mumble 1.5), which afaik is not yet implemented in libmumble.

[Pawel]

On 5/11/2023 at 5:49 PM, Krzmbrzl said:

the way Davide sowed, I suppose - only with the UDPTunnel message type.

the tunnel message is sent via tcp and thus has to be parsed as such

Hi, I did something like:

 

    Opus::Decoder decoder(1);
    Code code = decoder.init();

   // ....

 

const auto type = Message::type(pack);

else if(type == Type::UDPTunnel)
 {

    Message::UDPTunnel msg;

    if(pack(msg))

    {

                 const BufViewConst bfv = pack.data();
                 printf("  B.size: %lu\n", bfv.size());
                 printf("  B.data: %p\n", bfv.data());       

 

                 printf("  packet channels:   %d\n", decoder.packetChannels(bfv)); // 1
                 printf("  packet frames:     %d\n", decoder.packetFrames(bfv));  // returns 1
                 printf("  samples per frame: %d\n", decoder.packetSamplesPerFrame(bfv, 1)); // returns 0
                 //printf("  packet samples: %d\n", decoder.packetSamples(bfv)); // CRASHES
                 printf("  packet samples:    %d\n", Opus::packetSamples(bfv, 1)); // returns 0

    }

}

 

Is it correct ? Do bfv.size() and bfv.data() give us access to incoming voice data buffer ?

Should we decode it using

 

IntegerView operator()(const IntegerView out, const BufViewConst in, const bool decodeFEC = false);

 

When I tried to do:

 

Opus::Decoder::IntegerView iv = decoder(iv, bfv);

 

it crashed (invalid read of size 8, in valgrind, I am on Linux ).

And after decoding (how ?) should it be sent to some output ?

 

Thanks in advance,

Edited May 23 by Pawel

[Krzmbrzl]

@davidebeatrici^

[davidebeatrici]

tcp::Message::UDPTunnel msg;
udp::Message::Audio audio;

if (pack(msg) && msg.pack(audio)) {
	const auto nSamples = decoder.packetSamples(audio.opusData);
	if (nSamples) {
		std::vector< float > floatData(nSamples);
		const auto floatDataWritten = decoder(floatData, audio.opusData);

		std::vector< int16_t > integerData(nSamples);
		const auto integerDataWritten = decoder(integerData, audio.opusData);
	}
}

 

For reference: https://github.com/mumble-voip/libmumble/pull/15

[Pawel]

Thanks a lot but msg.pack(audio) is false for me - only pack(msg) is true, so the above condition does not work at all. Are you sure it is correct ?

Also, assuming it works, we get 2 vectors of samples - float and integer. Do we need both or is it just a demonstration that decoder may return samples in various formats ?

And finally, if we have samples, I guess we should send them somehow to local output sound device.