Raspberry Pi Server Disconnection issue

[hitsuke]

So here is my problem,

I purchased a raspberry pi (model B) half a year ago to setup a mumble server. After getting things installed, working, and connected properly it all seemed like things were fine and dandy. However, I noticed a few days later that users would disconnect and reconnect less than a second after. This all happened at random intervals and I had no idea what was causing the problem. At one point I had to switch modems and needed to change all appropriate information to get my server running again. The problem only worsened after that point and the disconnects became more frequent. I've tried doing things like getting my users to force TCP mode,disabling QoS, putting the IP of the server in DMZ, disabling DoS protection on my router, disabling SIP ALG. For a few hours each would seem to have solved the problem, but soon after someone would disconnect and reconnect regardless of what they did. I stumbled upon a forum post detailing a similar issue to mine and the suggestion was that it might have to do with the NTP. I found that my time was off and fiddled with the settings to make sure the date was in sync. For a short time it seemed like that fixed it as well, but bad news followed.

Today, when users tried connecting to the server they would remain on for about 10 seconds and then get disconnected. No reconnects this time around, and I'm at a loss of what I need to do in order to fix everything. Any help would be appreciated. Thanks