I've found what is causing the issue or what I believe that is causing it, which seems to be the caching code, but unhappily couldn't further look into it as I needed some answers, which I wasn't getting.
Regardless I've post what I've found here for whoever takes over it.
There is also a fix(or rather workaround) for those compiling the server them self.