Workaround: ./configure --with-gnutls=no

log: level: 0, area: 8, connection encryption active
log: level: 0, area: 8, Sending xml string... SK
GnuTLSBase::encrypt: <?xml version='1.0' ?><stream:stream 
to='debxwoody.de' xmlns='jabber:client' 
xmlns:stream='http://etherx.jabber.org/streams'  xml:lang='en' 
version='1.0'>
GnuTLSBase::encrypt - gnutls_record_send
GnuTLSBase::encrypt done
log: level: 0, area: 262144, <?xml version='1.0' ?><stream:stream 
to='debxwoody.de' xmlns='jabber:client' 
xmlns:stream='http://etherx.jabber.org/streams'  xml:lang='en' 
version='1.0'>
log: level: 0, area: 8, Sending xml string...done SK
log: level: 2, area: 8, handleHandshakeResult() - Done
GnuTLSBase::handshake() true done
ConnectionTCPClient::recv Data: D6D700A0
ConnectionTCPClient::recv Data:
ConnectionTCPClient::recv timeout = -1
dataAvailable
ConnectionTCPClient::recv: 79 size
Calling handleReceivedData()
ClientBase::handleReceivedData
ClientBase::handleReceivedData - decrypt
GnuTLSBase::decrypt
GnuTLSBase:  gnutls_record_recv 17000
GnuTLSBase:decrypt() None data? -28
Fehler: Resource temporarily unavailable, try 
again.ConnectionTCPClient::recv Data: D6D6F170
ConnectionTCPClient::recv Data:
ConnectionTCPClient::recv timeout = -1
dataAvailable
ConnectionTCPClient::recv: 557 size
Calling handleReceivedData()
ClientBase::handleReceivedData
ClientBase::handleReceivedData - decrypt
GnuTLSBase::decrypt
GnuTLSBase:  gnutls_record_recv 17000
GnuTLSBase:decrypt() None data? -28
Fehler: Resource temporarily unavailable, try 
again.ConnectionTCPClient::recv Data: D6D7C640
ConnectionTCPClient::recv Data:
ConnectionTCPClient::recv timeout = -1


I changed the implementation:

       ret = static_cast<int>( gnutls_record_recv( *m_session, m_buf, 
m_bufsize ) );
       if(ret == GNUTLS_E_AGAIN) {
         ret = static_cast<int>( gnutls_record_recv( *m_session, 
m_buf, m_bufsize ) );
       }

I looks littel bit better:

<challenge xmlns='urn:ietf:params:xml:ns:xmpp-sasl'>.....
Processing SASL challenge
...

GnuTLSBase:  gnutls_record_recv 17000
log: level: 0, area: 8, ClientBase::handleDecryptedData
ClientBase::parse
Parser::streamEvent
log: level: 0, area: 8, handleTag
log: level: 0, area: 131072, <failure 
xmlns='urn:ietf:params:xml:ns:xmpp-sasl'><malformed-request/><text>Invalid 
channel binding value.</text></failure>
log: level: 0, area: 4, handleNormalNode
log: level: 0, area: 4, failure
log: level: 2, area: 4, SASL authentication failed!
log: level: 0, area: 8, Sending xml string... SK
GnuTLSBase::encrypt: </stream:stream>

prosody 0.11.2

With which XMPP server software does this happen?

On a Debian stretch it is working, but not on Debian buster.

I'm not sure, but maybe there are changes in the TLS Version or 
implementation.

File: tlsgnutlsbase.cpp line 78 (int GnuTLSBase::decrypt( const 
std::string& data ) )
It seems on my env there is a return of GNUTLS_E_AGAIN, which is not 
handled correctly.