This would be a valid point if the client source code wasn't available; you can build the app from source and sideload it onto your Android phone or verify [0] that the build available for your platform matches the code you've audited for compliance to the protocol. Granted I don't know if anyone's performed such an audit, but it's at least an option.

[0] https://core.telegram.org/reproducible-builds