Is it possible to establish a secure connection using only client certificate
Speaking generally, not necessarily of TLS, the answer is “it depends”. If you consider a connection secure if it’s between two known parties, then no: with only one side authenticating (the client side), you don’t know if you’re talking to the real server or to an impostor. Same with HTTPS as it is usually deployed (without client-side certificate): the server does not know whom they are talking to, only the client knows they are talking to the right server.
If you don’t care about authenticating the server to the client, it doesn’t really matter whether TLS supports your specific setup: you can always let the client accept any certificate, which is equally secure to the server not presenting any certificate.
generate a token using A’s private key and demand B to prove that decrypt it using A’s public key
So it sounds like you do want mutual authentication.
Treating a public key as private is probably going to go wrong at some point. It reminds me of the fairy tale of Rumpelstiltskin, where nobody is allowed to know your name (public key). And in case of RSA, an attacker (having knowledge of neither the public nor private key) could submit any message to B and demand they ‘decrypt’ (
message^e mod n) it using A’s public key. So an attacker does not know A’s public key, but they can ask B to encrypt anything they want to send to A using that public key. It depends on the scenario if that is actually useful for anything, but the whole scenario feels wrong.
or what’s even simpler, just demand B to send back the A’s public key
An attacker could just capture the transmission once and replay it ever after. Are you assuming there is already an encrypted channel based on the client certificate? Because then you’re not using the TLS handshake but just building something inside the TLS tunnel.
I don’t know the entire TLS specification by heart to be able to tell you that this is supported by the protocol, or all features of a popular client to be able to tell you that a specific client cannot do this, but it seems doubtful. I don’t see a need to support this: your scenario cannot really exist, since if no data can flow from B back to A, there can be no handshake. Therefore, when you exchange the key from A to B via something like SSH, B can also communicate back to A. If you want mutual authentication and there is, at some point in time, an established encrypted tunnel that was mutually authenticated, then that would be the opportunity to exchange keys. And if you’re not looking for mutual authentication but only for “a secure TLS connection using only client certificate” (i.e. only one side is authenticated), then you can use TLS as normal but ignore the validity of the server’s certificate (so like