RSA SHA2 256/512 key upgrade support libssh2 #536 - #626

I'm coming from issue #634.

This PR is about adding hostkey support for SHA2 and the problem I'm trying to fix is related but different in that I'm trying to add SHA2 support to RSA public key user authentication.

It is possible that this is the same problem and I just don't understand libssh2 well enough to see this. I apologise if that is the case.

I'm testing with willco007:master where public key authentication uses ssh-rsa as the public key method for RSA keys when LIBSSH2_RSA_SHA2 = 1 which makes OpenSSH 8.8 servers reject authentcation attempts. RSA is uncommon for hostkeys but quite common for user keys.

I'm working on this right now and need to sort out some assumptions where the public key method is always the first word of the public key at the moment. Hope to have something working soon.

Please let me know if this PR already supports SHA2 for user authentication. Perhaps I'm building it wrong.

I have something working that builds on top of your work in this PR: palmin@9cc16ee

I'm going to be beta testing this with the Working Copy and Secure ShellFish users.

@palmin let me dig back into this, it's been a while since I wrote this. Thanks for your interest in getting it going! :)

@palmin If I'm reading your changes correctly, you're basically doing this (from openssh):

The signature itself is made using the SSH signature algorithm and
encoding rules for the chosen key type. For RSA signatures, the
signature algorithm must be "rsa-sha2-512" or "rsa-sha2-256" (i.e.
not the legacy RSA-SHA1 "ssh-rsa").

By forcing the key to use rsa-sha2-256. However, your change won't be compatible with older servers which only support sha-rsa.

@willco007 You are correct.

My next steps are to follow the instructions for Discovery of Public Key Algorithms Supported by Servers in rfc8332.

This requires reading the list of supported public key algorithms with the "server-sig-algs" extension.

@palmin OK, sounds like you're on the right track. I was doing more research on this last night and upgrading the keys is certainly more involved than I originally thought. I was also going to look at implementing the extension RFC today and designing an API to get that information down into the backends.

Why not implement ssh-ed25519 and ecdsa-sha2-nistp521 support by the way?

@icebluey Libssh2 already supports both of those. OpenSSH has created a compatibility layer for existing RSA keys which improves security while allowing people to keep using already generated keys.

I'm now able to request and parse the server-sig-algs list in palmin@d43a8d4 and in palmin@1e66d2f upgrade_publickey_method() will only upgrade when server-sig-algs contains one of the relevant methods.

I tried to incorporate all your feedback in palmin@e180c06

I have started beta testing this in both my apps.

@willco007 I'm not able to find ext-info-c mentioned anywhere which makes me wonder if the servers will ever send the SSH_MSG_EXT_INFO packet.

This is the change I have in kex.c:
palmin@d43a8d4#diff-2060d014fa38220550b67c00d442ce6b1b40cbf351d106913a4fb55f35b4212fR3029-R3040

@palmin Oops, missed porting that over. Sort of the important bit! :)

@willco007 I have looked through the changed files and compiled willco007@9d1a3a5 where I can verify that rsa-sha2-512 is used.

Looks good.

@willco007 I have met servers where the hostkey type passed to hostkey_method_ssh_rsa_init() is rsa-sha2-256 or rsa-sha2-512 instead of the expected ssh-rsa causing negotiation to fail.

Here are some adjustments taking this into account: palmin@3579f21

@palmin good catch, will update PR.

Anything I can do to help merge this pull request?

Do we need additional people to review it?

Give that github is deprecating SHA-1 RSA today, 2022-1-11, and ssh-keygen's default is SHA-2 RSA, I hope that this feature can be added to a patch release soon.

@tsengjun Yup, I totally missed porting this over when moving from my private branch. I also missed sign_algo_prefs. I've submitted a PR and will land it shortly.

I setted in the .h files the LIBSSH2_RSA_SHA2 to 1 and LIBSSH2_RSA to 0 but it isnt working

@wastez The LIBSSH2_RSA_SHA2 and LIBSSH2_RSA flags are not something you normally change. These flags tell the library the crypto backend has implemented the feature. Unless you're implementing the RSA SHA2 backend you wouldn't change these values. Lastly, LIBSSH2_RSA_SHA2 relies on LIBSSH2_RSA being enabled.

If i enable both of them compilation runs into a error.
/usr/bin/ld: ../src/.libs/libssh2.so: undefined reference to '_libssh2_rsa_sha2_verify'
/usr/bin/ld: ../src/.libs/libssh2.so: undefined reference to '_libssh2_rsa_sha2_sign'
collect2: error: ld returned 1 exit status

What i´ve to do to use rsa-sha2-256 and rsa-sha2-512, sorry but i'm no coder.

Edit: Ok now i get it, just the support is implemented but the backend isn't implemented until now for the usage of rsa-sha2-256 or rsa-sha2-256.
For this my knowledge about coding is too weak.
Is somebody working on it?