I had this error this morning, it has something to do with later versions of Java disabling support for SSLv3, I know there have been some serious SSL vulnerabilities of late so it may be related to that.
Anyway, it seems that in order to get the remote KVM console redirection working with DRAC 5, you have to enable SSLv3. To do this find the java.security file on your system and edit it. I found it in C:\Program Files (x86)\Java\jre1.8.0_66\lib\security.
Search for the line that starts “jdk.tls.disabledAlgorithms” and comment it out by putting a hash at the start. Save the file. The line ended up looking like this in my java.security file:
#jdk.tls.disabledAlgorithms=SSLv3, RC4, DH keySize < 768
Now try your remote console again and hopefully it’ll work.