Public-key authentication is often the next step in improving the security of a system that is protected by passwords. It addresses some of the security concerns with password authentication.

Public-key authentication is based on the use of public-key (asymmetric) cryptography. In public-key cryptography, messages are encrypted and decrypted with different keys. This means that each entity (person or device) that uses public-key cryptography has a key pair that consists of a public key and a private key.

Private keys are secret and known only to their owners. They are protected by a passphrase and can be stored on separate hardware cryptographic devices such as smart cards. Private keys are used for proving the identity of the entity.

Public keys are, as the name implies, public and should be distributed to all hosts with which the entity wants to communicate securely.

The two keys are mathematically dependent but the private key cannot be derived from the public key. Furthermore, the two keys possess a distinct quality: data encrypted with the public key can only be decrypted with the private key (and vice versa).

Before public-key operations can be made, the public key of the other entity has to be received securely, so that no one can substitute the genuine key with a tampered one. In basic public-key authentication (without certificates), an entity (for example, a user) can simply send its public key via e-mail to the other entity (for example, an administrator of the remote host). The receiving entity must then verify by using an out-of-band method (for example, a telephone call) that the public key is correct and the entity really is who it claims to be. Every public key has a unique fingerprint that can be used in verification.

This initial verification of the identity is very important as it is the basis for all subsequent security policy decisions. If the identity of the entity who sent the public key and the validity of the received public key are not verified, the wrong entity might be trusted. This process of proving the initial identity is called identity establishment.

If the receiving entity is a server host, the administrator can, for example, configure the server to associate the public key with a specific user account. This could be done by copying the public key to a special directory in the user's home directory, or by maintaining a database of usernames and their public keys. The owners of the key pairs will be able to authenticate themselves to the server with the key pair, and then be authorized to access the system based on the server policy.

If the receiving entity is a client host, the user can, for example, add the public key of the (server) entity to a local database and associate it with the server's IP address. The client host will now trust the server in the future when it authenticates itself with the same key pair.