Disclaimer: This blog entry is more of a note of some simple recipes used over and over again.

Introduction

Sometimes one has some Python scripts running and wants to transfer data to or fetch data from a remote machine - or simply execute commands. The best way to run commands on remote machines usually is SSH, the best ways for file transfer SFTP, SCP or simply copying to an NFS share (or of course using WebDAV / DeltaV). A simple method to realize this is the usage of the excellent paramiko library - this is a Python implementation of the SSHv2 protocol that builds on top of the cryptography package for native cryptography primitives.

This library is pretty flexible - this short blog entry will only show the most basic usage and also assumes that one wants to use public key based authentication for the remote user. As soon as one has grasped the basic ideas one can figure out everything necessary from the excellent documentation.

Installation

One can directly install paramiko from PyPi:

pip install paramiko

Importing the library, loading the private key file

First as usual one has to import the library:

import paramiko

Then one can use the RSAKey.from_private_key_file method to load the private key part for authentication:

try:
    sshkey = paramiko.RSAKey.from_private_key_file("/path/to/keyfile")
except IOError:
    # The file has not been found or any other kind of I/O error has happened
    pass
except PasswordRequiredException:
    # No password has been supplied by the keyfile is password protected
    pass
except SSHException
    # Keyfile is invalid
    pass

To supply a password one can add the password argument to the from_private_key_file method. Please keep in mind that handling passwords usually should be done with care - using separate memory locations that are not pageable, etc. - this is pretty hard to do correct from Pythons environment.

Executing remote commands

To execute remote commands one just has to:

• Load the private key file (see above)
• Take care of host key handling to make sure the remote machine is well known
• Run commands supplying the stdin, stdout and stderr pipes from the remote shell
• Shutting down

Host key handling tells the client how to react when one sees an unknown or changed remote host key. The typical policies are:

• RejectPolicy (the default policy) that simply rejects connections to any unknown host. One has to connect using the same user first manually to the remote machine to get a program running using this policy.
• AutoAddPolicy is one of the more popular approaches - when a host first sees a new host key it accepts it when no other hostkey is known for the given destination. In case the key changes the connection is rejected though.
• WarningPolicy always connects and only warns in case of unknown or changed host keys. This is rather unsafe and allows for man in the middle attacks …

# First load the SSHKey - see above for details:
sshkey = paramiko.RSAKey.from_private_key_file("/path/to/keyfile")

# Create the SSH client and setup handling of new host keys
sshClient = paramiko.SSHClient()
sshClient.set_missing_host_key_policy(paramiko.AutoAddPolicy())

# Connect to the remote host
sshClient.connect("203.0.113.1", username = "example", pkey = sshkey)

To execute commands one uses the exec_command method that returns a tuple of the standard pipes:

stdin, stdout, stderr = sshClient.exec_command(command)

To dump all information sent by a client and simply terminate afterwards one can dump information from stdout and invoke all destructors on the pipes. Note that this is required!:

stdin, stdout, stderr = sshClient.exec_command("uname -a")
print(stdout.read())
del stdin, stdout, stderr

In the end one has to close the client:

sshClient.close()

File transfer via SFTP

To transfer files one can use the sftp subsystem of SSH. Paramiko provides a full blown implementation of the SFTP protocol including directory traversal, permission management, reading and writing files without storing them locally, etc. The following example will limit itself to uploading and downloading of files as well as deleting and listing.

Bootstrapping of the public key, client instantiating and connecting works the same as before:

# First load the SSHKey - see above for details:
sshkey = paramiko.RSAKey.from_private_key_file("/path/to/keyfile")

# Create the SSH client and setup handling of new host keys
sshClient = paramiko.SSHClient()
sshClient.set_missing_host_key_policy(paramiko.AutoAddPolicy())

# Connect to the remote host
sshClient.connect("203.0.113.1", username = "example", pkey = sshkey)

Then one has to open the SFTP client:

sftpClient = sshClient.open_sftp()

Upload and download

• Uploading files uses the put operation: sftpClient.put("/local/source", "/remote/destination")
• Downloading files uses the get operation: sftpClient.get("/remote/source", "/local/destination")

Both methods allow one to set a callback = callable that receives two integer parameters callback(bytesTransferred, bytesTotal). One could use this to provide status updates for example:

def showTransferStatus(bytesTransferred, bytesTotal):
    print(f"{bytesTransferred} out of {bytesTotal} bytes transferred")

sftpClient.put("/local/source", "/remote/destination", callback = showTransferStatus)

Delete

To delete a file one can simply call the remove function:

sftpClient.remove("/path/remote")

Listing files

There are a variety of methods to list files on the remote machine using the SFTP client. The easiest for small directories is listdir(path = '.'). This method returns a simple Python list containing all filenames and directory names on the remote side. It does not contain entries for . and .. though.

print(sftpClient.listdir())

In case one also wants to receive the attributes one might use listdir_attr(path = '.'). This returns an wrapper containing additional information per entry:

• Two representations of the filename:
  • filename
  • longname
• The information known from stat(2):
  • st_atime - the last accessed time (not updated on all filesystems)
  • st_gid as the owning group ID
  • st_uid is the owning user ID
  • st_mode contains the Unix permission flags
  • st_mtime specifies the last modified time
  • st_size, the size in bytes

A more advanced method to list files uses an iterator / generator pattern. This method is called listdir_iter(path = '.', read_aheads = 50). This method can be used in a typical iterator pattern like a for each loop:

for f in sftpClient.listdir_iter():
	print(f.filename)

The objects iterated over have the same structure as the result of listdir_attr.

Shutting down

In the end the SFTP client as well as the SSH client has to be shut down:

sftpClient.close()
sshClient.close()

This article is tagged: Python, Basics, Programming