Cook Book

While in many ways, pysftp is just a thin wrapper over paramiko’s SFTPClient, there are a number of ways that we make it more productive and easier to accomplish common, higher-level tasks. The following snippets show where we add value to this great module. See the API docs for a complete listing.

pysftp.Connection()

The Connection object is the base of pysftp. It supports connections via username and password.

import pysftp
sftp = pysftp.Connection('hostname', username='me', password='secret')
#
# ... do sftp operations
#
sftp.close()    # close your connection to hostname

The Connection object is also context aware so you can use it with a with statement.

import pysftp
with pysftp.Connection('hostname', username='me', password='secret') as sftp:
    #
    # ... do sftp operations
    #
# connection closed automatically at the end of the with-block

Want to use an RSA or DSA key pair, that is simple too.

import pysftp
with pysftp.Connection('hostname', username='me', private_key='/path/to/keyfile') as sftp:
    #
    # ... do sftp operations
    #

If you key is password protected, just add private_key_pass to the argument list.

How about a paramiko.AgentKey ? no problem, just set the private_key equal to it.

import pysftp
with pysftp.Connection('hostname', username='me', private_key=my_agentkey) as sftp:
    #
    # ... do sftp operations
    #

The connection object also allows you to use an IP Address for the host and you can set the port which defaults to 22, as well.

pysftp.CnOpts

You can also specify additional connection options using the pysftp.CnOpts object. These options are advanced and not applicable to most uses, because of this they have been segmented from the Connection parameter list and made available via CnOpts obj/parameter.

Host Key checking is enabled by default. It will use ~/.ssh/known_hosts by default. If you wish to disable host key checking (NOT ADVISED) you will need to modify the default CnOpts and set the .hostkeys to None.

import pysftp
cnopts = pysftp.CnOpts()
cnopts.hostkeys = None
with pysftp.Connection('host', username='me', password='pass', cnopts=cnopts):
    # do stuff here

To use a completely different known_hosts file, you can override CnOpts looking for ~/.ssh/known_hosts by specifying the file when instantiating.

import pysftp
cnopts = pysftp.CnOpts(knownhosts='path/to/your/knownhostsfile')
cnopts.hostkeys = None
with pysftp.Connection('host', username='me', password='pass', cnopts=cnopts):
    # do stuff here

If you wish to use ~/.ssh/known_hosts but add additional known host keys you can merge with update additional known_host format files by using .load method.

import pysftp
cnopts = pysftp.CnOpts()
cnopts.hostkeys.load('path/to/your/extra_knownhosts')
with pysftp.Connection('host', username='me', password='pass', cnopts=cnopts):
    # do stuff here

For both the knownhost parameter and the load argument, pysftp expands user, so you can use tilde notation in your pathing.

OTHER AVAILABLE CONNECTION OPTIONS via CnOpts:

  • .log - replaces the log parameter in the Connection method
  • .compression - False (Default) no compression, True - enable compression
  • .ciphers - replaces the ciphers parameter in the Connection method.
  • log and ciphers in the Connection parameter list are deprecated and will be removed in version 0.3.0 Use the CnOpts to specify them.

Here is a common scenario, you have your connection information stored in a persistence mechanism, like yamjam and when you access it, it is returned in dictionary form. {'host':'myhost', username:'me', ...} Just send the dict into the connection object like so:

import pysftp
cinfo = {'host':'hostname', 'username':'me', 'password':'secret', 'port':2222}
with pysftp.Connection(**cinfo) as sftp:
    #
    # ... do sftp operations
    #

pysftp.Connection.get()

In addition to the normal paramiko call, you can optionally set the preserve_mtime parameter to True and the operation will make sure that the modification times on the local copy match those on the server.

# ...
sftp.get('myfile', preserve_mtime=True)

pysftp.Connection.get_d()

This pysftp method is an abstraction above get() that allows you to copy all the files in a remote directory to a local path.

# copy all files under public to a local path, preserving modification time
sftp.get_d('public', 'local-backup', preserve_mtime=True)

pysftp.Connection.get_r()

This pysftp method is an abstraction that recursively copies files and directories from the remote to a local path.

# copy all files AND directories under public to a local path
sftp.get_r('public', 'local-backup', preserve_mtime=True)

pysftp.Connection.put()

In addition to the normal paramiko call, you can optionally set the preserve_mtime parameter to True and the operation will make sure that the modification times on the server copy match those on the local.

# copy myfile, to the current working directory on the server, preserving modification time
sftp.put('myfile', preserve_mtime=True)

pysftp.Connection.put_d()

The opposite of get_d(), put_d allows you to copy the contents of a local directory to a remote one via SFTP.

# copy files from images, to remote static/images directory, preserving modification time
sftp.put_d('images', 'static/images', preserve_mtime=True)

pysftp.Connection.put_r()

This method copies all files and directories from a local path to a remote path. It creates directories, and happily succeeds even if the target directories already exist.

# recursively copy files and directories from local static, to remote static,
# preserving modification times on the files
sftp.put_r('static', 'static', preserve_mtime=True)

pysftp.Connection.cd()

This method is a with-context capable version of chdir(). Restoring the original directory when the with statement goes out of scope. It can be called with a remote directory to temporarily change to

with sftp.cd('static'):     # now in ./static
    sftp.chdir('here')      # now in ./static/here
    sftp.chdir('there')     # now in ./static/here/there
# now back to the original current working directory

Or it can be called without a remote directory to just act as a bookmark you want to return to later.

with sftp.cd():             # still in .
    sftp.chdir('static')    # now in ./static
    sftp.chdir('here')      # now in ./static/here
# now back to the original current working directory

pysftp.Connection.chmod()

chmod() is a wrapper around paramiko’s except for the fact it will takes an integer representation of the octal mode. No leading 0 or 0o wanted. We know it’s suppose to be an octal, but who really remembers that?

This way it is just like a command line chmod 644 readme.txt

user group other
rwx  rwx   rwx
421  421   421

user  - read/write = 4+2 = 6
group - read       = 4   = 4
other - read       = 4   = 4
sftp.chmod('readme.txt', 644)

pysftp.st_mode_to_int()

converts an octal mode result back to an integer representation. The .st_mode information returned in SFTPAttribute object .stat(fname).st_mode contains extra things you probably don’t care about, in a form that has been converted from octal to int so you won’t recognize it at first. This function clips the extra bits and hands you the file mode bits in a way you’ll recognize.

>>> attr = sftp.stat('readme.txt')
>>> attr.st_mode
33188
>>> pysftp.st_mode_to_int(attr.st_mode)
644

pysftp.Connection.chown()

pysftp’s method allows you to specify just, gid or the uid or both. If either gid or uid is None (default), then pysftp does a stat to get the current ids and uses that to fill in the missing parameter because the underlying paramiko method requires that you explicitly set both.

NOTE uid and gid are integers and relative to each system. Just because you are uid 102 on your local system, a uid of 102 on the remote system most likely won’t be your login. You will need to do some homework to make sure that you are setting these values as you intended.

pysftp.Connection.cwd()

cwd() is a synonym for chdir(). Its purpose is to make transposing hand typed commands at an sftp command line into those used by pysftp, easier to do.

...
sftp.cwd('public')  # is equivalent to sftp.chdir('public')

pysftp.Connection.pwd

Returns the current working directory. It returns the result of .normalize(‘.’) but makes your code and intention easier to read. Paramiko has a method, getcwd(), that we expose, but that method returns None if chdir() has not been called prior.

...
>>> print(sftp.getcwd())
None
>>> sftp.pwd
u'/home/test'

pysftp.Connection.listdir()

The difference here, is that pysftp’s version returns a sorted list instead of paramiko’s arbitrary order. Sorted by filename.

...
>>> sftp.listdir()
[u'pub', u'readme.sym', u'readme.txt']

pysftp.Connection.listdir_attr()

The difference here, is that pysftp’s version returns a sorted list instead of paramiko’s arbitrary order. Sorted by SFTPAttribute.filename.

...
>>> for attr in sftp.listdir_attr():
...     print attr.filename, attr
...
pub dr-xrwxr-x   1 501      502             5 19 May 23:22 pub
readme.sym lrwxr-xr-x   1 501      502            10 21 May 23:29 readme.sym
readme.txt -r--r--r--   1 501      502          8192 26 May 23:32 readme.txt

pysftp.Connection.makedirs()

A common scenario where you need to create all directories in a path as needed, setting their mode, if created. Takes a mode argument, just like chmod(), that is an integer representation of the mode you want.

...
sftp.makedirs('pub/show/off')  # will happily make all non-existing directories

pysftp.Connection.mkdir()

Just like chmod(), the mode is an integer representation of the octal number to use. Just like the unix cmd, chmod you use 744 not 0744 or 0o744.

...
sftp.mkdir('show', mode=644)  # user r/w, group and other read-only

pysftp.Connection.isdir()

Does all the busy work of stat’ing and dealing with the stat module returning a simple True/False.

...
>>> sftp.isdir('pub')
True

pysftp.Connection.isfile()

Does all the busy work of stat’ing and dealing with the stat module returning a simple True/False.

...
>>> sftp.isfile('pub')
False

pysftp.Connection.exists()

Returns True if a remote entity exists

...
>>> sftp.exists('readme.txt')   # a file
True
>>> sftp.exists('pub')          # a dir
True

pysftp.Connection.lexists()

Like exists(), but returns True for a broken symbolic link

pysftp.Connection.truncate()

Like the underlying .truncate method, by pysftp returns the file’s new size after the operation.

>>> sftp.truncate('readme.txt', 4096)
4096

pysftp.Connection.walktree()

Is a powerful method that can recursively (default) walk a remote directory structure and calls a user-supplied callback functions for each file, directory or unknown entity it encounters. It is used in the get_x methods of pysftp and can be used with great effect to do your own bidding. Each callback is supplied the pathname of the entity. (form: func(str))

pysftp.Connection.sftp_client

Don’t like how we have over-ridden or modified a paramiko method? Use this attribute to get at paramiko’s original version. Remember, our goal is to augment not supplant paramiko.

pysftp.path_advance

generator to iterate over a file path

...
>>> list(pysftp.path_advance('./pub/example/example01'))
['.', './pub', './pub/example', './pub/example/example01']

pysftp.path_retreat

generator to iterate over a file path in reverse

...
>>> list(pysftp.path_retreat('./pub/example/example01'))
['./pub/example/example01', './pub/example', './pub', '.']

pysftp.reparent

Pythons os.path.join('backup', '/home/test/pub') returns ‘/home/test/pub’, but if you want to copy a directory structure to a new path this won’t do what you want. But, reparent will.

...
 >>> pysftp.reparent('backup', '/home/test/pub')
'backup/./home/test/pub'

pysftp.walktree

Is similar to pysftp.Connection.walktree() except that it walks a local directory structure. It has the same callback mechanism.

pysftp.cd

A with-context aware version of os.chdir for use on the local file system. The yin to pysftp.Connection.cd() yang.

...
>>> import os
>>> os.getcwd()
'/home/jlh/Projects/pysftp/src'
>>> with pysftp.cd('docs'):
...     print os.getcwd()
...
/home/jlh/Projects/pysftp/src/docs
>>> os.getcwd()
'/home/jlh/Projects/pysftp/src'

Remarks

We think paramiko is a great python library and it is the backbone of pysftp. The methods pysftp has created are abstractions that serve a programmer’s productivity by encapsulating many of the higher function use cases of interacting with SFTP. Instead of writing your own code to walk directories and call get and put, dealing with not only paramiko but Python’s own os and stat modules and writing tests (many code snippets on the net are incomplete and don’t account for edge cases) pysftp supplies a complete library for dealing with all three. Leaving you to focus on your primary task.

Paramiko also tries very hard to stay true to Python’s os module, which means sometimes, things are weird or a bit too low level. We think paramiko’s goals are good and don’t believe they should change. Those changes are for an abstraction library like pysftp.