Helper class to source any file-like object

[bertday]

I have a use case where I'm picking up CSV over SFTP (using a library called pysftp) and running it through a pipeline. Initially I thought it would look something like this:

import petl as etl
import pysftp

with pysftp.Connection(hostname, username=username, password=password) as conn:
    with conn.open('path/to/file.csv') as f:
            # f is a file-like object
            (etl.fromcsv(f)
                ...
            )

But that raised an error since f doesn't implement open() and therefore can't be opened. So I ended up writing a wrapper like:

class SftpSource(object):
    def __init__(self, conn, path):
        self.conn = conn
        self.path = path

    @contextmanager
    def open(self, mode='r'):
        if not mode.startswith('r'):
            raise ArgumentError('source is read-only')
        f = self.conn.open(self.path)
        try:
            yield f
        finally:
            f.close()

Which wasn't much trouble at all, but I was trying to think of how you could generalize that (and wrap it into petl) to handle basically any file-like object. Something like:

class FloSource(object):
    def __init__(self, open_, *args, **kwargs):
        self.open_ = open_
        self.args = args
        self.kwargs = kwargs

    @contextmanager
    def open(self, mode='r'):
        f = self.open_(*args, **kwargs)
        try:
            yield f
        finally:
            f.close()

And used like:

conn = pysftp.Connection(...)
source = etl.FloSource(conn.open, '/path/to/data.csv')
(etl.fromcsv(source)
     ...
)

Just a thought, in case this might be helpful to others. Would be happy to work on a PR.

[bertday]

Actually, this looks a lot like the existing FileSource class. I suppose you could just give FileSource a kwarg for an open function. This worked for my purposes:

class FileSource(object):
    def __init__(self, path, opener=None, **kwargs):
        self.path = path
        self.opener = opener
        self.kwargs = kwargs

    def open(self, mode='r'):
        if self.opener:
            return self.opener(self.path, **self.kwargs)
        return io.open(self.filename, mode, **self.kwargs)

# test
with pysftp.Connection(hostname, username=username, password=password) as conn:
    source = FileSource(path, opener=conn.open)
    (etl.fromcsv(source)
        ...
    )

[alimanfoo]

Thanks Robert. I'd be happy to include some generalisation to support this. Would just need to make sure it works properly with the context manager protocol to ensure whatever is opened is closed finally. Using the pattern below I think this means that whatever is returned by opener would need to support the context manager protocol. Is this a reasonable expectation do you think?

…

On Wednesday, January 25, 2017, Robert Martin ***@***.***> wrote: Actually, this looks a lot like the existing FileSource class. I suppose you could also give FileSource a kwarg for an open function. This worked for my purposes: class FileSource(object): def __init__(self, path, opener=None, **kwargs): self.path = path self.opener = opener self.kwargs = kwargs def open(self, mode='r'): if self.opener: return self.opener(self.path, **self.kwargs) return io.open(self.filename, mode, **self.kwargs) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#409 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAq8QmiCYCRI_E-pQQhvTEEG7XvBCvgLks5rV82vgaJpZM4LuEb-> .

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health <http://cggh.org> The Wellcome Trust Centre for Human Genetics Roosevelt Drive Oxford OX3 7BN United Kingdom Email: [email protected] Web: http://purl.org/net/aliman Twitter: https://twitter.com/alimanfoo Tel: +44 (0)1865 287721

[bertday]

I would say so - from memory it seems like most I/O libraries out there are context-aware. I suppose the other consideration is how many accept some kind of file path as their first positional arg. Not totally unreasonable IMO, but I'll look around at some other APIs to make sure that's the norm.

[alimanfoo]

Thanks. We could always change FileSource to accept *args, **kwargs, then try to pop 'opener' out of kwargs with default io.open. Then just pass through *args and remaining **kwargs when calling opener? Or trying to be too clever?

…

On Wednesday, January 25, 2017, Robert Martin ***@***.***> wrote: I would say so - from memory it seems like most I/O libraries out there are context-aware. I suppose the other consideration is how many accept some kind of file path as their first positional arg. Not totally unreasonable IMO, but I'll look around at some other APIs to make sure that's the norm. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#409 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAq8QuFdVuPnw4Ie2ZEOwHjHaTrrj6KWks5rV9JDgaJpZM4LuEb-> .

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health <http://cggh.org> The Wellcome Trust Centre for Human Genetics Roosevelt Drive Oxford OX3 7BN United Kingdom Email: [email protected] Web: http://purl.org/net/aliman Twitter: https://twitter.com/alimanfoo Tel: +44 (0)1865 287721

[bertday]

I think that could work. One other option - you could allow the first arg to FileSource to be either a string or callable, similar to fromdb/todb.

class FileSource(object):
    def __init__(self, path_or_opener, **kwargs):
        self.path_or_opener = path_or_opener
        self.kwargs = kwargs

    def open(self, mode='r'):
        if callable(self.path_or_opener):
            opener = self.path_or_opener()
            if not hasattr(opener, '__exit__'):
                raise ArgumentError('File opener must be a context manager')
            return opener

        return io.open(self.path_or_opener, mode, **self.kwargs)

#######

conn = pysftp.Connection(...)
source = etl.FileSource(lambda: conn.open('path/to/data.csv'))
(etl.fromcsv(source)
     ...
)

[bertday]

I submitted a PR based on that last idea, but feel free to edit/let me know if that's too off base.

Thanks!

[alimanfoo]

Thanks Robert, appreciated. I'm a bit snowed under at the moment but will review asap.

…

On Thursday, January 26, 2017, Robert Martin ***@***.***> wrote: I submitted a PR <#410> based on that last idea, but feel free to edit/let me know if that's too off base. Thanks! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#409 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAq8QspbFUOQoJOPgvqm5czeNFmvVAwGks5rWNc9gaJpZM4LuEb-> .

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health <http://cggh.org> The Wellcome Trust Centre for Human Genetics Roosevelt Drive Oxford OX3 7BN United Kingdom Email: [email protected] Web: http://purl.org/net/aliman Twitter: https://twitter.com/alimanfoo Tel: +44 (0)1865 287721

[bertday]

@alimanfoo no problem, I've been shimming that code in as needed so I'm set for now :) Thanks.

[juarezr]

@rbrtmrtn ,

With release v1.5.0 there is new function for registering custom sources that fill this role.

Also with release v1.6.0 there is support for reading for remote sources by using the package fsspec, including SFTP servers.

For this working is required:

1. Install petl: pip install petl
2. Install fsspec: pip install fsspec
3. Install paramiko: pip install paramiko
4. Use a full url pointing to the file in the remote server in from...() and to...() functions.

E.g:

import petl as etl

myurl = "sftp://myuser:mypassword@myserver/path/to/myfile.csv"
table2 = etl.fromcsv(myurl)
# ...

Do you think this closes this issue?

Although #410 could be useful for other custom cases.