feat(dataset): adding data from external files without copy

[mohammad-alisafaee]

Description

To add an external file to a dataset pass --external option to the dataset add command. To check for modification in external files run renku dataset update --external.

Fixes #815

[rokroskar]

This is starting to look nice! I just tried it and one thing I notice is that in the resulting graph there is no link between the pointer file and the file in the dataset (see below).

Another thought I had was that if we are adding a pointer reference in .renku maybe it should be generic? That way I could do something like renku run --external-input <path-outside-of-repo> myscript.sh

[emmjab]

I tried it out in the following way:

1. created a simple text file in a temporary location
2. added to the dataset with the new "external" flag
3. ran a renku run a script to take that file as an input; write the contents into a new file in the repo
4. pushed the repo to dev.renku.ch

I was able to view the lineage from the output file (but not the data file or the pointer file) [1]
The pointer files were effectively not there in the repo, but the metadata was preserved.

renku init
renku dataset create test
renku dataset add test /tmp/testing_renku/data.txt --external
renku run python scripts/modify_test.py data/test/data.txt out.txt

[1] https://dev.renku.ch/projects/e.jablonski/test_external/files/lineage/out.txt
[2] https://dev.renku.ch/projects/e.jablonski/test_external/files/blob/.renku/datasets/f6826b9c-f9a3-49ad-92ed-5464780874be/pointers/a1e20fa1-8ace-4fd8-8797-472113f6785a-e8e09898d91cc3a412ba36751fdba4ac931acb5c

Seems good to me! I didn't try changing data in the input file or code and running a renku update yet

[rokroskar]

Awesome, thanks for making the project @emmjab - This looks like a good starting point!

[mohammad-alisafaee]

@rokroskar I'd say the pointer file should not show up in the graph or lineage at all. It's just an implementation detail that users shouldn't care about. I'm not sure though if it is possible to exclude it.

[emmjab]

Also just tested the renku dataset update --external and called renku update; also works as I would expect & generates a linage: https://dev.renku.ch/projects/e.jablonski/test_external/files/lineage/out.txt

[rokroskar]

I'd say the pointer file should not show up in the graph or lineage at all. It's just an implementation detail that users shouldn't care about. I'm not sure though if it is possible to exclude it.

I see - I suppose at the moment there isn't a way to exclude it. But if it's there, maybe a link between the two entities would make sense?

Two other points that come to mind at the moment:

1. naming the external file nodes: should we have some stable naming convention? As it stands, I could use the same data from two different projects but this would not be obvious from the KG (apart from maybe the last part of the path being the same). I guess eventually they could be linked via their sha256 or similar. Have you thought about that?

2. Could there be a mechanism to easily update the link in case the location of the external data changes?

Actually, I have a third :)

3. Could this be extended to allow something like:

pwd
  <a renku project root>
ln -s ../../../data/path/file.csv
renku external add file.csv

So now in my renku project, I have a symbolic link file.csv pointing to the external data and the command renku external add registered this file and created the a file in .renku/external (?) that holds the metadata (checksum, etc.) Alternatively, the registration could be implicit, i.e. only when the link is used in a renku command. So I could create the symbolic link and commit it, but only when I did

renku run --input file.csv script.py 

we would see that it's a symbolic link and create the requisite metadata. This is very similar to the dataset functionality but it allows me to do it without having to explicitly create a dataset.

[ciyer]

This is very nice, and works pretty well!

I tested a case where I have code in one repo and a file that is processed just in a regular directory. The process and results are documented here:

https://dev.renku.ch/projects/renku-external-data-one-repo/renku-external-data-code/

I found the same thing that Emma found:

• The pointer file is shown as an upstream input to the result, but it does not have lineage itself, which is probably a bug

Question
Is there a reason the pointer files are stored in git-lfs? I would think they can/should be stored in plain git.

[mohammad-alisafaee]

Thanks all for your comments!
The issue with lineage of external file is solved and the lineage is available now.

1. naming the external file nodes: should we have some stable naming convention? As it stands, I could use the same data from two different projects but this would not be obvious from the KG (apart from maybe the last part of the path being the same). I guess eventually they could be linked via their sha256 or similar. Have you thought about that?

As we discussed this will be implemented later.

2. Could there be a mechanism to easily update the link in case the location of the external data changes?

You can re-add the files with --force flag from the new location.

[rokroskar]

This works really well! I only have a few minor wording suggestions for the documentation.

One thing I realized is that it won't work for directories. So if I want to add a directory of files and then want renku to respond to any new data added to the directory I basically have to re-add the data. Maybe we can have a follow-up issue for that.

[rokroskar]

Suggested change

	actual files to your repository. This is useful for example when external data
	is too large to store locally. The external data must exist (i.e. mounted) on
	your filesystem. Renku create a symbolic to your data and you can use this
	actual files to your repository. This is useful for example when external data
	is too large to store locally. The external data must exist (i.e. be mounted) on
	your filesystem. Renku creates a symbolic to your data and you can use this

[rokroskar]

Suggested change

	help='Update only external storage files (symlinks).'
	help='Update only external data.'

Maybe we can keep this more generic?

[rokroskar]

👍

[rokroskar]

This works really nicely! Just a minor wording suggestion - in most cases it might not be a mounted volume so the user might wonder what that's all about.

Suggested change

	' (make sure that external volume is mounted and accessible)' +
	' (make sure that the external path is accessible)' +