attempt to not pattern match on ConstructedAtom but directly on haskell types when making AtomFunction

[YuMingLiao]

I was trying to omit the "pattern match on ConstructedAtom" process. And here is what I have so far.
code on gist
To make it compile, some packages need jailbreak and minor api updates. (I use nix, and sometimes snack to build project-m36)

It compiles, but sadly tutd couldn't find a symbol when it is a .o and get a LoadFunctionError when it is a .so.

Not sure if it is worth all the effort. I just pose this WIP here in case someone is interested.

[agentm]

Very cool! I didn't even realize this sort of type trickery could work. Making server-side functions should be easier, so I'll see how best to integrate this.

Dynamically-loaded functions can be tricky, especially since one has to keep track of which version of project-m36 the running server has loaded. I can probably help with the error if you post it here, though.

[YuMingLiao]

Very cool! I didn't even realize this sort of type trickery could work.

Glad you like it!

Dynamically-loaded functions can be tricky, especially since one has to keep track of which version of project-m36 the running server has loaded.

Yes, and I am aware of the option of put functions directly in project-m36 and re-compiling. But I have to import my csv again due to MerkleHash check. Hard to develop fluently.

I can probably help with the error if you post it here, though.

[nix-shell:~/DynamicAtomFunctions]# ghc DynamicAtomFunctions.hs -package project-m36
[1 of 2] Compiling HeteroListUtil   ( HeteroListUtil.hs, HeteroListUtil.o )                                                         
[2 of 2] Compiling DynamicAtomFunctions ( DynamicAtomFunctions.hs, DynamicAtomFunctions.o )                                         

[nix-shell:~/DynamicAtomFunctions]# tutd
Project:M36 TutorialD Interpreter 0.9.4
Type ":help" for more information.
A full tutorial is available at:
https://github.com/agentm/project-m36/blob/master/docs/tutd_tutorial.markdown                                                       
TutorialD (master/main): loadatomfunctions "DynamicAtomFunctions" "haskellFunctions" "DynamicAtomFunctions.o"
tutd: DynamicAtomFunctions.o: unknown symbol `polydatazm0zi3zi0zi0zm5LV4Kscn6cpFJtPsOXhy8A_DataziPolyziFunction_mkPolyFunc1_closure'
tutd: Could not load Object Code DynamicAtomFunctions.o.

TutorialD (master/main): Segmentation fault (core dumped)

[nix-shell:~/DynamicAtomFunctions]# ghc DynamicAtomFunctions.hs -package project-m36 -package polydata -dynamic -shared -fPIC -o DynamicAtomFunctions.so 
[1 of 2] Compiling HeteroListUtil   ( HeteroListUtil.hs, HeteroListUtil.o )
[2 of 2] Compiling DynamicAtomFunctions ( DynamicAtomFunctions.hs, DynamicAtomFunctions.o )
Linking DynamicAtomFunctions.so ...

[nix-shell:~/DynamicAtomFunctions]# tutd
Project:M36 TutorialD Interpreter 0.9.4
Type ":help" for more information.
A full tutorial is available at:
https://github.com/agentm/project-m36/blob/master/docs/tutd_tutorial.markdown
TutorialD (master/main): loadatomfunctions "DynamicAtomFunctions" "haskellFunctions" "DynamicAtomFunctions.so"
ERR: LoadFunctionError
TutorialD (master/main): 

With or without -package polydata, error message seems the same.

[agentm]

Shoot- it looks like polydata isn't maintained. I have emailed the maintainer to ask if he is accepting pull requests on his modules. But, is polydata necessary or just handy here? Could this be reimplemented using straight data families? I think this would be a great native feature to include in Project:M36.

[agentm]

I received a reply from the polydata maintainer that he is willing to accept pull requests, so I'll relax the polydata base constraints and upstream them.

[YuMingLiao]

Not sure if it helps but, according to my nix configuration, I've tried those things to make my code compile. (jailbreak, specific version, fixing apis)

             HList = self.callPackage ./HList-0.5.2.0.nix {};
             indextype = doJailbreak (self.callPackage /root/fix/indextype/indextype.nix {});
             polydata-core = doJailbreak super.polydata-core;
             polydata = doJailbreak super.polydata;
             constraint-manip = doJailbreak super.constraint-manip;
             heterolist = doJailbreak (self.callPackage /root/fix/heterolist/heterolist.nix {});
             type-list = doJailbreak (super.singletons);
             tuple-hlist = doJailbreak (self.callPackage /root/fix/tuple-hlist/tuple-hlist.nix {});

is polydata necessary or just handy here?

The purpose of mkPolyFunc1 here is to create a poly fromAtom function that can be mapped to a HeteroList. And HeteroList is here for get types from a function's argement and assigning result types of mapping fromAtom to a [Atom]. The normal fromAtom seems to deal with one type only at a time. If there is a way to "bypass" HeteroList but assign result types, then polydata may be able to omitted.

Could this be reimplemented using straight data families?

I think polyFromAtom works for any Atomable a if the instance is in the scope. Not sure how data familes could do this type trickery.

[agentm]

Shoot, I tested polydata with GHC 9.2 and it already broken due the PolyKinds changes in the indextype package dependency. Luckily, all these modules are maintained by the same guy, so hopefully we can push all these changes through.

[YuMingLiao]

Okay, I use ghc8107 and I have pulled requests for fixes I have done for indextypes, heterolist and tuple-hlist. Not sure how to bump bounds for other packages 'cause I haven't tried to compile them with ghc921.

[YuMingLiao]

Hi! @agentm
Good News. It works!

.o file with original example.

TutorialD (master/main): loadatomfunctions "DynamicAtomFunctions" "someFunctions" "DynamicAtomFunctions.o"
(successfully loaded)  

.so file with type trickery.

TutorialD (master/main): loadatomfunctions "DynamicAtomFunctions" "haskellFunctions" "DynamicAtomFunctions.so"                      
ERR: LoadFunctionError                                                                                                              
TutorialD (master/main): loadatomfunctions "DynamicAtomFunctions" "haskellFunctions" "/root/DynamicAtomFunctions/DynamicAtomFunctions
.so"
TutorialD (master/main): a := relation{tuple{a int(1), b int(2)}}
TutorialD (master/main): :showexpr a:{c := plus(@a,@b)}
┌──────┬──────┬──────┐
│a::Int│b::Int│c::Int│
├──────┼──────┼──────┤
│1     │2     │3     │
└──────┴──────┴──────┘

It seems .o accepts a relative path while .so needs a absolute path.

Anyway, the type trickery works!

[agentm]

Hm, the intention is that the module paths should be relative to the database directory's modules directory, so it looks like that is not working reliably. That could be a security issue, so I'll take a closer look at that code path.