Add the ability to create a dynamic library

[sheosi]

As of now, crystal lacks any possibility of making dynamic libraries, which is a bump for this language as it would open a great number of possibilities, theoritically, it's possible I've made myself Crystal code called by C code, and as much as I've tried (not that much, anyway) there's no problem, so far the only show-stopper I found was the libraries were compiled without pic, making dynamic libraries impossible.
Maybe, by compiling the .bc/.ll by hand there's a possiblity this can be workarounded, but I couldn't get there myself (version from git failed to build).

[bcardiff]

It would be really nice. There have been some efforts in that direction, but Crystal is not there yet

Maybe #241 and this Makefile could help a bit. But have in mind this last comment

Again, I agree it would be really nice.

[sheosi]

Sorry, it seems I didn't dig enough, a shame anyways.

[bcardiff]

@sheosi Not at all! It wasn't an easy search.

Also gave in mind that fun functions are like C externals, they can be referenced by name when loading. Other def have some mangling due to template like expansion. If you deal with the GC you might be able to come with a solution.

It will be really good to expose Crystal code to the outside world. There is just no clear roadmap for this yet.

[sheosi]

If this helps anyone all that I did was to copy crystal's main inside a
initialize function,also removed yield keyword from the main( it gave some
strange issues, I couldn't print from crystal).

[asterite]

@sheosi Crystal requires a GC, does it really make sense to use it for dynamic libraries?

[sheosi]

While I don't know much about GC side-effects, I guess it should be usable
anywhere where Lua is usable, particularly I'm interested in plugins, as
thanks to crystal's power they can be written in a declarative-style.

[ozra]

It wouldn't be impossible despite GC. But you'd get the reverse problem of manual memory managment (remembering to free memory), namely ensuring that memory is kept, keeping lists of pointers to everything that might still be alive in the libs using code, so that the GC still has references to them. Or, doing manual allocations for everything that is passed over to using code.

[fabianloewe]

I'm working on this and got a shared library by emitting an object file and linking it to such a library. My only problem is all Crystal symbols are hidden/local so one can't load them dynamically or link to them.
What in the compiler causes this?

BTW, I don't think the GC is a problem. We could just let all libraries use the executable's GC so all objects are tracked at one place.
But I haven't got so far...

[ysbaddaden]

@hyronx You need to write an interface between the Crystal code and the outside. That is you must write a C API, much like you would to expose a C++ library. Thankfully you can write it in Crystal using fun instead of def, taking care to Box references to Crystal objects.

Since I was interested, I put together a Proof of Concept that shows that this is possible and that it does work! But it requires a bunch of boilerplate to be manually written (maybe macros could help): https://github.com/ysbaddaden/crystal_library

It only requires a small hack to avoid loading the original main.cr and expose an alternative method to manually initialize the GC and the Crystal world (constants, ...). Maybe it could interesting to have a -Dlibrary flag in the original main.cr @asterite? Even though we don't support this explicitly, there are some usage of embedding Crystal into another language (Ruby extension, plugins, etc) and it's a tiny change.

[fabianloewe]

@ysabbaden That's really nice work. I would like to make linking Crystal with Crystal code possible without the C layer between it. Maybe I get so far in the next days with the help of your PoC. It would help me if somebody could explain why all Crystal symbols are local.

Shared library support and linking to all kinds of existing code could help making Crystal more popular I suppose.

[ysbaddaden]

No, creating libraries for Crystal code isn't possible without hacking the Crystal compiler, and this is probably impossible to achieve:

• It would miss all the class/module/method signatures, and it would need a mechanism to read them from (like a header file).
• Furthermore, method argument types are dynamic, and creating a library would fail, unless all method arguments are fully typed beforehand (killing most of the pleasure of using Crystal).

If you want dynamically loadable plugins written in crystal into a crystal application, you'll need to have a C API exposed and a lib binding to this exposed API. This isn't pretty, but it could work. Bonus: plugins could be written in C, Rust or C++ now.

[fabianloewe]

The compiler could get its definitions from the library .cr files and then look for the binary containing this code in standard paths. If it finds one it could instead of compiling the code into the executable, link to this shared library. Otherwise it just compiles the library and links then to it (possibly also installs it). So far my idea. A bit more research and compiler modding could show what's possible but I believe that is achievable.

Anyway having a C API between it is definitely better than nothing and I'll go this way at the moment for my stuff.

Update: I found an answer to my question myself. Switching context.fun.linkage = LLVM::Linkage::Internal to External makes all symbols visible. This breaks a lot of other things but a step forward for me. 💃

[fabianloewe]

Some more progress: I made my first Crystal library which links to a Crystal application. The GC's are still seperated and I haven't tested any parameters but puts works. 😄
I will further play with the compiler in the next days when I have some time to spend.

If somebody is interested: https://github.com/hyronx/crystal-shared-lib

[ozra]

I wish dynamic linking was dead already B-) And also, that everyone shared their source, and everyone's source was in Crystal B-) (Pipe dream disintegrates and I'm suddenly back in the bitter reality.. hehe)

No but seriously: it's definitely cool PoC to fiddle with, but why on earth not take advantage of using shards and letting Crystal do it's whole-program-magic - if usage in Crystal was your aim? I'd get it if it was to expose a great crystal lib to C / insert-esoteric-language-here users...

Anyways, keep hacking :-) You'll probably invent something cool.

[ysbaddaden]

Plugins are really the single use case I can think of. Build support for countless providers but allow to only load a few of them; allowing external plugins to be developed without having to recompile the whole application (and all the plugins you want).

[fabianloewe]

Okay, imagine Crystal has grown in the last year/s. There are already multiple applications written in Crystal pre-installed. Do you really want to have every application include in its executable the same parts of code from the std library? Furthermore there will be some often used shards and their code will be included in every executable that's using them.

This is a huge storage wasting so the decision to separate code into shared libraries were already made lots of years ago. Why should we know do the opposite by not gaining anything from my point of view?

And what I try to reach: Exactlly that. Naturally compiling libraries downloaded by shards to shared libraries. Not for dynamic loading but linking as we are linking against pthread but everything more automatically through the compiler.

[ozra]

Do you really want to have every application include in its executable the same parts of code from the std library?

Yes, and no. Yes: I'd prefer all applications on my entire system statically linked. I think that's a much safer and more sustainable way of keeping a system intact: Decentralize, decouple, distribute.
No: Not the same parts - with today's linking methods only the used parts are included in the binary.

This is a huge storage wasting...

Storage is cheap, and cold pages of code (unused...) will be released from memory (the kernel knows where to find them).

...so the decision to separate code into shared libraries were already made lots of years ago.

The decision to separate code in to shared libraries was made yeeeeears ago when storage space was expensive and hard to cram in to an architecture at all. When there wasn't such a multitude of desktops and frameworks. If all apps used libc x.y.z and xlib u.v.w - then fine. When one app use KDE, one QT4 alone, one QT5 alone, one Gnome 3, one GTK3 alone, one GTK2... ad nauseam - the overhead outweighs having compiled them statically and having only the functionality actually used in the apps take space in mem, instead of entire chunks of dead code paths of the entire lib loaded as dynlib (which of course to be fair, would also be paged out though).

Why should we know do the opposite by not gaining anything from my point of view?

Maintaining ever changing dynamic dependencies is expensive, especially when they break.
As a fresh anecdote I upgraded one of my distros just two days ago, and the new version of libc refused to install, but the distro-upgrade continued with just a warning. At the end app after app fucked up. When doing some hell-holish gymnastics to get that package in manually after and salvage it, it turns out it had refused to install because "/lib" was in my LD_LIBRARY_PATH and that was a "don't-like-wont-install" prerequisite. That's brittle system integrity in my view. (Note my opinions are not based on this incident, I've held them for a while)

And on a flip side, comparisons for Linux distros aiming for everything statically compiled (in this case with musl as libc) indicate (aside from the obvious faster startups) that memory usage is actually lower. That's probably only true for core utils and not for all cases, (and perhaps because the comparison is dyn-linked glibc ;-) ). But this is not of immense weight to me.

There's always the ASLR argument, but that can easily be torn into shreds.

I don't mean to discourage you at all - these are just my two cents, and of course a lot of studies on this, on a wide variety of setups, would be needed to get a truely clear picture. I guess my main point is: not being able to dynlib cr-code shouldn't be a worry even if you don't share my view that it's actually a "feature" ;-)

[asterite]

I agree with what @ozra says. For example Mac OSX apps are mostly self-contained, and these apps contain duplicate dependencies between each other. This is fine, storage is cheap. And the good thing is that when you uninstall an app all of its dependencies go away too. And there's no conflict with versions and so on. So it's much simpler and you don't get the version messes you have in linux.

[ozra]

The version messes in Linux distros not using all statically linked binaries you mean ;-)

[fabianloewe]

Yes, storage is cheap but we also have Raspberry Pis (we can connect an external device, so not such a huge problem here) and smartphones today which only have 32GB of storage. I would really like to see Crystal applications running on them, too. (at least on rpi)

For desktops and laptops you are completely right. With shared libraries I also had in mind to link to existing code written in other languages in the future so we don't have to reinvent the wheel.

But maybe I'm on the wrong way... Currently there is no native static library support, right?

[ozra]

I was about to mention in my rant that even RPi's and hand helds wouldn't be a problem - but you might build enormously bigger applications than me B-)

[fabianloewe]

Not really. 😃

Okay okay, I drop this stuff. At least I've learned much about the compiler.

[RX14]

My interest would basically be to embed Crystal-written functionality (and presumably, the runtime) into a C program

Then why not embed the C program into the Crystal one? If you're only ever going to be using one crystal codebase in a given executable, this is equivalent and supported today.

[jemc]

It's not really equivalent because it doesn't meet the use case of distributing the functionality as a shared-library to be used in arbitrary C programs that I may or may not personally control or have access to the source code of.

[RX14]

that I may or may not personally control or have access to the source code of

Then you cannot guarantee they do not use another Crystal shared library.

[jemc]

Does the Crystal runtime rely on static/global state that pollutes the global namespace? Otherwise, I wouldn't expect the mere presence of another Crystal shared library to be a problem.

[rdp]

Yeah it might work...there's some confusion for me in terms of the threading model when it's called as embedded...

[RX14]

Does the Crystal runtime rely on static/global state that pollutes the global namespace? Otherwise, I wouldn't expect the mere presence of another Crystal shared library to be a problem.

Yes. It has a runtime, and that runtime doesn't expect to be loaded twice.

[jemc]

Yes. It has a runtime and [...]

Obviously I know that Crystal has a runtime, it's right there in my question about "the Crystal runtime". There's no need for this part of the comment, unless you are trying to imply that "doesn't expect to be loaded twice" is a universal feature of language runtimes. But that isn't the case. Not all language runtimes rely on static/global state, and some are properly isolated for such uses.

[rdp]

It's definitely possible. Just will take quite some work :) And probably not a high prio for the core guys, but PR's welcome! I still think this ticket should be reopened as a feature request :)

…

On Tue, Feb 18, 2020 at 1:27 PM Joe Eli McIlvain ***@***.***> wrote: Yes. It has a runtime and [...] Obviously I know that Crystal has a runtime, it's right there in my question about "the Crystal runtime". There's no need for this part of the comment, unless you are trying to imply that "doesn't expect to be loaded twice" is a universal feature of language runtimes. But that isn't the case. Not all language runtimes rely on static/global state, and some are properly isolated for such uses. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#921?email_source=notifications&email_token=AAADBUD2GKFVOBKIW44FO23RDRAC3A5CNFSM4BKEO2UKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMEP5PA#issuecomment-587792060>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAADBUCDCQTI3GKWQ7BZOTDRDRAC3ANCNFSM4BKEO2UA> .

[konovod]

Yes. It has a runtime and [...]

There's no need for this part of the comment, unless you are trying to imply that "doesn't expect to be loaded twice" is a universal feature of language runtimes.

It is not a universal feature, but it is a feature of Crystal runtime. Thing isn't only in polluting global namespace - i haven't looked in details, but at least two problems come in mind - crystal runtime sets signals handler (so second one will override first one) and boehm gc have global state that won't work nice with two gc users. Of course it is solvable.

[watzon]

Definitely should be reopened and probably tagged with "feature request" and "help wanted". Being able to export a C interface would be nice for a number of reasons, not the least of which is making something like a metacall loader really easy.

And with something like that we'd have full Crystal/Ruby interoperability.

[infoman]

I'd prefer all applications on my entire system statically linked.

IMO the entire concept of bundling/static-linking everything is horribly broken, because if one tiny library has a critical security issue, then suddenly all projects that depends on it need to recompile. And if it's a proprietary project and the author just don't care — we now have an application that is vulnerable by design.

[bellaz89]

@asterite
Hi! First of all, thank you very much for your effort! The language seems very promising and nice :)
Sorry if I ask dumb questions. Just curious..

I'm closing this issue. There's no way Crystal can be used to generate dynamic libraries. It has a GC.

I don't get the point here. boehm was originally made for C. Plus it has functions to arbitrarly remove/add roots and from the boehm gc.h header multiple invocations of GC_INIT are harmless.

There's simply no use trying to invest effort in this. It's simply not Crystal's target. It's like trying to creating a dynamic library with Ruby.

Well, Ruby modules can be and are used by C/C++ programs by linking the runtime (as for Lua/CPython and other GCed interpreters). Having the possibility to generate Crystal dynamic libraries would open the possibility to generate plugins and modules for other languages.

I know that the directiony you want to give to the language is different but I would e.g. totally love to use Crystal to speed up python code without relying on 'heavy' languages such as C/C++/Rust. This IMO would expose Crystal to a much bigger audience that is stuck with other languages for a variety of reasons.
For example: Why just not using only Crystal in the first place instead of Python? Because Python has still a much-much bigger ecosystem.

[RX14]

The tricky part is not GC, as far as I can see, it's that crystal programs, or any dynamic library which you make in Crystal, includes the standard library and all other libraries. Loading a single crystal dynamic library (in its own thread so it can own the event loop) is not that difficult to achieve. However it's inadvisable to do this, since linking the executable would break as soon as another dynamic library written in crystal is added, because there would be symbol clashes.

The core team isn't looking at this near-term, but as far as I know there have been a few proof of concepts for crystal dynamic libraries. There are clearly remaining problems to be solved, and I think they can be, but the effort will most likely come from the community and anyone interested and not the core team.

Some ideas to look into for anyone who wants to look into this:

• dynamic library mode which prefixes all symbols with a nonce, to allow multiple crystal programs to link together
• support wrapper for dynamic libraries which handles spawning the event loop in a new thread, and "portalling" calls from exported symbols in a calling thread to crystal functions running in a crystal thread (is this neccesary?)

[shayneoneill]

I realise this is probably beating on a dead horse, but it would be nice to have this facility. I write music software, and the primary market for independents is VST/AU/LV2 plugins, usually in the form of a .dll , .so or dylib. Still , the problem sounds like its a bit hard to really solve so oh well. But just wanna point out that the arguments philosophically against dylibs really arent generally applicable, there really are use cases for dlls that cant be solved other ways.

[rdp]

So you want to create dylib's is that right?

…

On Wed, Apr 20, 2022 at 4:37 AM shayneoneill ***@***.***> wrote: I realise this is probably beating on a dead horse, but it *would* be nice to have this facility. I write music software, and the primary market for independents is VST/AU/LV2 plugins, usually in the form of a .dll , .so or dylib. Still , the problem sounds like its a bit hard to really solve so oh well. But just wanna point out that the arguments philosophically against dylibs really arent generally applicable, there really are use cases for dlls that cant be solved other ways. — Reply to this email directly, view it on GitHub <#921 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAADBUGBKZSPDOW2FJEEOC3VF7M77ANCNFSM4BKEO2UA> . You are receiving this because you commented.Message ID: ***@***.***>

[axvm]

I want to vote for this feature. And I can't figure out why crystal is apparently trying to become a golang for Rubyists but can't have the same features as golang. I.e., golang has a build mode for dynamic libs. And I would love to do some computations in crystal and call it in ruby.

I believe this is an essential feature in the context of mass adoption.

[HertzDevil]

Today I have tried to build a true Android app written in pure Crystal from my Termux / Android NDK branch. The entry point must be defined through a shared library, and it basically boils down to this:

fun on_create = ANativeActivity_onCreate(activity : LibAndroid::ANativeActivity*, savedState : Void*, savedStateSize : LibC::SizeT)
  GC.init

  # we must pass something here otherwise `PROGRAM_NAME` will fail to initialize
  # this is _not_ `--prelude=empty`! we already have virtually the entire stdlib working
  args = ["(???)"]
  LibCrystalMain.__crystal_main(args.size, args.map(&.to_unsafe).to_unsafe)

  activity.value.callbacks.value.onStart = ->(activity : LibAndroid::ANativeActivity*) do
    # ...
  end
end

Crystal's runtime libraries (GC, libevent etc.) are then linked statically. This and the VST example are perfectly valid use cases where the shared library containing the Crystal runtime is effectively the only running Crystal instance, as the library still manages the whole lifecycle of an application, except that the entry point is no longer that of the usual command line. Thus building a shared library does not necessarily imply that we are consuming Crystal code from other Crystal code via this mechanism.

So I'd vote to reopen this issue.

(Now this scenario is solved by cross-compiling, but it won't be the case for VSTs, nor if someone decides to build the Android app on Termux itself.)

[crysbot]

This issue has been mentioned on Crystal Forum. There might be relevant details there:

https://forum.crystal-lang.org/t/is-it-possible-to-create-a-postgresql-hook-module-with-crystal/7209/2

[HertzDevil]

As always, it is helpful to first try this with the empty prelude before moving on to the Crystal runtime, since at the minimum we should be able to mimic simple C libraries. Our library will consist of a single function:

fun foo(x : Int32, y : Int32) : Int32
  x &+ y
end

You can now build this dynamic library today:

crystal build --prelude=empty foo.cr -o libfoo.so --link-flags=-shared

crystal build --prelude=empty foo.cr -o foo.dll --link-flags=/DLL --link-flags=/EXPORT:foo

And you can consume it right away: (on MSVC you'll have to add the current directory to %CRYSTAL_LIBRARY_PATH% first)

@[Link("foo")]
{% unless flag?(:msvc) %}
  @[Link(ldflags: "-L#{__DIR__}/.. -Wl,-rpath,#{__DIR__}/..")]
{% end %}
lib LibFoo
  fun foo(x : Int32, y : Int32) : Int32
end

LibFoo.foo(3, 5) # => 8

Note that loading the dynamic library will not call __crystal_main, which contains all the top-level code, e.g. constant and class variable initializers, including strings like Crystal::VERSION. Nor is it possible to call it directly, because __crystal_main simply resolves to the current program's symbol, rather than that of the dynamic library:

lib LibFoo
  fun __crystal_main(argc : Int32, argv : UInt8**)
end

LibFoo.__crystal_main(0, Pointer(UInt8*).null) # infinite loop

Likewise, if we are linking against two different dynamic libraries built by Crystal, their __crystal_main symbols will conflict. The solution might be adding the top-level code to @llvm.global_ctors instead, corresponding to __attribute__((constructor)) in GCC or Clang.

[ysbaddaden]

I've been looking into symbol linkage in Crystal:

• globals (constants, class variables, static strings, ...) are always declared with internal or private linkage;

• def when --single-module is specified are declared with internal linkage; without single module we need each object to be able to call to other objects so we need the symbols to be visible;

• oddity: def with the @[NoInline] annotation are declared with the external linkage. I fail to see the point; a quick clang / llvm-ir check proved that the noinline attribute is enough to prevent LLVM from inlining the function.

• fun are always declared with external linkage, which makes sense because fun are mostly calling to extern library symbols; yet the compiler expects a number __crystal_* funs whose usage is internal.

I'd like to introduce a @[Linkage("type")] annotation where value can be internal or external? With it we could control the linkage of individual fun and def.

I'd make all fun, except for extern lib funs, be internal by default, to make the external API explicit, but that's another topic.

If the generated __crystal_main function and all the fun __crystal_* have internal linkage, then there shouldn't be any more conflicts? Albeit it requires the library to expose a fun that calls __crystal_main to properly initialize the constants, class vars and global user code in general:

@[Linkage("external")]
fun mylib_init : Nil
  __crystal_main
end

[ysbaddaden]

Having the compiler set the linkage of __crystal_main to internal is enough to resolve the linkage issue, at least on Linux:

diff --git a/src/compiler/crystal/codegen/codegen.cr b/src/compiler/crystal/codegen/codegen.cr
index 7e15b1bdc..e4d05ba4a 100644
--- a/src/compiler/crystal/codegen/codegen.cr
+++ b/src/compiler/crystal/codegen/codegen.cr
@@ -283,6 +283,7 @@ module Crystal
       ret_type = @llvm_typer.llvm_return_type(@main_ret_type)
       main_type = LLVM::Type.function([llvm_context.int32, llvm_context.void_pointer.pointer], ret_type)
       @main = @llvm_mod.functions.add(MAIN_NAME, main_type)
+      @main.linkage = LLVM::Linkage::Internal if @single_module
       @fun_types = { {@llvm_mod, MAIN_NAME} => main_type }

       if @program.has_flag?("msvc")

The library can declare a fun mylib_init that calls LibCrystalMain.__crystal_main(0, nil) and it will call the symbol internal to the object 👍

Let's note that the empty prelude always declares a main fun, which means linking will fail (both objects have a conflicting main symbol). The solution is easy: the library musn't declare a main function 👍

[ysbaddaden]

The above, along with a Linkage annotation should be enough for my use case: porting my GC to Crystal using an alternative stdlib (for example nano).

In order to use the stdlib directly we could add the @[Linkage("internal")] annotation to all the fun __crystal_* definitions. As per __crystal_main as explained above, that would avoid conflicts.

Aparte: I checked (on linux) where two objects declare the same symbol, one internal and the other external and that doesn't affect linking (I assume issues arise when different objects declare the same symbol as external). Lucky, as it would affect my use case when using the GC in non release mode 😅

Aparte: there's one huge issue left for using the stdlib: the Boehm GC is global. We can't initialize multiple instances in the same process 😭

That would only affect using a Crystal library in a Crystal program, or using multiple Crystal libraries in the same executable, which doesn't have to be the case (e.g. Android NDK application).