llama : add xcframework build script #11996
Conversation
This commit adds a script to build an XCFramework for iOS. The script uses CMake to build the library for both the simulator and device and then creates an XCFramework that contains both builds. The XCFramework can then be added to a project and used in the same way as a regular framework. The llama.swiftui example project has been updated to use the XCFramework and can be started using the following command: ```console $ open examples/llama.swiftui/llama.swiftui.xcodeproj/ ``` Refs: ggml-org#10747
Remove the unnecessary whitespace in the build-xcframework.sh script.
Package.swift
Outdated
| .binaryTarget( | ||
| name: "llama", | ||
| path: "build-ios/llama.xcframework" | ||
| ), | ||
| //.systemLibrary(name: "llama", pkgConfig: "llama"), |
There was a problem hiding this comment.
Something that is not clear to me yet is now that we build a XCFramework, do we need this SPM package? Could we simply add the framework to the project and skip the package?
There was a problem hiding this comment.
The way we are using SPM where we expect llama.cpp to be installed on the local system, my understanding is that these are two ways of doing the same thing and perhaps we should just use framework which seems to be pretty simple.
The option of using SPM for source code distribution (not binary distribution or what is might be called when using a system library) would enable end users to include llama.cpp in their projects using something like this their projects Package.swift:
dependencies: [
.package(url: "https://github.com/ggerganov/llama.cpp.git", from: "1.0.0")
]
SPM would then download the source code, compile it, etc. But I'm not sure how much work it would require to get this working and maintain.
There was a problem hiding this comment.
Yes, so we want to move away from the workflow where the SPM package builds the llama from source for 2 reasons:
- We would need to maintain a second build system (i.e. the SPM package)
- It will not work when
ggmlbecomes a submodule in the future (SPM does not support submodules)
So it seems to me that we should probably remove the Package.swift all together and use the framework approach.
There was a problem hiding this comment.
So it seems to me that we should probably remove the
Package.swiftall together and use the framework approach.
This sounds like a good to me.
There was a problem hiding this comment.
@ggerganov I've removed Package.swift now as it was causing issues with xcodebuild and also produced an error in xcode (but it was still possible to build/run). I've updated the CI build to remove the install of llama.cpp as it should no longer be required, and also added a FRAMEWORKD_FOLDER_PATH for the xcodebuild commands.
There was a problem hiding this comment.
Both, there is also https://github.com/ggml-org/llama.cpp/tree/master/Sources
There was a problem hiding this comment.
I'll remove spm-headers and Sources shortly. (Package.swift has been removed and in llama : remove Package.swift)
There was a problem hiding this comment.
Package dependency management is not just an easy way to integrate dependencies.
It's also there to manage them. SPM actually makes use of the Git version tags, so, when using SPM, one can explicitly define a version they want (need) to use, and distribute that knowledge to others in a team.
If you remove Package.swift, that capability gets lost.
Also, if there ever will be support for executing build scripts on installation of SPM dependencies, this build script could be referenced there.
(cough In that regard, I would like to promote CocoaPods again, as that is capable of exactly such a feature…)
Currently, the nicest way would be to have a Github Action build the xcframework during a release and add that to a Github release page.
That could be referenced from a Package.swift and suddenly people would have a really easy time to integrate without needing a lot of knowledge on how to compile C++ things.
There was a problem hiding this comment.
Just to make sure I've understood this correctly, I as a user would be able to add the following to my project's Package.swift:
dependencies: [
.package(url: "https://github.com/ggml-org/llama.cpp.git", from: "b4777")
]And the Swift Package Manager would then clone the repository and look for a Package.swift file in the root of llama.cpp. This Package.swift could then looks something like the following:
import PackageDescription
let package = Package(
name: "llama",
platforms: [
.iOS(.v14),
.macOS(.v10_15),
.tvOS(.v14),
.visionOS(.v1)
],
products: [
.library(name: "llama", targets: ["llama"])
],
targets: [
.binaryTarget(
name: "llama",
url: "https://github.com/ggml-org/llama.cpp/releases/download/b4777/llama-b4777-xcframework.zip",
checksum: "the-sha256-checksum-here"
)
]
)Without Swift Package (Manual Integration):
- User has to manually go to our GitHub releases page
- Find the correct version of the xcframework.zip
- Download it
- Extract the zip file
- Drag the XCFramework into their Xcode project
- Configure build settings (embedding, signing, etc.)
- Repeat all these steps whenever they want to update to a newer version
With Swift Package (Automated Integration):
- User adds a single line to their dependencies
- Swift Package Manager automatically downloads the right binary
- Proper linking and integration happens automatically
- Updating is as simple as changing a version number
A very basic example can be found here
There was a problem hiding this comment.
Exactly.
To be completely honest, though, I should mention this one:
The checksum is mandatory and is only known after creating the xcframework.zip.
So, someone or something would need to update that in the Package.swift file after creating the new xcframework and, at best, then tag the release accordingly, or force move an already tagged release.
If that is too much hassle, the Package.swift part could be done in another repo which releases separately. Like https://github.com/srgtuszy/llama-cpp-swift.
github-actions
bot
added
the
devops
This commit removes the Package.swift file, as we are now building an XCFramework for the project.
This commit removes the reference to llama.cpp from the project.pbxproj file since Package.swift has been removed.
This commit adds FRAMEWORK_FOLDER_PATH to xcodebuild command in the xcodebuild commands so that the framwork can be found. It also removes the installing of llama.cpp to the system as that is not needed anymore and the xcodebuild/xcode builds will use the framework instead.
|
Tagging people who participated in recent iOS/Swift build discussions - please provide feedback about the proposed build changes in this PR. The new XCFramework approach proposed here should resolve all build issues for 3rd-party Swift projects depending on @jiabochao @nvoter @pgorzelany @Animaxx @MrMage @jhen0409 @hnipps @yhondri @tladesignz |
|
with xcframework I am able to build the swift example project, but when run it got error: when I trying to use in other project got error: |
|
@Animaxx Thanks for trying this out!
I'm trying to figure this out. Can you tell me if you are running this using the iphone simulator in xcode or some other way I can reproduce this?
I've been able to reproduce this and looking into it now. Thanks! Update: I've been looking at this issue and have updated the script to use static libraries which I should probably have done in the first place. With these changes, using the generated XCFramework I'm able to run the llama.swiftui example, and I also created a standalone example that imported the import SwiftUI
import llama
struct ContentView: View {
var body: some View {
VStack {
Image(systemName: "globe")
.imageScale(.large)
.foregroundStyle(.tint)
Text("Hello, world!")
}
.padding()
.onAppear {
// Test llama framework functionality
let params = llama_context_default_params()
print("Llama context params initialized:")
print("n_ctx: \(params.n_ctx)")
print("n_threads: \(params.n_threads)")
let start = ggml_time_us()
print("start: \(start)")
}
}
}
#Preview {
ContentView()
}I ran out of time today and I'll be out Mon-Tue, but I'll revisit the issue @Animaxx reported when I'm back on Wednesday. |
This commit updates the build script to use static libraries for the XCFramework build. This is done as linking with dynamic libraries is not supported on all platforms.
|
Hi @danbev thank you for the PR! The error I got is from debugging on the real device, not sure if that related to xcframework sign |
|
with the latest commit works on device now! trying to have validate archives got the error |
|
also when I use the xcframework to build for MacOS app, got error: with unsigned alert |
Ah, I missed building for macos, it currently only builds for ios. I'll try adding a bulid for macos. |
|
Thank you so much, @danbev, for starting this! I took your work for a test drive:
It ran on the following devices:
I didn't try out visionOS, since I don't have a device and I was too lazy to download the simulator. I stumbled over various minor things, however, so I suggest to take care of the following:
|
|
Im not sure what issue I'm running into on my m4 pro MacBook running latest version of Sequoia. I have all of the Xcode dependencies updated and I cannot get passed this. I ran into it the other day when I tried too but the same commit that had issues that day worked once today when I tried again and after pulling in the latest macOS changes from your repo I'm back to the same issues. |
|
@blaineam, that looks like your C compiler toolchain (LLVM) is broken. Run this: xcode-select --install |
This commit add support for visionOS in xcframework build script. This includes building for visionOS and visionOS simulator. Currently there is some duplication in the build script which I'll address in a follow-up commit as I think it should probably be refactored now that more platforms are being added.
This commmit changes the build directory to build-apple for the xcframework build script. It also introduces variables for the minimum OS versions for iOS, macOS, and visionOS. This is a work in progress in an attempt to cleanup/refactor the script.
|
🫶 |
This commit adds the ability to create a GitHub release with the xcframework build artifact.
yeah that didn't fix it, I already had that installed. needed to run The build made it farther, however it did end up failing on: |
This commit adds the Metal and Foundation frameworks to the xcframework module. This is necessary for the xcframework to be able to be used in projects that use Metal or Foundation. I did not notice this with projects where I added the xcframework to the project, but I did notice this when trying out Swift Package Manager with the xcframework (I've not committed the Package.swift file yet as there are still some discussions in progress about this)
Unfortunately, this log line doesn't tell how it failed. Anyway, this brings me to an idea:
Did you just upgrade from Sonoma to Sequoia? In that case, a lot of brew installed build tools are broken. You might want to run brew reinstall installed(That'll take time a lot of time, depending on the stuff you installed via brew. Maybe |
its not happy about something just not sure what. it seems cmake fails to find the c compiler and cxx compiler identifications sometimes and if either of those fail it cannot build. on rare occasion it finds them though ... had one almost successful run and then it failed too. then subsequent builds fail each time. no clue what is wrong on my MacBook to cause such intermittent issues but something is. I tried running the following: brew list | xargs brew reinstall
sudo rm -rf /Library/Developer/CommandLineTools;
brew install cmake
sudo xcodebuild -license accept
sudo xcodebuild -runFirstLaunch
sudo xcode-select --reset
sudo xcode-select --installCLI OutputAri/llama.cpp-temp/llama.cpp on xcframework-build-10747 took 8s Ari/llama.cpp-temp/llama.cpp on xcframework-build-10747
/Users/blainemiller/Documents/mine/Personal/Apps/Ari/llama.cpp-temp/llama.cpp/ggml/src/ggml-cpu/ggml-cpu-quants.c:741:22: warning: implicit CompileC /Users/blainemiller/Documents/mine/Personal/Apps/Ari/llama.cpp-temp/llama.cpp/build-ios-device/build/ggml-cpu.build/Release-iphoneos/Objects-normal/arm64/ggml-cpu-traits.o /Users/blainemiller/Documents/mine/Personal/Apps/Ari/llama.cpp-temp/llama.cpp/ggml/src/ggml-cpu/ggml-cpu-traits.cpp normal arm64 c++ com.apple.compilers.llvm.clang.1_0.compiler (in target 'ggml-cpu' from project 'llama.cpp') CompileC /Users/blainemiller/Documents/mine/Personal/Apps/Ari/llama.cpp-temp/llama.cpp/build-ios-device/build/ggml-cpu.build/Release-iphoneos/Objects-normal/arm64/ggml-cpu-hbm.o /Users/blainemiller/Documents/mine/Personal/Apps/Ari/llama.cpp-temp/llama.cpp/ggml/src/ggml-cpu/ggml-cpu-hbm.cpp normal arm64 c++ com.apple.compilers.llvm.clang.1_0.compiler (in target 'ggml-cpu' from project 'llama.cpp') CompileC /Users/blainemiller/Documents/mine/Personal/Apps/Ari/llama.cpp-temp/llama.cpp/build-ios-device/build/ggml-cpu.build/Release-iphoneos/Objects-normal/arm64/ggml-cpu-c0b5ffc4095df3ac83da75871c8389dd.o /Users/blainemiller/Documents/mine/Personal/Apps/Ari/llama.cpp-temp/llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c normal arm64 c com.apple.compilers.llvm.clang.1_0.compiler (in target 'ggml-cpu' from project 'llama.cpp') CompileC /Users/blainemiller/Documents/mine/Personal/Apps/Ari/llama.cpp-temp/llama.cpp/build-ios-device/build/ggml-cpu.build/Release-iphoneos/Objects-normal/arm64/ggml-cpu-aarch64.o /Users/blainemiller/Documents/mine/Personal/Apps/Ari/llama.cpp-temp/llama.cpp/ggml/src/ggml-cpu/ggml-cpu-aarch64.cpp normal arm64 c++ com.apple.compilers.llvm.clang.1_0.compiler (in target 'ggml-cpu' from project 'llama.cpp') note: Run script build phase 'Generate CMakeFiles/ZERO_CHECK' will be run during every build because the option to run the script phase "Based on dependency analysis" is unchecked. (in target 'ZERO_CHECK' from project 'llama.cpp') The following build commands failed: ... -- The C compiler identification is unknown |
|
I am getting an error when I try to compile this. Lots of lines that look like I also see these lines in the output I should note that I tested this before you added mac support (which I am very happy about) and it worked fine. I haven't tried reinstalling all my brew packages because it seems a bit heavy since it was working. |
|
so I noticed one of the errors was about Disk IO errors so I moved the llama.cpp repo to a non iCloud Drive synced folder and now builds are consistent and mostly working each time. It fails to build macOS with the same issues @schlu pointed out every time. iOS and iOS sim seem to build ok now that Im not using a icloud synced folder. |
|
@schlu Thanks for reporting this, I'm looking into this now. |
This commit sets GGML_NATIVE=OFF and adds -mcpu=apple-a12 to the C and C++ flags for the iOS simulator, iOS device, and macOS builds. This is an attempt to avoid native compilation and also find a suitable target architecture with regards to features tha the CPU supports.
This commit suppresses the warnings and reduces the output from the build commands. The motivation for this is that currrently there is so much output that it makes it hard to see the actual errors/issues.
|
Much worse now. I stops right away. I rolled back to commit 14d48be to make sure it was still working and it was still compiling the iOS version fine. Here is the entire output |
|
Ran into the same error as well. Didn't have time to test a previous commit though. |
This commit removes the -mcpu=apple-a12 flag from the ios-sim and
ios-device builds. This should not be required now that we are setting
GGML_NATIVE=OFF.
The motivation for this is that these flags were causing the following
error when building the xcframework:
```console
clang: error: unsupported option '-mcpu=' for target 'x86_64-apple-ios16.4-simulator'
```
Sorry about this (I am running the build locally and checking that the builds work in various projects to try to avoid wasting your time). I've pushed another commit which I hope will address this 🤞 |
|
It built fully this time with no errors I could see. |
Don't worry, thank you for your time. I am getting this error now: I went and downloaded the visonOS stuff in Xcode but the error persisted:
|
|
I do see this adding my app with the llama.xcframework to App Store Connect. And they emailed me these issues: Here the link they reference too: |
|
@schlu Can you tell me the version of cmake you are using? I'm asking because I think CMake added official support for visionOS in version 3.28.0. Based on this in the error you are seeing, perhaps that is the issue: System is unknown to cmake, create:
Platform/visionOS to use this system, please post your config file on discourse.cmake.org so it can be added to cmake
CMake Error at CMakeLists.txt:2 (project):If this is the case I'll add a check to the script. |
I've not tried this but I'll take a closer look at this tomorrow 👍 |
Thanks, I updated cmake and it worked! I should have tried this before I reported, I don't really use CMake much, I preferred to just use SPM but alas I am not a maintainer. |
This commit updates the message displayed when the required version of CMake is not installed. The message now specifies that CMake 3.28.0 or later is required so that visionOS is a system recognized by CMake.
This commit updates the structure of the XCFramework to meet the requirements for App Store submission. This involved generating debug symbols and the creation of dSYM bundles for each platform. While having the dSYM bundles should allow this framework to pass App Store validation. One thing to note though is that the UUID for the dSYM is missing for this combined static library but the object files them selfes contains debugging symbols which I've verified by running a simple macos application in lldb and stepping through the llama library code. Hopefully this will be enough for reports that are generated by the App Store but I don't now for sure yet as I currently don't have a developer account set up to test this.
|
With the latest commit I was able to create a project and validate it that uses the xcodeframework: 
@blaineam Would you be able to try this as well to see if there is anything we are missing for App Store Connect? |
This commit updates the project and removed the old reference to the xcframework in the build-ios director (now moved to build-apple). It also removes some of the development team and provisioning profile that were added in previous commit by mistake.
looks like its even less happy now: |
|
@blaineam Could you give me some pointers to how to recreate these errors? |
|
@danbev I do know Xcode does some stuff differently than apple's server side validation. Sometimes it's best to just upload something to confirm it will be validated by Apple's server side requirements. As for the minimum Xcode version that's probably tied to my app directly because my app has its minimum to like iOS 17 or 18 (can't remember which) |
|
@blaineam Thanks for that! I just validated an iphone app (the one I tried previously was a macos app) and I'm seeing the same errors. I'll dig into them in the morning. |
This commit adds a script to build an XCFramework for iOS. The script uses CMake to build the library for both the simulator and device and then creates an XCFramework that contains both builds.
The XCFramework can then be added to a project and used in the same way as a regular framework. The llama.swiftui example project has been updated to use the XCFramework and can be started using the following command:
$ open examples/llama.swiftui/llama.swiftui.xcodeproj/Refs: #10747