Issue with packaging as an native dependency and rebuilding with electron-rebuild -- bufferutils/node-gyp

[lmick002]

Bug report

Describe the bug

When adding @supabase/realtime-js to an electron project where the native dependencies have to be rebuild the install process fails due to bufferutil

To Reproduce

1. Clone https://github.com/electron-react-boilerplate/electron-react-boilerplate
2. Install the realtime library as a native dependency (see: https://electron-react-boilerplate.js.org/docs/native-modules)
3. View error when running install

Expected behavior

Able to install and rebuild superbase successfully within electron project.

System information

• OS: MacOS Big Sir
• Version of supabase-js: latest
• Python: Python 2.7.18
• Python3: Python 3.9.7
• Version of Node.js: v14.18.0

Additional context

Error Log:

running npm install -s "@supabase/supabase-js" in the native dependency directory installs the dependency but throws the below error during the electron-rebuild process:

`$ yarn electron-rebuild && yarn link-modules
yarn run v1.22.10
$ node -r ../../.erb/scripts/babel-register.js ../../.erb/scripts/electron-rebuild.js
⠏ Building modules: 0/2Makefile:343: *** missing separator. Stop.
Makefile:343: *** missing separator. Stop.
✖ Rebuild Failed

An unhandled error occurred inside electron-rebuild
node-gyp failed to rebuild '/Users/macbookpro/Documents/My-Company/app/app-desktop/build/app/node_modules/utf-8-validate'.
For more information, rerun with the DEBUG environment variable set to "electron-rebuild".

Error: make failed with exit code: 2

Error: node-gyp failed to rebuild '/Users/macbookpro/Documents/My-Company/app/app-desktop/build/app/node_modules/utf-8-validate'.
For more information, rerun with the DEBUG environment variable set to "electron-rebuild".

Error: make failed with exit code: 2

at NodeGyp.rebuildModule (/Users/macbookpro/Documents/My-Company/app/app-desktop/node_modules/electron-rebuild/lib/src/module-type/node-gyp.js:109:19)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async ModuleRebuilder.rebuildNodeGypModule (/Users/macbookpro/Documents/My-Company/app/app-desktop/node_modules/electron-rebuild/lib/src/module-rebuilder.js:94:9)
at async Rebuilder.rebuildModuleAt (/Users/macbookpro/Documents/My-Company/app/app-desktop/node_modules/electron-rebuild/lib/src/rebuild.js:226:9)
at async Promise.all (index 177)
at async Rebuilder.rebuild (/Users/macbookpro/Documents/My-Company/app/app-desktop/node_modules/electron-rebuild/lib/src/rebuild.js:180:13)
at async /Users/macbookpro/Documents/My-Company/app/app-desktop/node_modules/electron-rebuild/lib/src/cli.js:154:9
node:child_process:903
throw err;
^

Error: Command failed: ../../node_modules/.bin/electron-rebuild --parallel --force --types prod,dev,optional --module-dir .
at checkExecSyncError (node:child_process:826:11)
at execSync (node:child_process:900:15)
at Object. (/Users/macbookpro/Documents/My-Company/app/app-desktop/.erb/scripts/electron-rebuild.js:17:3)
at Module._compile (node:internal/modules/cjs/loader:1101:14)
at Module._compile (/Users/macbookpro/Documents/My-Company/app/app-desktop/node_modules/pirates/lib/index.js:99:24)
macbookpro@MacbookPros-MBP app % yarn
yarn install v1.22.10
[1/4] 🔍 Resolving packages...
success Already up-to-date.
$ yarn electron-rebuild && yarn link-modules
yarn run v1.22.10
$ node -r ../../.erb/scripts/babel-register.js ../../.erb/scripts/electron-rebuild.js
⠇ Building modules: 0/2 CC(target) Release/obj.target/bufferutil/src/bufferutil.o
CC(target) Release/obj.target/bufferutil/src/bufferutil.o
⠹ Building modules: 0/2error: /Library/Developer/CommandLineTools/usr/bin/lipo: can't move temporary file: Release/obj.target/bufferutil/src/bufferutil.o to file: Release/obj.target/bufferutil/src/bufferutil.o.lipo (No such file or directory)
clang: error: lipo command failed with exit code 1 (use -v to see invocation)
make: *** [Release/obj.target/bufferutil/src/bufferutil.o] Error 1
✖ Rebuild Failed

An unhandled error occurred inside electron-rebuild
node-gyp failed to rebuild '/Users/macbookpro/Documents/My-Company/app/app-desktop/build/app/node_modules/bufferutil'.
For more information, rerun with the DEBUG environment variable set to "electron-rebuild".

Error: make failed with exit code: 2

Error: node-gyp failed to rebuild '/Users/macbookpro/Documents/My-Company/app/app-desktop/build/app/node_modules/bufferutil'.
For more information, rerun with the DEBUG environment variable set to "electron-rebuild".

Error: make failed with exit code: 2

at NodeGyp.rebuildModule (/Users/macbookpro/Documents/My-Company/app/app-desktop/node_modules/electron-rebuild/lib/src/module-type/node-gyp.js:109:19)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async ModuleRebuilder.rebuildNodeGypModule (/Users/macbookpro/Documents/My-Company/app/app-desktop/node_modules/electron-rebuild/lib/src/module-rebuilder.js:94:9)
at async Rebuilder.rebuildModuleAt (/Users/macbookpro/Documents/My-Company/app/app-desktop/node_modules/electron-rebuild/lib/src/rebuild.js:226:9)
at async Promise.all (index 91)
at async Rebuilder.rebuild (/Users/macbookpro/Documents/My-Company/app/app-desktop/node_modules/electron-rebuild/lib/src/rebuild.js:180:13)
at async /Users/macbookpro/Documents/My-Company/app/app-desktop/node_modules/electron-rebuild/lib/src/cli.js:154:9
node:child_process:903
throw err;
^

Error: Command failed: ../../node_modules/.bin/electron-rebuild --parallel --force --types prod,dev,optional --module-dir .
at checkExecSyncError (node:child_process:826:11)
at execSync (node:child_process:900:15)
at Object. (/Users/macbookpro/Documents/My-Company/app/app-desktop/.erb/scripts/electron-rebuild.js:17:3)
at Module._compile (node:internal/modules/cjs/loader:1101:14)
at Module._compile (/Users/macbookpro/Documents/My-Company/app/app-desktop/node_modules/pirates/lib/index.js:99:24)
at Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
at Object.newLoader [as .js] (/Users/macbookpro/Documents/My-Company/app/app-desktop/node_modules/pirates/lib/index.js:104:7)
at Module.load (node:internal/modules/cjs/loader:981:32)
at Function.Module._load (node:internal/modules/cjs/loader:822:12)
at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:79:12) {
status: 255,
signal: null,
output: [ null, null, null ],
pid: 39612,
stdout: null,
stderr: null
}
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
SOLINK_MODULE(target) Release/bufferutil.node
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
macbookpro@MacbookPros-MBP app % clang: error: no such file or directory: 'Release/obj.target/bufferutil/src/bufferutil.o'
make: *** [Release/bufferutil.node] Error 1
`

$ yarn electron-rebuild && yarn link-modules
yarn run v1.22.10
$ node -r ../../.erb/scripts/babel-register.js ../../.erb/scripts/electron-rebuild.js
⠏ Building modules: 0/2Makefile:343: *** missing separator. Stop.
Makefile:343: *** missing separator. Stop.
✖ Rebuild Failed

An unhandled error occurred inside electron-rebuild
node-gyp failed to rebuild '/Users/macbookpro/Documents/Audio-Sundae/sundae/sundae-desktop/build/app/node_modules/utf-8-validate'.
For more information, rerun with the DEBUG environment variable set to "electron-rebuild".

Error: make failed with exit code: 2

Error: node-gyp failed to rebuild '/Users/macbookpro/Documents/Audio-Sundae/sundae/sundae-desktop/build/app/node_modules/utf-8-validate'.
For more information, rerun with the DEBUG environment variable set to "electron-rebuild".

Error: make failed with exit code: 2

at NodeGyp.rebuildModule (/Users/macbookpro/Documents/Audio-Sundae/sundae/sundae-desktop/node_modules/electron-rebuild/lib/src/module-type/node-gyp.js:109:19)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async ModuleRebuilder.rebuildNodeGypModule (/Users/macbookpro/Documents/Audio-Sundae/sundae/sundae-desktop/node_modules/electron-rebuild/lib/src/module-rebuilder.js:94:9)
at async Rebuilder.rebuildModuleAt (/Users/macbookpro/Documents/Audio-Sundae/sundae/sundae-desktop/node_modules/electron-rebuild/lib/src/rebuild.js:226:9)
at async Promise.all (index 177)
at async Rebuilder.rebuild (/Users/macbookpro/Documents/Audio-Sundae/sundae/sundae-desktop/node_modules/electron-rebuild/lib/src/rebuild.js:180:13)
at async /Users/macbookpro/Documents/Audio-Sundae/sundae/sundae-desktop/node_modules/electron-rebuild/lib/src/cli.js:154:9
node:child_process:903
throw err;
^

Error: Command failed: ../../node_modules/.bin/electron-rebuild --parallel --force --types prod,dev,optional --module-dir .
at checkExecSyncError (node:child_process:826:11)
at execSync (node:child_process:900:15)
at Object. (/Users/macbookpro/Documents/Audio-Sundae/sundae/sundae-desktop/.erb/scripts/electron-rebuild.js:17:3)
at Module._compile (node:internal/modules/cjs/loader:1101:14)
at Module._compile (/Users/macbookpro/Documents/Audio-Sundae/sundae/sundae-desktop/node_modules/pirates/lib/index.js:99:24)
macbookpro@MacbookPros-MBP app % yarn
yarn install v1.22.10
[1/4] 🔍 Resolving packages...
success Already up-to-date.
$ yarn electron-rebuild && yarn link-modules
yarn run v1.22.10
$ node -r ../../.erb/scripts/babel-register.js ../../.erb/scripts/electron-rebuild.js
⠇ Building modules: 0/2 CC(target) Release/obj.target/bufferutil/src/bufferutil.o
CC(target) Release/obj.target/bufferutil/src/bufferutil.o
⠹ Building modules: 0/2error: /Library/Developer/CommandLineTools/usr/bin/lipo: can't move temporary file: Release/obj.target/bufferutil/src/bufferutil.o to file: Release/obj.target/bufferutil/src/bufferutil.o.lipo (No such file or directory)
clang: error: lipo command failed with exit code 1 (use -v to see invocation)
make: *** [Release/obj.target/bufferutil/src/bufferutil.o] Error 1
✖ Rebuild Failed

An unhandled error occurred inside electron-rebuild
node-gyp failed to rebuild '/Users/macbookpro/Documents/Audio-Sundae/sundae/sundae-desktop/build/app/node_modules/bufferutil'.
For more information, rerun with the DEBUG environment variable set to "electron-rebuild".

Error: make failed with exit code: 2

Error: node-gyp failed to rebuild '/Users/macbookpro/Documents/Audio-Sundae/sundae/sundae-desktop/build/app/node_modules/bufferutil'.
For more information, rerun with the DEBUG environment variable set to "electron-rebuild".

Error: make failed with exit code: 2

at NodeGyp.rebuildModule (/Users/macbookpro/Documents/Audio-Sundae/sundae/sundae-desktop/node_modules/electron-rebuild/lib/src/module-type/node-gyp.js:109:19)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async ModuleRebuilder.rebuildNodeGypModule (/Users/macbookpro/Documents/Audio-Sundae/sundae/sundae-desktop/node_modules/electron-rebuild/lib/src/module-rebuilder.js:94:9)
at async Rebuilder.rebuildModuleAt (/Users/macbookpro/Documents/Audio-Sundae/sundae/sundae-desktop/node_modules/electron-rebuild/lib/src/rebuild.js:226:9)
at async Promise.all (index 91)
at async Rebuilder.rebuild (/Users/macbookpro/Documents/Audio-Sundae/sundae/sundae-desktop/node_modules/electron-rebuild/lib/src/rebuild.js:180:13)
at async /Users/macbookpro/Documents/Audio-Sundae/sundae/sundae-desktop/node_modules/electron-rebuild/lib/src/cli.js:154:9
node:child_process:903
throw err;
^

Error: Command failed: ../../node_modules/.bin/electron-rebuild --parallel --force --types prod,dev,optional --module-dir .
at checkExecSyncError (node:child_process:826:11)
at execSync (node:child_process:900:15)
at Object. (/Users/macbookpro/Documents/Audio-Sundae/sundae/sundae-desktop/.erb/scripts/electron-rebuild.js:17:3)
at Module._compile (node:internal/modules/cjs/loader:1101:14)
at Module._compile (/Users/macbookpro/Documents/Audio-Sundae/sundae/sundae-desktop/node_modules/pirates/lib/index.js:99:24)
at Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
at Object.newLoader [as .js] (/Users/macbookpro/Documents/Audio-Sundae/sundae/sundae-desktop/node_modules/pirates/lib/index.js:104:7)
at Module.load (node:internal/modules/cjs/loader:981:32)
at Function.Module._load (node:internal/modules/cjs/loader:822:12)
at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:79:12) {
status: 255,
signal: null,
output: [ null, null, null ],
pid: 39612,
stdout: null,
stderr: null
}
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
SOLINK_MODULE(target) Release/bufferutil.node
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
macbookpro@MacbookPros-MBP app % clang: error: no such file or directory: 'Release/obj.target/bufferutil/src/bufferutil.o'
make: *** [Release/bufferutil.node] Error 1

[w3b6x9]

@lmick002 thanks for reporting. We'll investigate.

[w3b6x9]

@lmick002 you can install @supabase/realtime-js as a normal js dependency instead of a native dependency. Was there a specific reason you decided to install the realtime js lib as a native dependency?

[lmick002]

@w3b6x9 thanks for the quick response - whenever u install the @supabase/realtime-js as in the base directory and not native directory it throws this error:

`

postinstall
ts-node .erb/scripts/check-native-dep.js && electron-builder install-app-deps && cross-env NODE_ENV=development TS_NODE_TRANSPILE_ONLY=true webpack --config ./.erb/configs/webpack.config.renderer.dev.dll.ts && opencollective-postinstall

Webpack does not work with native dependencies.
@supabase/realtime-js is a native dependency and should be installed inside of the "./release/app" folder.
First, uninstall the packages from "./package.json":
npm uninstall your-package
Then, instead of installing the package to the root "./package.json":
npm install your-package
Install the package to "./release/app/package.json"
cd ./release/app && npm install your-package
Read more about native dependencies at:
https://electron-react-boilerplate.js.org/docs/adding-dependencies/#module-structure
`

[w3b6x9]

@w3b6x9 thanks for the quick response - whenever u install the @supabase/realtime-js as in the base directory and not native directory it throws this error:

`

postinstall
ts-node .erb/scripts/check-native-dep.js && electron-builder install-app-deps && cross-env NODE_ENV=development TS_NODE_TRANSPILE_ONLY=true webpack --config ./.erb/configs/webpack.config.renderer.dev.dll.ts && opencollective-postinstall

Webpack does not work with native dependencies. @supabase/realtime-js is a native dependency and should be installed inside of the "./release/app" folder. First, uninstall the packages from "./package.json": npm uninstall your-package Then, instead of installing the package to the root "./package.json": npm install your-package Install the package to "./release/app/package.json" cd ./release/app && npm install your-package Read more about native dependencies at: https://electron-react-boilerplate.js.org/docs/adding-dependencies/#module-structure `

@lmick002 got it, thanks for following up! I am able to reproduce your issue now and after investigating for a while I still haven't found a solution yet. I'll leave this open to see if anyone in the community has any ideas and I'll try to investigate more later on.

[hyferg]

The websocket dependency is evidently native and ERB can't handle that. This is the relevant bit of code from CheckNativeDeps.js that gets mad.

try {
    // Find the reason for why the dependency is installed. If it is installed
    // because of a devDependency then that is okay. Warn when it is installed
    // because of a dependency
    const { dependencies: dependenciesObject } = JSON.parse(
      execSync(`npm ls ${nativeDeps.join(' ')} --json`).toString()
    );
    const rootDependencies = Object.keys(dependenciesObject);
    const filteredRootDependencies = rootDependencies.filter((rootDependency) =>
      dependenciesKeys.includes(rootDependency)
    );
    if (filteredRootDependencies.length > 0) {
      const plural = filteredRootDependencies.length > 1;
      console.log(`
 ${chalk.whiteBright.bgYellow.bold(
   'Webpack does not work with native dependencies.'
 )}

You could for example list @supabase/realtime-js as a dev dependency to make this block of code happy. Not sure what that would mean in production though.

Probably the problem here is that the websocket library has both native and web code.

[w3b6x9]

@lmick002 were you able to find a workaround / get this working?

[ComputerDen]

anyone further forward with this? currently just uninstall and reinsatlling it where its needed

[lucasjstifano]

Currently having this exact same issue... anyone happen to find a work around?

[lmick002]

@w3b6x9 yes. I ended up having to update my python environment (at the time I was running ~v2..) but I'll double check to make sure it's fixed.

[SantoshKumarRavi]

I am having same issue with "fabric" native dependency. if anyone fixed these native deps , let me know .