importing UTF-8 encoded files

[tjaartvdwalt]

Hi,
I am using plugin-sass, which uses sass.js for parsing.
I logged dougludlow/plugin-sass#74, but the authors seem to think the problem is upstream.

The problem is that I cannot seem to import files that contain @charset "UTF-8";.

If you are interested, I have created a test repository to illustrate the problem: https://github.com/tjaartvdwalt/systemjs-sass-plugin-example

[rodneyrehm]

After running npm install sass.js on your repository, the following will serve as the minimal test case:

var fs = require('fs');
var Sass = require('sass.js');

Sass.importer(function(request, done) {
  // sass.js works in the "/sass/" directory, make that relative to CWD
  var path = '.' + request.resolved.replace(/^\/sass\//, '/' );
  var file = Sass.findPathVariation(fs.statSync, path);
  done({
    path: file,
    content: fs.readFileSync(file, {encoding: 'utf8'}),
  });
});

Sass.compile('@import "bourbon/_bourbon";', function(res) {
  console.log(res);
});

This produces the (absurd and unhelpful) error message

Error: property "C" must be followed by a ':'
        on line 37 of bourbon/bourbon/library/_margin.scss
>> C\u0003
   ^

Fiddling around with the content of _margin.scss, I figured out that the characters “ (copepoint 8220) and ” (codepoint 8221) caused the problem. Once they were replaced by regular double quotes the code was correctly parsed. Fiddling around some more it seems any non-ASCII character causes problems.

I'm not aware of any UTF-8 problems in the past so I'm a bit stumped by this.

Which node version are you running? (I'm on 7.1.0).

[jgillich]

What about this: https://github.com/jgillich/sassjs-bulma

Error: Invalid CSS after "@charset": expected 1 selector or at-rule, was '"utf-8"'
        on line 2 of node_modules/bulma/bulma.sass
>> @charset "utf-8"
   --------^

Something definitely seems off with UTF-8 handling.

[rodneyrehm]

Ok, things are beginning to make some sense now. The short answer is that content returned by the importer callback directly is treated differently from the content libsass reads from emscripten's FS (i.e. when using Sass.writeFile(), et al). The workaround is to have the importer callback pass the content to Sass.writeFile() and only return the path, but not the content, so that libsass will go through emscripten's FS. see the demo code below.

@tjaartvdwalt's problem has to do with UTF-8 not being ingested properly when the content is passed as string from the importer callback. I guess that's the fault of stringToPointer(), because '”'.length === 1 (1 character, but 2 bytes) looks like we're not allocating enough memory. I assume that's why for every multi-byte character we lose a character at the end and come to that incredibly useless error message.

@jgillich's problem has to do with libsass not realizing that the content provided by the importer callback is SASS, but attempts to parse everything as SCSS. That's why the @charset "UTF-8" is considered invalid, because there's no ; at the end.

---

I assume the UTF-8 problem can be dealt with somehow. However, I have no idea why the importer callback doesn't like SASS. Both problems disappear if the content from the real FS is imported in the following way:

var fs = require('fs');
var Sass = require('sass.js');

function fileExists(path) {
  var stat = fs.statSync(path);
  return stat && stat.isFile();
}

function importFileToSass(path, done) {
  // any path must be relative to CWD to work in both environments (real FS, and emscripten FS)
  var requestedPath = './' + path;
  // figure out the *actual* path of the file
  var filesystemPath = Sass.findPathVariation(fileExists, requestedPath);
  if (!filesystemPath) {
    done({
      error: 'File "' + requestedPath + '" not found',
    });

    return;
  }

  // write the file to emscripten FS so libsass internal FS handling
  // can engage the scss/sass switch, which apparently does not happen
  // for content provided through the importer callback directly
  var content = fs.readFileSync(filesystemPath, {encoding: 'utf8'});
  Sass.writeFile(filesystemPath, content, function() {
    done({
      path: filesystemPath,
    });
  });
}

Sass.importer(function(request, done) {
  // sass.js works in the "/sass/" directory, make that relative to CWD
  var requestedPath = request.resolved.replace(/^\/sass\//, '' );
  importFileToSass(requestedPath, done);
});

var rootFile = 'node_modules/bulma/bulma.sass';
importFileToSass(rootFile, function() {
  Sass.compileFile(rootFile, function(res) {
    console.log(res);
  });
});

[rodneyrehm]

I just released 0.10.0. This release fixes the UTF-8 problems encountered by @tjaartvdwalt. @jgillich's problem is not resolved, but the workaround I posted has been included in the the library

[jgillich]

Great news, thanks!

[tjaartvdwalt]

thanks!