-
-
Notifications
You must be signed in to change notification settings - Fork 115
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support arbitrary encodings for XML documents #10
Comments
This, as you correctly identified in the comment to another issue, is likely a BOM mark which terminates the processing (BTW, there is no such encoding as DOS-UTF8, there is only just UTF-8). Unfortunately, I don't think I want to add special handling for it in Another reason I don't want to add BOM handling directly into the library is that BOM mark should not be used in UTF-8 encoded files - it is not even recommended by W3C. For now, your best bet will be dropping first character from the buffer if it is a BOM mark. However, as far as I'm aware, Rust |
Hi, have you mentioned this to the Rust developers? Philippe ----- Mail original ----- This, as you correctly identified in the comment to another issue, is likely a BOM mark which terminates the processing (BTW, there is no such encoding as DOS-UTF8, there is only just UTF-8). Unfortunately, I don't think I want to add special handling for it in rust-xml . Ideally this should be handled on the low level by I/O streams library, and rust-xml should use properly decoded streams of characters. There is an encoding library, rust-encode , which probably can be a basis for this, but I'm not sure if it provides proper streams and not only buffered decoding. Another reason I don't want to add BOM handling directly into the library is that BOM mark should not be used in UTF-8 encoded files - it is not even recommended by W3C. For now, your best bet will be dropping first character from the buffer if it is a BOM mark. However, as far as I'm aware, Rust Buffer does not provide an ability to "unread" characters (though it would be quite natural), so you will have to either to read your document into memory entirely and just slice it from the second byte onward, or you can write your own Buffer or even Reader implementation which filters out the first several bytes if they are UTF-8 BOM mark. — |
@PHIROC, there was an RFC on something like that, but it was postponed. I now looked more closely into |
There is a new library, |
latin1, ASCII, and UTF-16 are now supported. I'm not sure if there's demand for other encodings like legacy codepages or non-Unicode CJK encodings. Please open new issue if you need other encodings. |
I get the following error messages when I read a DOS-UTF8 XML file in a Rust program that uses rust-xml, on Windows 7:
If I convert the XML file to ANSI LATIN 1, the issue disappears.
My version of rustc is
rustc 0.12.0-pre-nightly (0bdac78da 2014-09-01 21:31:00 +0000)
The text was updated successfully, but these errors were encountered: