with_toc_data with Japanese

[NathanHazout]

I have a markdown file in Japanese.

A header might look like:

## アプリケーション・ストリングの翻訳

The generated HTML looks like:
<h2 id=">アプリケーション・ストリングの翻訳</h2>

As you can see, it's not just that it leaves it empty, it generates invalid HTML (missing closing quote), breaking the entire page.

PS. This is happening inside a Jekyll 3 project

[eliot-akira]

I was just now experiencing the same issue using Jekyll, and came here to investigate.

### ヒューマンリーダブル

...becomes...

<h3 id=">ヒューマンリーダブル</h3>

Trial and error showed that even one Japanese character will produce this issue.

Possibly related to this commit and the regular expression used to strip inline markup from headers: 8ec8275

[KitaitiMakoto]

I also encountered the same problem. Minimal code to reproduce the problem is here:

# coding: utf-8                                                                                                                                
require 'redcarpet'

puts Redcarpet::VERSION

renderer = Redcarpet::Render::HTML.new(with_toc_data: true)
markdown = Redcarpet::Markdown.new(renderer, {})
puts markdown.render('# 見出し')

$ ruby broken-html.rb
3.3.4
<h1 id=">見出し</h1>

As an aside, "見出し" means heading(s).

[NathanHazout]

According to last comment, it means it is purely redcarpet and not Jekyll-related.
@robin850 any ideas?

[IdanAdar]

Any idea if this issue is being tracked by the redcarpet team?

[scvthedefect]

The heading id is generated by its content. For content do not match /^\w$/ will cause this bug, id is incomplete and not closed in the right way.

I use heading tags explicitly (<h1></h1>) to avoid this but don't feel good.

[NathanHazout]

@robin850 @vmg or anyone at redcarpet - any news with this?
Can someone review the attached Pull Request see if it's good?
We want to go live soon with our project and this is a problem...

[snkrheadz]

@nasht00 hi, this is my solution for now.

class MyRender < Redcarpet::Render::HTML
  def header(text, header_level)
    %Q{<h#{header_level} id="#{text.downcase.gsub(" ", "-")}">#{text}</h#{header_level}>}.html_safe
  end
end

[NathanHazout]

@akinrt I tried your fix.
It works is most cases, however it breaks if the header contains a link.

For example:

### [Setting up Your Development Environment](../setting-up-your-development-environment/)

Generates:

<h3 id="<a-href="../setting-up-your-development-environment/">setting-up-your-development-environment</a>"><a href="../setting-up-your-development-environment/">Setting up Your Development Environment</a></h3>

Which is obviously no good. It used to work before including this fix.

[robin850]

Hi guys!

We've a bunch of lacks regarding non-ASCII languages, sorry about that and sorry for the late answer here. I will try to have a look as soon as possible. Thanks for reporting !

[ryush00]

I also have the same issue.

### 한글 테스트

=>

<h3 id=">한글 테스트</h3>

Any updates?

[IdanAdar]

@robin850 Could this issue please be treated with the appropriate severity, given the many reports?

[ryush00]

There are some alternatives.

https://www.ruby-toolbox.com/categories/markup_processors

[ermaker]

Any update?

[lengerfulluse]

face the same problem with toc extension.
I think the root cause is https://github.com/vmg/redcarpet/blob/master/ext/redcarpet/html.c#L291. The isascii() check is not appropriate since it will cause generate empty header id for unicode character(CJK etc). Two possible solutions.

• use url encode for non-ascii character.
• use alternative isutf8() or some others check instead.

[gnujoow]

facing same problem redcarpet 3.3.4 in korean.

[robin850]

Hi everyone !

#591 should solve this problem ; I'm so sorry for the delay, thank you very much for your patience guys ! This will be part of Redcarpet 3.4.0 that should be released to Rubgems very soon ! Merry Christmas and happy new year ! ❤️ 🎄 🎅 ❄️ ⭐️

[robin850]

Redcarpet 3.4.0 is available on Rubygems ! Enjoy ! ❤️

[KitaitiMakoto]

Great!