New implicit-str-concat-in-sequence crashes

[xavfernandez]

Current behavior

00:02:04.979 Traceback (most recent call last):
00:02:04.979   File "/home/jenkins/venv/4/bin/pylint", line 11, in <module>
00:02:04.979     sys.exit(run_pylint())
00:02:04.979   File "/home/jenkins/venv/4/lib/python3.6/site-packages/pylint/__init__.py", line 20, in run_pylint
00:02:04.979     Run(sys.argv[1:])
00:02:04.979   File "/home/jenkins/venv/4/lib/python3.6/site-packages/pylint/lint.py", line 1608, in __init__
00:02:04.979     linter.check(args)
00:02:04.979   File "/home/jenkins/venv/4/lib/python3.6/site-packages/pylint/lint.py", line 938, in check
00:02:04.979     self._do_check(files_or_modules)
00:02:04.980   File "/home/jenkins/venv/4/lib/python3.6/site-packages/pylint/lint.py", line 1071, in _do_check
00:02:04.980     self.check_astroid_module(ast_node, walker, rawcheckers, tokencheckers)
00:02:04.980   File "/home/jenkins/venv/4/lib/python3.6/site-packages/pylint/lint.py", line 1154, in check_astroid_module
00:02:04.980     walker.walk(ast_node)
00:02:04.980   File "/home/jenkins/venv/4/lib/python3.6/site-packages/pylint/utils.py", line 1269, in walk
00:02:04.980     self.walk(child)
00:02:04.980   File "/home/jenkins/venv/4/lib/python3.6/site-packages/pylint/utils.py", line 1269, in walk
00:02:04.980     self.walk(child)
00:02:04.980   File "/home/jenkins/venv/4/lib/python3.6/site-packages/pylint/utils.py", line 1269, in walk
00:02:04.980     self.walk(child)
00:02:04.980   File "/home/jenkins/venv/4/lib/python3.6/site-packages/pylint/utils.py", line 1266, in walk
00:02:04.980     cb(astroid)
00:02:04.980   File "/home/jenkins/venv/4/lib/python3.6/site-packages/pylint/checkers/strings.py", line 613, in visit_tuple
00:02:04.980     self.check_for_concatenated_strings(node, "tuple")
00:02:04.980   File "/home/jenkins/venv/4/lib/python3.6/site-packages/pylint/checkers/strings.py", line 622, in check_for_concatenated_strings
00:02:04.980     (elt.lineno, elt.col_offset)
00:02:04.980 KeyError: (5, 39)

I'll try to find the crash-causing code.

Expected behavior

Do not crash ;)

pylint --version output

pylint 2.2.0
astroid 2.1.0
Python 3.6.7 (default, Nov 22 2018, 09:26:21)
[GCC 8.2.1 20180831]

[xavfernandez]

cc @Lucas-C of #1655 😇

[xavfernandez]

$ cat ko 
TOTO = (
    ('foo', 'bar',
     "Café", "Café"),
)
$ pylint ko
************* Module ko
ko:1:0: C0111: Missing module docstring (missing-docstring)
Traceback (most recent call last):
  File "/home/xfernandez/.virtualenvs/tmp/bin/pylint", line 11, in <module>
    sys.exit(run_pylint())
  File "/home/xfernandez/.virtualenvs/tmp/lib/python3.6/site-packages/pylint/__init__.py", line 20, in run_pylint
    Run(sys.argv[1:])
  File "/home/xfernandez/.virtualenvs/tmp/lib/python3.6/site-packages/pylint/lint.py", line 1608, in __init__
    linter.check(args)
  File "/home/xfernandez/.virtualenvs/tmp/lib/python3.6/site-packages/pylint/lint.py", line 938, in check
    self._do_check(files_or_modules)
  File "/home/xfernandez/.virtualenvs/tmp/lib/python3.6/site-packages/pylint/lint.py", line 1071, in _do_check
    self.check_astroid_module(ast_node, walker, rawcheckers, tokencheckers)
  File "/home/xfernandez/.virtualenvs/tmp/lib/python3.6/site-packages/pylint/lint.py", line 1154, in check_astroid_module
    walker.walk(ast_node)
  File "/home/xfernandez/.virtualenvs/tmp/lib/python3.6/site-packages/pylint/utils.py", line 1269, in walk
    self.walk(child)
  File "/home/xfernandez/.virtualenvs/tmp/lib/python3.6/site-packages/pylint/utils.py", line 1269, in walk
    self.walk(child)
  File "/home/xfernandez/.virtualenvs/tmp/lib/python3.6/site-packages/pylint/utils.py", line 1269, in walk
    self.walk(child)
  File "/home/xfernandez/.virtualenvs/tmp/lib/python3.6/site-packages/pylint/utils.py", line 1266, in walk
    cb(astroid)
  File "/home/xfernandez/.virtualenvs/tmp/lib/python3.6/site-packages/pylint/checkers/strings.py", line 613, in visit_tuple
    self.check_for_concatenated_strings(node, "tuple")
  File "/home/xfernandez/.virtualenvs/tmp/lib/python3.6/site-packages/pylint/checkers/strings.py", line 623, in check_for_concatenated_strings
    (elt.lineno, elt.col_offset)
KeyError: (3, 14)

while:

$  cat ok 
TOTO = (
    ('foo', 'bar',
     "Cafe", "Cafe"),
)

does not crash.

[Lucas-C]

Thanks for the minimal example reproducing the error.
This an encoding-related issue: due to the "é" character, .col_offset is wrong because astroid compute it as a number of bytes.
I'm working on a fix right now.

[Lucas-C]

I have a working solution for UTF-8 in #2611,
but I am struggling with my Latin1-encoding test now...
A "é" in Latin1 is a single-character, but it seems that astroid considers it as a 2-bytes...
Our Const.col_offset is set in TreeBuilder.visit_const from https://github.com/PyCQA/astroid/blob/master/astroid/rebuilder.py#L423, which comes from a, _ast3.Str created in the typed_ast module.

[Lucas-C]

I have created an issue in the typed_ast project to ask for some help with this:
python/typed_ast#65

[Lucas-C]

I added a workaround to not crash in the meantime

[Lucas-C]

The PR is ready to be merged.
Currently this checker will simply not work on latin1-encoded files.