macOS Mojave: Include clang header files when using compile_commands.json

[joelfrederico]

Issue Prelude

Please complete these steps and check these boxes (by putting an x inside
the brackets) before filing your issue:

• I have read and understood YCM's CONTRIBUTING document.
• I have read and understood YCM's CODE_OF_CONDUCT document.
• I have read and understood YCM's README, especially the
  Frequently Asked Questions section.
• I have searched YCM's issue tracker to find issues similar to the one I'm
  about to report and couldn't find an answer to my problem. (Example Google
  search.)
• If filing a bug report, I have included the output of vim --version.
• If filing a bug report, I have included the output of :YcmDebugInfo.
• If filing a bug report, I have attached the contents of the logfiles using
  the :YcmToggleLogs command.
• If filing a bug report, I have included which OS (including specific OS
  version) I am using.
• If filing a bug report, I have included a minimal test case that reproduces
  my issue, including what I expected to happen and what actually happened.
• If filing a installation failure report, I have included the entire output
  of install.py (or cmake/make/ninja) including its invocation
• I understand this is an open-source project staffed by volunteers and
  that any help I receive is a selfless, heartfelt gift of their free time. I
  know I am not entitled to anything and will be polite and courteous.
• I understand my issue may be closed if it becomes obvious I didn't
  actually perform all of these steps.

Thank you for adhering to this process! It ensures your issue is resolved
quickly and that neither your nor our time is needlessly wasted.

Issue Details

Provide a clear description of the problem, including the following key
questions:

• What did you do?

I included C++ headers.

• What did you expect to happen?

Locate the headers, understand the stuff in them.

• What actually happened?

Couldn't find the headers

Diagnostic data

Output of vim --version

Place the output here, or a link to a gist.

Output of YcmDebugInfo

Printing YouCompleteMe debug information...
-- Client logfile: /var/folders/8k/6z0xcptd00gbx6mgqvr0xgy00000gn/T/ycm_UcoIcf.log
-- Server Python interpreter: /opt/local/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
-- Server Python version: 2.7.15
-- Server has Clang support compiled in: True
-- Clang version: clang version 6.0.0 (tags/RELEASE_600/final)
-- No extra configuration file found
-- C-family completer debug information:
-- Compilation database path: /Users//Dev/Testing
-- Flags: [u'/opt/local/bin/clang++-mp-6.0', u'-isysroot', u'/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Dev
eloper/SDKs/MacOSX10.14.sdk', u'-std=gnu++14', u'-resource-dir=/Users//.vim/bundle/YouCompleteMe/third_party/ycmd/ycmd/../
clang_includes', u'-fspell-checking']
-- Translation unit: /Users//Dev/Testing/src/main.cpp
-- Server running at: http://127.0.0.1:55432
-- Server process ID: 10731
-- Server logfiles:

Contents of YCM, ycmd and completion engine logfiles

Add let g:ycm_log_level = 'debug' to vimrc, restart Vim, reproduce the
issue, and include link here to a gist containing the entire logfiles for
ycm, ycmd and any completer logfiles listed by :YcmToggleLogs.

OS version, distribution, etc.

macOS Mojave 10.14

Output of build/install commands

Include link to a gist containing the invocation and entire output of
install.py if reporting an installation issue.

[joelfrederico]

So from all the documentation I've read, this issue is that LibClang doesn't know where its STL is, while Clang does. There was some hack done to get this working for macOs. AFAICT, it looks like this isn't working for Mojave. I don't know where this code exists in YCM, or I might try to submit a PR.

Of course I lost the issue where Valloric mentions this issue. I'll keep looking for it...

[joelfrederico]

Ah it's here: #303 (comment)

[joelfrederico]

Also, this is sort of a unique problem for me. I have a custom installed clang (from MacPorts) in /opt/local, so the flags I need to get added are:

/opt/local/libexec/llvm-6.0/include/c++/v1
/opt/local/libexec/llvm-6.0/lib/clang/6.0.1/include

The problem is (obviously) that I'm using compile_commands.json, so even the usual hack of just assuming where the internal libs are won't be correct for me. This isn't so much a problem with iostream, but it becomes a problem when I want to build with advanced C++17 or C++2a with the latest branch of clang.

I think I can work around this by manually including the paths in my CMake so they show up in compile_commands.json, but this is... still a workaround.

Why not try at least to use my compile_commands.json to use my compiler (in this case, I know not all cases will have a compiler available) in order to figure out what those default include paths are? The full path to the clang I'm using is there, so we have all the info we need. We could scrape the -v output (https://stackoverflow.com/a/23658940) and get (and cache?) the paths we need to add so that libclang knows where to look. Yeah, it's ugly and I wish libclang would come with clang's STL and do the includes just like clang does, but... it would work...

Thoughts?

[bstaletic]

So from all the documentation I've read, this issue is that LibClang doesn't know where its STL is, while Clang does.

It's a little more complicated than that, but essentially that is correct.

There was some hack done to get this working for macOs. AFAICT, it looks like this isn't working for Mojave.

I love this. I really think Apple just likes to shuffle stuff around every few years just to screw with libclang users like we are.

I don't know where this code exists in YCM, or I might try to submit a PR.

Take a look at YouCompleteMe/third_party/ycmd/completers/cpp/flags.py. However, this is currently going through a rewrite. Take a look at ycm-core/ycmd#853.

I think I can work around this by manually including the paths in my CMake so they show up in compile_commands.json, but this is... still a workaround.

I agree that it is a workaround. However, sucessfully finding the user's headers on all OS's is hard and so has been mostly up to users to do that for now.

We could scrape the -v output

clang -E -v is almost exactly what ycm-core/ycmd#853 is doing, but again, it's mote complex than that.

and cache?

This is an open question in ycm-core/ycmd#853. So far we are leaning towards not caching, because at worst it takes 40ms to get the clang output and process it.

I have a custom installed clang (from MacPorts) in /opt/local

This is an interesting point. We might want to allow users to somehow tell us which clang, not just aim for clang from $PATH.

---

Bottom line is that you should definitely give ycm-core/ycmd#853 a try and report what you find.

[chrisniael]

I have same problem, after I have updated the macOS.
I solved the problem by change the source code here
make line
if OnMac() and not _SysRootSpecifedIn( flags ):
be
if OnMac():

I found the option -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk is added in file compile_commands.json after upgrading the macOS.
If I mannually remove the option from the compile_commands.json, then YCM works normally.

The function _SysRootSpecifedIn( flags ) is aim to judge if there is a -sysroot option in flags.

[joelfrederico]

@chrisniael I think the proper thing to do would be to fix _SysRootSpecifiedIn, yes? Naively, I think it's failing on my system due to a unicode issue. I may fix this and submit a PR...

FYI, the workaround does work.

@bstaletic Thanks for the info. I think I will request the ability to specify Clang, or deduce Clang from compile_commands.json in that thread.

[bstaletic]

Just a note about changing if OnMac() and not _SysRootSpecifiedIn( flags): to if OnMac():. It will make ycmd find wrong paths if one is cross compiling.

[xuyuheng]

+1, for Chromium Project, in both macOS 10.13 & 10.14.
Fix issues by add -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk to .ycm_extra_conf.py, and modified third_party/ycmd/ycmd/completers/cpp/flags.py as mentioned above.

Thanks @joelfrederico

[joelfrederico]

I should have been more specific, my workaround works. Modifying CMake to include the paths clang includes automatically does put them in the compile_commands.json, which ycm used to find the files.

[bstaletic]

My previous comment was really vague. I was refering to removing the _SysRootSpecifiedIn() part. Previous comment updated.

[evilpan]

The '-isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk' option added in compile_commands.json by cmake would prevent ycmd from finding std library such as iostream. Currently I just removed it like what @chrisniael said as a workaround. Maybe there're better solutions.

[progers]

My experience is similar to @xuyuheng's above. I'm using YCM on MacOS 10.13.6 with Chromium where we have a custom .ycm_extra_conf.py script that ends up adding:

-isysroot [chromiumdir]/src/build/mac_files/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk

By modifying _AddMacIncludePaths in third_party/ycmd/ycmd/completers/cpp/flags.py to be:

def _AddMacIncludePaths( flags ):
  if OnMac():
    flags.extend( MAC_INCLUDE_PATHS )
  return flags

as a workaround, I was able to get YCM working again.