Unable to build v1.8.x with cpp_package in VS2019

[justanotheruser]

Description

mxnet_60 project fails to build

Error Message

Lots of them but only of two kinds:
Code Description Project File Line

calling a __host__ function("__floorf") from a __global__ function("mxnet::op::adaptiveaveragepool<float> ") is not allowed	mxnet_60	D:\clean_version\src\src\operator\contrib\adaptive_avg_pooling.cu	73	

identifier "__floorf" is undefined in device code	mxnet_60	D:\clean_version\src\src\operator\contrib\adaptive_avg_pooling.cu	73	

Rest of the errors are the same except for name of the function or file

Steps to reproduce

1. mkdir build
2. git clone --branch v1.8.x https://github.com/apache/incubator-mxnet.git src
3. cmake -G "Visual Studio 16 2019" -A x64^
   -T cuda=10.2,host=x64^
   -DCMAKE_POLICY_DEFAULT_CMP0104=NEW^
   -DCMAKE_BUILD_TYPE=Release^
   -DUSE_CPP_PACKAGE=1^
   -DUSE_CUDA=1^
   -DUSE_CUDNN=1^
   -DUSE_OPENCV=0^
   -DUSE_OPENMP=1^
   -DUSE_BLAS=mkl^
   -DMKL_ROOT="C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2020\windows\mkl"^
   -DMXNET_CUDA_ARCH=6.0;6.2;7.0;7.2;7.5^
   -DCUDNN_INCLUDE="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\include"^
   -DCUDNN_LIBRARY="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\lib\x64\cudnn.lib"^
   -S .\src -B .\build
4. Build solution in VS 2019

What have you tried to solve it?

1. Updating to latest patch of Visual Studio Community 2019
2. Reinstalling CUDA and cuDNN (after VS). Currently installed:
   Version 16.9.2
   VisualStudio.16.Release/16.9.2+31112.23
   Microsoft .NET Framework
   Version 4.8.04084
   Installed Version: Community
   Visual C++ 2019 00435-60000-00000-AA092
   Microsoft Visual C++ 2019
   NVIDIA CUDA 10.2 Wizards 10.2
   NVIDIA Nsight Visual Studio Edition - CUDA support 2019.4.0.19274
3. Building with VS2017
   Actually that how I wanted to build it initially but I've got compiler is out of heap space in pass 2 error for the same project which I couldn't resolve (as suggested somewhere here) by settings 'Link Time Code Optimization' to '/LTCG:PGOptimize' because VS can't find mxnet_60.pgd file. However, I've successfully built version without CUDA with VS 2019.

Environment

Environment Information

``` (base) C:\Users\User\Desktop>python diagnose.py ----------Python Info---------- Version : 3.8.3 Compiler : MSC v.1916 64 bit (AMD64) Build : ('default', 'Jul 2 2020 17:30:36') Arch : ('64bit', 'WindowsPE') ------------Pip Info----------- Version : 20.1.1 Directory : C:\Users\User\anaconda3\lib\site-packages\pip ----------MXNet Info----------- An error occured trying to import mxnet. This is very likely due to missing missing or incompatible library files. Traceback (most recent call last): File "diagnose.py", line 96, in check_mxnet print('Version :', mxnet.__version__) AttributeError: module 'mxnet' has no attribute '__version__'

----------System Info----------
Platform : Windows-10-10.0.19041-SP0
system : Windows
node : Slepcov-PC
release : 10
version : 10.0.19041
----------Hardware Info----------
machine : AMD64
processor : Intel64 Family 6 Model 165 Stepping 5, GenuineIntel
Name
Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz

----------Network Test----------
Setting timeout: 10
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0338 sec, LOAD: 0.5731 sec.
Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.2435 sec, LOAD: 0.3744 sec.
Error open Gluon Tutorial(cn): https://zh.gluon.ai, <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)>, DNS finished in 0.18741607666015625 sec.
Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.1526 sec, LOAD: 0.7710 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0227 sec, LOAD: 0.7252 sec.
Error open Conda: https://repo.continuum.io/pkgs/free/, HTTP Error 403: Forbidden, DNS finished in 0.14266681671142578 sec.
----------Environment----------
MXNET_CUDA="D:\Recognition Technologies\AvtoUragan ver 3.8\ExtLibs\MxNet\Win64"

</details>

[github-actions]

Welcome to Apache MXNet (incubating)! We are on a mission to democratize AI, and we are glad that you are contributing to it by opening this issue.
Please make sure to include all the relevant context, and one of the @apache/mxnet-committers will be here shortly.
If you are interested in contributing to our project, let us know! Also, be sure to check out our guide on contributing to MXNet and our development guides wiki.

[leezu]

Please try using the settings in https://github.com/apache/incubator-mxnet/blob/master/ci/build_windows.py

[justanotheruser]

@leezu Will (should) it work if I add USE_CPP_PACKAGE=1 in that script?
Thanks.

[leezu]

Let's try that. Would you like to open a PR targeting the v1.x branch that enables USE_CPP_PACKAGE=1 in above file? Then CI will test the USE_CPP_PACKAGE=1 build and answer the question

[justanotheruser]

I tried to build with settings in https://github.com/apache/incubator-mxnet/blob/master/ci/build_windows.py
This is exact command that I used: python build_windows.py --flavour WIN_GPU_MKLDNN
Build log (start)

(venv) D:\incubator-mxnet\ci>python build_windows.py --flavour WIN_GPU_MKLDNN
2021-03-29 02:03:58,868 MXNet Windows build helper
2021-03-29 02:03:58,868 Build flavour: WIN_GPU_MKLDNN
2021-03-29 02:03:58,868 Detected Windows platform
2021-03-29 02:03:58,868 Using vcvars environment:
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvars64.bat
2021-03-29 02:03:58,868 Found MXNet root: D:\incubator-mxnet
2021-03-29 02:03:59,158 Generating project with CMake:
"C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvars64.bat" && cmake -GNinja -DCMAKE_C_COMPILER=cl -DCMAKE_CXX_COMPILER=cl -DUSE_CUDA=ON -DUSE_CUDNN=ON
-DENABLE_CUDA_RTC=ON -DUSE_OPENCV=ON -DUSE_OPENMP=ON -DUSE_BLAS=open -DUSE_LAPACK=ON -DUSE_DIST_KVSTORE=OFF -DMXNET_CUDA_ARCH="5.2" -DUSE_MKLDNN=ON -DCMAKE_BUILD_TYPE=Release D:\incubato
r-mxnet

---

** Visual Studio 2019 Developer Command Prompt v16.9.2
** Copyright (c) 2021 Microsoft Corporation

---

[vcvarsall.bat] Environment initialized for: 'x64'
-- The C compiler identification is MSVC 19.28.29913.0
-- The CXX compiler identification is MSVC 19.28.29913.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.28.29910/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.28.29910/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- CMAKE_CROSSCOMPILING FALSE
-- CMAKE_HOST_SYSTEM_PROCESSOR AMD64
-- CMAKE_SYSTEM_PROCESSOR AMD64
-- CMAKE_SYSTEM_NAME Windows
-- CMake version '3.20.0-rc4' using generator 'Ninja'
-- Looking for a CUDA compiler
-- Looking for a CUDA compiler - C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.2/bin/nvcc.exe
-- The CUDA compiler identification is NVIDIA 10.2.89
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.2/bin/nvcc.exe - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- The ASM_MASM compiler identification is MSVC
-- Found assembler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.28.29910/bin/Hostx64/x64/ml64.exe
CMake Deprecation Warning at 3rdparty/mkldnn/CMakeLists.txt:17 (cmake_minimum_required):
Compatibility with CMake < 2.8.12 will be removed from a future version of
CMake.

Update the VERSION argument value or use a ... suffix to tell
CMake that the project does not need compatibility with older versions.

-- Intel MKL-DNN compat: set DNNL_BUILD_EXAMPLES to MKLDNN_BUILD_EXAMPLES with value OFF
-- Intel MKL-DNN compat: set DNNL_BUILD_TESTS to MKLDNN_BUILD_TESTS with value OFF
-- Intel MKL-DNN compat: set DNNL_ENABLE_JIT_PROFILING to MKLDNN_ENABLE_JIT_PROFILING with value OFF
-- Intel MKL-DNN compat: set DNNL_LIBRARY_TYPE to MKLDNN_LIBRARY_TYPE with value STATIC
-- Intel MKL-DNN compat: set DNNL_ARCH_OPT_FLAGS to MKLDNN_ARCH_OPT_FLAGS with value ``
-- Looking for pthread.h
-- Looking for pthread.h - not found
-- Found Threads: TRUE
-- Found OpenMP_C: -openmp (found version "2.0")
-- Found OpenMP_CXX: -openmp (found version "2.0")
-- Found OpenMP: TRUE (found version "2.0")
-- GPU support is disabled
-- Found Git: C:/Program Files/Git/cmd/git.exe (found version "2.29.2.windows.3")
-- Primitive cache is enabled
-- Found OpenBLAS libraries: D:/OpenBLAS-v0.2.19-Win64-int32/lib/libopenblas.dll.a
-- Found OpenBLAS include: D:/OpenBLAS-v0.2.19-Win64-int32/include
-- OpenCV ARCH: x64
-- OpenCV RUNTIME: vc15
-- OpenCV STATIC: OFF
-- Found OpenCV: C:/.conan/87a46b/1 (found version "4.1.1") found components: core highgui imgproc imgcodecs
-- Found OpenCV 4.1.1 in C:/.conan/87a46b/1/cmake/x64/vc15/lib
-- You might need to add C:.conan\87a46b\1\cmake\x64\vc15\bin to your PATH to be able to run your applications.
-- OpenCV 4.1.1 found (C:/.conan/87a46b/1/cmake/x64/vc15/lib)
-- OpenCV_LIBS=opencv_core;opencv_highgui;opencv_imgproc;opencv_imgcodecs
USE_LAPACK is ON
CMake Warning at 3rdparty/googletest/googletest/CMakeLists.txt:47 (project):
VERSION keyword not followed by a value or was followed by a value that
expanded to nothing.

CMake Deprecation Warning at 3rdparty/googletest/googletest/CMakeLists.txt:49 (cmake_minimum_required):
Compatibility with CMake < 2.8.12 will be removed from a future version of
CMake.

Update the VERSION argument value or use a ... suffix to tell
CMake that the project does not need compatibility with older versions.

-- Found PythonInterp: D:/incubator-mxnet/ci/venv/Scripts/python.exe (found version "3.6.8")
-- Found GTest: gtest
-- Found CUDNN: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.2/lib/x64/cudnn.lib
-- Looking for clock_gettime in rt
-- Looking for clock_gettime in rt - not found
-- Looking for fopen64
-- Looking for fopen64 - not found
-- Looking for C++ include cxxabi.h
-- Looking for C++ include cxxabi.h - not found
-- Looking for nanosleep
-- Looking for nanosleep - not found
-- Looking for backtrace
-- Looking for backtrace - not found
-- Could NOT find Backtrace (missing: Backtrace_LIBRARY Backtrace_INCLUDE_DIR)
-- D:/incubator-mxnet/3rdparty/dmlc-core/cmake/build_config.h.in -> include/dmlc/build_config.h
-- CUDA: Using the following NVCC architecture flags -gencode;arch=compute_52,code=sm_52
-- Found CUDAToolkit: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.2/include (found version "10.2.89")
-- Performing Test SUPPORT_MSSE3
-- Performing Test SUPPORT_MSSE3 - Failed
-- Performing Test SUPPORT_MSSE2
-- Performing Test SUPPORT_MSSE2 - Failed
-- Determining F16C support
F16C instruction set is not yet supported for MSVC
-- CUDA: Adding NVCC options: --fatbin-options --compress-all
-- Google Test not found
-- Configuring done
CMake Warning (dev) in CMakeLists.txt:
Policy CMP0104 is not set: CMAKE_CUDA_ARCHITECTURES now detected for NVCC,
empty CUDA_ARCHITECTURES not allowed. Run "cmake --help-policy CMP0104"
for policy details. Use the cmake_policy command to set the policy and
suppress this warning.

CUDA_ARCHITECTURES is empty for target "mxnet_52".
This warning is for project developers. Use -Wno-dev to suppress it.

CMake Warning (dev) in CMakeLists.txt:
Policy CMP0104 is not set: CMAKE_CUDA_ARCHITECTURES now detected for NVCC,
empty CUDA_ARCHITECTURES not allowed. Run "cmake --help-policy CMP0104"
for policy details. Use the cmake_policy command to set the policy and
suppress this warning.

CUDA_ARCHITECTURES is empty for target "customop_gpu_lib".
This warning is for project developers. Use -Wno-dev to suppress it.

-- Generating done
-- Build files have been written to: D:/incubator-mxnet/ci/build
2021-03-29 02:04:21,538 Building:
"C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvars64.bat" && ninja

---

** Visual Studio 2019 Developer Command Prompt v16.9.2
** Copyright (c) 2021 Microsoft Corporation

---

[vcvarsall.bat] Environment initialized for: 'x64'
[2/877] Building CXX object CMakeFiles\mxnet_52.dir\src\operator\identity_attach_KL_sparse_reg.cc.obj

After 1800000 more lines build log ends with ninja: build stopped: subcommand failed. Quick search by words "error" or "failed" gives this:

[571/877] Building CUDA object CMakeFiles\mxnet_52.dir\src\operator\contrib\adaptive_avg_pooling.cu.obj
FAILED: CMakeFiles/mxnet_52.dir/src/operator/contrib/adaptive_avg_pooling.cu.obj
C:\PROGRA1\NVIDIA2\CUDA\v10.2\bin\nvcc.exe -forward-unknown-to-host-compiler -DCUB_IGNORE_DEPRECATED_CPP_DIALECT -DDMLC_CORE_USE_CMAKE -DDMLC_LOG_FATAL_THROW=1 -DDMLC_LOG_STACK_TRACE_SIZE=0 -DDMLC_MODERN_THREAD_LOCAL=0 -DDMLC_STRICT_CXX11 -DDMLC_USE_CXX11 -DDMLC_USE_CXX11=1 -DMSHADOW_FORCE_STREAM -DMSHADOW_INT64_TENSOR_SIZE=0 -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_CUDA=1 -DMSHADOW_USE_CUDNN=1 -DMSHADOW_USE_F16C=0 -DMSHADOW_USE_MKL=0 -DMSHADOW_USE_SSE=0 -DMXNET_ENABLE_CUDA_RTC=1 -DMXNET_EXPORTS -DMXNET_USE_BLAS_OPEN=1 -DMXNET_USE_CUDA=1 -DMXNET_USE_LAPACK=1 -DMXNET_USE_MKLDNN=1 -DMXNET_USE_OPENCV=1 -DMXNET_USE_OPENMP=1 -DMXNET_USE_SIGNAL_HANDLER=1 -DNNVM_EXPORTS -DNOMINMAX -DUSE_CUDNN -DWIN32_LEAN_AND_MEAN -D_CRT_SECURE_NO_WARNINGS -D_SCL_SECURE_NO_WARNINGS -D__USE_XOPEN2K8 -Dmxnet_52_EXPORTS -I....\3rdparty\mkldnn\include -I3rdparty\mkldnn\include -I....\include -I....\src -I....\3rdparty\tvm\nnvm\include -I....\3rdparty\tvm\include -I....\3rdparty\dmlc-core\include -I....\3rdparty\dlpack\include -I....\3rdparty\nvidia_cub -I....\3rdparty\mshadow -I....\3rdparty\mkldnn\src..\include -I3rdparty\dmlc-core\include -isystem=D:\OpenBLAS-v0.2.19-Win64-int32\include -isystem=C:.conan\87a46b\1\include -isystem="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\include" -D_WINDOWS -Xcompiler="/W3 /GR /EHsc" --fatbin-options --compress-all -Xcompiler="-MD -O2 -Ob2" -DNDEBUG --gpu-architecture=compute_52 --gpu-code=sm_52,compute_52 "-Xcompiler=-MT -Gy /bigobj" -MD -MT CMakeFiles\mxnet_52.dir\src\operator\contrib\adaptive_avg_pooling.cu.obj -MF CMakeFiles\mxnet_52.dir\src\operator\contrib\adaptive_avg_pooling.cu.obj.d -x cu -c ....\src\operator\contrib\adaptive_avg_pooling.cu -o CMakeFiles\mxnet_52.dir\src\operator\contrib\adaptive_avg_pooling.cu.obj -Xcompiler=-FdCMakeFiles\mxnet_52.dir,-FS

There's also the same error message for bilinear_sampler.cu.obj and deformable_psroi_pooling.cu.obj. Can you advise next step?

[leezu]

Please check the failing files in the v1.x / master branch. There might be fixes that need to be backported for v1.8.x to work

[justanotheruser]

Unfortunately I'm still facing same errors. Perhaps something is wrong with CUDA installation on my machine, or with CUDA integration with Visual Studio. Are you actually cross-compiling MXNet in Docker container? I want to try to build this way now, but I don't quite understand what exactly should I do.

[leezu]

You could try updating to latest cuda and latest msvc. We don't cross-compile for windows

[justanotheruser]

@leezu Turns out the solution was not to update to the latest msvs but to downgrade to the previous version: https://devtalk.blender.org/t/cuda-compile-error-windows-10/17886/5
When I switched from 16.9 to 16.8.6, these errors disappeared. But I still got some (in 3 build attempts in a row):

D:\incubator-mxnet\3rdparty\nvidia_cub\cub\thread../util_type.cuh(900): error C2993: "T": illegal type for non-type template parameter "__formal"
D:\incubator-mxnet\3rdparty\nvidia_cub\cub\thread../util_type.cuh(900): error C2065: __T289: undeclared identifier
D:\incubator-mxnet\3rdparty\nvidia_cub\cub\thread../util_type.cuh(900): error C2923: std::_Select<__formal>::_Apply: "__T289" is not a valid template type argument for parameter ""
D:\incubator-mxnet\3rdparty\nvidia_cub\cub\thread../util_type.cuh(900): error C2062: type 'unknown-type' unexpected

Quick search led to this issue NVIDIA/thrust#1090 but I didn't find any Thrust headers in util_type.cuh. I'm also not sure if error depends on USE_CPP_PACKAGE=ON setting, so I opened a PR just to test it: #20108
In the meantime, will try to build with CUDA 11 and report back.
I honestly think that nightly builds of libmxnet.dll/.so would greatly improve MXNet accessibility.

[justanotheruser]

Btw, what is the exact version of MSVS that CI uses?

[leezu]

D:\incubator-mxnet\3rdparty\nvidia_cub\cub\thread../util_type.cuh(900): error C2993: "T": illegal type for non-type template parameter "__formal"
D:\incubator-mxnet\3rdparty\nvidia_cub\cub\thread../util_type.cuh(900): error C2065: __T289: undeclared identifier
D:\incubator-mxnet\3rdparty\nvidia_cub\cub\thread../util_type.cuh(900): error C2923: std::_Select<__formal>::_Apply: "__T289" is not a valid template type argument for parameter ""
D:\incubator-mxnet\3rdparty\nvidia_cub\cub\thread../util_type.cuh(900): error C2062: type 'unknown-type' unexpected

This is a bug in CUDA - MSVC integration and should be solved by upgrading to Cuda 11. Note that this issue is not thrust specific, but a bug in nvcc compiler. That's why it occurs for you even without thrust. For more details, see NVIDIA/thrust#1090

@josephevans may know the precise version. Alternativel, I think you can look at the compilation log of the CI build which should contain a line with the version number

[justanotheruser]

@leezu Yes, building with CUDA 11 did help. Maybe CUDA version in build_windows.py should reflect that? Anyway, I'm too lazy for another PR