Enzyme error

[swilliamson7]

Again not sure what to title this, but I'm facing an issue when I use Enzyme that previously didn't happen. Basically, the code used to work and now gives me:

jERROR: Enzyme compilation failed due to an internal error.
 Please open an issue with the code to reproduce and full error log on github.com/EnzymeAD/Enzyme.jl
 To toggle more information for debugging (needed for bug reports), set Enzyme.Compiler.VERBOSE_ERRORS[] = true (default false)

Illegal replace ficticious phi for:   %_replacementA32 = phi {} addrspace(10)* , !dbg !143 of   %33 = call nonnull "enzyme_type"="{[-1]:Pointer, [-1,0]:Integer, [-1,1]:Integer, [-1,2]:Integer, [-1,3]:Integer, [-1,4]:Integer, [-1,5]:Integer, [-1,6]:Integer, [-1,7]:Integer, [-1,8]:Pointer, [-1,8,-1]:Float@double}" {} addrspace(10)* @jl_genericmemory_copy_slice({} addrspace(10)* noundef %85, i64 %32, i64 %9) #87, !dbg !156

Stacktrace:
 [1] copy
   @ ./array.jl:350
 [2] unaliascopy
   @ ./abstractarray.jl:1516
 [3] unalias
   @ ./abstractarray.jl:1500
 [4] copyto!
   @ ./abstractarray.jl:1060

Stacktrace:
  [1] julia_error(msg::String, val::Ptr{…}, errtype::Enzyme.API.ErrorType, data::Ptr{…}, data2::Ptr{…}, B::Ptr{…})
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/7i3U8/src/errors.jl:384
  [2] julia_error(cstr::Cstring, val::Ptr{…}, errtype::Enzyme.API.ErrorType, data::Ptr{…}, data2::Ptr{…}, B::Ptr{…})
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/7i3U8/src/errors.jl:210
  [3] EnzymeCreatePrimalAndGradient(logic::Enzyme.Logic, todiff::LLVM.Function, retType::Enzyme.API.CDIFFE_TYPE, constant_args::Vector{…}, TA::Enzyme.TypeAnalysis, returnValue::Bool, dretUsed::Bool, mode::Enzyme.API.CDerivativeMode, runtimeActivity::Bool, width::Int64, additionalArg::Ptr{…}, forceAnonymousTape::Bool, typeInfo::Enzyme.FnTypeInfo, uncacheable_args::Vector{…}, augmented::Ptr{…}, atomicAdd::Bool)
    @ Enzyme.API ~/.julia/packages/Enzyme/7i3U8/src/api.jl:268
  [4] enzyme!(job::GPUCompiler.CompilerJob{…}, mod::LLVM.Module, primalf::LLVM.Function, TT::Type, mode::Enzyme.API.CDerivativeMode, width::Int64, parallel::Bool, actualRetType::Type, wrap::Bool, modifiedBetween::NTuple{…} where N, returnPrimal::Bool, expectedTapeType::Type, loweredArgs::Set{…}, boxedArgs::Set{…})
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/7i3U8/src/compiler.jl:1706
  [5] codegen(output::Symbol, job::GPUCompiler.CompilerJob{…}; libraries::Bool, deferred_codegen::Bool, optimize::Bool, toplevel::Bool, strip::Bool, validate::Bool, only_entry::Bool, parent_job::Nothing)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/7i3U8/src/compiler.jl:4550
  [6] codegen
    @ ~/.julia/packages/Enzyme/7i3U8/src/compiler.jl:3353 [inlined]
  [7] _thunk(job::GPUCompiler.CompilerJob{Enzyme.Compiler.EnzymeTarget, Enzyme.Compiler.EnzymeCompilerParams}, postopt::Bool)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/7i3U8/src/compiler.jl:5410
  [8] _thunk
    @ ~/.julia/packages/Enzyme/7i3U8/src/compiler.jl:5410 [inlined]
  [9] cached_compilation
    @ ~/.julia/packages/Enzyme/7i3U8/src/compiler.jl:5462 [inlined]
 [10] thunkbase(mi::Core.MethodInstance, World::UInt64, FA::Type{…}, A::Type{…}, TT::Type, Mode::Enzyme.API.CDerivativeMode, width::Int64, ModifiedBetween::NTuple{…} where N, ReturnPrimal::Bool, ShadowInit::Bool, ABI::Type, ErrIfFuncWritten::Bool, RuntimeActivity::Bool, edges::Vector{…})
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/7i3U8/src/compiler.jl:5573
 [11] thunk_generator(world::UInt64, source::LineNumberNode, FA::Type, A::Type, TT::Type, Mode::Enzyme.API.CDerivativeMode, Width::Int64, ModifiedBetween::NTuple{…} where N, ReturnPrimal::Bool, ShadowInit::Bool, ABI::Type, ErrIfFuncWritten::Bool, RuntimeActivity::Bool, self::Any, fakeworld::Any, fa::Type, a::Type, tt::Type, mode::Type, width::Type, modifiedbetween::Type, returnprimal::Type, shadowinit::Type, abi::Type, erriffuncwritten::Type, runtimeactivity::Type)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/7i3U8/src/compiler.jl:5758
 [12] autodiff
    @ ~/.julia/packages/Enzyme/7i3U8/src/Enzyme.jl:485 [inlined]
 [13] autodiff
    @ ~/.julia/packages/Enzyme/7i3U8/src/Enzyme.jl:544 [inlined]
 [14] autodiff
    @ ~/.julia/packages/Enzyme/7i3U8/src/Enzyme.jl:516 [inlined]
 [15] error1()
    @ Main ~/Documents/GitHub/data_assimilation_2/three_mass_spring/enzyme_error.jl:142
 [16] top-level scope
    @ REPL[10]:1
Some type information was truncated. Use `show(err)` to see complete types.

I updated Enzyme in this project yesterday to v0.13.29, and according to my git history the same code was running with Enzyme v0.11.13 (so, granted, quite a while back). I made a quick somewhat minimal example to run to reproduce the error:

using Enzyme
using Random
using Parameters
using LinearAlgebra

@with_kw mutable struct mso_params_ops{F<:Function}

    T::Int               # Total steps to integrate
    t::Int = 0           # placeholder for current step
    dt::Float64 = 0.001  # timestep

    x::Vector{Float64} = zeros(6)        # placeholder for state vector
    u::Matrix{Float64} = zeros(6, T+1)   # random forcing (if any), otherwise just leave zero 
    n::Matrix{Float64} = zeros(6, T+1)   # placeholder for noise to add to the data

    k::Float64 = 30          # spring constant
    r::Float64 = 0.5         # Rayleigh friction coefficient

    q::F                     # forcing function

    J::Float64 = 0.0         # cost function evaluation

    states::Matrix{Float64}    # placeholder for computed states

    A::Matrix{Float64} = zeros(6,6)                            # Time-step (x(t+1) = A x(t))
    B::Matrix{Float64} = diagm([1., 0., 0., 0., 0., 0.])       # Distributes known forcing  
    Gamma::Matrix{Float64} = zeros(6,6)                        # Distrubutes unknown (random) forcing

    E::Matrix{Float64} = zeros(6,6)           # Acts on data vector, generally the identity (e.g. full info on all positions/velocities)

    Q::Matrix{Float64} = zeros(6,6)           # Covariance matrix for unknown (random) forcing
    Q_inv::Float64
    R::Matrix{Float64} = zeros(6,6)           # Covariance matrix for noise in data
    R_inv::Matrix{Float64}                    # Inverse of operator R

    K::Matrix{Float64} = zeros(6,6)           # Placeholder for Kalman gain matrix
    Kc::Matrix{Float64} = zeros(6,6)     

end

function build_ops(params; E = zeros(6,6), R = zeros(6,6), K = zeros(6,6))
    
    dt = params.dt
    k = params.k
    r = params.r

    Rc = -r .* diagm(ones(3))
    Kc = [-2*k k 0; k -3*k k; 0 k -2*k]
    Ac = [zeros(3,3) diagm(ones(3))
         Kc Rc
    ]
    
    A = diagm(ones(6)) + dt .* Ac
    
    ms_ops = mso_operators(A=A,
    E=E,
    R=R,
    K=K,
    Kc=Kc
    )

    return ms_ops
end

function integrate(mso_struct::mso_params_ops)

    @unpack A, B, Gamma, E, R, R_inv, Kc, Q, Q_inv = mso_struct
    @unpack T, x, u, dt, states,q =  mso_struct
    @unpack J = mso_struct

    states[:,1] .= x

    # run the model forward to get estimates for the states 
    temp = 0.0
    for t = 2:T+1

        x .= A * x + B * [q(temp); 0.; 0.; 0.; 0.; 0.] + Gamma * u[:, t-1]
        states[:, t] .= copy(x)

        temp += dt

    end

    return nothing 

end

function error1()

    # Parameter choices 
    T = 10000          # Total number of steps to integrate
    r = 0.5
    q_true(t) = 0.1 * cos(2 * pi * t / (2.5 / r))                       # known forcing function 
    q_kf(t) = 0.5 * q_true(t)                                           # forcing seen by KF (and adjoint)
    data_steps = [1500 + k*1000 for k in 1:5]      # steps where data will be assimilated

    rand_forcing = 0.1 .* randn(T+1)
    u = zeros(6, T+1)
    u[1, :] .= rand_forcing

    Q_inv = 1
    # R_inv = ops.R^(-1)
    params_adjoint = mso_params_ops(T=T,
        t = 0,
        x = [1.0, 0.0, 0.0, 0.0, 0.0, 0.0],
        u = 0.0 .* u,
        n = 0.001 .* randn(6, T+1),
        q = q_kf,
        J = 0.0,
        states = zeros(6, T+1),
        Q_inv = Q_inv,
        R_inv = Diagonal(ones(6))
    )

    dparams = Enzyme.make_zero(params_adjoint)
    dparams.J = 1.0

    autodiff(Reverse, integrate, Duplicated(params_adjoint, dparams))

end

error1()

I'm going to try switching to a branch with all the code as it was back when things were running for now, but I tried reverting to an older Enzyme and that didn't seem to fix it

[wsmoses]

What Julia are you using, we should fix this but that error only can arise on Julia 1.11

[swilliamson7]

Ahhhh that could be why reverting to an old Enzyme didn't work! I recently updated to 1.11

[wsmoses]

Yeah generally I’d recommend Julia 1.10 (1.11 is still working things out)

[swilliamson7]

Okay, I'll go back to an older Julia. I'd tried updating because I'd heard that Enzyme might run faster on 1.11

[wsmoses]

Oh where did you hear that, I’d expect the opposite (Julia 1.11 made arrays into triple pointers so the generated code has less info and has to cache more), that said precompile time could be improved.

[swilliamson7]

I don't want to name names lol, but I'll go back to Julia 1.10! Maybe the Reactant stuff will suddenly work (big hope but not likely 😅 )