fix(interpreter): avoid overflow when checking if mem limit reached

fix(interpreter): avoid overflow when checking if mem limit reached #3433