Cached runs crashes on typechecking .filter-kwargs with IndexError

[flaeppe]

Bug report

Cached runs crashes on typechecking QuerySet.filter()-kwargs

What's wrong

error: Unexpected keyword argument "shipment_id" for "filter" of "ShipmentLineManager"  [call-arg]
                    ShipmentLine.objects.filter(
                    ^
error: INTERNAL ERROR -- Please try using mypy master on Github:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.931
Traceback (most recent call last):
  File "mypy/checkexpr.py", line 3962, in accept
  File "mypy/nodes.py", line 1713, in accept
  File "mypy/checkexpr.py", line 288, in visit_call_expr
  File "mypy/checkexpr.py", line 371, in visit_call_expr_inner
  File "mypy/checkexpr.py", line 878, in check_call_expr_with_callee_type
  File "mypy/checkexpr.py", line 938, in check_call
  File "mypy/checkexpr.py", line 1068, in check_callable_call
  File "mypy/checkexpr.py", line 758, in apply_function_plugin
  File "/app/venv/lib/python3.10/site-packages/mypy_django_plugin/transformers/orm_lookups.py", line 12, in typecheck_queryset_filter
    lookup_kwargs = ctx.arg_names[1]
IndexError: list index out of range

How is that should be

.filter-kwargs should be properly typechecked on cached mypy runs.

System information

• OS:
• python version: 3.10.1
• django version: 4.0.1
• mypy version: 0.931
• django-stubs version: 1.9.0 (but also failing on master)
• django-stubs-ext version: 0.3.1

[flaeppe]

I think I've hunted down the issue. I realised that on cached runs, any manager/queryset methods ended up with a MethodContext like this:

MethodContext(type=warehouse.models.ShipmentLineManager[warehouse.models.ShipmentLine], arg_types=[], arg_kinds=[], callee_arg_names=[], arg_names=[], default_return_type=warehouse.managers.ShipmentLineQuerySet[warehouse.models.ShipmentLine*], args=[], context=<mypy.nodes
.CallExpr object at 0x7fb13789d3f0>, api=<mypy.checker.TypeChecker object at 0x7fb136dc4500>)

Which is the source of the index-error. There are no arguments there(all lists are empty). Also, when inspecting the objects.filter-method on a cached round, it looked like this:

(Pdb++) ctx.type.type.names["filter"].node.type
def (self: warehouse.models.ShipmentLineManager[_T`1]) -> warehouse.managers.ShipmentLineQuerySet[_T`1]

While on non-cached rounds it looks like this (which is what typecheck_queryset_filter expects), which looks correct

def (self: warehouse.models.ShipmentLineManager[_T`1], *args: Any, **kwargs: Any) -> warehouse.managers.ShipmentLineQuerySet[_T`1]

I also extracted the mypy cache object for the .filter-method. Expand below

Cache object for filter method

          "filter": {
            ".class": "SymbolTableNode",
            "kind": "Mdef",
            "node": {
              ".class": "FuncDef",
              "arg_kinds": [
                0,
                2,
                4
              ],
              "arg_names": [
                "self",
                "args",
                "kwargs"
              ],
              "flags": [],
              "fullname": "warehouse.models.ShipmentLineManager.filter",
              "name": "filter",
              "type": {
                ".class": "CallableType",
                "arg_kinds": [
                  0,
                  2,
                  4
                ],
                "arg_names": [
                  "self",
                  "args",
                  "kwargs"
                ],
                "arg_types": [
                  {
                    ".class": "Instance",
                    "args": [
                      {
                        ".class": "TypeVarType",
                        "fullname": "django.db.models.manager._T",
                        "id": 1,
                        "name": "_T",
                        "upper_bound": "django.db.models.base.Model",
                        "values": [],
                        "variance": 1
                      }
                    ],
                    "type_ref": "warehouse.models.ShipmentLineManager"
                  },
                  {
                    ".class": "AnyType",
                    "missing_import_name": null,
                    "source_any": null,
                    "type_of_any": 2
                  },
                  {
                    ".class": "AnyType",
                    "missing_import_name": null,
                    "source_any": null,
                    "type_of_any": 2
                  }
                ],
                "bound_args": [],
                "def_extras": {},
                "fallback": "builtins.function",
                "implicit": false,
                "is_ellipsis_args": false,
                "name": "filter of ShipmentLineManager",
                "ret_type": {
                  ".class": "Instance",
                  "args": [
                    {
                      ".class": "TypeVarType",
                      "fullname": "django.db.models.manager._T",
                      "id": 1,
                      "name": "_T",
                      "upper_bound": "django.db.models.base.Model",
                      "values": [],
                      "variance": 1
                    }
                  ],
                  "type_ref": "warehouse.managers.ShipmentLineQuerySet"
                },
                "type_guard": null,
                "variables": []
              }
            },
            "plugin_generated": true
          }

Right, so the cached object looks fine. But if one pays a bit extra attention, there's no "arguments" key serialized for the FuncDef. Also briefly mentioned in mypy at these places:

• https://github.com/python/mypy/blob/5d765e71fa1313830b485a4e9f18a6553b221426/mypy/nodes.py#L632
• https://github.com/python/mypy/blob/5d765e71fa1313830b485a4e9f18a6553b221426/mypy/nodes.py#L729-L744

That's the reason we get a weird function-definition for e.g. .filter (but in reality on all copied manager methods) on cached runs. Because the lib.helpers.copy_method_to_another_class function (used for dynamically generating managers) expects the method_node: FuncDef argument to have .arguments. But that attribute is never cached by mypy..

django-stubs/mypy_django_plugin/lib/helpers.py

Lines 409 to 412 in 220f0e4

	try:
	original_arguments = method_node.arguments[1:]
	except AttributeError:
	original_arguments = []

Or, more precisely, it doesn't take into consideration that mypy always sets arguments to an empty list and then delete the attribute when a FuncDef instance is created from it's deserialize method:

https://github.com/python/mypy/blob/5d765e71fa1313830b485a4e9f18a6553b221426/mypy/nodes.py#L750-L762