Support TypeScript-style "evolving" types for []-initialized values

[danvk]

Is your feature request related to a problem? Please describe.

Here's some valid Python code that pyright is unhappy with in strict mode:

# pyright: strict

xs = []  # type is list[Unknown]
xs.append(1) # error: Type of "append" is partially unknown
xs.append(2) # error: Type of "append" is partially unknown

print(sum(xs))  # error: Argument type is partially unknown

(pyright playground)

Describe the solution you’d like

Coming from the TypeScript world, I'd expect the type of xs to evolve as you append to it:

const xs = [];  // type is any[]
xs.push(1);
xs.push(2);

xs  // type is number[]

const sum = xs.reduce((a, b) => a + b);  // ok, type is number
console.log(sum);

(TypeScript playground)

This construct could be quite helpful for typing Python code without annotations.

It looks like pyright already does this for values initialized to None:

val = None

if random.random() < 0.5:
    val = 12
elif random.random() < 0.5:
    val = "abc"

val  # (variable) val: Literal[12, 'abc'] | None

So this is a feature request to add a similar behavior to values initialized to [] / empty list.

[erictraut]

This feature has been proposed previously for pyright. Although I like the feature in TypeScript, I'm very reluctant to add it in pyright for the following reasons.

1. In TypeScript, the Array type is a base type in the type system and is therefore special. In Python, the list type is not a base type. It's simply a class that derives from object. One could argue that it is somewhat special because the language includes syntax for constructing a list object — which is also true for sets and dictionaries. If a feature like this were ever added to pyright, it would be hard to justify adding it for only list types. At a minimum, it would also need to apply to set and dict as well.
2. In TypeScript, there are only a few methods that are commonly used to add items to an array: push, unshift, and assignment via index (x[n] = value). In Python, there are many methods and operators that are commonly used, and these differ across list, set and dict types. These would need to be special-cased and hard-coded in the type checker, which would override the type information found in the typeshed stubs.
3. TypeScript's type system tends to be a bit more "loose" than Python's. For example, arrays are treated as covariant when they really should be invariant for type safety. I personally think that TypeScript makes better tradeoffs between safety and usability, but many people in the Python world do not agree with me. The Python type system tends to prioritize type safety over usability in cases like this. Here's an example of where TypeScript's evolving type feature permits buggy code to go undetected even in strict mode:

function push_string(x: (number | string)[]) {
    x.push('');
}

function test() {
    const x = [];
    x.push(1);
    push_string(x);

    // x's type is inferred to be "number[]", but the array contains strings!
    return x;
}

I'll give this a bit more thought, but I wanted to share my current thinking.

[erictraut]

After thinking about this further, I've come up with even more reasons against implementing this in pyright.

4. I've found additional ways that TypeScript's approach results in type checking "holes". I don't think these are acceptable for Python given other rules that are codified in the Python type system.

function test(cond: boolean) {
    const x = [];

    if (cond) {
        x.push(1);
        // The inferred type of x here is "number[]"
    } else {
        x.push("hello");
        // The inferred type of x here is "string[]"
    }

    // The type of x here is "(number | string)[]"
    return x;
}

5. There are ways to close these holes, but they would limit where this approach could be used — meaning that there would be apparent inconsistencies. Also, while TypeScript's approach is cheap, implementing this in a way that closes all typing holes would be very expensive at analysis time. This additional analysis overhead would negatively impact many common use cases, including the latency of pylance completion suggestions.
6. This breaks the mental model that the type of an object is established at construction time and all methods called on that object are specialized based on the object's type.For all of these reasons, I'm going to reject this enhancement request.

The recommended approach is to use an explicit type annotation to specify the intended type of the list.

xs: list[int] = []
xs.append(1)

[danvk]

Thanks for considering the proposal @erictraut, and for the detailed response. Regarding point 4, perhaps I've just internalized how TypeScript handles these situations, but what is the "hole"? The type of x at each location is sound in that it includes all possible values at that location.

The TS feature has many of the same issues that you described. [] is handled specially, but Set, Map and {} are not. This feels natural in TS because Set and Map aren't used as frequently in JS as set and dict are in Python. The issue with variance is a general one that's not related to "evolving" types specifically. There's some work towards stricter assignability checks with readonly in microsoft/TypeScript#58296, but I think it will be a long time until TS catches the bug in your push_string example.

In any case, thanks for the rationale, and for the fantastic tool!