Type system¶
Pyanalyze supports most of the Python type system, as specified in PEP 484 and various later PEPs and in the Python documentation. It uses type annotations to infer types and checks for type compatibility in calls and return types. Supported type system features include generics like List[int], NewType, TypedDict, TypeVar, and Callable.
Extensions¶
In addition to the standard Python type system, pyanalyze supports a number of non-standard extensions:
Callable literals: you can declare a parameter as
Literal[some_function]and it will accept any callable assignable tosome_function. Pyanalyze also supports Literals of various other types in addition to those supported by PEP 586.pyanalyze.extensions.AsynqCallableis a variant ofCallablethat applies toasynqfunctions.pyanalyze.extensions.ParameterTypeGuardis a generalization of PEP 649’sTypeGuardthat allows guards on any parameter to a function. To use it, returnAnnotated[bool, ParameterTypeGuard["arg", SomeType]].pyanalyze.extensions.HasAttrGuardis a similar mechanism that allows indicating that an object has a particular attribute. To use it, returnAnnotated[bool, HasAttrGuard["arg", "attribute", SomeType]].pyanalyze.extensions.ExternalTypeis a way to refer to a type that cannot be referenced by name in contexts where usingif TYPE_CHECKINGis not possible.pyanalyze.extensions.CustomCheckis a powerful mechanism to extend the type system with custom user-defined checks.
They are explained in more detail below.
Extended literals¶
Literal types are specified by PEP 586. The PEP only supports Literals of int, str, bytes, bool, Enum, and None objects, but pyanalyze accepts Literals over all Python objects.
As an extension, pyanalyze accepts any compatible callable for a Literal over a function type. This allows more flexible callable types.
For example:
from typing_extensions import Literal
def template(x: int, y: str = "") -> None:
pass
def takes_template(func: Literal[template]) -> None:
func(x=1, y="x")
def good_callable(x: int, y: str = "default", z: float = 0.0) -> None:
pass
takes_template(good_callable) # accepted
def bad_callable(not_x: int, y: str = "") -> None:
pass
takes_template(bad_callable) # rejected
AsynqCallable¶
The @asynq() callable in the asynq framework produces a special callable that can either be called directly (producing a synchronous call) or through the special .asynq() attribute (producing an asynchronous call). The AsynqCallable special form is similar to Callable, but describes a callable with this extra .asynq() attribute.
For example, this construct can be used to implement the asynq.tools.amap helper function:
from asynq import asynq
from pyanalyze.extensions import AsynqCallable
from typing import TypeVar, List, Iterable
T = TypeVar("T")
U = TypeVar("U")
@asynq()
def amap(function: AsynqCallable[[T], U], sequence: Iterable[T]) -> List[U]:
return (yield [function.asynq(elt) for elt in sequence])
Because of limitations in the runtime typing library, some generic aliases involving AsynqCallable will not work at runtime. For example, given a generic alias L = List[AsynqCallable[[T], int]], L[str] will throw an error. Quoting the type annotation works around this.
ParameterTypeGuard¶
PEP 647 added support for type guards, a mechanism to narrow the type of a variable. However, it only supports narrowing the first argument to a function.
Pyanalyze supports an extended version that combines with PEP 593’s Annotated type to support guards on any function parameter.
For example, the below function narrows the type of two of its parameters:
from typing import Iterable, Annotated
from pyanalyze.extensions import ParameterTypeGuard
from pyanalyze.value import KnownValue, Value
def _can_perform_call(
args: Iterable[Value], keywords: Iterable[Value]
) -> Annotated[
bool,
ParameterTypeGuard["args", Iterable[KnownValue]],
ParameterTypeGuard["keywords", Iterable[KnownValue]],
]:
return all(isinstance(arg, KnownValue) for arg in args) and all(
isinstance(kwarg, KnownValue) for kwarg in keywords
)
HasAttrGuard¶
HasAttrGuard is similar to ParameterTypeGuard and TypeGuard, but instead of narrowing a type, it indicates that an object has a particular attribute. For example, consider this function:
from typing import Literal, Annotated
from pyanalyze.extensions import HasAttrGuard
def has_time(arg: object) -> Annotated[bool, HasAttrGuard["arg", Literal["time"], int]]:
attr = getattr(arg, "time", None)
return isinstance(attr, int)
After a call to has_time(o) succeeds, pyanalyze will know that o.time exists and is of type int.
In practice the main use of this type is to implement the type of hasattr itself. In pure Python hasattr could look like this:
from typing import Any, TypeVar, Annotated
from pyanalyze.extensions import HasAttrGuard
T = TypeVar("T", bound=str)
def hasattr(obj: object, name: T) -> Annotated[bool, HasAttrGuard["obj", T, Any]]:
try:
getattr(obj, name)
return True
except AttributeError:
return False
As currently implemented, HasAttrGuard does not narrow types; instead it preserves the previous type of a variable and adds the additional attribute.
ExternalType¶
ExternalType is a way to refer to a type that is not imported at runtime.
The type must be fully qualified.
from pyanalyze.extensions import ExternalType
def function(arg: ExternalType["other_module.Type"]) -> None:
pass
To resolve the type, pyanalyze will import other_module, but the module
using ExternalType does not have to import other_module.
typing.TYPE_CHECKING can be used in a similar fashion, but ExternalType
can be more convenient when programmatically generating types. Our motivating
use case is our database schema definition file: we would like to map each
column to the enum it corresponds to, but those enums are defined in code
that should not be imported by the schema definition.
CustomCheck¶
CustomCheck is a mechanism that allows users to define additional checks
that are not natively supported by the type system. To use it, create a
new subclass of CustomCheck that overrides the can_assign method. Such
objects can then be placed in Annotated annotations.
For example, the following creates a custom check that allows only literal values:
from pyanalyze.extensions import CustomCheck
from pyanalyze.value import Value, CanAssign, CanAssignContext, CanAssignError, KnownValue, flatten_values
class LiteralOnly(CustomCheck):
def can_assign(self, value: Value, ctx: CanAssignContext) -> CanAssign:
for subval in flatten_values(value):
if not isinstance(subval, KnownValue):
return CanAssignError("Value must be a literal")
return {}
It is used as follows:
def func(arg: Annotated[str, LiteralOnly()]) -> None:
...
func("x") # ok
func(str(some_call())) # error
Custom checks can also be generic. For example, the following custom check implements basic support for integers with a limited range:
from dataclasses import dataclass
from pyanalyze.extensions import CustomCheck
from pyanalyze.value import (
AnyValue,
flatten_values,
CanAssign,
CanAssignError,
CanAssignContext,
KnownValue,
TypeVarMap,
TypeVarValue,
Value,
)
from typing_extensions import Annotated, TypeGuard
from typing import Iterable, TypeVar, Union
# Annotated[] annotations must be hashable
@dataclass(frozen=True)
class GreaterThan(CustomCheck):
# The value can be either an integer or a TypeVar. In the latter case,
# the check hasn't been specified yet, and we let everything through.
value: Union[int, TypeVar]
def can_assign(self, value: Value, ctx: CanAssignContext) -> CanAssign:
if isinstance(self.value, TypeVar):
return {}
# flatten_values() unwraps unions, but we don't want to unwrap
# Annotated, so we can accept other Annotated objects.
for subval in flatten_values(value, unwrap_annotated=False):
if isinstance(subval, AnnotatedValue):
# If the inner value isn't valid, error immediately (for example,
# if it's an int that's too small).
can_assign = self._can_assign_inner(subval.value)
if not isinstance(can_assign, CanAssignError):
return can_assign
gts = list(subval.get_custom_check_of_type(GreaterThan))
if not gts:
# We reject values that are just ints with no GreaterThan
# annotation.
return CanAssignError(f"Size of {value} is not known")
# If a value winds up with multiple GreaterThan annotations,
# we allow it if at least one is bigger than or equal to our value.
if not any(
check.value >= self.value
for check in gts
if isinstance(check.value, int)
):
return CanAssignError(f"{subval} is too small")
else:
can_assign = self._can_assign_inner(subval)
if isinstance(can_assign, CanAssignError):
return can_assign
return {}
def _can_assign_inner(self, value: Value) -> CanAssign:
if isinstance(value, KnownValue):
if not isinstance(value.val, int):
return CanAssignError(f"Value {value.val!r} is not an int")
if value.val <= self.value:
return CanAssignError(
f"Value {value.val!r} is not greater than {self.value}"
)
elif isinstance(value, AnyValue):
# We let Any through.
return {}
else:
# Should be mostly TypedValue.
return CanAssignError(f"Size of {value} is not known")
def walk_values(self) -> Iterable[Value]:
if isinstance(self.value, TypeVar):
yield TypeVarValue(self.value)
def substitute_typevars(self, typevars: TypeVarMap) -> "GreaterThan":
if isinstance(self.value, TypeVar) and self.value in typevars:
value = typevars[self.value]
if isinstance(value, KnownValue) and isinstance(value.val, int):
return GreaterThan(value.val)
return self
def more_than_two(x: Annotated[int, GreaterThan(2)]) -> None:
pass
IntT = TypeVar("IntT", bound=int)
def is_greater_than(
x: int, limit: IntT
) -> TypeGuard[Annotated[int, GreaterThan(IntT)]]:
return x > limit
def caller(x: int) -> None:
more_than_two(x) # E: incompatible_argument
if is_greater_than(x, 2):
more_than_two(x) # ok
more_than_two(3) # ok
more_than_two(2) # E: incompatible_argument
This is not a full, usable implementation of ranged integers; for that we would
also need to add support for this check to operators like int.__add__.
Two custom checks are exposed by pyanalyze.extensions:
pyanalyze.extensions.LiteralOnly, which allows only literal values (as discussed above)pyanalyze.extensions.NoAny, which disallows passing untyped values
Limitations¶
Although pyanalyze aims to support the full Python type system, support for some features is still missing or incomplete, including:
Variance of TypeVars
NewTypeover non-trivial typesParamSpec(PEP 612)TypeVarTuple(PEP 646)
More generally, Python is sufficiently dynamic that almost any check like the ones run by pyanalyze will inevitably have false positives: cases where the script sees an error, but the code in fact runs fine. Attributes may be added at runtime in hard-to-detect ways, variables may be created by direct manipulation of the globals() dictionary, and the unittest.mock module can change anything into anything. Although pyanalyze has a number of configuration mechanisms to deal with these false positives, it is usually better to write code in a way that doesn’t require use of these knobs: code that’s easier for the script to understand is probably also easier for humans to understand.
Just as the tool inevitably has false positives, it equally inevitably cannot find all code that will throw a runtime error. It is generally impossible to statically determine what a program does or whether it runs successfully without actually running the program. Pyanalyze doesn’t check program logic and it cannot always determine exactly what value a variable will have. It is no substitute for unit tests.