PyCQA/pylint

Pylint crashing: AstroidError -- RecursionError in node_classes because of scipy #7543

hongha912 posted onGitHub

Bug description

pylint crashes while running on a file with a recursion error thrown by astroid in node_classes.py.

Configuration

[MASTER]

load-plugins= pylint_quotes

string-quote=single-avoid-escape
triple-quote=double
docstring-quote=double

[REPORTS]

reports = no

[MESSAGES CONTROL]

disable=
    C,  # convention
    R,  # refactor
    W,  # warning
    no-member,
    invalid-unary-operand-type,
    unpacking-non-sequence 

enable=
    anomalous-backslash-in-string,
    bad-format-string,
    bad-open-mode,
    binary-op-exception,
    consider-using-dict-comprehension,
    consider-using-in,
    consider-using-set-comprehension,
    duplicate-key,
    global-variable-not-assigned,
    inconsistent-return-statements,
    line-too-long,
    no-default-true,
    string_quotes,
    too-many-arguments,
    undefined-loop-variable,
    ungrouped-imports,
    unnecessary-semicolon,
    unreachable,
    unused-import,
    unused-variable,
    useless-return,
    wildcard-import,
    wrong-import-order

extension-pkg-whitelist=
    numpy,
    scipy.spatial,

Command used

pylint <file_path.py>

Content

from scipy import spatial

Pylint output

pylint crashed with a ``AstroidError`` and with the following stacktrace:

Traceback (most recent call last):
  File "<venv folder>/lib/python3.8/site-packages/pylint/checkers/imports.py", line 798, in _get_imported_module
    return importnode.do_import_module(modname)
  File "<venv folder>/lib/python3.8/site-packages/astroid/nodes/_base_nodes.py", line 148, in do_import_module
    return mymodule.import_module(
  File "<venv folder>/lib/python3.8/site-packages/astroid/nodes/scoped_nodes/scoped_nodes.py", line 521, in import_module
    return AstroidManager().ast_from_module_name(absmodname)
  File "<venv folder>/lib/python3.8/site-packages/astroid/manager.py", line 209, in ast_from_module_name
    return self.ast_from_file(found_spec.location, modname, fallback=False)
  File "<venv folder>/lib/python3.8/site-packages/astroid/manager.py", line 118, in ast_from_file
    return AstroidBuilder(self).file_build(filepath, modname)
  File "<venv folder>/lib/python3.8/site-packages/astroid/builder.py", line 135, in file_build
    return self._post_build(module, builder, encoding)
  File "<venv folder>/lib/python3.8/site-packages/astroid/builder.py", line 154, in _post_build
    self.add_from_names_to_locals(from_node)
  File "<venv folder>/lib/python3.8/site-packages/astroid/builder.py", line 210, in add_from_names_to_locals
    imported = node.do_import_module()
  File "<venv folder>/lib/python3.8/site-packages/astroid/nodes/_base_nodes.py", line 148, in do_import_module
    return mymodule.import_module(
  File "<venv folder>/lib/python3.8/site-packages/astroid/nodes/scoped_nodes/scoped_nodes.py", line 521, in import_module
    return AstroidManager().ast_from_module_name(absmodname)
  File "<venv folder>/lib/python3.8/site-packages/astroid/manager.py", line 184, in ast_from_module_name
    return self.ast_from_module(module, modname)
  File "<venv folder>/lib/python3.8/site-packages/astroid/manager.py", line 286, in ast_from_module
    return AstroidBuilder(self).module_build(module, modname)
  File "<venv folder>/lib/python3.8/site-packages/astroid/builder.py", line 92, in module_build
    node = self.inspect_build(module, modname=modname, path=path)
  File "<venv folder>/lib/python3.8/site-packages/astroid/raw_building.py", line 373, in inspect_build
    self.object_build(node, module)
  File "<venv folder>/lib/python3.8/site-packages/astroid/raw_building.py", line 433, in object_build
    self.object_build(module, member)
  File "<venv folder>/lib/python3.8/site-packages/astroid/raw_building.py", line 433, in object_build
    self.object_build(module, member)
  File "<venv folder>/lib/python3.8/site-packages/astroid/raw_building.py", line 433, in object_build
    self.object_build(module, member)
  [Previous line repeated 1 more time]
  File "<venv folder>/lib/python3.8/site-packages/astroid/raw_building.py", line 424, in object_build
    attach_const_node(node, name, member)
  File "<venv folder>/lib/python3.8/site-packages/astroid/raw_building.py", line 82, in attach_const_node
    _attach_local_node(node, nodes.const_factory(value), name)
  File "<venv folder>/lib/python3.8/site-packages/astroid/nodes/node_classes.py", line 5413, in const_factory
    instance.postinit(_create_dict_items(value, instance))
  File "<venv folder>/lib/python3.8/site-packages/astroid/nodes/node_classes.py", line 5387, in _create_dict_items
    value_node = const_factory(value)
  File "<venv folder>/lib/python3.8/site-packages/astroid/nodes/node_classes.py", line 5413, in const_factory
    instance.postinit(_create_dict_items(value, instance))
  File "<venv folder>/lib/python3.8/site-packages/astroid/nodes/node_classes.py", line 5387, in _create_dict_items
    value_node = const_factory(value)
  File "<venv folder>/lib/python3.8/site-packages/astroid/nodes/node_classes.py", line 5413, in const_factory
    instance.postinit(_create_dict_items(value, instance))
  File "<venv folder>/lib/python3.8/site-packages/astroid/nodes/node_classes.py", line 5387, in _create_dict_items
    value_node = const_factory(value)
  File "<venv folder>/lib/python3.8/site-packages/astroid/nodes/node_classes.py", line 5413, in const_factory
    instance.postinit(_create_dict_items(value, instance))
  File "<venv folder>/lib/python3.8/site-packages/astroid/nodes/node_classes.py", line 5387, in _create_dict_items
    value_node = const_factory(value)
  File "<venv folder>/lib/python3.8/site-packages/astroid/nodes/node_classes.py", line 5413, in const_factory
    instance.postinit(_create_dict_items(value, instance))
  File "<venv folder>/lib/python3.8/site-packages/astroid/nodes/node_classes.py", line 5387, in _create_dict_items
    value_node = const_factory(value)
  File "<venv folder>/lib/python3.8/site-packages/astroid/nodes/node_classes.py", line 5413, in const_factory
    instance.postinit(_create_dict_items(value, instance))
  File "<venv folder>/lib/python3.8/site-packages/astroid/nodes/node_classes.py", line 5387, in _create_dict_items
    value_node = const_factory(value)
  File "<venv folder>/lib/python3.8/site-packages/astroid/nodes/node_classes.py", line 5413, in const_factory
    instance.postinit(_create_dict_items(value, instance))
(continue like this)

This continues -- tail end of the stack trace:

  File "<venv folder>/lib/python3.8/site-packages/astroid/nodes/node_classes.py", line 5413, in const_factory
    instance.postinit(_create_dict_items(value, instance))
  File "<venv folder>/lib/python3.8/site-packages/astroid/nodes/node_classes.py", line 5385, in _create_dict_items
    key_node = const_factory(key)
  File "<venv folder>/lib/python3.8/site-packages/astroid/nodes/node_classes.py", line 5407, in const_factory
    if issubclass(initializer_cls, (List, Set, Tuple)):
  File "/usr/local/google/home/hongvu/.pyenv/versions/3.8.12/lib/python3.8/abc.py", line 102, in __subclasscheck__
    return _abc_subclasscheck(cls, subclass)
RecursionError: maximum recursion depth exceeded in comparison

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<venv folder>/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 782, in _lint_file
    check_astroid_module(module)
  File "<venv folder>/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 1049, in check_astroid_module
    retval = self._check_astroid_module(
  File "<venv folder>/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 1099, in _check_astroid_module
    walker.walk(node)
  File "<venv folder>/lib/python3.8/site-packages/pylint/utils/ast_walker.py", line 93, in walk
    self.walk(child)
  File "<venv folder>/lib/python3.8/site-packages/pylint/utils/ast_walker.py", line 90, in walk
    callback(astroid)
  File "<venv folder>/lib/python3.8/site-packages/pylint/checkers/imports.py", line 502, in visit_importfrom
    imported_module = self._get_imported_module(node, basename)
  File "<venv folder>/lib/python3.8/site-packages/pylint/checkers/imports.py", line 823, in _get_imported_module
    raise astroid.AstroidError from e
astroid.exceptions.AstroidError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<venv folder>/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 747, in _lint_files
    self._lint_file(fileitem, module, check_astroid_module)
  File "<venv folder>/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 784, in _lint_file
    raise astroid.AstroidError from e
astroid.exceptions.AstroidError

Expected behavior

Pylint should not crash.

Pylint version

pylint 2.15.3
astroid 2.12.10
Python 3.8.12 (default, Mar 14 2022, 20:18:12) 
[GCC 11.2.0]

OS / Environment

Debian 5

Additional dependencies

scipy==1.4.1

Without any code example it will be very hard to reproduce and fix this. Are you able to share the code that reproduces this?

posted by DanielNoord over 2 years ago

@DanielNoord I was able to isolate the problem down actually -- seems like the issue is with scipy 1.4.1! Just a one-liner file from scipy import spatial can cause this crash.

Hope this helps.

posted by hongha912 over 2 years ago

Also updated the ticket with a sanitized version of our configuration -- was able to repro with the given one-liner file and the configuration.

posted by hongha912 over 2 years ago

I did a little investigation, but I couldn't find a quick fix. Sorry!

There does seem to be some actual recursion going on: a print on the values in _create_dict_items seems to print similar values after some time.

posted by DanielNoord over 2 years ago

@DanielNoord is there a way for me to work around this? Can I ignore scipy somehow? We only care about linting our own code, not external libraries' code.

posted by hongha912 about 2 years ago

Does the crash still happen if you remove scipy from the extension pkg allowlist option in the configuration ?

posted by Pierre-Sassoulas about 2 years ago

I ran into a similar issue today after a version upgrade of xarray from 2022.9.0 to 2022.10.0. With the old xarray version installed pylint ran through fine as expected, but then with the only change of installing the new xarray version in the environment suddenly pylint crashes:

<details> Stacktrace:

Fatal Python error: Cannot recover from stack overflow.
Python runtime state: initialized

Current thread 0x00007f53e18b9740 (most recent call first):
  File "/home/lukas/.local/share/virtualenvs/project-oePfdNug/lib/python3.8/site-packages/astroid/nodes/node_ng.py", line 350 in frame
  File "/home/lukas/.local/share/virtualenvs/project-oePfdNug/lib/python3.8/site-packages/astroid/nodes/node_ng.py", line 350 in frame
  File "/home/lukas/.local/share/virtualenvs/project-oePfdNug/lib/python3.8/site-packages/astroid/filter_statements.py", line 69 in _filter_stmts
  File "/home/lukas/.local/share/virtualenvs/project-oePfdNug/lib/python3.8/site-packages/astroid/nodes/scoped_nodes/mixin.py", line 74 in _scope_lookup
  File "/home/lukas/.local/share/virtualenvs/project-oePfdNug/lib/python3.8/site-packages/astroid/nodes/scoped_nodes/scoped_nodes.py", line 378 in scope_lookup
  File "/home/lukas/.local/share/virtualenvs/project-oePfdNug/lib/python3.8/site-packages/astroid/nodes/scoped_nodes/mixin.py", line 85 in _scope_lookup
  File "/home/lukas/.local/share/virtualenvs/project-oePfdNug/lib/python3.8/site-packages/astroid/nodes/scoped_nodes/scoped_nodes.py", line 1222 in scope_lookup
  File "/home/lukas/.local/share/virtualenvs/project-oePfdNug/lib/python3.8/site-packages/astroid/nodes/scoped_nodes/scoped_nodes.py", line 1782 in scope_lookup
  File "/home/lukas/.local/share/virtualenvs/project-oePfdNug/lib/python3.8/site-packages/astroid/nodes/node_classes.py", line 359 in lookup
  File "/home/lukas/.local/share/virtualenvs/project-oePfdNug/lib/python3.8/site-packages/astroid/inference.py", line 218 in infer_name
  File "/home/lukas/.local/share/virtualenvs/project-oePfdNug/lib/python3.8/site-packages/astroid/decorators.py", line 109 in wrapped
  File "/home/lukas/.local/share/virtualenvs/project-oePfdNug/lib/python3.8/site-packages/astroid/decorators.py", line 140 in raise_if_nothing_inferred
  File "/home/lukas/.local/share/virtualenvs/project-oePfdNug/lib/python3.8/site-packages/astroid/nodes/node_ng.py", line 182 in infer
  File "/home/lukas/.local/share/virtualenvs/project-oePfdNug/lib/python3.8/site-packages/astroid/inference.py", line 343 in infer_attribute
  File "/home/lukas/.local/share/virtualenvs/project-oePfdNug/lib/python3.8/site-packages/astroid/decorators.py", line 109 in wrapped
  File "/home/lukas/.local/share/virtualenvs/project-oePfdNug/lib/python3.8/site-packages/astroid/decorators.py", line 140 in raise_if_nothing_inferred
  File "/home/lukas/.local/share/virtualenvs/project-oePfdNug/lib/python3.8/site-packages/astroid/nodes/node_ng.py", line 182 in infer
  File "/home/lukas/.local/share/virtualenvs/project-oePfdNug/lib/python3.8/site-packages/astroid/inference.py", line 255 in infer_call
  File "/home/lukas/.local/share/virtualenvs/project-oePfdNug/lib/python3.8/site-packages/astroid/decorators.py", line 109 in wrapped
  File "/home/lukas/.local/share/virtualenvs/project-oePfdNug/lib/python3.8/site-packages/astroid/decorators.py", line 140 in raise_if_nothing_inferred
  File "/home/lukas/.local/share/virtualenvs/project-oePfdNug/lib/python3.8/site-packages/astroid/nodes/node_ng.py", line 182 in infer
  File "/home/lukas/.local/share/virtualenvs/project-oePfdNug/lib/python3.8/site-packages/astroid/bases.py", line 165 in _infer_stmts
  File "/home/lukas/.local/share/virtualenvs/project-oePfdNug/lib/python3.8/site-packages/astroid/decorators.py", line 109 in wrapped
  File "/home/lukas/.local/share/virtualenvs/project-oePfdNug/lib/python3.8/site-packages/astroid/decorators.py", line 140 in raise_if_nothing_inferred
... 

fish: 'pylint tests/test_data.py' terminated by signal SIGABRT (Abort)

</details>

Unfortunately it was not easily possible to isolate the test enough to easily share it here.

But I was wondering if it is possible to tell pylint to just not lint the source code of xarray (or any third party packages really), which it is apparently doing since only a version upgrade suddenly results in failure?

Environment

pylint 2.15.4
astroid 2.12.11
Python 3.8.10 (default, Jan 28 2022, 09:41:12)
[GCC 9.3.0]
posted by lukasbindreiter about 2 years ago

But I was wondering if it is possible to tell pylint to just not lint the source code of xarray (or any third party packages really), which it is apparently doing since only a version upgrade suddenly results in failure?

If it's a c-binding causing the problem it's possible to remove a lib from the "extension pkg allowlist", otherwise no afaik. (We're not supposed to crash, but adding this option could also be useful for code that are slow to parse.

posted by Pierre-Sassoulas about 2 years ago

If it's a c-binding causing the problem it's possible to remove a lib from the "extension pkg allowlist", otherwise no afaik. (We're not supposed to crash, but adding this option could also be useful for code that are slow to parse.

Ah sorry I forgot to mention, this is my .pylintrc:

[MASTER]


[MESSAGES CONTROL]

disable=missing-docstring,
        too-many-arguments,
        too-many-function-args,
        too-few-public-methods,
        logging-format-interpolation

So I explicitly don't have an extension-pkg-whitelist=xarray in there. Do you know why pylint still tries to look at xarray source code? Is there some default extension-pkg-whitelist if none is specified?

posted by lukasbindreiter about 2 years ago

Here's some documentation on extension-pkg-allowlist but it's only used for C extension modules for security reasons, there's no option to prevent parsing python module.

posted by Pierre-Sassoulas about 2 years ago

Do you know why pylint still tries to look at xarray source code?

Yes, it's a fundamental design aspect of pylint/astroid that it inspects your imports to find out what types variables take.

black_box = FromSomeLibrary()
for x in black_box:  # pylint complains "not-an-iterable" because it inspected FromSomeLibrary
   ...
posted by jacobtylerwalls about 2 years ago

Just so I can remove the needs reproduce label, FYI this can be reproduced as stated originally in scipy 1.4.1 but surprisingly cannot be reproduced with pip install scipy==1.9.*... Not sure what that means maintainers?

posted by clavedeluna about 2 years ago

Means that scipy has become easier for us to infer and we no longer recurse to deeply 🎉

Closing!

posted by DanielNoord about 2 years ago

Fund this Issue

$0.00
Funded

Pull requests