[Matplotlib-devel] High priority for 3.2: figure cleanup

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[Matplotlib-devel] High priority for 3.2: figure cleanup

Eric Firing

In today's weekly meeting, with Hannah, Thomas, and myself, we discussed
https://github.com/matplotlib/matplotlib/issues/8519, which points to
the long-standing problem that matplotlib relies on garbage collection
to release memory after a figure is closed.  With some work-flows the
automatic collection doesn't happen often enough and the user must
explicitly trigger it.  It would be better if pyplot cleaned up after
itself, breaking reference cycles when a figure is closed.  I think it
is time to make this change, so I marked the issue "Release-critical".
I am hoping we can get this done as a high-priority task, *early* in the
3.2 cycle.

The central point is that closing a figure should clear it, and clearing
it should recursively walk through the children, breaking connections
and reference cycles all the way down.  One possibility would be to
modify or replace the figure object such that any attempt to operate on
the figure after closing would fail by raising an informative exception.

Question: should closing the gui window also close the figure?  No,
because that would prevent the user from calling savefig on it.  This
would interfere with a reasonable use-case, in which plt.show is used in
blocking mode, the window is closed, and the figure is saved via
savefig.  Problem: if closing the gui window doesn't close the figure,
how can it ever be closed?

This leads to the suggestion that the pyplot state manager keep a list
of "latent figures" which are no longer displayed in a gui window but
have not been closed.  They could then be explicitly closed by
"plt.close('all')".  This also opens the possibility for an API allowing
one to re-display a figure after closing its gui window. For example,
one might do a calculation, show it in a figure in a blocking window
with "plt.show", close the window, do additional calculations, add them
to the same figure, and bring the figure back in a new gui window.  This
might be done in a loop, incrementing the contents of the figure.

See also:
https://github.com/matplotlib/matplotlib/issues/6793
https://github.com/matplotlib/matplotlib/issues/6982

Eric
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
Reply | Threaded
Open this post in threaded view
|

Re: High priority for 3.2: figure cleanup

Jody Klymak
I think if it makes our lives easier the paradigm that a closed window means the figure is closed programmatically is pretty straightforward and standard across GUI interfaces.  If folks need a workflow where they access the figure while it’s open they can turn off blocking.  I vote we keep it simple and predictable rather than tie ourselves in knots with complicated ways of keeping track of what we think the user may want to do.  

Cheers.  Jody

Sent from my iPhone

> On Apr 16, 2019, at 12:19 AM, Eric Firing <[hidden email]> wrote:
>
>
> In today's weekly meeting, with Hannah, Thomas, and myself, we discussed https://github.com/matplotlib/matplotlib/issues/8519, which points to the long-standing problem that matplotlib relies on garbage collection to release memory after a figure is closed.  With some work-flows the automatic collection doesn't happen often enough and the user must explicitly trigger it.  It would be better if pyplot cleaned up after itself, breaking reference cycles when a figure is closed.  I think it is time to make this change, so I marked the issue "Release-critical". I am hoping we can get this done as a high-priority task, *early* in the 3.2 cycle.
>
> The central point is that closing a figure should clear it, and clearing it should recursively walk through the children, breaking connections and reference cycles all the way down.  One possibility would be to modify or replace the figure object such that any attempt to operate on the figure after closing would fail by raising an informative exception.
>
> Question: should closing the gui window also close the figure?  No, because that would prevent the user from calling savefig on it.  This would interfere with a reasonable use-case, in which plt.show is used in blocking mode, the window is closed, and the figure is saved via savefig.  Problem: if closing the gui window doesn't close the figure, how can it ever be closed?
>
> This leads to the suggestion that the pyplot state manager keep a list of "latent figures" which are no longer displayed in a gui window but have not been closed.  They could then be explicitly closed by "plt.close('all')".  This also opens the possibility for an API allowing one to re-display a figure after closing its gui window. For example, one might do a calculation, show it in a figure in a blocking window with "plt.show", close the window, do additional calculations, add them to the same figure, and bring the figure back in a new gui window.  This might be done in a loop, incrementing the contents of the figure.
>
> See also:
> https://github.com/matplotlib/matplotlib/issues/6793
> https://github.com/matplotlib/matplotlib/issues/6982
>
> Eric
> _______________________________________________
> Matplotlib-devel mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/matplotlib-devel

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
Reply | Threaded
Open this post in threaded view
|

Re: High priority for 3.2: figure cleanup

Antony Lee-3
Hi,
It's actually not clear to me how GUI-closing and really closing are related (unless we decided that GUI closing means teardown, but I don't think we *need* that), and what the "latent figures" concept is supposed to achieve.
Doesn't this approximately come down to
1) when calling plt.close(fig), do fig.clf() -> ax.cla()
2) in ax.cla(), before doing ax.lines = [], first do for line in ax.lines: line.remove()? (and similarly for other artists)
Also, note that if one just wants to dump figures to your hard drive in a loop, I would strongly suggest not bothering with pyplot as all and just do "fig = matplotlib.figure.Figure(); ax = fig.subplots(); etc."  The main issue with this approach is that it's a bit awkward to pop up a figure if one decides, after all, that they want to interactively play with it; a possibility would be to allow pyplot to "adopt" figures in a post-hoc manner via plt.show([fig1, fig2, ...]) (https://github.com/matplotlib/matplotlib/pull/13590#issuecomment-469497636).
Antony

On Tue, Apr 16, 2019 at 5:41 PM Jody Klymak <[hidden email]> wrote:
I think if it makes our lives easier the paradigm that a closed window means the figure is closed programmatically is pretty straightforward and standard across GUI interfaces.  If folks need a workflow where they access the figure while it’s open they can turn off blocking.  I vote we keep it simple and predictable rather than tie ourselves in knots with complicated ways of keeping track of what we think the user may want to do. 

Cheers.  Jody

Sent from my iPhone

> On Apr 16, 2019, at 12:19 AM, Eric Firing <[hidden email]> wrote:
>
>
> In today's weekly meeting, with Hannah, Thomas, and myself, we discussed https://github.com/matplotlib/matplotlib/issues/8519, which points to the long-standing problem that matplotlib relies on garbage collection to release memory after a figure is closed.  With some work-flows the automatic collection doesn't happen often enough and the user must explicitly trigger it.  It would be better if pyplot cleaned up after itself, breaking reference cycles when a figure is closed.  I think it is time to make this change, so I marked the issue "Release-critical". I am hoping we can get this done as a high-priority task, *early* in the 3.2 cycle.
>
> The central point is that closing a figure should clear it, and clearing it should recursively walk through the children, breaking connections and reference cycles all the way down.  One possibility would be to modify or replace the figure object such that any attempt to operate on the figure after closing would fail by raising an informative exception.
>
> Question: should closing the gui window also close the figure?  No, because that would prevent the user from calling savefig on it.  This would interfere with a reasonable use-case, in which plt.show is used in blocking mode, the window is closed, and the figure is saved via savefig.  Problem: if closing the gui window doesn't close the figure, how can it ever be closed?
>
> This leads to the suggestion that the pyplot state manager keep a list of "latent figures" which are no longer displayed in a gui window but have not been closed.  They could then be explicitly closed by "plt.close('all')".  This also opens the possibility for an API allowing one to re-display a figure after closing its gui window. For example, one might do a calculation, show it in a figure in a blocking window with "plt.show", close the window, do additional calculations, add them to the same figure, and bring the figure back in a new gui window.  This might be done in a loop, incrementing the contents of the figure.
>
> See also:
> https://github.com/matplotlib/matplotlib/issues/6793
> https://github.com/matplotlib/matplotlib/issues/6982
>
> Eric
> _______________________________________________
> Matplotlib-devel mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/matplotlib-devel

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
Reply | Threaded
Open this post in threaded view
|

Re: High priority for 3.2: figure cleanup

Benjamin Root
I don't think the ability to save after closing the GUI window has always worked. In fact, I have always taught students to perform a savefig prior to doing `plt.show()` if you want to programmatically save a figure and display it. I'd be perfectly fine with killing off that mis-feature. To me, closing a figure window should be equivalent to calling `fig.close()`. If you wanted to save the figure after closing the gui window, you could easily do it from the save file button.


On Tue, Apr 16, 2019 at 12:02 PM Antony Lee <[hidden email]> wrote:
Hi,
It's actually not clear to me how GUI-closing and really closing are related (unless we decided that GUI closing means teardown, but I don't think we *need* that), and what the "latent figures" concept is supposed to achieve.
Doesn't this approximately come down to
1) when calling plt.close(fig), do fig.clf() -> ax.cla()
2) in ax.cla(), before doing ax.lines = [], first do for line in ax.lines: line.remove()? (and similarly for other artists)
Also, note that if one just wants to dump figures to your hard drive in a loop, I would strongly suggest not bothering with pyplot as all and just do "fig = matplotlib.figure.Figure(); ax = fig.subplots(); etc."  The main issue with this approach is that it's a bit awkward to pop up a figure if one decides, after all, that they want to interactively play with it; a possibility would be to allow pyplot to "adopt" figures in a post-hoc manner via plt.show([fig1, fig2, ...]) (https://github.com/matplotlib/matplotlib/pull/13590#issuecomment-469497636).
Antony

On Tue, Apr 16, 2019 at 5:41 PM Jody Klymak <[hidden email]> wrote:
I think if it makes our lives easier the paradigm that a closed window means the figure is closed programmatically is pretty straightforward and standard across GUI interfaces.  If folks need a workflow where they access the figure while it’s open they can turn off blocking.  I vote we keep it simple and predictable rather than tie ourselves in knots with complicated ways of keeping track of what we think the user may want to do. 

Cheers.  Jody

Sent from my iPhone

> On Apr 16, 2019, at 12:19 AM, Eric Firing <[hidden email]> wrote:
>
>
> In today's weekly meeting, with Hannah, Thomas, and myself, we discussed https://github.com/matplotlib/matplotlib/issues/8519, which points to the long-standing problem that matplotlib relies on garbage collection to release memory after a figure is closed.  With some work-flows the automatic collection doesn't happen often enough and the user must explicitly trigger it.  It would be better if pyplot cleaned up after itself, breaking reference cycles when a figure is closed.  I think it is time to make this change, so I marked the issue "Release-critical". I am hoping we can get this done as a high-priority task, *early* in the 3.2 cycle.
>
> The central point is that closing a figure should clear it, and clearing it should recursively walk through the children, breaking connections and reference cycles all the way down.  One possibility would be to modify or replace the figure object such that any attempt to operate on the figure after closing would fail by raising an informative exception.
>
> Question: should closing the gui window also close the figure?  No, because that would prevent the user from calling savefig on it.  This would interfere with a reasonable use-case, in which plt.show is used in blocking mode, the window is closed, and the figure is saved via savefig.  Problem: if closing the gui window doesn't close the figure, how can it ever be closed?
>
> This leads to the suggestion that the pyplot state manager keep a list of "latent figures" which are no longer displayed in a gui window but have not been closed.  They could then be explicitly closed by "plt.close('all')".  This also opens the possibility for an API allowing one to re-display a figure after closing its gui window. For example, one might do a calculation, show it in a figure in a blocking window with "plt.show", close the window, do additional calculations, add them to the same figure, and bring the figure back in a new gui window.  This might be done in a loop, incrementing the contents of the figure.
>
> See also:
> https://github.com/matplotlib/matplotlib/issues/6793
> https://github.com/matplotlib/matplotlib/issues/6982
>
> Eric
> _______________________________________________
> Matplotlib-devel mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/matplotlib-devel

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
Reply | Threaded
Open this post in threaded view
|

Re: High priority for 3.2: figure cleanup

tcaswell

On Tue, Apr 16, 2019, 20:23 Benjamin Root <[hidden email]> wrote:
I don't think the ability to save after closing the GUI window has always worked. In fact, I have always taught students to perform a savefig prior to doing `plt.show()` if you want to programmatically save a figure and display it. I'd be perfectly fine with killing off that mis-feature. To me, closing a figure window should be equivalent to calling `fig.close()`. If you wanted to save the figure after closing the gui window, you could easily do it from the save file button.


On Tue, Apr 16, 2019 at 12:02 PM Antony Lee <[hidden email]> wrote:
Hi,
It's actually not clear to me how GUI-closing and really closing are related (unless we decided that GUI closing means teardown, but I don't think we *need* that), and what the "latent figures" concept is supposed to achieve.
Doesn't this approximately come down to
1) when calling plt.close(fig), do fig.clf() -> ax.cla()
2) in ax.cla(), before doing ax.lines = [], first do for line in ax.lines: line.remove()? (and similarly for other artists)
Also, note that if one just wants to dump figures to your hard drive in a loop, I would strongly suggest not bothering with pyplot as all and just do "fig = matplotlib.figure.Figure(); ax = fig.subplots(); etc."  The main issue with this approach is that it's a bit awkward to pop up a figure if one decides, after all, that they want to interactively play with it; a possibility would be to allow pyplot to "adopt" figures in a post-hoc manner via plt.show([fig1, fig2, ...]) (https://github.com/matplotlib/matplotlib/pull/13590#issuecomment-469497636).
Antony

On Tue, Apr 16, 2019 at 5:41 PM Jody Klymak <[hidden email]> wrote:
I think if it makes our lives easier the paradigm that a closed window means the figure is closed programmatically is pretty straightforward and standard across GUI interfaces.  If folks need a workflow where they access the figure while it’s open they can turn off blocking.  I vote we keep it simple and predictable rather than tie ourselves in knots with complicated ways of keeping track of what we think the user may want to do. 

Cheers.  Jody

Sent from my iPhone

> On Apr 16, 2019, at 12:19 AM, Eric Firing <[hidden email]> wrote:
>
>
> In today's weekly meeting, with Hannah, Thomas, and myself, we discussed https://github.com/matplotlib/matplotlib/issues/8519, which points to the long-standing problem that matplotlib relies on garbage collection to release memory after a figure is closed.  With some work-flows the automatic collection doesn't happen often enough and the user must explicitly trigger it.  It would be better if pyplot cleaned up after itself, breaking reference cycles when a figure is closed.  I think it is time to make this change, so I marked the issue "Release-critical". I am hoping we can get this done as a high-priority task, *early* in the 3.2 cycle.
>
> The central point is that closing a figure should clear it, and clearing it should recursively walk through the children, breaking connections and reference cycles all the way down.  One possibility would be to modify or replace the figure object such that any attempt to operate on the figure after closing would fail by raising an informative exception.
>
> Question: should closing the gui window also close the figure?  No, because that would prevent the user from calling savefig on it.  This would interfere with a reasonable use-case, in which plt.show is used in blocking mode, the window is closed, and the figure is saved via savefig.  Problem: if closing the gui window doesn't close the figure, how can it ever be closed?
>
> This leads to the suggestion that the pyplot state manager keep a list of "latent figures" which are no longer displayed in a gui window but have not been closed.  They could then be explicitly closed by "plt.close('all')".  This also opens the possibility for an API allowing one to re-display a figure after closing its gui window. For example, one might do a calculation, show it in a figure in a blocking window with "plt.show", close the window, do additional calculations, add them to the same figure, and bring the figure back in a new gui window.  This might be done in a loop, incrementing the contents of the figure.
>
> See also:
> https://github.com/matplotlib/matplotlib/issues/6793
> https://github.com/matplotlib/matplotlib/issues/6982
>
> Eric
> _______________________________________________
> Matplotlib-devel mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/matplotlib-devel

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
Reply | Threaded
Open this post in threaded view
|

Re: High priority for 3.2: figure cleanup

Eric Firing
I think that this cleanup can be approached in parts, some of which
involve refactoring so as to reduce the number of circular references.
Looking at ticker.py, for example, I suspect that a cleaner
implementation would eliminate the need for a Formatter or Locator to
have an actual or dummy Axis as an attribute.  Instead, the Axis
reference, or, ideally, the information actually needed from an Axis,
could be passed as kwargs to the methods that require that information.

Can we get there from here, or would the
deprecation/backwards-compatibility dance be too exhausting?

Eric

On 2019/04/20 12:44 PM, Thomas Caswell wrote:

> Some of the tools in
>
> https://medium.com/zendesk-engineering/hunting-for-memory-leaks-in-python-applications-6824d0518774
>
> May be useful for working on this.
>
> Tom
>
> On Tue, Apr 16, 2019, 20:23 Benjamin Root <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     I don't think the ability to save after closing the GUI window has
>     always worked. In fact, I have always taught students to perform a
>     savefig prior to doing `plt.show()` if you want to programmatically
>     save a figure and display it. I'd be perfectly fine with killing off
>     that mis-feature. To me, closing a figure window should be
>     equivalent to calling `fig.close()`. If you wanted to save the
>     figure after closing the gui window, you could easily do it from the
>     save file button.
>
>
>     On Tue, Apr 16, 2019 at 12:02 PM Antony Lee
>     <[hidden email]
>     <mailto:[hidden email]>> wrote:
>
>         Hi,
>         It's actually not clear to me how GUI-closing and really closing
>         are related (unless we decided that GUI closing means teardown,
>         but I don't think we *need* that), and what the "latent figures"
>         concept is supposed to achieve.
>         Doesn't this approximately come down to
>         1) when calling plt.close(fig), do fig.clf() -> ax.cla()
>         2) in ax.cla(), before doing ax.lines = [], first do for line in
>         ax.lines: line.remove()? (and similarly for other artists)
>         Also, note that if one just wants to dump figures to your hard
>         drive in a loop, I would strongly suggest not bothering with
>         pyplot as all and just do "fig = matplotlib.figure.Figure(); ax
>         = fig.subplots(); etc."  The main issue with this approach is
>         that it's a bit awkward to pop up a figure if one decides, after
>         all, that they want to interactively play with it; a possibility
>         would be to allow pyplot to "adopt" figures in a post-hoc manner
>         via plt.show([fig1, fig2, ...])
>         (https://github.com/matplotlib/matplotlib/pull/13590#issuecomment-469497636).
>         Antony
>
>         On Tue, Apr 16, 2019 at 5:41 PM Jody Klymak <[hidden email]
>         <mailto:[hidden email]>> wrote:
>
>             I think if it makes our lives easier the paradigm that a
>             closed window means the figure is closed programmatically is
>             pretty straightforward and standard across GUI interfaces.
>             If folks need a workflow where they access the figure while
>             it’s open they can turn off blocking.  I vote we keep it
>             simple and predictable rather than tie ourselves in knots
>             with complicated ways of keeping track of what we think the
>             user may want to do.
>
>             Cheers.  Jody
>
>             Sent from my iPhone
>
>              > On Apr 16, 2019, at 12:19 AM, Eric Firing
>             <[hidden email] <mailto:[hidden email]>> wrote:
>              >
>              >
>              > In today's weekly meeting, with Hannah, Thomas, and
>             myself, we discussed
>             https://github.com/matplotlib/matplotlib/issues/8519, which
>             points to the long-standing problem that matplotlib relies
>             on garbage collection to release memory after a figure is
>             closed.  With some work-flows the automatic collection
>             doesn't happen often enough and the user must explicitly
>             trigger it.  It would be better if pyplot cleaned up after
>             itself, breaking reference cycles when a figure is closed.
>             I think it is time to make this change, so I marked the
>             issue "Release-critical". I am hoping we can get this done
>             as a high-priority task, *early* in the 3.2 cycle.
>              >
>              > The central point is that closing a figure should clear
>             it, and clearing it should recursively walk through the
>             children, breaking connections and reference cycles all the
>             way down.  One possibility would be to modify or replace the
>             figure object such that any attempt to operate on the figure
>             after closing would fail by raising an informative exception.
>              >
>              > Question: should closing the gui window also close the
>             figure?  No, because that would prevent the user from
>             calling savefig on it.  This would interfere with a
>             reasonable use-case, in which plt.show is used in blocking
>             mode, the window is closed, and the figure is saved via
>             savefig.  Problem: if closing the gui window doesn't close
>             the figure, how can it ever be closed?
>              >
>              > This leads to the suggestion that the pyplot state
>             manager keep a list of "latent figures" which are no longer
>             displayed in a gui window but have not been closed.  They
>             could then be explicitly closed by "plt.close('all')".  This
>             also opens the possibility for an API allowing one to
>             re-display a figure after closing its gui window. For
>             example, one might do a calculation, show it in a figure in
>             a blocking window with "plt.show", close the window, do
>             additional calculations, add them to the same figure, and
>             bring the figure back in a new gui window.  This might be
>             done in a loop, incrementing the contents of the figure.
>              >
>              > See also:
>              > https://github.com/matplotlib/matplotlib/issues/6793
>              > https://github.com/matplotlib/matplotlib/issues/6982
>              >
>              > Eric
>              > _______________________________________________
>              > Matplotlib-devel mailing list
>              > [hidden email]
>             <mailto:[hidden email]>
>              > https://mail.python.org/mailman/listinfo/matplotlib-devel
>
>             _______________________________________________
>             Matplotlib-devel mailing list
>             [hidden email] <mailto:[hidden email]>
>             https://mail.python.org/mailman/listinfo/matplotlib-devel
>
>         _______________________________________________
>         Matplotlib-devel mailing list
>         [hidden email] <mailto:[hidden email]>
>         https://mail.python.org/mailman/listinfo/matplotlib-devel
>
>     _______________________________________________
>     Matplotlib-devel mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://mail.python.org/mailman/listinfo/matplotlib-devel
>
>
> _______________________________________________
> Matplotlib-devel mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/matplotlib-devel
>

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel