[Matplotlib-devel] Proposal to amend the PR merge rules

classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[Matplotlib-devel] Proposal to amend the PR merge rules

Antony Lee-4
Hi all,

During the weekly dev call, I proposed to amend the PR merge rules, following an initial comment on Github [https://github.com/matplotlib/matplotlib/pull/13173#issuecomment-453921220].  There was general agreement among the devs present (Tom, Hannah, Jody, and myself), so I'm putting it here for discussion.  The objective of this change is to prevent "uncontroversial but uninteresting" PRs from falling into oblivion, and try to decrease the size of the open PR stack.

The current rule is that a PR needs positive reviews (from two committers, and excluding the author if a committer) to be merged, except that doc-only PRs (docstrings, rst) only needs a single positive review.

I am proposing that, if a (non-doc) PR already has one positive review, but no activity on that PR has occurred for two weeks (exact time interval up to bikeshedding) [Jody suggests: and the PR has 100% code coverage], then a committer (either the first reviewer, or the author if a committer) can suggest that it be merged on the basis of that single review.  To do so, the "sponsor" should ping all developers (@matplotlib/developers) on that issue indicating that intent, and add a "single-review-merge" label on the PR (so that these PRs can easily be found).  Committers are encouraged to review the PR to accept and merge it or reject it or request changes on it; but they can also just indicate that they consider the PR sufficiently complex that a proper second review is needed before merging, or request an extension, etc.  To do so they should still leave a "reject" review, even if just saying "objecting to single-review merged; anyone can dismiss after a second review".  However, if within another two weeks, no committer voiced any objection (explicitly, i.e. by rejecting), then the PR can indeed be merged on the basis of that single review.

To avoid overwhelming the system, any committer can only "sponsor" a single PR at a time.

Thoughts?

Antony

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to amend the PR merge rules

Nelle Varoquaux
Hello,

Overall, I think this is a good idea, but I would like specifics on what "uncontroversial but uninteresting" PR mean. IMO, that should exclude any PR with API changes.

Cheers,
N

On Tue, 15 Jan 2019 at 06:00, Antony Lee <[hidden email]> wrote:
Hi all,

During the weekly dev call, I proposed to amend the PR merge rules, following an initial comment on Github [https://github.com/matplotlib/matplotlib/pull/13173#issuecomment-453921220].  There was general agreement among the devs present (Tom, Hannah, Jody, and myself), so I'm putting it here for discussion.  The objective of this change is to prevent "uncontroversial but uninteresting" PRs from falling into oblivion, and try to decrease the size of the open PR stack.

The current rule is that a PR needs positive reviews (from two committers, and excluding the author if a committer) to be merged, except that doc-only PRs (docstrings, rst) only needs a single positive review.

I am proposing that, if a (non-doc) PR already has one positive review, but no activity on that PR has occurred for two weeks (exact time interval up to bikeshedding) [Jody suggests: and the PR has 100% code coverage], then a committer (either the first reviewer, or the author if a committer) can suggest that it be merged on the basis of that single review.  To do so, the "sponsor" should ping all developers (@matplotlib/developers) on that issue indicating that intent, and add a "single-review-merge" label on the PR (so that these PRs can easily be found).  Committers are encouraged to review the PR to accept and merge it or reject it or request changes on it; but they can also just indicate that they consider the PR sufficiently complex that a proper second review is needed before merging, or request an extension, etc.  To do so they should still leave a "reject" review, even if just saying "objecting to single-review merged; anyone can dismiss after a second review".  However, if within another two weeks, no committer voiced any objection (explicitly, i.e. by rejecting), then the PR can indeed be merged on the basis of that single review.

To avoid overwhelming the system, any committer can only "sponsor" a single PR at a time.

Thoughts?

Antony
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to amend the PR merge rules

Paul Hobson-2
I think this would be a good policy as well. Can I get some clarification on the flow? The way I understand it:

1) Someone (committer, contributor, or first-timer) opens a simple PR
2) Committer A reviews it, adds a "single-review-merge" label, then digitally walks away
3) No less than two weeks later, Committer B sees the PR, notices the label, opens it up, and merges without reviewing.

Does that capture a successful "single-review-merge" lifecycle?
-paul

On Tue, Jan 15, 2019 at 9:57 AM Nelle Varoquaux <[hidden email]> wrote:
Hello,

Overall, I think this is a good idea, but I would like specifics on what "uncontroversial but uninteresting" PR mean. IMO, that should exclude any PR with API changes.

Cheers,
N

On Tue, 15 Jan 2019 at 06:00, Antony Lee <[hidden email]> wrote:
Hi all,

During the weekly dev call, I proposed to amend the PR merge rules, following an initial comment on Github [https://github.com/matplotlib/matplotlib/pull/13173#issuecomment-453921220].  There was general agreement among the devs present (Tom, Hannah, Jody, and myself), so I'm putting it here for discussion.  The objective of this change is to prevent "uncontroversial but uninteresting" PRs from falling into oblivion, and try to decrease the size of the open PR stack.

The current rule is that a PR needs positive reviews (from two committers, and excluding the author if a committer) to be merged, except that doc-only PRs (docstrings, rst) only needs a single positive review.

I am proposing that, if a (non-doc) PR already has one positive review, but no activity on that PR has occurred for two weeks (exact time interval up to bikeshedding) [Jody suggests: and the PR has 100% code coverage], then a committer (either the first reviewer, or the author if a committer) can suggest that it be merged on the basis of that single review.  To do so, the "sponsor" should ping all developers (@matplotlib/developers) on that issue indicating that intent, and add a "single-review-merge" label on the PR (so that these PRs can easily be found).  Committers are encouraged to review the PR to accept and merge it or reject it or request changes on it; but they can also just indicate that they consider the PR sufficiently complex that a proper second review is needed before merging, or request an extension, etc.  To do so they should still leave a "reject" review, even if just saying "objecting to single-review merged; anyone can dismiss after a second review".  However, if within another two weeks, no committer voiced any objection (explicitly, i.e. by rejecting), then the PR can indeed be merged on the basis of that single review.

To avoid overwhelming the system, any committer can only "sponsor" a single PR at a time.

Thoughts?

Antony
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to amend the PR merge rules

Jody Klymak


On 15 Jan 2019, at 10:20, Paul Hobson <[hidden email]> wrote:

I think this would be a good policy as well. Can I get some clarification on the flow? The way I understand it:

1) Someone (committer, contributor, or first-timer) opens a simple PR
2) Committer A reviews it, adds a "single-review-merge" label, then digitally walks away
3) No less than two weeks later, Committer B sees the PR, notices the label, opens it up, and merges without reviewing.

If there is a Committer B, I think they should at least look at the PR to make sure it doesn’t have any hidden API changes, or introduce anything hostile.  

The point here is that getting the third person to look at a PR is proving quite difficult, and if the rest of the folks with the commit bit haven’t looked at a PR for a month, even after being warned, then silence indicates consent.  

Note that only folks with the commit bit can label PRs, so anyone flagging a PR like this is implicitly trusted to not be wreaking havoc, and not doing this for major changes.  Since we all curate our own PRs, I rather expect that the PR contributor will often be the person who adds the flag.  Whether we want to allow a self-merge at that point is up for debate, but if not, then folks need to get in the habit of looking at old flagged PRs and merging them.  

I think a huge problem we have is the LIFO default sort of the github PR queue, and that any PR not on the first page might as well be closed for all the attention it will get without constant nagging.  

Cheers,   Jody

Does that capture a successful "single-review-merge" lifecycle?
-paul

On Tue, Jan 15, 2019 at 9:57 AM Nelle Varoquaux <[hidden email]> wrote:
Hello,

Overall, I think this is a good idea, but I would like specifics on what "uncontroversial but uninteresting" PR mean. IMO, that should exclude any PR with API changes.

Cheers,
N

On Tue, 15 Jan 2019 at 06:00, Antony Lee <[hidden email]> wrote:
Hi all,

During the weekly dev call, I proposed to amend the PR merge rules, following an initial comment on Github [https://github.com/matplotlib/matplotlib/pull/13173#issuecomment-453921220].  There was general agreement among the devs present (Tom, Hannah, Jody, and myself), so I'm putting it here for discussion.  The objective of this change is to prevent "uncontroversial but uninteresting" PRs from falling into oblivion, and try to decrease the size of the open PR stack.

The current rule is that a PR needs positive reviews (from two committers, and excluding the author if a committer) to be merged, except that doc-only PRs (docstrings, rst) only needs a single positive review.

I am proposing that, if a (non-doc) PR already has one positive review, but no activity on that PR has occurred for two weeks (exact time interval up to bikeshedding) [Jody suggests: and the PR has 100% code coverage], then a committer (either the first reviewer, or the author if a committer) can suggest that it be merged on the basis of that single review.  To do so, the "sponsor" should ping all developers (@matplotlib/developers) on that issue indicating that intent, and add a "single-review-merge" label on the PR (so that these PRs can easily be found).  Committers are encouraged to review the PR to accept and merge it or reject it or request changes on it; but they can also just indicate that they consider the PR sufficiently complex that a proper second review is needed before merging, or request an extension, etc.  To do so they should still leave a "reject" review, even if just saying "objecting to single-review merged; anyone can dismiss after a second review".  However, if within another two weeks, no committer voiced any objection (explicitly, i.e. by rejecting), then the PR can indeed be merged on the basis of that single review.

To avoid overwhelming the system, any committer can only "sponsor" a single PR at a time.

Thoughts?

Antony
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel

--
Jody Klymak    






_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to amend the PR merge rules

Nelle Varoquaux
The proposed scheme by Paul doesn't seem reasonable to me. Core contributor A or committers needs to actively reach out to other core contributors: labeling is not enough IMO.


On Tue, 15 Jan 2019 at 10:37, Jody Klymak <[hidden email]> wrote:


On 15 Jan 2019, at 10:20, Paul Hobson <[hidden email]> wrote:

I think this would be a good policy as well. Can I get some clarification on the flow? The way I understand it:

1) Someone (committer, contributor, or first-timer) opens a simple PR
2) Committer A reviews it, adds a "single-review-merge" label, then digitally walks away
3) No less than two weeks later, Committer B sees the PR, notices the label, opens it up, and merges without reviewing.

If there is a Committer B, I think they should at least look at the PR to make sure it doesn’t have any hidden API changes, or introduce anything hostile.  

The point here is that getting the third person to look at a PR is proving quite difficult, and if the rest of the folks with the commit bit haven’t looked at a PR for a month, even after being warned, then silence indicates consent.  

Note that only folks with the commit bit can label PRs, so anyone flagging a PR like this is implicitly trusted to not be wreaking havoc, and not doing this for major changes.  Since we all curate our own PRs, I rather expect that the PR contributor will often be the person who adds the flag.  Whether we want to allow a self-merge at that point is up for debate, but if not, then folks need to get in the habit of looking at old flagged PRs and merging them.  

I think a huge problem we have is the LIFO default sort of the github PR queue, and that any PR not on the first page might as well be closed for all the attention it will get without constant nagging.  

Cheers,   Jody

Does that capture a successful "single-review-merge" lifecycle?
-paul

On Tue, Jan 15, 2019 at 9:57 AM Nelle Varoquaux <[hidden email]> wrote:
Hello,

Overall, I think this is a good idea, but I would like specifics on what "uncontroversial but uninteresting" PR mean. IMO, that should exclude any PR with API changes.

Cheers,
N

On Tue, 15 Jan 2019 at 06:00, Antony Lee <[hidden email]> wrote:
Hi all,

During the weekly dev call, I proposed to amend the PR merge rules, following an initial comment on Github [https://github.com/matplotlib/matplotlib/pull/13173#issuecomment-453921220].  There was general agreement among the devs present (Tom, Hannah, Jody, and myself), so I'm putting it here for discussion.  The objective of this change is to prevent "uncontroversial but uninteresting" PRs from falling into oblivion, and try to decrease the size of the open PR stack.

The current rule is that a PR needs positive reviews (from two committers, and excluding the author if a committer) to be merged, except that doc-only PRs (docstrings, rst) only needs a single positive review.

I am proposing that, if a (non-doc) PR already has one positive review, but no activity on that PR has occurred for two weeks (exact time interval up to bikeshedding) [Jody suggests: and the PR has 100% code coverage], then a committer (either the first reviewer, or the author if a committer) can suggest that it be merged on the basis of that single review.  To do so, the "sponsor" should ping all developers (@matplotlib/developers) on that issue indicating that intent, and add a "single-review-merge" label on the PR (so that these PRs can easily be found).  Committers are encouraged to review the PR to accept and merge it or reject it or request changes on it; but they can also just indicate that they consider the PR sufficiently complex that a proper second review is needed before merging, or request an extension, etc.  To do so they should still leave a "reject" review, even if just saying "objecting to single-review merged; anyone can dismiss after a second review".  However, if within another two weeks, no committer voiced any objection (explicitly, i.e. by rejecting), then the PR can indeed be merged on the basis of that single review.

To avoid overwhelming the system, any committer can only "sponsor" a single PR at a time.

Thoughts?

Antony
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel

--
Jody Klymak    






_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to amend the PR merge rules

Antony Lee-3
I'm obviously more aware of my own PRs, so here are a few I'd have put up for single-review-merge (note that it's quite possible that they'd have attracted more attention and gotten merged faster under that system; that's also the goal...).  Obviously other devs may think about other PRs that could have benefitted from the same process.

- https://github.com/matplotlib/matplotlib/pull/9787 adds support for the ttc font format (requested since 2014) for png/svg output (that's provided basically for free by FreeType) but not for pdf/ps (that would require more difficult changes).  The entire PR comes down to:
  * adding "ttc" as an extension that should be treated like "ttf"/"otf",
  * adding error handling to pdf/ps to signal these formats are not supported there,
  * tests.
  The PR sat open for 10 months before a first positive review (because no one cares about fonts, I guess) and took another 3 months to attract a second positive review.

- https://github.com/matplotlib/matplotlib/pull/12472 does a fairly trivial fix to fontList.json to make it reusable for Matplotlibs installed in different venvs, got a positive review in one day and waited another 2.5 months for a second positive review.

- https://github.com/matplotlib/matplotlib/pull/9867 deduplicates completely duplicated code between the pdf and ps backends, mostly for maintainability's sake; it took 7.5 months (and 5 rebases due to conflicts) to get a first positive review and another 6 (and around as many rebases) to get a second one.

-----

As for the specifics (as I see it):
- Someone opens a PR, it gets a positive review, no negative review, and no activity occurs for two weeks (significant activity -- excluding trivial chat on the thread).
- Sponsor (first reviewer, or author if a committer) *pings* all devs by mentioning @matplotlib/developers on the PR thread with the intent to single-review-merge, and labels the PR accordingly.
- The PR can still be reviewed/merged/rejected by the normal review process.  However, if no one explicitly opposes the merge within two weeks, anyone can merge it at that point (including by self-merge).

I would not exclude all API changes from the process.  For example, https://github.com/matplotlib/matplotlib/pull/13173/commits/e90d264f3e5a263ef57243a2af86b46ab74ccc16 deprecates (and prepares for deletion) two parameters to imshow() that have had no effect since 2006.  Let's pretend for a second this commit was a single independent PR (and not an example to showcase the parameter-deprecating decorator...); if it started being forgotten I would have proposed it for single-review-merge (again, note that if you're worried about the change, you don't need to reject the PR; you can just say, I think this API change needs to be approved by a second reviewer and that'll block the single-review-merge just as well).

Antony

On Tue, Jan 15, 2019 at 7:46 PM Nelle Varoquaux <[hidden email]> wrote:
The proposed scheme by Paul doesn't seem reasonable to me. Core contributor A or committers needs to actively reach out to other core contributors: labeling is not enough IMO.


On Tue, 15 Jan 2019 at 10:37, Jody Klymak <[hidden email]> wrote:


On 15 Jan 2019, at 10:20, Paul Hobson <[hidden email]> wrote:

I think this would be a good policy as well. Can I get some clarification on the flow? The way I understand it:

1) Someone (committer, contributor, or first-timer) opens a simple PR
2) Committer A reviews it, adds a "single-review-merge" label, then digitally walks away
3) No less than two weeks later, Committer B sees the PR, notices the label, opens it up, and merges without reviewing.

If there is a Committer B, I think they should at least look at the PR to make sure it doesn’t have any hidden API changes, or introduce anything hostile.  

The point here is that getting the third person to look at a PR is proving quite difficult, and if the rest of the folks with the commit bit haven’t looked at a PR for a month, even after being warned, then silence indicates consent.  

Note that only folks with the commit bit can label PRs, so anyone flagging a PR like this is implicitly trusted to not be wreaking havoc, and not doing this for major changes.  Since we all curate our own PRs, I rather expect that the PR contributor will often be the person who adds the flag.  Whether we want to allow a self-merge at that point is up for debate, but if not, then folks need to get in the habit of looking at old flagged PRs and merging them.  

I think a huge problem we have is the LIFO default sort of the github PR queue, and that any PR not on the first page might as well be closed for all the attention it will get without constant nagging.  

Cheers,   Jody

Does that capture a successful "single-review-merge" lifecycle?
-paul

On Tue, Jan 15, 2019 at 9:57 AM Nelle Varoquaux <[hidden email]> wrote:
Hello,

Overall, I think this is a good idea, but I would like specifics on what "uncontroversial but uninteresting" PR mean. IMO, that should exclude any PR with API changes.

Cheers,
N

On Tue, 15 Jan 2019 at 06:00, Antony Lee <[hidden email]> wrote:
Hi all,

During the weekly dev call, I proposed to amend the PR merge rules, following an initial comment on Github [https://github.com/matplotlib/matplotlib/pull/13173#issuecomment-453921220].  There was general agreement among the devs present (Tom, Hannah, Jody, and myself), so I'm putting it here for discussion.  The objective of this change is to prevent "uncontroversial but uninteresting" PRs from falling into oblivion, and try to decrease the size of the open PR stack.

The current rule is that a PR needs positive reviews (from two committers, and excluding the author if a committer) to be merged, except that doc-only PRs (docstrings, rst) only needs a single positive review.

I am proposing that, if a (non-doc) PR already has one positive review, but no activity on that PR has occurred for two weeks (exact time interval up to bikeshedding) [Jody suggests: and the PR has 100% code coverage], then a committer (either the first reviewer, or the author if a committer) can suggest that it be merged on the basis of that single review.  To do so, the "sponsor" should ping all developers (@matplotlib/developers) on that issue indicating that intent, and add a "single-review-merge" label on the PR (so that these PRs can easily be found).  Committers are encouraged to review the PR to accept and merge it or reject it or request changes on it; but they can also just indicate that they consider the PR sufficiently complex that a proper second review is needed before merging, or request an extension, etc.  To do so they should still leave a "reject" review, even if just saying "objecting to single-review merged; anyone can dismiss after a second review".  However, if within another two weeks, no committer voiced any objection (explicitly, i.e. by rejecting), then the PR can indeed be merged on the basis of that single review.

To avoid overwhelming the system, any committer can only "sponsor" a single PR at a time.

Thoughts?

Antony
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel

--
Jody Klymak    





_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to amend the PR merge rules

Eric Firing
Antony,

Your examples illustrate that the problem is often that we don't have
enough people who understand some areas, like fonts and pdf/ps file
formats, to get 2 thorough and knowledgeable reviews in a reasonable time.

I'm fine with your proposed rule.  It should help at least a little bit.

Eric


On 2019/01/15 9:11 AM, Antony Lee wrote:

> I'm obviously more aware of my own PRs, so here are a few I'd have put
> up for single-review-merge (note that it's quite possible that they'd
> have attracted more attention and gotten merged faster under that
> system; that's also the goal...).  Obviously other devs may think about
> other PRs that could have benefitted from the same process.
>
> - https://github.com/matplotlib/matplotlib/pull/9787 adds support for
> the ttc font format (requested since 2014) for png/svg output (that's
> provided basically for free by FreeType) but not for pdf/ps (that would
> require more difficult changes).  The entire PR comes down to:
>    * adding "ttc" as an extension that should be treated like "ttf"/"otf",
>    * adding error handling to pdf/ps to signal these formats are not
> supported there,
>    * tests.
>    The PR sat open for 10 months before a first positive review (because
> no one cares about fonts, I guess) and took another 3 months to attract
> a second positive review.
>
> - https://github.com/matplotlib/matplotlib/pull/12472 does a fairly
> trivial fix to fontList.json to make it reusable for Matplotlibs
> installed in different venvs, got a positive review in one day and
> waited another 2.5 months for a second positive review.
>
> - https://github.com/matplotlib/matplotlib/pull/9867 deduplicates
> completely duplicated code between the pdf and ps backends, mostly for
> maintainability's sake; it took 7.5 months (and 5 rebases due to
> conflicts) to get a first positive review and another 6 (and around as
> many rebases) to get a second one.
>
> -----
>
> As for the specifics (as I see it):
> - Someone opens a PR, it gets a positive review, no negative review, and
> no activity occurs for two weeks (significant activity -- excluding
> trivial chat on the thread).
> - Sponsor (first reviewer, or author if a committer) *pings* all devs by
> mentioning @matplotlib/developers on the PR thread with the intent to
> single-review-merge, and labels the PR accordingly.
> - The PR can still be reviewed/merged/rejected by the normal review
> process.  However, if no one explicitly opposes the merge within two
> weeks, anyone can merge it at that point (including by self-merge).
>
> I would not exclude all API changes from the process.  For example,
> https://github.com/matplotlib/matplotlib/pull/13173/commits/e90d264f3e5a263ef57243a2af86b46ab74ccc16 
> deprecates (and prepares for deletion) two parameters to imshow() that
> have had no effect since 2006.  Let's pretend for a second this commit
> was a single independent PR (and not an example to showcase the
> parameter-deprecating decorator...); if it started being forgotten I
> would have proposed it for single-review-merge (again, note that if
> you're worried about the change, you don't need to reject the PR; you
> can just say, I think this API change needs to be approved by a second
> reviewer and that'll block the single-review-merge just as well).
>
> Antony
>
> On Tue, Jan 15, 2019 at 7:46 PM Nelle Varoquaux
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     The proposed scheme by Paul doesn't seem reasonable to me. Core
>     contributor A or committers needs to actively reach out to other
>     core contributors: labeling is not enough IMO.
>
>
>     On Tue, 15 Jan 2019 at 10:37, Jody Klymak <[hidden email]
>     <mailto:[hidden email]>> wrote:
>
>
>
>>         On 15 Jan 2019, at 10:20, Paul Hobson <[hidden email]
>>         <mailto:[hidden email]>> wrote:
>>
>>         I think this would be a good policy as well. Can I get some
>>         clarification on the flow? The way I understand it:
>>
>>         1) Someone (committer, contributor, or first-timer) opens a
>>         simple PR
>>         2) Committer A reviews it, adds a "single-review-merge" label,
>>         then digitally walks away
>>         3) No less than two weeks later, Committer B sees the PR,
>>         notices the label, opens it up, and merges without reviewing.
>
>         If there is a Committer B, I think they should at least look at
>         the PR to make sure it doesn’t have any hidden API changes, or
>         introduce anything hostile.
>
>         The point here is that getting the third person to look at a PR
>         is proving quite difficult, and if the rest of the folks with
>         the commit bit haven’t looked at a PR for a month, even after
>         being warned, then silence indicates consent.
>
>         Note that only folks with the commit bit can label PRs, so
>         anyone flagging a PR like this is implicitly trusted to not be
>         wreaking havoc, and not doing this for major changes.  Since we
>         all curate our own PRs, I rather expect that the PR contributor
>         will often be the person who adds the flag.  Whether we want to
>         allow a self-merge at that point is up for debate, but if not,
>         then folks need to get in the habit of looking at old flagged
>         PRs and merging them.
>
>         I think a huge problem we have is the LIFO default sort of the
>         github PR queue, and that any PR not on the first page might as
>         well be closed for all the attention it will get without
>         constant nagging.
>
>         Cheers,   Jody
>
>>         Does that capture a successful "single-review-merge" lifecycle?
>>         -paul
>>
>>         On Tue, Jan 15, 2019 at 9:57 AM Nelle Varoquaux
>>         <[hidden email] <mailto:[hidden email]>>
>>         wrote:
>>
>>             Hello,
>>
>>             Overall, I think this is a good idea, but I would like
>>             specifics on what "uncontroversial but uninteresting" PR
>>             mean. IMO, that should exclude any PR with API changes.
>>
>>             Cheers,
>>             N
>>
>>             On Tue, 15 Jan 2019 at 06:00, Antony Lee
>>             <[hidden email] <mailto:[hidden email]>> wrote:
>>
>>                 Hi all,
>>
>>                 During the weekly dev call, I proposed to amend the PR
>>                 merge rules, following an initial comment on Github
>>                 [https://github.com/matplotlib/matplotlib/pull/13173#issuecomment-453921220].
>>                 There was general agreement among the devs present
>>                 (Tom, Hannah, Jody, and myself), so I'm putting it
>>                 here for discussion.  The objective of this change is
>>                 to prevent "uncontroversial but uninteresting" PRs
>>                 from falling into oblivion, and try to decrease the
>>                 size of the open PR stack.
>>
>>                 The current rule is that a PR needs positive reviews
>>                 (from two committers, and excluding the author if a
>>                 committer) to be merged, except that doc-only PRs
>>                 (docstrings, rst) only needs a single positive review.
>>
>>                 I am proposing that, if a (non-doc) PR already has one
>>                 positive review, but no activity on that PR has
>>                 occurred for two weeks (exact time interval up to
>>                 bikeshedding) [Jody suggests: and the PR has 100% code
>>                 coverage], then a committer (either the first
>>                 reviewer, or the author if a committer) can suggest
>>                 that it be merged on the basis of that single review.
>>                 To do so, the "sponsor" should ping all developers
>>                 (@matplotlib/developers) on that issue indicating that
>>                 intent, and add a "single-review-merge" label on the
>>                 PR (so that these PRs can easily be found).
>>                 Committers are encouraged to review the PR to accept
>>                 and merge it or reject it or request changes on it;
>>                 but they can also just indicate that they consider the
>>                 PR sufficiently complex that a proper second review is
>>                 needed before merging, or request an extension, etc.
>>                 To do so they should still leave a "reject" review,
>>                 even if just saying "objecting to single-review
>>                 merged; anyone can dismiss after a second review".
>>                 However, if within another two weeks, no committer
>>                 voiced any objection (explicitly, i.e. by rejecting),
>>                 then the PR can indeed be merged on the basis of that
>>                 single review.
>>
>>                 To avoid overwhelming the system, any committer can
>>                 only "sponsor" a single PR at a time.
>>
>>                 Thoughts?
>>
>>                 Antony
>>                 _______________________________________________
>>                 Matplotlib-devel mailing list
>>                 [hidden email]
>>                 <mailto:[hidden email]>
>>                 https://mail.python.org/mailman/listinfo/matplotlib-devel
>>
>>             _______________________________________________
>>             Matplotlib-devel mailing list
>>             [hidden email]
>>             <mailto:[hidden email]>
>>             https://mail.python.org/mailman/listinfo/matplotlib-devel
>>
>>         _______________________________________________
>>         Matplotlib-devel mailing list
>>         [hidden email] <mailto:[hidden email]>
>>         https://mail.python.org/mailman/listinfo/matplotlib-devel
>
>         --
>         Jody Klymak
>         http://web.uvic.ca/~jklymak/
>
>
>
>
>
>     _______________________________________________
>     Matplotlib-devel mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://mail.python.org/mailman/listinfo/matplotlib-devel
>
>
> _______________________________________________
> Matplotlib-devel mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/matplotlib-devel
>

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to amend the PR merge rules

Antony Lee-3
Eric,
Neither the ttc font, nor the pdf/ps common code PRs actually touch any "complex" point regarding fonts or pdf/ps.  I have described the ttc font in my previous message; the pdf/ps PR is really just, look, these two classes share 80 lines worth of code that's literally copy-pasted, let's just put that in a common parent class and inherit from it.
Conversely, one font-related PR for which I actually have low hopes of getting properly reviewed (except on an "sure, I trust you" basis) is https://github.com/matplotlib/matplotlib/pull/12928, which actually touches a rather complex point about font encodings (some of which I had to clarify on the FreeType mailing list).  (If anyone wants to have a look at it, though... :p)
Antony

On Tue, Jan 15, 2019 at 10:36 PM Eric Firing <[hidden email]> wrote:
Antony,

Your examples illustrate that the problem is often that we don't have
enough people who understand some areas, like fonts and pdf/ps file
formats, to get 2 thorough and knowledgeable reviews in a reasonable time.

I'm fine with your proposed rule.  It should help at least a little bit.

Eric


On 2019/01/15 9:11 AM, Antony Lee wrote:
> I'm obviously more aware of my own PRs, so here are a few I'd have put
> up for single-review-merge (note that it's quite possible that they'd
> have attracted more attention and gotten merged faster under that
> system; that's also the goal...).  Obviously other devs may think about
> other PRs that could have benefitted from the same process.
>
> - https://github.com/matplotlib/matplotlib/pull/9787 adds support for
> the ttc font format (requested since 2014) for png/svg output (that's
> provided basically for free by FreeType) but not for pdf/ps (that would
> require more difficult changes).  The entire PR comes down to:
>    * adding "ttc" as an extension that should be treated like "ttf"/"otf",
>    * adding error handling to pdf/ps to signal these formats are not
> supported there,
>    * tests.
>    The PR sat open for 10 months before a first positive review (because
> no one cares about fonts, I guess) and took another 3 months to attract
> a second positive review.
>
> - https://github.com/matplotlib/matplotlib/pull/12472 does a fairly
> trivial fix to fontList.json to make it reusable for Matplotlibs
> installed in different venvs, got a positive review in one day and
> waited another 2.5 months for a second positive review.
>
> - https://github.com/matplotlib/matplotlib/pull/9867 deduplicates
> completely duplicated code between the pdf and ps backends, mostly for
> maintainability's sake; it took 7.5 months (and 5 rebases due to
> conflicts) to get a first positive review and another 6 (and around as
> many rebases) to get a second one.
>
> -----
>
> As for the specifics (as I see it):
> - Someone opens a PR, it gets a positive review, no negative review, and
> no activity occurs for two weeks (significant activity -- excluding
> trivial chat on the thread).
> - Sponsor (first reviewer, or author if a committer) *pings* all devs by
> mentioning @matplotlib/developers on the PR thread with the intent to
> single-review-merge, and labels the PR accordingly.
> - The PR can still be reviewed/merged/rejected by the normal review
> process.  However, if no one explicitly opposes the merge within two
> weeks, anyone can merge it at that point (including by self-merge).
>
> I would not exclude all API changes from the process.  For example,
> https://github.com/matplotlib/matplotlib/pull/13173/commits/e90d264f3e5a263ef57243a2af86b46ab74ccc16
> deprecates (and prepares for deletion) two parameters to imshow() that
> have had no effect since 2006.  Let's pretend for a second this commit
> was a single independent PR (and not an example to showcase the
> parameter-deprecating decorator...); if it started being forgotten I
> would have proposed it for single-review-merge (again, note that if
> you're worried about the change, you don't need to reject the PR; you
> can just say, I think this API change needs to be approved by a second
> reviewer and that'll block the single-review-merge just as well).
>
> Antony
>
> On Tue, Jan 15, 2019 at 7:46 PM Nelle Varoquaux
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     The proposed scheme by Paul doesn't seem reasonable to me. Core
>     contributor A or committers needs to actively reach out to other
>     core contributors: labeling is not enough IMO.
>
>
>     On Tue, 15 Jan 2019 at 10:37, Jody Klymak <[hidden email]
>     <mailto:[hidden email]>> wrote:
>
>
>
>>         On 15 Jan 2019, at 10:20, Paul Hobson <[hidden email]
>>         <mailto:[hidden email]>> wrote:
>>
>>         I think this would be a good policy as well. Can I get some
>>         clarification on the flow? The way I understand it:
>>
>>         1) Someone (committer, contributor, or first-timer) opens a
>>         simple PR
>>         2) Committer A reviews it, adds a "single-review-merge" label,
>>         then digitally walks away
>>         3) No less than two weeks later, Committer B sees the PR,
>>         notices the label, opens it up, and merges without reviewing.
>
>         If there is a Committer B, I think they should at least look at
>         the PR to make sure it doesn’t have any hidden API changes, or
>         introduce anything hostile.
>
>         The point here is that getting the third person to look at a PR
>         is proving quite difficult, and if the rest of the folks with
>         the commit bit haven’t looked at a PR for a month, even after
>         being warned, then silence indicates consent.
>
>         Note that only folks with the commit bit can label PRs, so
>         anyone flagging a PR like this is implicitly trusted to not be
>         wreaking havoc, and not doing this for major changes.  Since we
>         all curate our own PRs, I rather expect that the PR contributor
>         will often be the person who adds the flag.  Whether we want to
>         allow a self-merge at that point is up for debate, but if not,
>         then folks need to get in the habit of looking at old flagged
>         PRs and merging them.
>
>         I think a huge problem we have is the LIFO default sort of the
>         github PR queue, and that any PR not on the first page might as
>         well be closed for all the attention it will get without
>         constant nagging.
>
>         Cheers,   Jody
>
>>         Does that capture a successful "single-review-merge" lifecycle?
>>         -paul
>>
>>         On Tue, Jan 15, 2019 at 9:57 AM Nelle Varoquaux
>>         <[hidden email] <mailto:[hidden email]>>
>>         wrote:
>>
>>             Hello,
>>
>>             Overall, I think this is a good idea, but I would like
>>             specifics on what "uncontroversial but uninteresting" PR
>>             mean. IMO, that should exclude any PR with API changes.
>>
>>             Cheers,
>>             N
>>
>>             On Tue, 15 Jan 2019 at 06:00, Antony Lee
>>             <[hidden email] <mailto:[hidden email]>> wrote:
>>
>>                 Hi all,
>>
>>                 During the weekly dev call, I proposed to amend the PR
>>                 merge rules, following an initial comment on Github
>>                 [https://github.com/matplotlib/matplotlib/pull/13173#issuecomment-453921220].
>>                 There was general agreement among the devs present
>>                 (Tom, Hannah, Jody, and myself), so I'm putting it
>>                 here for discussion.  The objective of this change is
>>                 to prevent "uncontroversial but uninteresting" PRs
>>                 from falling into oblivion, and try to decrease the
>>                 size of the open PR stack.
>>
>>                 The current rule is that a PR needs positive reviews
>>                 (from two committers, and excluding the author if a
>>                 committer) to be merged, except that doc-only PRs
>>                 (docstrings, rst) only needs a single positive review.
>>
>>                 I am proposing that, if a (non-doc) PR already has one
>>                 positive review, but no activity on that PR has
>>                 occurred for two weeks (exact time interval up to
>>                 bikeshedding) [Jody suggests: and the PR has 100% code
>>                 coverage], then a committer (either the first
>>                 reviewer, or the author if a committer) can suggest
>>                 that it be merged on the basis of that single review.
>>                 To do so, the "sponsor" should ping all developers
>>                 (@matplotlib/developers) on that issue indicating that
>>                 intent, and add a "single-review-merge" label on the
>>                 PR (so that these PRs can easily be found).
>>                 Committers are encouraged to review the PR to accept
>>                 and merge it or reject it or request changes on it;
>>                 but they can also just indicate that they consider the
>>                 PR sufficiently complex that a proper second review is
>>                 needed before merging, or request an extension, etc.
>>                 To do so they should still leave a "reject" review,
>>                 even if just saying "objecting to single-review
>>                 merged; anyone can dismiss after a second review".
>>                 However, if within another two weeks, no committer
>>                 voiced any objection (explicitly, i.e. by rejecting),
>>                 then the PR can indeed be merged on the basis of that
>>                 single review.
>>
>>                 To avoid overwhelming the system, any committer can
>>                 only "sponsor" a single PR at a time.
>>
>>                 Thoughts?
>>
>>                 Antony
>>                 _______________________________________________
>>                 Matplotlib-devel mailing list
>>                 [hidden email]
>>                 <mailto:[hidden email]>
>>                 https://mail.python.org/mailman/listinfo/matplotlib-devel
>>
>>             _______________________________________________
>>             Matplotlib-devel mailing list
>>             [hidden email]
>>             <mailto:[hidden email]>
>>             https://mail.python.org/mailman/listinfo/matplotlib-devel
>>
>>         _______________________________________________
>>         Matplotlib-devel mailing list
>>         [hidden email] <mailto:[hidden email]>
>>         https://mail.python.org/mailman/listinfo/matplotlib-devel
>
>         --
>         Jody Klymak
>         http://web.uvic.ca/~jklymak/
>
>
>
>
>
>     _______________________________________________
>     Matplotlib-devel mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://mail.python.org/mailman/listinfo/matplotlib-devel
>
>
> _______________________________________________
> Matplotlib-devel mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/matplotlib-devel
>

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to amend the PR merge rules

Eric Firing
On 2019/01/15 11:44 AM, Antony Lee wrote:
> Neither the ttc font, nor the pdf/ps common code PRs actually touch any
> "complex" point regarding fonts or pdf/ps.

I understand.  Nevertheless, there is an understandable tendency for
someone like me to look at such a PR and back off, hoping someone more
familiar with those areas will take care of it. I think your proposal
will help with this problem.

Eric

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to amend the PR merge rules

tcaswell
In reply to this post by Antony Lee-3
I think a distinction between "no one cares" and "no one has bandwidth" needs to be made.

If you look at the 'pulse' page, we merging 100+ PRs a month (https://github.com/matplotlib/matplotlib/pulse/monthly) so reviews are happening, the issue is just that some are falling through the cracks.

Tom

On Tue, Jan 15, 2019 at 4:45 PM Antony Lee <[hidden email]> wrote:
Eric,
Neither the ttc font, nor the pdf/ps common code PRs actually touch any "complex" point regarding fonts or pdf/ps.  I have described the ttc font in my previous message; the pdf/ps PR is really just, look, these two classes share 80 lines worth of code that's literally copy-pasted, let's just put that in a common parent class and inherit from it.
Conversely, one font-related PR for which I actually have low hopes of getting properly reviewed (except on an "sure, I trust you" basis) is https://github.com/matplotlib/matplotlib/pull/12928, which actually touches a rather complex point about font encodings (some of which I had to clarify on the FreeType mailing list).  (If anyone wants to have a look at it, though... :p)
Antony

On Tue, Jan 15, 2019 at 10:36 PM Eric Firing <[hidden email]> wrote:
Antony,

Your examples illustrate that the problem is often that we don't have
enough people who understand some areas, like fonts and pdf/ps file
formats, to get 2 thorough and knowledgeable reviews in a reasonable time.

I'm fine with your proposed rule.  It should help at least a little bit.

Eric


On 2019/01/15 9:11 AM, Antony Lee wrote:
> I'm obviously more aware of my own PRs, so here are a few I'd have put
> up for single-review-merge (note that it's quite possible that they'd
> have attracted more attention and gotten merged faster under that
> system; that's also the goal...).  Obviously other devs may think about
> other PRs that could have benefitted from the same process.
>
> - https://github.com/matplotlib/matplotlib/pull/9787 adds support for
> the ttc font format (requested since 2014) for png/svg output (that's
> provided basically for free by FreeType) but not for pdf/ps (that would
> require more difficult changes).  The entire PR comes down to:
>    * adding "ttc" as an extension that should be treated like "ttf"/"otf",
>    * adding error handling to pdf/ps to signal these formats are not
> supported there,
>    * tests.
>    The PR sat open for 10 months before a first positive review (because
> no one cares about fonts, I guess) and took another 3 months to attract
> a second positive review.
>
> - https://github.com/matplotlib/matplotlib/pull/12472 does a fairly
> trivial fix to fontList.json to make it reusable for Matplotlibs
> installed in different venvs, got a positive review in one day and
> waited another 2.5 months for a second positive review.
>
> - https://github.com/matplotlib/matplotlib/pull/9867 deduplicates
> completely duplicated code between the pdf and ps backends, mostly for
> maintainability's sake; it took 7.5 months (and 5 rebases due to
> conflicts) to get a first positive review and another 6 (and around as
> many rebases) to get a second one.
>
> -----
>
> As for the specifics (as I see it):
> - Someone opens a PR, it gets a positive review, no negative review, and
> no activity occurs for two weeks (significant activity -- excluding
> trivial chat on the thread).
> - Sponsor (first reviewer, or author if a committer) *pings* all devs by
> mentioning @matplotlib/developers on the PR thread with the intent to
> single-review-merge, and labels the PR accordingly.
> - The PR can still be reviewed/merged/rejected by the normal review
> process.  However, if no one explicitly opposes the merge within two
> weeks, anyone can merge it at that point (including by self-merge).
>
> I would not exclude all API changes from the process.  For example,
> https://github.com/matplotlib/matplotlib/pull/13173/commits/e90d264f3e5a263ef57243a2af86b46ab74ccc16
> deprecates (and prepares for deletion) two parameters to imshow() that
> have had no effect since 2006.  Let's pretend for a second this commit
> was a single independent PR (and not an example to showcase the
> parameter-deprecating decorator...); if it started being forgotten I
> would have proposed it for single-review-merge (again, note that if
> you're worried about the change, you don't need to reject the PR; you
> can just say, I think this API change needs to be approved by a second
> reviewer and that'll block the single-review-merge just as well).
>
> Antony
>
> On Tue, Jan 15, 2019 at 7:46 PM Nelle Varoquaux
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     The proposed scheme by Paul doesn't seem reasonable to me. Core
>     contributor A or committers needs to actively reach out to other
>     core contributors: labeling is not enough IMO.
>
>
>     On Tue, 15 Jan 2019 at 10:37, Jody Klymak <[hidden email]
>     <mailto:[hidden email]>> wrote:
>
>
>
>>         On 15 Jan 2019, at 10:20, Paul Hobson <[hidden email]
>>         <mailto:[hidden email]>> wrote:
>>
>>         I think this would be a good policy as well. Can I get some
>>         clarification on the flow? The way I understand it:
>>
>>         1) Someone (committer, contributor, or first-timer) opens a
>>         simple PR
>>         2) Committer A reviews it, adds a "single-review-merge" label,
>>         then digitally walks away
>>         3) No less than two weeks later, Committer B sees the PR,
>>         notices the label, opens it up, and merges without reviewing.
>
>         If there is a Committer B, I think they should at least look at
>         the PR to make sure it doesn’t have any hidden API changes, or
>         introduce anything hostile.
>
>         The point here is that getting the third person to look at a PR
>         is proving quite difficult, and if the rest of the folks with
>         the commit bit haven’t looked at a PR for a month, even after
>         being warned, then silence indicates consent.
>
>         Note that only folks with the commit bit can label PRs, so
>         anyone flagging a PR like this is implicitly trusted to not be
>         wreaking havoc, and not doing this for major changes.  Since we
>         all curate our own PRs, I rather expect that the PR contributor
>         will often be the person who adds the flag.  Whether we want to
>         allow a self-merge at that point is up for debate, but if not,
>         then folks need to get in the habit of looking at old flagged
>         PRs and merging them.
>
>         I think a huge problem we have is the LIFO default sort of the
>         github PR queue, and that any PR not on the first page might as
>         well be closed for all the attention it will get without
>         constant nagging.
>
>         Cheers,   Jody
>
>>         Does that capture a successful "single-review-merge" lifecycle?
>>         -paul
>>
>>         On Tue, Jan 15, 2019 at 9:57 AM Nelle Varoquaux
>>         <[hidden email] <mailto:[hidden email]>>
>>         wrote:
>>
>>             Hello,
>>
>>             Overall, I think this is a good idea, but I would like
>>             specifics on what "uncontroversial but uninteresting" PR
>>             mean. IMO, that should exclude any PR with API changes.
>>
>>             Cheers,
>>             N
>>
>>             On Tue, 15 Jan 2019 at 06:00, Antony Lee
>>             <[hidden email] <mailto:[hidden email]>> wrote:
>>
>>                 Hi all,
>>
>>                 During the weekly dev call, I proposed to amend the PR
>>                 merge rules, following an initial comment on Github
>>                 [https://github.com/matplotlib/matplotlib/pull/13173#issuecomment-453921220].
>>                 There was general agreement among the devs present
>>                 (Tom, Hannah, Jody, and myself), so I'm putting it
>>                 here for discussion.  The objective of this change is
>>                 to prevent "uncontroversial but uninteresting" PRs
>>                 from falling into oblivion, and try to decrease the
>>                 size of the open PR stack.
>>
>>                 The current rule is that a PR needs positive reviews
>>                 (from two committers, and excluding the author if a
>>                 committer) to be merged, except that doc-only PRs
>>                 (docstrings, rst) only needs a single positive review.
>>
>>                 I am proposing that, if a (non-doc) PR already has one
>>                 positive review, but no activity on that PR has
>>                 occurred for two weeks (exact time interval up to
>>                 bikeshedding) [Jody suggests: and the PR has 100% code
>>                 coverage], then a committer (either the first
>>                 reviewer, or the author if a committer) can suggest
>>                 that it be merged on the basis of that single review.
>>                 To do so, the "sponsor" should ping all developers
>>                 (@matplotlib/developers) on that issue indicating that
>>                 intent, and add a "single-review-merge" label on the
>>                 PR (so that these PRs can easily be found).
>>                 Committers are encouraged to review the PR to accept
>>                 and merge it or reject it or request changes on it;
>>                 but they can also just indicate that they consider the
>>                 PR sufficiently complex that a proper second review is
>>                 needed before merging, or request an extension, etc.
>>                 To do so they should still leave a "reject" review,
>>                 even if just saying "objecting to single-review
>>                 merged; anyone can dismiss after a second review".
>>                 However, if within another two weeks, no committer
>>                 voiced any objection (explicitly, i.e. by rejecting),
>>                 then the PR can indeed be merged on the basis of that
>>                 single review.
>>
>>                 To avoid overwhelming the system, any committer can
>>                 only "sponsor" a single PR at a time.
>>
>>                 Thoughts?
>>
>>                 Antony
>>                 _______________________________________________
>>                 Matplotlib-devel mailing list
>>                 [hidden email]
>>                 <mailto:[hidden email]>
>>                 https://mail.python.org/mailman/listinfo/matplotlib-devel
>>
>>             _______________________________________________
>>             Matplotlib-devel mailing list
>>             [hidden email]
>>             <mailto:[hidden email]>
>>             https://mail.python.org/mailman/listinfo/matplotlib-devel
>>
>>         _______________________________________________
>>         Matplotlib-devel mailing list
>>         [hidden email] <mailto:[hidden email]>
>>         https://mail.python.org/mailman/listinfo/matplotlib-devel
>
>         --
>         Jody Klymak
>         http://web.uvic.ca/~jklymak/
>
>
>
>
>
>     _______________________________________________
>     Matplotlib-devel mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://mail.python.org/mailman/listinfo/matplotlib-devel
>
>
> _______________________________________________
> Matplotlib-devel mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/matplotlib-devel
>

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel


--
Thomas Caswell
[hidden email]

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to amend the PR merge rules

Nelle Varoquaux
For the way GitHub sorts pull requests, it is indeed a problem, but there's a way to sort by "most recently updated." I sometimes use it to make sure that I review recently updated pull request that don't appear on the first page.

It seems to me that this "uncontroversial but uninteresting" rule is currently not based on objective criteria. I'm personally not in favor of removing two core contributors approval for anything that modifies the API.  We have had too many close calls on PR almost merged that had backward incompatible changes: I realize that some core developers believe that we are not moving fast enough, on a project with hundreds of thousands of users, changes to the API should not be considered lightly.

Here's an attempt at better defining an "uncontroversial but uninteresting" rule based on objective criteria:
- Has not been commented on by more than one core developer;
- Doesn't change the public API;
- Doesn't add or remove a functionality;
- Doesn't add or remove a dependency;
- Is fully tested;
- All parameters are fully documented;
- Doesn't fix a release-critical bug;
- Is less than XX lines of code. (Long PR are harder to review, and thus the likelihood of one core dev not noticing a problem is higher). Threshold needs to be defined.

To be honest, it also doesn't help that some people have 50+ pull requests open (Antony? :p). It also doesn't help that some PR contain the same code (13117, 13128) and are being hold off until the other one is merged (also, why is 13117 being hold off for a PR that has been opened afterwards?). IMO, if a PR can be merged, it should be merged. But we shouldn't make too many exceptions to the two core contributors rules. It's easier to have two core contributor review a pull request, than to try to fix API issues on code that has been merged and released. Having two pairs of eyes on a PR is also a good way to avoid having a bus factor of 1 on some areas of the library.


On Tue, 15 Jan 2019 at 15:19, Thomas Caswell <[hidden email]> wrote:
I think a distinction between "no one cares" and "no one has bandwidth" needs to be made.

If you look at the 'pulse' page, we merging 100+ PRs a month (https://github.com/matplotlib/matplotlib/pulse/monthly) so reviews are happening, the issue is just that some are falling through the cracks.

Tom

On Tue, Jan 15, 2019 at 4:45 PM Antony Lee <[hidden email]> wrote:
Eric,
Neither the ttc font, nor the pdf/ps common code PRs actually touch any "complex" point regarding fonts or pdf/ps.  I have described the ttc font in my previous message; the pdf/ps PR is really just, look, these two classes share 80 lines worth of code that's literally copy-pasted, let's just put that in a common parent class and inherit from it.
Conversely, one font-related PR for which I actually have low hopes of getting properly reviewed (except on an "sure, I trust you" basis) is https://github.com/matplotlib/matplotlib/pull/12928, which actually touches a rather complex point about font encodings (some of which I had to clarify on the FreeType mailing list).  (If anyone wants to have a look at it, though... :p)
Antony

On Tue, Jan 15, 2019 at 10:36 PM Eric Firing <[hidden email]> wrote:
Antony,

Your examples illustrate that the problem is often that we don't have
enough people who understand some areas, like fonts and pdf/ps file
formats, to get 2 thorough and knowledgeable reviews in a reasonable time.

I'm fine with your proposed rule.  It should help at least a little bit.

Eric


On 2019/01/15 9:11 AM, Antony Lee wrote:
> I'm obviously more aware of my own PRs, so here are a few I'd have put
> up for single-review-merge (note that it's quite possible that they'd
> have attracted more attention and gotten merged faster under that
> system; that's also the goal...).  Obviously other devs may think about
> other PRs that could have benefitted from the same process.
>
> - https://github.com/matplotlib/matplotlib/pull/9787 adds support for
> the ttc font format (requested since 2014) for png/svg output (that's
> provided basically for free by FreeType) but not for pdf/ps (that would
> require more difficult changes).  The entire PR comes down to:
>    * adding "ttc" as an extension that should be treated like "ttf"/"otf",
>    * adding error handling to pdf/ps to signal these formats are not
> supported there,
>    * tests.
>    The PR sat open for 10 months before a first positive review (because
> no one cares about fonts, I guess) and took another 3 months to attract
> a second positive review.
>
> - https://github.com/matplotlib/matplotlib/pull/12472 does a fairly
> trivial fix to fontList.json to make it reusable for Matplotlibs
> installed in different venvs, got a positive review in one day and
> waited another 2.5 months for a second positive review.
>
> - https://github.com/matplotlib/matplotlib/pull/9867 deduplicates
> completely duplicated code between the pdf and ps backends, mostly for
> maintainability's sake; it took 7.5 months (and 5 rebases due to
> conflicts) to get a first positive review and another 6 (and around as
> many rebases) to get a second one.
>
> -----
>
> As for the specifics (as I see it):
> - Someone opens a PR, it gets a positive review, no negative review, and
> no activity occurs for two weeks (significant activity -- excluding
> trivial chat on the thread).
> - Sponsor (first reviewer, or author if a committer) *pings* all devs by
> mentioning @matplotlib/developers on the PR thread with the intent to
> single-review-merge, and labels the PR accordingly.
> - The PR can still be reviewed/merged/rejected by the normal review
> process.  However, if no one explicitly opposes the merge within two
> weeks, anyone can merge it at that point (including by self-merge).
>
> I would not exclude all API changes from the process.  For example,
> https://github.com/matplotlib/matplotlib/pull/13173/commits/e90d264f3e5a263ef57243a2af86b46ab74ccc16
> deprecates (and prepares for deletion) two parameters to imshow() that
> have had no effect since 2006.  Let's pretend for a second this commit
> was a single independent PR (and not an example to showcase the
> parameter-deprecating decorator...); if it started being forgotten I
> would have proposed it for single-review-merge (again, note that if
> you're worried about the change, you don't need to reject the PR; you
> can just say, I think this API change needs to be approved by a second
> reviewer and that'll block the single-review-merge just as well).
>
> Antony
>
> On Tue, Jan 15, 2019 at 7:46 PM Nelle Varoquaux
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     The proposed scheme by Paul doesn't seem reasonable to me. Core
>     contributor A or committers needs to actively reach out to other
>     core contributors: labeling is not enough IMO.
>
>
>     On Tue, 15 Jan 2019 at 10:37, Jody Klymak <[hidden email]
>     <mailto:[hidden email]>> wrote:
>
>
>
>>         On 15 Jan 2019, at 10:20, Paul Hobson <[hidden email]
>>         <mailto:[hidden email]>> wrote:
>>
>>         I think this would be a good policy as well. Can I get some
>>         clarification on the flow? The way I understand it:
>>
>>         1) Someone (committer, contributor, or first-timer) opens a
>>         simple PR
>>         2) Committer A reviews it, adds a "single-review-merge" label,
>>         then digitally walks away
>>         3) No less than two weeks later, Committer B sees the PR,
>>         notices the label, opens it up, and merges without reviewing.
>
>         If there is a Committer B, I think they should at least look at
>         the PR to make sure it doesn’t have any hidden API changes, or
>         introduce anything hostile.
>
>         The point here is that getting the third person to look at a PR
>         is proving quite difficult, and if the rest of the folks with
>         the commit bit haven’t looked at a PR for a month, even after
>         being warned, then silence indicates consent.
>
>         Note that only folks with the commit bit can label PRs, so
>         anyone flagging a PR like this is implicitly trusted to not be
>         wreaking havoc, and not doing this for major changes.  Since we
>         all curate our own PRs, I rather expect that the PR contributor
>         will often be the person who adds the flag.  Whether we want to
>         allow a self-merge at that point is up for debate, but if not,
>         then folks need to get in the habit of looking at old flagged
>         PRs and merging them.
>
>         I think a huge problem we have is the LIFO default sort of the
>         github PR queue, and that any PR not on the first page might as
>         well be closed for all the attention it will get without
>         constant nagging.
>
>         Cheers,   Jody
>
>>         Does that capture a successful "single-review-merge" lifecycle?
>>         -paul
>>
>>         On Tue, Jan 15, 2019 at 9:57 AM Nelle Varoquaux
>>         <[hidden email] <mailto:[hidden email]>>
>>         wrote:
>>
>>             Hello,
>>
>>             Overall, I think this is a good idea, but I would like
>>             specifics on what "uncontroversial but uninteresting" PR
>>             mean. IMO, that should exclude any PR with API changes.
>>
>>             Cheers,
>>             N
>>
>>             On Tue, 15 Jan 2019 at 06:00, Antony Lee
>>             <[hidden email] <mailto:[hidden email]>> wrote:
>>
>>                 Hi all,
>>
>>                 During the weekly dev call, I proposed to amend the PR
>>                 merge rules, following an initial comment on Github
>>                 [https://github.com/matplotlib/matplotlib/pull/13173#issuecomment-453921220].
>>                 There was general agreement among the devs present
>>                 (Tom, Hannah, Jody, and myself), so I'm putting it
>>                 here for discussion.  The objective of this change is
>>                 to prevent "uncontroversial but uninteresting" PRs
>>                 from falling into oblivion, and try to decrease the
>>                 size of the open PR stack.
>>
>>                 The current rule is that a PR needs positive reviews
>>                 (from two committers, and excluding the author if a
>>                 committer) to be merged, except that doc-only PRs
>>                 (docstrings, rst) only needs a single positive review.
>>
>>                 I am proposing that, if a (non-doc) PR already has one
>>                 positive review, but no activity on that PR has
>>                 occurred for two weeks (exact time interval up to
>>                 bikeshedding) [Jody suggests: and the PR has 100% code
>>                 coverage], then a committer (either the first
>>                 reviewer, or the author if a committer) can suggest
>>                 that it be merged on the basis of that single review.
>>                 To do so, the "sponsor" should ping all developers
>>                 (@matplotlib/developers) on that issue indicating that
>>                 intent, and add a "single-review-merge" label on the
>>                 PR (so that these PRs can easily be found).
>>                 Committers are encouraged to review the PR to accept
>>                 and merge it or reject it or request changes on it;
>>                 but they can also just indicate that they consider the
>>                 PR sufficiently complex that a proper second review is
>>                 needed before merging, or request an extension, etc.
>>                 To do so they should still leave a "reject" review,
>>                 even if just saying "objecting to single-review
>>                 merged; anyone can dismiss after a second review".
>>                 However, if within another two weeks, no committer
>>                 voiced any objection (explicitly, i.e. by rejecting),
>>                 then the PR can indeed be merged on the basis of that
>>                 single review.
>>
>>                 To avoid overwhelming the system, any committer can
>>                 only "sponsor" a single PR at a time.
>>
>>                 Thoughts?
>>
>>                 Antony
>>                 _______________________________________________
>>                 Matplotlib-devel mailing list
>>                 [hidden email]
>>                 <mailto:[hidden email]>
>>                 https://mail.python.org/mailman/listinfo/matplotlib-devel
>>
>>             _______________________________________________
>>             Matplotlib-devel mailing list
>>             [hidden email]
>>             <mailto:[hidden email]>
>>             https://mail.python.org/mailman/listinfo/matplotlib-devel
>>
>>         _______________________________________________
>>         Matplotlib-devel mailing list
>>         [hidden email] <mailto:[hidden email]>
>>         https://mail.python.org/mailman/listinfo/matplotlib-devel
>
>         --
>         Jody Klymak
>         http://web.uvic.ca/~jklymak/
>
>
>
>
>
>     _______________________________________________
>     Matplotlib-devel mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://mail.python.org/mailman/listinfo/matplotlib-devel
>
>
> _______________________________________________
> Matplotlib-devel mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/matplotlib-devel
>

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel


--
Thomas Caswell
[hidden email]
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to amend the PR merge rules

dstansby
I think it is good remember that (as far as I know) everyone submitting PRs and everyone reviewing them isn't paid and does so in their spare time; therefore is no obligation for anyone to review anything on any particular timescale. One thing that hasn't been discussed is how to increase the reviewing capacity of the community, which I guess would involve working out how to attract and retain more regular contributors.

re. the proposed changes, I'm +1 modulo objective criteria being agreed upon. Personally I agree with Nelle's list, with the addition "has 100% code coverage".

David



On Tue, 15 Jan 2019, 23:46 Nelle Varoquaux <[hidden email] wrote:
For the way GitHub sorts pull requests, it is indeed a problem, but there's a way to sort by "most recently updated." I sometimes use it to make sure that I review recently updated pull request that don't appear on the first page.

It seems to me that this "uncontroversial but uninteresting" rule is currently not based on objective criteria. I'm personally not in favor of removing two core contributors approval for anything that modifies the API.  We have had too many close calls on PR almost merged that had backward incompatible changes: I realize that some core developers believe that we are not moving fast enough, on a project with hundreds of thousands of users, changes to the API should not be considered lightly.

Here's an attempt at better defining an "uncontroversial but uninteresting" rule based on objective criteria:
- Has not been commented on by more than one core developer;
- Doesn't change the public API;
- Doesn't add or remove a functionality;
- Doesn't add or remove a dependency;
- Is fully tested;
- All parameters are fully documented;
- Doesn't fix a release-critical bug;
- Is less than XX lines of code. (Long PR are harder to review, and thus the likelihood of one core dev not noticing a problem is higher). Threshold needs to be defined.

To be honest, it also doesn't help that some people have 50+ pull requests open (Antony? :p). It also doesn't help that some PR contain the same code (13117, 13128) and are being hold off until the other one is merged (also, why is 13117 being hold off for a PR that has been opened afterwards?). IMO, if a PR can be merged, it should be merged. But we shouldn't make too many exceptions to the two core contributors rules. It's easier to have two core contributor review a pull request, than to try to fix API issues on code that has been merged and released. Having two pairs of eyes on a PR is also a good way to avoid having a bus factor of 1 on some areas of the library.


On Tue, 15 Jan 2019 at 15:19, Thomas Caswell <[hidden email]> wrote:
I think a distinction between "no one cares" and "no one has bandwidth" needs to be made.

If you look at the 'pulse' page, we merging 100+ PRs a month (https://github.com/matplotlib/matplotlib/pulse/monthly) so reviews are happening, the issue is just that some are falling through the cracks.

Tom

On Tue, Jan 15, 2019 at 4:45 PM Antony Lee <[hidden email]> wrote:
Eric,
Neither the ttc font, nor the pdf/ps common code PRs actually touch any "complex" point regarding fonts or pdf/ps.  I have described the ttc font in my previous message; the pdf/ps PR is really just, look, these two classes share 80 lines worth of code that's literally copy-pasted, let's just put that in a common parent class and inherit from it.
Conversely, one font-related PR for which I actually have low hopes of getting properly reviewed (except on an "sure, I trust you" basis) is https://github.com/matplotlib/matplotlib/pull/12928, which actually touches a rather complex point about font encodings (some of which I had to clarify on the FreeType mailing list).  (If anyone wants to have a look at it, though... :p)
Antony

On Tue, Jan 15, 2019 at 10:36 PM Eric Firing <[hidden email]> wrote:
Antony,

Your examples illustrate that the problem is often that we don't have
enough people who understand some areas, like fonts and pdf/ps file
formats, to get 2 thorough and knowledgeable reviews in a reasonable time.

I'm fine with your proposed rule.  It should help at least a little bit.

Eric


On 2019/01/15 9:11 AM, Antony Lee wrote:
> I'm obviously more aware of my own PRs, so here are a few I'd have put
> up for single-review-merge (note that it's quite possible that they'd
> have attracted more attention and gotten merged faster under that
> system; that's also the goal...).  Obviously other devs may think about
> other PRs that could have benefitted from the same process.
>
> - https://github.com/matplotlib/matplotlib/pull/9787 adds support for
> the ttc font format (requested since 2014) for png/svg output (that's
> provided basically for free by FreeType) but not for pdf/ps (that would
> require more difficult changes).  The entire PR comes down to:
>    * adding "ttc" as an extension that should be treated like "ttf"/"otf",
>    * adding error handling to pdf/ps to signal these formats are not
> supported there,
>    * tests.
>    The PR sat open for 10 months before a first positive review (because
> no one cares about fonts, I guess) and took another 3 months to attract
> a second positive review.
>
> - https://github.com/matplotlib/matplotlib/pull/12472 does a fairly
> trivial fix to fontList.json to make it reusable for Matplotlibs
> installed in different venvs, got a positive review in one day and
> waited another 2.5 months for a second positive review.
>
> - https://github.com/matplotlib/matplotlib/pull/9867 deduplicates
> completely duplicated code between the pdf and ps backends, mostly for
> maintainability's sake; it took 7.5 months (and 5 rebases due to
> conflicts) to get a first positive review and another 6 (and around as
> many rebases) to get a second one.
>
> -----
>
> As for the specifics (as I see it):
> - Someone opens a PR, it gets a positive review, no negative review, and
> no activity occurs for two weeks (significant activity -- excluding
> trivial chat on the thread).
> - Sponsor (first reviewer, or author if a committer) *pings* all devs by
> mentioning @matplotlib/developers on the PR thread with the intent to
> single-review-merge, and labels the PR accordingly.
> - The PR can still be reviewed/merged/rejected by the normal review
> process.  However, if no one explicitly opposes the merge within two
> weeks, anyone can merge it at that point (including by self-merge).
>
> I would not exclude all API changes from the process.  For example,
> https://github.com/matplotlib/matplotlib/pull/13173/commits/e90d264f3e5a263ef57243a2af86b46ab74ccc16
> deprecates (and prepares for deletion) two parameters to imshow() that
> have had no effect since 2006.  Let's pretend for a second this commit
> was a single independent PR (and not an example to showcase the
> parameter-deprecating decorator...); if it started being forgotten I
> would have proposed it for single-review-merge (again, note that if
> you're worried about the change, you don't need to reject the PR; you
> can just say, I think this API change needs to be approved by a second
> reviewer and that'll block the single-review-merge just as well).
>
> Antony
>
> On Tue, Jan 15, 2019 at 7:46 PM Nelle Varoquaux
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     The proposed scheme by Paul doesn't seem reasonable to me. Core
>     contributor A or committers needs to actively reach out to other
>     core contributors: labeling is not enough IMO.
>
>
>     On Tue, 15 Jan 2019 at 10:37, Jody Klymak <[hidden email]
>     <mailto:[hidden email]>> wrote:
>
>
>
>>         On 15 Jan 2019, at 10:20, Paul Hobson <[hidden email]
>>         <mailto:[hidden email]>> wrote:
>>
>>         I think this would be a good policy as well. Can I get some
>>         clarification on the flow? The way I understand it:
>>
>>         1) Someone (committer, contributor, or first-timer) opens a
>>         simple PR
>>         2) Committer A reviews it, adds a "single-review-merge" label,
>>         then digitally walks away
>>         3) No less than two weeks later, Committer B sees the PR,
>>         notices the label, opens it up, and merges without reviewing.
>
>         If there is a Committer B, I think they should at least look at
>         the PR to make sure it doesn’t have any hidden API changes, or
>         introduce anything hostile.
>
>         The point here is that getting the third person to look at a PR
>         is proving quite difficult, and if the rest of the folks with
>         the commit bit haven’t looked at a PR for a month, even after
>         being warned, then silence indicates consent.
>
>         Note that only folks with the commit bit can label PRs, so
>         anyone flagging a PR like this is implicitly trusted to not be
>         wreaking havoc, and not doing this for major changes.  Since we
>         all curate our own PRs, I rather expect that the PR contributor
>         will often be the person who adds the flag.  Whether we want to
>         allow a self-merge at that point is up for debate, but if not,
>         then folks need to get in the habit of looking at old flagged
>         PRs and merging them.
>
>         I think a huge problem we have is the LIFO default sort of the
>         github PR queue, and that any PR not on the first page might as
>         well be closed for all the attention it will get without
>         constant nagging.
>
>         Cheers,   Jody
>
>>         Does that capture a successful "single-review-merge" lifecycle?
>>         -paul
>>
>>         On Tue, Jan 15, 2019 at 9:57 AM Nelle Varoquaux
>>         <[hidden email] <mailto:[hidden email]>>
>>         wrote:
>>
>>             Hello,
>>
>>             Overall, I think this is a good idea, but I would like
>>             specifics on what "uncontroversial but uninteresting" PR
>>             mean. IMO, that should exclude any PR with API changes.
>>
>>             Cheers,
>>             N
>>
>>             On Tue, 15 Jan 2019 at 06:00, Antony Lee
>>             <[hidden email] <mailto:[hidden email]>> wrote:
>>
>>                 Hi all,
>>
>>                 During the weekly dev call, I proposed to amend the PR
>>                 merge rules, following an initial comment on Github
>>                 [https://github.com/matplotlib/matplotlib/pull/13173#issuecomment-453921220].
>>                 There was general agreement among the devs present
>>                 (Tom, Hannah, Jody, and myself), so I'm putting it
>>                 here for discussion.  The objective of this change is
>>                 to prevent "uncontroversial but uninteresting" PRs
>>                 from falling into oblivion, and try to decrease the
>>                 size of the open PR stack.
>>
>>                 The current rule is that a PR needs positive reviews
>>                 (from two committers, and excluding the author if a
>>                 committer) to be merged, except that doc-only PRs
>>                 (docstrings, rst) only needs a single positive review.
>>
>>                 I am proposing that, if a (non-doc) PR already has one
>>                 positive review, but no activity on that PR has
>>                 occurred for two weeks (exact time interval up to
>>                 bikeshedding) [Jody suggests: and the PR has 100% code
>>                 coverage], then a committer (either the first
>>                 reviewer, or the author if a committer) can suggest
>>                 that it be merged on the basis of that single review.
>>                 To do so, the "sponsor" should ping all developers
>>                 (@matplotlib/developers) on that issue indicating that
>>                 intent, and add a "single-review-merge" label on the
>>                 PR (so that these PRs can easily be found).
>>                 Committers are encouraged to review the PR to accept
>>                 and merge it or reject it or request changes on it;
>>                 but they can also just indicate that they consider the
>>                 PR sufficiently complex that a proper second review is
>>                 needed before merging, or request an extension, etc.
>>                 To do so they should still leave a "reject" review,
>>                 even if just saying "objecting to single-review
>>                 merged; anyone can dismiss after a second review".
>>                 However, if within another two weeks, no committer
>>                 voiced any objection (explicitly, i.e. by rejecting),
>>                 then the PR can indeed be merged on the basis of that
>>                 single review.
>>
>>                 To avoid overwhelming the system, any committer can
>>                 only "sponsor" a single PR at a time.
>>
>>                 Thoughts?
>>
>>                 Antony
>>                 _______________________________________________
>>                 Matplotlib-devel mailing list
>>                 [hidden email]
>>                 <mailto:[hidden email]>
>>                 https://mail.python.org/mailman/listinfo/matplotlib-devel
>>
>>             _______________________________________________
>>             Matplotlib-devel mailing list
>>             [hidden email]
>>             <mailto:[hidden email]>
>>             https://mail.python.org/mailman/listinfo/matplotlib-devel
>>
>>         _______________________________________________
>>         Matplotlib-devel mailing list
>>         [hidden email] <mailto:[hidden email]>
>>         https://mail.python.org/mailman/listinfo/matplotlib-devel
>
>         --
>         Jody Klymak
>         http://web.uvic.ca/~jklymak/
>
>
>
>
>
>     _______________________________________________
>     Matplotlib-devel mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://mail.python.org/mailman/listinfo/matplotlib-devel
>
>
> _______________________________________________
> Matplotlib-devel mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/matplotlib-devel
>

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel


--
Thomas Caswell
[hidden email]
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to amend the PR merge rules

Antony Lee-3
Replying to Nelle's points first:
 
It seems to me that this "uncontroversial but uninteresting" rule is currently not based on objective criteria. I'm personally not in favor of removing two core contributors approval for anything that modifies the API.  We have had too many close calls on PR almost merged that had backward incompatible changes: I realize that some core developers believe that we are not moving fast enough, on a project with hundreds of thousands of users, changes to the API should not be considered lightly. 

To be clear (and even though it is probably common knowledge that I would like to evolve the API faster), there is no "secret agenda" of using this to sneak API breaks through the system.
 
Here's an attempt at better defining an "uncontroversial but uninteresting" rule based on objective criteria:
- Has not been commented on by more than one core developer;

Sounds reasonable (with a reasonably flexible interpretation -- similarly to the "no activity since two weeks, excluding trivial chat"), basically "no other core dev expressed interest in reviewing it".
 
- Doesn't change the public API;
- Doesn't add or remove a functionality;
- Doesn't add or remove a dependency;

I expect all changes of API to go through the normal route of deprecation and API change note, which even the first reviewer should make sure are present before sponsoring the PR for single-review-merge (as an aside, I guess the first reviewer could even say, "approving, but I don't want to take the responsibility for being the single reviewer"...).  So now you have a two week period where any other core dev passing by can just see whether there is an API change note and make their own judgment, including deciding to block the PR if necessary, based on that change note.
I guess it is reasonable to say "API deprecations or changes and new functionality should be clearly documented" to help that "skim-through" review.
As for *adding* functionality, I guess you would have rejected the ttc-font PR on that premise?  (Actually I now see that the additional functionality was actually not documented :))  Note that this is not some obscure edge case that no one cares about; it's a feature that has been requested for years on the tracker.

- Is fully tested;
 
Also proposed by Jody and now by David, sounds good to me.
 
- All parameters are fully documented;

(sounds contradictory with "doesn't add a functionality"... if there is no new functionality it's unlikely there's a new parameter)
 
- Doesn't fix a release-critical bug;

These tend to all be merged just before the next release is cut anyways (review dynamics are a bit different in these periods), so I wouldn't worry about it, but sure.
 
- Is less than XX lines of code. (Long PR are harder to review, and thus the likelihood of one core dev not noticing a problem is higher). Threshold needs to be defined.

I don't actually think this is a good rule; the pdf/ps merge PR is +138 lines, -182 lines but as explained above the change is fairly trivial; obviously I could have split it into smaller chunks but if anything that would have made it harder to review.
Conversely (to use an example I have already raised elsewhere) https://github.com/matplotlib/matplotlib/pull/11530 (making sure that LocationEvent.x and LocationEvent.y are integers (as they refer to pixel locations) rather than floats) is 2 lines of changes in the codebase and 20 lines of test, was reviewed by 2.5 core devs (I left a comment in passing) and we all missed that there was "problem" with it (it broke mplcursors and some other pieces of code I have that assume that it is possible to synthetize LocationEvents pointing to arbitrary places in the canvas).  (Not to derail this discussion into how to evaluate API breaks once again... but I think these kinds of changes are much more insidious than, say, removing a function, as the latter would just trigger an AttributeError that is easily noticed and at worst you just go back to the previous Matplotlib release, whereas the LocationEvent change was only caught because it was covered by the mplcursors test suite; otherwise I'd have been silently generating wrong results due to it...)
 
To be honest, it also doesn't help that some people have 50+ pull requests open (Antony? :p).

I guess if the way to fix Matplotlib is to not contribute to Matplotlib anymore, we have a bigger problem.  In fact, I have some pretty big patches fixing long-standing issues (e.g. for #8550 I have a pure-Python reimplementation of the fontconfig font weight deduction algorithm, translated from fontconfig's C implementation) that I don't even bother turning into PRs, as I know they have no chance of being reviewed and merged under the current system at least, because no one would care (see also the bus-factor point below).
 
It also doesn't help that some PR contain the same code (13117, 13128) and are being hold off until the other one is merged (also, why is 13117 being hold off for a PR that has been opened afterwards?). IMO, if a PR can be merged, it should be merged.

That's discussed in https://github.com/matplotlib/matplotlib/pull/13117#discussion_r245503693 (FTR I actually agreed on merging 13117 directly, but apparently that's an API break...)
 
But we shouldn't make too many exceptions to the two core contributors rules. It's easier to have two core contributor review a pull request, than to try to fix API issues on code that has been merged and released. Having two pairs of eyes on a PR is also a good way to avoid having a bus factor of 1 on some areas of the library.

There are already huge and central chunks of the library (Agg internals, PDF/PS) that have a bus factor close to zero.  If anything, I hope that this "drive-by review" system will encourage reviewers to start looking at places they are less familiar with ("oh yes, I guess this change makes sense" -- see Eric's reply above) and decrease that.

And then to David's (the code-coverage point is already covered above):

I think it is good remember that (as far as I know) everyone submitting PRs and everyone reviewing them isn't paid and does so in their spare time; therefore is no obligation for anyone to review anything on any particular timescale. One thing that hasn't been discussed is how to increase the reviewing capacity of the community, which I guess would involve working out how to attract and retain more regular contributors.

Given the premises "no one is paid to review PRs" and "the reviewing capacity is too small" (which I agree with), I think there are two reasonable solutions:
- Make PRs easier to merge (this proposal),
- Increasing to number of core devs (your proposal, but it's not clear to me whether this number can increase significantly in a short period of time...).

Antony

On Wed, Jan 16, 2019 at 10:34 AM David Stansby <[hidden email]> wrote:
I think it is good remember that (as far as I know) everyone submitting PRs and everyone reviewing them isn't paid and does so in their spare time; therefore is no obligation for anyone to review anything on any particular timescale. One thing that hasn't been discussed is how to increase the reviewing capacity of the community, which I guess would involve working out how to attract and retain more regular contributors.

re. the proposed changes, I'm +1 modulo objective criteria being agreed upon. Personally I agree with Nelle's list, with the addition "has 100% code coverage".

David



On Tue, 15 Jan 2019, 23:46 Nelle Varoquaux <[hidden email] wrote:
For the way GitHub sorts pull requests, it is indeed a problem, but there's a way to sort by "most recently updated." I sometimes use it to make sure that I review recently updated pull request that don't appear on the first page.

It seems to me that this "uncontroversial but uninteresting" rule is currently not based on objective criteria. I'm personally not in favor of removing two core contributors approval for anything that modifies the API.  We have had too many close calls on PR almost merged that had backward incompatible changes: I realize that some core developers believe that we are not moving fast enough, on a project with hundreds of thousands of users, changes to the API should not be considered lightly.

Here's an attempt at better defining an "uncontroversial but uninteresting" rule based on objective criteria:
- Has not been commented on by more than one core developer;
- Doesn't change the public API;
- Doesn't add or remove a functionality;
- Doesn't add or remove a dependency;
- Is fully tested;
- All parameters are fully documented;
- Doesn't fix a release-critical bug;
- Is less than XX lines of code. (Long PR are harder to review, and thus the likelihood of one core dev not noticing a problem is higher). Threshold needs to be defined.

To be honest, it also doesn't help that some people have 50+ pull requests open (Antony? :p). It also doesn't help that some PR contain the same code (13117, 13128) and are being hold off until the other one is merged (also, why is 13117 being hold off for a PR that has been opened afterwards?). IMO, if a PR can be merged, it should be merged. But we shouldn't make too many exceptions to the two core contributors rules. It's easier to have two core contributor review a pull request, than to try to fix API issues on code that has been merged and released. Having two pairs of eyes on a PR is also a good way to avoid having a bus factor of 1 on some areas of the library.


On Tue, 15 Jan 2019 at 15:19, Thomas Caswell <[hidden email]> wrote:
I think a distinction between "no one cares" and "no one has bandwidth" needs to be made.

If you look at the 'pulse' page, we merging 100+ PRs a month (https://github.com/matplotlib/matplotlib/pulse/monthly) so reviews are happening, the issue is just that some are falling through the cracks.

Tom

On Tue, Jan 15, 2019 at 4:45 PM Antony Lee <[hidden email]> wrote:
Eric,
Neither the ttc font, nor the pdf/ps common code PRs actually touch any "complex" point regarding fonts or pdf/ps.  I have described the ttc font in my previous message; the pdf/ps PR is really just, look, these two classes share 80 lines worth of code that's literally copy-pasted, let's just put that in a common parent class and inherit from it.
Conversely, one font-related PR for which I actually have low hopes of getting properly reviewed (except on an "sure, I trust you" basis) is https://github.com/matplotlib/matplotlib/pull/12928, which actually touches a rather complex point about font encodings (some of which I had to clarify on the FreeType mailing list).  (If anyone wants to have a look at it, though... :p)
Antony

On Tue, Jan 15, 2019 at 10:36 PM Eric Firing <[hidden email]> wrote:
Antony,

Your examples illustrate that the problem is often that we don't have
enough people who understand some areas, like fonts and pdf/ps file
formats, to get 2 thorough and knowledgeable reviews in a reasonable time.

I'm fine with your proposed rule.  It should help at least a little bit.

Eric


On 2019/01/15 9:11 AM, Antony Lee wrote:
> I'm obviously more aware of my own PRs, so here are a few I'd have put
> up for single-review-merge (note that it's quite possible that they'd
> have attracted more attention and gotten merged faster under that
> system; that's also the goal...).  Obviously other devs may think about
> other PRs that could have benefitted from the same process.
>
> - https://github.com/matplotlib/matplotlib/pull/9787 adds support for
> the ttc font format (requested since 2014) for png/svg output (that's
> provided basically for free by FreeType) but not for pdf/ps (that would
> require more difficult changes).  The entire PR comes down to:
>    * adding "ttc" as an extension that should be treated like "ttf"/"otf",
>    * adding error handling to pdf/ps to signal these formats are not
> supported there,
>    * tests.
>    The PR sat open for 10 months before a first positive review (because
> no one cares about fonts, I guess) and took another 3 months to attract
> a second positive review.
>
> - https://github.com/matplotlib/matplotlib/pull/12472 does a fairly
> trivial fix to fontList.json to make it reusable for Matplotlibs
> installed in different venvs, got a positive review in one day and
> waited another 2.5 months for a second positive review.
>
> - https://github.com/matplotlib/matplotlib/pull/9867 deduplicates
> completely duplicated code between the pdf and ps backends, mostly for
> maintainability's sake; it took 7.5 months (and 5 rebases due to
> conflicts) to get a first positive review and another 6 (and around as
> many rebases) to get a second one.
>
> -----
>
> As for the specifics (as I see it):
> - Someone opens a PR, it gets a positive review, no negative review, and
> no activity occurs for two weeks (significant activity -- excluding
> trivial chat on the thread).
> - Sponsor (first reviewer, or author if a committer) *pings* all devs by
> mentioning @matplotlib/developers on the PR thread with the intent to
> single-review-merge, and labels the PR accordingly.
> - The PR can still be reviewed/merged/rejected by the normal review
> process.  However, if no one explicitly opposes the merge within two
> weeks, anyone can merge it at that point (including by self-merge).
>
> I would not exclude all API changes from the process.  For example,
> https://github.com/matplotlib/matplotlib/pull/13173/commits/e90d264f3e5a263ef57243a2af86b46ab74ccc16
> deprecates (and prepares for deletion) two parameters to imshow() that
> have had no effect since 2006.  Let's pretend for a second this commit
> was a single independent PR (and not an example to showcase the
> parameter-deprecating decorator...); if it started being forgotten I
> would have proposed it for single-review-merge (again, note that if
> you're worried about the change, you don't need to reject the PR; you
> can just say, I think this API change needs to be approved by a second
> reviewer and that'll block the single-review-merge just as well).
>
> Antony
>
> On Tue, Jan 15, 2019 at 7:46 PM Nelle Varoquaux
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     The proposed scheme by Paul doesn't seem reasonable to me. Core
>     contributor A or committers needs to actively reach out to other
>     core contributors: labeling is not enough IMO.
>
>
>     On Tue, 15 Jan 2019 at 10:37, Jody Klymak <[hidden email]
>     <mailto:[hidden email]>> wrote:
>
>
>
>>         On 15 Jan 2019, at 10:20, Paul Hobson <[hidden email]
>>         <mailto:[hidden email]>> wrote:
>>
>>         I think this would be a good policy as well. Can I get some
>>         clarification on the flow? The way I understand it:
>>
>>         1) Someone (committer, contributor, or first-timer) opens a
>>         simple PR
>>         2) Committer A reviews it, adds a "single-review-merge" label,
>>         then digitally walks away
>>         3) No less than two weeks later, Committer B sees the PR,
>>         notices the label, opens it up, and merges without reviewing.
>
>         If there is a Committer B, I think they should at least look at
>         the PR to make sure it doesn’t have any hidden API changes, or
>         introduce anything hostile.
>
>         The point here is that getting the third person to look at a PR
>         is proving quite difficult, and if the rest of the folks with
>         the commit bit haven’t looked at a PR for a month, even after
>         being warned, then silence indicates consent.
>
>         Note that only folks with the commit bit can label PRs, so
>         anyone flagging a PR like this is implicitly trusted to not be
>         wreaking havoc, and not doing this for major changes.  Since we
>         all curate our own PRs, I rather expect that the PR contributor
>         will often be the person who adds the flag.  Whether we want to
>         allow a self-merge at that point is up for debate, but if not,
>         then folks need to get in the habit of looking at old flagged
>         PRs and merging them.
>
>         I think a huge problem we have is the LIFO default sort of the
>         github PR queue, and that any PR not on the first page might as
>         well be closed for all the attention it will get without
>         constant nagging.
>
>         Cheers,   Jody
>
>>         Does that capture a successful "single-review-merge" lifecycle?
>>         -paul
>>
>>         On Tue, Jan 15, 2019 at 9:57 AM Nelle Varoquaux
>>         <[hidden email] <mailto:[hidden email]>>
>>         wrote:
>>
>>             Hello,
>>
>>             Overall, I think this is a good idea, but I would like
>>             specifics on what "uncontroversial but uninteresting" PR
>>             mean. IMO, that should exclude any PR with API changes.
>>
>>             Cheers,
>>             N
>>
>>             On Tue, 15 Jan 2019 at 06:00, Antony Lee
>>             <[hidden email] <mailto:[hidden email]>> wrote:
>>
>>                 Hi all,
>>
>>                 During the weekly dev call, I proposed to amend the PR
>>                 merge rules, following an initial comment on Github
>>                 [https://github.com/matplotlib/matplotlib/pull/13173#issuecomment-453921220].
>>                 There was general agreement among the devs present
>>                 (Tom, Hannah, Jody, and myself), so I'm putting it
>>                 here for discussion.  The objective of this change is
>>                 to prevent "uncontroversial but uninteresting" PRs
>>                 from falling into oblivion, and try to decrease the
>>                 size of the open PR stack.
>>
>>                 The current rule is that a PR needs positive reviews
>>                 (from two committers, and excluding the author if a
>>                 committer) to be merged, except that doc-only PRs
>>                 (docstrings, rst) only needs a single positive review.
>>
>>                 I am proposing that, if a (non-doc) PR already has one
>>                 positive review, but no activity on that PR has
>>                 occurred for two weeks (exact time interval up to
>>                 bikeshedding) [Jody suggests: and the PR has 100% code
>>                 coverage], then a committer (either the first
>>                 reviewer, or the author if a committer) can suggest
>>                 that it be merged on the basis of that single review.
>>                 To do so, the "sponsor" should ping all developers
>>                 (@matplotlib/developers) on that issue indicating that
>>                 intent, and add a "single-review-merge" label on the
>>                 PR (so that these PRs can easily be found).
>>                 Committers are encouraged to review the PR to accept
>>                 and merge it or reject it or request changes on it;
>>                 but they can also just indicate that they consider the
>>                 PR sufficiently complex that a proper second review is
>>                 needed before merging, or request an extension, etc.
>>                 To do so they should still leave a "reject" review,
>>                 even if just saying "objecting to single-review
>>                 merged; anyone can dismiss after a second review".
>>                 However, if within another two weeks, no committer
>>                 voiced any objection (explicitly, i.e. by rejecting),
>>                 then the PR can indeed be merged on the basis of that
>>                 single review.
>>
>>                 To avoid overwhelming the system, any committer can
>>                 only "sponsor" a single PR at a time.
>>
>>                 Thoughts?
>>
>>                 Antony
>>                 _______________________________________________
>>                 Matplotlib-devel mailing list
>>                 [hidden email]
>>                 <mailto:[hidden email]>
>>                 https://mail.python.org/mailman/listinfo/matplotlib-devel
>>
>>             _______________________________________________
>>             Matplotlib-devel mailing list
>>             [hidden email]
>>             <mailto:[hidden email]>
>>             https://mail.python.org/mailman/listinfo/matplotlib-devel
>>
>>         _______________________________________________
>>         Matplotlib-devel mailing list
>>         [hidden email] <mailto:[hidden email]>
>>         https://mail.python.org/mailman/listinfo/matplotlib-devel
>
>         --
>         Jody Klymak
>         http://web.uvic.ca/~jklymak/
>
>
>
>
>
>     _______________________________________________
>     Matplotlib-devel mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://mail.python.org/mailman/listinfo/matplotlib-devel
>
>
> _______________________________________________
> Matplotlib-devel mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/matplotlib-devel
>

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel


--
Thomas Caswell
[hidden email]
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to amend the PR merge rules

tcaswell
I want to push back strongly on the notion that "did not review --> does not care".  As as already been discussed above there are many other reasons someone does not comment (for example "I _do_ care about this, but I {do not have time, am too tired and cranky, ...} to properly review this now").

It seems to me that there are two classes of PRs that are getting grouped together here: small PRs that just fall through the cracks and big PRs that are hard to review, let's stay focused on the small PRs.

One habit related thing that may help is if we all bookmark:

as where to start when we sit down to do some Matplotlib work.  I would also remind everyone that if you have a commit bit, then we trust you to use it!

As we write down the list of things that need to be true for single-review merging, it seems the effort to determine that a given PR checks (or does not) all of the boxes is equal to the effort to just review the PR.  This make me think that the biggest benefit of this change would be to get more eyes on PRs than any policy change.

To a fair degree this comes down to a project management / resource allocation problem (which are hard even in situations where everyone is paid!) 

Tom

On Wed, Jan 16, 2019 at 6:11 AM Antony Lee <[hidden email]> wrote:
Replying to Nelle's points first:
 
It seems to me that this "uncontroversial but uninteresting" rule is currently not based on objective criteria. I'm personally not in favor of removing two core contributors approval for anything that modifies the API.  We have had too many close calls on PR almost merged that had backward incompatible changes: I realize that some core developers believe that we are not moving fast enough, on a project with hundreds of thousands of users, changes to the API should not be considered lightly. 

To be clear (and even though it is probably common knowledge that I would like to evolve the API faster), there is no "secret agenda" of using this to sneak API breaks through the system.
 
Here's an attempt at better defining an "uncontroversial but uninteresting" rule based on objective criteria:
- Has not been commented on by more than one core developer;

Sounds reasonable (with a reasonably flexible interpretation -- similarly to the "no activity since two weeks, excluding trivial chat"), basically "no other core dev expressed interest in reviewing it".
 
- Doesn't change the public API;
- Doesn't add or remove a functionality;
- Doesn't add or remove a dependency;

I expect all changes of API to go through the normal route of deprecation and API change note, which even the first reviewer should make sure are present before sponsoring the PR for single-review-merge (as an aside, I guess the first reviewer could even say, "approving, but I don't want to take the responsibility for being the single reviewer"...).  So now you have a two week period where any other core dev passing by can just see whether there is an API change note and make their own judgment, including deciding to block the PR if necessary, based on that change note.
I guess it is reasonable to say "API deprecations or changes and new functionality should be clearly documented" to help that "skim-through" review.
As for *adding* functionality, I guess you would have rejected the ttc-font PR on that premise?  (Actually I now see that the additional functionality was actually not documented :))  Note that this is not some obscure edge case that no one cares about; it's a feature that has been requested for years on the tracker.

- Is fully tested;
 
Also proposed by Jody and now by David, sounds good to me.
 
- All parameters are fully documented;

(sounds contradictory with "doesn't add a functionality"... if there is no new functionality it's unlikely there's a new parameter)
 
- Doesn't fix a release-critical bug;

These tend to all be merged just before the next release is cut anyways (review dynamics are a bit different in these periods), so I wouldn't worry about it, but sure.
 
- Is less than XX lines of code. (Long PR are harder to review, and thus the likelihood of one core dev not noticing a problem is higher). Threshold needs to be defined.

I don't actually think this is a good rule; the pdf/ps merge PR is +138 lines, -182 lines but as explained above the change is fairly trivial; obviously I could have split it into smaller chunks but if anything that would have made it harder to review.
Conversely (to use an example I have already raised elsewhere) https://github.com/matplotlib/matplotlib/pull/11530 (making sure that LocationEvent.x and LocationEvent.y are integers (as they refer to pixel locations) rather than floats) is 2 lines of changes in the codebase and 20 lines of test, was reviewed by 2.5 core devs (I left a comment in passing) and we all missed that there was "problem" with it (it broke mplcursors and some other pieces of code I have that assume that it is possible to synthetize LocationEvents pointing to arbitrary places in the canvas).  (Not to derail this discussion into how to evaluate API breaks once again... but I think these kinds of changes are much more insidious than, say, removing a function, as the latter would just trigger an AttributeError that is easily noticed and at worst you just go back to the previous Matplotlib release, whereas the LocationEvent change was only caught because it was covered by the mplcursors test suite; otherwise I'd have been silently generating wrong results due to it...)
 
To be honest, it also doesn't help that some people have 50+ pull requests open (Antony? :p).

I guess if the way to fix Matplotlib is to not contribute to Matplotlib anymore, we have a bigger problem.  In fact, I have some pretty big patches fixing long-standing issues (e.g. for #8550 I have a pure-Python reimplementation of the fontconfig font weight deduction algorithm, translated from fontconfig's C implementation) that I don't even bother turning into PRs, as I know they have no chance of being reviewed and merged under the current system at least, because no one would care (see also the bus-factor point below).
 
It also doesn't help that some PR contain the same code (13117, 13128) and are being hold off until the other one is merged (also, why is 13117 being hold off for a PR that has been opened afterwards?). IMO, if a PR can be merged, it should be merged.

That's discussed in https://github.com/matplotlib/matplotlib/pull/13117#discussion_r245503693 (FTR I actually agreed on merging 13117 directly, but apparently that's an API break...)
 
But we shouldn't make too many exceptions to the two core contributors rules. It's easier to have two core contributor review a pull request, than to try to fix API issues on code that has been merged and released. Having two pairs of eyes on a PR is also a good way to avoid having a bus factor of 1 on some areas of the library.

There are already huge and central chunks of the library (Agg internals, PDF/PS) that have a bus factor close to zero.  If anything, I hope that this "drive-by review" system will encourage reviewers to start looking at places they are less familiar with ("oh yes, I guess this change makes sense" -- see Eric's reply above) and decrease that.

And then to David's (the code-coverage point is already covered above):

I think it is good remember that (as far as I know) everyone submitting PRs and everyone reviewing them isn't paid and does so in their spare time; therefore is no obligation for anyone to review anything on any particular timescale. One thing that hasn't been discussed is how to increase the reviewing capacity of the community, which I guess would involve working out how to attract and retain more regular contributors.

Given the premises "no one is paid to review PRs" and "the reviewing capacity is too small" (which I agree with), I think there are two reasonable solutions:
- Make PRs easier to merge (this proposal),
- Increasing to number of core devs (your proposal, but it's not clear to me whether this number can increase significantly in a short period of time...).

Antony

On Wed, Jan 16, 2019 at 10:34 AM David Stansby <[hidden email]> wrote:
I think it is good remember that (as far as I know) everyone submitting PRs and everyone reviewing them isn't paid and does so in their spare time; therefore is no obligation for anyone to review anything on any particular timescale. One thing that hasn't been discussed is how to increase the reviewing capacity of the community, which I guess would involve working out how to attract and retain more regular contributors.

re. the proposed changes, I'm +1 modulo objective criteria being agreed upon. Personally I agree with Nelle's list, with the addition "has 100% code coverage".

David



On Tue, 15 Jan 2019, 23:46 Nelle Varoquaux <[hidden email] wrote:
For the way GitHub sorts pull requests, it is indeed a problem, but there's a way to sort by "most recently updated." I sometimes use it to make sure that I review recently updated pull request that don't appear on the first page.

It seems to me that this "uncontroversial but uninteresting" rule is currently not based on objective criteria. I'm personally not in favor of removing two core contributors approval for anything that modifies the API.  We have had too many close calls on PR almost merged that had backward incompatible changes: I realize that some core developers believe that we are not moving fast enough, on a project with hundreds of thousands of users, changes to the API should not be considered lightly.

Here's an attempt at better defining an "uncontroversial but uninteresting" rule based on objective criteria:
- Has not been commented on by more than one core developer;
- Doesn't change the public API;
- Doesn't add or remove a functionality;
- Doesn't add or remove a dependency;
- Is fully tested;
- All parameters are fully documented;
- Doesn't fix a release-critical bug;
- Is less than XX lines of code. (Long PR are harder to review, and thus the likelihood of one core dev not noticing a problem is higher). Threshold needs to be defined.

To be honest, it also doesn't help that some people have 50+ pull requests open (Antony? :p). It also doesn't help that some PR contain the same code (13117, 13128) and are being hold off until the other one is merged (also, why is 13117 being hold off for a PR that has been opened afterwards?). IMO, if a PR can be merged, it should be merged. But we shouldn't make too many exceptions to the two core contributors rules. It's easier to have two core contributor review a pull request, than to try to fix API issues on code that has been merged and released. Having two pairs of eyes on a PR is also a good way to avoid having a bus factor of 1 on some areas of the library.


On Tue, 15 Jan 2019 at 15:19, Thomas Caswell <[hidden email]> wrote:
I think a distinction between "no one cares" and "no one has bandwidth" needs to be made.

If you look at the 'pulse' page, we merging 100+ PRs a month (https://github.com/matplotlib/matplotlib/pulse/monthly) so reviews are happening, the issue is just that some are falling through the cracks.

Tom

On Tue, Jan 15, 2019 at 4:45 PM Antony Lee <[hidden email]> wrote:
Eric,
Neither the ttc font, nor the pdf/ps common code PRs actually touch any "complex" point regarding fonts or pdf/ps.  I have described the ttc font in my previous message; the pdf/ps PR is really just, look, these two classes share 80 lines worth of code that's literally copy-pasted, let's just put that in a common parent class and inherit from it.
Conversely, one font-related PR for which I actually have low hopes of getting properly reviewed (except on an "sure, I trust you" basis) is https://github.com/matplotlib/matplotlib/pull/12928, which actually touches a rather complex point about font encodings (some of which I had to clarify on the FreeType mailing list).  (If anyone wants to have a look at it, though... :p)
Antony

On Tue, Jan 15, 2019 at 10:36 PM Eric Firing <[hidden email]> wrote:
Antony,

Your examples illustrate that the problem is often that we don't have
enough people who understand some areas, like fonts and pdf/ps file
formats, to get 2 thorough and knowledgeable reviews in a reasonable time.

I'm fine with your proposed rule.  It should help at least a little bit.

Eric


On 2019/01/15 9:11 AM, Antony Lee wrote:
> I'm obviously more aware of my own PRs, so here are a few I'd have put
> up for single-review-merge (note that it's quite possible that they'd
> have attracted more attention and gotten merged faster under that
> system; that's also the goal...).  Obviously other devs may think about
> other PRs that could have benefitted from the same process.
>
> - https://github.com/matplotlib/matplotlib/pull/9787 adds support for
> the ttc font format (requested since 2014) for png/svg output (that's
> provided basically for free by FreeType) but not for pdf/ps (that would
> require more difficult changes).  The entire PR comes down to:
>    * adding "ttc" as an extension that should be treated like "ttf"/"otf",
>    * adding error handling to pdf/ps to signal these formats are not
> supported there,
>    * tests.
>    The PR sat open for 10 months before a first positive review (because
> no one cares about fonts, I guess) and took another 3 months to attract
> a second positive review.
>
> - https://github.com/matplotlib/matplotlib/pull/12472 does a fairly
> trivial fix to fontList.json to make it reusable for Matplotlibs
> installed in different venvs, got a positive review in one day and
> waited another 2.5 months for a second positive review.
>
> - https://github.com/matplotlib/matplotlib/pull/9867 deduplicates
> completely duplicated code between the pdf and ps backends, mostly for
> maintainability's sake; it took 7.5 months (and 5 rebases due to
> conflicts) to get a first positive review and another 6 (and around as
> many rebases) to get a second one.
>
> -----
>
> As for the specifics (as I see it):
> - Someone opens a PR, it gets a positive review, no negative review, and
> no activity occurs for two weeks (significant activity -- excluding
> trivial chat on the thread).
> - Sponsor (first reviewer, or author if a committer) *pings* all devs by
> mentioning @matplotlib/developers on the PR thread with the intent to
> single-review-merge, and labels the PR accordingly.
> - The PR can still be reviewed/merged/rejected by the normal review
> process.  However, if no one explicitly opposes the merge within two
> weeks, anyone can merge it at that point (including by self-merge).
>
> I would not exclude all API changes from the process.  For example,
> https://github.com/matplotlib/matplotlib/pull/13173/commits/e90d264f3e5a263ef57243a2af86b46ab74ccc16
> deprecates (and prepares for deletion) two parameters to imshow() that
> have had no effect since 2006.  Let's pretend for a second this commit
> was a single independent PR (and not an example to showcase the
> parameter-deprecating decorator...); if it started being forgotten I
> would have proposed it for single-review-merge (again, note that if
> you're worried about the change, you don't need to reject the PR; you
> can just say, I think this API change needs to be approved by a second
> reviewer and that'll block the single-review-merge just as well).
>
> Antony
>
> On Tue, Jan 15, 2019 at 7:46 PM Nelle Varoquaux
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     The proposed scheme by Paul doesn't seem reasonable to me. Core
>     contributor A or committers needs to actively reach out to other
>     core contributors: labeling is not enough IMO.
>
>
>     On Tue, 15 Jan 2019 at 10:37, Jody Klymak <[hidden email]
>     <mailto:[hidden email]>> wrote:
>
>
>
>>         On 15 Jan 2019, at 10:20, Paul Hobson <[hidden email]
>>         <mailto:[hidden email]>> wrote:
>>
>>         I think this would be a good policy as well. Can I get some
>>         clarification on the flow? The way I understand it:
>>
>>         1) Someone (committer, contributor, or first-timer) opens a
>>         simple PR
>>         2) Committer A reviews it, adds a "single-review-merge" label,
>>         then digitally walks away
>>         3) No less than two weeks later, Committer B sees the PR,
>>         notices the label, opens it up, and merges without reviewing.
>
>         If there is a Committer B, I think they should at least look at
>         the PR to make sure it doesn’t have any hidden API changes, or
>         introduce anything hostile.
>
>         The point here is that getting the third person to look at a PR
>         is proving quite difficult, and if the rest of the folks with
>         the commit bit haven’t looked at a PR for a month, even after
>         being warned, then silence indicates consent.
>
>         Note that only folks with the commit bit can label PRs, so
>         anyone flagging a PR like this is implicitly trusted to not be
>         wreaking havoc, and not doing this for major changes.  Since we
>         all curate our own PRs, I rather expect that the PR contributor
>         will often be the person who adds the flag.  Whether we want to
>         allow a self-merge at that point is up for debate, but if not,
>         then folks need to get in the habit of looking at old flagged
>         PRs and merging them.
>
>         I think a huge problem we have is the LIFO default sort of the
>         github PR queue, and that any PR not on the first page might as
>         well be closed for all the attention it will get without
>         constant nagging.
>
>         Cheers,   Jody
>
>>         Does that capture a successful "single-review-merge" lifecycle?
>>         -paul
>>
>>         On Tue, Jan 15, 2019 at 9:57 AM Nelle Varoquaux
>>         <[hidden email] <mailto:[hidden email]>>
>>         wrote:
>>
>>             Hello,
>>
>>             Overall, I think this is a good idea, but I would like
>>             specifics on what "uncontroversial but uninteresting" PR
>>             mean. IMO, that should exclude any PR with API changes.
>>
>>             Cheers,
>>             N
>>
>>             On Tue, 15 Jan 2019 at 06:00, Antony Lee
>>             <[hidden email] <mailto:[hidden email]>> wrote:
>>
>>                 Hi all,
>>
>>                 During the weekly dev call, I proposed to amend the PR
>>                 merge rules, following an initial comment on Github
>>                 [https://github.com/matplotlib/matplotlib/pull/13173#issuecomment-453921220].
>>                 There was general agreement among the devs present
>>                 (Tom, Hannah, Jody, and myself), so I'm putting it
>>                 here for discussion.  The objective of this change is
>>                 to prevent "uncontroversial but uninteresting" PRs
>>                 from falling into oblivion, and try to decrease the
>>                 size of the open PR stack.
>>
>>                 The current rule is that a PR needs positive reviews
>>                 (from two committers, and excluding the author if a
>>                 committer) to be merged, except that doc-only PRs
>>                 (docstrings, rst) only needs a single positive review.
>>
>>                 I am proposing that, if a (non-doc) PR already has one
>>                 positive review, but no activity on that PR has
>>                 occurred for two weeks (exact time interval up to
>>                 bikeshedding) [Jody suggests: and the PR has 100% code
>>                 coverage], then a committer (either the first
>>                 reviewer, or the author if a committer) can suggest
>>                 that it be merged on the basis of that single review.
>>                 To do so, the "sponsor" should ping all developers
>>                 (@matplotlib/developers) on that issue indicating that
>>                 intent, and add a "single-review-merge" label on the
>>                 PR (so that these PRs can easily be found).
>>                 Committers are encouraged to review the PR to accept
>>                 and merge it or reject it or request changes on it;
>>                 but they can also just indicate that they consider the
>>                 PR sufficiently complex that a proper second review is
>>                 needed before merging, or request an extension, etc.
>>                 To do so they should still leave a "reject" review,
>>                 even if just saying "objecting to single-review
>>                 merged; anyone can dismiss after a second review".
>>                 However, if within another two weeks, no committer
>>                 voiced any objection (explicitly, i.e. by rejecting),
>>                 then the PR can indeed be merged on the basis of that
>>                 single review.
>>
>>                 To avoid overwhelming the system, any committer can
>>                 only "sponsor" a single PR at a time.
>>
>>                 Thoughts?
>>
>>                 Antony
>>                 _______________________________________________
>>                 Matplotlib-devel mailing list
>>                 [hidden email]
>>                 <mailto:[hidden email]>
>>                 https://mail.python.org/mailman/listinfo/matplotlib-devel
>>
>>             _______________________________________________
>>             Matplotlib-devel mailing list
>>             [hidden email]
>>             <mailto:[hidden email]>
>>             https://mail.python.org/mailman/listinfo/matplotlib-devel
>>
>>         _______________________________________________
>>         Matplotlib-devel mailing list
>>         [hidden email] <mailto:[hidden email]>
>>         https://mail.python.org/mailman/listinfo/matplotlib-devel
>
>         --
>         Jody Klymak
>         http://web.uvic.ca/~jklymak/
>
>
>
>
>
>     _______________________________________________
>     Matplotlib-devel mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://mail.python.org/mailman/listinfo/matplotlib-devel
>
>
> _______________________________________________
> Matplotlib-devel mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/matplotlib-devel
>

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel


--
Thomas Caswell
[hidden email]
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel


--
Thomas Caswell
[hidden email]

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to amend the PR merge rules

Jody Klymak

There is a mathematical necessity here; if you have the commit bit and submit 50 PRs a month, you need to review 100 PRs a month, and merge or close 50 of those PRs a month.  If all the folks with commit bit do this, then obviously PRs will pile up.

Cheers,   Jody


On Jan 16, 2019, at  6:32 AM, Thomas Caswell <[hidden email]> wrote:

I want to push back strongly on the notion that "did not review --> does not care".  As as already been discussed above there are many other reasons someone does not comment (for example "I _do_ care about this, but I {do not have time, am too tired and cranky, ...} to properly review this now").

It seems to me that there are two classes of PRs that are getting grouped together here: small PRs that just fall through the cracks and big PRs that are hard to review, let's stay focused on the small PRs.

One habit related thing that may help is if we all bookmark:

as where to start when we sit down to do some Matplotlib work.  I would also remind everyone that if you have a commit bit, then we trust you to use it!

As we write down the list of things that need to be true for single-review merging, it seems the effort to determine that a given PR checks (or does not) all of the boxes is equal to the effort to just review the PR.  This make me think that the biggest benefit of this change would be to get more eyes on PRs than any policy change.

To a fair degree this comes down to a project management / resource allocation problem (which are hard even in situations where everyone is paid!) 

Tom

On Wed, Jan 16, 2019 at 6:11 AM Antony Lee <[hidden email]> wrote:
Replying to Nelle's points first:
 
It seems to me that this "uncontroversial but uninteresting" rule is currently not based on objective criteria. I'm personally not in favor of removing two core contributors approval for anything that modifies the API.  We have had too many close calls on PR almost merged that had backward incompatible changes: I realize that some core developers believe that we are not moving fast enough, on a project with hundreds of thousands of users, changes to the API should not be considered lightly. 

To be clear (and even though it is probably common knowledge that I would like to evolve the API faster), there is no "secret agenda" of using this to sneak API breaks through the system.
 
Here's an attempt at better defining an "uncontroversial but uninteresting" rule based on objective criteria:
- Has not been commented on by more than one core developer;

Sounds reasonable (with a reasonably flexible interpretation -- similarly to the "no activity since two weeks, excluding trivial chat"), basically "no other core dev expressed interest in reviewing it".
 
- Doesn't change the public API;
- Doesn't add or remove a functionality;
- Doesn't add or remove a dependency;

I expect all changes of API to go through the normal route of deprecation and API change note, which even the first reviewer should make sure are present before sponsoring the PR for single-review-merge (as an aside, I guess the first reviewer could even say, "approving, but I don't want to take the responsibility for being the single reviewer"...).  So now you have a two week period where any other core dev passing by can just see whether there is an API change note and make their own judgment, including deciding to block the PR if necessary, based on that change note.
I guess it is reasonable to say "API deprecations or changes and new functionality should be clearly documented" to help that "skim-through" review.
As for *adding* functionality, I guess you would have rejected the ttc-font PR on that premise?  (Actually I now see that the additional functionality was actually not documented :))  Note that this is not some obscure edge case that no one cares about; it's a feature that has been requested for years on the tracker.

- Is fully tested;
 
Also proposed by Jody and now by David, sounds good to me.
 
- All parameters are fully documented;

(sounds contradictory with "doesn't add a functionality"... if there is no new functionality it's unlikely there's a new parameter)
 
- Doesn't fix a release-critical bug;

These tend to all be merged just before the next release is cut anyways (review dynamics are a bit different in these periods), so I wouldn't worry about it, but sure.
 
- Is less than XX lines of code. (Long PR are harder to review, and thus the likelihood of one core dev not noticing a problem is higher). Threshold needs to be defined.

I don't actually think this is a good rule; the pdf/ps merge PR is +138 lines, -182 lines but as explained above the change is fairly trivial; obviously I could have split it into smaller chunks but if anything that would have made it harder to review.
Conversely (to use an example I have already raised elsewhere) https://github.com/matplotlib/matplotlib/pull/11530 (making sure that LocationEvent.x and LocationEvent.y are integers (as they refer to pixel locations) rather than floats) is 2 lines of changes in the codebase and 20 lines of test, was reviewed by 2.5 core devs (I left a comment in passing) and we all missed that there was "problem" with it (it broke mplcursors and some other pieces of code I have that assume that it is possible to synthetize LocationEvents pointing to arbitrary places in the canvas).  (Not to derail this discussion into how to evaluate API breaks once again... but I think these kinds of changes are much more insidious than, say, removing a function, as the latter would just trigger an AttributeError that is easily noticed and at worst you just go back to the previous Matplotlib release, whereas the LocationEvent change was only caught because it was covered by the mplcursors test suite; otherwise I'd have been silently generating wrong results due to it...)
 
To be honest, it also doesn't help that some people have 50+ pull requests open (Antony? :p).

I guess if the way to fix Matplotlib is to not contribute to Matplotlib anymore, we have a bigger problem.  In fact, I have some pretty big patches fixing long-standing issues (e.g. for #8550 I have a pure-Python reimplementation of the fontconfig font weight deduction algorithm, translated from fontconfig's C implementation) that I don't even bother turning into PRs, as I know they have no chance of being reviewed and merged under the current system at least, because no one would care (see also the bus-factor point below).
 
It also doesn't help that some PR contain the same code (13117, 13128) and are being hold off until the other one is merged (also, why is 13117 being hold off for a PR that has been opened afterwards?). IMO, if a PR can be merged, it should be merged.

That's discussed in https://github.com/matplotlib/matplotlib/pull/13117#discussion_r245503693 (FTR I actually agreed on merging 13117 directly, but apparently that's an API break...)
 
But we shouldn't make too many exceptions to the two core contributors rules. It's easier to have two core contributor review a pull request, than to try to fix API issues on code that has been merged and released. Having two pairs of eyes on a PR is also a good way to avoid having a bus factor of 1 on some areas of the library.

There are already huge and central chunks of the library (Agg internals, PDF/PS) that have a bus factor close to zero.  If anything, I hope that this "drive-by review" system will encourage reviewers to start looking at places they are less familiar with ("oh yes, I guess this change makes sense" -- see Eric's reply above) and decrease that.

And then to David's (the code-coverage point is already covered above):

I think it is good remember that (as far as I know) everyone submitting PRs and everyone reviewing them isn't paid and does so in their spare time; therefore is no obligation for anyone to review anything on any particular timescale. One thing that hasn't been discussed is how to increase the reviewing capacity of the community, which I guess would involve working out how to attract and retain more regular contributors.

Given the premises "no one is paid to review PRs" and "the reviewing capacity is too small" (which I agree with), I think there are two reasonable solutions:
- Make PRs easier to merge (this proposal),
- Increasing to number of core devs (your proposal, but it's not clear to me whether this number can increase significantly in a short period of time...).

Antony

On Wed, Jan 16, 2019 at 10:34 AM David Stansby <[hidden email]> wrote:
I think it is good remember that (as far as I know) everyone submitting PRs and everyone reviewing them isn't paid and does so in their spare time; therefore is no obligation for anyone to review anything on any particular timescale. One thing that hasn't been discussed is how to increase the reviewing capacity of the community, which I guess would involve working out how to attract and retain more regular contributors.

re. the proposed changes, I'm +1 modulo objective criteria being agreed upon. Personally I agree with Nelle's list, with the addition "has 100% code coverage".

David



On Tue, 15 Jan 2019, 23:46 Nelle Varoquaux <[hidden email] wrote:
For the way GitHub sorts pull requests, it is indeed a problem, but there's a way to sort by "most recently updated." I sometimes use it to make sure that I review recently updated pull request that don't appear on the first page.

It seems to me that this "uncontroversial but uninteresting" rule is currently not based on objective criteria. I'm personally not in favor of removing two core contributors approval for anything that modifies the API.  We have had too many close calls on PR almost merged that had backward incompatible changes: I realize that some core developers believe that we are not moving fast enough, on a project with hundreds of thousands of users, changes to the API should not be considered lightly. 

Here's an attempt at better defining an "uncontroversial but uninteresting" rule based on objective criteria:
- Has not been commented on by more than one core developer;
- Doesn't change the public API;
- Doesn't add or remove a functionality;
- Doesn't add or remove a dependency;
- Is fully tested;
- All parameters are fully documented;
- Doesn't fix a release-critical bug;
- Is less than XX lines of code. (Long PR are harder to review, and thus the likelihood of one core dev not noticing a problem is higher). Threshold needs to be defined.

To be honest, it also doesn't help that some people have 50+ pull requests open (Antony? :p). It also doesn't help that some PR contain the same code (13117, 13128) and are being hold off until the other one is merged (also, why is 13117 being hold off for a PR that has been opened afterwards?). IMO, if a PR can be merged, it should be merged. But we shouldn't make too many exceptions to the two core contributors rules. It's easier to have two core contributor review a pull request, than to try to fix API issues on code that has been merged and released. Having two pairs of eyes on a PR is also a good way to avoid having a bus factor of 1 on some areas of the library.


On Tue, 15 Jan 2019 at 15:19, Thomas Caswell <[hidden email]> wrote:
I think a distinction between "no one cares" and "no one has bandwidth" needs to be made.

If you look at the 'pulse' page, we merging 100+ PRs a month (https://github.com/matplotlib/matplotlib/pulse/monthly) so reviews are happening, the issue is just that some are falling through the cracks.

Tom

On Tue, Jan 15, 2019 at 4:45 PM Antony Lee <[hidden email]> wrote:
Eric,
Neither the ttc font, nor the pdf/ps common code PRs actually touch any "complex" point regarding fonts or pdf/ps.  I have described the ttc font in my previous message; the pdf/ps PR is really just, look, these two classes share 80 lines worth of code that's literally copy-pasted, let's just put that in a common parent class and inherit from it.
Conversely, one font-related PR for which I actually have low hopes of getting properly reviewed (except on an "sure, I trust you" basis) is https://github.com/matplotlib/matplotlib/pull/12928, which actually touches a rather complex point about font encodings (some of which I had to clarify on the FreeType mailing list).  (If anyone wants to have a look at it, though... :p)
Antony

On Tue, Jan 15, 2019 at 10:36 PM Eric Firing <[hidden email]> wrote:
Antony,

Your examples illustrate that the problem is often that we don't have 
enough people who understand some areas, like fonts and pdf/ps file 
formats, to get 2 thorough and knowledgeable reviews in a reasonable time.

I'm fine with your proposed rule.  It should help at least a little bit.

Eric


On 2019/01/15 9:11 AM, Antony Lee wrote:

> I'm obviously more aware of my own PRs, so here are a few I'd have put 
> up for single-review-merge (note that it's quite possible that they'd 
> have attracted more attention and gotten merged faster under that 
> system; that's also the goal...).  Obviously other devs may think about 
> other PRs that could have benefitted from the same process.
> 
> - https://github.com/matplotlib/matplotlib/pull/9787 adds support for 
> the ttc font format (requested since 2014) for png/svg output (that's 
> provided basically for free by FreeType) but not for pdf/ps (that would 
> require more difficult changes).  The entire PR comes down to:
>    * adding "ttc" as an extension that should be treated like "ttf"/"otf",
>    * adding error handling to pdf/ps to signal these formats are not 
> supported there,
>    * tests.
>    The PR sat open for 10 months before a first positive review (because 
> no one cares about fonts, I guess) and took another 3 months to attract 
> a second positive review.
> 
> - https://github.com/matplotlib/matplotlib/pull/12472 does a fairly 
> trivial fix to fontList.json to make it reusable for Matplotlibs 
> installed in different venvs, got a positive review in one day and 
> waited another 2.5 months for a second positive review.
> 
> - https://github.com/matplotlib/matplotlib/pull/9867 deduplicates 
> completely duplicated code between the pdf and ps backends, mostly for 
> maintainability's sake; it took 7.5 months (and 5 rebases due to 
> conflicts) to get a first positive review and another 6 (and around as 
> many rebases) to get a second one.
> 
> -----
> 
> As for the specifics (as I see it):
> - Someone opens a PR, it gets a positive review, no negative review, and 
> no activity occurs for two weeks (significant activity -- excluding 
> trivial chat on the thread).
> - Sponsor (first reviewer, or author if a committer) *pings* all devs by 
> mentioning @matplotlib/developers on the PR thread with the intent to 
> single-review-merge, and labels the PR accordingly.
> - The PR can still be reviewed/merged/rejected by the normal review 
> process.  However, if no one explicitly opposes the merge within two 
> weeks, anyone can merge it at that point (including by self-merge).
> 
> I would not exclude all API changes from the process.  For example, 
> https://github.com/matplotlib/matplotlib/pull/13173/commits/e90d264f3e5a263ef57243a2af86b46ab74ccc16
> deprecates (and prepares for deletion) two parameters to imshow() that 
> have had no effect since 2006.  Let's pretend for a second this commit 
> was a single independent PR (and not an example to showcase the 
> parameter-deprecating decorator...); if it started being forgotten I 
> would have proposed it for single-review-merge (again, note that if 
> you're worried about the change, you don't need to reject the PR; you 
> can just say, I think this API change needs to be approved by a second 
> reviewer and that'll block the single-review-merge just as well).
> 
> Antony
> 
> On Tue, Jan 15, 2019 at 7:46 PM Nelle Varoquaux 
> <[hidden email] <mailto:[hidden email]>> wrote:
> 
>     The proposed scheme by Paul doesn't seem reasonable to me. Core
>     contributor A or committers needs to actively reach out to other
>     core contributors: labeling is not enough IMO.
> 
> 
>     On Tue, 15 Jan 2019 at 10:37, Jody Klymak <[hidden email]
>     <mailto:[hidden email]>> wrote:
> 
> 
> 
>>         On 15 Jan 2019, at 10:20, Paul Hobson <[hidden email]
>>         <mailto:[hidden email]>> wrote:
>>
>>         I think this would be a good policy as well. Can I get some
>>         clarification on the flow? The way I understand it:
>>
>>         1) Someone (committer, contributor, or first-timer) opens a
>>         simple PR
>>         2) Committer A reviews it, adds a "single-review-merge" label,
>>         then digitally walks away
>>         3) No less than two weeks later, Committer B sees the PR,
>>         notices the label, opens it up, and merges without reviewing.
> 
>         If there is a Committer B, I think they should at least look at
>         the PR to make sure it doesn’t have any hidden API changes, or
>         introduce anything hostile.
> 
>         The point here is that getting the third person to look at a PR
>         is proving quite difficult, and if the rest of the folks with
>         the commit bit haven’t looked at a PR for a month, even after
>         being warned, then silence indicates consent.
> 
>         Note that only folks with the commit bit can label PRs, so
>         anyone flagging a PR like this is implicitly trusted to not be
>         wreaking havoc, and not doing this for major changes.  Since we
>         all curate our own PRs, I rather expect that the PR contributor
>         will often be the person who adds the flag.  Whether we want to
>         allow a self-merge at that point is up for debate, but if not,
>         then folks need to get in the habit of looking at old flagged
>         PRs and merging them.
> 
>         I think a huge problem we have is the LIFO default sort of the
>         github PR queue, and that any PR not on the first page might as
>         well be closed for all the attention it will get without
>         constant nagging.
> 
>         Cheers,   Jody
> 
>>         Does that capture a successful "single-review-merge" lifecycle?
>>         -paul
>>
>>         On Tue, Jan 15, 2019 at 9:57 AM Nelle Varoquaux
>>         <[hidden email] <mailto:[hidden email]>>
>>         wrote:
>>
>>             Hello,
>>
>>             Overall, I think this is a good idea, but I would like
>>             specifics on what "uncontroversial but uninteresting" PR
>>             mean. IMO, that should exclude any PR with API changes.
>>
>>             Cheers,
>>             N
>>
>>             On Tue, 15 Jan 2019 at 06:00, Antony Lee
>>             <[hidden email] <mailto:[hidden email]>> wrote:
>>
>>                 Hi all,
>>
>>                 During the weekly dev call, I proposed to amend the PR
>>                 merge rules, following an initial comment on Github
>>                 [https://github.com/matplotlib/matplotlib/pull/13173#issuecomment-453921220]. 
>>                 There was general agreement among the devs present
>>                 (Tom, Hannah, Jody, and myself), so I'm putting it
>>                 here for discussion.  The objective of this change is
>>                 to prevent "uncontroversial but uninteresting" PRs
>>                 from falling into oblivion, and try to decrease the
>>                 size of the open PR stack.
>>
>>                 The current rule is that a PR needs positive reviews
>>                 (from two committers, and excluding the author if a
>>                 committer) to be merged, except that doc-only PRs
>>                 (docstrings, rst) only needs a single positive review.
>>
>>                 I am proposing that, if a (non-doc) PR already has one
>>                 positive review, but no activity on that PR has
>>                 occurred for two weeks (exact time interval up to
>>                 bikeshedding) [Jody suggests: and the PR has 100% code
>>                 coverage], then a committer (either the first
>>                 reviewer, or the author if a committer) can suggest
>>                 that it be merged on the basis of that single review. 
>>                 To do so, the "sponsor" should ping all developers
>>                 (@matplotlib/developers) on that issue indicating that
>>                 intent, and add a "single-review-merge" label on the
>>                 PR (so that these PRs can easily be found). 
>>                 Committers are encouraged to review the PR to accept
>>                 and merge it or reject it or request changes on it;
>>                 but they can also just indicate that they consider the
>>                 PR sufficiently complex that a proper second review is
>>                 needed before merging, or request an extension, etc. 
>>                 To do so they should still leave a "reject" review,
>>                 even if just saying "objecting to single-review
>>                 merged; anyone can dismiss after a second review". 
>>                 However, if within another two weeks, no committer
>>                 voiced any objection (explicitly, i.e. by rejecting),
>>                 then the PR can indeed be merged on the basis of that
>>                 single review.
>>
>>                 To avoid overwhelming the system, any committer can
>>                 only "sponsor" a single PR at a time.
>>
>>                 Thoughts?
>>
>>                 Antony
>>                 _______________________________________________
>>                 Matplotlib-devel mailing list
>>                 [hidden email]
>>                 <mailto:[hidden email]>
>>                 https://mail.python.org/mailman/listinfo/matplotlib-devel
>>
>>             _______________________________________________
>>             Matplotlib-devel mailing list
>>             [hidden email]
>>             <mailto:[hidden email]>
>>             https://mail.python.org/mailman/listinfo/matplotlib-devel
>>
>>         _______________________________________________
>>         Matplotlib-devel mailing list
>>         [hidden email] <mailto:[hidden email]>
>>         https://mail.python.org/mailman/listinfo/matplotlib-devel
> 
>         --
>         Jody Klymak
>         http://web.uvic.ca/~jklymak/
> 
> 
> 
> 
> 
>     _______________________________________________
>     Matplotlib-devel mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://mail.python.org/mailman/listinfo/matplotlib-devel
> 
> 
> _______________________________________________
> Matplotlib-devel mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/matplotlib-devel
> 

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel


-- 
Thomas Caswell
[hidden email]
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel


-- 
Thomas Caswell
[hidden email]
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel


_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to amend the PR merge rules

Nelle Varoquaux
In reply to this post by Antony Lee-3

I didn't mean to imply that you had a secret agenda in changing the API in a backward incompatible way. But mistakes happen, and looking for backward incompatibility has not been a top priority for some core developers recently, and  over time it creates a culture where looking carefully for backward incompatibility is not part of our reviewing process. Having two core developers review changes limits the chance of such pull requests being merged with backward incompatibility. I also think that every API change, including the addition of keywords, needs to be considered carefully. Part of the issues we are having in terms of maintenance are that too many functions are public and many of those don't have a consistent API with the rest of the library.

To summarize the points that have been agreed upon and the points that still need to be discussed. I've removed the one about documentation, but if we accept API changes in this system, it should probably added back as a point to be discussed.

Agreed upon:
- Has not been commented on by more than one core developer;
- Doesn't fix a release-critical bug;
- Is fully tested.

Still needs to be discussed:
1. Doesn't change the public API;
2. Doesn't add or remove a functionality;
3. Doesn't add or remove a dependency;
4. Is less than XX lines of code. (Long PR are harder to review, and thus the likelihood of one core dev not noticing a problem is higher). Threshold needs to be defined.

I'd also like to add:
5. All CI passes (because we have, in the past, ignored some errors on Travis when we felt tests were flaky)

I maintain that any pull requests that add or remove a functionality should be reviewed by 2 core developers. I guess our BDFL may have to make the final call on that one :)

My argument for 4. is that long pull requests are harder to review, even if they are trivial (there are studies on this.). It is thus easier to make a mistake in reviewing.

In terms of number of pull request per core dev, it is just that more pull requests dilute our effort in reviewing. The more someone has pull request open, the less likely they are to be reviewed and the more likely that developer is to "forget" about them. I don't think we should have a rule on the number of pull request open per person, but on the different project I contribute to, I have a rule of thumb to not have more than 5 pull requests opened. It forces me to finish up the work on the pull requests I have opened before working on something else.

As for David's point, it just happens that some collaborators and I are working on understanding factors for newcomer retention and developer burnout. We've got preliminary results that we are not ready to share but we are hoping that it'll help on that point.

Cheers,
N


On Wed, 16 Jan 2019 at 03:11, Antony Lee <[hidden email]> wrote:
Replying to Nelle's points first:
 
It seems to me that this "uncontroversial but uninteresting" rule is currently not based on objective criteria. I'm personally not in favor of removing two core contributors approval for anything that modifies the API.  We have had too many close calls on PR almost merged that had backward incompatible changes: I realize that some core developers believe that we are not moving fast enough, on a project with hundreds of thousands of users, changes to the API should not be considered lightly. 

To be clear (and even though it is probably common knowledge that I would like to evolve the API faster), there is no "secret agenda" of using this to sneak API breaks through the system.
 
Here's an attempt at better defining an "uncontroversial but uninteresting" rule based on objective criteria:
- Has not been commented on by more than one core developer;

Sounds reasonable (with a reasonably flexible interpretation -- similarly to the "no activity since two weeks, excluding trivial chat"), basically "no other core dev expressed interest in reviewing it".
 
- Doesn't change the public API;
- Doesn't add or remove a functionality;
- Doesn't add or remove a dependency;

I expect all changes of API to go through the normal route of deprecation and API change note, which even the first reviewer should make sure are present before sponsoring the PR for single-review-merge (as an aside, I guess the first reviewer could even say, "approving, but I don't want to take the responsibility for being the single reviewer"...).  So now you have a two week period where any other core dev passing by can just see whether there is an API change note and make their own judgment, including deciding to block the PR if necessary, based on that change note.
I guess it is reasonable to say "API deprecations or changes and new functionality should be clearly documented" to help that "skim-through" review.
As for *adding* functionality, I guess you would have rejected the ttc-font PR on that premise?  (Actually I now see that the additional functionality was actually not documented :))  Note that this is not some obscure edge case that no one cares about; it's a feature that has been requested for years on the tracker.

- Is fully tested;
 
Also proposed by Jody and now by David, sounds good to me.
 
- All parameters are fully documented;

(sounds contradictory with "doesn't add a functionality"... if there is no new functionality it's unlikely there's a new parameter)
 
- Doesn't fix a release-critical bug;

These tend to all be merged just before the next release is cut anyways (review dynamics are a bit different in these periods), so I wouldn't worry about it, but sure.
 
- Is less than XX lines of code. (Long PR are harder to review, and thus the likelihood of one core dev not noticing a problem is higher). Threshold needs to be defined.

I don't actually think this is a good rule; the pdf/ps merge PR is +138 lines, -182 lines but as explained above the change is fairly trivial; obviously I could have split it into smaller chunks but if anything that would have made it harder to review.
Conversely (to use an example I have already raised elsewhere) https://github.com/matplotlib/matplotlib/pull/11530 (making sure that LocationEvent.x and LocationEvent.y are integers (as they refer to pixel locations) rather than floats) is 2 lines of changes in the codebase and 20 lines of test, was reviewed by 2.5 core devs (I left a comment in passing) and we all missed that there was "problem" with it (it broke mplcursors and some other pieces of code I have that assume that it is possible to synthetize LocationEvents pointing to arbitrary places in the canvas).  (Not to derail this discussion into how to evaluate API breaks once again... but I think these kinds of changes are much more insidious than, say, removing a function, as the latter would just trigger an AttributeError that is easily noticed and at worst you just go back to the previous Matplotlib release, whereas the LocationEvent change was only caught because it was covered by the mplcursors test suite; otherwise I'd have been silently generating wrong results due to it...)
 
To be honest, it also doesn't help that some people have 50+ pull requests open (Antony? :p).

I guess if the way to fix Matplotlib is to not contribute to Matplotlib anymore, we have a bigger problem.  In fact, I have some pretty big patches fixing long-standing issues (e.g. for #8550 I have a pure-Python reimplementation of the fontconfig font weight deduction algorithm, translated from fontconfig's C implementation) that I don't even bother turning into PRs, as I know they have no chance of being reviewed and merged under the current system at least, because no one would care (see also the bus-factor point below).
 
It also doesn't help that some PR contain the same code (13117, 13128) and are being hold off until the other one is merged (also, why is 13117 being hold off for a PR that has been opened afterwards?). IMO, if a PR can be merged, it should be merged.

That's discussed in https://github.com/matplotlib/matplotlib/pull/13117#discussion_r245503693 (FTR I actually agreed on merging 13117 directly, but apparently that's an API break...)
 
But we shouldn't make too many exceptions to the two core contributors rules. It's easier to have two core contributor review a pull request, than to try to fix API issues on code that has been merged and released. Having two pairs of eyes on a PR is also a good way to avoid having a bus factor of 1 on some areas of the library.

There are already huge and central chunks of the library (Agg internals, PDF/PS) that have a bus factor close to zero.  If anything, I hope that this "drive-by review" system will encourage reviewers to start looking at places they are less familiar with ("oh yes, I guess this change makes sense" -- see Eric's reply above) and decrease that.

And then to David's (the code-coverage point is already covered above):

I think it is good remember that (as far as I know) everyone submitting PRs and everyone reviewing them isn't paid and does so in their spare time; therefore is no obligation for anyone to review anything on any particular timescale. One thing that hasn't been discussed is how to increase the reviewing capacity of the community, which I guess would involve working out how to attract and retain more regular contributors.

Given the premises "no one is paid to review PRs" and "the reviewing capacity is too small" (which I agree with), I think there are two reasonable solutions:
- Make PRs easier to merge (this proposal),
- Increasing to number of core devs (your proposal, but it's not clear to me whether this number can increase significantly in a short period of time...).

Antony

On Wed, Jan 16, 2019 at 10:34 AM David Stansby <[hidden email]> wrote:
I think it is good remember that (as far as I know) everyone submitting PRs and everyone reviewing them isn't paid and does so in their spare time; therefore is no obligation for anyone to review anything on any particular timescale. One thing that hasn't been discussed is how to increase the reviewing capacity of the community, which I guess would involve working out how to attract and retain more regular contributors.

re. the proposed changes, I'm +1 modulo objective criteria being agreed upon. Personally I agree with Nelle's list, with the addition "has 100% code coverage".

David



On Tue, 15 Jan 2019, 23:46 Nelle Varoquaux <[hidden email] wrote:
For the way GitHub sorts pull requests, it is indeed a problem, but there's a way to sort by "most recently updated." I sometimes use it to make sure that I review recently updated pull request that don't appear on the first page.

It seems to me that this "uncontroversial but uninteresting" rule is currently not based on objective criteria. I'm personally not in favor of removing two core contributors approval for anything that modifies the API.  We have had too many close calls on PR almost merged that had backward incompatible changes: I realize that some core developers believe that we are not moving fast enough, on a project with hundreds of thousands of users, changes to the API should not be considered lightly.

Here's an attempt at better defining an "uncontroversial but uninteresting" rule based on objective criteria:
- Has not been commented on by more than one core developer;
- Doesn't change the public API;
- Doesn't add or remove a functionality;
- Doesn't add or remove a dependency;
- Is fully tested;
- All parameters are fully documented;
- Doesn't fix a release-critical bug;
- Is less than XX lines of code. (Long PR are harder to review, and thus the likelihood of one core dev not noticing a problem is higher). Threshold needs to be defined.

To be honest, it also doesn't help that some people have 50+ pull requests open (Antony? :p). It also doesn't help that some PR contain the same code (13117, 13128) and are being hold off until the other one is merged (also, why is 13117 being hold off for a PR that has been opened afterwards?). IMO, if a PR can be merged, it should be merged. But we shouldn't make too many exceptions to the two core contributors rules. It's easier to have two core contributor review a pull request, than to try to fix API issues on code that has been merged and released. Having two pairs of eyes on a PR is also a good way to avoid having a bus factor of 1 on some areas of the library.


On Tue, 15 Jan 2019 at 15:19, Thomas Caswell <[hidden email]> wrote:
I think a distinction between "no one cares" and "no one has bandwidth" needs to be made.

If you look at the 'pulse' page, we merging 100+ PRs a month (https://github.com/matplotlib/matplotlib/pulse/monthly) so reviews are happening, the issue is just that some are falling through the cracks.

Tom

On Tue, Jan 15, 2019 at 4:45 PM Antony Lee <[hidden email]> wrote:
Eric,
Neither the ttc font, nor the pdf/ps common code PRs actually touch any "complex" point regarding fonts or pdf/ps.  I have described the ttc font in my previous message; the pdf/ps PR is really just, look, these two classes share 80 lines worth of code that's literally copy-pasted, let's just put that in a common parent class and inherit from it.
Conversely, one font-related PR for which I actually have low hopes of getting properly reviewed (except on an "sure, I trust you" basis) is https://github.com/matplotlib/matplotlib/pull/12928, which actually touches a rather complex point about font encodings (some of which I had to clarify on the FreeType mailing list).  (If anyone wants to have a look at it, though... :p)
Antony

On Tue, Jan 15, 2019 at 10:36 PM Eric Firing <[hidden email]> wrote:
Antony,

Your examples illustrate that the problem is often that we don't have
enough people who understand some areas, like fonts and pdf/ps file
formats, to get 2 thorough and knowledgeable reviews in a reasonable time.

I'm fine with your proposed rule.  It should help at least a little bit.

Eric


On 2019/01/15 9:11 AM, Antony Lee wrote:
> I'm obviously more aware of my own PRs, so here are a few I'd have put
> up for single-review-merge (note that it's quite possible that they'd
> have attracted more attention and gotten merged faster under that
> system; that's also the goal...).  Obviously other devs may think about
> other PRs that could have benefitted from the same process.
>
> - https://github.com/matplotlib/matplotlib/pull/9787 adds support for
> the ttc font format (requested since 2014) for png/svg output (that's
> provided basically for free by FreeType) but not for pdf/ps (that would
> require more difficult changes).  The entire PR comes down to:
>    * adding "ttc" as an extension that should be treated like "ttf"/"otf",
>    * adding error handling to pdf/ps to signal these formats are not
> supported there,
>    * tests.
>    The PR sat open for 10 months before a first positive review (because
> no one cares about fonts, I guess) and took another 3 months to attract
> a second positive review.
>
> - https://github.com/matplotlib/matplotlib/pull/12472 does a fairly
> trivial fix to fontList.json to make it reusable for Matplotlibs
> installed in different venvs, got a positive review in one day and
> waited another 2.5 months for a second positive review.
>
> - https://github.com/matplotlib/matplotlib/pull/9867 deduplicates
> completely duplicated code between the pdf and ps backends, mostly for
> maintainability's sake; it took 7.5 months (and 5 rebases due to
> conflicts) to get a first positive review and another 6 (and around as
> many rebases) to get a second one.
>
> -----
>
> As for the specifics (as I see it):
> - Someone opens a PR, it gets a positive review, no negative review, and
> no activity occurs for two weeks (significant activity -- excluding
> trivial chat on the thread).
> - Sponsor (first reviewer, or author if a committer) *pings* all devs by
> mentioning @matplotlib/developers on the PR thread with the intent to
> single-review-merge, and labels the PR accordingly.
> - The PR can still be reviewed/merged/rejected by the normal review
> process.  However, if no one explicitly opposes the merge within two
> weeks, anyone can merge it at that point (including by self-merge).
>
> I would not exclude all API changes from the process.  For example,
> https://github.com/matplotlib/matplotlib/pull/13173/commits/e90d264f3e5a263ef57243a2af86b46ab74ccc16
> deprecates (and prepares for deletion) two parameters to imshow() that
> have had no effect since 2006.  Let's pretend for a second this commit
> was a single independent PR (and not an example to showcase the
> parameter-deprecating decorator...); if it started being forgotten I
> would have proposed it for single-review-merge (again, note that if
> you're worried about the change, you don't need to reject the PR; you
> can just say, I think this API change needs to be approved by a second
> reviewer and that'll block the single-review-merge just as well).
>
> Antony
>
> On Tue, Jan 15, 2019 at 7:46 PM Nelle Varoquaux
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     The proposed scheme by Paul doesn't seem reasonable to me. Core
>     contributor A or committers needs to actively reach out to other
>     core contributors: labeling is not enough IMO.
>
>
>     On Tue, 15 Jan 2019 at 10:37, Jody Klymak <[hidden email]
>     <mailto:[hidden email]>> wrote:
>
>
>
>>         On 15 Jan 2019, at 10:20, Paul Hobson <[hidden email]
>>         <mailto:[hidden email]>> wrote:
>>
>>         I think this would be a good policy as well. Can I get some
>>         clarification on the flow? The way I understand it:
>>
>>         1) Someone (committer, contributor, or first-timer) opens a
>>         simple PR
>>         2) Committer A reviews it, adds a "single-review-merge" label,
>>         then digitally walks away
>>         3) No less than two weeks later, Committer B sees the PR,
>>         notices the label, opens it up, and merges without reviewing.
>
>         If there is a Committer B, I think they should at least look at
>         the PR to make sure it doesn’t have any hidden API changes, or
>         introduce anything hostile.
>
>         The point here is that getting the third person to look at a PR
>         is proving quite difficult, and if the rest of the folks with
>         the commit bit haven’t looked at a PR for a month, even after
>         being warned, then silence indicates consent.
>
>         Note that only folks with the commit bit can label PRs, so
>         anyone flagging a PR like this is implicitly trusted to not be
>         wreaking havoc, and not doing this for major changes.  Since we
>         all curate our own PRs, I rather expect that the PR contributor
>         will often be the person who adds the flag.  Whether we want to
>         allow a self-merge at that point is up for debate, but if not,
>         then folks need to get in the habit of looking at old flagged
>         PRs and merging them.
>
>         I think a huge problem we have is the LIFO default sort of the
>         github PR queue, and that any PR not on the first page might as
>         well be closed for all the attention it will get without
>         constant nagging.
>
>         Cheers,   Jody
>
>>         Does that capture a successful "single-review-merge" lifecycle?
>>         -paul
>>
>>         On Tue, Jan 15, 2019 at 9:57 AM Nelle Varoquaux
>>         <[hidden email] <mailto:[hidden email]>>
>>         wrote:
>>
>>             Hello,
>>
>>             Overall, I think this is a good idea, but I would like
>>             specifics on what "uncontroversial but uninteresting" PR
>>             mean. IMO, that should exclude any PR with API changes.
>>
>>             Cheers,
>>             N
>>
>>             On Tue, 15 Jan 2019 at 06:00, Antony Lee
>>             <[hidden email] <mailto:[hidden email]>> wrote:
>>
>>                 Hi all,
>>
>>                 During the weekly dev call, I proposed to amend the PR
>>                 merge rules, following an initial comment on Github
>>                 [https://github.com/matplotlib/matplotlib/pull/13173#issuecomment-453921220].
>>                 There was general agreement among the devs present
>>                 (Tom, Hannah, Jody, and myself), so I'm putting it
>>                 here for discussion.  The objective of this change is
>>                 to prevent "uncontroversial but uninteresting" PRs
>>                 from falling into oblivion, and try to decrease the
>>                 size of the open PR stack.
>>
>>                 The current rule is that a PR needs positive reviews
>>                 (from two committers, and excluding the author if a
>>                 committer) to be merged, except that doc-only PRs
>>                 (docstrings, rst) only needs a single positive review.
>>
>>                 I am proposing that, if a (non-doc) PR already has one
>>                 positive review, but no activity on that PR has
>>                 occurred for two weeks (exact time interval up to
>>                 bikeshedding) [Jody suggests: and the PR has 100% code
>>                 coverage], then a committer (either the first
>>                 reviewer, or the author if a committer) can suggest
>>                 that it be merged on the basis of that single review.
>>                 To do so, the "sponsor" should ping all developers
>>                 (@matplotlib/developers) on that issue indicating that
>>                 intent, and add a "single-review-merge" label on the
>>                 PR (so that these PRs can easily be found).
>>                 Committers are encouraged to review the PR to accept
>>                 and merge it or reject it or request changes on it;
>>                 but they can also just indicate that they consider the
>>                 PR sufficiently complex that a proper second review is
>>                 needed before merging, or request an extension, etc.
>>                 To do so they should still leave a "reject" review,
>>                 even if just saying "objecting to single-review
>>                 merged; anyone can dismiss after a second review".
>>                 However, if within another two weeks, no committer
>>                 voiced any objection (explicitly, i.e. by rejecting),
>>                 then the PR can indeed be merged on the basis of that
>>                 single review.
>>
>>                 To avoid overwhelming the system, any committer can
>>                 only "sponsor" a single PR at a time.
>>
>>                 Thoughts?
>>
>>                 Antony
>>                 _______________________________________________
>>                 Matplotlib-devel mailing list
>>                 [hidden email]
>>                 <mailto:[hidden email]>
>>                 https://mail.python.org/mailman/listinfo/matplotlib-devel
>>
>>             _______________________________________________
>>             Matplotlib-devel mailing list
>>             [hidden email]
>>             <mailto:[hidden email]>
>>             https://mail.python.org/mailman/listinfo/matplotlib-devel
>>
>>         _______________________________________________
>>         Matplotlib-devel mailing list
>>         [hidden email] <mailto:[hidden email]>
>>         https://mail.python.org/mailman/listinfo/matplotlib-devel
>
>         --
>         Jody Klymak
>         http://web.uvic.ca/~jklymak/
>
>
>
>
>
>     _______________________________________________
>     Matplotlib-devel mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://mail.python.org/mailman/listinfo/matplotlib-devel
>
>
> _______________________________________________
> Matplotlib-devel mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/matplotlib-devel
>

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel


--
Thomas Caswell
[hidden email]
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to amend the PR merge rules

Antony Lee-3


On Wed, Jan 16, 2019 at 6:26 PM Nelle Varoquaux <[hidden email]> wrote:

I didn't mean to imply that you had a secret agenda in changing the API in a backward incompatible way. But mistakes happen, and looking for backward incompatibility has not been a top priority for some core developers recently, and  over time it creates a culture where looking carefully for backward incompatibility is not part of our reviewing process. Having two core developers review changes limits the chance of such pull requests being merged with backward incompatibility. I also think that every API change, including the addition of keywords, needs to be considered carefully. Part of the issues we are having in terms of maintenance are that too many functions are public and many of those don't have a consistent API with the rest of the library.

Again, you may note that I have fairly few PRs that introduce new functionality; most of the API changes I PR'd actually deprecate and later remove functions that "should not have been public" (I have sort of given up on making things consistent in an backcompatible manner).  So in that respect I am mostly trying to improve maintanability...
 
To summarize the points that have been agreed upon and the points that still need to be discussed. I've removed the one about documentation, but if we accept API changes in this system, it should probably added back as a point to be discussed.

Agreed upon:
- Has not been commented on by more than one core developer;
- Doesn't fix a release-critical bug;
- Is fully tested.

Still needs to be discussed:
1. Doesn't change the public API;
2. Doesn't add or remove a functionality;
3. Doesn't add or remove a dependency;
4. Is less than XX lines of code. (Long PR are harder to review, and thus the likelihood of one core dev not noticing a problem is higher). Threshold needs to be defined.

I'd also like to add:
5. All CI passes (because we have, in the past, ignored some errors on Travis when we felt tests were flaky)

Agreed on 5.
 
I maintain that any pull requests that add or remove a functionality should be reviewed by 2 core developers. I guess our BDFL may have to make the final call on that one :)

Happy to let Tom decide, but again I would say that "add or remove a functionality" is not always an objective feature either (e.g., does the LocationEvent-integer PR qualify?).
 
My argument for 4. is that long pull requests are harder to review, even if they are trivial (there are studies on this.). It is thus easier to make a mistake in reviewing.

As I already mentioned, there have been cases of 2-line PRs (+ 20 lines of *new* tests) making backwards incompatible changes, whereas some 100+ lines PR are essentially copy-paste trivialities, so I really don't buy the argument.
 
In terms of number of pull request per core dev, it is just that more pull requests dilute our effort in reviewing. The more someone has pull request open, the less likely they are to be reviewed and the more likely that developer is to "forget" about them. I don't think we should have a rule on the number of pull request open per person, but on the different project I contribute to, I have a rule of thumb to not have more than 5 pull requests opened. It forces me to finish up the work on the pull requests I have opened before working on something else.

Frankly it's not clear what "finish up the work" is supposed to mean.  If you look at the PRs I've mentioned above they have been merged with (close to) no changes (except for repeated rebases on master...) after the 6-to-12-month review period, so essentially all the work was done from the beginning.
 
As for David's point, it just happens that some collaborators and I are working on understanding factors for newcomer retention and developer burnout. We've got preliminary results that we are not ready to share but we are hoping that it'll help on that point.

Happy to hear anything that could be done to help on that point.  But I guess I'll take advantage of this to reply to Tom's remark as well

[Tom] I want to push back strongly on the notion that "did not review --> does not care".  As as already been discussed above there are many other reasons someone does not comment (for example "I _do_ care about this, but I {do not have time, am too tired and cranky, ...} to properly review this now").
 
I'm not pointing fingers at anyone not doing their review share, and once again totally agree with David's point regarding no one being paid.
However, you have to realize that the consequences of insufficient review volume is that real improvements are not getting in (either because the PRs stall out, or (in my case) because I don't even bother PR'ing them anymore and just live-patch matplotlib locally), and saying "it's no-one's fault" (which is true) is not going to help.

Cheers,
N

Antony

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to amend the PR merge rules

dstansby
I think real improvements not getting in at the rate they are submitted is just how it works; if I started submitting a PR a day for the next 50 days I would expect a good chunk of them sit there un-reviewed based on our current rate of merging things, and that's fine. The developers as a whole have a total reviewing rate, which is unpredictable and finite by the very nature of this being an open-source project where no-one is obliged to do anything. If we want to adjust reviewing rules to increase 'productivity' that seems fine to me, but I don't think we should be doing it with the end goal of being able to review at the rate that code is submitted (or any particular other rate).

re. large diff PRs, I find 100+ line PRs that are just copy and paste really hard to review - it's pretty dull for me trying to work out if code has been copy/pasted exactly without any errors (if there's an easier way to review stuff like this instead of checking every line please let me know!).

David

On Wed, 16 Jan 2019 at 17:54, Antony Lee <[hidden email]> wrote:


On Wed, Jan 16, 2019 at 6:26 PM Nelle Varoquaux <[hidden email]> wrote:

I didn't mean to imply that you had a secret agenda in changing the API in a backward incompatible way. But mistakes happen, and looking for backward incompatibility has not been a top priority for some core developers recently, and  over time it creates a culture where looking carefully for backward incompatibility is not part of our reviewing process. Having two core developers review changes limits the chance of such pull requests being merged with backward incompatibility. I also think that every API change, including the addition of keywords, needs to be considered carefully. Part of the issues we are having in terms of maintenance are that too many functions are public and many of those don't have a consistent API with the rest of the library.

Again, you may note that I have fairly few PRs that introduce new functionality; most of the API changes I PR'd actually deprecate and later remove functions that "should not have been public" (I have sort of given up on making things consistent in an backcompatible manner).  So in that respect I am mostly trying to improve maintanability...
 
To summarize the points that have been agreed upon and the points that still need to be discussed. I've removed the one about documentation, but if we accept API changes in this system, it should probably added back as a point to be discussed.

Agreed upon:
- Has not been commented on by more than one core developer;
- Doesn't fix a release-critical bug;
- Is fully tested.

Still needs to be discussed:
1. Doesn't change the public API;
2. Doesn't add or remove a functionality;
3. Doesn't add or remove a dependency;
4. Is less than XX lines of code. (Long PR are harder to review, and thus the likelihood of one core dev not noticing a problem is higher). Threshold needs to be defined.

I'd also like to add:
5. All CI passes (because we have, in the past, ignored some errors on Travis when we felt tests were flaky)

Agreed on 5.
 
I maintain that any pull requests that add or remove a functionality should be reviewed by 2 core developers. I guess our BDFL may have to make the final call on that one :)

Happy to let Tom decide, but again I would say that "add or remove a functionality" is not always an objective feature either (e.g., does the LocationEvent-integer PR qualify?).
 
My argument for 4. is that long pull requests are harder to review, even if they are trivial (there are studies on this.). It is thus easier to make a mistake in reviewing.

As I already mentioned, there have been cases of 2-line PRs (+ 20 lines of *new* tests) making backwards incompatible changes, whereas some 100+ lines PR are essentially copy-paste trivialities, so I really don't buy the argument.
 
In terms of number of pull request per core dev, it is just that more pull requests dilute our effort in reviewing. The more someone has pull request open, the less likely they are to be reviewed and the more likely that developer is to "forget" about them. I don't think we should have a rule on the number of pull request open per person, but on the different project I contribute to, I have a rule of thumb to not have more than 5 pull requests opened. It forces me to finish up the work on the pull requests I have opened before working on something else.

Frankly it's not clear what "finish up the work" is supposed to mean.  If you look at the PRs I've mentioned above they have been merged with (close to) no changes (except for repeated rebases on master...) after the 6-to-12-month review period, so essentially all the work was done from the beginning.
 
As for David's point, it just happens that some collaborators and I are working on understanding factors for newcomer retention and developer burnout. We've got preliminary results that we are not ready to share but we are hoping that it'll help on that point.

Happy to hear anything that could be done to help on that point.  But I guess I'll take advantage of this to reply to Tom's remark as well

[Tom] I want to push back strongly on the notion that "did not review --> does not care".  As as already been discussed above there are many other reasons someone does not comment (for example "I _do_ care about this, but I {do not have time, am too tired and cranky, ...} to properly review this now").
 
I'm not pointing fingers at anyone not doing their review share, and once again totally agree with David's point regarding no one being paid.
However, you have to realize that the consequences of insufficient review volume is that real improvements are not getting in (either because the PRs stall out, or (in my case) because I don't even bother PR'ing them anymore and just live-patch matplotlib locally), and saying "it's no-one's fault" (which is true) is not going to help.

Cheers,
N

Antony

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to amend the PR merge rules

Antony Lee-3
On Wed, Jan 16, 2019 at 7:36 PM David Stansby <[hidden email]> wrote:
I think real improvements not getting in at the rate they are submitted is just how it works; if I started submitting a PR a day for the next 50 days I would expect a good chunk of them sit there un-reviewed based on our current rate of merging things, and that's fine. The developers as a whole have a total reviewing rate, which is unpredictable and finite by the very nature of this being an open-source project where no-one is obliged to do anything. If we want to adjust reviewing rules to increase 'productivity' that seems fine to me, but I don't think we should be doing it with the end goal of being able to review at the rate that code is submitted (or any particular other rate).

The end goal is not to set a specific rate, but it is indeed to try to raise that rate up.  Obviously we could also throw our hands up and say nothing can be done about it, but that seems a bit defeatist.

Right now, I would bet that a lot of PRs go in with exactly two reviewers having gone through them and no other reviewer having even skimmed them.  If anything, I believe that the system I propose will increase the number of core-devs that'll have at least skimmed through the PR (all you need to do, if you want to participate, is to check your github notifications once every other week, whereas right now you'd actually need to pay attention to each PR as it is being submitted), and it is quite likely that 1 in-depth reviewer + 3-4 skim-throughs works as well, if not better, than 2 "in-depth" reviews (especially when the second one is sometimes "oh, that looks fine and it's already been approved once, let's just commit it").
 
re. large diff PRs, I find 100+ line PRs that are just copy and paste really hard to review - it's pretty dull for me trying to work out if code has been copy/pasted exactly without any errors (if there's an easier way to review stuff like this instead of checking every line please let me know!).

Frankly, for a PR like that, I would just check that no function is missing and that everything is test-covered (without changes to the tests), and (in case you want to be sure) skim through the code to see that the PR author has not added a sneaky vulnerability that allows them to take over the user's computer... (In other words, I don't think a line-by-line check matters for such cases.)

Antony

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to amend the PR merge rules

dstansby
Apologies, my previous email was unnecessarily negative...

I think the new system is a good idea, so here's where we seem to be. In terms of process:

1) PR opened
2) PR gets 1 positive review
3) 2 weeks after review the PR gets no other significant activity, and qualifies for simpler review
4) PR opener or reviewer 1 pings all Matplotlib devs
5) If 2 weeks after pinging no other significant activity has occurred, PR can be merged by PR opener or reviewer 1

Does everyone agree on this? The bits I've put in bold still need to be defined; I don't have time now, but can provide a summary of these later.

David



On Thu, 17 Jan 2019 at 00:48, Antony Lee <[hidden email]> wrote:
On Wed, Jan 16, 2019 at 7:36 PM David Stansby <[hidden email]> wrote:
I think real improvements not getting in at the rate they are submitted is just how it works; if I started submitting a PR a day for the next 50 days I would expect a good chunk of them sit there un-reviewed based on our current rate of merging things, and that's fine. The developers as a whole have a total reviewing rate, which is unpredictable and finite by the very nature of this being an open-source project where no-one is obliged to do anything. If we want to adjust reviewing rules to increase 'productivity' that seems fine to me, but I don't think we should be doing it with the end goal of being able to review at the rate that code is submitted (or any particular other rate).

The end goal is not to set a specific rate, but it is indeed to try to raise that rate up.  Obviously we could also throw our hands up and say nothing can be done about it, but that seems a bit defeatist.

Right now, I would bet that a lot of PRs go in with exactly two reviewers having gone through them and no other reviewer having even skimmed them.  If anything, I believe that the system I propose will increase the number of core-devs that'll have at least skimmed through the PR (all you need to do, if you want to participate, is to check your github notifications once every other week, whereas right now you'd actually need to pay attention to each PR as it is being submitted), and it is quite likely that 1 in-depth reviewer + 3-4 skim-throughs works as well, if not better, than 2 "in-depth" reviews (especially when the second one is sometimes "oh, that looks fine and it's already been approved once, let's just commit it").
 
re. large diff PRs, I find 100+ line PRs that are just copy and paste really hard to review - it's pretty dull for me trying to work out if code has been copy/pasted exactly without any errors (if there's an easier way to review stuff like this instead of checking every line please let me know!).

Frankly, for a PR like that, I would just check that no function is missing and that everything is test-covered (without changes to the tests), and (in case you want to be sure) skim through the code to see that the PR author has not added a sneaky vulnerability that allows them to take over the user's computer... (In other words, I don't think a line-by-line check matters for such cases.)

Antony

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
12