[Matplotlib-devel] Reorganising the test files?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

[Matplotlib-devel] Reorganising the test files?

Nelle Varoquaux
Hello everyone,

Now that the pytest migration is almost over (thanks Elliott!), I was wondering if this could be a good opportunity to reorganize the test files to follow a "normal" python package structure.
The recommended python package structure is to place a tests folder in each submodule, and the relevant tests files in this folder.

To take a concrete example, it would for example require moving the tests corresponding of matplotlib.sphinx_ext.mathmpl.py  to matplotlib/sphinxext/tests/test_mathmpl.py

This would be a more or less incompatible change in the sense that matplotlib.tests seems to be an importable module (it probably shouldn't), with about a dozen functions. We could easily make this backward compatible in case anyone does use this module anywhere.

['__builtins__',
 '__cached__',
 '__doc__',
 '__file__',
 '__loader__',
 '__name__',
 '__package__',
 '__path__',
 '__spec__',
 '_multiprocess_can_split_',
 'absolute_import',
 'assert_str_equal',
 'difflib',
 'division',
 'os',
 'print_function',
 'setup',
 'six',
 'unicode_literals']

If we go down that route, we need to completely finish the pytest migration first.

Cheers,
N


_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
Reply | Threaded
Open this post in threaded view
|

Re: Reorganising the test files?

tcaswell
I am against moving the test files into the sub-modules. 

It seems like a lot of thrashing (on the down-stream packagers who package the main library and tests separately and on peoples brains) for no obvious gain.

Tom

On Wed, Jan 25, 2017 at 1:15 PM Nelle Varoquaux <[hidden email]> wrote:
Hello everyone,

Now that the pytest migration is almost over (thanks Elliott!), I was wondering if this could be a good opportunity to reorganize the test files to follow a "normal" python package structure.
The recommended python package structure is to place a tests folder in each submodule, and the relevant tests files in this folder.

To take a concrete example, it would for example require moving the tests corresponding of matplotlib.sphinx_ext.mathmpl.py  to matplotlib/sphinxext/tests/test_mathmpl.py

This would be a more or less incompatible change in the sense that matplotlib.tests seems to be an importable module (it probably shouldn't), with about a dozen functions. We could easily make this backward compatible in case anyone does use this module anywhere.

['__builtins__',
 '__cached__',
 '__doc__',
 '__file__',
 '__loader__',
 '__name__',
 '__package__',
 '__path__',
 '__spec__',
 '_multiprocess_can_split_',
 'absolute_import',
 'assert_str_equal',
 'difflib',
 'division',
 'os',
 'print_function',
 'setup',
 'six',
 'unicode_literals']

If we go down that route, we need to completely finish the pytest migration first.

Cheers,
N

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
Reply | Threaded
Open this post in threaded view
|

Re: Reorganising the test files?

Nelle Varoquaux

I am against moving the test files into the sub-modules. 

It seems like a lot of thrashing (on the down-stream packagers who package the main library and tests separately and on peoples brains) for no obvious gain.

Actually, it is really a pain that Matplotlib doesn't follow python standards for any contributors that contributes to any other python packages. I would suspect that it is also annoying for downstream packages that Matplotlib does not follow standards.
The reason I brought this up in the first place is because it took me a couple of minutes to look for a test for cbook.mkdirs function while reviewing a PR, and had to resort to doing a git grep to convince me it wasn't tested. If we had followed the standards in the first place, it would probably have taken me 5seconds.

I have the same feeling for Matplotlib's unstandard  packaging layout.

N
 

Tom

On Wed, Jan 25, 2017 at 1:15 PM Nelle Varoquaux <[hidden email]> wrote:
Hello everyone,

Now that the pytest migration is almost over (thanks Elliott!), I was wondering if this could be a good opportunity to reorganize the test files to follow a "normal" python package structure.
The recommended python package structure is to place a tests folder in each submodule, and the relevant tests files in this folder.

To take a concrete example, it would for example require moving the tests corresponding of matplotlib.sphinx_ext.mathmpl.py  to matplotlib/sphinxext/tests/test_mathmpl.py

This would be a more or less incompatible change in the sense that matplotlib.tests seems to be an importable module (it probably shouldn't), with about a dozen functions. We could easily make this backward compatible in case anyone does use this module anywhere.

['__builtins__',
 '__cached__',
 '__doc__',
 '__file__',
 '__loader__',
 '__name__',
 '__package__',
 '__path__',
 '__spec__',
 '_multiprocess_can_split_',
 'absolute_import',
 'assert_str_equal',
 'difflib',
 'division',
 'os',
 'print_function',
 'setup',
 'six',
 'unicode_literals']

If we go down that route, we need to completely finish the pytest migration first.

Cheers,
N

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel


_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
Reply | Threaded
Open this post in threaded view
|

Re: Reorganising the test files?

Sandro Tosi-4
In reply to this post by Nelle Varoquaux
On Wed, Jan 25, 2017 at 1:15 PM, Nelle Varoquaux
<[hidden email]> wrote:
> matplotlib.tests seems to be an importable module (it probably shouldn't)


there other external modules (cant recall which ones, but i can look
that up if you want) depending on mpl.tests being importable as they
are using its functions to run their tests

--
Sandro "morph" Tosi
My website: http://sandrotosi.me/
Me at Debian: http://wiki.debian.org/SandroTosi
G+: https://plus.google.com/u/0/+SandroTosi
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
Reply | Threaded
Open this post in threaded view
|

Re: Reorganising the test files?

Chris Barker - NOAA Federal
In reply to this post by Nelle Varoquaux
On Wed, Jan 25, 2017 at 2:38 PM, Nelle Varoquaux <[hidden email]> wrote:
Actually, it is really a pain that Matplotlib doesn't follow python standards for any contributors that contributes to any other python packages.

very true, but...

1) If we were starting from scratcth that would be one thing, but a lot of thrash jsut to meet a "standard" is probably not worth it.

2) I don't know how standardized this concept is, and personally, I don''t like it (if I understand it correctly, anyway)

I prefer that ALL tests be in one single directory tree, ideally with that tree mtaching the package / subpacakge/module tree of the project, ie:

project
    __init__.py
    proj_mod1.py
    sub_package1
      __init__.py
      sub_mod1.py
    ...

test_project
    test_proj_mod1.py
    test_sub_package1
        test_sub_mod1.py

i.e. one test file for each module file.

With this method, it would not be hard to find the tests that correspond to a particular python module.

I'm ambivalent about whether the test files should be importable packages, and I think there is a lot to be said for keeping the test code outside the installable package tree altogether, but I definitely don't like them all intermingled.

though I still don't advocate a lot of churn for this...

-CHB

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

[hidden email]

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
Reply | Threaded
Open this post in threaded view
|

Re: Reorganising the test files?

Nelle Varoquaux
Actually, it is really a pain that Matplotlib doesn't follow python standards for any contributors that contributes to any other python packages.

very true, but...

1) If we were starting from scratcth that would be one thing, but a lot of thrash jsut to meet a "standard" is probably not worth it.

The benefits of following standards for maintainability has been demonstrated over and over.
 
2) I don't know how standardized this concept is, and personally, I don''t like it (if I understand it correctly, anyway)

It is extremely standard. Matplotlib is not the only project not following standards, but it's probably in the 1% of projects not following standards.
More importantly, it is how we (the community) teach how to organize packages and tests.

numpy, scipy, skimage, pandas, sklearn, sympy, ipython all follow this standard.
The only other mainstream scientific python package that may not entirely follow this convention is cython.
 
I prefer that ALL tests be in one single directory tree, ideally with that tree mtaching the package / subpacakge/module tree of the project, ie:

The whole point of convention is to avoid  this.
 

project
    __init__.py
    proj_mod1.py
    sub_package1
      __init__.py
      sub_mod1.py
    ...

test_project
    test_proj_mod1.py
    test_sub_package1
        test_sub_mod1.py

i.e. one test file for each module file.

With this method, it would not be hard to find the tests that correspond to a particular python module.

I'm ambivalent about whether the test files should be importable packages, and I think there is a lot to be said for keeping the test code outside the installable package tree altogether, but I definitely don't like them all intermingled.

though I still don't advocate a lot of churn for this...

-CHB

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

[hidden email]


_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
Reply | Threaded
Open this post in threaded view
|

Re: Reorganising the test files?

Matthias BUSSONNIER
> numpy, scipy, skimage, pandas, sklearn, sympy, ipython all follow this standard.

We do our best in IPython to try to "catch up" on standard, and right
now we are not the best example.
Like matplotlib our test harness is relatively complex (for different
reasons than matplotlib). But we are doing
our best to push things forward. Migrating to PyTest is one of the
things we hope to one day achieve (or already start).

"The bit split" helped, but in 2014 out tests layout was kind of horrible.

I am not involved enough in matplotlib to tell you whether you should
(or not) decide to change (that's your choice, and only core dev can
likely correctly weight on cost/benefits balance), but IMHO, and the
experience I have:

  - With IPython : non traditional package layout and test harness
make it difficult for new/occasional contributors.
     - E.g in IPython a couple of time per year, we have people
running `nosetest` from the root of the directory.
     - we have a specific guard against running iptest from source directory.
     - contributing to our test suite is harder

That does not seem much, but integrating something small through time
can lead to quite a lot.

  - You may not realise it but a lot of people when wondering how to
do things will look up to "core" packages.
     - They either will not understand, potentially giving up.
     - Or will replicate a complex setup

That add to the myth that "python packaging is hard", and I think that
weighting on the overall community impact is important.
I definitely feel a "layout context switching" when working on
matplotlib. Test layout is not the only one, but it definitively
interfere with my editor which smart search on nearby files.

Catching up on "Standard" does not mean either changing all at once.
AFAIU you do that for pep8 where the number of violation can only go down,
so you can perfectly start a policy:
   - new test need to be next to thing they test.

Sidenote: I don't see why a testing submodule should not be importable.
Don't you have utility function that would be useful for others ?
Wouldn't it bother you that only a test submodule can be imported but
only by testing tools ?

I'm going to guess "just post 2.0" might not be a good time to
actually shuffle things around, but looking at "what would it break"
(beyond people habits) can be a way to see a couple of assumptions
that are not supposed to be here*, and lay out the foundation to
moving things around if one day you decide to.

Thanks !
--
Matthias

[*] source: PyCon 2014, I deleted the test suite. IPython was refusing
to start.




On Sat, Jan 28, 2017 at 10:27 AM, Nelle Varoquaux
<[hidden email]> wrote:

>>> Actually, it is really a pain that Matplotlib doesn't follow python
>>> standards for any contributors that contributes to any other python
>>> packages.
>>
>>
>> very true, but...
>>
>> 1) If we were starting from scratcth that would be one thing, but a lot of
>> thrash jsut to meet a "standard" is probably not worth it.
>
>
> The benefits of following standards for maintainability has been
> demonstrated over and over.
>
>>
>> 2) I don't know how standardized this concept is, and personally, I don''t
>> like it (if I understand it correctly, anyway)
>
>
> It is extremely standard. Matplotlib is not the only project not following
> standards, but it's probably in the 1% of projects not following standards.
> More importantly, it is how we (the community) teach how to organize
> packages and tests.
>
> numpy, scipy, skimage, pandas, sklearn, sympy, ipython all follow this
> standard.
> The only other mainstream scientific python package that may not entirely
> follow this convention is cython.
>
>>
>> I prefer that ALL tests be in one single directory tree, ideally with that
>> tree mtaching the package / subpacakge/module tree of the project, ie:
>
>
> The whole point of convention is to avoid  this.
>
>>
>>
>> project
>>     __init__.py
>>     proj_mod1.py
>>     sub_package1
>>       __init__.py
>>       sub_mod1.py
>>     ...
>>
>> test_project
>>     test_proj_mod1.py
>>     test_sub_package1
>>         test_sub_mod1.py
>>
>> i.e. one test file for each module file.
>>
>> With this method, it would not be hard to find the tests that correspond
>> to a particular python module.
>>
>> I'm ambivalent about whether the test files should be importable packages,
>> and I think there is a lot to be said for keeping the test code outside the
>> installable package tree altogether, but I definitely don't like them all
>> intermingled.
>>
>> though I still don't advocate a lot of churn for this...
>>
>> -CHB
>>
>> --
>>
>> Christopher Barker, Ph.D.
>> Oceanographer
>>
>> Emergency Response Division
>> NOAA/NOS/OR&R            (206) 526-6959   voice
>> 7600 Sand Point Way NE   (206) 526-6329   fax
>> Seattle, WA  98115       (206) 526-6317   main reception
>>
>> [hidden email]
>
>
>
> _______________________________________________
> Matplotlib-devel mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/matplotlib-devel
>
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
Reply | Threaded
Open this post in threaded view
|

Re: Reorganising the test files?

Antony Lee
It looks like there are two issues at hand here.

- Should there be one test module per real module?  My opinion is that the answer is generally yes, but there are cases where this would be awkward, so we shouldn't make it a hard rule.  For example, `test_pickle.py` tests the picklability of various objects, from simple ones to entire figures.  Of course, each of the tests could be moved to the test module corresponding to the module where the object is defined, but I doubt this would make things clearer.

- Should there be a `matplotlib.tests` package containing all the tests (possibly replicating the matplotlib hierarchy) or one `.tests` subpackage per package?  Let's realize that matplotlib has a very flat hierarchy compared to the other projects Nelle mentioned.  Basically, the change would entail moving
    - test_axes to axes/tests
    - test_backend_* to backends/tests
    - test_cbook to cbook/tests
    - test_triangulation to tri/tests
    - test_style to style/tests
The other subpackages are projections and sphinxext, for which there does not appear to be a dedicated test module.  Frankly, which such a small number of exceptions, I feel it would be more confusing to move the corresponding tests out to their own subpackage.  The situation is not the same as in numpy, scipy, skimage, pandas, sklearn and sympy, which (I just checked) all have more subpackages (i.e. folders) than submodules (i.e. files) directly under the main package.  (IPython has more direct submodules, but many of them seem to be just shims left over from the big split).  For these projects, subpackages provide a clear partitioning between parts of the code, and it is thus natural to split the tests in the same manner.

Antony

2017-01-28 17:40 GMT-08:00 Matthias Bussonnier <[hidden email]>:
> numpy, scipy, skimage, pandas, sklearn, sympy, ipython all follow this standard.

We do our best in IPython to try to "catch up" on standard, and right
now we are not the best example.
Like matplotlib our test harness is relatively complex (for different
reasons than matplotlib). But we are doing
our best to push things forward. Migrating to PyTest is one of the
things we hope to one day achieve (or already start).

"The bit split" helped, but in 2014 out tests layout was kind of horrible.

I am not involved enough in matplotlib to tell you whether you should
(or not) decide to change (that's your choice, and only core dev can
likely correctly weight on cost/benefits balance), but IMHO, and the
experience I have:

  - With IPython : non traditional package layout and test harness
make it difficult for new/occasional contributors.
     - E.g in IPython a couple of time per year, we have people
running `nosetest` from the root of the directory.
     - we have a specific guard against running iptest from source directory.
     - contributing to our test suite is harder

That does not seem much, but integrating something small through time
can lead to quite a lot.

  - You may not realise it but a lot of people when wondering how to
do things will look up to "core" packages.
     - They either will not understand, potentially giving up.
     - Or will replicate a complex setup

That add to the myth that "python packaging is hard", and I think that
weighting on the overall community impact is important.
I definitely feel a "layout context switching" when working on
matplotlib. Test layout is not the only one, but it definitively
interfere with my editor which smart search on nearby files.

Catching up on "Standard" does not mean either changing all at once.
AFAIU you do that for pep8 where the number of violation can only go down,
so you can perfectly start a policy:
   - new test need to be next to thing they test.

Sidenote: I don't see why a testing submodule should not be importable.
Don't you have utility function that would be useful for others ?
Wouldn't it bother you that only a test submodule can be imported but
only by testing tools ?

I'm going to guess "just post 2.0" might not be a good time to
actually shuffle things around, but looking at "what would it break"
(beyond people habits) can be a way to see a couple of assumptions
that are not supposed to be here*, and lay out the foundation to
moving things around if one day you decide to.

Thanks !
--
Matthias

[*] source: PyCon 2014, I deleted the test suite. IPython was refusing
to start.




On Sat, Jan 28, 2017 at 10:27 AM, Nelle Varoquaux
<[hidden email]> wrote:
>>> Actually, it is really a pain that Matplotlib doesn't follow python
>>> standards for any contributors that contributes to any other python
>>> packages.
>>
>>
>> very true, but...
>>
>> 1) If we were starting from scratcth that would be one thing, but a lot of
>> thrash jsut to meet a "standard" is probably not worth it.
>
>
> The benefits of following standards for maintainability has been
> demonstrated over and over.
>
>>
>> 2) I don't know how standardized this concept is, and personally, I don''t
>> like it (if I understand it correctly, anyway)
>
>
> It is extremely standard. Matplotlib is not the only project not following
> standards, but it's probably in the 1% of projects not following standards.
> More importantly, it is how we (the community) teach how to organize
> packages and tests.
>
> numpy, scipy, skimage, pandas, sklearn, sympy, ipython all follow this
> standard.
> The only other mainstream scientific python package that may not entirely
> follow this convention is cython.
>
>>
>> I prefer that ALL tests be in one single directory tree, ideally with that
>> tree mtaching the package / subpacakge/module tree of the project, ie:
>
>
> The whole point of convention is to avoid  this.
>
>>
>>
>> project
>>     __init__.py
>>     proj_mod1.py
>>     sub_package1
>>       __init__.py
>>       sub_mod1.py
>>     ...
>>
>> test_project
>>     test_proj_mod1.py
>>     test_sub_package1
>>         test_sub_mod1.py
>>
>> i.e. one test file for each module file.
>>
>> With this method, it would not be hard to find the tests that correspond
>> to a particular python module.
>>
>> I'm ambivalent about whether the test files should be importable packages,
>> and I think there is a lot to be said for keeping the test code outside the
>> installable package tree altogether, but I definitely don't like them all
>> intermingled.
>>
>> though I still don't advocate a lot of churn for this...
>>
>> -CHB
>>
>> --
>>
>> Christopher Barker, Ph.D.
>> Oceanographer
>>
>> Emergency Response Division
>> NOAA/NOS/OR&R            <a href="tel:%28206%29%20526-6959" value="+12065266959">(206) 526-6959   voice
>> 7600 Sand Point Way NE   (206) 526-6329   fax
>> Seattle, WA  98115       (206) 526-6317   main reception
>>
>> [hidden email]
>
>
>
> _______________________________________________
> Matplotlib-devel mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/matplotlib-devel
>
_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel


_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel
Reply | Threaded
Open this post in threaded view
|

Re: Reorganising the test files?

Nathaniel Smith
On Jan 29, 2017 1:44 AM, "Antony Lee" <[hidden email]> wrote:
It looks like there are two issues at hand here.

- Should there be one test module per real module?  My opinion is that the answer is generally yes, but there are cases where this would be awkward, so we shouldn't make it a hard rule.  For example, `test_pickle.py` tests the picklability of various objects, from simple ones to entire figures.  Of course, each of the tests could be moved to the test module corresponding to the module where the object is defined, but I doubt this would make things clearer.

- Should there be a `matplotlib.tests` package containing all the tests (possibly replicating the matplotlib hierarchy) or one `.tests` subpackage per package?  Let's realize that matplotlib has a very flat hierarchy compared to the other projects Nelle mentioned.  Basically, the change would entail moving
    - test_axes to axes/tests
    - test_backend_* to backends/tests
    - test_cbook to cbook/tests
    - test_triangulation to tri/tests
    - test_style to style/tests

Another thing to consider here is that some pytest features are biased towards layouts where all tests live in a single hierarchy. If you define a custom fixture in a conftest.py file, that fixture is available to tests in the directory and subdirectories, but not siblings. Maybe matplotlib doesn't use these features though, or maybe there's some way around this in pytest (if you figure it out I'd love to know! pytest is very, uh, "feature rich", and I haven't looked into this in depth, it's just something I ran into).

-n

_______________________________________________
Matplotlib-devel mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/matplotlib-devel