Discussion:
[PATCH 1/2] python: Properly uninstall __pycache__ in subdirectories
Lukas Fleischer
2018-05-20 15:28:51 UTC
Permalink
When uninstalling __pycache__ files in a subdirectory "sub", the
Makefile incorrectly removed the files from __pycache__/sub/ instead of
sub/__pycache__/.

* lib/am/python.am (uninstall-%DIR%PYTHON): Use the correct path when
installing byte-compiled files installed in '__pycache__'
subdirectories.
---
lib/am/python.am | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/am/python.am b/lib/am/python.am
index 21e6f842c..f315d9722 100644
--- a/lib/am/python.am
+++ b/lib/am/python.am
@@ -97,7 +97,7 @@ endif %?INSTALL%
if %?INSTALL%

?FIRST?am__pep3147_tweak = \
-?FIRST? sed -e 's|\.py$$||' -e 's|[^/]*$$|&.*.pyc\n&.*.pyo|'
+?FIRST? sed -e 's|\.py$$||' -e 's|[^/]*$$|__pycache__/&.*.pyc\n__pycache__/&.*.pyo|'

.PHONY uninstall-am: uninstall-%DIR%PYTHON
uninstall-%DIR%PYTHON:
@@ -116,7 +116,7 @@ uninstall-%DIR%PYTHON:
done; \
## This is somewhat tricky, because for newer pythons we have to take PEP-3147
## into account. Avoid exceeding the command-line length limit.
- dir='$(DESTDIR)$(%NDIR%dir)/__pycache__'; \
+ dir='$(DESTDIR)$(%NDIR%dir)'; \
echo "$$py_files" | $(am__pep3147_tweak) | $(am__base_list) | \
while read files; do \
$(am__uninstall_files_from_dir) || st=$$?; \
--
2.17.0
Lukas Fleischer
2018-05-20 15:28:52 UTC
Permalink
As of Python 3.5, but unoptimized and optimized bytecode are stored
within .pyc files; .pyo files are no longer generated. Update the Python
tests such that the test do not fail if .pyo files are missing.

* t/py-compile-basedir.sh: Remove all .pyo checks.
* t/py-compile-basic.sh: Likewise.
* t/py-compile-destdir.sh: Likewise.
* t/py-compile-option-terminate.sh: Likewise.
* t/python-virtualenv.sh: Likewise.
* t/python10.sh: Likewise.
* t/python12.sh: Likewise.
* t/python3.sh: Likewise.
---
t/py-compile-basedir.sh | 2 --
t/py-compile-basic.sh | 3 ---
t/py-compile-destdir.sh | 12 +++++-------
t/py-compile-option-terminate.sh | 5 -----
t/python-virtualenv.sh | 4 ----
t/python10.sh | 6 ------
t/python12.sh | 3 +--
t/python3.sh | 1 -
8 files changed, 6 insertions(+), 30 deletions(-)

diff --git a/t/py-compile-basedir.sh b/t/py-compile-basedir.sh
index 6921d16d5..9f927d066 100644
--- a/t/py-compile-basedir.sh
+++ b/t/py-compile-basedir.sh
@@ -40,9 +40,7 @@ for d in foo foo/bar "$(pwd)/foo" . .. ../foo ''; do
./py-compile --basedir "$d" "$f.py" "sub/$f.py"
find "$d2" # For debugging.
py_installed "$d2/$f.pyc"
- py_installed "$d2/$f.pyo"
py_installed "$d2/sub/$f.pyc"
- py_installed "$d2/sub/$f.pyo"
files=$(find "$d2" | grep '\.py[co]$')
test $(echo "$files" | wc -l) -eq 4
case $d2 in
diff --git a/t/py-compile-basic.sh b/t/py-compile-basic.sh
index 24df600cb..4c5cf57aa 100644
--- a/t/py-compile-basic.sh
+++ b/t/py-compile-basic.sh
@@ -61,10 +61,7 @@ mkdir -p 1/_/2/_/3/_

./py-compile foo.py sub/bar.py 1/_/2/_/3/_/0.py
py_installed foo.pyc
-py_installed foo.pyo
py_installed sub/bar.pyc
-py_installed sub/bar.pyo
py_installed 1/_/2/_/3/_/0.pyc
-py_installed 1/_/2/_/3/_/0.pyo

:
diff --git a/t/py-compile-destdir.sh b/t/py-compile-destdir.sh
index 07aa1f44f..d2ccafee5 100644
--- a/t/py-compile-destdir.sh
+++ b/t/py-compile-destdir.sh
@@ -34,13 +34,11 @@ echo 'def bar (): return "bar"' > $destdir/sub/bar.py

find $destdir # For debugging.
st=0
-for x in c o; do
- for b in foo sub/bar; do
- f=$(pyc_location -p "$destdir/$b.py$x")
- test -f "$f"
- strings "$f" || : # For debugging.
- $FGREP $destdir $f && { echo BAD: $f; st=1; }
- done
+for b in foo sub/bar; do
+ f=$(pyc_location -p "$destdir/$b.pyc")
+ test -f "$f"
+ strings "$f" || : # For debugging.
+ $FGREP $destdir $f && { echo BAD: $f; st=1; }
done
exit $st

diff --git a/t/py-compile-option-terminate.sh b/t/py-compile-option-terminate.sh
index 1aaeec21b..80ae55a88 100644
--- a/t/py-compile-option-terminate.sh
+++ b/t/py-compile-option-terminate.sh
@@ -27,17 +27,12 @@ cp "$am_scriptdir/py-compile" . \
: > ./--foo.py
./py-compile -- -o.py --foo.py
py_installed ./-o.pyc
-py_installed ./-o.pyo
py_installed ./--foo.pyc
-py_installed ./--foo.pyo
rm -f ./-*.py[co]
: > x.py
./py-compile x.py -o.py --foo.py
py_installed ./x.pyc
-py_installed ./x.pyo
py_installed ./-o.pyc
-py_installed ./-o.pyo
py_installed ./--foo.pyc
-py_installed ./--foo.pyo

:
diff --git a/t/python-virtualenv.sh b/t/python-virtualenv.sh
index 3bcd6ec76..7f9708866 100644
--- a/t/python-virtualenv.sh
+++ b/t/python-virtualenv.sh
@@ -124,10 +124,8 @@ check_install ()

test -f "$py_site"/am_foo.py
py_installed "$py_site"/am_foo.pyc
- py_installed "$py_site"/am_foo.pyo
py_installed "$py_site"/am_virtenv/__init__.py
py_installed "$py_site"/am_virtenv/__init__.pyc
- py_installed "$py_site"/am_virtenv/__init__.pyo
test -f "$py_site"/libquux.a
test -f "$py_site"/am_virtenv/libzardoz.a
}
@@ -138,10 +136,8 @@ check_uninstall ()

test ! -e "$py_site"/am_foo.py
py_installed --not "$py_site"/am_foo.pyc
- py_installed --not "$py_site"/am_foo.pyo
test ! -e "$py_site"/am_virtenv/__init__.py
py_installed --not "$py_site"/am_virtenv/__init__.pyc
- py_installed --not "$py_site"/am_virtenv/__init__.pyo
test ! -e "$py_site"/libquux.a
test ! -e "$py_site"/am_virtenv/libzardoz.a
}
diff --git a/t/python10.sh b/t/python10.sh
index 4def103ff..9c7a46175 100644
--- a/t/python10.sh
+++ b/t/python10.sh
@@ -60,27 +60,21 @@ cwd=$(pwd) || fatal_ "getting current working directory"
$MAKE install
test -f "$inst/your/two.py"
py_installed "$inst/your/two.pyc"
-py_installed "$inst/your/two.pyo"
py_installed --not "$inst/my/one.py"
py_installed --not "$inst/my/one.pyc"
-py_installed --not "$inst/my/one.pyo"
$MAKE uninstall
py_installed --not "$inst/your/two.py"
py_installed --not "$inst/your/two.pyc"
-py_installed --not "$inst/your/two.pyo"

../configure --prefix=$cwd/"$inst" one=1
$MAKE install
py_installed --not "$inst/your/two.py"
py_installed --not "$inst/your/two.pyc"
-py_installed --not "$inst/your/two.pyo"
test -f "$inst/my/one.py"
py_installed "$inst/my/one.pyc"
-py_installed "$inst/my/one.pyo"
$MAKE uninstall
py_installed --not "$inst/my/one.py"
py_installed --not "$inst/my/one.pyc"
-py_installed --not "$inst/my/one.pyo"

$MAKE disttest

diff --git a/t/python12.sh b/t/python12.sh
index d998ae8ad..cca057e6d 100644
--- a/t/python12.sh
+++ b/t/python12.sh
@@ -44,11 +44,10 @@ $MAKE install DESTDIR=$destdir

# Perfunctory test that the files were created.
test -f "$destdir/usr/share/my/my.py"
-pyo=$(pyc_location -p "$destdir/usr/share/my/my.pyo")
pyc=$(pyc_location -p "$destdir/usr/share/my/my.pyc")

# If DESTDIR has made it into the byte compiled files, fail the test.
-st=0; $FGREP "$destdir" "$pyc" "$pyo" || st=$?
+st=0; $FGREP "$destdir" "$pyc" || st=$?
test $st -eq 1

:
diff --git a/t/python3.sh b/t/python3.sh
index 0008eadd7..131aeb48d 100644
--- a/t/python3.sh
+++ b/t/python3.sh
@@ -42,6 +42,5 @@ cd build
$MAKE install
py_installed inst/my/one.py
py_installed inst/my/one.pyc
-py_installed inst/my/one.pyo

:
--
2.17.0
Mathieu Lirzin
2018-07-07 21:45:27 UTC
Permalink
Post by Lukas Fleischer
As of Python 3.5, but unoptimized and optimized bytecode are stored
within .pyc files; .pyo files are no longer generated. Update the Python
tests such that the test do not fail if .pyo files are missing.
Like in other message, it is important that the test suite passes
with older python version. Have you checked?
Post by Lukas Fleischer
* t/py-compile-basedir.sh: Remove all .pyo checks.
* t/py-compile-basic.sh: Likewise.
* t/py-compile-destdir.sh: Likewise.
* t/py-compile-option-terminate.sh: Likewise.
* t/python-virtualenv.sh: Likewise.
* t/python10.sh: Likewise.
* t/python12.sh: Likewise.
* t/python3.sh: Likewise.
---
t/py-compile-basedir.sh | 2 --
t/py-compile-basic.sh | 3 ---
t/py-compile-destdir.sh | 12 +++++-------
t/py-compile-option-terminate.sh | 5 -----
t/python-virtualenv.sh | 4 ----
t/python10.sh | 6 ------
t/python12.sh | 3 +--
t/python3.sh | 1 -
8 files changed, 6 insertions(+), 30 deletions(-)
Thanks for the patch.
--
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761 070D 0ADE E100 9460 4D37
Lukas Fleischer
2018-07-08 10:57:11 UTC
Permalink
Post by Mathieu Lirzin
Post by Lukas Fleischer
As of Python 3.5, but unoptimized and optimized bytecode are stored
within .pyc files; .pyo files are no longer generated. Update the Python
tests such that the test do not fail if .pyo files are missing.
Like in other message, it is important that the test suite passes
with older python version. Have you checked?
Note that this patch only removes some checks, so after this patch,
everything is strictly less restrictive. I don't see how this could
possibly break backwards compatibility.

Nevertheless, I just made sure that everything still works with Python
2.7.15.
Mathieu Lirzin
2018-07-08 20:27:44 UTC
Permalink
Post by Lukas Fleischer
Post by Mathieu Lirzin
Post by Lukas Fleischer
As of Python 3.5, but unoptimized and optimized bytecode are stored
within .pyc files; .pyo files are no longer generated. Update the Python
tests such that the test do not fail if .pyo files are missing.
Like in other message, it is important that the test suite passes
with older python version. Have you checked?
Note that this patch only removes some checks, so after this patch,
everything is strictly less restrictive. I don't see how this could
possibly break backwards compatibility.
Nevertheless, I just made sure that everything still works with Python
2.7.15.
OK I overlooked that. I think a inlined comment in the concerned tests
explaining why python object files presence is not tested would help
future code reader to understand.

Can you send an updated patch?

Thanks.
--
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761 070D 0ADE E100 9460 4D37
Mathieu Lirzin
2018-07-08 21:29:47 UTC
Permalink
Post by Mathieu Lirzin
Post by Lukas Fleischer
Post by Mathieu Lirzin
Post by Lukas Fleischer
As of Python 3.5, but unoptimized and optimized bytecode are stored
within .pyc files; .pyo files are no longer generated. Update the Python
tests such that the test do not fail if .pyo files are missing.
Like in other message, it is important that the test suite passes
with older python version. Have you checked?
Note that this patch only removes some checks, so after this patch,
everything is strictly less restrictive. I don't see how this could
possibly break backwards compatibility.
Nevertheless, I just made sure that everything still works with Python
2.7.15.
OK I overlooked that. I think a inlined comment in the concerned tests
explaining why python object files presence is not tested would help
future code reader to understand.
Can you send an updated patch?
Please forget the updated patch fornow. I have just remember about
about bug#30556 [1] where test suite is known to fail when ‘python’ is
‘python3’ and not ‘python2’ which is the case on Arch Linux. Sorry for
the confusion.

You seem to have a better understanding of how various version of python
handles compilation and caches than I do. Maybe can you summarize (give
some pointers) the various evolution in that regard?

[1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=30556
--
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761 070D 0ADE E100 9460 4D37
Lukas Fleischer
2018-07-09 14:59:36 UTC
Permalink
Post by Mathieu Lirzin
Post by Mathieu Lirzin
Post by Lukas Fleischer
Post by Mathieu Lirzin
Post by Lukas Fleischer
As of Python 3.5, but unoptimized and optimized bytecode are stored
within .pyc files; .pyo files are no longer generated. Update the Python
tests such that the test do not fail if .pyo files are missing.
Like in other message, it is important that the test suite passes
with older python version. Have you checked?
Note that this patch only removes some checks, so after this patch,
everything is strictly less restrictive. I don't see how this could
possibly break backwards compatibility.
Nevertheless, I just made sure that everything still works with Python
2.7.15.
OK I overlooked that. I think a inlined comment in the concerned tests
explaining why python object files presence is not tested would help
future code reader to understand.
Can you send an updated patch?
Please forget the updated patch fornow. I have just remember about
about bug#30556 [1] where test suite is known to fail when ‘python’ is
‘python3’ and not ‘python2’ which is the case on Arch Linux. Sorry for
the confusion.
I don't quite understand. Note that it is the automake test suite, not
Arch Linux, that does not conform with PEP 394. While PEP 394 mentions
that all distributions *should* ensure that python refers to python2, it
also mentions that this is not necessarily always the case. The document
also says that "[...] python should be used in the shebang line only for
scripts that are source compatible with both Python 2 and 3" and "in
preparation for an eventual change in the default version of Python,
Python 2 only scripts should either be updated to be source compatible
with Python 3 or else to use python2 in the shebang line."

So, we should either rewrite the test cases to explicitly use python2
or, even better, keep using python and make the tests work with both
Python 2 and Python 3, the latter of which is achieved by the two
patches I submitted. Any rationale on why they should be "forgotten for
now"?
Post by Mathieu Lirzin
You seem to have a better understanding of how various version of python
handles compilation and caches than I do. Maybe can you summarize (give
some pointers) the various evolution in that regard?
__pycache__ directories were introduced in Python 3.2 (for details, see
PEP 3147 [1]). PYO files were removed in Python 3.5 (see PEP 488 [2]).

Best regards,
Lukas

[1] https://www.python.org/dev/peps/pep-3147/
[2] https://www.python.org/dev/peps/pep-0488/
Lukas Fleischer
2018-07-22 09:05:19 UTC
Permalink
Post by Lukas Fleischer
So, we should either rewrite the test cases to explicitly use python2
or, even better, keep using python and make the tests work with both
Python 2 and Python 3, the latter of which is achieved by the two
patches I submitted. Any rationale on why they should be "forgotten for
now"?
Any news on this? This is still blocking the automake package on Arch
Linux...

Mathieu Lirzin
2018-07-07 21:43:28 UTC
Permalink
Post by Lukas Fleischer
When uninstalling __pycache__ files in a subdirectory "sub", the
Makefile incorrectly removed the files from __pycache__/sub/ instead of
sub/__pycache__/.
* lib/am/python.am (uninstall-%DIR%PYTHON): Use the correct path when
installing byte-compiled files installed in '__pycache__'
subdirectories.
Tell me if I am overlooking something, but I guess this change will
imply that the uninstallation does not properly work with older python
version? Have you tried?

This support is important in the context of Automake.

Thanks for the patch.
--
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761 070D 0ADE E100 9460 4D37
Lukas Fleischer
2018-07-08 11:05:22 UTC
Permalink
Post by Mathieu Lirzin
Post by Lukas Fleischer
When uninstalling __pycache__ files in a subdirectory "sub", the
Makefile incorrectly removed the files from __pycache__/sub/ instead of
sub/__pycache__/.
* lib/am/python.am (uninstall-%DIR%PYTHON): Use the correct path when
installing byte-compiled files installed in '__pycache__'
subdirectories.
Tell me if I am overlooking something, but I guess this change will
imply that the uninstallation does not properly work with older python
version? Have you tried?
I don't think so. My guess is that uninstalling __pycache__ from
subdirectories has always been broken --- unless there has been a
behavioral change in recent Python releases that affects the location of
__pycache__ in strange ways.

When running the test suite with Python 2, everything still works (which
is not surprising given that __pycache__ was only introduced in Python
3.2).

Unfortunately, when I run nobase-python.sh with Python 3.4, that test
fails both with and without the patch applied, so it is hard to tell
whether __pycache__ handling is broken with that Python release.

Regards,
Lukas
Loading...