Quantcast

Background processes in GNU Autotest

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Background processes in GNU Autotest

Olaf Mandel
Hello,

I am trying to use GNU Autotest (via AX_GNU_AUTOTEST()) to run
end-to-end tests on a network server. My current test script looks
somewhat like this:

AT_CHECK([server&],        , [ignore])
AT_CHECK([client --cmd]),  , [expected-output])
AT_CHECK([killall server], , [ignore])

This is far from perfect, however: there is no cleanup if the client
check fails or if the executable is called differently (see below).

Now I want to combine this with valgrind checking (from
AX_VALGRIND_CHECK()): I defined the check-TESTS target in the Makefile
so it calls "make check-autotest" and renames the log files to the
expected names and I added a macro to the test script to use an
environment variable RUNNER:

AC_DEFUN([WRAP_CMD], [${RUNNER} $(which $1) $2])
AT_CHECK(WRAP_CMD([server], [&]),       , [ignore])
AT_CHECK(WRAP_CMD([client], [--cmd]),   , [expected-output])
AT_CHECK(WRAP_CMD([killall], [server]), , [ignore])

But this fails to kill the server simply because the actual command
being executed by the first AT_CHECK with make check-valgrind is:

valgrind --error-exitcode=1 --quiet ./server &

So the simple killall call is no longer feasible.

Now my question: is there a standard and approved way to start a
background process in a GNU Autotest script and to kill it at or before
the AT_CLEANUP ?

Thank you for any suggestions,
Olaf Mandel


_______________________________________________
Autoconf mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/autoconf

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Background processes in GNU Autotest

Mike Frysinger
On 22 Jun 2016 11:03, Olaf Mandel wrote:

> I am trying to use GNU Autotest (via AX_GNU_AUTOTEST()) to run
> end-to-end tests on a network server. My current test script looks
> somewhat like this:
>
> AT_CHECK([server&],        , [ignore])
> AT_CHECK([client --cmd]),  , [expected-output])
> AT_CHECK([killall server], , [ignore])
>
> This is far from perfect, however: there is no cleanup if the client
> check fails or if the executable is called differently (see below).
>
> Now I want to combine this with valgrind checking (from
> AX_VALGRIND_CHECK()): I defined the check-TESTS target in the Makefile
> so it calls "make check-autotest" and renames the log files to the
> expected names and I added a macro to the test script to use an
> environment variable RUNNER:
>
> AC_DEFUN([WRAP_CMD], [${RUNNER} $(which $1) $2])
> AT_CHECK(WRAP_CMD([server], [&]),       , [ignore])
> AT_CHECK(WRAP_CMD([client], [--cmd]),   , [expected-output])
> AT_CHECK(WRAP_CMD([killall], [server]), , [ignore])
>
> But this fails to kill the server simply because the actual command
> being executed by the first AT_CHECK with make check-valgrind is:
>
> valgrind --error-exitcode=1 --quiet ./server &
>
> So the simple killall call is no longer feasible.
>
> Now my question: is there a standard and approved way to start a
> background process in a GNU Autotest script and to kill it at or before
> the AT_CLEANUP ?
wouldn't you want the test itself to spin up/down the server as need
be ?  that way you can write multiple end-to-end tests and have them
all run in parallel.

i'm guessing your method described above also doesn't work when you
try to run all the tests in parallel.
-mike

_______________________________________________
Autoconf mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/autoconf

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Background processes in GNU Autotest

Olaf Mandel
Hello Mike,

On 22.06.2016 21:12, Mike Frysinger wrote:
> On 22 Jun 2016 11:03, Olaf Mandel wrote:
>> I am trying to use GNU Autotest (via AX_GNU_AUTOTEST()) to run
>> end-to-end tests on a network server. [...]
>>
>> AT_CHECK([server&],        , [ignore])
>> AT_CHECK([client --cmd]),  , [expected-output])
>> AT_CHECK([killall server], , [ignore])
>>
-Snipp-
>> Now I want to combine this with valgrind checking [...]
>>
>> valgrind --error-exitcode=1 --quiet ./server &
>>
-Snipp-
>
> wouldn't you want the test itself to spin up/down the server as need
> be ?  that way you can write multiple end-to-end tests and have them
> all run in parallel.
>
You mean my "client" program starting the server itself? Didn't think of
that... it would solve the cleanup question during testing. I see that
valgrind provides the --trace-children=yes option: I will have to try if
this gives suitable debugging information (I am mostly interested in
memory debugging the real server, not the client that was only written
for testing).

> i'm guessing your method described above also doesn't work when you
> try to run all the tests in parallel.

Right. This would already fail because the TCP port would be blocked and
there is no reliable way to feed back the port number from the server to
the client (and no: I can't use Unix sockets instead of TCP ports
without extending some external library). But here the suggestion of
forking the server from the client may help as well: the client can try
starting the server on different ports until one works.

Thank you for the suggestion,
Olaf


_______________________________________________
Autoconf mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/autoconf

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Background processes in GNU Autotest

Mike Frysinger
On 22 Jun 2016 22:52, Olaf Mandel wrote:

> On 22.06.2016 21:12, Mike Frysinger wrote:
> > On 22 Jun 2016 11:03, Olaf Mandel wrote:
> >> I am trying to use GNU Autotest (via AX_GNU_AUTOTEST()) to run
> >> end-to-end tests on a network server. [...]
> >>
> >> AT_CHECK([server&],        , [ignore])
> >> AT_CHECK([client --cmd]),  , [expected-output])
> >> AT_CHECK([killall server], , [ignore])
> >>
> -Snipp-
> >> Now I want to combine this with valgrind checking [...]
> >>
> >> valgrind --error-exitcode=1 --quiet ./server &
> >>
> -Snipp-
> >
> > wouldn't you want the test itself to spin up/down the server as need
> > be ?  that way you can write multiple end-to-end tests and have them
> > all run in parallel.
> >
> You mean my "client" program starting the server itself? Didn't think of
> that... it would solve the cleanup question during testing. I see that
> valgrind provides the --trace-children=yes option: I will have to try if
> this gives suitable debugging information (I am mostly interested in
> memory debugging the real server, not the client that was only written
> for testing).
>
> > i'm guessing your method described above also doesn't work when you
> > try to run all the tests in parallel.
>
> Right. This would already fail because the TCP port would be blocked and
> there is no reliable way to feed back the port number from the server to
> the client (and no: I can't use Unix sockets instead of TCP ports
> without extending some external library). But here the suggestion of
> forking the server from the client may help as well: the client can try
> starting the server on different ports until one works.
probing for an open port would be one way to handle it.  if you can
conditionally rely on Linux namespace features, you could create a
new network namespace for each test which would give you a completely
blank slate with localhost available.
$ unshare -Urn
# ifconfig lo up
# ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group
default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1
    link/sit 0.0.0.0 brd 0.0.0.0
3: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN group default qlen 1
    link/tunnel6 :: brd ::
# netstat -nat
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
-mike

_______________________________________________
Autoconf mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/autoconf

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Background processes in GNU Autotest

Olaf Mandel
Hello Mike,

Am 23.06.2016 um 07:19 schrieb Mike Frysinger:
> On 22 Jun 2016 22:52, Olaf Mandel wrote:
>> On 22.06.2016 21:12, Mike Frysinger wrote:
-Snipp-

>>> wouldn't you want the test itself to spin up/down the server as need
>>> be ?  that way you can write multiple end-to-end tests and have them
>>> all run in parallel.
>>>
>> You mean my "client" program starting the server itself? [...]
>> I will have to try if
>> this gives suitable debugging information (I am mostly interested in
>> memory debugging the real server, not the client that was only written
>> for testing).
>>
I finally got it working exactly as I want: thank you for the solution.

>>> i'm guessing your method described above also doesn't work when you
>>> try to run all the tests in parallel.
-Snipp-
>
> probing for an open port would be one way to handle it.  if you can
> conditionally rely on Linux namespace features, you could create a
> new network namespace for each test which would give you a completely
> blank slate with localhost available.
> $ unshare -Urn
-Snipp-

Decided not to do that because I didn't need the parallel testing on one
machine and because my default user does not have the required
permissions. But now that I learned of this I will keep it in mind
should the need arise: thanks for pointing it out.

Best regards,
Olaf


_______________________________________________
Autoconf mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/autoconf

signature.asc (836 bytes) Download Attachment
Loading...