Discussion:
Language choice for Lustre tests
Nathan Rutman
2012-10-23 18:36:02 UTC
Permalink
At LAD'12 we proposed a plan for improving the Lustre test framework as an important part of the Lustre quality story. One of the discussion points there was that the bash language of the current tests was lacking in a variety of areas. We're moving forward with this work but need community agreement on the best course.

Given the requirements and language options below, the reasonable choices rapidly diminish to a showdown between perl and python. I think we're leaning at this point toward perl, based on it's superior speed and inter-version compatibility. The final piece of the puzzle is the knowledge of existing Lustre test writers, so please chime in. (But note that "popularity" is the reason we chose bash the first time, and look where that got us...)

requirements
1. easy to use
2. strict structure
3. universally available
4. widely maintained
5. widely understood
6. good filesystem interface: posix API
7. fast - replace e.g. createmany with embedded function
8. operate remote instances
9. inter-version compatibility
options
bash - capable, but too flexible, easy to abuse
perl - forward compatible, universal, more widely understood, xperior, compact; hard to read later
posix::open, opendir, lseek, etc.
parallel::MPI
~2x faster than python
more version compatible
python - very clear structure, swig module for c inclusion; non-standard installations, support
os.open: all c flags
MPI bindings
tab/space requirements make remote editing more difficult
cucumber - ruby based, difficult deployment
java - easy deployment, dev environ, debugger, fast; must compile
ruby - compact


http://silicainsilico.wordpress.com/2012/03/26/switching-from-perl-to-python-speed/
http://tenser.typepad.com/tenser_said_the_tensor/2006/08/python_vs_perl_.html
http://opennomad.com/content/performance-different-scripting-languages-shell-v-perl-v-python-v-ruby
http://hentenaar.com/serendipity/index.php?/archives/27-Benchmark-PHP-vs.-Python-vs.-Perl-vs.-Ruby.html
Jacques-Charles Lafoucriere
2012-10-23 19:39:49 UTC
Permalink
hello all

I personally always had difficulties with perl ...
the code is generally very hard to read, and the language brings very
quickly to tricky optimization only experts understood
python is a more generic language, very easy to learn and if we make the
right basic object definitions, it will help a lot for future
improvement and code structure
at CEA our admin tools like shine or cluster-shell are python based and
this choice allows us in getting scallable/reliable tools
For example, the initial version of shine (~6 years ago) was perl based
and we were pleased to move to python for the new design

The target is lustre developers (ie kernel developers), it will be much
easier for them to learn python than perl.
it will also be easier for them to get the right "python" way of
programming than the right "perl" way of programming
(in both cases the worst way is to use the new language as bash, which
can arrive much quickly with perl than with python)

about python, what do you mean by non standard installation ?
if your python configuration is right, the local differences should be
hidden to the test framework

Bye

JC
Post by Nathan Rutman
At LAD'12 we proposed a plan for improving the Lustre test framework
as an important part of the Lustre quality story. One of the
discussion points there was that the bash language of the current
tests was lacking in a variety of areas. We're moving forward with
this work but need community agreement on the best course.
Given the requirements and language options below, the reasonable
choices rapidly diminish to a showdown between perl and python. I
think we're leaning at this point toward perl, based on it's superior
speed and inter-version compatibility. The final piece of the puzzle
is the knowledge of existing Lustre test writers, so please chime in.
(But note that "popularity" is the reason we chose bash the first
time, and look where that got us...)
requirements
1. easy to use
2. strict structure
3. universally available
4. widely maintained
5. widely understood
6. good filesystem interface: posix API
7. fast - replace e.g. createmany with embedded function
8. operate remote instances
9. inter-version compatibility
options
bash - capable, but too flexible, easy to abuse
perl - forward compatible, universal, more widely understood, xperior,
compact; hard to read later
posix::open, opendir, lseek, etc.
parallel::MPI
~2x faster than python
more version compatible
python - very clear structure, swig module for c inclusion;
non-standard installations, support
os.open: all c flags
MPI bindings
tab/space requirements make remote editing more difficult
cucumber - ruby based, difficult deployment
java - easy deployment, dev environ, debugger, fast; must compile
ruby - compact
http://silicainsilico.wordpress.com/2012/03/26/switching-from-perl-to-python-speed/
http://tenser.typepad.com/tenser_said_the_tensor/2006/08/python_vs_perl_.html
http://opennomad.com/content/performance-different-scripting-languages-shell-v-perl-v-python-v-ruby
http://hentenaar.com/serendipity/index.php?/archives/27-Benchmark-PHP-vs.-Python-vs.-Perl-vs.-Ruby.html
_______________________________________________
Lustre-devel mailing list
http://lists.lustre.org/mailman/listinfo/lustre-devel
Bruce Korb
2012-10-23 20:31:27 UTC
Permalink
Hi,

On Tue, Oct 23, 2012 at 12:39 PM, Jacques-Charles Lafoucriere <
Post by Jacques-Charles Lafoucriere
hello all
I personally always had difficulties with perl ...
the code is generally very hard to read, and the language brings very
quickly to tricky optimization only experts understood
At first blush, I had a hard time reading C code. I was looking at the
first implementation of
the C preprocessor and nobody had figured out how to write clean code yet.
Truly awful.
I also had a professor who claimed he could make any language look like
FORTRAN.
I believe him to this day. It takes effort, care, examples and some
training to write
clean comprehensible code in any language, beit C or bash or perl or python
or ruby.

it will also be easier for them to get the right "python" way of
Post by Jacques-Charles Lafoucriere
programming than the right "perl" way of programming
(in both cases the worst way is to use the new language as bash, which can
arrive much quickly with perl than with python)
In both cases, you need to find good examples of language usage,
characterize what makes the good examples good and then emulate that.
Perl can be clean and comprehensible. Even bash can be clean and
comprehensible.
And Python can be illegible. It takes effort.
Roman Grigoryev
2012-10-24 06:31:58 UTC
Permalink
Hi JC, all,

I should say more words about compatibility.

In comparing with server-only tools(which are often pretty good
maintained and controlled
on limited node set), test tools often work on wider sets. Minimally,
it should work
on some clients and also on clients with different versions then
servers, include clients in real clusters,
developers virtual clusters and so on.

As I know, Lustre users could have environment with Lustre latest
servers and 1.8 clients,
some companies use RedHat5.x clients and RedHat6.x(SL6.x) server, other
use Ubuntu.

RH5 has only python2.4, SL61 has python2.4 and python2.6, and looks
like only
last Fedora will have python3. In same time, Ubuntu says that from next
release want to have only Python 3.
Which version should we use and how long backward compatibility will be
supported by Python and distros//for
selected version/?/

Precision Python version could be installed from non standard repos,
compiled from sources also as used
"non standard installation". Last item also mean testing own
installation on wide set of distros. Also we should
remember about external Python libraries which also could be touched by
breaking legacy compatibility.

I think, testing system should be friendly to developers as possible and
pushing to install precision version while
one or more pythons already in os could not be the simplest solution.

Now Lustre tests compatibility for wide set of system is solved by shell
and standard utilities. Perl also has great
compatibility history, many scripts could work on latest version as 10
years ago. It is reason why I see Perl as
good decision.
From compatibility(and my) point of view, also Java is preferable
solution then Python. It has few good described ways
of installation, proved compatibility, great library managers (maven,
ant+ivy) and could support scripting
languages(JPython, JRuby and more).
But it needs more memory and pretty big start time.

Thanks,
Roman
hello all
.......
about python, what do you mean by non standard installation ?
if your python configuration is right, the local differences should be
hidden to the test framework
Bye
JC
Post by Nathan Rutman
At LAD'12 we proposed a plan for improving the Lustre test framework
as an important part of the Lustre quality story. One of the
discussion points there was that the bash language of the current
tests was lacking in a variety of areas. We're moving forward with
this work but need community agreement on the best course.
Given the requirements and language options below, the reasonable
choices rapidly diminish to a showdown between perl and python. I
think we're leaning at this point toward perl, based on it's superior
speed and inter-version compatibility. The final piece of the puzzle
is the knowledge of existing Lustre test writers, so please chime in.
(But note that "popularity" is the reason we chose bash the first
time, and look where that got us...)
requirements
1. easy to use
2. strict structure
3. universally available
4. widely maintained
5. widely understood
6. good filesystem interface: posix API
7. fast - replace e.g. createmany with embedded function
8. operate remote instances
9. inter-version compatibility
options
bash - capable, but too flexible, easy to abuse
perl - forward compatible, universal, more widely understood,
xperior, compact; hard to read later
posix::open, opendir, lseek, etc.
parallel::MPI
~2x faster than python
more version compatible
python - very clear structure, swig module for c inclusion;
non-standard installations, support
os.open: all c flags
MPI bindings
tab/space requirements make remote editing more difficult
cucumber - ruby based, difficult deployment
java - easy deployment, dev environ, debugger, fast; must compile
ruby - compact
http://silicainsilico.wordpress.com/2012/03/26/switching-from-perl-to-python-speed/
http://tenser.typepad.com/tenser_said_the_tensor/2006/08/python_vs_perl_.html
http://opennomad.com/content/performance-different-scripting-languages-shell-v-perl-v-python-v-ruby
http://hentenaar.com/serendipity/index.php?/archives/27-Benchmark-PHP-vs.-Python-vs.-Perl-vs.-Ruby.html
_______________________________________________
Lustre-devel mailing list
http://lists.lustre.org/mailman/listinfo/lustre-devel
Kilian Cavalotti
2012-10-24 07:18:34 UTC
Permalink
Hi Roman,

On Wed, Oct 24, 2012 at 8:31 AM, Roman Grigoryev
RH5 has only python2.4, SL61 has python2.4 and python2.6, and looks like
only last Fedora will have python3.
I agree that inter-version compatibility could be a problem with
Python. But to take this argument literally, the best way to avoid
compatibility issues is to use the widest-spread and most version
consistent language across supported distributions, ie. bash.
In same time, Ubuntu says that from next
release want to have only Python 3.
Since only RHEL, CentOS and SLES are supported, should we really care
about python versions in Ubuntu?
When installing Lustre in Ubuntu, users already have to do their own
packaging, so I'm not sure that having to install a specific version
of a scripting language would make much of a difference.
Now Lustre tests compatibility for wide set of system is solved by shell and
standard utilities. Perl also has great
compatibility history, many scripts could work on latest version as 10 years
ago. It is reason why I see Perl as
good decision.
So what's wrong with bash again? If it's just "too flexible, easy to
abuse", I'm afraid there's little in Perl to prevent falling into the
same pitfalls. :)

Cheers,
--
Kilian
Roman Grigoryev
2012-10-24 08:44:57 UTC
Permalink
Hi Kilian,

if you want I could try to explain why not bash from my point of view.

I agree that bash is language which could be used for test frameworks
and test-framework.sh prove it. But some
bash feature make current framework support pretty hard. For example, I
don't see good way to use bash unit tests for test-framework.sh

More structured requests:

1) Language features. Perl/Python/Java has many features which allows
simple write complex logic (OOP,AOP and so on)
2) Pretty wide set of libraries.
3) powerful and useful unit test frameworks.
4) support tools, e.g. inline documentation, schema generator based on
code, coverage collectiors, copyright checkers

Ubuntu is building by Intel
(http://build.whamcloud.com/job/lustre-b2_3/), so looks like it is
important platform.

Thanks,
Roman
Post by Kilian Cavalotti
Hi Roman,
On Wed, Oct 24, 2012 at 8:31 AM, Roman Grigoryev
RH5 has only python2.4, SL61 has python2.4 and python2.6, and looks like
only last Fedora will have python3.
I agree that inter-version compatibility could be a problem with
Python. But to take this argument literally, the best way to avoid
compatibility issues is to use the widest-spread and most version
consistent language across supported distributions, ie. bash.
In same time, Ubuntu says that from next
release want to have only Python 3.
Since only RHEL, CentOS and SLES are supported, should we really care
about python versions in Ubuntu?
When installing Lustre in Ubuntu, users already have to do their own
packaging, so I'm not sure that having to install a specific version
of a scripting language would make much of a difference.
Now Lustre tests compatibility for wide set of system is solved by shell and
standard utilities. Perl also has great
compatibility history, many scripts could work on latest version as 10 years
ago. It is reason why I see Perl as
good decision.
So what's wrong with bash again? If it's just "too flexible, easy to
abuse", I'm afraid there's little in Perl to prevent falling into the
same pitfalls. :)
Cheers,
DEGREMONT Aurelien
2012-10-24 16:30:43 UTC
Permalink
This post might be inappropriate. Click to display it.
Christopher J. Morrone
2012-10-24 20:32:15 UTC
Permalink
We can pick a completely new language so no gets their current favorite.
I nominate go (http://golang.org/). :)
Post by DEGREMONT Aurelien
Hello
I've developed in Perl and Python for several years in both languages. I
enjoyed both.
Regarding starting a new project, for testing framework, here is several
points which are the key points for choosing the right language.
- Longevity
Perl as a long history behind it, is available in all distros, that's
true. But Perl is there for compatibility for running projects which
were starting a long time ago. Nobody is starting a really big and fancy
project in Perl nowadays. All new hackers are only speaking of Python or
Ruby. You will not attract contributors with Perl. Perl 5 is very
compatible because no new changes is really added to this language.
And do not tell me that Perl 6 is coming. If Perl 6 is an option, then
there is no issue with Python 3 and forget your compatibility.
- Compat
Developping in Python, for a large number of environment means coding
for Python 2.4+. This will run nicely on Python up to 2.7
As long as only Python 2 only is concerned, compat for Python is fine.
But I agree that Python 3 is coming, and at long term, it should be
taken in account even if no major distro is using it right now.
(No Python 3 before RHEL7 or next Ubuntu LTS (14.04))
-Lustre is for system people
Lustre is developped by system guys, which mainly use vim or emacs to
develop in Lustre. Very few of them are using IDE or stuff like Maven.
Population developing Java with Maven is the exact opposite of guys
coding in C, for Kernel code. Moreover, JVM are awful to install
regarding a standard Python, Perl or Ruby interpreter in all major distro.
My choice would go for Python 2 with a clear path to migrate to Python 3
when needed (in 1 year)
Aurélien
Post by Roman Grigoryev
Hi Kilian,
if you want I could try to explain why not bash from my point of view.
I agree that bash is language which could be used for test frameworks
and test-framework.sh prove it. But some
bash feature make current framework support pretty hard. For example, I
don't see good way to use bash unit tests for test-framework.sh
1) Language features. Perl/Python/Java has many features which allows
simple write complex logic (OOP,AOP and so on)
2) Pretty wide set of libraries.
3) powerful and useful unit test frameworks.
4) support tools, e.g. inline documentation, schema generator based on
code, coverage collectiors, copyright checkers
Ubuntu is building by Intel
(http://build.whamcloud.com/job/lustre-b2_3/), so looks like it is
important platform.
Thanks,
Roman
Post by Kilian Cavalotti
Hi Roman,
On Wed, Oct 24, 2012 at 8:31 AM, Roman Grigoryev
RH5 has only python2.4, SL61 has python2.4 and python2.6, and looks like
only last Fedora will have python3.
I agree that inter-version compatibility could be a problem with
Python. But to take this argument literally, the best way to avoid
compatibility issues is to use the widest-spread and most version
consistent language across supported distributions, ie. bash.
In same time, Ubuntu says that from next
release want to have only Python 3.
Since only RHEL, CentOS and SLES are supported, should we really care
about python versions in Ubuntu?
When installing Lustre in Ubuntu, users already have to do their own
packaging, so I'm not sure that having to install a specific version
of a scripting language would make much of a difference.
Now Lustre tests compatibility for wide set of system is solved by shell and
standard utilities. Perl also has great
compatibility history, many scripts could work on latest version as 10 years
ago. It is reason why I see Perl as
good decision.
So what's wrong with bash again? If it's just "too flexible, easy to
abuse", I'm afraid there's little in Perl to prevent falling into the
same pitfalls. :)
Cheers,
_______________________________________________
Lustre-devel mailing list
http://lists.lustre.org/mailman/listinfo/lustre-devel
_______________________________________________
lustre-devel mailing list
http://lists.opensfs.org/listinfo.cgi/lustre-devel-opensfs.org
.
Kilian Cavalotti
2012-10-23 20:05:35 UTC
Permalink
Hi,

On Tue, Oct 23, 2012 at 8:36 PM, Nathan Rutman
Post by Nathan Rutman
Given the requirements and language options below, the reasonable choices
rapidly diminish to a showdown between perl and python. I think we're
leaning at this point toward perl, based on it's superior speed and
inter-version compatibility.
I myself tend towards Python, but that's perfectly knowing this
couldn't be very different from the recurring, almost religious "emacs
vs vim" or "kde vs gnome" flame wars. It's a matter of taste, and it's
almost pointless to try to convince somebody to adopt the other one
once their choice is made.

I would only recommend reading this :
http://www.strombergers.com/python/python_perl_slurp_file.html.

And the obligatory XKCD: http://xkcd.com/353/

Cheers,
--
Kilian
Gearing, Chris
2012-10-24 20:02:08 UTC
Permalink
Nathan,

I'm not 100% sure what you are proposing here, your LAD presentation suggested a 'tune-up' of the current test framework rather than a complete re-write. Which of the two are we discussing?

Which ever you are intending to undertake I think it is vital that before decisions are made on things such as languages a clear specification/definition of the activity is created and distributed. I am in fact working with Roman to bring such a document to the working group on the tune-up, making me somewhat confused about what is proposed here.

Going back to your LAD presentation for the tune-up your intention is to write some libraries to better address the requirements of the framework than is possible in bash, but those libraries would be called from the existing bash tests. This illustrates that the language used to develop the framework may not be the same language used for writing the tests, in fact I'm guessing that for both a tune-up and rewrite the requirements placed on the framework writer are different than those place on the test writer and so the languages required might well be different, and in fact if this is a tune-up then surely the tests must continue to be written in bash even if the libraries are in something more applicable, we do not want a single framework with mixed languages (do we?)

So to return to my original question could you please provide some greater depth of insight as to what you are embarking upon, and then we can make more objective input as the language required.

On the complete-rewrite topic, I will post the slides I sent you before LAD to the Whamcloud wiki and then provide a link to this list so that people can share our thoughts. (Network access prevents me doing this immediately)

Regards

Chris
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
Nathan Rutman
2012-10-24 22:05:04 UTC
Permalink
Post by Gearing, Chris
Nathan,
I'm not 100% sure what you are proposing here, your LAD presentation suggested a 'tune-up' of the current test framework rather than a complete re-write. Which of the two are we discussing?
Both...
Post by Gearing, Chris
Which ever you are intending to undertake I think it is vital that before decisions are made on things such as languages a clear specification/definition of the activity is created and distributed. I am in fact working with Roman to bring such a document to the working group on the tune-up, making me somewhat confused about what is proposed here.
Going back to your LAD presentation for the tune-up your intention is to write some libraries to better address the requirements of the framework than is possible in bash, but those libraries would be called from the existing bash tests. This illustrates that the language used to develop the framework may not be the same language used for writing the tests, in fact I'm guessing that for both a tune-up and rewrite the requirements placed on the framework writer are different than those place on the test writer and so the languages required might well be different,
and in fact if this is a tune-up then surely the tests must continue to be written in bash even if the libraries are in something more applicable, we do not want a single framework with mixed languages (do we?)
The ultimate goal is to produce higher-quality, more robust, well-controlled, and safer tests. To that end, I think the eventual language of the tests must change to something meeting the requirements I stated before:
1. easy to use
2. strict structure
3. universally available/portable
4. widely maintained
5. widely understood
6. fully-featured filesystem interface: posix API
7. fast - replace e.g. createmany with embedded function
8. operate remote instances
9. inter-version compatibility
The requirements on the framework language are more relaxed, but for ease of development and developer sanity, I assume that the framework language should match the test language. So I'm using the test language as the requirements driver, and to gage community preference for that test language.

Based on the responses so far, it seems that there is a fairly clear preference for Python as a test language, and so I'll propose that Python should be used shorter-term to start replacing test-framework.
Post by Gearing, Chris
So to return to my original question could you please provide some greater depth of insight as to what you are embarking upon, and then we can make more objective input as the language required.
On the complete-rewrite topic, I will post the slides I sent you before LAD to the Whamcloud wiki and then provide a link to this list so that people can share our thoughts. (Network access prevents me doing this immediately)
Regards
Chris
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
_______________________________________________
Lustre-devel mailing list
http://lists.lustre.org/mailman/listinfo/lustre-devel
Colin Faber
2012-10-24 22:09:43 UTC
Permalink
My vote here is perl, there's a few [lustre] tools already written in
perl, it also meets all of the stated requirements. Yes, perl can look
like core dump. Yes perl can be easily obfuscated and impossibly hard to
understand (even for expert class developers). However the enormous
range of module support, as well as the maturity and cross platform
portability make it ideal for this task.

-cf
Post by Nathan Rutman
Post by Gearing, Chris
Nathan,
I'm not 100% sure what you are proposing here, your LAD presentation
suggested a 'tune-up' of the current test framework rather than a
complete re-write. Which of the two are we discussing?
Both...
Post by Gearing, Chris
Which ever you are intending to undertake I think it is vital that
before decisions are made on things such as languages a clear
specification/definition of the activity is created and distributed.
I am in fact working with Roman to bring such a document to the
working group on the tune-up, making me somewhat confused about what
is proposed here.
Going back to your LAD presentation for the tune-up your intention is
to write some libraries to better address the requirements of the
framework than is possible in bash, but those libraries would be
called from the existing bash tests. This illustrates that the
language used to develop the framework may not be the same language
used for writing the tests, in fact I'm guessing that for both a
tune-up and rewrite the requirements placed on the framework writer
are different than those place on the test writer and so the
languages required might well be different,
and in fact if this is a tune-up then surely the tests must continue
to be written in bash even if the libraries are in something more
applicable, we do not want a single framework with mixed languages
(do we?)
The ultimate goal is to produce higher-quality, more robust,
well-controlled, and safer tests. To that end, I think the eventual
language of the tests must change to something meeting the
1. easy to use
2. strict structure
3. universally available/portable
4. widely maintained
5. widely understood
6. fully-featured filesystem interface: posix API
7. fast - replace e.g. createmany with embedded function
8. operate remote instances
9. inter-version compatibility
The requirements on the framework language are more relaxed, but for
ease of development and developer sanity, I assume that the framework
language should match the test language. So I'm using the test
language as the requirements driver, and to gage community preference
for that test language.
Based on the responses so far, it seems that there is a fairly clear
preference for Python as a test language, and so I'll propose that
Python should be used shorter-term to start replacing test-framework.
Post by Gearing, Chris
So to return to my original question could you please provide some
greater depth of insight as to what you are embarking upon, and then
we can make more objective input as the language required.
On the complete-rewrite topic, I will post the slides I sent you
before LAD to the Whamcloud wiki and then provide a link to this list
so that people can share our thoughts. (Network access prevents me
doing this immediately)
Regards
Chris
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
_______________________________________________
Lustre-devel mailing list
http://lists.lustre.org/mailman/listinfo/lustre-devel
_______________________________________________
Lustre-devel mailing list
http://lists.lustre.org/mailman/listinfo/lustre-devel
Brian Behlendorf
2012-10-25 17:23:13 UTC
Permalink
Post by Nathan Rutman
Post by Gearing, Chris
Nathan,
I'm not 100% sure what you are proposing here, your LAD presentation suggested a 'tune-up' of the current test framework rather than a complete re-write. Which of the two are we discussing?
Both...
The requirements on the framework language are more relaxed, but for ease of development and developer sanity, I assume that the framework language should match the test language. So I'm using the test language as the requirements driver, and to gage community preference for that test language.
Before embarking on building yet another new and custom framework for
Lustre we should evaluate some existing frameworks. For example, the
Autotest project was specifically designed to test the Linux kernel.
It's open source, looks active, is flexible, and there is detailed
documentation on how to write tests. Plus it was designed specifically
for testing the kernel so there are likely existing file system tests.

http://autotest.github.com/

"Autotest is a framework for fully automated testing. It is
designed primarily to test the Linux kernel, though it is useful
for many other functions such as qualifying new hardware. It's an
open-source project under the GPL and is used and developed by a
number of organizations, including Google, IBM, Red Hat, and many
others."
Post by Nathan Rutman
Based on the responses so far, it seems that there is a fairly clear preference for Python as a test language, and so I'll propose that Python should be used shorter-term to start replacing test-framework.
If we decide the Autotest framework is a good fit then we'll want to
write the tests in python to be consistent with the framework language.
However, for a first cut it looks like you could use the existing bash
tests largely unmodified.
--
Thanks,
Brian
Nathan Rutman
2012-10-25 18:04:55 UTC
Permalink
There are several levels of "test framework" that are involved with automated testing of Lustre:
unit tests themselves, written in bash, e.g. sanity 1a
test-framework.sh & friends, which provides some support/setup/communal library for unit tests
automated test systems, which automatically execute the unit tests, record and report the results.

I'm only speaking about the language for the first of these items at the moment.
For more context, including the motivation for changing things, see my LAD'12 presentation (more narrowly focused) on the OpenSFS Wiki.
Post by Brian Behlendorf
Post by Nathan Rutman
Post by Gearing, Chris
Nathan,
I'm not 100% sure what you are proposing here, your LAD presentation suggested a 'tune-up' of the current test framework rather than a complete re-write. Which of the two are we discussing?
Both...
The requirements on the framework language are more relaxed, but for ease of development and developer sanity, I assume that the framework language should match the test language. So I'm using the test language as the requirements driver, and to gage community preference for that test language.
Before embarking on building yet another new and custom framework for
Lustre we should evaluate some existing frameworks. For example, the
Autotest project was specifically designed to test the Linux kernel.
It's open source, looks active, is flexible, and there is detailed
documentation on how to write tests. Plus it was designed specifically
for testing the kernel so there are likely existing file system tests.
http://autotest.github.com/
"Autotest is a framework for fully automated testing. It is
designed primarily to test the Linux kernel, though it is useful
for many other functions such as qualifying new hardware. It's an
open-source project under the GPL and is used and developed by a
number of organizations, including Google, IBM, Red Hat, and many
others."
Post by Nathan Rutman
Based on the responses so far, it seems that there is a fairly clear preference for Python as a test language, and so I'll propose that Python should be used shorter-term to start replacing test-framework.
If we decide the Autotest framework is a good fit then we'll want to
write the tests in python to be consistent with the framework language.
However, for a first cut it looks like you could use the existing bash
tests largely unmodified.
--
Thanks,
Brian
_______________________________________________
Lustre-devel mailing list
http://lists.lustre.org/mailman/listinfo/lustre-devel
Prakash Surya
2012-10-25 21:17:44 UTC
Permalink
Post by Nathan Rutman
There are several levels of "test framework" that are involved with
* unit tests themselves, written in bash, e.g. sanity 1a
What do you mean by "unit tests" here? Because as I understand it,
Lustre has no unit testing at all. It is all functional testing,
which is related but completely different.
Post by Nathan Rutman
* test-framework.sh & friends, which provides some
support/setup/communal library for unit tests
Again, unit tests? I really wish there _were_ unit tests, but aren't,
and I haven't seen any discussion about creating them.
Post by Nathan Rutman
* automated test systems, which automatically execute the unit tests,
record and report the results.
I don't quite understand why there is so much discussion regarding a new
testing framework specifically designed for testing Lustre. Is there
really nothing already out there we can make use of?

Brian linked to the Autotest framework, does that not do what we want?
If it doesn't work, why not? And how is our home brewed solution going
to differ? Are there any other alternatives out there? I don't see
benefit in spending development on a new framework if it's unnecessary.

There is a lot of discussion regarding a tool (the language) and little
involving the problem (lack of useful functional and unit tests to
properly stress the system). Pick something, move on, and start
improving the codes stability.

And if somebody *really* wants to build a new testing framework, have it
live out of tree. Create it as a standalone project which interfaces
with Lustre.
--
Cheers, Prakash
Post by Nathan Rutman
I'm only speaking about the language for the first of these items at the moment.
For more context, including the motivation for changing things, see my
LAD'12 presentation (more narrowly focused) on the [1]OpenSFS Wiki.
On Oct 25, 2012, at 10:23 AM, Brian Behlendorf
On Oct 24, 2012, at 1:02 PM, "Gearing, Chris"
Nathan,
I'm not 100% sure what you are proposing here, your LAD presentation
suggested a 'tune-up' of the current test framework rather than a
complete re-write. Which of the two are we discussing?
Both...
The requirements on the framework language are more relaxed, but for
ease of development and developer sanity, I assume that the
framework language should match the test language. So I'm using the
test language as the requirements driver, and to gage community
preference for that test language.
Before embarking on building yet another new and custom framework for
Lustre we should evaluate some existing frameworks. For example, the
Autotest project was specifically designed to test the Linux kernel.
It's open source, looks active, is flexible, and there is detailed
documentation on how to write tests. Plus it was designed specifically
for testing the kernel so there are likely existing file system tests.
[4]http://autotest.github.com/
"Autotest is a framework for fully automated testing. It is
designed primarily to test the Linux kernel, though it is useful
for many other functions such as qualifying new hardware. It's an
open-source project under the GPL and is used and developed by a
number of organizations, including Google, IBM, Red Hat, and many
others."
Based on the responses so far, it seems that there is a fairly clear
preference for Python as a test language, and so I'll propose that
Python should be used shorter-term to start replacing
test-framework.
If we decide the Autotest framework is a good fit then we'll want to
write the tests in python to be consistent with the framework language.
However, for a first cut it looks like you could use the existing bash
tests largely unmodified.
--
Thanks,
Brian
_______________________________________________
Lustre-devel mailing list
http://lists.lustre.org/mailman/listinfo/lustre-devel
References
1. http://www.opensfs.org/foswiki/bin/view/Lustre/TestingImprovements
4. http://autotest.github.com/
_______________________________________________
Lustre-devel mailing list
http://lists.lustre.org/mailman/listinfo/lustre-devel
Nathan Rutman
2012-10-25 21:36:02 UTC
Permalink
Post by Prakash Surya
Post by Nathan Rutman
There are several levels of "test framework" that are involved with
* unit tests themselves, written in bash, e.g. sanity 1a
What do you mean by "unit tests" here? Because as I understand it,
Lustre has no unit testing at all. It is all functional testing,
which is related but completely different.
Let's please not get into this. I just meant the individual tests.
Post by Prakash Surya
Post by Nathan Rutman
* test-framework.sh & friends, which provides some
support/setup/communal library for unit tests
* automated test systems, which automatically execute the unit tests,
record and report the results.
I don't quite understand why there is so much discussion regarding a new
testing framework specifically designed for testing Lustre. Is there
really nothing already out there we can make use of?
Brian linked to the Autotest framework, does that not do what we want?
That covers the third meaning, but not the first two. I'm only concerned about the
first two at the moment.
Post by Prakash Surya
If it doesn't work, why not?
It doesn't contain a Lustre-specific functional library, it doesn't safely limit the tests
that may be run on a specific system, it doesn't handle interoperability testing, etc.
Post by Prakash Surya
And how is our home brewed solution going
to differ? Are there any other alternatives out there? I don't see
benefit in spending development on a new framework if it's unnecessary.
There is a lot of discussion regarding a tool (the language) and little
involving the problem (lack of useful functional and unit tests to
properly stress the system). Pick something, move on, and start
improving the codes stability.
That was what I was trying to do.
Post by Prakash Surya
And if somebody *really* wants to build a new testing framework, have it
live out of tree. Create it as a standalone project which interfaces
with Lustre.
As stated in my presentation, that is part of the plan.
Post by Prakash Surya
--
Cheers, Prakash
Post by Nathan Rutman
I'm only speaking about the language for the first of these items at the moment.
For more context, including the motivation for changing things, see my
LAD'12 presentation (more narrowly focused) on the [1]OpenSFS Wiki.
On Oct 25, 2012, at 10:23 AM, Brian Behlendorf
On Oct 24, 2012, at 1:02 PM, "Gearing, Chris"
Nathan,
I'm not 100% sure what you are proposing here, your LAD presentation
suggested a 'tune-up' of the current test framework rather than a
complete re-write. Which of the two are we discussing?
Both...
The requirements on the framework language are more relaxed, but for
ease of development and developer sanity, I assume that the
framework language should match the test language. So I'm using the
test language as the requirements driver, and to gage community
preference for that test language.
Before embarking on building yet another new and custom framework for
Lustre we should evaluate some existing frameworks. For example, the
Autotest project was specifically designed to test the Linux kernel.
It's open source, looks active, is flexible, and there is detailed
documentation on how to write tests. Plus it was designed specifically
for testing the kernel so there are likely existing file system tests.
[4]http://autotest.github.com/
"Autotest is a framework for fully automated testing. It is
designed primarily to test the Linux kernel, though it is useful
for many other functions such as qualifying new hardware. It's an
open-source project under the GPL and is used and developed by a
number of organizations, including Google, IBM, Red Hat, and many
others."
Based on the responses so far, it seems that there is a fairly clear
preference for Python as a test language, and so I'll propose that
Python should be used shorter-term to start replacing
test-framework.
If we decide the Autotest framework is a good fit then we'll want to
write the tests in python to be consistent with the framework language.
However, for a first cut it looks like you could use the existing bash
tests largely unmodified.
--
Thanks,
Brian
_______________________________________________
Lustre-devel mailing list
http://lists.lustre.org/mailman/listinfo/lustre-devel
References
1. http://www.opensfs.org/foswiki/bin/view/Lustre/TestingImprovements
4. http://autotest.github.com/
_______________________________________________
Lustre-devel mailing list
http://lists.lustre.org/mailman/listinfo/lustre-devel
Nathan Rutman
2012-10-25 21:50:05 UTC
Permalink
Thank you all for your input; I think I've got what I need for now.
I'll continue to raise discussion points here as our plans solidify.
Prakash Surya
2012-10-25 22:13:09 UTC
Permalink
Post by Nathan Rutman
Post by Prakash Surya
Post by Nathan Rutman
There are several levels of "test framework" that are involved with
* unit tests themselves, written in bash, e.g. sanity 1a
What do you mean by "unit tests" here? Because as I understand it,
Lustre has no unit testing at all. It is all functional testing,
which is related but completely different.
Let's please not get into this. I just meant the individual tests.
OK, I just wanted to make sure we're on the same page. I would *love*
to see unit testing incorporated into the test suite. I think that would
greatly benefit the stability of the project.
Post by Nathan Rutman
Post by Prakash Surya
Post by Nathan Rutman
* test-framework.sh & friends, which provides some
support/setup/communal library for unit tests
* automated test systems, which automatically execute the unit tests,
record and report the results.
I don't quite understand why there is so much discussion regarding a new
testing framework specifically designed for testing Lustre. Is there
really nothing already out there we can make use of?
Brian linked to the Autotest framework, does that not do what we want?
That covers the third meaning, but not the first two. I'm only concerned about the
first two at the moment.
Well, I think it spills into the first and second as well.

1. Since Autotest is already implemented in python, it would make sense to use
that as the language for the individual tests as well.

2. And it _may_ deal with set up of tests. But I haven't looked into
Autotest enough to know its limitations. The shared library would, of
course, be specific to Lustre.
Post by Nathan Rutman
Post by Prakash Surya
If it doesn't work, why not?
It doesn't contain a Lustre-specific functional library, it doesn't safely limit the tests
that may be run on a specific system, it doesn't handle interoperability testing, etc.
Right, the test library will be specific to Lustre.

Limiting the number of tests to run, which ones run, and interoperability
might be partially handled by whatever automated framework is used. Of
course it might not, as well.
Post by Nathan Rutman
Post by Prakash Surya
And how is our home brewed solution going
to differ? Are there any other alternatives out there? I don't see
benefit in spending development on a new framework if it's unnecessary.
There is a lot of discussion regarding a tool (the language) and little
involving the problem (lack of useful functional and unit tests to
properly stress the system). Pick something, move on, and start
improving the codes stability.
That was what I was trying to do.
Great! :). I'm really looking forward to seeing a solid testing
infrastructure implemented.
Post by Nathan Rutman
Post by Prakash Surya
And if somebody *really* wants to build a new testing framework, have it
live out of tree. Create it as a standalone project which interfaces
with Lustre.
As stated in my presentation, that is part of the plan.
Ah, I didn't catch that part. Good to know.
--
Cheers, Prakash
Post by Nathan Rutman
Post by Prakash Surya
--
Cheers, Prakash
Post by Nathan Rutman
I'm only speaking about the language for the first of these items at the moment.
For more context, including the motivation for changing things, see my
LAD'12 presentation (more narrowly focused) on the [1]OpenSFS Wiki.
On Oct 25, 2012, at 10:23 AM, Brian Behlendorf
On Oct 24, 2012, at 1:02 PM, "Gearing, Chris"
Nathan,
I'm not 100% sure what you are proposing here, your LAD presentation
suggested a 'tune-up' of the current test framework rather than a
complete re-write. Which of the two are we discussing?
Both...
The requirements on the framework language are more relaxed, but for
ease of development and developer sanity, I assume that the
framework language should match the test language. So I'm using the
test language as the requirements driver, and to gage community
preference for that test language.
Before embarking on building yet another new and custom framework for
Lustre we should evaluate some existing frameworks. For example, the
Autotest project was specifically designed to test the Linux kernel.
It's open source, looks active, is flexible, and there is detailed
documentation on how to write tests. Plus it was designed specifically
for testing the kernel so there are likely existing file system tests.
[4]http://autotest.github.com/
"Autotest is a framework for fully automated testing. It is
designed primarily to test the Linux kernel, though it is useful
for many other functions such as qualifying new hardware. It's an
open-source project under the GPL and is used and developed by a
number of organizations, including Google, IBM, Red Hat, and many
others."
Based on the responses so far, it seems that there is a fairly clear
preference for Python as a test language, and so I'll propose that
Python should be used shorter-term to start replacing
test-framework.
If we decide the Autotest framework is a good fit then we'll want to
write the tests in python to be consistent with the framework language.
However, for a first cut it looks like you could use the existing bash
tests largely unmodified.
--
Thanks,
Brian
_______________________________________________
Lustre-devel mailing list
http://lists.lustre.org/mailman/listinfo/lustre-devel
References
1. http://www.opensfs.org/foswiki/bin/view/Lustre/TestingImprovements
4. http://autotest.github.com/
_______________________________________________
Lustre-devel mailing list
http://lists.lustre.org/mailman/listinfo/lustre-devel
Roman Grigoryev
2012-10-25 22:19:46 UTC
Permalink
This post might be inappropriate. Click to display it.
Gearing, Chris
2012-10-25 18:20:20 UTC
Permalink
Hi,
Post by Nathan Rutman
Based on the responses so far, it seems that there is a fairly clear preference for Python as a test language, and so I'll propose that Python should be used shorter-term to start replacing test-framework.
I don't think that is the case Nathan, because unless I am mistaken

Jacques-Charles, Kilian Cavalotti and if it is a binary choice today my preference is Python (but please see below)
Bruce was neutral
Roman and Colin (after your email) expressed Perl as a choice.

So the preference would seem to be for Python, counting by people and organization although obviously it's a statistically suspect poll.

Chris

NOTES OF MY VOTE: Given the importance of the topic I believe a much broader piece of analysis needs to be done to select the right language, also if this is for an existing framework update the tests must all be in bash or all converted to the new language of choice.

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
Nathan Rutman
2012-10-25 19:24:56 UTC
Permalink
Hi,
Post by Nathan Rutman
Based on the responses so far, it seems that there is a fairly clear preference for Python as a test language, and so I'll propose that Python should be used shorter-term to start replacing test-framework.
I don’t think that is the case Nathan, because unless I am mistaken
Jacques-Charles, Kilian Cavalotti and if it is a binary choice today my preference is Python (but please see below)
Aurelien - python 2
Bruce was neutral
Roman and Colin (after your email) expressed Perl as a choice.
So the preference would seem to be for Python, counting by people and organization although obviously it’s a statistically suspect poll.
Indeed. As I stated before, a poll is not the best way to choose the language; it's just one data point. But as a) we must choose something to start making any headway toward improvements, and b) bash doesn't meet the requirements, I'm trying to come up with some kind of baseline.
Chris
NOTES OF MY VOTE: Given the importance of the topic I believe a much broader piece of analysis needs to be done to select the right language,
I'm just trying to keep things moving from theoretical optimum to practical conclusion. We feel a need today for much better Lustre testing, and there's a huge pile of work waiting, and so I feel we must move forward.
also if this is for an existing framework update the tests must all be in bash or all converted to the new language of choice.
Short term the idea would be to translate one of the test suites, for example conf-sanity, to the new language/framework. The two systems (old and new) would live side-by-side for some period. New tests should be implemented under the new system (e.g. sanity2). Other old tests could be converted piecemeal subsequently.
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
Christopher J. Morrone
2012-10-25 21:13:46 UTC
Permalink
It really sounds like you are putting the cart before the horse.
Picking a favorite language is probably a bad way to go. First we
should figure out why we're changing things, and what the scope and
goals of the new design are.

Brian Behlendorf pointed out the kernel Autotest system as an option,
which sounds worthy of investigation to me. If we were to decide on
Perl today, that would make working with that Autotest more difficult,
because that is all done in Python. But if choose Python and then try
to use an entirely Visual Basic based framework (Ha! I kid!) we also
probably made the wrong choice.

In other words, it is important to know something about where we are
going before we pick a language.
Post by Nathan Rutman
Post by Nathan Rutman
Hi,
Post by Nathan Rutman
Based on the responses so far, it seems that there is a fairly clear
preference for Python as a test language, and so I'll propose that
Python should be used shorter-term to start replacing test-framework.
I don’t think that is the case Nathan, because unless I am mistaken
Jacques-Charles, Kilian Cavalotti and if it is a binary choice today
my preference is Python (but please see below)
Aurelien - python 2
Post by Nathan Rutman
Bruce was neutral
Roman and Colin (after your email) expressed Perl as a choice.
So the preference would seem to be for Python, counting by people and
organization although obviously it’s a statistically suspect poll.
Indeed. As I stated before, a poll is not the best way to choose the
language; it's just one data point. But as a) we must choose something
to start making any headway toward improvements, and b) bash doesn't
meet the requirements, I'm trying to come up with some kind of baseline.
Post by Nathan Rutman
Chris
NOTES OF MY VOTE: Given the importance of the topic I believe a much
broader piece of analysis needs to be done to select the right language,
I'm just trying to keep things moving from theoretical optimum to
practical conclusion. We feel a need today for much better Lustre
testing, and there's a huge pile of work waiting, and so I feel we must
move forward.
Post by Nathan Rutman
also if this is for an existing framework update the tests must all be
in bash or all converted to the new language of choice.
Short term the idea would be to translate one of the test suites, for
example conf-sanity, to the new language/framework. The two systems
(old and new) would live side-by-side for some period. New tests should
be implemented under the new system (e.g. sanity2). Other old tests
could be converted piecemeal subsequently.
Post by Nathan Rutman
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
_______________________________________________
Lustre-devel mailing list
http://lists.lustre.org/mailman/listinfo/lustre-devel
Nathan Rutman
2012-10-25 21:28:52 UTC
Permalink
It really sounds like you are putting the cart before the horse. Picking a favorite language is probably a bad way to go. First we should figure out why we're changing things, and what the scope and goals of the new design are.
Brian Behlendorf pointed out the kernel Autotest system as an option, which sounds worthy of investigation to me. If we were to decide on Perl today, that would make working with that Autotest more difficult, because that is all done in Python. But if choose Python and then try to use an entirely Visual Basic based framework (Ha! I kid!) we also probably made the wrong choice.
In other words, it is important to know something about where we are going before we pick a language.
For more context, including the motivation for changing things, see my LAD'12 presentation (more narrowly focused) on the OpenSFS Wiki.
As promised: http://wiki.whamcloud.com/download/attachments/7634975/new+test+framework.pdf
Picking a language is just one part of it, and definitely not the most important part. I'll be posting more on some other proposed changes in the future.
Post by Nathan Rutman
Post by Gearing, Chris
Nathan,
I'm not 100% sure what you are proposing here, your LAD presentation suggested a 'tune-up' of the current test framework rather than a complete re-write. Which of the two are we discussing?
Both...
The requirements on the framework language are more relaxed, but for ease of development and developer sanity, I assume that the framework language should match the test language. So I'm using the test language as the requirements driver, and to gage community preference for that test language.
Before embarking on building yet another new and custom framework for
Lustre we should evaluate some existing frameworks. For example, the
Autotest project was specifically designed to test the Linux kernel.
It's open source, looks active, is flexible, and there is detailed
documentation on how to write tests. Plus it was designed specifically
for testing the kernel so there are likely existing file system tests.
http://autotest.github.com/
"Autotest is a framework for fully automated testing. It is
designed primarily to test the Linux kernel, though it is useful
for many other functions such as qualifying new hardware. It's an
open-source project under the GPL and is used and developed by a
number of organizations, including Google, IBM, Red Hat, and many
others."
Post by Nathan Rutman
Based on the responses so far, it seems that there is a fairly clear preference for Python as a test language, and so I'll propose that Python should be used shorter-term to start replacing test-framework.
If we decide the Autotest framework is a good fit then we'll want to
write the tests in python to be consistent with the framework language.
However, for a first cut it looks like you could use the existing bash
tests largely unmodified.
--
Thanks,
Brian
_______________________________________________
Lustre-devel mailing list
http://lists.lustre.org/mailman/listinfo/lustre-devel
Post by Nathan Rutman
Post by Gearing, Chris
Hi,
Post by Nathan Rutman
Based on the responses so far, it seems that there is a fairly clear
preference for Python as a test language, and so I'll propose that
Python should be used shorter-term to start replacing test-framework.
I don’t think that is the case Nathan, because unless I am mistaken
Jacques-Charles, Kilian Cavalotti and if it is a binary choice today
my preference is Python (but please see below)
Aurelien - python 2
Post by Gearing, Chris
Bruce was neutral
Roman and Colin (after your email) expressed Perl as a choice.
So the preference would seem to be for Python, counting by people and
organization although obviously it’s a statistically suspect poll.
Indeed. As I stated before, a poll is not the best way to choose the
language; it's just one data point. But as a) we must choose something
to start making any headway toward improvements, and b) bash doesn't
meet the requirements, I'm trying to come up with some kind of baseline.
Post by Gearing, Chris
Chris
NOTES OF MY VOTE: Given the importance of the topic I believe a much
broader piece of analysis needs to be done to select the right language,
I'm just trying to keep things moving from theoretical optimum to
practical conclusion. We feel a need today for much better Lustre
testing, and there's a huge pile of work waiting, and so I feel we must
move forward.
Post by Gearing, Chris
also if this is for an existing framework update the tests must all be
in bash or all converted to the new language of choice.
Short term the idea would be to translate one of the test suites, for
example conf-sanity, to the new language/framework. The two systems
(old and new) would live side-by-side for some period. New tests should
be implemented under the new system (e.g. sanity2). Other old tests
could be converted piecemeal subsequently.
Post by Gearing, Chris
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
_______________________________________________
Lustre-devel mailing list
http://lists.lustre.org/mailman/listinfo/lustre-devel
Prakash Surya
2012-10-25 21:52:43 UTC
Permalink
On Oct 25, 2012, at 2:13 PM, Christopher J. Morrone
It really sounds like you are putting the cart before the horse.
Picking a favorite language is probably a bad way to go. First we
should figure out why we're changing things, and what the scope and
goals of the new design are.
Brian Behlendorf pointed out the kernel Autotest system as an
option, which sounds worthy of investigation to me. If we were to
decide on Perl today, that would make working with that Autotest
more difficult, because that is all done in Python. But if choose
Python and then try to use an entirely Visual Basic based framework
(Ha! I kid!) we also probably made the wrong choice.
In other words, it is important to know something about where we are
going before we pick a language.
For more context, including the motivation for changing things, see my
LAD'12 presentation (more narrowly focused) on the [2]OpenSFS Wiki.
Commenting on your presentation slides (slide 3 specifically) as I think
it relates to this discusstion:

* "Bash languages is not modular or portable"

Why do you think this? Bash is extremely modular using the "source"
command. Write code in a file, source it in another file, and there you
go. The "sourced" file is a separate reusable module, is it not? Sure it
might be global in the namespace, but that's another issue.

Not portable? Really? Bash is available on just about every Linux
distribution that comes to mind. Definitely every one Lustre cares
about.

* "Test scripts are not coherent"

That is not Bash's fault. Bad code can be written in any language. It's
up to the programmer to write "coherent" test scripts.

How would switching languages improve on either of these two fronts?
--
Cheers, Prakash
[3]http://wiki.whamcloud.com/download/attachments/7634975/new+test+f
ramework.pdf
Picking a language is just one part of it, and definitely not the most
important part. I'll be posting more on some other proposed changes in
the future.
On Oct 25, 2012, at 10:23 AM, Brian Behlendorf
On Oct 24, 2012, at 1:02 PM, "Gearing, Chris"
Nathan,
I'm not 100% sure what you are proposing here, your LAD presentation
suggested a 'tune-up' of the current test framework rather than a
complete re-write. Which of the two are we discussing?
Both...
The requirements on the framework language are more relaxed, but for
ease of development and developer sanity, I assume that the
framework language should match the test language. So I'm using the
test language as the requirements driver, and to gage community
preference for that test language.
Before embarking on building yet another new and custom framework for
Lustre we should evaluate some existing frameworks. For example, the
Autotest project was specifically designed to test the Linux kernel.
It's open source, looks active, is flexible, and there is detailed
documentation on how to write tests. Plus it was designed specifically
for testing the kernel so there are likely existing file system tests.
[6]http://autotest.github.com/
"Autotest is a framework for fully automated testing. It is
designed primarily to test the Linux kernel, though it is useful
for many other functions such as qualifying new hardware. It's an
open-source project under the GPL and is used and developed by a
number of organizations, including Google, IBM, Red Hat, and many
others."
Based on the responses so far, it seems that there is a fairly clear
preference for Python as a test language, and so I'll propose that
Python should be used shorter-term to start replacing
test-framework.
If we decide the Autotest framework is a good fit then we'll want to
write the tests in python to be consistent with the framework language.
However, for a first cut it looks like you could use the existing bash
tests largely unmodified.
--
Thanks,
Brian
_______________________________________________
Lustre-devel mailing list
[8]http://lists.lustre.org/mailman/listinfo/lustre-devel
On Oct 25, 2012, at 11:20 AM, "Gearing, Chris"
Hi,
Post by Nathan Rutman
Based on the responses so far, it seems that there is a fairly
clear
preference for Python as a test language, and so I'll propose that
Python should be used shorter-term to start replacing
test-framework.
I don’t think that is the case Nathan, because unless I am mistaken
Jacques-Charles, Kilian Cavalotti and if it is a binary choice today
my preference is Python (but please see below)
Aurelien - python 2
Bruce was neutral
Roman and Colin (after your email) expressed Perl as a choice.
So the preference would seem to be for Python, counting by people and
organization although obviously it’s a statistically suspect poll.
Indeed. As I stated before, a poll is not the best way to choose the
language; it's just one data point. But as a) we must choose something
to start making any headway toward improvements, and b) bash doesn't
meet the requirements, I'm trying to come up with some kind of baseline.
Chris
NOTES OF MY VOTE: Given the importance of the topic I believe a much
broader piece of analysis needs to be done to select the right language,
I'm just trying to keep things moving from theoretical optimum to
practical conclusion. We feel a need today for much better Lustre
testing, and there's a huge pile of work waiting, and so I feel we must
move forward.
also if this is for an existing framework update the tests must all be
in bash or all converted to the new language of choice.
Short term the idea would be to translate one of the test suites, for
example conf-sanity, to the new language/framework. The two systems
(old and new) would live side-by-side for some period. New tests should
be implemented under the new system (e.g. sanity2). Other old tests
could be converted piecemeal subsequently.
--------------------------------------------------------------------
-
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or
distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
_______________________________________________
Lustre-devel mailing list
http://lists.lustre.org/mailman/listinfo/lustre-devel
References
2. http://www.opensfs.org/foswiki/bin/view/Lustre/TestingImprovements
3. http://wiki.whamcloud.com/download/attachments/7634975/new+test+framework.pdf
6. http://autotest.github.com/
8. http://lists.lustre.org/mailman/listinfo/lustre-devel
_______________________________________________
Lustre-devel mailing list
http://lists.lustre.org/mailman/listinfo/lustre-devel
Kilian Cavalotti
2012-10-26 06:24:08 UTC
Permalink
Hi Prakash,
Post by Prakash Surya
Why do you think this? Bash is extremely modular using the "source"
command.
Not portable? Really? Bash is available on just about every Linux
distribution that comes to mind. Definitely every one Lustre cares
about.
That is not Bash's fault. Bad code can be written in any language. It's
up to the programmer to write "coherent" test scripts.
How would switching languages improve on either of these two fronts?
I couldn't agree more. I'm still not sure why Bash has to be gotten
rid of, and what the improvement would be rewriting a whole new
testing framework from scratch in an other language, when a working
one currently exist. It can certainly be improved, but I'm pretty sure
Bash has been chosen for a reason at some point, and I'm not convinced
the reasons that lead to this choice are not valid anymore.

Cheers,
--
Kilian
Roman Grigoryev
2012-10-26 08:52:42 UTC
Permalink
Hi Kilian,
Post by Kilian Cavalotti
I couldn't agree more. I'm still not sure why Bash has to be gotten
rid of, and what the improvement would be rewriting a whole new
testing framework from scratch in an other language, when a working
one currently exist. It can certainly be improved, but I'm pretty sure
Bash has been chosen for a reason at some point, and I'm not convinced
the reasons that lead to this choice are not valid anymore.
Few samples, which comparably hard to implement on bash than on other
languages from my point of view (especially via evolving tes-framework.sh):

- how could be implemented for current framework random and repeatable
test execution?

- how could be implemented observation of remote node status while
command executed on it and killed it by timeout?

- how simple add machine-readable meta information for test, e.f. tags
and timeouts?


Thanks,
Roman
Kilian Cavalotti
2012-10-26 11:53:26 UTC
Permalink
Hi Roman,
Post by Roman Grigoryev
Few samples, which comparably hard to implement on bash than on other
I don't really want to engage in a code battle I would definitely
loose, and I'm also not very familiar with the current test framework
inner workings, but here are a few points.
Post by Roman Grigoryev
- how could be implemented for current framework random and repeatable
test execution?
Not sure what's required here, but how about storing the list of tests
in a Bash array, generating a list of random indexes, and using it to
loop over the test array?
Post by Roman Grigoryev
- how could be implemented observation of remote node status while
command executed on it and killed it by timeout?
One can read a SSH output stream with the appropriate redirections and
act upon results, stdout or stderr contents, and/or return codes. The
read Bash builtin takes a timeout parameter, and "timeout" is also an
often overlooked coreutils command. :)
Post by Roman Grigoryev
- how simple add machine-readable meta information for test, e.f. tags
and timeouts?
With an associative array, for instance:
$ declare -A TESTS
$ TESTS=( [test1]="tag1:timeout1"
[test2]="tag2:timeout2"
[test3]="tag3:timeout3" )
$ for test in ${!TESTS[@]}; do
value=${TESTS[$test]}
tag=${value%%:*}
timeout=${value##*:}
printf "%s is tagged with %s and has a timeout of %s.\n" $test
$tag $timeout
done
test1 is tagged with tag1 and has a timeout of timeout1.
test2 is tagged with tag2 and has a timeout of timeout2.
test3 is tagged with tag3 and has a timeout of timeout3.

Cheers,
--
Kilian

Gearing, Chris
2012-10-25 18:38:36 UTC
Permalink
Hi,
Post by Gearing, Chris
On the complete-rewrite topic, I will post the slides I sent you before LAD to the Whamcloud wiki and then provide a link to this list so that people can share our thoughts. (Network access prevents me doing this immediately)
As promised: http://wiki.whamcloud.com/download/attachments/7634975/new+test+framework.pdf

Regards

Chris

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
Continue reading on narkive:
Loading...