| View previous topic :: View next topic |
| Author |
Message |
David Brown
Guest
|
 Posted: Tue Nov 18, 2008 10:59 pm Post subject: Re: ARM IDE |
 |
|
Chris H wrote:
| Quote: | In message <491c4570$0$20349$8404b019@news.wineasy.se>, David Brown
david@westcontrol.removethisbit.com> writes
They are also happy to run Plum Hall and other such validation
suites on their tools.
Why don't they? Plum Hall or Perennial
For example, IAR's web site says they test with Plum Hall and
Perennial - they don't give any results for these tests for their
compilers. I'm sure they would tell me if I ask, especially if I'm
offering lots of money - and I'm sure the same applies to CodeSourcery.
Actually they can't publish the results due to the licensing for both
test suites .
|
Yes, I've had a little look around with Google - it seems there is not
much anyone can say except that they "test with Plum Hall". I guess
Plum Hall wants interested parties to buy their own license and test
themselves.
| Quote: | Personally, I am not interested in such big-name test suites.
Fine but they are the only recognised ones.
I have no a priori reason to think that an expensive closed-source
test suite is any better than an open source test suite,
You missed out the "of similar standing". Is there an Open Source
compiler test suite of similar standing to Pum-Hall or Perennial?
and plenty of reason to think that open source test suites are better
in some ways (for example, if a bug is found in gcc, then a test can
be added to the regression test suite to ensure that the bug is not
repeated in future versions).
You are confusing a build test suite and a language test suite. Most
compiler companies have test suites for checking the build. Pull-Hall
and Perennial are language test suites.
|
Yes, I that was my mistake. I asked CodeSourcery about their testing,
and they made this point as well. They actively use Plum Hall to test
for language conformance, and have found and fixed issues as a result.
Because of licensing issues, they can't give out details, of course.
Many of the issues that will be found using something like Plum Hall
will be for unusual language uses - things that don't occur in normal
real-world programming, but are nonetheless part of the language
standards. That's why I don't feel these tests are of direct interest
for me - if a flaw is so obscure that it is only found by such complete
language tests rather than common test suites and common usage, then
that flaw will not be triggered by *my* code, because I don't write
obfuscated code.
| Quote: | Certainly there are times when it is legally important to have
certifications from independent well-known third parties
Quite
- but I don't think it is likely to make any realistic difference to
the reliability of the end product (it is *far* more likely that any
bugs are do to *my* programming, not the compiler).
And you are prepared to stand up in court on a corporate manslaughter
charge with that argument?
|
If I make a system that contains a bug that leads to death, am I less
responsible if I can claim that the compiler used passes Plum Hall
tests? If the Plum Hall tests are considered proof that the compiler is
correct, that only increases the evidence that it was *my* code that
caused the failure! The only legal benefit from having the Plum Hall
certification is if the fault really was in the compiler - I could claim
that I didn't need to check the compiler because Plum Hall said it was OK.
| Quote: | The law in the UK changed on the 6th April 2008 and has a bearing on SW
development,
validation they have for the different parts of their toolkits -
that's more in the realms of "professional" support.
It is quite difficult to do for GCC. Also it would only be for a
specific version and build. As so as you change anything you have to
re-test. Also it only applies to the binary. If you release the
source for some one lese to build it is not covered. (Because any
one could change the source or build it with a different compiler.
CodeSourcery releases compiler builds - you download the pre-packaged
binary and install it just as you would for any closed source tool.
They release new versions about twice a year (with faster updates for
paying customers), just like for closed source tools. They run all
their internal tests and validation (whatever these may be) on these
builds, just like for closed source tools.
Fair enough. These binaries can be tested and validated.
Have a look at this post - it explains pretty well why you don't see
many gcc Plum Hall results published:
http://gcc.gnu.org/ml/gcc/2003-02/msg00652.html
http://gcc.gnu.org/ml/gcc/2003-02/msg01206.html
I know that. However..... as I have said it ONLY applies to the specific
binary you test it with. Not the compiler per say.
|
I agree on that (although where do you stop? Does it only apply when
the compiler binary is run on the same kind of processor as when it was
tested?) |
|
| |
|
 Back to top |
Chris H
Guest
|
| Posted: Wed Nov 19, 2008 12:40 am Post subject: Re: ARM IDE |
|
|
In message <2ZmdnVQyYdTzab_UnZ2dnUVZ8qjinZ2d@lyse.net>, David Brown
<david.brown@hesbynett.removethisbit.no> writes
| Quote: | Chris H wrote:
In message <491c4570$0$20349$8404b019@news.wineasy.se>, David Brown
david@westcontrol.removethisbit.com> writes
They are also happy to run Plum Hall and other such validation
suites on their tools.
Why don't they? Plum Hall or Perennial
For example, IAR's web site says they test with Plum Hall and
Perennial - they don't give any results for these tests for their
compilers. I'm sure they would tell me if I ask, especially if I'm
offering lots of money - and I'm sure the same applies to CodeSourcery.
Actually they can't publish the results due to the licensing for
both test suites .
Yes, I've had a little look around with Google - it seems there is not
much anyone can say except that they "test with Plum Hall". I guess
Plum Hall wants interested parties to buy their own license and test
themselves.
|
Quite. They need to eat too  Time and effort costs.
These test suites are not small or insignificant. The current Perennial
C test suite has over 68000 tests and the C++ has 124,000 (C and C++ are
different languages that parted company back in the 170's)
| Quote: | and plenty of reason to think that open source test suites are
better in some ways (for example, if a bug is found in gcc, then a
test can be added to the regression test suite to ensure that the bug
repeated in future versions).
You are confusing a build test suite and a language test suite.
Most compiler companies have test suites for checking the build.
Pull-Hall and Perennial are language test suites.
Yes, I that was my mistake. I asked CodeSourcery about their testing,
and they made this point as well.
|
It does get confusing. You need to check the thing is built right then
did we build the right thing. (Verification and validation) Then you get
on to testing the maths libraries, the assembler etc :-)
| Quote: | They actively use Plum Hall to test for language conformance, and have
found and fixed issues as a result. Because of licensing issues, they
can't give out details, of course.
|
Quite... BUT it only applies to that specific binary that was tested
under a set of specific conditions. When we tested an ARM binary we did
30 sets of tests on the one compiler for 30 different ARM targets.
| Quote: | Many of the issues that will be found using something like Plum Hall
will be for unusual language uses - things that don't occur in normal
real-world programming, but are nonetheless part of the language
standards.
|
This is a problem... "don't normally occur" the world has a differing
30% of the language they use. Overall they use about 99% of it. More
to the point there are a lot of things they don't know they use. How
well do you know the internal workings of the library for your compiler?
Do you know what the library uses? Probably best not to in some cases
:-)
The problem is you either test or you don't. There is no halfway house.
If you test you test all of it or not at all. If you test all you can
list AND DOCUMENT the areas you do not meet. This is quite common with
embedded compilers (and virtually all C99 compilers We meet the
standard here BUT in the following places we do something different.
More importantly this is what we do where we don't meet the standard.
| Quote: | That's why I don't feel these tests are of direct interest for me -
if a flaw is so obscure that it is only found by such complete language
tests rather than common test suites and common usage, then that flaw
will not be triggered by *my* code, because I don't write obfuscated code.
|
Not true...... It depends on how the compiler works internally and how
it optimises. Also it depends what you are doing. There are parts of
C99 that the majority have not implemented. However they were put in
because small pressure groups got them in and you can bet that they use
them in the one or two compilers that implemented them. However what
does your compiler do when it meets those constructs?
| Quote: | - but I don't think it is likely to make any realistic difference to
the reliability of the end product (it is *far* more likely that any
bugs are do to *my* programming, not the compiler).
And you are prepared to stand up in court on a corporate
manslaughter charge with that argument?
If I make a system that contains a bug that leads to death, am I less
responsible if I can claim that the compiler used passes Plum Hall
tests?
|
It depends on the accident. However it does show that you have taken
reasonable care to ensure the compiler meets the specification of an ISO
C compiler.
| Quote: | If the Plum Hall tests are considered proof that the compiler is
correct,
Not at all. It is simply that given a set of inputs as described in |
the standard you will get a set of outputs as specified in the standard.
OR not and if not what it does do.
| Quote: | that only increases the evidence that it was *my* code that caused the
failure!
|
I am Not a Lawyer
Yes. However if you are not using a tested compiler, or you roll your
own from source then you are more likely to be seen as liable as have
used untested or unsuitable tools. How do we know they are "unsuitable"
tools? Because they are not validated or tested.
You wan to be hanged for one crime or guillotined for the other? :-)
| Quote: | The only legal benefit from having the Plum Hall certification is if
the fault really was in the compiler - I could claim that I didn't need
to check the compiler because Plum Hall said it was OK.
|
Yes. However for safety critical use you have to show due diligence on
the tools. Plum Hall (or perennial) is only part of a validation of a
compiler. However having taken reasonable steps if the fault is in the
compiler I would think that (not being a Lawyer) that it would lessen
your liability somewhat.
BTW the new Corporate manslaughter Act that came into force this year
says that there are fine of up to 15% of *TURNOVER* (not profit) and
jail sentences for directors and responsible managers.
| Quote: | I know that. However..... as I have said it ONLY applies to the
specific binary you test it with. Not the compiler per say.
I agree on that (although where do you stop? Does it only apply when
the compiler binary is run on the same kind of processor as when it was
tested?)
|
Not sure what you mean here. Normally you will run the compiler binary
on for example a windows platform and test it. If that compiler binary
is distributed on various versions of windows then there should be no
problem.
For MAC's I assume you would test (at least we would) on both the PPC
and the Intel platforms.
If you have a compiler that you distribute on Power PC, Sparc, Intel,
Apha, MIPS etc then you build and test on EACH of those.
For the Embedded Arm compiler we test on 30 different targets. Though
in all cases the compiler binary runs on the same Windows host (XP SP2
so far.
--
\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
\/\/\/\/\ Chris Hills Staffs England /\/\/\/\/
\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/ |
|
| |
|
| Back to top |
FreeRTOS.org
Guest
|
| Posted: Wed Nov 19, 2008 12:52 am Post subject: Re: ARM IDE |
|
|
| Quote: | If I make a system that contains a bug that leads to death, am I less
responsible if I can claim that the compiler used passes Plum Hall tests?
|
You are liable only if you have been neglegent. To not be neglegent you
have to show that you have used due dilligence to identify and mitigate
hazards.
The compiler is a potential source of problems, so if you fail to identify
this I suppose you could be considered neglegent.
Once you have identified the compiler you have to do what is necessary to
remove it as a potential problem. Running the compiler through a test suite
is one way of showing that you are taking care, but there are others (as per
my previous post in this thread) and test suites have their limitations.
Somebody very high up in the FAA once said to me - "its impossible to
validate a compiler as there are an infinite number of inputs" [BTW: I
don't buy that statement myself]. I have spoken with people who have
attempted a formal (i.e. mathematical) proof of a compiler, but this is too
big a task to be viable unless you seriously restrict the inputs.
| Quote: | If the Plum Hall tests are considered proof that the compiler is correct,
|
Its not proof, but it is using the "state of the art", and doing all that is
practical *to show language compliance*, and therefore a worth while
exercise if you are worried *about language compliance*. If you test your
code fully from requirements to object code then the compiler can be
completely non "standard" and your code still be shown to be completely
conformant to its specified behaviour.
On the assumption that, if your code has the potential to cause death, then
you are going to test it pretty damn well, then the compiler makes little
difference. Bum code generation will be picked up when a test fails.
| Quote: | that only increases the evidence that it was *my* code that caused the
failure!
|
If somebody is dead, then an investigation will find the source of the
problem no matter whether you try to hind behind somebody elses unpublished
results or not. At least this is the case in the risk adverse, knee jerk
reaction, British society.
| Quote: | The only legal benefit from having the Plum Hall certification is if the
fault really was in the compiler - I could claim that I didn't need to
check the compiler because Plum Hall said it was OK.
|
Hmm.
Out of interest, I have seen most bugs in generated code where non standard
features are used (like __interrupt qualifiers, etc.) and particularly
subtle stack frame problems that can be very reliant on particual events
occurring just at the wrong moment (temporal effects) - for example an
interrupt being taken just as another interrupt is being exited. The
interrupt entry/exit code being very hardware specific. Do the Plum Hall
tests pick up on that sort of thing?
--
Regards,
Richard.
+ http://www.FreeRTOS.org & http://www.FreeRTOS.org/shop
17 official architecture ports, more than 6000 downloads per month.
+ http://www.SafeRTOS.com
Certified by TÜV as meeting the requirements for safety related systems. |
|
| |
|
| Back to top |
David Brown
Guest
|
| Posted: Wed Nov 19, 2008 5:36 am Post subject: Re: ARM IDE |
|
|
Chris H wrote:
| Quote: | In message <2ZmdnVQyYdTzab_UnZ2dnUVZ8qjinZ2d@lyse.net>, David Brown
david.brown@hesbynett.removethisbit.no> writes
Chris H wrote:
In message <491c4570$0$20349$8404b019@news.wineasy.se>, David Brown
david@westcontrol.removethisbit.com> writes
|
I'm snipping a lot of this because it's getting a bit unwieldy - you can
assume that I basically agree with your comments if I've snipped them.
| Quote: |
Many of the issues that will be found using something like Plum Hall
will be for unusual language uses - things that don't occur in normal
real-world programming, but are nonetheless part of the language
standards.
This is a problem... "don't normally occur" the world has a differing
30% of the language they use. Overall they use about 99% of it. More
to the point there are a lot of things they don't know they use. How
well do you know the internal workings of the library for your compiler?
Do you know what the library uses? Probably best not to in some cases :-)
The problem is you either test or you don't. There is no halfway house.
If you test you test all of it or not at all. If you test all you can
list AND DOCUMENT the areas you do not meet. This is quite common with
embedded compilers (and virtually all C99 compilers We meet the
standard here BUT in the following places we do something different.
More importantly this is what we do where we don't meet the standard.
That's why I don't feel these tests are of direct interest for me -
if a flaw is so obscure that it is only found by such complete
language tests rather than common test suites and common usage, then
that flaw will not be triggered by *my* code, because I don't write
obfuscated code.
Not true...... It depends on how the compiler works internally and how
it optimises. Also it depends what you are doing. There are parts of
C99 that the majority have not implemented. However they were put in
because small pressure groups got them in and you can bet that they use
them in the one or two compilers that implemented them. However what
does your compiler do when it meets those constructs?
|
First off, I don't care how my compiler reacts to these few language
features put in for a small pressure group - I'm not in one of these
groups, and I don't use those features.
More generally, if you code to a particular standard (say, MISRA), and
you have tests and code reviews in place that enforce those standards,
then you don't need to consider how your compiler deals with code
outside those standards.
Consider the clichéd car analogy. If you buy a car in Britain, and only
ever drive it in Britain, then you don't care if it has been tested in
outside temperatures of over 45 C or under -25 C. As you won't be using
it outside these parameters, you don't have to worry about them. If you
know that you are always careful about checking the oil levels, then you
don't care how the car reacts to a lack of oil - that corner-case
situation does not apply to you. I'm sure the car manufacturer will do
more testing - but *you* don't care about such tests.
<snip>
| Quote: | I know that. However..... as I have said it ONLY applies to the
specific binary you test it with. Not the compiler per say.
I agree on that (although where do you stop? Does it only apply when
the compiler binary is run on the same kind of processor as when it
was tested?)
Not sure what you mean here. Normally you will run the compiler binary
on for example a windows platform and test it. If that compiler binary
is distributed on various versions of windows then there should be no
problem.
For MAC's I assume you would test (at least we would) on both the PPC
and the Intel platforms.
If you have a compiler that you distribute on Power PC, Sparc, Intel,
Apha, MIPS etc then you build and test on EACH of those.
For the Embedded Arm compiler we test on 30 different targets. Though
in all cases the compiler binary runs on the same Windows host (XP SP2
so far.
|
Sometimes processors have bugs (like the Pentium FDIV bug), or perhaps
differences in the way they handle apparently identical instructions.
Do you test your compiler for compliance on all possible processors, or
do you assume it works the same on each one? Sometimes the various
versions of windows have differences that might affect the compiler
behaviour (it's unlikely, of course, but can you be sure? Different
system libraries might give different results in odd cases). Do you
test them all? What about less usual circumstances, like running the
windows binary under Wine, or using some sort of virtualisation
software? Perhaps the compiler has a bug in its __DATE__ macro that is
only apparent on 29th February - do you have to test the compiler on
each day? Libraries need to be validated too - if you have different
libraries, do you need to validate each compiler/library combination
individually? What about compiler switches - do they also need to be
considered?
My point is that testing a binary, or testing a binary/target
combination, is an arbitrary boundary (albeit a reasonable choice). You
could also argue that you should not consider a compiler Plum Hall
validated unless you have run Plum Hall on the compiler running on
*your* development PC. Or you could argue that it is fine to validate
it for a particular source code / configuration combination (as long as
it is then compiled with a validated compiler, of course). |
|
| |
|
| Back to top |
Chris H
Guest
|
| Posted: Wed Nov 19, 2008 9:28 am Post subject: Re: ARM IDE |
|
|
In message <dbadnUQoMPAdzL7UnZ2dneKdnZydnZ2d@lyse.net>, David Brown
<david.brown@hesbynett.removethisbit.no> writes
| Quote: | Chris H wrote:
In message <2ZmdnVQyYdTzab_UnZ2dnUVZ8qjinZ2d@lyse.net>, David Brown
david.brown@hesbynett.removethisbit.no> writes
Chris H wrote:
In message <491c4570$0$20349$8404b019@news.wineasy.se>, David Brown
david@westcontrol.removethisbit.com> writes
I'm snipping a lot of this because it's getting a bit unwieldy - you
can assume that I basically agree with your comments if I've snipped
them.
|
OK... I was getting a similar problem :-)
| Quote: | Many of the issues that will be found using something like Plum Hall
will be for unusual language uses - things that don't occur in normal
real-world programming, but are nonetheless part of the language
standards.
This is a problem... "don't normally occur" the world has a
differing 30% of the language they use. Overall they use about 99% of
it. More to the point there are a lot of things they don't know they
use. How well do you know the internal workings of the library for
your compiler? Do you know what the library uses? Probably best not
to in some cases
The problem is you either test or you don't. There is no halfway
house.
If you test you test all of it or not at all. If you test all you
can list AND DOCUMENT the areas you do not meet. This is quite common
with embedded compilers (and virtually all C99 compilers We meet
standard here BUT in the following places we do something different.
More importantly this is what we do where we don't meet the standard.
That's why I don't feel these tests are of direct interest for me -
if a flaw is so obscure that it is only found by such complete
language tests rather than common test suites and common usage, then
that flaw will not be triggered by *my* code, because I don't write
obfuscated code.
Not true...... It depends on how the compiler works internally and
how it optimises. Also it depends what you are doing. There are parts
C99 that the majority have not implemented. However they were put in
because small pressure groups got them in and you can bet that they
use them in the one or two compilers that implemented them. However
what does your compiler do when it meets those constructs?
First off, I don't care how my compiler reacts to these few language
features put in for a small pressure group - I'm not in one of these
groups, and I don't use those features.
|
How do you know that? It may use some of those constructs indirectly
in the library. Which features?
Actually there are two sets of testing one for hosted and one for
free-standing compilers. However the differences are specified in the
standard. This is for compilers that target a system that uses an OS and
for systems that target a system without an OS (the majority of cases)
| Quote: | More generally, if you code to a particular standard (say, MISRA),
|
MISRA is neither a standard nor a full subset.
| Quote: | and you have tests and code reviews in place that enforce those
standards, then you don't need to consider how your compiler deals with
code outside those standards.
|
This is not true. It depends.... The whole point of compiler testing is
that you remove the "it depends" Also I am willing to bet that you use
many unspecified and undefined parts of the standard... these are the
implementation defined parts.
For example there are three types of char.
The two integer types signed char and unsigned char.
Then there is the plain char. Used for characters. Is it signed or
unsigned? This depends on your implementation. This has an effect
across the whole standard library.
| Quote: | Consider the clichéd car analogy. If you buy a car in Britain, and
only ever drive it in Britain, then you don't care if it has been
tested in outside temperatures of over 45 C or under -25 C. As you
won't be using it outside these parameters, you don't have to worry
about them.
|
Yes you will. You can not guarantee that the UK temperature will never
go outside those limits. More to the point you can not guarantee that
some one will not drive it to the Munich Oktober Fest. It gets below -25
there.
| Quote: | If you know that you are always careful about checking the oil
levels, then you don't care how the car reacts to a lack of oil - that
corner-case situation does not apply to you.
|
Correct. However should you knock the sump on a speed hump (and you
can't miss them these days) and loose oil no matter how careful you are
checking the oil before you travel you will be driving the car without
oil.
Assuming the world is perfect means you don't have to test at all.
However you are talking about fault conditions not does the car perform
as specified.
Compiler testing is does it perform as specified not what happens if I
break parts of it.
| Quote: | I'm sure the car manufacturer will do more testing - but *you* don't
care about such tests.
|
So you have a set of tests for a car to be driven in the UK and a set of
tests for a car to be driven in Germany...
What if I only want to drive it in Surrey on Tuesdays? For many years
people could get (unofficially and illegally ) death trap cars MOT'ed
because they were only using them locally etc.
That is what you are arguing for.
Of course the real world and other vehicles and people intruded in to
theis local world.
| Quote: | I know that. However..... as I have said it ONLY applies to the
specific binary you test it with. Not the compiler per say.
I agree on that (although where do you stop? Does it only apply
when the compiler binary is run on the same kind of processor as when
was tested?)
Not sure what you mean here. Normally you will run the compiler
binary on for example a windows platform and test it. If that
compiler binary is distributed on various versions of windows then
there should be no problem.
For MAC's I assume you would test (at least we would) on both the
PPC and the Intel platforms.
If you have a compiler that you distribute on Power PC, Sparc,
Intel, Apha, MIPS etc then you build and test on EACH of those.
For the Embedded Arm compiler we test on 30 different targets.
Though in all cases the compiler binary runs on the same Windows host
(XP SP2 so far.
Sometimes processors have bugs (like the Pentium FDIV bug), or perhaps
differences in the way they handle apparently identical instructions.
Do you test your compiler for compliance on all possible processors, or
do you assume it works the same on each one?
|
Interesting point. Do you mean as a host or as a target? If for
targets then yes we test on multiple targets. The compiler targeted for
ARM processors is tested on about 30 different Arm targets.
| Quote: | Sometimes the various versions of windows have differences that might
affect the compiler behaviour (it's unlikely, of course, but can you be
sure?
|
No we specify the host OS. Normally Windows XP in our case. The
compilers only run on windows
| Quote: | Different system libraries might give different results in odd cases).
Do you test them all?
|
No. You are thinking GCC again. The compilers we test are supplied with
a single standard library. We test that. However it does raise the
point that will the GCC you have multiple libraries for many sources
This makes it even more difficult to test GCC.
| Quote: | What about less usual circumstances, like running the windows binary
under Wine, or using some sort of virtualisation software?
|
We do not test it under those hosts. We state a specific host OS.
Emulations and virtual systems are up to you and you would need to test
on those systems for higher SIL.
Again it makes the use of these systems far more difficult for Safety
critical systems. Hence the reason why it is dammed difficult to
validate GCC systems.
| Quote: | Perhaps the compiler has a bug in its __DATE__ macro that is only
apparent on 29th February - do you have to test the compiler on each
day?
|
Not every day, no. I will see if I can find out what the testing for the
__DATE__ macro is since you raise it.
| Quote: | Libraries need to be validated too - if you have different libraries,
do you need to validate each compiler/library combination individually?
|
Yes. However most commercial compilers come with a single system
library. If you are using additional libraries you have to validate
them too. However that would not be part of the compiler validation.
| Quote: | What about compiler switches - do they also need to be considered?
|
Definitely. Tests are run for different target memory configurations
etc.
This is why for C there are over 68000 tests and you run them multiple
times for different configurations.
| Quote: | My point is that testing a binary, or testing a binary/target
combination, is an arbitrary boundary (albeit a reasonable choice).
Not at all. |
| Quote: | You could also argue that you should not consider a compiler Plum
Hall validated unless you have run Plum Hall on the compiler running on
*your* development PC.
|
For SIL4 that is exactly what you do. For SIL 1-3 you can use a
reference platform i.e. WinXP- SP2 as long as you are using a windows
host for development..
| Quote: | Or you could argue that it is fine to validate it for a particular
source code / configuration combination (as long as it is then compiled
with a validated compiler, of course).
|
No. You validate the binary because the source can be altered.
--
\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
\/\/\/\/\ Chris Hills Staffs England /\/\/\/\/
\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/ |
|
| |
|
| Back to top |
David Brown
Guest
|
| Posted: Wed Nov 19, 2008 9:28 am Post subject: Re: ARM IDE |
|
|
Chris H wrote:
| Quote: | In message <dbadnUQoMPAdzL7UnZ2dneKdnZydnZ2d@lyse.net>, David Brown
david.brown@hesbynett.removethisbit.no> writes
Chris H wrote:
In message <2ZmdnVQyYdTzab_UnZ2dnUVZ8qjinZ2d@lyse.net>, David Brown
david.brown@hesbynett.removethisbit.no> writes
Chris H wrote:
In message <491c4570$0$20349$8404b019@news.wineasy.se>, David Brown
david@westcontrol.removethisbit.com> writes
I'm snipping a lot of this because it's getting a bit unwieldy - you
can assume that I basically agree with your comments if I've snipped
them.
OK... I was getting a similar problem :-)
Many of the issues that will be found using something like Plum Hall
will be for unusual language uses - things that don't occur in
normal real-world programming, but are nonetheless part of the
language standards.
This is a problem... "don't normally occur" the world has a
differing 30% of the language they use. Overall they use about 99%
of it. More to the point there are a lot of things they don't know
they use. How well do you know the internal workings of the library
for your compiler? Do you know what the library uses? Probably best
not to in some cases
The problem is you either test or you don't. There is no halfway
house.
If you test you test all of it or not at all. If you test all you
can list AND DOCUMENT the areas you do not meet. This is quite
common with embedded compilers (and virtually all C99 compilers
We meet standard here BUT in the following places we do something
different. More importantly this is what we do where we don't meet
the standard.
That's why I don't feel these tests are of direct interest for me -
if a flaw is so obscure that it is only found by such complete
language tests rather than common test suites and common usage, then
that flaw will not be triggered by *my* code, because I don't write
obfuscated code.
Not true...... It depends on how the compiler works internally and
how it optimises. Also it depends what you are doing. There are
parts C99 that the majority have not implemented. However they were
put in because small pressure groups got them in and you can bet that
they use them in the one or two compilers that implemented them.
However what does your compiler do when it meets those constructs?
First off, I don't care how my compiler reacts to these few language
features put in for a small pressure group - I'm not in one of these
groups, and I don't use those features.
How do you know that? It may use some of those constructs indirectly
in the library. Which features?
|
Any constructs used in common libraries automatically fall under the
label of commonly used and well-tested features. Incorrect behaviour
(either because the compiler writers did not properly interpret the
standards, or because the compiler does not implement their
interpretation) will be quickly spotted and handled by the build test
suites.
| Quote: | Actually there are two sets of testing one for hosted and one for
free-standing compilers. However the differences are specified in the
standard. This is for compilers that target a system that uses an OS and
for systems that target a system without an OS (the majority of cases)
More generally, if you code to a particular standard (say, MISRA),
MISRA is neither a standard nor a full subset.
and you have tests and code reviews in place that enforce those
standards, then you don't need to consider how your compiler deals
with code outside those standards.
This is not true. It depends.... The whole point of compiler testing is
that you remove the "it depends" Also I am willing to bet that you use
many unspecified and undefined parts of the standard... these are the
implementation defined parts.
|
My point is that there is no need for Plum Hall validation for common
features and common language constructs - the compiler's standard test
suites should cover all these features. Don't get me wrong - I am still
glad that my compiler suppliers use Plum Hall to test and improve the
tools. More testing, especially testing in different and independent
ways, is always good. But I don't see any benefit for *me* to know the
details of the Plum Hall validation tests.
Add to this mix, you have the problem that most embedded compilers have
some non-standard additions or extensions that are essential for their
use and effectively invalidate independent tests. If a compiler has an
extra "flash" keyword, for example, then you must either test with the
keyword disabled (which means testing with different settings from when
you use the compiler), or with the keyword enabled (which means the
compiler is no longer compliant, as you can't use "flash" as an
identifier). Even if there is a middle ground such as "__flash" as the
keyword, the Plum Hall tests will not cover this feature, which will be
heavily used in real programs, meaning that their worth as an
independent test tool is greatly diminished.
| Quote: | For example there are three types of char.
The two integer types signed char and unsigned char.
Then there is the plain char. Used for characters. Is it signed or
unsigned? This depends on your implementation. This has an effect
across the whole standard library.
|
Well-written libraries will function identically independently of the
sign of plain "char".
But it is certainly important to remember that there are parts of the C
standards that are not fully specified, and are implementation
dependent. Plum Hall cannot validate these (although perhaps it can
report on them in some way) - with multiple correct answers, there is no
pass/fail test. So source code that is dependent on these features may
work on one Plum Hall validated compiler, and fail on another. The only
way to get around this is to avoid using constructs that have such
dependencies (for example, using "signed char" or "unsigned char"
explicitly when it is relevant). That's much the same thing as avoiding
obscure (but standards-compliant) language constructs that are unlikely
to be well reviewed and tested by the compiler's standard test suites.
| Quote: | Consider the clichéd car analogy. If you buy a car in Britain, and
only ever drive it in Britain, then you don't care if it has been
tested in outside temperatures of over 45 C or under -25 C. As you
won't be using it outside these parameters, you don't have to worry
about them.
Yes you will. You can not guarantee that the UK temperature will never
go outside those limits. More to the point you can not guarantee that
some one will not drive it to the Munich Oktober Fest. It gets below -25
there.
If you know that you are always careful about checking the oil
levels, then you don't care how the car reacts to a lack of oil - that
corner-case situation does not apply to you.
Correct. However should you knock the sump on a speed hump (and you
can't miss them these days) and loose oil no matter how careful you are
checking the oil before you travel you will be driving the car without oil.
Assuming the world is perfect means you don't have to test at all.
However you are talking about fault conditions not does the car perform
as specified.
Compiler testing is does it perform as specified not what happens if I
break parts of it.
|
Perhaps I'm not making myself clear. As a compiler user, or car driver,
I am concerned that the tools work as I expect them to do, when *I* use
them. If I don't drive in temperatures under -25 C, then I am not
concerned about how the car reacts in those circumstances. That's all
there is to it - you can't start adding fantasies about how I *might*
theoretically be able to go outside my assumptions. A car that explodes
when you start the engine at -26 C is still within my required
specifications. A C++ compiler that generates subtle bugs for a
six-layer deep class hierarchy with multiple virtual inheritance and
overloaded virtual "friend" operators is also within my required
specifications for a compiler - it's not code that I would use or
consider safe.
| Quote: | I'm sure the car manufacturer will do more testing - but *you* don't
care about such tests.
So you have a set of tests for a car to be driven in the UK and a set of
tests for a car to be driven in Germany...
What if I only want to drive it in Surrey on Tuesdays? For many years
people could get (unofficially and illegally ) death trap cars MOT'ed
because they were only using them locally etc.
That is what you are arguing for.
Of course the real world and other vehicles and people intruded in to
theis local world.
|
Again, you are misunderstanding me. I'm in favour of compiler
manufacturers using Plum Hall (as CodeSourcery and other gcc testers
do), and doing as much testing as reasonably practical - I just don't
care about validations or certifications that are not relevant in my use
of the tools. A car manufacturer will use the same tests of the UK and
Germany (although they might distinguish between Norway and Oman
markets), but I don't care about the results outside my area of interest.
| Quote: | I know that. However..... as I have said it ONLY applies to the
specific binary you test it with. Not the compiler per say.
I agree on that (although where do you stop? Does it only apply
when the compiler binary is run on the same kind of processor as
when was tested?)
Not sure what you mean here. Normally you will run the compiler
binary on for example a windows platform and test it. If that
compiler binary is distributed on various versions of windows then
there should be no problem.
For MAC's I assume you would test (at least we would) on both the
PPC and the Intel platforms.
If you have a compiler that you distribute on Power PC, Sparc,
Intel, Apha, MIPS etc then you build and test on EACH of those.
For the Embedded Arm compiler we test on 30 different targets.
Though in all cases the compiler binary runs on the same Windows
host (XP SP2 so far.
Sometimes processors have bugs (like the Pentium FDIV bug), or perhaps
differences in the way they handle apparently identical instructions.
Do you test your compiler for compliance on all possible processors,
or do you assume it works the same on each one?
Interesting point. Do you mean as a host or as a target? If for
targets then yes we test on multiple targets. The compiler targeted for
ARM processors is tested on about 30 different Arm targets.
|
I was thinking of the host here (as you already talked about different
targets). That's what's relevant when you decide that it is the
compiler binary that's important.
| Quote: | Sometimes the various versions of windows have differences that might
affect the compiler behaviour (it's unlikely, of course, but can you
be sure?
No we specify the host OS. Normally Windows XP in our case. The
compilers only run on windows
|
There are dozens of variants of XP (different service packs, different
languages, different choice of additional software that can give
different versions of system libraries).
And what about compilers that run under different OS's? There are
plenty of commercial tools that run under a variety of *nix's.
| Quote: | Different system libraries might give different results in odd cases).
Do you test them all?
No. You are thinking GCC again. The compilers we test are supplied with
a single standard library. We test that. However it does raise the
point that will the GCC you have multiple libraries for many sources
This makes it even more difficult to test GCC.
|
Perhaps the compilers you sell are limited in this way. Other tools may
come with multiple libraries, or multiple variants of their libraries
(perhaps maths libraries designed for small size, high speed, or high
accuracy). If you add in support for different operating systems, you
get a whole new set of library issues - even the basic C libraries can
come in variants for different OS's, and with or without support for
things like multi-threading.
I think it is fair to say that if you want a compiler that can claim to
be third-party "validated" or "certified" in some way, you are talking
about a *very* restricted set of circumstances - bare-bones with no
operating system, a specific small library, specific target processors,
specific compiler settings, and specific host environments (including
OS, processor, and installed additional software).
It's like Windows NT's famous "C2" security levels - they are only valid
on a machine that is so locked-down that it is barely usable, and don't
apply in the real world.
If my company felt that Plum Hall validation was relevant to our work,
there would be no possible choice except to buy the test suite and run
it ourselves on our workstations. Of course, it would be useful to know
that our compiler suppliers had run the tests themselves - then we would
know what to expect. But only our own results would have any real weight.
| Quote: | What about less usual circumstances, like running the windows binary
under Wine, or using some sort of virtualisation software?
We do not test it under those hosts. We state a specific host OS.
Emulations and virtual systems are up to you and you would need to test
on those systems for higher SIL.
Again it makes the use of these systems far more difficult for Safety
critical systems. Hence the reason why it is dammed difficult to
validate GCC systems.
|
Validating a software design process and a software project for SIL is
*always* difficult. I haven't dealt with higher SIL levels, but I did
work on a project with lower SIL levels (the software was written in
assembly) - continuous and extensive functional testing and a design and
development methodology that avoids or detects flaws, are the key
elements. Choice of tools is obviously important - but tool validation
by Plum Hall is neither necessary nor sufficient.
| Quote: | Perhaps the compiler has a bug in its __DATE__ macro that is only
apparent on 29th February - do you have to test the compiler on each day?
Not every day, no. I will see if I can find out what the testing for the
__DATE__ macro is since you raise it.
|
Don't worry too much about it - it was just a random example.
| Quote: | Libraries need to be validated too - if you have different libraries,
do you need to validate each compiler/library combination individually?
Yes. However most commercial compilers come with a single system
library. If you are using additional libraries you have to validate
them too. However that would not be part of the compiler validation.
What about compiler switches - do they also need to be considered?
Definitely. Tests are run for different target memory configurations etc.
This is why for C there are over 68000 tests and you run them multiple
times for different configurations.
My point is that testing a binary, or testing a binary/target
combination, is an arbitrary boundary (albeit a reasonable choice).
Not at all.
You could also argue that you should not consider a compiler Plum
Hall validated unless you have run Plum Hall on the compiler running
on *your* development PC.
For SIL4 that is exactly what you do. For SIL 1-3 you can use a
reference platform i.e. WinXP- SP2 as long as you are using a windows
host for development..
Or you could argue that it is fine to validate it for a particular
source code / configuration combination (as long as it is then
compiled with a validated compiler, of course).
No. You validate the binary because the source can be altered.
|
So can the binary if you try hard enough (or have a virus on your
windows machine, as many do).
Clearly any testing or validation on a source code bundle will only be
valid as long as the source code is not modified. |
|
| |
|
| Back to top |
Chris H
Guest
|
| Posted: Wed Nov 19, 2008 9:28 am Post subject: Re: ARM IDE |
|
|
In message <4923d0c8$0$2063$8404b019@news.wineasy.se>, David Brown
<david@westcontrol.removethisbit.com> writes
| Quote: | Chris H wrote:
onstructs?
First off, I don't care how my compiler reacts to these few language
features put in for a small pressure group - I'm not in one of these
groups, and I don't use those features.
How do you know that? It may use some of those constructs
indirectly in the library. Which features?
Any constructs used in common libraries automatically fall under the
label of commonly used and well-tested features.
|
Not all are implemented or commonly used. Complex maths?
| Quote: | Incorrect behaviour (either because the compiler writers did not
properly interpret the standards, or because the compiler does not
implement their interpretation) will be quickly spotted and handled by
the build test suites.
|
Only if you do the full tests. And document all the unspecified,
undefined and implementation specific behaviour the standard permits/
| Quote: | My point is that there is no need for Plum Hall validation for common
features and common language constructs - the compiler's standard test
suites should cover all these features.
|
Not at all. Also the Plum-Hall and Perennial are independent tests.
Writing your own tests proves little.
| Quote: | Don't get me wrong - I am still glad that my compiler suppliers use
Plum Hall to test and improve the tools. More testing, especially
testing in different and independent ways, is always good. But I don't
see any benefit for *me* to know the details of the Plum Hall
validation tests.
|
Correct. There is generally no need for users doing no safety critical
or mission critical work to know the test results. You just need to
know that the compiler manufacturer is taking all reasonable steps to go
a good compiler.
When you get to Safety Critical the burden of checking is on you not the
tool company to check suitability.
| Quote: | Add to this mix, you have the problem that most embedded compilers have
some non-standard additions or extensions that are essential for their
use and effectively invalidate independent tests.
|
Certainly not.
| Quote: | If a compiler has an extra "flash" keyword, for example, then you must
either test with the keyword disabled (which means testing with
different settings from when you use the compiler), or with the keyword
enabled (which means the compiler is no longer compliant, as you can't
use "flash" as an identifier). Even if there is a middle ground such
as "__flash" as the keyword, the Plum Hall tests will not cover this
feature, which will be heavily used in real programs, meaning that
their worth as an independent test tool is greatly diminished.
|
Plum-Hall and Perennial test the standard C. Then you document the
differences between the compiler and the Standard C. There are also
other test suites internal to the compiler company such as build tests
etc that you look at.
Validating a compiler usually requires an NDA with the compiler
manufacturer to look at their other tests, development and build
procedures.
This is not required for the majority but it is for Safety Critical. If
there is an error in the system people get killed and you end up in
court.
| Quote: | For example there are three types of char.
The two integer types signed char and unsigned char.
Then there is the plain char. Used for characters. Is it signed or
unsigned? This depends on your implementation. This has an effect
across the whole standard library.
Well-written libraries will function identically independently of the
sign of plain "char".
|
They should... but is [plain] char singed or unsigned ? It's not just
the libraries. It is YOUR code as well. Do you pass a signed char or
unsigned char to the libraries?
This is a problem if you use MISRA-C and or static analysis. It *may*
also change if you change libraries.
| Quote: | But it is certainly important to remember that there are parts of the C
standards that are not fully specified, and are implementation
dependent.
|
Yes.
| Quote: | Plum Hall cannot validate these (although perhaps it can report on
them in some way) - with multiple correct answers, there is no
pass/fail test.
|
Quite.
| Quote: | So source code that is dependent on these features may work on one Plum
Hall validated compiler, and fail on another.
|
Yes. There is more to it than just the one test suite.
| Quote: | The only way to get around this is to avoid using constructs that have
such dependencies (for example, using "signed char" or "unsigned char"
explicitly when it is relevant).
|
Yes. However the libraries use [plain] char. This is something we have
debated several times in the MISRA meetings.
The problem is the ISO -C teams do not want to enforce it one way or the
other as it will break many compilers and much source code.
| Quote: | That's much the same thing as avoiding obscure (but
standards-compliant) language constructs
|
I agree. Best to avoid some things.
| Quote: | that are unlikely to be well reviewed and tested by the compiler's
standard test suites.
|
No if specified it will be well tested.
| Quote: | Perhaps I'm not making myself clear. As a compiler user, or car
driver, I am concerned that the tools work as I expect them to do, when
*I* use them.
|
Yes but there are several thousand "I"s who all want to use the compiler
differently
| Quote: | If I don't drive in temperatures under -25 C, then I am not concerned
about how the car reacts in those circumstances.
|
But what about those who do and never drive it in conditions over +2?
The problem is (and I have see this many times) is what is "normal" for
one users is abnormal for another. There is no "normal" that 95% of
users agree on.
Otherwise we would ditch the hosted versions of C. The vast majority of
C development is on embedded systems.....
| Quote: | Again, you are misunderstanding me. I'm in favour of compiler
manufacturers using Plum Hall (as CodeSourcery and other gcc testers
do), and doing as much testing as reasonably practical - I just don't
care about validations or certifications that are not relevant in my
use of the tools.
|
You don't need them. Only those doing safety critical work do.
| Quote: | A car manufacturer will use the same tests of the UK and Germany
(although they might distinguish between Norway and Oman markets), but
I don't care about the results outside my area of interest.
|
You don't but others do. The compiler company needs to test to all
users. The validation for critical use also has to test the whole tool
| Quote: | I was thinking of the host here (as you already talked about different
targets). That's what's relevant when you decide that it is the
compiler binary that's important.
|
You can only test the binary. The source is as everyone points out
modifiable by the user.
| Quote: | There are dozens of variants of XP (different service packs, different
languages, different choice of additional software that can give
different versions of system libraries).
|
Standard compiler system libraries?
| Quote: | And what about compilers that run under different OS's? There are
plenty of commercial tools that run under a variety of *nix's.
|
Then you specify which you test on.
| Quote: | Different system libraries might give different results in odd
cases). Do you test them all?
No. You are thinking GCC again. The compilers we test are supplied
with a single standard library. We test that. However it does raise
the point that will the GCC you have multiple libraries for many
sources This makes it even more difficult to test GCC.
Perhaps the compilers you sell are limited in this way. Other tools
may come with multiple libraries, or multiple variants of their
libraries (perhaps maths libraries designed for small size, high speed,
or high accuracy).
|
Then you test all of them if that is what is supplied. BTW we do I
misunderstood what you meant.
| Quote: | If you add in support for different operating systems, you get a whole
new set of library issues - even the basic C libraries can come in
variants for different OS's, and with or without support for things
like multi-threading.
|
Yes. That is why compiler Validation is not a simple thing.
| Quote: | I think it is fair to say that if you want a compiler that can claim to
be third-party "validated" or "certified" in some way, you are talking
about a *very* restricted set of circumstances - bare-bones with no
operating system, a specific small library, specific target processors,
specific compiler settings, and specific host environments (including
OS, processor, and installed additional software).
|
No. Not VERY restricted. For example we test the IAR compilers. They run
on Windows. We use XP for that. We test with all variants of their
standard library. Usually on multiple targets. This can take a couple
of days.
| Quote: | It's like Windows NT's famous "C2" security levels - they are only
valid on a machine that is so locked-down that it is barely usable, and
don't apply in the real world.
|
No idea.
| Quote: | If my company felt that Plum Hall validation was relevant to our work,
there would be no possible choice except to buy the test suite and run
it ourselves on our workstations.
|
For SIL4 you need to do that. For SIL0-3 you can use independent
validation on a reference platform similar to the one you use.
| Quote: | Of course, it would be useful to know that our compiler suppliers had
run the tests themselves - then we would know what to expect. But only
our own results would have any real weight.
|
Yes.
| Quote: | Validating a software design process and a software project for SIL is
*always* difficult.
|
Yes. Not simple either. For the tools you are not validating the process
used to build the tools just that for the following set of legal inputs
you get the following legal outputs. However looking at the compiler
manufactures system does increase confidence and due diligence.
| Quote: | I haven't dealt with higher SIL levels, but I did work on a project
with lower SIL levels (the software was written in assembly) -
continuous and extensive functional testing and a design and
development methodology that avoids or detects flaws, are the key
elements. Choice of tools is obviously important - but tool validation
by Plum Hall is neither necessary nor sufficient.
|
Plum Hall does not do assembler testing AFAIK Neither does Perennial
| Quote: | Perhaps the compiler has a bug in its __DATE__ macro that is only
apparent on 29th February - do you have to test the compiler on each
Not every day, no. I will see if I can find out what the testing for
the __DATE__ macro is since you raise it.
Don't worry too much about it - it was just a random example.
|
Of course I will worry about it... I am curious. Not you have mentioned
it :-)
--
\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
\/\/\/\/\ Chris Hills Staffs England /\/\/\/\/
\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/ |
|
| |
|
| Back to top |
Walter Banks
Guest
|
| Posted: Wed Nov 19, 2008 7:46 pm Post subject: Re: ARM IDE |
|
|
David Brown wrote:
| Quote: | With commercial closed source tools, it's a rarity that you can get in
touch with the developers directly. Often you deal with dedicated
support staff who know less about the tools and target than expert users
|
The compiler business is small and it is generally quite
possible to contact the developers directly. Support staff
deals with the day to day stuff but developers are generally
very much involved in dealing with real issues. Unlike
open source it is generally easy to determine who actually
is responsible for specific design and implementation decisions.
| Quote: | Personally, I am not interested in such big-name test suites. I have no
a priori reason to think that an expensive closed-source test suite is
any better than an open source test suite, and plenty of reason to think
that open source test suites are better in some ways (for example, if a
bug is found in gcc, then a test can be added to the regression test
suite to ensure that the bug is not repeated in future versions). Of
course, it is always better to test with as many testsuites as
conveniently possible (none of them will cover everything).
|
"big-name test suites" offer organized methodical language testing to
a specific standard. There are no similar test FOSS test suites
available that I know of. GCC's regression tests the last time I looked
were a small scatted collection of random code fragments.
One of the biggest problems facing compiler testing is the ability
to deal with testing multiple code generation sequences that come
from syntactically identical sources. Code generation for example
that deals with variable placement context for example. Outside
commercial tools very little serious work has been done to address
this problem.
Regards
| |
