Home Original page

to XFAIL or not to XFAIL

Chris G. Demetriou cgd@sibyte.com
Mon Oct 9 12:20:00 GMT 2000 
Ian Lance Taylor <ian@zembu.com> writes:
> The main use of the testsuite is to see whether the binutils are
> behaving as expected.  That means that `make check' should normally
> succeed, which means it should not generate any FAIL messages.
> Otherwise, you have to remember the expected number of FAIL messages,
> which is error-prone and tedious.
>
> However, we should not lose track of whether some binutils behaviour,
> though expected, actually indicates a bug.  Hence, I believe XFAIL
> should be used for your case (a) (and your patch should not be
> applied).

hmm.  OK, so, based on this, i'd expect that after some amount of
time, tests that show bugs would be marked XFAIL?  8-)

BTW, this patch -- and my few previous patches -- came about because i
ran 'gmake check' over a cleanly built mips-elf binutils tree, and I
got N failures, and was surprised by that.  In fact, the fact that I
hit FAILs prompted me to go in and look what was happening and try to
fix it...  XFAILs don't have nearly that effect.


On a related note, do you have any comments about which targets one
should try (and expect to work 8-) when submitting MIPS patches for
inclusion into the source tree?  (as noted, mips-elf works with a
small number of FAILs or XFAILS, mipsel-elf falls over significantly
more often, due to lack of endianness checking/setting in the
tests... I've not tried other MIPS targets yet.)


> If anything, I think a test which does not indicate a bug, your case
> (b), should result in a PASS.  Perhaps DejaGNU should provide some
> easy way to say ``reverse the sense of this test for this target.''

Well, I don't think that reversing the sense is such a good idea, in
the cases i was thinking of.

For instance, in the readelf tests, really, they need to be enhanced
(somehow) to adequately test either more generically, or with specific
tests per target or cpu.

In the mean time, however, how about the patch below (apply in
binutils/testsuite).



chris
===================================================================
2000-10-09  Chris Demetriou  <cgd@sibyte.com>

	* binutils-all/readelf.exp (readelf_test): Work properly
	when multiple targets expect failtures.
	(readelf -S test, readelf -s test): Expect mips*-*-*
	to fail.

Index: binutils-all/readelf.exp
===================================================================
RCS file: /cvs/src/src/binutils/testsuite/binutils-all/readelf.exp,v
retrieving revision 1.3
diff -c -r1.3 readelf.exp
*** readelf.exp	2000/06/14 01:21:35	1.3
--- readelf.exp	2000/10/09 19:17:01
***************
*** 166,173 ****
      send_log "exec $READELF $READELFFLAGS $options $binary_file > readelf.out\n"
      catch "exec $READELF $READELFFLAGS $options $binary_file > readelf.out" got
  
!     if { [llength $xfails] != 0 } then {
! 	setup_xfail $xfails
      }
      
      if ![string match "" $got] then {
--- 166,173 ----
      send_log "exec $READELF $READELFFLAGS $options $binary_file > readelf.out\n"
      catch "exec $READELF $READELFFLAGS $options $binary_file > readelf.out" got
  
!     foreach xfail $xfails {
! 	setup_xfail $xfail
      }
      
      if ![string match "" $got] then {
***************
*** 230,239 ****
  
  # The v850 fails the next two tests because it creates two special
  # sections of its own: .call_table_data and .call_table_text
  # The regexp scripts are not expecting these sections...
  
! readelf_test -S $tempfile readelf.s  {v850*-*-*}
! readelf_test -s $tempfile readelf.ss {v850*-*-*}
  readelf_test -r $tempfile readelf.r  {}
  
  
--- 230,241 ----
  
  # The v850 fails the next two tests because it creates two special
  # sections of its own: .call_table_data and .call_table_text
+ # MIPS targets fail because they add .rela.txt (empty), .mdebug,
+ # and .reginfo sections.
  # The regexp scripts are not expecting these sections...
  
! readelf_test -S $tempfile readelf.s  {v850*-*-* mips*-*-*}
! readelf_test -s $tempfile readelf.ss {v850*-*-* mips*-*-*}
  readelf_test -r $tempfile readelf.r  {}
More information about the Binutils mailing list