Home Original page

Decimal arithmetic, with example code

Chris Gonnerman chris.gonnerman at newcenturycomputers.net
Tue Oct 1 21:43:50 EDT 2002 
----- Original Message ----- 
From: "Paul Boddie" <paul at boddie.net>


> Here are some example results from an extended
> version of the original program:
> 
> PCT    AMT    Fixed  Fixed  Fixed  Scaled Scaled Float
>               2dp    4dp    -> 2dp 2dp    4dp
<< hack >>
> 0.10   0.45   0.04   0.0450 0.04   0.05   0.0450 0.05
> 0.10   0.65   0.06   0.0650 0.06   0.07   0.0650 0.07
> 0.10   0.85   0.08   0.0850 0.08   0.09   0.0850 0.09

Whoops... can someone on the FixedPoint team explain this
to me?  When did 0.85 * 0.10, rounded to 2 places, become
0.08?  It should be 0.09.

Perhaps I misunderstood.  Am I supposed to use higher 
precision and then lower it after computation?

This isn't any better than floats.

Chris Gonnerman -- chris.gonnerman at newcenturycomputers.net
http://newcenturycomputers.net
More information about the Python-list mailing list