User Manual

20050501
4 Number of Digits and Precision
k Number of Digits
Standard Mode
The following applies when the check box next to the “Decimal Calculation” item on the
Basic Format dialog box is not selected.
•Up to 611 digits are stored in memory for integer values.
•Decimal values up to 15 digits are converted to fraction format and saved in memory. When
a mathematical expression cannot be converted to fraction format, the result is displayed in
decimal format.
•Values stored in memory are displayed as-is, regardless of how [Display] settings
(Normal 1, Normal 2, Fix 0 – 9, Sci 0 – 9) are configured (except when a decimal value is
displayed).
Decimal Mode
The following applies when the check box next to the “Decimal Calculation” item on the
Basic Format dialog box is selected.
•Values stored in Ans memory have the same number of digits as they have when
displayed. A value that is assigned to a variable has the same number of digits as defined
for Standard mode values.
•Values are displayed in accordance with how [Display] settings (Normal 1, Normal 2,
Fix 0 – 9, Sci 0 – 9) are configured.
•Displayed values are rounded to the appropriate number of decimal places.
•Some applications store values using a mantissa up to 15 digits long and a 3-digit
exponent.
k Precision
• Internal calculations are performed using 15 digits.
• The error for a single mathematical expression (Decimal mode calculation error) is ±1 at
the 10th digit. In the case of exponential display, calculation error is ±1 at the least
significant digit. Note that performing consecutive calculations causes error to be
cumulative. Error is also cumulative for internal consecutive calculations performed for:
^(x
y
),
x
, x!, nPr, nCr, etc.
•Error is cumulative and tends to be larger in the vicinity of a function’s singular point(s) and
inflection point(s), and the vicinity of zero. With sinh(x) and tanh(x), for example, the
inflection point occurs when x = 0. In this vicinity, error is cumulative and precision is poor.
α -4-1
Number of Digits and Precision