Calculation Ranges, Number of Digits, and Precision
The calculation range, number of digits used for internal calculation, and calculation precision depend on the type of calculation you are performing.
Calculation Range and Precision
Calculation Range | ±1 × 10-99 to ±9.999999999 × 1099 or 0 |
Number of Digits for Internal Calculation | 23 digits |
Precision | In general, ±1 at the 10th digit for a single calculation. Precision for exponential display is ±1 at the least significant digit. Errors are cumulative in the case of consecutive calculations. |
Function Calculation Input Ranges and Precision
Functions | Input Range | |
---|---|---|
sinx cosx |
Degree | 0 ≤ |x| < 9 × 109 |
Radian | 0 ≤ |x| < 157079632.7 | |
Gradian | 0 ≤ |x| < 1 × 1010 | |
tanx | Degree | Same as sinx, except when |x| = (2n - 1) × 90. |
Radian | Same as sinx, except when |x| = (2n - 1) × π/2. | |
Gradian | Same as sinx, except when |x| = (2n - 1) × 100. | |
sin-1x, cos-1x | 0 ≤ |x| ≤ 1 | |
tan-1x | 0 ≤ |x| ≤ 9.999999999 × 1099 | |
sinhx, coshx | 0 ≤ |x| ≤ 230.2585092 | |
sinh-1x | 0 ≤ |x| ≤ 4.999999999 × 1099 | |
cosh-1x | 1 ≤ x ≤ 4.999999999 × 1099 | |
tanhx | 0 ≤ |x| ≤ 9.999999999 × 1099 | |
tanh-1x | 0 ≤ |x| ≤ 9.999999999 × 10-1 | |
logx, lnx | 0 < x ≤ 9.999999999 × 1099 | |
10x | -9.999999999 × 1099 ≤ x ≤ 99.99999999 | |
√x | 0 ≤ x < 1 × 10100 | |
x2 | |x| < 1 × 1050 | |
x-1 | |x| < 1 × 10100; x ≠ 0 | |
x! | 0 ≤ x ≤ 69 (x is an integer) | |
nPr | 0 ≤ n < 1 × 1010, 0 ≤ r ≤ n (n, r are integers) 1 ≤ {n!/(n - r)!} < 1 × 10100 |
|
nCr | 0 ≤ n < 1 × 1010, 0 ≤ r ≤ n (n, r are integers) 1 ≤ n!/r! < 1 × 10100 or 1 ≤ n!/(n - r)! < 1 × 10100 |
|
Pol(x, y) | |x|, |y| ≤ 9.999999999 × 1099 √x2 + y2 ≤ 9.999999999 × 1099 |
|
Rec(r, θ) | 0 ≤ r ≤ 9.999999999 × 1099 θ: Same as sinx |
|
a°b’c” | |a|, b, c < 1 × 10100; 0 ≤ b, c |
|
a°b’c” = x | 0°0’0” ≤ |x| ≤ 9999999°59’59” |
|
xy | x > 0: -1 × 10100 < ylogx < 100 |
|
x√y | y > 0: x ≠ 0, -1 × 10100 < 1/x logy < 100 |
|
a b/c | Total of integer, numerator, and denominator must be 10 digits or less (including separator symbol). | |
RanInt#(a, b) | a < b; |a|, |b| < 1 × 1010; b - a < 1 × 1010 | |
GCD(a, b) | |a|, |b| < 1 × 1010 (a, b are integers) | |
LCM(a, b) | 0 ≤ a, b < 1 × 1010 (a, b are integers) |
Precision is basically the same as that described under "Calculation Range and Precision", above.
xy, x√y, x!, nPr, nCr type functions require consecutive internal calculation, which can cause accumulation of errors that occur with each calculation.
Error is cumulative and tends to be large in the vicinity of a function’s singular point and inflection point.
The range for calculation results that can be displayed in π form when MathI/MathO is selected for Input/Output on the SETTINGS menu is |x| < 106. Note, however, that internal calculation error can make it impossible to display some calculation results in π form. It also can cause calculation results that should be in decimal form to appear in π form.