What is Precision: Definition and 124 Discussions

Extended precision refers to floating-point number formats that provide greater precision than the basic floating-point formats. Extended precision formats support a basic format by minimizing roundoff and overflow errors in intermediate values of expressions on the base format. In contrast to extended precision, arbitrary-precision arithmetic refers to implementations of much larger numeric types (with a storage count that usually is not a power of two) using special software (or, rarely, hardware).

View More On Wikipedia.org
Recent contents Top users
  1. G

    Step Function: Numerical Precision Matters

    Hi, Friends How to numerically do with step function? I mean, this function is very sensitive to the precision. For example, f(x) is a step function. When x is greater than some very small positive number, \delta, the numerical answer is 1, which agrees with the expected value. But when x <...
  2. F

    Precise Vehicle Control Using Electronics

    I do not know if this post belongs here, but it does deal with electronics. I am building a car about 30cm long and 20cm wide. This car has to go a specified distance and must stop as closely as possible to this distance. I have built previous vehicle and found out that there are only two things...
  3. N

    Mathematica A Problem with Mathematica's Precision

    Hi all, Perhaps it is just me that is bugged by this, but I can never get the N function to work in Mathematica. For example, try doing: N[6.50889261991, 2] and you get: 6.50889 Which clearly isn't to 2 significant figures or decimal places... Does anyone know if there is...
  4. A

    What is the level of precision in the universe?

    When I was first learning programming I was surprised that computers only hold numbers to 32 or 64 bits of precision (sometimes more). What I am wondering is to what precision does stuff in the universe happen? For example: if you had a super slow motion microscope and could zoom into a...
  5. I

    How do I construct a set of concentric circles with millimeter precision?

    How do I construct a set of "concentric" circles with millimeter precision? I would need a few real life circular objects with the following specifications: They should all be "semi-concentric" so that they all share the same central axis (at right angle to the planes of the circles), but...
  6. C

    Help me choose an arbitrary precision program

    This is a crosspost from General Math; I wasn't sure what place was more appropriate. I'm looking for a tool to solve a problem I've been working on. In particular, I want something that implements subquadratic trig functions -- in my case I'm looking for the tangent. I've been using Pari...
  7. G

    What does it mean to determine the precision, expressed as percent

    In my lab class, I needed to find k for a spring in two ways (hanging a mass from a spring & through simple harmonic motion). I've done this and now need to compare the two values - the question asks me to determine "the precision, expressed as a percent" of my measurements for k - I don't...
  8. T

    Can someone check my precision for a meter stick, vernier cailper, and micro

    [SOLVED] Can someone check my precision for a meter stick, vernier cailper, and micro Meterstick Precision: 1 mm Example: 607 mm Vernier Caliper Precision: .1 mm Example: 41.5 mm Micrometer Precision : .01 mm Example: .05 mm what I'm asking basically is, is the precision I...
  9. J

    Fortran Double precision computation time in fortran

    I've written a program that repeats a calculation a certain number of times in single precision, then in double precision. The times are recorded by the cpu_time(real t1) subroutine. (Specifically, the calculation is aReal/(aReal+0.01) .) It seems that the time for the double calculation is 1.20...
  10. D

    What are the capabilities of CNC machines for precision steel machining?

    I am currently investigating ways to make a mold for hot-embossing. Normally this is done by for instance SU8 on a silicon wafer, or DRIE of a silicon wafer. But my design is deeper than a normal wafer, around 5 mm deep. Because the devices are actually fairly large (for microsystems) I...
  11. -Job-

    Troubleshooting a Dell Precision Machine After a Power Outage

    Great, i get home from work and find my fairly expensive dell precision machine, which i use as a web server, in some state. All three fans spinning very loudly at full power like a jet engine and the computer completely dead. When i restart the computer no characters are printed, the monitor...
  12. C

    Calculating constants to high precision

    I'm trying to calculate some simple constants to high precision -- a few tens of millions of decimal places, maybe even a few hundred million. Are there any good programs out there that are fast at doing this? I'm using PARI/GP right now, but it's taking quadratic time to calculate ln(2)...
  13. F

    C/C++ Setting Precision in C++: Floor Rounding to 2 Decimals

    Hey.. I'm having trouble achieving this in C++... I'm trying to do an opperation but keeping only 2 decimals, and floor rounding. ((3.52 * 4.32)/3.26) + 2.34 So I have 3.52 * 4.32 = 15.2064, but then I just round to 15.20 then 15.20/3.26 = 4.662576687116, but then I just round to...
  14. A

    What level of precision is implied in a order-of-magnitude calculation

    What level of precision is implied in an order-of-magnitude calculation?
  15. Q

    Calculating Error in s for Free Fall Lab Experiment

    I am measuring the position of photogates in a free fall lab experiment. The question is: What is the error in s in terms of reading error? Now add in the error in the metal scale, a precision of one part in 4000, in quadrature to give an expression for the total error in s. My reading...
  16. J

    Low noise, precision amp?

    help! with building a circuit. i need a gain of 300 (not exact, but somewhere around there, 200-500) for a small signal (typ. 0.1mV to 5mV) AC input with a bandwidth from 0.5Hz to 120Hz. I need it low-noise, with high precision and reliability... has anyone got a reference for any sort of help...
  17. K

    Converting 29 and 0.7 to IEEE single precision floating point

    I am having difficulty in converting 29 and 0.7 to IEEE single precision floating point. For 29, I represent 29 as a product of a power of 2 using 58/4 * 21. However, I am having trouble getting the mantissa value because 58/4 is 14.5, and I need something in the format of 1.[numbers]. Any...
  18. T

    Accuracy & Precision: What's the Difference?

    What is the difference between accuracy and precision?
  19. M

    MATLAB Determine the machine precision in Matlab

    hi, how to determine the machine precision in MATLAB and estimate from this how many bits are used in the mantissa ??
  20. H

    Understanding Precision and Uncertainty in Physics Problem Calculations

    Hi. I was wondering how do you calcultate physics problems when you must also note the error of measurement. I mean if you have a weight-scale that has a percison of 20 g, and let's say that the scale shows 0,7 kg. Does that mean you have 0,7 kg +- 0,02 kg or 0,7 +- 0,01 kg. Or what. And...
  21. B

    Mathematica High precision calculation in Mathematica

    I'm trying to calculate the velocity of 10^20 eV cosmic rays, but the precision in Mathematica and Google is not high enough: both give velocities of exactly 1*c. I've tried putting SetAccuracy[...,1000] around all the constants and the final result, and it still doesn't work. It now...
  22. K

    Numeric precision with iterative matrix rotations

    I couldn't find a forum section on numerical analysis, so I'm writing this here. I'm on the lookout for simple matrix rotation/multiplication methods that can overcome the precision problems associated with poorly conditioned matrices. In my case I'm trying to simulate the rotational...
  23. S

    Evaluation items in precision machine

    hi there exis a relationship between dimensional precision and surface roughness as show in the attach file. however, i can't find any other relationship among the others evaluation items. Some example of evaluation items for precision machine are as below: 1)dimensional precision 2)angular...
  24. R

    Precision of The Big Bang Theory with John Gibbin's "In the Beginning"

    During the last few years I have read a number of books on The Big Bang including recently one By John Gibbin; "In the Beginning." Most if not all of the books mentioned the precision with which the initial values of some important parameters had to be for the universe to have formed as it has...
Back
Top