# What is Precision: Definition and 125 Discussions

Extended precision refers to floating-point number formats that provide greater precision than the basic floating-point formats. Extended precision formats support a basic format by minimizing roundoff and overflow errors in intermediate values of expressions on the base format. In contrast to extended precision, arbitrary-precision arithmetic refers to implementations of much larger numeric types (with a storage count that usually is not a power of two) using special software (or, rarely, hardware).

View More On Wikipedia.org
1. ### Comp Sci IEEE-754 Precision Format

Hello! (Note : I put the prefix as comp sci, since at my uni this is a computer science class but I am in EE so it could be put under both prefixes. If anyone feels its more appropriate as Engineering feel free to change) So here is my attempt at the solution The formula that we are given is...
2. ### Is my textbook wrong about absolute uncertainties?

First off, I will set out what I think I know so that any misconceptions of mine can be put right. Definitions: Precision: a quality denoting the closeness of agreement between (consistency, low variability of) measured values obtained by repeated measurements Accuracy: A quality denoting the...
3. ### What can I add to a Taekwondo body protector scoring systeme to be more stable?

I am developing a Taekwondo body protector scoring system using a textile piezoelectric sensor. The sensor is designed to detect mechanical force and convert it into an electrical charge. However, it is sensitive to temperature variations, leading to potential inaccuracies and instability in the...
4. ### I Measurement Error Analysis in Gaussian distribution

I am new to statistics and recently learned about ISO guidelines for Accuracy & Precision and Uncertainty & Error. But there are some graphs of probability distribution I found on internet which I am not able to grasp. image Source Q. In this graph(above) if...
5. ### Need a Precision Voltage Divider for -10V to 10V Input and -4V to 4V Output?

Hello! I need a voltage divider where the input is between -10 and 10 V and the output is between -4 and 4 V. I would need something with a bandwidth of 1 (maybe even 10) MHz. Can someone recommend me a reliable product (the price plays a role, too, so being cheaper would be also useful)? Thank you!
6. ### A Do moon phases affect weighing precision on the microgram range?

Hi, I'm looking to improve the precision of a system over a several hour cycle of operation of +50°C operation. The mass is ~11g and is suspended below the balance via a factory provided point. The "sample" is enclosed within a large borosilicate flask 150mm dia x 450 mm height that is trace...
7. ### Comp Sci Computer Arithmetic for Double Precision Numbers

I know that this expression evaluates to 1 when a is equal to 0. Also for when a is equal to 1/n when n is a positive number, but I'm confused about how to go about this in double precision?
8. ### I Number of Sig Figures and Precision

Hello, When we measure something, using a measuring instrument, our measurement will be a number with a finite number of significant figures. The rightmost digit is the uncertain digit affected by uncertainty. A measurement should always be reported as the best estimate +-error range where the...
9. ### A Why low temperature is better in many precision experiments?

Hello! In many precision experiment, especially those overlapping with quantum computing techniques, such as trapping a few ions in a Paul trap, they use cryogenic systems (around 1K). I am not totally sure I fully understand the advantage of that, compared to room temperature. For example, a...
10. ### I Decimal Precision - Multiplying, Rounding, and Adding

Hi, I have \$150. This value is multiplied by 2 or more factors. Each factor is less than or equal to 1 and greater than or equal to 0. Each factor has a maximum of 9 decimal places. The sum of the factors equals 1. For example, the following set of factors meets these 3 conditions: 0.500033333...
11. ### Precision Temperature Measurement

What is the most precise way to log temperature in the laboratory? I am concerned with resolution and precision here, not accuracy. RTDs don't seem to cut it. Almost all that I have looked at and tried only have a resolution of around 0.1 deg C. I was hoping for 10X better than that- Thanks...
12. ### Relative error and measurement precision

NOTE: this is a programming exercise (Python). I started adding to ##x_{true}## an error related to a (for example) 10% relative error, obtaining ##x_{measurement}##. Then i computed ##y_{measurement}##. To find the precision, i calculated ##(y_{true}-y_{measurement})/y_{measurement}##. If it is...
13. ### Fortran Help with fortran numerical precision

Hello. I need help to understand why my code is not giving me double numerical precision. In this piece of code, I create a function that is analytically normalized, but when I calculate the numerical normalization factor, it seems to be with single precision. Here it is: program dyna...
14. ### Number Precision When Adding/Subtracting

If you read any introductory physics or chemistry course, in the very first chapter, you'll come across measurement, uncertainty, etc. There, you learn some rules about rounding off numbers, etc. One of the rules discussed there is that if you add/subtract two numbers with different decimal...
15. ### Compare Precision & Accuracy: Speed of Light Measurements

Homework Statement Two different scientists measured the speed of light four times. Scientist A 3.80*108m/s 3.32*108m/s 3.10*108m/s 3.05*108m/s Average 3.32*108m/s Scientist B 2.90*108m/s 3.10*108m/s 3.24*108m/s 2.95*108m/s Average...
16. ### Dealing with 1000 digit numbers

Hi, Mostly i work with Octave / Matlab but i am trying to get into python also. Lately i have a couple problems where my numbers can't be represented in binary64 or float64 default format because they exceed the max of 1.8x10^308. Is there a common way people deal with this (without toolkits)...
17. ### Fortran Print correct value of Real number with gfortran?

The following program is printing wrong value of RWTSED. How can I print correct value?? program inpdat c IMPLICIT NONE REAL RHOMN,RWTSED,VOLSED VOLSED = 17424.0 RHOMN = 2.42 !0000076293945 RWTSED= VOLSED*RHOMN*1.0E6 2000 FORMAT(/,3F40.5)...
18. ### B Quantum Theory and its Precision

Hi, I was watching a really interesting old video from Richard Feynman who was talking about how precise/accurate the predictions of quantum theory agree with observation. The number of decimal places it agrees he said is astounding. He likened it to measuring the circumference of the Earth to...
19. ### B Difference Between Trueness & Bias Measurement

Can someone explain the difference between trueness and bias, as used in measurement?
20. ### I Regarding the precision and accuracy of Atomic mass

The website of CIAAW list the 2012 atomic mass in Daltons (Atomic Mass Units) and mention the atomic mass of stable isotopes followed by an additional number between braces. For example Helium-4: 4.002 603 2541(4) My first question is what does the number between braces mean? I assume it is some...
21. ### The making of vintage precision instruments

Ever when I see one of these beautiful brass scientific instruments, like microscopes, telescopes, goniometers etc. from the 19th century, I wonder how they have been actually made and whether I could learn how to do this. I think that most of gears used in the construction of these pieces were...
22. ### Use math to prove which lab equipment was precise/accurate?

Homework Statement We performed an experiment to see which laboratory glassware was more accurate and which was more precise. Question: (a): Which glassware was more precise and why? (b) Which glassware was more accurate and why? Use the calculations to support your statement. I won't post the...
23. ### B How Precise Can QED Predict the Motion of Charged Particles?

It is claimed that QED could predict certain quantities, like the lamb shift and magnetic moment of electron with astonishing precision of 12 digits. My question is the following: does QED predict only a few things to this precision, or does it predict, this precisely, the entire arbitrary...
24. ### Pulse echo technique to find the distance of the Moon

Homework Statement [/B] Over 40 years ago, the Apollo astronauts placed reflectors on the surface of the Moon. These are still used by a number of observatories on Earth to monitor the distance to the Moon by reflecting pulses of laser light from them and detecting the reflected signal...
25. ### I Measurement precision question

Supposing we have 2 circles in a 2 d space that can bounce off each other like balls. These circles are made of an infinite number of points. We put the centre of each circle on a line and send them towards each other along the line. Are they going to bounce off keeping their trajectory on the...
26. ### Insights Representations and Why Precision is Important - Comments

fresh_42 submitted a new PF Insights post Representations and Why Precision is Important Continue reading the Original PF Insights Post.
27. ### C/C++ What is the difference between double and int64_t in C/C++ for precision?

Hello I am currently using double type for numbers with the format 6f8 (6 digits before the point, 8 digits after the point). Double structure is enough for this format, or should I use int64_t instead? Thank you in advance :) Greetings. PS: I tried with this example code, but the result...
28. ### Implied Precision: Is It 0.005 or 0.01?

Homework Statement Some sources say the implied precision of a measurement is half of the least significant figure, while others say the implied precision is "1" in the place of the least significant figure. For example, a length is measured as 3.28 m. According to some sources, the implied...
29. ### Advice on milligram/centigram precision scale

Any suggestion on a good (but as usual not that expensive) scale that can measure at least to centigram precision? Searching online (e.g. Amazon) I did find some options but the reviews are usually mixed regarding precision (users will say that scales with milligram precision specs are not that...
30. ### The precision in measurements of radial velocities by DE?

Hello, I have an exercise here that I need help with. The precision in measurements of radial velocities by the Doppler effect is currently 1 m/s. Can a Jupiter like planet orbiting a star similar to the Sun at a distance from the mother star equal to the Sun-Jupiter distance be detected? (Use...
31. ### Difference between uncertainty and precision?

Is precision the smallest division in measurement instruments? Is uncertainty one-tenth or a half of the smallest division in measurement instruments? I'm confused some say one-tenth, some say a half... Thanks in advance
32. ### Need more precision in the calculations....

Hello friends, today i have this question about precision. Actually I'm trying to do a simple use of the pow function for a small number x=0.005 squared. By entrying pow ( x, 2) my console does not output the proper number 0.000025, but some memory error. I've read about it and i may be able to...
33. ### Fortran Modern variable precision and integer multipliers

I learned Fortran from ancient people who disregard modern standards. As such, my code consists mainly of that older syntax. I am wondering how well this meshes with current Fortran environments, specifically regarding double precision calculations. This comes up mainly because I have been...
34. ### Feynman's approach to precision QED

Feynman's approach to precision QED calculations, see his book "Quantum Electrodynamics", was shown(?) to be equivalent to the method of quantum fields. Both methods get the right the get the right answers to some problems. What quantum problems does Feynman's method (there is a name for that I...
35. ### Why low precision of the radius of Sun's orbit in MW?

It seems that the value has only 1 1/2 significant digits. Why are values for the measurement of this radius not more precise? What prevents a greater precision? I have recently been trying to learn about the sun's orbit in the Milky Way (MW) in order to calculate estimates re DM for another...
36. ### Measuring a paper clip with precision

Homework Statement You are asked to find the weight of a paper clip as accurately as possible and given ten paper clips and a scale that has a precision of 0.1 g. (A paper clip weighs about 1 g.) Which of the following proceedures would give you the most precise measurement the weight of a...
37. ### Cleaning copper connections for high precision instruments

Hi all, Here's my first post with a question I've been trying to solve for a while. I work with electronic components of very high quality and we try to make all our connections using thin copper ribbons of various thickness. I have started cleaning my copper connections with a solution of...
38. ### Significant figures vs precision (I think)

Homework Statement I've got a doppler shift question about calculating the frequency heard by a receiver moving 5.0m/s towards a sound source that is emitting 3000Hz sound. I assumed speed of sound is 343 m/s for an air temperature of 20C. Homework Equations Freceiver = Fsender((V +...
39. ### Precision hole making methods for small holes

I'm designing a fixture for quality control and need a few precision holes. Their diameters range from 1/8" to 5/16" and need to be ±0.0002" with a depth of 1/2". I've been looking into EDM hole making and honing, but neither seem to be great for that range of diameters. Anyone know of any other...
40. ### Using MKL to Switch Between Single and Double Precision

So I have a header file containing things like #ifndef _params_h #define _params_h #define REAL double // precision of data #define ALIGN 64 // byte size of alignment #define DBG 1 // 1 prints out results #define READIN 1 // 1 reads in white paper data, 0 randomizes data #define...
41. ### How Does IEEE Single Precision Format Store 7 Decimal Digits?

Could someone please show me some examples how IEEE single precision format can used to store approximately 7 decimal digits of a number when the number is written in decimal format? How this is related to the statement ##2^{-23}=1.19\times10^{-7}##?
42. ### Precision Rectifiers: Get Perfect DC from Sinusoidal Wave

If you are given a pure sinusoidal wave and a perfect DC is required from it. then what is the best way to get it? should we go for full bridge rectifiers? or any better way is there?
43. ### What determines the precision of uncertainties?

What limits the precision with which you can describe the uncertainty of a measurement? I will describe two examples that feel qualitatively different, but I am not sure if they are quantitatively different in how you would deal with the uncertainty. 1) Measuring the length of a pencil with a...
44. C

### Can Magnetic Beams Enable Faster Space Travel?

Hey all, So I've been writing a hard science fiction novel and need real answers from someone who understands the fields better than I ( I have no formal education outside basic high school classes in physics) The problem I have is I'm not a fan of faster than light travel but I need a way for...
45. ### Precision measurements of atomic mass.

Hello I have been trying to find out how the data for the atomic mass of different isotopes is processed. I am vaguely familiar with mass spectrometry and other techniques used for the measurements and I know that the experimental results are coordinated by the Atomic Mass Data Centre. I am...
46. ### Precision and accuracy of a solution

Homework Statement so let's say that we have a solution S=0.4±0.01/250ml i want to do a dilution so i pick 5ml and pour it into 100ml beaker how should i calculate precision of the solution (mathematically or with some other equations)? Homework Equations so basically we have F=5*S/100 The...
47. ### How does the OpAmp bypass Vf limitations in precision rectifiers?

Hi all, I am currently studying OpAmps and one of the topics that came up was the applications of OpAmp circuits. In particular, I was learning about precision rectifiers (using non-inverting version of it, thus the diode is pointing away from the "out" terminal of the OpAmp). The thing I...
48. ### Assessing a measurement's precision

Homework Statement I'm doing a quality of measurements coursework at the moment where I have to find the efficiency of the different volumes of water. To do this, I needed to use this formula: After measuring the efficiency of different volumes of water, I'm asked to assess the...
49. ### How Can I Compute High Precision Square Roots for Special Relativity Homework?

Hello, i'm having trouble evaluating my gamma factor for my special relativity homework, because I need to compute 1 minus a very small number (8.57*10^-13). My calculator treats this value as simply 1, as does Mathematica. Although I don't know much about it, and maybe there's a way to force...
50. ### Finding the Error in z for q = -0.6 \pm 10\% and z = 0.2

Homework Statement For the equation q = \frac{z(z+2) - 2DH}{z^2} q = -0.6 \pm 10\% , and z = 0.2. D and H are known exactly. I have to find the error in z that will give an answer of q = -0.6 \pm 10\% Homework EquationsThe Attempt at a Solution I have considered rewriting the equation...