Representing long decimals in hex in Java

[Paul Uszak]

Does anyone know how to represent the following (example) base 10 number as hexadecimal in the Java language?

2.67367134065437661762356872350435000732582367230003208735000765782365523111637787869816671091367351854359435034651476543984089764356140092091709145675149665165167871324671303963400256123013656501396325676347820

 

[phinds]

I'm not sure I understand the question. It seems like a straightforward algorithm to convert a decimal floating point number to a base 16 floating point number, or is that not what you mean? And it's the algorithm that matters, not what language it is implemented in. Certainly the standard math pack won't do it, if that's what you're asking ...an extended precision algorithm will be required but that's just programming.

 

[Paul Uszak]

No - it's exactly as you say. I just don't know how to do it. Any specific pointers /code fragments..? (I mention Java as that's my programming skill set.)

 

[phinds]

Try Planet Source Code. I haven't used it in years but when I used to go there I could almost always find something that helped.

 

[Filip Larsen]

If you don't mind handling the decimal point yourself you can use the BigInteger class [1] which has a constructor that accept a long string of numbers (without decimal point) and a toString method that takes a radix (use 16 for hexadecimal).

If you like to parse your number with a decimal point but don't need the decimal point on output you can also use the BigDecimal class [2] which can read in decimal numbers with a decimal point and convert the whole number to an unscaled BigInteger which you then can print in hexadecimal. If you really need the decimal point on output too you should be able to extract the integer part and fractional part of that BigDecimal as BigIntegers which you can then print separated by a decimal point.

[1] https://docs.oracle.com/javase/7/docs/api/java/math/BigInteger.html
[2[ https://docs.oracle.com/javase/7/docs/api/java/math/BigDecimal.html

 

[pbuk]

That does not work Filip Larsen - 1.1₁₀ ≠ 1.1₁₆ (and also ≠ 0.B₁₆).

You need the apfloat (arbitrary precision floating point) library rather than BigInteger.

 

[Filip Larsen]

That does not work Filip Larsen - 1.1₁₀ ≠ 1.1₁₆ (and also ≠ 0.B₁₆).

You need the apfloat (arbitrary precision floating point) library rather than BigInteger.

Apparently you read some else into the OP's question than I do.

It seems like a straightforward algorithm to convert a decimal floating point number to a base 16 floating point number, or is that not what you mean? [...]
Certainly the standard math pack won't do it, if that's what you're asking ...an extended precision algorithm will be required but that's just programming

No - it's exactly as you say.

As a follow-up to this exchange, my comment to use BigInteger/BigDecimal was to point out that there are standard classes for doing it (as long as the OP can manage to handle the decimal point himself). If this is not what the OP wants, then perhaps he should clarify what problem he is trying to solve with the hexadecimal number.

 

[phinds]

As a follow-up to this exchange, my comment to use BigInteger/BigDecimal was to point out that there are standard classes for doing it (as long as the OP can manage to handle the decimal point himself). If this is not what the OP wants, then perhaps he should clarify what problem he is trying to solve with the hexadecimal number.

I have to question whether you have really thought this through. Just take the simple decimal number 1.5 If you ignore the decimal point, planning to "take care" of it after the conversion, you have 15 which converts to hex F. Now take the ACTUAL conversion and you go from decimal 1.5 to hex 1.8 How do you "handle the decimal point" to get from F to 1.8 ? Suggesting that this is trivial does not strike me as a helpful approach. Am I missing something?

 

[Filip Larsen]

I have to question whether you have really thought this through. Just take the simple decimal number 1.5 If you ignore the decimal point, planning to "take care" of it after the conversion, you have 15 which converts to hex F. Now take the ACTUAL conversion and you go from decimal 1.5 to hex 1.8 How do you "handle the decimal point" to get from F to 1.8 ? Suggesting that this is trivial does not strike me as a helpful approach. Am I missing something?

Well, I have not been very clear on my assumptions. If you are trying to write up a floating point number in hex then you are right - what I suggest will not work. And the OP does indeed mention "floating point", although it is not clear to me why he wants to express it in hexadecimal.

Lacking any clear understanding about what the OP wants, I am merely suggesting that if he treat the integer and fractional part as two separate (hex) number strings that just happens to be separated by a decimal point (if he really needs it) then he can use BigInteger. The combined string will not be representing the number as floating point number in the way floating numbers are defined, but since there is no standard way in Java to specify a hex floating point number constant with an arbitrarily number of precision he will have to use a library of some sorts anyway in order to use this number for anything related to arithmetic.

Granted, without any reason for having the number specified in hex I should probably just have suggested to include the number directly as a BigDecimal with something like

BigDecimal myConstant = new BigDecimal("2.67367134065437661762356872350435000732582367230003208735000765782365523111637787869816671091367351854359435034651476543984089764356140092091709145675149665165167871324671303963400256123013656501396325676347820");

after which myConstant is available for calculations or convertion using the BigDecimal class.

 

[pbuk]

Lacking any clear understanding about what the OP wants

The OP states that he wants to represent rational numbers (of which he gives an example) in hex in Java.

... I should probably just have suggested to include the number directly as a BigDecimal with something like

BigDecimal myConstant = new BigDecimal("2.67367134065437661762356872350435000732582367230003208735000765782365523111637787869816671091367351854359435034651476543984089764356140092091709145675149665165167871324671303963400256123013656501396325676347820");

after which myConstant is available for calculations or convertion using the BigDecimal class.

That won't work - BigDecimal does not do number base (radix) conversions: apfloat does which is why I suggested it.

 

[Filip Larsen]

The OP states that he wants to represent rational numbers (of which he gives an example) in hex in Java.

I see no mention of rational numbers by the OP in this thread?

That won't work - BigDecimal does not do number base (radix) conversions: apfloat does which is why I suggested it.

The code snippet I gave has no need for a radix. It is meant to illustrate that lacking any particular reason for writing the number in hex you may as well include number directly as a decimal number using BigDecimal (or the Apfloat library, if that fits your bill).

 

[pbuk]

The OP gave an example of a number he wanted to represent: the example is a rational number. As the OP has not returned and his question has been answered I have no more to say on this.

 

[Paul Uszak]

The OP...

Side (Silly) Question:-

Sorry, but this is the second site that's called me the "OP". What does OP mean please? Is it rude ?

 

[pbuk]

Original Poster - not rude at all

 

[DrZoidberg]

You realize that a number with a finite amount of digits in base 10 may have an infinite amount of digits in base 16?
Anyway, here is an example in Java.

import java.math.BigInteger;
import java.math.BigDecimal;

public class Hex {
  public static void main(String[] args) {
    String input = "2.67367134065437661762356872350435000732582367230003208735000765782365523111637787869816671091367351854359435034651476543984089764356140092091709145675149665165167871324671303963400256123013656501396325676347820";
    int maxDigits = 200;
    BigDecimal bd = new BigDecimal(input);
   
    BigInteger bi = bd.toBigInteger();
    String hexString = bi.toString(16);
    bd = bd.subtract(new BigDecimal(bi));
   
    if(!bd.equals(BigInteger.ZERO)) hexString += ".";
   
    while(!bd.equals(BigDecimal.ZERO) && maxDigits-- > 0) {
      bd = bd.multiply(new BigDecimal(16));
      bi = bd.toBigInteger();
      hexString += bi.toString(16);
      bd = bd.subtract(new BigDecimal(bi));
    }
   
    System.out.println(hexString);
  }
}

 

[Paul Uszak]

Thanks for the code. It's really useful.

I gave a simple example because I didn't want the thread to drift off in a tangential direction before consensus was reached on an answer.

I'm trying to convert a billion (10^9) digit file of the decimal expansion of pi into a binary file of 0-255 value bytes. This will give me a file of what I believe to be truly random bits for comparison purposes for some other work I'm undertaking. I can adapt the code for my needs. Just hope that I have enough RAM...

 

[DrZoidberg]

With a billion digits my code is going to need a lot of RAM and it will take forever to run.
But if all you want is random bytes you can just use Random.nextBytes

 

[Paul Uszak]

Will say a week and 8 GB be sufficient do you think?

Can't use Random.nextBytes; it's not really. I need undisputedly random.

 

[DrZoidberg]

PI is not any more random than the output of the Random class. But there is also SecureRandom which produces better random numbers.

 

[pbuk]

This applet http://www.apfloat.org/apfloat_java/applet/pi.html should generate pi in hex directly.

 

[Paul Uszak]

PI is not any more random than the output of the Random class

Err, is this a serious comment? The mathematical consensus is that the sequence of digits of pi, e, golden ratio, root 2 are infinite in all bases and entirely random. There's a mountain of published research substantiating this.

I want to use pi as a bench mark for some tests I'd like to do, just don't have it as bytes.

 

[pbuk]

Did you look at generating it rather than converting it?

 

[DrZoidberg]

If all you want to do is convert a completely random sequence of decimal digits into a completely random sequence of bytes, you don't actually have to do a proper base conversion. You can use a simpler, faster algorithm that ends up producing a smaller number of bytes than theoretically possible but is quite fast and needs only little memory.
Here is an example in Java. Whenever a decimal digit is between 0 and 7 it is converted to binary directly. If it's 8 or 9 it is converted into a 0 or a 1. The result should be every bit as random as the input digits. Producing on average 2.6 bits for each decimal digit instead of the theoretically possible 3.3 bits per digit.
As input you can use a FileReader object and as output a FileOutputStream.
btw. there is a program you can use to test the randomness of the output at http://www.fourmilab.ch/random/

public static void randDecToBin(Reader in, OutputStream out) {
  try {
    int b = 0, bits = 0;
    while(true) {
      int c = in.read();
      if(c < 0) break;
      int d = c - '0';
      if(d < 0 || d > 9) continue;
      if(d >= 0 && d <= 7) {
        b = (b << 3) | d;
        bits += 3;
      } else {
        if(d == 8) b = b << 1;
        else b = (b << 1) | 1;
        bits++;
      }
      if(bits >= 8) {
        out.write(b & 0xFF);
        b >>>= 8;
        bits -= 8;
      }
    }
  } catch(Exception e) {
    e.printStackTrace();
  }
}

 

[pbuk]

No, that will produce twice as many 0s and 1s as any other (octal) digit. Your method is basically sound (and will be just as random as a true representation of pi in any number base), you just need to ignore 8s and 9s.

 

[DrZoidberg]

Why would it produce twice as many bits? It works fine and the output tests as extremely random.

 

[pbuk]

Oh sorry, I misread your explanation and found your code a bit hard to read so I gave up - the method you describe does indeed produce an average of 2.6 random bits for each decimal digit.

 

[Paul Uszak]

Did you look at generating it rather than converting it?

I did briefly. I don't have the skill or kit to generate pi to that number of digits. I already have it in the form of ASCII text. I'd be more confident of having the correct sequence if I just do a simple bit manipulation exercise. One of the features of randomness and cryptography is large numbers, so a roughly Gigabyte file would be really useful.

 

[Paul Uszak]

...the output tests as extremely random.

If I may, what was your input for the test, and how did you test the output? ENT?

 

[D H]

Can't use Random.nextBytes; it's not really. I need undisputedly random.

What, exactly, makes you think you need "undisputedly random"? What, exactly, makes you think a finite sequence of the hexadecimal representation of pi is "undisputedly random"?

Some applications do indeed need a cryptographically secure pseudo random number generator. Note well: Bits generated by the binary representation of pi are not cryptographically secure.

I want to use pi as a bench mark for some tests I'd like to do, just don't have it as bytes.

There is a formula that let's you do exactly that, the Bailey-Borwein–Plouffe formula. With this, one can, for example, calculate the quadrillionth (10¹⁵) hexadecimal digit of pi without having to calculate all of the 10¹⁵-1 hexadecimal digits that precede it.

 

[DrZoidberg]

If I may, what was your input for the test, and how did you test the output? ENT?

Yes, I used ENT for the test. My input was a text file containing the first 1 billion decimal digits of pi.

 

[Paul Uszak]

I think that we both probably have the same pi file! Thanks for your code example. It's a clever idea that I hadn't thought of.

Thing is, my spider sense is tingling because I'm having a hard time understanding how the output can be pi like with less entropy that in the original (2.6 bits versus 3.3). My fear is that the code does not represent pi, but has generated a pseudo random file with characteristics determined exclusively by your code, and not by inherent mathematics. Is there any possibility that your code is acting as a pseudo random generator acting upon an ASCII pi seed?

 

[DrZoidberg]

The file generated by my code is determined by both - pi and the way in which my code does the conversion. There are many different ways to extract random bits out of pi. What's important here is that the output is perfectly random as long as the input is. The amount of information is reduced simply because some of it is thrown away in order to allow the algorithm to be faster.

 

[Paul Uszak]

one can, for example, calculate the quadrillionth (10¹⁵) hexadecimal digit of pi without having to calculate all of the 10¹⁵-1 hexadecimal digits that precede it.

I suspect that it's not quite as easy as you've written it. There's quite a summation to be calculated using arbitrary precision mathematics isn't there..?

 

[Paul Uszak]

you can use the BigInteger class

I thought that this was the silver bullet. Wow, was I wrong

I used BigInteger and output 4MB of pi bytes. It worked perfectly and I've attached a piccy for anyone who wants to see what 3MB of pi looks like. BUT.

The BigInteger technique won't work in practice to process the 1GB file. It tool me 3 hours to process 4MB. I did some tests, and have attached a spreadsheet of my findings. It will take 4ish years to process the full 1GB file. This result is unexpected and not understood. I didn't even bother with memory requirements. It would actually be quicker to generate pi as bytes directly. I'll have to think of something else...

 

[pbuk]

What you are trying to do is ridiculous. The only property of π you are interested in is the randomness of its digits (in any base). The place value of its decimal digits is irrelevant and you have been given a perfectly good method of transforming the decimal digits to binary while preserving entropy (just reducing the overall volume of information), plus a pointer towards methods to generate π in binary directly if that is really what you want to do.

Did you really have to do all that work to realize that performing 10⁹ multiplications of 10⁹ digit numbers (which is what base conversion entails) is not computationally feasible?