unsigned int elements = 256 * 1024 * 1024 * 50 * 0.01You're multiplying 256 by 2^20 and by 0.5, which again equals 128 million. Not so fast! Because the operations happen from left to right, at one point you'll end up with a number that's over 13 billion (13,421,772,800 to be exact). Unsigned ints only go up to around 4 billion, so you'll overflow. The problem is you get an intermediate large number before your final answer. A safer approach is to be multiply the 0.01 before the 50 so you get a smaller intermediate number before your final answer. Yes, even though this is obvious, I still made this mistake.
Showing posts with label order of operations. Show all posts
Showing posts with label order of operations. Show all posts
Friday, 12 February 2016
Order of operations and overflow
Let's say you want 50% of 256 million (as in powers of two though like mebibytes, so 2 ^28). Easy, that's 128 million ( 2^27). One could easily write it out as
Subscribe to:
Posts (Atom)