Friday 12 February 2016

Order of operations and overflow

Let's say you want 50% of 256 million (as in powers of two though like mebibytes, so 2 ^28). Easy, that's 128 million ( 2^27). One could easily write it out as
unsigned int elements = 256 * 1024 * 1024 * 50 * 0.01
You're multiplying 256 by 2^20 and by 0.5, which again equals 128 million. Not so fast! Because the operations happen from left to right, at one point you'll end up with a number that's over 13 billion (13,421,772,800 to be exact). Unsigned ints only go up to around 4 billion, so you'll overflow. The problem is you get an intermediate large number before your final answer. A safer approach is to be multiply the 0.01 before the 50 so you get a smaller intermediate number before your final answer. Yes, even though this is obvious, I still made this mistake.

No comments:

Post a Comment