#98159

This is intended. I close the issue. cc @sobolevn

@corona10 The issue focuses on int, but it also reports the same problem with bool and tuple. For example:

>>> bool.__basicsize__
32
>>> True.__sizeof__()
28
>>> False.__sizeof__()
24

@kylehofmann

This code will explain why the intention is the same as int.

@corona10 Yes, bool inherits from int, so much of the semantics must be the same. Are you saying that, because of this, it would be impossible to provide bool__sizeof__ and bool__sizeof__impl functions that returned the actual size? Or would that cause some of the same problems as changing ob_size?

Also, tuple has the same issue. If we create a million tuple subclasses (for example using namedtuple) then used memory will get undercounted by the size of a million PyObject *.

@kylehofmann

Yeah, since the semantics of False and 0 in most of the cases are the same, I thought that this issue should be handled as a unified approach.
But @mdickinson may have the more accurate answer about this issue.

We could fix this for False and 0 (and the empty tuple), by providing type-specific overrides for __sizeof__.

What we can't do is fix it by messing with the value of the ob_size field, since that field is used for other purposes (giving the length in the case of a tuple, or the value k satisfying BASE**(k-1) <= abs(value) < BASE**k in the case of a nonzero int); that was the point of my comment in #98159.

The other thing we can't do is fix this for general ints, since the int object doesn't store information about how many limbs were allocated.

There are other aspects that sys.getsizeof doesn't handle, like the fact that the default allocator only ever allocates chunks in multiples of 8 bytes (for alignment reasons), so sys.getsizeof(1) (which currently returns 28 on most builds) is also kinda wrong - 32 would be more accurate.

@ionite34

creates unsafe scenarios when moving memory

Can you elaborate on what you're doing here? This may not be a task that sys.getsizeof is intended for, or well-suited to.

See also #47940 for previous discussion.

Given that people have been discovering and re-filing issues about this for well over a decade, maybe it's worth a remark in the source? Say, in int__sizeof__impl, the comment, "This is wrong for the zero singleton and bool objects, but for other reasons we can't change their ob_size."

We could fix this for False and 0 (and the empty tuple), by providing type-specific overrides for __sizeof__.

This was essentially what I was going for. I agree changes on ob_size would be impossible.

The other thing we can't do is fix this for general ints, since the int object doesn't store information about how many limbs were allocated.

This is true, but on the other hand, is there a reason 0, False, and empty tuples are even allocated and use a 1 length array? The code paths I've seen never touch the item array when ob_size is 0. So it seems it doesn't need to exist at all?

C99 supports declaring the respective structs with
https://www.ibm.com/docs/en/i/7.2?topic=declarations-flexible-array-members

digit ob_digit[];

Given that people have been discovering and re-filing issues about this for well over a decade, maybe it's worth a remark in the source? Say, in int__sizeof__impl, the comment, "This is wrong for the zero singleton and bool objects, but for other reasons we can't change their ob_size."

+1 on this as well, and also regarding the sys.getsizeof documentation, which states:

Return the size of an object in bytes. The object can be any type of object. All built-in objects will return correct results

This is true, but on the other hand, is there a reason 0, False, and empty tuples are even allocated and use a 1 length array?

For 0 and False, yes: this is a deliberate decision to make some of the fast paths for arithmetic simpler. E.g., in this code we access x->ob_digit[0] unconditionally. For tuples, it may well be just a holdover from before we were allowed to assume C99.

[...] empty tuples are even allocated and use a 1 length array [...]

Regarding tuples: it doesn't seem worth worrying too much about the empty tuple, since it's a singleton, so we're only taking about an extra 8 bytes per interpreter instance.

For tuple subclasses, I'm not seeing the behaviour you describe. On main, after instrumenting calls to PyObject_Malloc, if I do class mytuple(tuple): pass, then on my machine each mytuple instance asks for 64 + 8*tuple_length bytes, and the case of zero length is not exceptional (it asks for 64 bytes, not 72 bytes).

For 0 and False, yes: this is a deliberate decision to make some of the fast paths for arithmetic simpler. E.g., in this code we access x->ob_digit[0] unconditionally.

This leads me to think that the ob_digit[1] of [0] is not considered just an "extra allocation" but actually used by the int / bool structs. Such it seems that this should definitely be reported in __sizeof__

Despite int(0) being cached, user subclasses are still affected by the __sizeof__ impl, where the difference can be significant for many allocations.

class UserInt(int):
    ...

zero = 0
uzero = UserInt()

print(zero.__sizeof__())
>> 24
print(uzero.__sizeof__())
>> 24

In addition, since 24 falls within a 8 byte allocation boundary, users are mislead to think the allocated size is 24 when it is actually 32 due to the requested being 28

@ionite34

Such it seems that this should definitely be reported in __sizeof__.

Would you be interested in working on a PR for this?

the default allocator only ever allocates chunks in multiples of 8 bytes (for alignment reasons)

Just curious: Is that new or common? I thought it's 16 (at least usually), so 24 or 28 wouldn't make a difference, both actually costing 32.

@pochmann

I thought it's 16 (at least usually)

Sorry, yes; I think you're right:

Regarding tuples: it doesn't seem worth worrying too much about the empty tuple, since it's a singleton

Following up on this, sys.getsizeof is actually over-reporting for the empty tuple. On a 64-bit machine, it looks as though in the current code the empty tuple singleton uses just 24 bytes (8 each for the refcount, type pointer and size field), while sys.getsizeof reports 40 (it's allowing for two fields for gc-tracking, which don't exist for the non-gc-tracked immortal empty tuple).

@ionite34 It looks to me as though tuples should be out of scope for this issue. If there's behaviour that you think should be changed for tuple or tuple subclasses, I'd suggest opening a new issue for that, so that it can be dealt with separately.

Would you be interested in working on a PR for this?

Yes, sure.

Though it'd be my first contribution, not too sure about the process but if I'm reading the guide correctly I can just make the changes on a branch and do a PR?

And so the current scope of a fix would be changing int___sizeof___impl in longobject.c to return 28 instead of 24 for ob_size 0 empty ints and bools?

Implementation-wise I think it would be simplest just to take Py_MIN of the current absolute size and 1.

I'm reading the guide correctly I can just make the changes on a branch and do a PR?

Yep. If you ping me when it's done, I'll review.

And so the current scope of a fix would be changing int___sizeof___impl in longobject.c to return 28 instead of 24 for ob_size 0 empty ints and bools?

Yes, preferably along with a unit test or two. (I haven't checked the current tests, but I'd hope that making the suggested change would trigger a test failure, so that it's obvious which part of the tests need to be updated.)

Implementation-wise I think it would be simplest just to take Py_MIN of the current absolute size and 1.

SGTM (with Py_MAX in place of Py_MIN).

Last word on tuples: it turns out that there is an under-reporting issue for sys.getsizeof applied to instances of tuple subclasses, but it's not related to the PyObject *ob_item[1]; declaration, and not specific to length-0 tuples. I've opened #100659 for this.

This was fixed (for int and bool) by #100663 (and backported to 3.11 in #100717).