Scribd Logo
Upload

Welcome to Scribd!

  • 88 views

    Out of Space Error Messages Even Though DF Report VXVM

    Uploaded by

    shekhar785424
    The database reported an "out of space" error even though df showed 201GB of free space. Running fsck on the file system fixed the issue and showed many bitmap inconsistencies. The likely root cause was an I/O failure writing bitmaps, marking them as bad. This can cause the file system to report being full even when space is available. The disk was incorrectly presented to multiple servers, causing I/O issues during backups.

    Copyright:

    © All Rights Reserved
    Download as TXT, PDF, TXT or read online from Scribd

    Out of Space Error Messages Even Though DF Report VXVM

    Uploaded by

    shekhar785424
    88 views3 pages
    The database reported an "out of space" error even though df showed 201GB of free space. Running fsck on the file system fixed the issue and showed many bitmap inconsistencies. The likely root cause was an I/O failure writing bitmaps, marking them as bad. This can cause the file system to report being full even when space is available. The disk was incorrectly presented to multiple servers, causing I/O issues during backups.

    Original Description:

    out

    Original Title

    Out of Space Error Messages Even Though Df Report Vxvm

    Copyright

    © © All Rights Reserved

    Available Formats

    TXT, PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    The database reported an "out of space" error even though df showed 201GB of free space. Running fsck on the file system fixed the issue and showed many bitmap inconsistencies. The likely root cause was an I/O failure writing bitmaps, marking them as bad. This can cause the file system to report being full even when space is available. The disk was incorrectly presented to multiple servers, causing I/O issues during backups.

    Copyright:

    © All Rights Reserved
    Download as TXT, PDF, TXT or read online from Scribd
    Download as txt, pdf, or txt
    88 views3 pages

    Out of Space Error Messages Even Though DF Report VXVM

    Uploaded by

    shekhar785424
    The database reported an "out of space" error even though df showed 201GB of free space. Running fsck on the file system fixed the issue and showed many bitmap inconsistencies. The likely root cause was an I/O failure writing bitmaps, marking them as bad. This can cause the file system to report being full even when space is available. The disk was incorrectly presented to multiple servers, causing I/O issues during backups.

    Copyright:

    © All Rights Reserved
    Download as TXT, PDF, TXT or read online from Scribd
    Download as txt, pdf, or txt
    of 3

    "out of space" error messages even though df report an adequate amount of free

    space
    Written by Andy Han
    font size decrease font size increase font size
    Print
    E-mail
    Rate this item
    1
    2
    3
    4
    5
    (1 Vote)
    Our database reported out of space error:
    Errors in file /xxx/ora/admin/xxx/bdump/xxx_p050_15602.trc:
    ORA-01114: IO error writing block to file 401 (block # 790630)
    ORA-27072: File I/O error
    Linux-x86_64 Error: 11: Resource temporarily unavailable Additional information:
    4 Additional information: 790630 Additional information: 16384
    ORA-01114: IO error writing block to file 401 (block # 790630)
    ORA-27072: File I/O error
    Linux-x86_64 Error: 11: Resource temporarily unavailable Additional information:
    4 Additional information: 790630 Additional information: 16384
    We use df to check the file system, it report 201GB free space.
    /dev/vx/dskxxxxxdb_PRD/xxxxx-ora 1.2T 987G 201G 84% /xxxdbprd/ora
    df -h shows system has 201G free space but Oracle reported filesystem is full.
    and nothing new can be created under that file system.
    we removed about 20 GB files under that fs but df still shows 201G free.
    we umount the file system and fsck it and then mount it back issue resolved.
    below is fsck logs, it shows lots of emap incorrect and summary incorrect.
    root@xxx# fsck.vxfs -o full -y /dev/vx/dsk/xxxxx_PRD/xxxdbprd-ora
    log replay in progress
    pass0 - checking structural files
    pass1 - checking inode sanity and blocks
    pass2 - checking directory linkage
    pass3 - checking reference counts
    pass4 - checking resource maps
    au 0 summary incorrect - fix? (ynq)y
    au 10 summary incorrect - fix? (ynq)y
    au 11 summary incorrect - fix? (ynq)y
    au 22 summary incorrect - fix? (ynq)y
    au 23 summary incorrect - fix? (ynq)y
    ....
    au 29571 emap incorrect - fix? (ynq)y
    au 29583 emap incorrect - fix? (ynq)y
    au 29584 emap incorrect - fix? (ynq)y
    au 29585 emap incorrect - fix? (ynq)y
    It appears to be a known issue:
    When the file system has bitmap inconsistencies, it reports "out of space" error
    messages
    even though df may report an adequate amount of free space.
    from logs we see:
    messages.1:Jul 19 09:53:14 xxx kernel: vxfs: msgcnt 14 mesg 056: V-2-56: vx_mapb
    ad - vx_smap_stateupd - /dev/vx/dsk/xxxxdb_PRD/xxx-ora file system extent alloca
    tion unit state bitmap number 9 marked bad
    messages.2:Jul 7 04:16:11 xxx kernel: vxfs: msgcnt 7 mesg 056: V-2-56: vx_mapba
    d - vx_smap_stateupd - /dev/vx/dsk/xxxxdb_PRD/xxx-ora file system extent allocat
    ion unit state bitmap number 7 marked bad
    messages.2:Jul 7 04:16:11 xxx kernel: vxfs: msgcnt 8 mesg 096: V-2-96: vx_setfs
    flags - /dev/vx/dsk/xxxxdb_PRD/xxxxdbprd-ora file system fullfsck flag set - vx_
    mapbad
    If there is an I/O failure while writing a bitmap, the map is marked bad.
    The kernel considers the maps to be invalid, so does not do any more resource al
    location from maps.
    This situation can cause the file system to report "out of space" or "out of ino
    de" error messages
    even though df may report an adequate amount of free space.
    We will follow up with Storage team with possible I/O issue on the disk.
    Article URL http://www.symantec.com/docs/TECH1213
    =====
    root cause identified today:
    data disk present to application server is wrongly present to media server.
    so when backup started, disk will be mounted and fscked on media server. this ca
    used I/O issue on application server.
    we should present snapshot/BCV volume to media server instead of data volume.

    You might also like