About deleting a VM

When operating the vSphere Client, the right-click menu is Delete from inventory and Delete from disk are next to each other.

Both give warnings, but I inadvertently intend to remove them from my inventory. If you delete it from the disk, the VM data will be buried and you will not be able to use it.

Since the deletion itself ends immediately, it is presumed that 0 has not been overwritten. I think the data itself remains I would like to find out if the deleted files can be recovered.

Survey flow

Since the specification of VMFS is not published, Try to investigate by referring to the source of vmfs-tools. It doesn't seem to be perfect for handling VMFS5, but it's a challenge for the time being.

Install vSphere ESXi as a guest on the host vSphere ESXi
(The VMFS area of the vmdk file with this guest ESXi installed will be investigated)
Install CentOS as a Nested VM
(Add CentOS as a guest to guest ESXi)
Shut down Netsed VM and guest ESXi and copy vmdk of guest ESXi
Check the situation in advance with the Nested VM file existing.
Find out what changed after removing the Nested VM from disk
(Also copy the deleted vmdk file and check the VMFS area)

In the Nested environment, the VMFS-formatted partition under investigation is in the vmdk file, so Easy to take to other places for work.

Survey started

The <vmname> .vmdk file is called a virtual disk descriptor file and is not the body of the disk data, so Copy the <vmname> -flat.vmdk file. The target here is named __backup_vsphere-flat.vmdk.

Well, how should I find it? First, with the Nested VM not deleted Let's see if we can find the data for the Nested VM.

The search target is the MBR of the Nested VM. Transfer the VMDK of guest ESXi to another Linux and start investigating there.

However, I don't know much about MBR, so let's check the structure first. Roughly understand the following based on the reference materials.

446 bytes from the beginning is the boot loader
64 bytes behind the bootloader is a partition table (4 records of 16 bytes each)
The last 2 bytes are the magic number

If you read the MBR of CentOS with GRUB installed at hand, you will find the following:

A string that GRUB is likely to use
Partition table
Magic number

Use these as clues to find the MBR. For Nested VM, I created a 10GB VMDK and installed two of them. I cut each partition differently. The guest ESXi vmdk is made with 40GB, and I search from end to end for the time being.

`searchmbr.py`


#!/usr/bin/env python
# -*- coding: utf-8 -*-

import sys, os, re, struct
#target = './mbr.dd'
target = './__backup_vsphere-flat.vmdk'
MAGIC_OF_MBR = '\x55\xaa'
MARK_OF_GRUB = 'GRUB \x00Geom\x00Hard Disk\x00Read\x00 Error'

try:
    if sys.argv[1] == '-d':
        DEBUG = 1
except:
    pass

def D(mesg):
    try:
        if DEBUG:
            import inspect
            print "[DEBUG:%d] %s" % (inspect.currentframe().f_back.f_lineno, mesg)
    except NameError:
        pass

def convert_num(num):
    if len(str(int(num))) > 12:
        return '%.1fT' % (num/(1024.0**4))
    elif len(str(int(num))) > 9:
        return '%.1fG' % (num/(1024.0**3))
    elif len(str(int(num))) > 6:
        return '%.1fM' % (num/(1024.0**2))
    elif len(str(int(num))) > 3:
        return '%.1fk' % (num/(1024.0**1))
    else:
        return str(num)


def check(data):
    bootloader = data[0:446]
    part_table = data[446:510]
    magic = data[510:512]
    if magic != MAGIC_OF_MBR:
        return False
    if re.search(MARK_OF_GRUB, bootloader):
        print 'This is maybe MBR !!'
    else:
        return False

    splitter = lambda s, n: [s[i:i+n] for i in range(0, len(s), n)]
    print "Partition    Boot    %s    %s    Id    Size" % ('Start'.rjust(10), 'End'.rjust(10))
    i=1
    for record in splitter(part_table, 16):
        boot = record[0]
        D('flag: %s' % hex(ord(boot)))
        if boot == '\x80':
            boot = '*'
        elif boot == '\x00':
            boot = ''
        else:
            print 'boot flag is wrong'
            boot = hex(ord(boot))
        id = record[4:5]
        #Check only LBA data(Don't look at CHS data)
        start_sector, num_sector = struct.unpack('II', record[8:16])
        print "%s    %s    %10d    %10d  %s    %sB" % (str(i).rjust(9), boot.rjust(4), start_sector, start_sector + num_sector, hex(ord(id)).rjust(4), convert_num(num_sector * 512))
        i+=1
    return True


with open(target , 'r') as f:
    try:
        fd = f.fileno()
    except Exception as e:
        print e
        raise Exception('cannot get file size')
    else:
        size = os.fstat(fd).st_size

    i=1
    data = f.read(512)
    while data:
        if check(data):
            print "position of above data: %d - %d" %  (f.tell() - 512, f.tell())
        data = f.read(512)
        if f.tell() > size / 100.0 * i:
            D("searching... %d / %d" % (f.tell(), size))
            i+=1

`MBR search`


# python searchmbr.py
This is maybe MBR !!
Partition    Boot         Start           End    Id    Size
        1       *          2048       1026048  0x83    500.0MB
        2               1026048       2074624  0x82    512.0MB
        3               2074624      20971520  0x83    9.0GB
        4                     0             0   0x0    0B
position of above data: 38196477952 - 38196478464
This is maybe MBR !!
Partition    Boot         Start           End    Id    Size
boot flag is wrong
        1    0x24     332251185    2480556663   0x0    1.0TB
boot flag is wrong
        2    0xf9    2548826057    4435176463  0xbe    899.5GB
boot flag is wrong
        3    0x70    3053499650    4967399938   0x0    912.6GB
boot flag is wrong
        4    0xd7             0             0  0x4f    0B
position of above data: 38792327168 - 38792327680
This is maybe MBR !!
Partition    Boot         Start           End    Id    Size
boot flag is wrong
        1    0x24     332251185    2480556663   0x0    1.0TB
boot flag is wrong
        2    0xf9    2548826057    4435176463  0xbe    899.5GB
boot flag is wrong
        3    0x70    3053499650    4967399938   0x0    912.6GB
boot flag is wrong
        4    0xd7             0             0  0x4f    0B
position of above data: 39116325888 - 39116326400
This is maybe MBR !!
Partition    Boot         Start           End    Id    Size
        1       *          2048       1026048  0x83    500.0MB
        2               1026048      20971520  0x8e    9.5GB
        3                     0             0   0x0    0B
        4                     0             0   0x0    0B
position of above data: 39419117568 - 39419118080
This is maybe MBR !!
Partition    Boot         Start           End    Id    Size
boot flag is wrong
        1    0x24     332251185    2480556663   0x0    1.0TB
boot flag is wrong
        2    0xf9    2548826057    4435176463  0xbe    899.5GB
boot flag is wrong
        3    0x70    3053499650    4967399938   0x0    912.6GB
boot flag is wrong
        4    0xd7             0             0  0x4f    0B
position of above data: 40218379264 - 40218379776
This is maybe MBR !!
Partition    Boot         Start           End    Id    Size
boot flag is wrong
        1    0x24     332251185    2480556663   0x0    1.0TB
boot flag is wrong
        2    0xf9    2548826057    4435176463  0xbe    899.5GB
boot flag is wrong
        3    0x70    3053499650    4967399938   0x0    912.6GB
boot flag is wrong
        4    0xd7             0             0  0x4f    0B
position of above data: 40242606080 - 40242606592

I was able to find it safely. Some strange flags are found, but the Nested VM I made it with 10GB vmdk, so I'm sure the two flags are correct.

VMFS exploration

Now let's read VMFS. From here is the production.

# parted __backup_vsphere-flat.vmdk unit s print
model:  (file)
disk/root/recover/__backup_vsphere-flat.vmdk: 83886080s
Sector size(logic/Physics): 512B/512B
Partition table: gpt

Number Start End Size File System Name Flag
 1    64s        8191s      8128s      fat16                   boot
 5    8224s      520191s    511968s    fat16
 6    520224s    1032191s   511968s    fat16
 7    1032224s   1257471s   225248s
 8    1257504s   1843199s   585696s    fat16
 2    1843200s   10229759s  8386560s   fat16
 3    10229760s  83886046s  73656287s

Partition table is GPT The largest area should be VMFS, but I'm not sure with parted.

In GPT, the GUID of VMFS is ʻAA31E02A400F11DB9590000C2911D1B8. If you change the notation, ʻAA31E02A-400F-11DB-9590-000C2911D1B8 Looking at the location corresponding to the large partition No. 3, The first three items are little endian. Certainly the GUIDs matched.

`GPT survey`


# hexdump -C -s `expr 512 \* 2 + 128 \* 2` -n 128 vsphere-flat.vmdk | less
00000500  2a e0 31 aa 0f 40 db 11  95 90 00 0c 29 11 d1 b8  |*.1..@......)...|
00000510  45 01 35 2b 41 d6 59 45  aa 77 33 40 c0 94 f4 0d  |E.5+A.YE.w3@....|
00000520  00 18 9c 00 00 00 00 00  de ff ff 04 00 00 00 00  |................|
00000530  00 00 00 00 00 00 00 00  00 00 ff ff ff ff ff ff  |................|
00000540  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|

I want to continue reading information about VMFS.

`vmfs_fs.h`


#define VMFS_FSINFO_MAGIC  0x2fabf15e

struct vmfs_fsinfo_raw {
   uint32_t magic;
   uint32_t volver;
   u_char ver;
   uuid_t uuid;
   uint32_t mode;
   char label[128];
   uint32_t dev_blocksize;
   uint64_t blocksize;
   uint32_t ctime; /* ctime? in seconds */
   uint32_t _unknown3;
   uuid_t lvm_uuid;
   u_char _unknown4[16];
   uint32_t fdc_header_size;
   uint32_t fdc_bitmap_count;
   uint32_t subblock_size;
} __attribute__((packed));

Looking at vmfs-tools, it seems that there is a magic number, so try to find it in the same way as for MBR.

`searchvmfsinfo.py`


#!/usr/bin/env python
# -*- coding: utf-8 -*-

import sys, os, re, struct
target = './__backup_vsphere-flat.vmdk'

MAGIC_OF_VMFSINFO = '\x5e\xf1\xab\x2f'

try:
    if sys.argv[1] == '-d':
        DEBUG = 1
except:
    pass

def D(mesg):
    try:
        if DEBUG:
            import inspect
            print "[DEBUG:%d] %s" % (inspect.currentframe().f_back.f_lineno, mesg)
    except NameError:
        pass

def check(data):
    if MAGIC_OF_VMFSINFO in data:
        print 'Magic Number found. This is maybe VMFS !!'
        return True
    else:
        return False

with open(target , 'r') as f:
    try:
        fd = f.fileno()
    except Exception as e:
        print e
        raise Exception('cannot get file size')
    else:
        size = os.fstat(fd).st_size

    i=1
    data = f.read(512)
    while data:
        if check(data):
            print "position of above data: %d - %d" %  (f.tell() - 512, f.tell())
        data = f.read(512)
        if f.tell() > size / 100.0 * i:
            D("searching... %d / %d" % (f.tell(), size))
            i+=1

`vmfsinfo search`


# python searchvmfs.py
Magic Number found. This is maybe VMFS !!
position of above data: 5257560064 - 5257560576
Magic Number found. This is maybe VMFS !!
position of above data: 37435702784 - 37435703296
Magic Number found. This is maybe VMFS !!
position of above data: 38707637760 - 38707638272
Magic Number found. This is maybe VMFS !!
position of above data: 39859326464 - 39859326976
Magic Number found. This is maybe VMFS !!
position of above data: 40373243392 - 40373243904

I found some. The beginning of the VMFS partition is 10229760 * 512 = 5237637120, so it doesn't seem to start right from the beginning. I'm not sure if there's a gap, but maybe the first place I found it was VMFS information. Did the others just happen to match? You can see 128 bytes of label, so let's check it out.

`Check label data`


# hexdump -C -s `expr 5257560064 + 29` -n 128 __backup_vsphere-flat.vmdk
13960001d  64 61 74 61 73 74 6f 72  65 31 00 00 00 00 00 00  |datastore1......|
13960002d  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
# hexdump -C -s `expr 37435702784 + 29` -n 128 __backup_vsphere-flat.vmdk
8b7577e1d  00 73 71 75 61 73 68 66  73 00 68 73 71 73 00 73  |.squashfs.hsqs.s|
8b7577e2d  68 73 71 00 25 75 2e 25  30 32 75 00 6e 73 73 00  |hsq.%u.%02u.nss.|
8b7577e3d  53 50 42 35 00 78 65 6e  69 78 00 2b 55 44 00 44  |SPB5.xenix.+UD.D|
8b7577e4d  55 2b 00 73 79 73 76 00  62 74 72 66 73 00 5f 42  |U+.sysv.btrfs._B|
8b7577e5d  48 52 66 53 5f 4d 00 4c  41 42 45 4c 4f 4e 45 00  |HRfS_M.LABELONE.|
8b7577e6d  4c 56 4d 32 5f 6d 65 6d  62 65 72 00 4c 56 4d 32  |LVM2_member.LVM2|
8b7577e7d  20 30 30 31 00 4c 56 4d  31 5f 6d 65 6d 62 65 72  | 001.LVM1_member|
8b7577e8d  00 48 4d 00 44 4d 5f 73  6e 61 70 73 68 6f 74 5f  |.HM.DM_snapshot_|
8b7577e9d
# hexdump -C -s `expr 38707637760 + 29` -n 128 __backup_vsphere-flat.vmdk
90327ae1d  00 73 71 75 61 73 68 66  73 00 68 73 71 73 00 73  |.squashfs.hsqs.s|
90327ae2d  68 73 71 00 25 75 2e 25  30 32 75 00 6e 73 73 00  |hsq.%u.%02u.nss.|
90327ae3d  53 50 42 35 00 78 65 6e  69 78 00 2b 55 44 00 44  |SPB5.xenix.+UD.D|
90327ae4d  55 2b 00 73 79 73 76 00  62 74 72 66 73 00 5f 42  |U+.sysv.btrfs._B|
90327ae5d  48 52 66 53 5f 4d 00 4c  41 42 45 4c 4f 4e 45 00  |HRfS_M.LABELONE.|
90327ae6d  4c 56 4d 32 5f 6d 65 6d  62 65 72 00 4c 56 4d 32  |LVM2_member.LVM2|
90327ae7d  20 30 30 31 00 4c 56 4d  31 5f 6d 65 6d 62 65 72  | 001.LVM1_member|
90327ae8d  00 48 4d 00 44 4d 5f 73  6e 61 70 73 68 6f 74 5f  |.HM.DM_snapshot_|
90327ae9d
# hexdump -C -s `expr 39859326464 + 29` -n 128 __backup_vsphere-flat.vmdk
947cd0e1d  00 73 71 75 61 73 68 66  73 00 68 73 71 73 00 73  |.squashfs.hsqs.s|
947cd0e2d  68 73 71 00 25 75 2e 25  30 32 75 00 6e 73 73 00  |hsq.%u.%02u.nss.|
947cd0e3d  53 50 42 35 00 78 65 6e  69 78 00 2b 55 44 00 44  |SPB5.xenix.+UD.D|
947cd0e4d  55 2b 00 73 79 73 76 00  62 74 72 66 73 00 5f 42  |U+.sysv.btrfs._B|
947cd0e5d  48 52 66 53 5f 4d 00 4c  41 42 45 4c 4f 4e 45 00  |HRfS_M.LABELONE.|
947cd0e6d  4c 56 4d 32 5f 6d 65 6d  62 65 72 00 4c 56 4d 32  |LVM2_member.LVM2|
947cd0e7d  20 30 30 31 00 4c 56 4d  31 5f 6d 65 6d 62 65 72  | 001.LVM1_member|
947cd0e8d  00 48 4d 00 44 4d 5f 73  6e 61 70 73 68 6f 74 5f  |.HM.DM_snapshot_|
947cd0e9d
# hexdump -C -s `expr 40373243392 + 29` -n 128 __backup_vsphere-flat.vmdk
9666ece1d  00 73 71 75 61 73 68 66  73 00 68 73 71 73 00 73  |.squashfs.hsqs.s|
9666ece2d  68 73 71 00 25 75 2e 25  30 32 75 00 6e 73 73 00  |hsq.%u.%02u.nss.|
9666ece3d  53 50 42 35 00 78 65 6e  69 78 00 2b 55 44 00 44  |SPB5.xenix.+UD.D|
9666ece4d  55 2b 00 73 79 73 76 00  62 74 72 66 73 00 5f 42  |U+.sysv.btrfs._B|
9666ece5d  48 52 66 53 5f 4d 00 4c  41 42 45 4c 4f 4e 45 00  |HRfS_M.LABELONE.|
9666ece6d  4c 56 4d 32 5f 6d 65 6d  62 65 72 00 4c 56 4d 32  |LVM2_member.LVM2|
9666ece7d  20 30 30 31 00 4c 56 4d  31 5f 6d 65 6d 62 65 72  | 001.LVM1_member|
9666ece8d  00 48 4d 00 44 4d 5f 73  6e 61 70 73 68 6f 74 5f  |.HM.DM_snapshot_|
9666ece9d

You can see the label datastore1 created by default from the first place you find it. I don't know what else, but for now it looks like it can be ignored.

To be honest, I don't know what to do from here, but there seems to be an inode.

`vmfs_inode.h`


struct vmfs_inode_raw {
   struct vmfs_metadata_hdr_raw mdh;
   uint32_t id;
   uint32_t id2; /* seems to be VMFS_BLK_FD_ITEM(id) + 1 */
   uint32_t nlink;
   uint32_t type;
   uint32_t flags;
   uint64_t size;
   uint64_t blk_size;
   uint64_t blk_count;
   uint32_t mtime;
   uint32_t ctime;
   uint32_t atime;
   uint32_t uid;
   uint32_t gid;
   uint32_t mode;
   uint32_t zla;
   uint32_t tbz;
   uint32_t cow;
   u_char _unknown2[432];
   union {
      uint32_t blocks[VMFS_INODE_BLK_COUNT];
      uint32_t rdm_id;
      char content[VMFS_INODE_BLK_COUNT * sizeof(uint32_t)];
   };
} __attribute__((packed));

`vmfs_metadata.h`


struct vmfs_metadata_hdr_raw {
   uint32_t magic; /* Magic number */
   uint64_t pos; /* Position in the volume */
   uint64_t hb_pos; /* Heartbeat position */
   uint64_t hb_seq; /* Heartbeat sequence */
   uint64_t obj_seq; /* Object sequence */
   uint32_t hb_lock; /* Heartbeat lock flag */
   uuid_t hb_uuid; /* UUID of locking server */
   uint64_t mtime;
   u_char pad1[0x1c0]; /* Padding/unknown */
} __attribute__((packed));

Anyway, if I can find an inode, I feel like I can do something about it.

`vmfs_inode.h`


#define VMFS_INODE_MAGIC  0x10c00001

The inode also seems to have a magic number, so look for this. It seems to be a clue if you display id and size by the way.

`searchinode.py`


#!/usr/bin/env python
# -*- coding: utf-8 -*-

import sys, os, re, struct
target = './__backup_vsphere-flat.vmdk'

POS_OF_VMFSINFOMAGIC = 5257560064
MAGIC_OF_VMFSINODE = '\x01\x00\xc0\x10'

try:
    if sys.argv[1] == '-d':
        DEBUG = 1
except:
    pass

def D(mesg):
    try:
        if DEBUG:
            import inspect
            print "[DEBUG:%d] %s" % (inspect.currentframe().f_back.f_lineno, mesg)
    except NameError:
        pass

def convert_num(num):
    if len(str(int(num))) > 12:
        return '%.1fT' % (num/(1024.0**4))
    elif len(str(int(num))) > 9:
        return '%.1fG' % (num/(1024.0**3))
    elif len(str(int(num))) > 6:
        return '%.1fM' % (num/(1024.0**2))
    elif len(str(int(num))) > 3:
        return '%.1fk' % (num/(1024.0**1))
    else:
        return str(num)


with open(target , 'r') as f:
    def check(data):
        if MAGIC_OF_VMFSINODE in data:
            print 'Magic Number found. This is maybe inode !!'
            data = f.read(512)
            id, id2, size = struct.unpack('IIQ', data[0:8] + data[20:28])
            print 'id = %d, id2 = %d, filesize = %sB' % (id, id2, convert_num(size))
            return True
        else:
            return False

    try:
        fd = f.fileno()
    except Exception as e:
        print e
        raise Exception('cannot get file size')
    else:
        size = os.fstat(fd).st_size

    f.seek(POS_OF_VMFSINFOMAGIC)

    i=1
    data = f.read(512)
    while data:
        if check(data):
            print "position of above data: %d - %d" %  (f.tell() - 1024, f.tell() - 512)
        data = f.read(512)
        if f.tell() > size / 100.0 * i:
            D("searching... %d / %d" % (f.tell(), size))
            i+=1

`inode search`


# python searchinode.py
Magic Number found. This is maybe inode !!
id = 4, id2 = 1, filesize = 1.4kB
position of above data: 5259788288 - 5259788800
Magic Number found. This is maybe inode !!
id = 4194308, id2 = 1, filesize = 256.0kB
position of above data: 5259790336 - 5259790848
Magic Number found. This is maybe inode !!
id = 8388612, id2 = 1, filesize = 254.7MB
position of above data: 5259792384 - 5259792896
Magic Number found. This is maybe inode !!
id = 12582916, id2 = 1, filesize = 256.0MB
position of above data: 5259794432 - 5259794944
Magic Number found. This is maybe inode !!
id = 16777220, id2 = 1, filesize = 250.6MB
position of above data: 5259796480 - 5259796992
Magic Number found. This is maybe inode !!
id = 20971524, id2 = 1, filesize = 4.0MB
position of above data: 5259798528 - 5259799040
Magic Number found. This is maybe inode !!
id = 25165828, id2 = 1, filesize = 1.1MB
position of above data: 5259800576 - 5259801088
Magic Number found. This is maybe inode !!
id = 32772, id2 = 1, filesize = 0B
position of above data: 5472124928 - 5472125440
Magic Number found. This is maybe inode !!
id = 4227076, id2 = 2, filesize = 0B
position of above data: 5472126976 - 5472127488
Magic Number found. This is maybe inode !!
id = 8421380, id2 = 3, filesize = 0B
position of above data: 5472129024 - 5472129536
Magic Number found. This is maybe inode !!
id = 12615684, id2 = 4, filesize = 0B
position of above data: 5472131072 - 5472131584
Magic Number found. This is maybe inode !!
id = 16809988, id2 = 5, filesize = 0B
position of above data: 5472133120 - 5472133632
Magic Number found. This is maybe inode !!
id = 21004292, id2 = 6, filesize = 0B
position of above data: 5472135168 - 5472135680
Magic Number found. This is maybe inode !!
id = 25198596, id2 = 7, filesize = 0B
position of above data: 5472137216 - 5472137728
Magic Number found. This is maybe inode !!
id = 29392900, id2 = 8, filesize = 0B
position of above data: 5472139264 - 5472139776
Magic Number found. This is maybe inode !!
id = 33587204, id2 = 9, filesize = 0B
position of above data: 5472141312 - 5472141824
Magic Number found. This is maybe inode !!
id = 37781508, id2 = 10, filesize = 0B
position of above data: 5472143360 - 5472143872
Magic Number found. This is maybe inode !!
id = 41975812, id2 = 11, filesize = 0B
position of above data: 5472145408 - 5472145920
Magic Number found. This is maybe inode !!
id = 46170116, id2 = 12, filesize = 0B
position of above data: 5472147456 - 5472147968
Magic Number found. This is maybe inode !!
id = 50364420, id2 = 14, filesize = 0B
position of above data: 5472149504 - 5472150016
Magic Number found. This is maybe inode !!
id = 54558724, id2 = 15, filesize = 0B
position of above data: 5472151552 - 5472152064
Magic Number found. This is maybe inode !!
id = 58753028, id2 = 17, filesize = 0B
position of above data: 5472153600 - 5472154112
Magic Number found. This is maybe inode !!
id = 62947332, id2 = 18, filesize = 0B
position of above data: 5472155648 - 5472156160
Magic Number found. This is maybe inode !!
id = 67141636, id2 = 20, filesize = 1.4kB
position of above data: 5472157696 - 5472158208
Magic Number found. This is maybe inode !!
id = 71335940, id2 = 21, filesize = 2.6kB
position of above data: 5472159744 - 5472160256
Magic Number found. This is maybe inode !!
id = 75530244, id2 = 22, filesize = 261B
position of above data: 5472161792 - 5472162304
Magic Number found. This is maybe inode !!
id = 79724548, id2 = 23, filesize = 0B
position of above data: 5472163840 - 5472164352
Magic Number found. This is maybe inode !!
id = 83918852, id2 = 24, filesize = 10.0GB
position of above data: 5472165888 - 5472166400
Magic Number found. This is maybe inode !!
id = 88113156, id2 = 25, filesize = 493B
position of above data: 5472167936 - 5472168448
Magic Number found. This is maybe inode !!
id = 92307460, id2 = 26, filesize = 0B
position of above data: 5472169984 - 5472170496
Magic Number found. This is maybe inode !!
id = 96501764, id2 = 27, filesize = 64.8kB
position of above data: 5472172032 - 5472172544
Magic Number found. This is maybe inode !!
id = 100696068, id2 = 28, filesize = 0B
position of above data: 5472174080 - 5472174592
Magic Number found. This is maybe inode !!
id = 104890372, id2 = 29, filesize = 8.5kB
position of above data: 5472176128 - 5472176640
Magic Number found. This is maybe inode !!
id = 109084676, id2 = 30, filesize = 0B
position of above data: 5472178176 - 5472178688
Magic Number found. This is maybe inode !!
id = 113278980, id2 = 31, filesize = 123.1kB
position of above data: 5472180224 - 5472180736
Magic Number found. This is maybe inode !!
id = 117473284, id2 = 33, filesize = 1.2kB
position of above data: 5472182272 - 5472182784
Magic Number found. This is maybe inode !!
id = 121667588, id2 = 34, filesize = 2.5kB
position of above data: 5472184320 - 5472184832
Magic Number found. This is maybe inode !!
id = 125861892, id2 = 35, filesize = 261B
position of above data: 5472186368 - 5472186880
Magic Number found. This is maybe inode !!
id = 130056196, id2 = 36, filesize = 0B
position of above data: 5472188416 - 5472188928
Magic Number found. This is maybe inode !!
id = 134250500, id2 = 37, filesize = 10.0GB
position of above data: 5472190464 - 5472190976
Magic Number found. This is maybe inode !!
id = 138444804, id2 = 38, filesize = 493B
position of above data: 5472192512 - 5472193024
Magic Number found. This is maybe inode !!
id = 142639108, id2 = 39, filesize = 0B
position of above data: 5472194560 - 5472195072
Magic Number found. This is maybe inode !!
id = 146833412, id2 = 40, filesize = 119.8kB
position of above data: 5472196608 - 5472197120
Magic Number found. This is maybe inode !!
id = 151027716, id2 = 41, filesize = 0B
position of above data: 5472198656 - 5472199168
Magic Number found. This is maybe inode !!
id = 155222020, id2 = 42, filesize = 8.5kB
position of above data: 5472200704 - 5472201216
^CTraceback (most recent call last):
  File "searchinode.py", line 66, in <module>
    data = f.read(512)
KeyboardInterrupt

I found something like that, so I stopped. If you look at the size, there is something that shows 10.0GB, so this is It looks like an inode in the <vmname> -flat.vmdk file. I can't keep up with how it is mapped to the actual block, This \ <vmname >-flat.vmdk inode at 0x400 is For pointer block id (in vmfs-tools), read the pointer block for that id As a result of various processing, it seems that the actual block can be found.

Anyway, it seems to be mapping using this area, so It seems that it can be recovered if such information remains after it is deleted.

Let's check this inode from the ESXi shell once. If 83918852 and 134250500 are <vmname> -flat.vmdk, that's what you expected.

`inode display`


# pwd
/vmfs/volumes/datastore1
# ls -li */*vmdk
83918852 -rw-------    1 root     root        10737418240 Jun 13 13:31 test01/test01-flat.vmdk
88113156 -rw-------    1 root     root                493 Jun 13 13:03 test01/test01.vmdk
134250500 -rw-------    1 root     root        10737418240 Jun 13 13:57 test02/test02-flat.vmdk
138444804 -rw-------    1 root     root                493 Jun 13 13:38 test02/test02.vmdk

There seems to be no fatal misunderstanding so far. Delete it immediately and check the data. Connect with the vSphere Client and delete with Delete from Disk. It seems that vmdk cannot be accessed when guest ESXi is running, so shut it down and check.

`Deleted from disk`


# pwd
/vmfs/volumes/datastore1
# ls -a
.        ..       .fbb.sf  .fdc.sf  .pb2.sf  .pbc.sf  .sbc.sf  .vh.sf

Compare with the data before deletion.

`Inode information before deletion`


## <vmname>-flat.inode indicating vmdk
# hexdump -C -s 5472165888 -n 2048 __backup_vsphere-flat.vmdk
1462aa000  01 00 c0 10 00 a0 ea 0c  00 00 00 00 00 00 3e 00  |..............>.|
1462aa010  00 00 00 00 1f 00 00 00  00 00 00 00 20 00 00 00  |............ ...|
1462aa020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
1462aa030  00 00 00 00 00 00 00 00  91 05 00 00 00 00 00 00  |................|
1462aa040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
1462aa200  04 80 00 05 18 00 00 00  01 00 00 00 03 00 00 00  |................|
1462aa210  00 00 00 00 00 00 00 80  02 00 00 00 00 00 10 00  |................|
1462aa220  00 00 00 00 52 04 00 00  00 00 00 00 4d fd 9a 53  |....R.......M..S|
1462aa230  f6 f8 9a 53 4d fd 9a 53  00 00 00 00 00 00 00 00  |...SM..S........|
1462aa240  80 01 00 00 03 00 00 00  00 00 00 00 00 00 00 00  |................|
1462aa250  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
1462aa400  03 ed 02 10 03 ed 02 30  03 ed 02 40 03 ed 02 50  |[email protected]|
1462aa410  03 ed 02 60 03 ed 02 70  03 ed 02 80 03 ed 02 90  |...`...p........|
1462aa420  03 ed 02 a0 03 ed 02 20  00 00 00 00 00 00 00 00  |....... ........|
1462aa430  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
1462aa800

`Inode information after deletion`


##I couldn't skip directly with hexdump on the vSphere shell, so skip it with dd and then pass it

## <vmname>-flat.inode indicating vmdk
# dd if=vsphere-flat.vmdk bs=512 skip=10687824 count=4
 2> /dev/null | hexdump -C
00000000  01 00 c0 10 00 a0 ea 0c  00 00 00 00 00 f0 3d 00  |..............=.|
00000010  00 00 00 00 11 00 00 00  00 00 00 00 24 00 00 00  |............$...|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000030  00 00 00 00 00 00 00 00  d3 00 00 00 00 00 00 00  |................|
00000040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200  04 80 00 05 18 00 00 00  00 00 00 00 03 00 00 00  |................|
00000210  00 00 00 00 00 00 00 00  00 00 00 00 00 00 10 00  |................|
00000220  00 00 00 00 00 00 00 00  00 00 00 00 4d fd 9a 53  |............M..S|
00000230  e8 a9 9d 53 e1 7b 9d 53  00 00 00 00 00 00 00 00  |...S.{.S........|
00000240  80 01 00 00 03 00 00 00  00 00 00 00 00 00 00 00  |................|
00000250  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000800

The location that is supposed to be used for mapping has been overwritten with 0s. ... I can't handle this, let's give up. And let's go to bed.

The small files seemed to fit directly in the inode I also checked that, but the data was found as it was in this file.

`Check the small file inode from the deleted vmdk`


## <vmname>.vmdk (descriptor)Indicates an inode
# dd if=vsphere-flat.vmdk bs=512 skip=10687828 count=4 2> /dev/null | hexdump -C
00000000  01 00 c0 10 00 a8 ea 0c  00 00 00 00 00 f0 3d 00  |..............=.|
00000010  00 00 00 00 11 00 00 00  00 00 00 00 2b 00 00 00  |............+...|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000030  00 00 00 00 00 00 00 00  d9 00 00 00 00 00 00 00  |................|
00000040  00 00 00 00 00 00 3e 00  00 00 00 00 1f 00 00 00  |......>.........|
00000050  00 00 00 00 ff e6 9a 53  cf 54 c0 67 9d 62 00 0c  |.......S.T.g.b..|
00000060  29 d5 43 a8 00 00 00 00  00 00 00 00 00 00 00 00  |).C.............|
00000070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200  04 80 40 05 19 00 00 00  00 00 00 00 03 00 00 00  |..@.............|
00000210  00 00 00 00 00 00 00 00  00 00 00 00 00 20 00 00  |............. ..|
00000220  00 00 00 00 00 00 00 00  00 00 00 00 a9 f6 9a 53  |...............S|
00000230  8e f5 9a 53 a9 f6 9a 53  00 00 00 00 00 00 00 00  |...S...S........|
00000240  80 01 00 00 d1 10 00 00  00 00 00 00 00 00 00 00  |................|
00000250  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000400  23 20 44 69 73 6b 20 44  65 73 63 72 69 70 74 6f  |# Disk Descripto|
00000410  72 46 69 6c 65 0a 76 65  72 73 69 6f 6e 3d 31 0a  |rFile.version=1.|
00000420  65 6e 63 6f 64 69 6e 67  3d 22 55 54 46 2d 38 22  |encoding="UTF-8"|
00000430  0a 43 49 44 3d 35 62 61  64 65 30 63 31 0a 70 61  |.CID=5bade0c1.pa|
00000440  72 65 6e 74 43 49 44 3d  66 66 66 66 66 66 66 66  |rentCID=ffffffff|
00000450  0a 69 73 4e 61 74 69 76  65 53 6e 61 70 73 68 6f  |.isNativeSnapsho|
00000460  74 3d 22 6e 6f 22 0a 63  72 65 61 74 65 54 79 70  |t="no".createTyp|
00000470  65 3d 22 76 6d 66 73 22  0a 0a 23 20 45 78 74 65  |e="vmfs"..# Exte|
00000480  6e 74 20 64 65 73 63 72  69 70 74 69 6f 6e 0a 52  |nt description.R|
00000490  57 20 32 30 39 37 31 35  32 30 20 56 4d 46 53 20  |W 20971520 VMFS |
000004a0  22 74 65 73 74 30 31 2d  66 6c 61 74 2e 76 6d 64  |"test01-flat.vmd|
000004b0  6b 22 0a 0a 23 20 54 68  65 20 44 69 73 6b 20 44  |k"..# The Disk D|
000004c0  61 74 61 20 42 61 73 65  20 0a 23 44 44 42 0a 0a  |ata Base .#DDB..|
000004d0  64 64 62 2e 61 64 61 70  74 65 72 54 79 70 65 20  |ddb.adapterType |
000004e0  3d 20 22 6c 73 69 6c 6f  67 69 63 22 0a 64 64 62  |= "lsilogic".ddb|
000004f0  2e 74 68 69 6e 50 72 6f  76 69 73 69 6f 6e 65 64  |.thinProvisioned|
00000500  20 3d 20 22 31 22 0a 64  64 62 2e 67 65 6f 6d 65  | = "1".ddb.geome|
00000510  74 72 79 2e 73 65 63 74  6f 72 73 20 3d 20 22 36  |try.sectors = "6|
00000520  33 22 0a 64 64 62 2e 67  65 6f 6d 65 74 72 79 2e  |3".ddb.geometry.|
00000530  68 65 61 64 73 20 3d 20  22 32 35 35 22 0a 64 64  |heads = "255".dd|
00000540  62 2e 67 65 6f 6d 65 74  72 79 2e 63 79 6c 69 6e  |b.geometry.cylin|
00000550  64 65 72 73 20 3d 20 22  31 33 30 35 22 0a 64 64  |ders = "1305".dd|
00000560  62 2e 75 75 69 64 20 3d  20 22 36 30 20 30 30 20  |b.uuid = "60 00 |
00000570  43 32 20 39 64 20 33 35  20 63 37 20 30 39 20 63  |C2 9d 35 c7 09 c|
00000580  66 2d 63 66 20 64 34 20  30 65 20 61 33 20 32 37  |f-cf d4 0e a3 27|
00000590  20 36 35 20 39 38 20 62  62 22 0a 64 64 62 2e 6c  | 65 98 bb".ddb.l|
000005a0  6f 6e 67 43 6f 6e 74 65  6e 74 49 44 20 3d 20 22  |ongContentID = "|
000005b0  37 64 35 30 66 36 36 32  33 30 37 32 30 38 37 61  |7d50f6623072087a|
000005c0  37 31 64 36 61 63 63 39  35 62 61 64 65 30 63 31  |71d6acc95bade0c1|
000005d0  22 0a 64 64 62 2e 76 69  72 74 75 61 6c 48 57 56  |".ddb.virtualHWV|
000005e0  65 72 73 69 6f 6e 20 3d  20 22 38 22 0a 00 00 00  |ersion = "8"....|
000005f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000800

Copy and investigate vmdk of guest ESXi after deletion The magic number of the inode can be found, so although the size is 0 bytes It seems that small files can be recovered by searching from one end.

`Search for inodes after removal from disk`


# python searchinode.py
Magic Number found. This is maybe inode !!
id = 4, id2 = 1, filesize = 1.1kB
position of above data: 5259788288 - 5259788800
Magic Number found. This is maybe inode !!
id = 4194308, id2 = 1, filesize = 256.0kB
position of above data: 5259790336 - 5259790848
Magic Number found. This is maybe inode !!
id = 8388612, id2 = 1, filesize = 254.7MB
position of above data: 5259792384 - 5259792896
Magic Number found. This is maybe inode !!
id = 12582916, id2 = 1, filesize = 256.0MB
position of above data: 5259794432 - 5259794944
Magic Number found. This is maybe inode !!
id = 16777220, id2 = 1, filesize = 250.6MB
position of above data: 5259796480 - 5259796992
Magic Number found. This is maybe inode !!
id = 20971524, id2 = 1, filesize = 4.0MB
position of above data: 5259798528 - 5259799040
Magic Number found. This is maybe inode !!
id = 25165828, id2 = 1, filesize = 1.1MB
position of above data: 5259800576 - 5259801088
Magic Number found. This is maybe inode !!
id = 32772, id2 = 1, filesize = 0B
position of above data: 5472124928 - 5472125440
Magic Number found. This is maybe inode !!
id = 4227076, id2 = 2, filesize = 0B
position of above data: 5472126976 - 5472127488
Magic Number found. This is maybe inode !!
id = 8421380, id2 = 3, filesize = 0B
position of above data: 5472129024 - 5472129536
Magic Number found. This is maybe inode !!
id = 12615684, id2 = 4, filesize = 0B
position of above data: 5472131072 - 5472131584
Magic Number found. This is maybe inode !!
id = 16809988, id2 = 5, filesize = 0B
position of above data: 5472133120 - 5472133632
Magic Number found. This is maybe inode !!
id = 21004292, id2 = 6, filesize = 0B
position of above data: 5472135168 - 5472135680
Magic Number found. This is maybe inode !!
id = 25198596, id2 = 7, filesize = 0B
position of above data: 5472137216 - 5472137728
Magic Number found. This is maybe inode !!
id = 29392900, id2 = 8, filesize = 0B
position of above data: 5472139264 - 5472139776
Magic Number found. This is maybe inode !!
id = 33587204, id2 = 9, filesize = 0B
position of above data: 5472141312 - 5472141824
Magic Number found. This is maybe inode !!
id = 37781508, id2 = 10, filesize = 0B
position of above data: 5472143360 - 5472143872
Magic Number found. This is maybe inode !!
id = 41975812, id2 = 11, filesize = 0B
position of above data: 5472145408 - 5472145920
Magic Number found. This is maybe inode !!
id = 46170116, id2 = 12, filesize = 0B
position of above data: 5472147456 - 5472147968
Magic Number found. This is maybe inode !!
id = 50364420, id2 = 14, filesize = 0B
position of above data: 5472149504 - 5472150016
Magic Number found. This is maybe inode !!
id = 54558724, id2 = 15, filesize = 0B
position of above data: 5472151552 - 5472152064
Magic Number found. This is maybe inode !!
id = 58753028, id2 = 17, filesize = 0B
position of above data: 5472153600 - 5472154112
Magic Number found. This is maybe inode !!
id = 62947332, id2 = 18, filesize = 0B
position of above data: 5472155648 - 5472156160
Magic Number found. This is maybe inode !!
id = 67141636, id2 = 20, filesize = 0B
position of above data: 5472157696 - 5472158208
Magic Number found. This is maybe inode !!
id = 71335940, id2 = 21, filesize = 0B
position of above data: 5472159744 - 5472160256
Magic Number found. This is maybe inode !!
id = 75530244, id2 = 22, filesize = 0B
position of above data: 5472161792 - 5472162304
Magic Number found. This is maybe inode !!
id = 79724548, id2 = 23, filesize = 0B
position of above data: 5472163840 - 5472164352
Magic Number found. This is maybe inode !!
id = 83918852, id2 = 24, filesize = 0B
position of above data: 5472165888 - 5472166400
Magic Number found. This is maybe inode !!
id = 88113156, id2 = 25, filesize = 0B
position of above data: 5472167936 - 5472168448
Magic Number found. This is maybe inode !!
id = 92307460, id2 = 26, filesize = 0B
position of above data: 5472169984 - 5472170496
Magic Number found. This is maybe inode !!
id = 96501764, id2 = 27, filesize = 0B
position of above data: 5472172032 - 5472172544
Magic Number found. This is maybe inode !!
id = 100696068, id2 = 28, filesize = 0B
position of above data: 5472174080 - 5472174592
Magic Number found. This is maybe inode !!
id = 104890372, id2 = 29, filesize = 0B
position of above data: 5472176128 - 5472176640
Magic Number found. This is maybe inode !!
id = 109084676, id2 = 30, filesize = 0B
position of above data: 5472178176 - 5472178688
Magic Number found. This is maybe inode !!
id = 113278980, id2 = 31, filesize = 0B
position of above data: 5472180224 - 5472180736
Magic Number found. This is maybe inode !!
id = 117473284, id2 = 33, filesize = 0B
position of above data: 5472182272 - 5472182784
Magic Number found. This is maybe inode !!
id = 121667588, id2 = 34, filesize = 0B
position of above data: 5472184320 - 5472184832
Magic Number found. This is maybe inode !!
id = 125861892, id2 = 35, filesize = 0B
position of above data: 5472186368 - 5472186880
Magic Number found. This is maybe inode !!
id = 130056196, id2 = 36, filesize = 0B
position of above data: 5472188416 - 5472188928
Magic Number found. This is maybe inode !!
id = 134250500, id2 = 37, filesize = 0B
position of above data: 5472190464 - 5472190976
Magic Number found. This is maybe inode !!
id = 138444804, id2 = 38, filesize = 0B
position of above data: 5472192512 - 5472193024
Magic Number found. This is maybe inode !!
id = 142639108, id2 = 39, filesize = 0B
position of above data: 5472194560 - 5472195072
Magic Number found. This is maybe inode !!
id = 146833412, id2 = 40, filesize = 0B
position of above data: 5472196608 - 5472197120
Magic Number found. This is maybe inode !!
id = 151027716, id2 = 41, filesize = 0B
position of above data: 5472198656 - 5472199168
Magic Number found. This is maybe inode !!
id = 155222020, id2 = 42, filesize = 0B
position of above data: 5472200704 - 5472201216
Magic Number found. This is maybe inode !!
id = 159416324, id2 = 43, filesize = 0B
position of above data: 5472202752 - 5472203264
Magic Number found. This is maybe inode !!
id = 163610628, id2 = 44, filesize = 0B
position of above data: 5472204800 - 5472205312
Magic Number found. This is maybe inode !!
id = 167804932, id2 = 45, filesize = 0B
position of above data: 5472206848 - 5472207360
Magic Number found. This is maybe inode !!
id = 171999236, id2 = 46, filesize = 0B
position of above data: 5472208896 - 5472209408
Magic Number found. This is maybe inode !!
id = 176193540, id2 = 47, filesize = 0B
position of above data: 5472210944 - 5472211456
Magic Number found. This is maybe inode !!
id = 180387844, id2 = 49, filesize = 0B
position of above data: 5472212992 - 5472213504
Magic Number found. This is maybe inode !!
id = 184582148, id2 = 50, filesize = 0B
position of above data: 5472215040 - 5472215552
Magic Number found. This is maybe inode !!
id = 188776452, id2 = 51, filesize = 0B
position of above data: 5472217088 - 5472217600
Magic Number found. This is maybe inode !!
id = 192970756, id2 = 52, filesize = 0B
position of above data: 5472219136 - 5472219648
Magic Number found. This is maybe inode !!
id = 197165060, id2 = 53, filesize = 0B
position of above data: 5472221184 - 5472221696
Magic Number found. This is maybe inode !!
id = 201359364, id2 = 54, filesize = 0B
position of above data: 5472223232 - 5472223744
^CTraceback (most recent call last):
  File "searchinode.py", line 64, in <module>
    data = f.read(512)
KeyboardInterrupt

Summary

The data itself is probably not overwritten (when deleted)
Inode information is partially erased
You can retrieve files that are small in size (1kB or less?) And that fit the data in the inode.
(About .vmxf or .vmdk for virtual disk descriptor file)
The VMDK file for <vmname> -flat.vmdk is almost impossible because the information of the corresponding block of inode is deleted.
If it is about MBR, it can be found and recovered directly regardless of the inode.
Make a backup of your important data

Reference material

VMFS related

http://glandium.org/projects/vmfs-tools/
http://buildvirtual.net/using-vmkfstools-to-manage-vmfs-datastores/

MBR related

http://ja.wikipedia.org/wiki/ Master boot record
http://thestarman.narod.ru/asm/mbr/index.html

GPT related

http://ja.wikipedia.org/wiki/GUID Partition Table

Exploring the possibility of recovering deleted VMDK files on VMFS

About deleting a VM

Survey flow

Survey started

searchmbr.py

MBR search

VMFS exploration

GPT survey

vmfs_fs.h

searchvmfsinfo.py

vmfsinfo search

Check label data

vmfs_inode.h

vmfs_metadata.h

vmfs_inode.h

searchinode.py

inode search

inode display

Deleted from disk

Inode information before deletion

Inode information after deletion

Check the small file inode from the deleted vmdk

Search for inodes after removal from disk

Summary

Reference material

`searchmbr.py`

`MBR search`

`GPT survey`

`vmfs_fs.h`

`searchvmfsinfo.py`

`vmfsinfo search`

`Check label data`

`vmfs_inode.h`

`vmfs_metadata.h`

`vmfs_inode.h`

`searchinode.py`

`inode search`

`inode display`

`Deleted from disk`

`Inode information before deletion`

`Inode information after deletion`

`Check the small file inode from the deleted vmdk`

`Search for inodes after removal from disk`