By: JD Durick
If you are responsible for collecting and analyzing digital evidence, you are already aware of how platform and desktop virtualization are changing the way organizations, corporations, and government agencies deliver, store, and manage digital content. Virtualization adds an enormous level of complexity to an already complex field. As forensic practitioners tasked with performing examinations on enterprise virtual environments, they are confronted with the challenge of understanding what VM-related files are of significant importance. Should they acquire the Virtual Machine Disk (VMDK) related files of the Virtual machine in question? What about the snapshot, memory, swap, configuration, metadata, and log files? Each one of these files are essential in running the virtual machine and could assist forensic examiners in understanding the Virtual machine’s function and potential compromise. Below is a step-by-step listing of a virtual machine’s life cycle detailing six major specific states:
Virtual Machine life cycle in VMware ESXi 4 update 1:
When a virtual machine is created, several important files are generated in the location specified by the user. These files include:
(.vmxf) – A configuration file for teaming features. Teaming is a feature used primarily with VMware workstation to allow administrators to logically group virtual machines for streamlined administration.
(.vmx) – The primary virtual machine configuration file. The .vmx file contains the bulk of the virtual machines settings, configuration, hardware support, and emulation features.
(.vmsd) – Virtual machine snapshot descriptor file. This empty file is created when you configure and generate a new virtual machine, and maintains information about the virtual machine’s snapshots.
(.vmdk) – The virtual machine disk descriptor file contains disk geometry, layout, structure, and physical properties. The disk descriptor will describe an extent that represents the physical storage used by the disk. The term “file extent” refers to how many different fragments or “data runs” there are for a file. At most, a logical VMFS-3 volume can have 32 physical extents.
(vmname-flat.vmdk) – This file is considered a monolithic flat disk that has been fully allocated with the start and end markers allocated. This type of disk could potentially contain old data that resided on the data store and was not properly cleared prior to allocation of the disk. This type of disk is the default standard when setting up a virtual machine via the vSphere client on ESXi 4.0 update 1.
Figure 1: Pre-creation snapshot and corresponding files of the Virtual Machine.
When the VM is started, several additional files are created including:
(.log) – VMware log files contain extensive diagnostic information, configuration and run-time messages of the virtual machine. Specifically, the vmware.log file holds detailed information about the virtual machine such as: network interfaces and corresponding IP addresses, hostname, ESXi kernel information, and detailed times in which snapshots were taken and restored.
(.nvram) – The virtual machines Basic Input Output System (BIOS) settings can be found in the non-volatile random access memory file.
(.vswp) – A virtual machine swap file. When the virtual machine requires more memory to be allocated, the swap file will be used instead of physical memory even if its RAM setting is not overcommitted. As soon as the VM is started, a swap file for the VM is created. The swap file is created with the following format: vmname-[8 character hexadecimal number].vswp. In our scenario, it was lifecycle_ubuntu-8bcd0f28.vswp and 256 MB in size.
Figure 2: Files created after installation of the Virtual Machine.
When a virtual machine is suspended, a suspended state file (.vmss) is created that represents the state of the machine at the time it was suspended, or paused; the .vswp file is then deleted. Again, the .vmss file contained the same eight character hexadecimal number as was applied to the .vswp file.
Figure 3: Files that were created after VM suspension occurred.
When a virtual machine is resumed from a suspended state, the .vmss file remains and the .vswp is regenerated.
Figure 4: Files changed after a VM resume was executed.
The previously empty .vmsd file is now populated with information about the new snapshot that was just created. A .vmsn file is generated, containing memory contents for the virtual machine, however, only if specified in the snapshot options menu of vSphere.
A snapshot descriptor file (.vmsd) and redo logs (.vmdk) are generated to represent changes made after generating the snapshot. The parent .vmdk disk descriptor and extents are untouched, meaning this operation will provide a forensically sound image file.
The vmname-######-delta.vmdk file stores changes made to a virtual disk while the virtual machine is running. According to VMware, there may be more than one such file.
Specifically, the .vmsn file contains “strings” of importance such as potential running process and hidden files that can be analyzed by the forensic examiner. Currently, tools exist to compare snapshots of the same virtual machine to track the changes of altered or suspect files. Additionally, Volatility has been known to be successful when analyzing and dissecting virtual machine memory files.
Figure 5: Changes to the files after a snapshot of the VM was created.
When a virtual machine is shutdown cleanly, the .vswp and .vmss files are deleted. During an investigation, a .vswp file may contain important data. However, after a clean shutdown it is deleted from the virtual machine directory.
Figure 6: Files that now exist after a clean shutdown are executed on the VM.
It is critical for forensic examiners to fully understand what information each file provides when conducting a forensic examination on virtual machines in a virtual enterprise environment. The above lifecycle has shown us the purpose and function of those VM files used during each of the six states. After the virtual machines corresponding files have been retrieved, numerous tools on the market today make it possible to mount, dissect, and examine their contents to glean valuable forensic evidence.