It's certainly possible to repurpose existing hardware for a virtualization endeavor, but it's generally a better bet to buy a new server designed for a role as a virtualization host. This server should have at least a single quad-core CPU (or two dual-core CPUs) and as much RAM as the budget will allow. If the file server will be virtualized, then of course the disks must be sized appropriately. Here, you'll want to make certain that the server has a RAID controller and that there are enough disks in the server to provide the requisite storage space when configured as a RAID5 array.
For our example, a Dell PowerEdge 2950 or HP DL380 G5 with a single quad-core CPU, 16GB of RAM, and four 72GB SAS drives in a RAID5 configuration would do the trick. This would provide for roughly 200GB of storage for all virtual servers and plenty of RAM. Additionally, these servers can be upgraded with another quad-core CPU in the future. At the time of this writing, both of these systems are included in the VMware ESXi hardware compatibility guide.
Building the box
The next step is to install the virtualization software. If it's VMware ESXi, then this is a free download that can be burned to a CD and installed simply by booting the CD and stepping through the installer. The management of VMware ESXi is handled via a Web interface or the VMware Infrastructure Client included with the package. Citrix's XenServer installs essentially the same way as VMware ESXi, and is also managed via an included client.
If VMware Server is your choice, then the underlying operating system must first be installed, whether that's Microsoft Windows or a Linux distribution.
There are several benefits to choosing Linux as the basis for a VMware Server installation. The installation footprint of the Linux distribution can be extremely small and still support VMware Server, and the performance of VMware Server is arguably better with a Linux foundation than with a Windows foundation. In addition, if you choose a community-supported Linux distribution such as CentOS, Fedora, or Ubuntu, you will not have to pay licensing fees as you would for Windows.
In either case, the VMware Server installation is generally very straightforward. On Windows, it's a matter of downloading the software from VMware and running the installer. On Linux, it's the same, though this is generally done from the command line and not via GUI.
Once VMware Server is installed, it can be managed from the Web UI or from the VMware Server Console application that is installed on a Linux or Windows workstation. Generally speaking, managing a VMware Server installation is simpler via the VMware Server Console application than via the Web interface.
If Hyper-V is the chosen solution, then it's simply a matter of installing Windows Server 2008 and choosing to install Hyper-V. The Hyper-V management console is built around commonly used Windows management frameworks, and anyone familiar with Windows servers should take to it quickly.
Making the move
After the virtualization layer is installed, your physical-to-virtual transition is simply a matter of building the virtual servers and migrating the services.
Although VMware, Microsoft, Citrix, and others offer "P2V" migration tools, you're frequently better off rebuilding these servers from scratch rather than migrating them from their physical hosts. This is especially true for domain controllers, which should always be fresh-built, and for any servers that have gathered the weight of a few years of operation. Taking the time to lay down fresh installs will reduce complications later on.