Building a high performance GPU computing workstation for deep learning – part II
I decided to configure the system as a dual boot, to maximize its forward utility. Win 10 Pro & Ubuntu 16.04LTS. Main packages installed will be: Python 3.5, R 3.4.1, R Studio, and Pycharm & Jupyter.
Steps we will take are as follows:
- Update ASUS Motherboard BIOS to most current UEFI version.
- Install Windows 10 Pro via USB to NVMe Drive.
- Partition (shrink) NVMe Drive in Windows to free up space for Ubuntu
- Format & Partition 8Tb data drive
- Create a bootable USB drive with Ubuntu 16.04LTS
- Turn off Secure Boot in UEFI BIOS
- Install Ubuntu 16.04LTS via USB drive to NVMe free partition
- Install Nvidia 1080Ti Drivers in Ubuntu
- Test both windows and Ubuntu for proper functio
- Install CUDA and cuDNN in Ubuntu
- Install Tensorflow, Keras, pytorch, and jupyter in Ubuntu
- Run MNIST, Deep Dream, and small VGG network to confirm proper operation of GPU
- Modify Grub Boot Loader (optional) & configure system
Of course, this ‘simple’ list is not as simple as it seems, mostly due to the fact that Ubuntu and Nvidia drivers don’t play well, and Nvidia’s debian package for CUDA throws you a curve ball with the 1080Ti.
If you are not very familiar with Linux/Ubuntu I would recommend having a 2nd computer connected to internet (with this blog post synced up, perhaps?) while you are configuring the new build. I would also recommend a reference book – I used Matthew Helmke’s Ubuntu unleased 2017 version by SAMS.
IMPORTANT:
Make sure you have a WIRED keyboard and a WIRED mouse and WIRED monitor connected for this process. Wireless devices may not work during the install! (Mine didn’t).
ASUS Motherboard BIOS update:
While a little bit nerve wracking, since you can theoretically brick your Mobo, it was really not that difficult. ASUS lets you do it at least 3 different ways. I updated by downloading the latest drivers from ASUS’s website, renamed the file as instructed, transferred it to an otherwise empty USB 2.0 drive and then plugged it into the USB slot outlined in blue-green on the backplate. There is also a program that will update it for you on the ASUS website. Probably not a good idea to do during a thunderstorm!
Installing Windows 10:
Plug in the USB, reset your computer, and enter the setup menu in BIOS/UEFI (hit F2, select USB to boot first in boot priority and hit save/exit F10). Windows installs fairly straightforward; select 64 bit Pro version and enter product key.
For guidance on how to install Win 10, I thought this was reasonable.
To tweak your system, Black Viper’s Windows services website is the best.
After install, Turn off Auto Defragment in Windows. This is believed to harm your NVMe drive’s life.
Partition C:\ in Windows 10
You will need to run diskmgmt.msc from the windows command line prompt and shrink the volume of your C:\ Drive (the M2 NVMe SSD). Linux seems more compact than windows, so I set the amount of available disk space to 100gb in Ubuntu. After some consideration, I would recommend partitioning the 1Tb SSD 500Gb for Windows, 400Gb for Linux, and 100GB fallow (said to help with SSD drive life). I have yet to have problems with the 100GB limit, but I have lots of space left over in Windows!
A guide to shrinking the C:\ partition is here as part of a dual-boot install guide. You might want to open this guide up on your 2nd computer too.
Format and Partition 8TB HDD:
Using dismgmt.msc I formatted D:\ . 7 of the 8TB HDD as NTFS, which is usable both by Windows and Linux. I reserved 1Tb for Linux. If I recall correctly, this step took a while.
Create a bootable Ubuntu 16.04LTS USB Drive
Take an empty USB drive (mine was a 4GB). Download the rufus freeware program in windows. Plug in your USB flashdrive, run the rufus program and create a bootable iso flash drive with 64-bit Ubuntu 16.04LTS. More detailed instructions are here.
I strongly suggest you read ahead through the rest of the blogpost and have an ample amount of time ahead of you before your proceed. If not, this would be a good stopping point.
Restart the computer, enter UEFI Bios with F2 and set it to boot off the USB.
Disable UEFI secure boot in the BIOS firmware
There is an obscure setting in the UEFI BIOS which interferes with dual booting and installation. Go into UEFI (F2) upon bootup and go to advanced settings (F7). Click on advanced and scroll down to secure boot. Enter it and select key management & clear secure boot keys to disable it. I followed directions but wasn’t entirely sure about what I was doing so you might want to write down the keys. You can read about it here.
The Black Screen of Death
Now that you are ready to install Ubuntu, you need to know about the black screen of death and what you can do about it. Basically, there is a default video driver built into Ubuntu/Linux – Nouveaux. And it does not play well with Nvidia cards. Ergo, when you install Ubuntu, update the Nvidia video driver, or run an apt-get update & apt-get upgrade and an updated Nouveax driver loads, you have a conflict and you get zero video output – i.e. the black screen of death. You need to be prepared for this, and know how to fix it. There are a bunch of different possible solutions. This page lists most of them. I really suggest you read through it first and prepare to refer to it.
Nomodeset: The easiest fix is to add the word nomodeset to the starting list of parameters in the GRUB bootloader (do-able once you have installed Ubuntu already). Once you see the grub bootloader show up with its choices to boot into either Windows 10 or Ubuntu 16.04, select Ubuntu and hit the letter ‘e’. This will take you into a rudimentary text editor with a few lines of code. Look for the word quiet splash and add “nomodeset” after it. Separate nomodeset with a space before and after. It is OK if other terms/parameters come after nomodeset. Exit with F10 and Ubuntu should boot.
Terminal Mode: If things aren’t going the way you want & you want to get a terminal, hit <ctrl>-<alt>-<f1> (holding down all 3). This should give you a login prompt and then a password. You will then be in root and able to access a command prompt. When you are done doing what you need to do (i.e. installing the nvidia drivers), you can exit by <ctrl>-<alt>-<f7>. I found this very helpful during the process when I was spinning my wheels.
I really have gotten screwed up somehow: If you are like me, and have limited experience with Ubuntu, driver installs, and the like, and have struggled through thing s with a partial install, it is possible that you make some serious errors along the way. If this happens, and you can’t get the drivers working, here is a script which may help:
#remove prior nvidia drivers
sudo apt-get install linux-headers-$(uname -r)
sudo apt-get purge nvidia*
#now update the nvidia driver – stop conflicting processes first
sudo lightdm stop
sudo init 3
sudo sh ~/Desktop/Nvidia-linx-x86_64-384.59.run #driver .run file downloaded 1st
sudo modprobe nvidia
sudo start lightdm
nvidia -smi
Ok, now you have the resources to help you when this bombs out, which it likely will in your first install or runthrough. If you are ready, you can now exit BIOS with the Ubuntu USB drive in your computer and see if you will boot.
Followup Note: The challenges of setting up Ubuntu 16 with NVIDIA drivers in 2017 was a very different experience than what exists now. The working LTS in 2020 is 18.04 and 20.04 has recently been released. Both seem to “play nicer” with the NVIDIA driver installs than the prior 16 version. As such, you will likely not need to go through these gyrations. Its left here as a historical account of what you needed to do once upon a time to get stuff working!
