Tuesday, 19 September 2017

Deep Learning HOWTO with IBM Minsky Power8 server, Ubuntu 16.04, Caffe-NV and Kitti (Part 2)

Minsky server has some differences when you compare with a regular one. By large the most time consuming one (if you do not know it) is the BMC port and how to connect to it.
Here is the explanation.

Interfaces in Minsky

IBM Minsky server (model 8335-GTB) has 3 NICs (Ethernet interfaces), however it only has two physical ports.

Watching the back of the server you will find both Ports (Port0 and Port1)



Port 1 is a common port that can be configured as a regular NIC in the OS.
Port 0 is a physical port that has two internal NICs, it shows 2 different MAC addresses to the network.
  1. The first MAC address is from a NIC that can be used in the OS as another regular eth adapters.
  2. The second MAC is from the BMC interface. It is configured as DHCP by default.

Implementing the connection

You can use whatever solution you prefer to connect, just checking the interface (see the first commands of our solution) for the acquired IP address but as we do not have DHCP server we will configure a static IP address in the same range as the NIC1, forgetting about the NIC0 interface.

From Ubuntu CLI

  1. Install the necessary packages in the host OS.

root@minsky:/etc/network# apt install ipmitool

  1. Then we have to install the ipmi driver

root@minsky:/etc/network# modprobe ipmi_devintf

  1. Now we will check the actual configuration
root@minsky:/etc/network# ipmitool lan print 1
Set in Progress         : Set Complete
Auth Type Support       : MD5 
Auth Type Enable        : Callback : MD5 
                        : User     : MD5 
                        : Operator : MD5 
                        : Admin    : MD5 
                        : OEM      : MD5 
IP Address Source       : DHCP Address
IP Address              : 0.0.0.0
Subnet Mask             : 0.0.0.0
MAC Address             : 70:e2:84:14:25:fd
SNMP Community String   : AMI
IP Header               : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
BMC ARP Control         : ARP Responses Enabled, Gratuitous ARP Disabled
Gratituous ARP Intrvl   : 0.0 seconds
Default Gateway IP      : 0.0.0.0
Default Gateway MAC     : 00:00:00:00:00:00
Backup Gateway IP       : 0.0.0.0
Backup Gateway MAC      : 00:00:00:00:00:00
802.1q VLAN ID          : Disabled
802.1q VLAN Priority    : 0
RMCP+ Cipher Suites     : 0,1,2,3,6,7,8,11,12,15,16,17
Cipher Suite Priv Max   : caaaaaaaaaaaXXX
                        :     X=Cipher Suite Unused
                        :     c=CALLBACK
                        :     u=USER
                        :     o=OPERATOR
                        :     a=ADMIN
                        :     O=OEM
Bad Password Threshold  : 0
Invalid password disable: no
Attempt Count Reset Int.: 0
User Lockout Interval   : 0
root@minsky:/etc/network# 


  1. Check the current IP address configuration
root@minsky:/etc/network# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enP9p7s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 70:e2:84:14:25:fe brd ff:ff:ff:ff:ff:ff
    inet6 fe80::72e2:84ff:fe14:25fe/64 scope link 
       valid_lft forever preferred_lft forever
3: enP9p7s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 70:e2:84:14:25:ff brd ff:ff:ff:ff:ff:ff
    inet 10.250.45.2/24 brd 10.250.45.255 scope global enP9p7s0f1
       valid_lft forever preferred_lft forever
    inet6 fe80::72e2:84ff:fe14:25ff/64 scope link 
       valid_lft forever preferred_lft forever

  1. Configuration of the BMC interface with a static IP
root@minsky:/etc/network# ipmitool lan set 1 ipsrc static
root@minsky:/etc/network# ipmitool lan set 1 ipaddr 10.250.45.3
Setting LAN IP Address to 10.250.45.3
root@minsky:/etc/network# ipmitool lan set 1 netmask 255.255.255.0
Setting LAN Subnet Mask to 255.255.255.0
root@minsky:/etc/network# ipmitool lan set 1 defgw ipaddr 10.250.45.1
Setting LAN Default Gateway IP to 10.250.45.1
root@minsky:/etc/network# ipmitool lan set 1 arp respond on
Enabling BMC-generated ARP responses
root@minsky:/etc/network# ipmitool lan set 1 auth ADMIN MD5
root@minsky:/etc/network# ipmitool lan set 1 access on
Set Channel Access for channel 1 was successful.
root@minsky:/etc/network# 

  1. Check the outcome
root@minsky:~# ipmitool lan print 1
Set in Progress         : Set Complete
Auth Type Support       : MD5 
Auth Type Enable        : Callback : MD5 
                        : User     : MD5 
                        : Operator : MD5 
                        : Admin    : MD5 
                        : OEM      : MD5 
IP Address Source       : Static Address
IP Address              : 10.250.45.3
Subnet Mask             : 255.255.255.0
MAC Address             : 70:e2:84:14:25:fd
SNMP Community String   : AMI
IP Header               : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
BMC ARP Control         : ARP Responses Enabled, Gratuitous ARP Disabled
Gratituous ARP Intrvl   : 0.0 seconds
Default Gateway IP      : 10.250.45.1
Default Gateway MAC     : 02:e0:52:8c:1f:01
Backup Gateway IP       : 0.0.0.0
Backup Gateway MAC      : 00:00:00:00:00:00
802.1q VLAN ID          : Disabled
802.1q VLAN Priority    : 0
RMCP+ Cipher Suites     : 0,1,2,3,6,7,8,11,12,15,16,17
Cipher Suite Priv Max   : caaaaaaaaaaaXXX
                        :     X=Cipher Suite Unused
                        :     c=CALLBACK
                        :     u=USER
                        :     o=OPERATOR
                        :     a=ADMIN
                        :     O=OEM
Bad Password Threshold  : 0
Invalid password disable: no
Attempt Count Reset Int.: 0
User Lockout Interval   : 0

  1. Now we can connect using a web browser


Credentials: username ADMIN password admin

You are in!!!




Deep Learning HOWTO with IBM Minsky Power8 server, Ubuntu 16.04, Caffe-NV and Kitti (Part 1)

Who should be reading this

This document is suited to anyone who wants to start testing deep learning in a supercomputer scale with an IBM Minsky server.
It will help installing all the latest and greatest components of this solution and then will help to run an example of deep learning as KITTI test is.

Justification of the document

You can follow IBM official doc (located here →https://www-01.ibm.com/common/ssi/cgi-bin/ssialias?htmlfid=POO03514WWEN ) to install the components necessary run this test.

However if you want the latest and greatest versions of all SW components you will find here a quick guide to install all of them with examples included

Ubuntu 16.04.02 OS installation


# Download Ubuntu 16.04.02 LTS from this URL (check if there is something newer)

# Burn the ISO into a USB drive and boot the system.

# Select the 4.4 kernel as in the photo (be aware that you may be presented with the option of installing HWE (shipped with kernel version 4.8. This is not supported at the time of writing this doc).

# Install only ssh and base packages.

# After first boot check the output is kernel 4.4 (not 4.8)

root@minsky:~# uname -a
Linux minsky 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18 14:09:19 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux
root@minsky:~#


# Keep the IP address/es to access later via ssh.

root@minsky:/sw/dw# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enP9p7s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 70:e2:84:14:25:fe brd ff:ff:ff:ff:ff:ff
    inet 9.6.112.113/25 brd 9.6.112.127 scope global enP9p7s0f0
       valid_lft forever preferred_lft forever
    inet6 fe80::72e2:84ff:fe14:25fe/64 scope link 
       valid_lft forever preferred_lft forever
3: enP9p7s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 70:e2:84:14:25:ff brd ff:ff:ff:ff:ff:ff
    inet 9.6.112.89/25 brd 9.6.112.127 scope global enP9p7s0f1
       valid_lft forever preferred_lft forever
    inet6 fe80::72e2:84ff:fe14:25ff/64 scope link 
       valid_lft forever preferred_lft forever
root@minsky:/sw/dw# 



Friday, 16 June 2017

Small Ethereum mining rig (44ish Mh/s)

44ish Mh/s, still running without firmware optimizations though.
Can donate here if you want :-) --> Link

Tuesday, 16 May 2017

Deep Learning using 2 Power8 CPUs (16 cores) @4Ghz and 2 Nvidia P100 GPUs (7186 cores)

Today we are breaking the time to complete a Kitti benchmark using 100 epochs with an IBM Minsky server.
Reference machine: Supermicro Intel based server with 2xP100 Nvidia cards. Test time: 300 minutes.
Our server: IBM Minsky Power8 based server with 2xP100 Nvidia cards. Predicted Test time: 131 minutes. (currently running) Final time 2h20mins (140 mins)

More details to come in next days...




Holidays are for the summer