Phoenix For advanced users: Install software packages for your own usage

亦凉 2022-04-13 07:19 147阅读 0赞

We recommend all Phoenix users to [contact us][] at the first instance for application enquiries. However, we are aware the diverse research requirement from various disciplines across the university, we are here to recommend a mechanism for our advanced cluster users to allow you to bypass the Phoenix administration to install many open source packages for your own research purposes. We are appreciate if you [inform us][contact us] your required packages at the first instance to allow us to provide the same packages access to other researchers.

## Contents ##

\[[hide][]\]

*  [1Installing Applications on the cluster][]
    
     *  [1.1Loading EasyBuild][]
     *  [1.2Searching with EasyBuild][]
     *  [1.3Writing a configuration EasyBuild file][]
     *  [1.4Installation][]

## Installing Applications on the cluster\[[edit][] | [edit source][]\] ##

Phoenix cluster administrators use EasyBuild tool to install applications in a safe and reproducible fashion. You can also use this tool to install your required packages locally (in your home directory) for your own usage.

### Loading EasyBuild\[[edit][edit 1] | [edit source][edit source 1]\] ###

In order to use EasyBuild, you must load the module first:

> module load EasyBuild
    > eb --version
    This is EasyBuild 2.5.0 (framework: 2.5.0, easyblocks: 2.5.0) on host l01.phoenix.adelaide.edu.au

EasyBuild can easily install an application using a configure file for it. In many cases there is a configure file ready to use.

Now, it is time to install an application! e.g. We want to install Lua programming language. These are the steps to do so.

### Searching with EasyBuild\[[edit][edit 2] | [edit source][edit source 2]\] ###

Search for Lua configuration:

> eb -S Lua

The search result is:

== temporary log file in case of crash /tmp/eb-Tx0A\_g/easybuild-Up59LX.log == Searching (case-insensitive) for 'lua' in /apps/software/EasyBuild/2.5.0/lib/python2.7/site-packages/easybuild\_easyconfigs-2.5.0-py2.7.egg/easybuild/easyconfigs CFGS1=/apps/software/EasyBuild/2.5.0/lib/python2.7/site-packages/easybuild\_easyconfigs-2.5.0-py2.7.egg/easybuild/easyconfigs/l

* $CFGS1/Lua/Lua-5.1.4-5-GCC-4.7.2.eb
    * $CFGS1/Lua/Lua-5.1.4-5-GCC-4.8.2.eb
    * $CFGS1/Lua/Lua-5.1.4-5-goolf-1.4.10.eb
    * $CFGS1/Lua/Lua-5.1.4-8-GCC-4.8.2.eb
    * $CFGS1/Lua/Lua-5.1.4-8-GCC-4.8.4.eb
    * $CFGS1/LuaJIT/LuaJIT-2.0.2-GCC-4.9.2.eb
    * $CFGS1/likwid/likwid-4.0.1-config-and-lua.patch

== Temporary log file(s) /tmp/eb-Tx0A\_g/easybuild-Up59LX.log\* have been removed. == Temporary directory /tmp/eb-Tx0A\_g has been removed.

We have found 5 different versions of configuration files to install Lua. We can realise that there are two versions of lua: 5.1.4-5 and 5.1.4-8, and different toolchains.

To check any further dependency, it is a good practice first see the dependencies before installing. The argument -Dx does that:

> eb Lua-5.1.4-8-GCC-4.8.4.eb -Dx

== temporary log file in case of crash /tmp/eb-qlHVTV/easybuild-UivGqc.log
    Dry run: printing build status of easyconfigs and dependencies
    CFGS=/apps/software/EasyBuild/2.5.0/lib/python2.7/site-packages/easybuild_easyconfigs-2.5.0-py2.7.egg/easybuild/easyconfigs
     * [ ] $CFGS/g/GCC/GCC-4.8.4.eb (module: GCC/4.8.4)
     * [ ] $CFGS/n/ncurses/ncurses-5.9-GCC-4.8.4.eb (module: ncurses/5.9-GCC-4.8.4)
     * [ ] $CFGS/l/Lua/Lua-5.1.4-8-GCC-4.8.4.eb (module: Lua/5.1.4-8-GCC-4.8.4)
    == Temporary log file(s) /tmp/eb-qlHVTV/easybuild-UivGqc.log* have been removed.
    == Temporary directory /tmp/eb-qlHVTV has been removed.
    > eb Lua-5.1.4-8-GCC-4.8.4.eb -Dx

The result shows that for installing lua using GCC-4.8.4 we need to install 2 additional applications. We need the GCC version 4.8.4 and ncurses library version 5.9. We can search what we actually have:

> module avail GCC
    GCC/4.9.3-binutils-2.25 GCC/5.1.0-binutils-2.25
    > module avail ncurses
    ncurses/5.9-foss-2015b ncurses/5.9-intel-2015c

First of all, ncurses has been built by foss-2015b toolchain, which is the defacto toolchain on Phoenix. Therefore, we need to write a customised configuration to use foss/2015b as toolchain, but not from scratch. We can modify the found configuration:

### Writing a configuration EasyBuild file\[[edit][edit 3] | [edit source][edit source 3]\] ###

EasyBuild comes with many configuration files. They are located at: /apps/software/EasyBuild/2.5.0/lib/python2.7/site-packages/easybuild\_easyconfigs-2.5.0-py2.7.egg/easybuild/easyconfigs/. We can define a variable to help us:

> export EASYCONFIGHOME=/apps/software/EasyBuild/2.5.0/lib/python2.7/site-packages/easybuild_easyconfigs-2.5.0-py2.7.egg/easybuild/easyconfigs/

> cp $EASYCONFIGHOME/l/Lua/Lua-5.1.4-8-GCC-4.8.4.eb Lua-5.1.4-8-foss-2015b.eb

Now we have a configuration file for Lua and foss toolchain, but we need to change its content. The copied file looks like:

easyblock = 'ConfigureMake'
    
    name = "Lua"
    version = "5.1.4-8"
    
    homepage = "http://www.lua.org/"
    description = """Lua is a powerful, fast, lightweight, embeddable scripting language.
     Lua combines simple procedural syntax with powerful data description constructs based
     on associative arrays and extensible semantics. Lua is dynamically typed,
     runs by interpreting bytecode for a register-based virtual machine,
     and has automatic memory management with incremental garbage collection,
     making it ideal for configuration, scripting, and rapid prototyping."""
    
    toolchain = {'name': 'GCC', 'version': '4.8.4'}
    
    sources = ['lua-%s.tar.gz' % version.replace('-', '.')]
    source_urls = ['http://sourceforge.net/projects/lmod/files/']
    
    dependencies = [('ncurses', '5.9')]
    
    sanity_check_paths = {
     'files': ["bin/lua"],
     'dirs': []
    }
    
    moduleclass = "lang"

After we have modified toolchain, the final version should look like:

easyblock = 'ConfigureMake'
    
    name = "Lua"
    version = "5.1.4-8"
    
    homepage = "http://www.lua.org/"
    description = """Lua is a powerful, fast, lightweight, embeddable scripting language.
     Lua combines simple procedural syntax with powerful data description constructs based
     on associative arrays and extensible semantics. Lua is dynamically typed,
     runs by interpreting bytecode for a register-based virtual machine,
     and has automatic memory management with incremental garbage collection,
     making it ideal for configuration, scripting, and rapid prototyping."""
    
    toolchain = {'name': 'foss', 'version': '2015b'}
    
    sources = ['lua-%s.tar.gz' % version.replace('-', '.')]
    source_urls = ['http://sourceforge.net/projects/lmod/files/']
    
    dependencies = [('ncurses', '5.9')]
    
    sanity_check_paths = {
     'files': ["bin/lua"],
     'dirs': []
    }
    
    moduleclass = "lang"

### Installation\[[edit][edit 4] | [edit source][edit source 4]\] ###

Now we can install the Lua in our personal home directory:

eb Lua-5.1.4-8-foss-2015b.eb

The installation output indicates what is going on...

== temporary log file in case of crash /tmp/eb-2LqRWW/easybuild-vCljJ5.log == processing EasyBuild easyconfig /home/a1634120/eb\_tutorial/Lua-5.1.4-8-foss-2015b.eb == building and installing Lua/5.1.4-8-foss-2015b... == fetching files... == creating build dir, resetting environment... == unpacking... == patching... == preparing... == configuring... == building... == testing... == installing... == taking care of extensions... == postprocessing... == sanity checking... == cleaning up... == creating module... == permissions... == packaging... == COMPLETED: Installation ended successfully == Results of the build can be found in the log file /home/a1634120/.local/easybuild/software/Lua/5.1.4-8-foss-2015b/easybuild/easybuild-Lua-5.1.4-8-20160303.110938.log == Build succeeded for 1 out of 1 == Temporary log file(s) /tmp/eb-2LqRWW/easybuild-vCljJ5.log\* have been removed. == Temporary directory /tmp/eb-2LqRWW has been removed.

EasyBuild download the source files, load the toolchain, configure, build and install the application. Easy! The local installation directory is ~/.local/easybuild/

To use locally installed packages

module use ~/.local/easybuild/modules/all
    module load Lua

[contact us]: mailto:hpcsupport@adelaide.edu.au
[hide]: https://wiki.adelaide.edu.au/hpc/For_advanced_users:_Install_software_packages_for_your_own_usage#
[1Installing Applications on the cluster]: https://wiki.adelaide.edu.au/hpc/For_advanced_users:_Install_software_packages_for_your_own_usage#Installing_Applications_on_the_cluster
[1.1Loading EasyBuild]: https://wiki.adelaide.edu.au/hpc/For_advanced_users:_Install_software_packages_for_your_own_usage#Loading_EasyBuild
[1.2Searching with EasyBuild]: https://wiki.adelaide.edu.au/hpc/For_advanced_users:_Install_software_packages_for_your_own_usage#Searching_with_EasyBuild
[1.3Writing a configuration EasyBuild file]: https://wiki.adelaide.edu.au/hpc/For_advanced_users:_Install_software_packages_for_your_own_usage#Writing_a_configuration_EasyBuild_file
[1.4Installation]: https://wiki.adelaide.edu.au/hpc/For_advanced_users:_Install_software_packages_for_your_own_usage#Installation
[edit]: https://wiki.adelaide.edu.au/hpc/index.php?title=For_advanced_users:_Install_software_packages_for_your_own_usage&veaction=edit&vesection=1
[edit source]: https://wiki.adelaide.edu.au/hpc/index.php?title=For_advanced_users:_Install_software_packages_for_your_own_usage&action=edit&section=1
[edit 1]: https://wiki.adelaide.edu.au/hpc/index.php?title=For_advanced_users:_Install_software_packages_for_your_own_usage&veaction=edit&vesection=2
[edit source 1]: https://wiki.adelaide.edu.au/hpc/index.php?title=For_advanced_users:_Install_software_packages_for_your_own_usage&action=edit&section=2
[edit 2]: https://wiki.adelaide.edu.au/hpc/index.php?title=For_advanced_users:_Install_software_packages_for_your_own_usage&veaction=edit&vesection=3
[edit source 2]: https://wiki.adelaide.edu.au/hpc/index.php?title=For_advanced_users:_Install_software_packages_for_your_own_usage&action=edit&section=3
[edit 3]: https://wiki.adelaide.edu.au/hpc/index.php?title=For_advanced_users:_Install_software_packages_for_your_own_usage&veaction=edit&vesection=4
[edit source 3]: https://wiki.adelaide.edu.au/hpc/index.php?title=For_advanced_users:_Install_software_packages_for_your_own_usage&action=edit&section=4
[edit 4]: https://wiki.adelaide.edu.au/hpc/index.php?title=For_advanced_users:_Install_software_packages_for_your_own_usage&veaction=edit&vesection=5
[edit source 4]: https://wiki.adelaide.edu.au/hpc/index.php?title=For_advanced_users:_Install_software_packages_for_your_own_usage&action=edit&section=5