Pawan Mishra
by Pawan Mishra
1 min read

Tags

This post is a quick tutorial on how you can easily manage & switch between different versions of libraries that you have configured in your machine with the help of symbolic links. I am currently working on Apache Spark based project & I have to download & configure not only spark but other libraries like hadoop, zeppelin & hive in my machine. These libraries are frequently released with major &minor updates.

SetUp


The idea here is to keep all the different versions of a given library in one place & create symbolic link against the version you want to use. Later, use that symbolic link in other places like .bash_profile or in custom scripts. In my machine, I have created a root level bin folder & inside that I have library specific directories. E.g. :

~ bin
~ bin/sparks/
~ bin/hdfs/
~ bin/zeppelins/
~ bin/hives/

Inside each of the library specific directory, I have the various version specific directory of that library. For e.g. inside ~/bin/sparks/, I have :

Dec 10  2016 spark-2.0.2-bin-hadoop2.7
Jan 29  2017 spark-2.0.0-bin-hadoop2.7
Feb 22  2017 spark-2.1.0-bin-hadoop2.7
Mar  2  2017 spark-1.6.1-bin-hadoop2.6
Sep  5 16:17 spark-2.2.0-bin-hadoop2.7

Next step is to create the symbolic link. I prefer to create symbolic links inside the ~/bin directory but outside any of the library specific directory. For e.g if I want to work with latest spark version, I will create symbolic link as :

➜  ln -snf ~/bin/sparks/spark-2.2.0-bin-hadoop2.7 ~/bin/spark

Once you have symbolic links in place, then you can easily switch it to point to another version e.g.

➜  ln -snf ~/bin/sparks/spark-2.0.2-bin-hadoop2.7 ~/bin/spark

Next, you can refer the symbolic link instead of actual library in places where you need the application reference. For e.g. I have the following _HOME_ variables configured in my .zshrc file :

export SPARK_HOME=/Users/pawan/bin/spark
export PATH=$PATH:$SPARK_HOME/bin

This is it. With the help of symbolic link,s you can not only manage different versions of libraries, you can also seamlessly navigate across different versions of a given library.