Python / Linux / VMware 3: Installing Python 3.6

I'm trying to expand my knowledge of Linux and Python. I'm taking a "start from scratch" approach, building some virtual environments at home, and documenting every step along the way. This is the third of five posts capturing those steps; the end-state is PyCharm IDE, Python 3.6, CentOS 7.3, VMware Workstation Pro 12 running on Windows 10.

Part 1: Installing VMware and CentOS
Part 2: Expanding CentOS's Minimal Install
This is Part 3: Installing Python 3.6
Part 4: Linking PyCharm to Github via SSH
Part 5: Running through Git-It again, this time with PyCharm

I'm going to install the most recent version of Python, which at the moment is 3.6.0. I want to use the latest version for two reasons:

• To align with the book that I'm using, which focuses on version 3
• To experience software installation that's a bit more involved than yum install

This means that I'm going to need to download the source and compile it.

It's important to know that Python 2.7 is already installed, since it's a default part of most Linux distributions. Because of this, if I just install Python 3 without changing some of the standard installation behavior, I'm going to create trouble for myself later; the default Python installation and version are used by some important Linux system tools, which I'd rather not break.

I'm also going to install an Integrated Development Environment (IDE) for Python, to make learning and coding a bit easier. I'm going to use the free community edition of PyCharm from JetBrains, because I'm already familiar and comfortable with some of their other products (like IntelliJ IDEA for Java).

Update Python 2.7

As I mentioned, the default install of Python 2.7 is used pretty extensively by much of the operating system itself. Because of that, I don't want to mess too much with the configuration or package versions of Python 2.7.

However, any Python IDE that I install is going to allow me to choose which version of Python I want to use for each of my development projects. If I ever do want to do any work in Python 2.7, I should at least have the ability to add libraries that may not currently be installed, or otherwise make minor tweaks to the Python 2.7 installation.

Python doesn't use yum to manage its internal plugins. Instead, it uses its own package manager called pip, which stands for either Pip Installs Python or Pip Installs Packages, depending on who you ask. It's also worth noting that pip is one of several Python packaging and installation tools. Extensive details and discussion can be found at the Python Packaging User Guide site, but the skinny of it is that there are four packages that we want to have installed and current for even our Python 2.7 installation:

• pip - "A tool for installing Python packages."  This is a must-have because it's what will allow you to keep other packages current within Python. Think of it as yum for your Python environment.
• setuptools - "A collection of enhancements to the Python distutils that allow you to more easily build and distribute Python distributions, especially ones that have dependencies on other packages."  This is about simplifying the process of creating Python packages that you can distribute elsewhere. If you intend to write code that might live somewhere other than just your own system, and might ultimately have dependencies on other packages, then this is how you'll accomplish that.
• wheel - "Primarily, the wheel project offers the bdist_wheel setuptools extension for creating wheel distributions. Additionally, it offers its own command line utility for creating and installing wheels."  A wheel is a ZIP-format package with a specially-formatted filename and a .whl extension; it is how packages are distributed. It's another critical ingredient if you're ever going to distribute your code (and probably a good thing to learn about even if you aren't).
• virtualenv - "A tool for creating isolated Python environments."  This allows you to create development environments in a way that won't break dependencies. What if Application A needs version 1 of Foo, but Application B needs version 2 of Foo? Virtualenv helps address this and similar challenges.

Because these are all Python packages, generally we would use pip to install and/or upgrade. However, if you try to run pip right now, you'll see bash: pip: command not found. This is because for Python versions earlier than 2.7.9, pip is not installed by default - and our CentOS 7 installation comes with 2.7.5.

Referring again to the Python Packaging User Guide, we see that pip is provided by a repository which isn't enabled in CentOS by default. It's called EPEL, or Extra Packages for Enterprise Linux. Let's install this repository now.

$ sudo yum -y install epel-release
 ## progress updates on the installation will appear
Complete!
$

Now let's install the latest version of pip that we can get with yum.

$ sudo yum -y install python-pip
 ## progress updates on the installation will appear
Complete!
$

Finally, let's use pip to update itself and to update or install the other Python packages we want to add.

$ sudo pip install --upgrade pip setuptools wheel virtualenv
 ## progress updates on the installation will appear
Successfully installed appdirs-1.4.0 packaging-16.8 pip-9.0.1 pyparsing-2.1.10 setuptools-34.2.0 six-1.10.0 virtualenv-15.1.0 wheel-0.29.0
$

Install Python 3

Download the latest version of Python 3

In your Linux VM, use a web browser to go to https://www.python.org/. Hover your cursor over Downloads in the menu bar, and then click on the latest version under Python Source to download Python.

Alternatively you can visit the Python download site directly at  https://www.python.org/downloads/release/ if you're looking for other versions, or for a .tgz archive instead of the .tar.xz archive.

Extract Python

Launch a Terminal session by clicking, in the top menu bar, Applications / Favorites / Terminal. Extract the Python source to a new folder in your home directory, then change to that directory prior to the next step.

$ tar xf ~/Downloads/Python-3.6.0.tar.xz -C ~
$ cd ~/Python-3.6.0/
[Python-3.6.0]$

Configure the Python source

Before we build the source, we need to make sure that all of the dependencies are there, and we need to prep the environment. This is done automatically by ./configure. Because this isn't making changes to any system directories with restricted permissions, it can be run as a regular user. From the directory that contains the Python source:

$ ./configure
 ## Lots of status updates as it checks dependencies and configures things
$

Compile the Python code

Now that we've configured things so that the code will compile properly, we're going to actually compile it with make. Like .configure, because this isn't making changes to any system directories with restricted permissions, it can be run as a regular user. In the same directory that you ran .configure:

$ make
 ## Lots of status updates as the code compiles
$

Install Python

Now that the code is compiled, we're going to install it with make altinstall. Note the altinstall there - that's the part I mentioned earlier. If you used a regular make install you'd overwrite some things upon which your system relies. Unlike .configure and make, this is actually going to make changes to some system directories with restricted permissions, and so must be run as root. In the same directory that you ran .configure and make:

$ sudo make altinstall
 ## Final batch of status updates as the compiled code is installed
$

Confirm that it works

Now that Python is installed, let's launch it and run one quick command, just to make sure that there are no immediate and obvious major problems. Launch Python with the full path to the version-specific file that you've just installed (/usr/local/bin/python3.6) and look at the next line to confirm that it's showing you the version number you expect; you should see Python 3.6.0.

This will put you interactively inside the Python interpreter. Type print("Hello World") and confirm that you see its output on the next line; then type quit() to exit the Python interpreter.

$ /usr/local/bin/python3.6
Python 3.6.0 (default, Feb  5 2017, 11:28:50) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> print("Hello World")
Hello World
>>> quit()
$ 

Upgrade packaging tools for Python 3.6

Just as we did for Python 2.7, we're going to install and/or update the core packaging tools that we want. Most of these should be installed and current, but to be complete, we'll use pip to attempt an upgrade on all four.

$ sudo /usr/local/bin/pip3.6 install -U pip setuptools wheel virtualenv
Requirement already up-to-date: pip in /usr/local/lib/python3.6/site-packages
Collecting setuptools
  Using cached setuptools-34.2.0-py2.py3-none-any.whl
Collecting wheel
  Using cached wheel-0.29.0-py2.py3-none-any.whl
Collecting virtualenv
  Using cached virtualenv-15.1.0-py2.py3-none-any.whl
Collecting six>=1.6.0 (from setuptools)
  Using cached six-1.10.0-py2.py3-none-any.whl
Collecting packaging>=16.8 (from setuptools)
  Using cached packaging-16.8-py2.py3-none-any.whl
Collecting appdirs>=1.4.0 (from setuptools)
  Using cached appdirs-1.4.0-py2.py3-none-any.whl
Collecting pyparsing (from packaging>=16.8->setuptools)
  Using cached pyparsing-2.1.10-py2.py3-none-any.whl
Installing collected packages: six, pyparsing, packaging, appdirs, setuptools, wheel, virtualenv
  Found existing installation: setuptools 28.8.0
    Uninstalling setuptools-28.8.0:
      Successfully uninstalled setuptools-28.8.0
Successfully installed appdirs-1.4.0 packaging-16.8 pyparsing-2.1.10 setuptools-34.2.0 six-1.10.0 virtualenv-15.1.0 wheel-0.29.0
$

Clean up

Finally, just for good housekeeping, let's get rid of the original download and the temporary directory where we extracted the Python source.

$ rm ~/Downloads/Python-3.6.0.tar.xz
$ sudo rm -rf ~/Python-3.6.0/
$ 

Install PyCharm (a Python IDE)

Download PyCharm (community edition)

Visit http://www.jetbrains.com/pycharm/download/ to retrieve the latest version of the Community edition.

After you click on the Download link, be sure to Save File, rather than choosing to open it with Archive Manager, then put it in your regular Downloads directory.

Extract PyCharm

We're going to put Python in /opt, which the Filesystem Hierarchy Standard tells us is for "the installation of add-on application software packages". This directory is for unbundled packages, each in its own subdirectory (like /opt/someapp). This differs from /usr/local, where we installed Python, in that /usr/local is a place to install files built by the administrator (typically with .configure, make, and make install).

$ sudo tar xzf ~/Downloads/pycharm-community-2016.3.2.tar.gz -C /opt/
$ rm ~/Downloads/pycharm-community-2016.3.2.tar.gz
$ 

Create a symbolic link to PyCharm

PyCharm is now installed in pycharm-community-2016.3.2, but the operating system won't know how to find it unless you manually specify that location each time you want to use it, or add that location to your $PATH environment variable. We don't want to use either of these options, so we're going to create a symbolic link to PyCharm in a location that is in the $PATH.

$ sudo ln -s /opt/pycharm-community-2016.3.2/bin/pycharm.sh /usr/bin/pycharm
$ 

Symbolic links are essentially files which are nothing more than pointers to absolute or relative paths. Take a look at the symbolic link that we just created, for example. See that it is 46 bytes in size? If you count the number of characters in the path it represents, you'll find that there are 46 of them.

$ cd /usr/bin/
$ ls -og pycharm
lrwxrwxrwx. 1 46 Feb  7 06:01 pycharm -> /opt/pycharm-community-2016.3.2/bin/pycharm.sh
$ 

Pretty cool, huh?

Run PyCharm for the first time

Back in your terminal, start PyCharm as a background process.

$ pycharm &
[1] 21742
$

Be patient for a bit while it prepares to run for the first time. You'll eventually get a pop-up window prompt to complete the installation; select "I do not have a previous version of PyCharm or I do not want to import my settings" and click on OK. Accept the privacy policy that appears after that.

Next you'll be prompted to complete the initial configuration of PyCharm. Be sure to select Create desktop entry, and also check For all users, before clicking on OK.

Once this has finished, close PyCharm - then launch it again, but this time by going to your Top Bar and selecting Applications / Programming / PyCharm Community Edition.

Once PyCharm is up and running, look at your dock extension; there's an icon for PyCharm (because it is currently running). Right-click on that icon and select Add to Favorites. This will keep a PyCharm button on your dash, so that you can launch it with only a single click hereafter.

Almost there.

Get rid of the IBus error

We're pretty much done, at this point, with the installation of Python and PyCharm. There's an error message that is appearing on the PyCharm startup screen, though, and that offends my sense of order. This is, after all, a brand new installation; we shouldn't really think of it as "done" if parts of it are telling us they may be broken.

Figure out whether to hide it or fix it

One of the options for this error is to clock on "Do not show again", which presumably will do just that. If this is actually an innocuous or spurious message, that's fine. However, if it represents a potential deeper problem, then we need to figure out how to fix it. Here's a running list of things I discover along the way as I tried to answer this question.

What is IBus, anyway? I found this site, which says that:

The Intelligent Input Bus (IBus) is an input method framework for multilingual input in Unix-like operating systems. It's called "Bus" because it has a bus-like architecture.

That's a little helpful, I suppose. It sounds like I may not need IBus, but I'm not going to make that call quite yet.

I also found a number of places that described a problem with keystrokes occasionally arriving out of order, not just being blocked, and tying it back to an IBus issue. I have been intermittently experiencing exactly this in my virtualized Linux environment, so it seems that I might be affected. That actually makes me feel kind of lucky, because this has pointed me towards a potential solution to a problem that might otherwise have left me scratching my head for much longer.

Looks like I should probably fix it, rather than try to ignore it.

Figure out how I want to fix it

From the IDEA-78860 bug description, it looks like I have three options:

• Disable IBus.
• Enable synchronization mode. From a terminal session, run one or the other (or both) of these:
• echo 'export IBUS_ENABLE_SYNC_MODE=1' >> ~/.profile
• IBUS_ENABLE_SYNC_MODE=1 ibus-daemon --xim --replace
• Upgrade IBus to at least version 1.5.11.

Because gnome.org indicates that IBus is now an integrated part of GNOME, I'd rather not disable IBus altogether. I'm still not entirely certain that I need it, but this approach feels a little like a cop-out, and also minimizes the fun of actually fixing something. If my other solutions do not prove fruitful, though, then I may come back to this and solve it with sudo yum remove ibus.

Synchronization mode looks to be a bit better, and looks like it might be a fairly easy fix, but something about it doesn't feel quite right to me - particularly since I've read several threads with users who have done this but continued to experience issues. I'm not sure that this is the approach I want to use, but may try it later.

I'm going to try upgrading IBus to the most recent version, this approach, as I tend to prefer running more recent versions on principle, anyway.

Upgrading IBus

The IBus ReadMe indicates (for Fedora, at least) that an upgrade is as simple as using yum. However, we need to be on at least version 1.5.11, and yum gets us no further than 1.5.3:

$ yum provides ibus
Loaded plugins: fastestmirror, langpacks, list-data
Loading mirror speeds from cached hostfile
 * base: mirrors.usc.edu
 * extras: repos.lax.quadranet.com
 * updates: mirror.sigmanet.com
ibus-1.5.3-13.el7.i686 : Intelligent Input Bus for Linux OS
Repo        : base

ibus-1.5.3-13.el7.x86_64 : Intelligent Input Bus for Linux OS
Repo        : base

ibus-1.5.3-13.el7.x86_64 : Intelligent Input Bus for Linux OS
Repo        : @base

$ 

Download and extract IBus

Download the latest IBus from its github home at https://github.com/ibus/ibus/wiki. Go to the Downloads section, and then to ibus core under "Source Files"; download the tar.gz file for the latest version (currently that would mean ibus-1.5.14.tar.gz).

In a Terminal, extract this file, then move to that directory.

$ tar -xf ~/Downloads/ibus-1.5.14.tar.gz
$ rm ~/Downloads/ibus-1.5.14.tar.gz 
$ cd ~/ibus-1.5.14/
$ 

Configure IBus (eventually)

Just as before, with Python, we're going to ./configure and make and make install to get IBus installed. There are a few options that we're going to add to ./configure this time around, though.

• -q to suppress the feedback about what is succeeding along the way. This will allow us to more easily find and fix problems with the script as it runs.
• --prefix=/usr and --sysconfdir=/etc because that's how the IBus documentation on github says to run this script.
• --disable-emoji-dict because we haven't installed the EmojiOne package, and don't want that to cause troubles.

$ ./configure -q --prefix=/usr --sysconfdir=/etc --disable-emoji-dict
configure: WARNING: no proper vala compiler found
configure: WARNING: you will not be able to compile vala source files
*** Could not run GLIB test program, checking why...
*** The test program failed to compile or link. See the file config.log for the
*** exact error that occured. This usually means GLIB is incorrectly installed.
configure: error: Package requirements (
    glib-2.0 >= 2.36.0
) were not met:

No package 'glib-2.0' found

Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.

Alternatively, you may set the environment variables GLIB2_CFLAGS
and GLIB2_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.
$ 

Oops - there are some warnings and errors there. I don't like warnings and errors, so I'm going to try to resolve these along the way.

I'll include here only excerpts from the long text outputs, but here is how I resolved these first two issues (for vala and glib2).

For vala, the warning message says that it cannot find a vala compiler, so I'll start there.

1. I used yum search 'vala compiler' on the off chance that it would find something, and it turned up some useful results: packages vala and vala-devel.
2. I used yum info vala to read about this package, and from its description I learned that the Vala compiler is actually named valac. I also observed here that the valapackage is not currently installed on my coupter
3. I used yum provides valac to then confirm which package actually contains that file (so I'll know what to install), and learned that it is the valapackage.

For glib-2.0, I followed a similar approach.

1. I used yum search glib2 to see what it would turn up. This was a bigger list of packages than when I searched for vala, but there are really only two that seem like they may be relevant: glib2 and glib2-devel.
2. I used yum info glib2 to read about that package, and the first thing I noticed was that it was already installed. It's not likely to be the one causing a problem with its absence, then.
3. I used yum info glib2-devel to read about that package, and it was not installed; I'll install it to see if it fixes the problem.

After installing these two packages, I ran ./configure again, and both of those problems had gone away - to be replaced by a new warning about something else.

I used this same loop to resolve each new missing dependency error as it came up. Rather than working through all of those repetitive steps here, I'll skip forward to the actual installation of the packages that are needed in order for the installation of IBus to succeed.

$ sudo yum -y install vala glib2-devel gtk2-devel gtk3-devel dconf-devel libnotify-devel iso-codes-devel gtk-doc
Complete!
$ 

Then it's forward again:

$ ./configure -q --prefix=/usr --sysconfdir=/etc --disable-emoji-dict
config.status: WARNING:  'ibus-1.0.pc.in' seems to ignore the --datarootdir setting
$ 

Aw, nuts. There's one last error. This is happening because in Autoconf 2.60, the set of directory variables was changed, and the defaults of some variables were adjusted to changes in the GNU Coding Standards. Notably, datadir, infodir, and mandir are now expressed in terms of datarootdir. We can correct this by manually making a quick adjustment to the ibus-1.0.pc.in file that's causing the error.

This sed command will insert a new line after the 5th line; this new line will contain datarootdir=@datarootdir@, which should resolve that last warning.

$ sed -i '6idatarootdir=@datarootdir@' ibus-1.0.pc.in
$ 

Finally, let's run ./configure again; if that works we'll move on to make and make install.

$ ./configure -q --prefix=/usr --sysconfdir=/etc --disable-emoji-dict
 ## sweet - not a single error
$ make
 ## all sorts of progress updates until it finishes
$ sudo make install
 ## remember to sudo, or it won't work
$ 

Did it work?

Finally, I'll re-launch PyCharm, to see if it's still giving me the IBus error message that started this work in the first place.

Look, ma, no errors!

Let's not forget our good housekeeping:

$ rm -rf ~/ibus-1.5.14/
$ 

Tweak PyCharm's Python configurations

Tell PyCharm about Python 3.6

By default, PyCharm is going to know only about the Python 2.7 installation - because that's what came with the OS, and because we didn't install Python 3.6 via yum. Let's fix that.

In the PyCharm main window, click on Configure and then on Settings.

On the left, select Project Interpreter, and then near the top right of the subsequent screen, click on the gear icon and select Add Local. Type the path to the Python 3.6 interpreter that we installed earlier - /usr/local/bin/python3.6 - and click on OK. Click on OK again, on the next screen, and you'll be back at the main PyCharm window.

We're all set now. We're The last thing I'll want to do is get github configured and working with PyCharm, and then I'll be able to get started; I'll cover github in my next post.

Ciao!

Part 1: Installing VMware and CentOS
Part 2: Expanding CentOS's Minimal Install
That was Part 3: Installing Python 3.6
Part 4: Linking PyCharm to Github via SSH
Part 5: Running through Git-It again, this time with PyCharm