LPTHW - Exercise 48

In this exercise, we created an intelligent input scanner, which we can use in our games. This scanner takes a line of input, and returns a list of tuples, each containg the 'type of word' and the word itself.

We were given the unit tests (lexicon_tests.py), and were supposed to write the actual code (lexicon.py) to fulfill the tests. This exercise was a lot of fun.

The test suite and code is embedded below:

LPTHW - Exercise 47

In this exercise we create a very simple class for a game, and then create unit tests for it.

Since, I have already done much programming (albeit in Java), I am familiar with the concept of unit testing, and why it is important.

Python seems to have something called 'nose' which is a test harness similar to JUnit for Java. Here is the production and test code modules.



In our project, we have a directory called tests, which holds all our tests. The test module name is 'actualmodule_test.py', where 'actualmodule' is the name of the module which is being tested.

The test script has setup and teardown functions, but when I ran the tests with nosetests, these functions were not run. I need to figure out why.

Creating and installing a script in my distribution module

In the previous blog post, I explained how to create and distribute a package distribution. If the package were a library, then most likely this is where it would end. However, since I have created something which will be used as a command line desktop application, it would be wonderful if I could bundle a script which the user can invoke from the command line.

First I made a shell script, which would invoke a Python module, however I ran into an issue where the interpreter could not locate the Python module. I think it should have since the package has already been installed with distutils. I need to spend some time understanding how the interpreter locates modules.

In the spirit of moving ahead, I decided to create a Python script and make it executable instead. Once I created the script, I again ran setup.py, which caused this script to be copied in the 'bin' directory of my ENV. Here is the simple Python script I created.

Creating a simple distribution module

To understand better all the things I learned in Exercise 46, I am now going to create a simple distribution module which will work with distutils. Here is the documentation of how to create such a module with distutils.

I am creating a simple Python Pomodoro.

First, I copied the skeleton directory into another directory called 'pypomodoro'. Next I renamed the package directory 'NAME' to 'pypomodoro'. Next, in the tests directory, I renamed NAME_tests.py to pypomodoro_tests.py. Next, in setup.py I added 'pypomodoro' to the packages names parameter.

Packaging is usually a nightmare on all platforms... and it is not exception with Python, at least when learning it.

Anyways, here are some notes which I am hoping will help me understand all this better.

In Python project names usually use camel case. So I have renamed my python-pomodoro project to PyPomodoro.

The project directory will usually contain a few more directories and files.

PythonProject

\bin

\docs

\tests

__init__.py

module_name_tests.py

\main_project_module

__init__.py

module1.py

module2.py

setup.py

I realized that we refer to the terminology module for individual python files, as well as to directories. Directories which are modules however, must have a file called __init__.py

So, this is a bit confusing. A bit of Googling takes me to this tutorial on Python packages. This tutorial states

"Packages are a way of structuring Python's module namespace by using dotted module names"

This makes sense. Looks similar to packages in Java, but the term 'packages' in Python seems to be overloaded. I believe I also encountered it in the context of the thing which contains setup.py. Perhaps this is not a package, but a 'package distribution'.

We can import sub-modules like this:

import A.B

where we expect B to be a submodule of A. Note that B cannot be a function or a member inside A. It has to be a submodule. What this means is that the directory A, must have a file called __init__.py and also if B is a directory, it must have a file called __init__.py However, if B can also be a file. The __init__.py file is invoked when the module containing it is loaded. Here is a tutorial explaining the usage of __init__.py

Question: What if the module is loaded multiple times? Perhaps I should also ask the question, if a module CAN be loaded multiple times.

Ways to install a Python package

After doing exercise 46 of LPTHW, I realized that there are multiple ways of installing a Python package.

Here are three ways that I know of.

pip install package_name

easy_install package_name

Download and unzip the tar file and then do
python setup.py install

LPTHW Exercise 46

In this exercise, we create a skeleton directory for future Python projects. This skeleton basically contains a directory structure, and an install script.

For all this to work properly, we need to have certain Python packages installed. The book recommends the following packages.

pip
distribute
nose
virtialenv

Now I am going to install all of the above packages.

Installing Pip:

pip is a Python package installer (which installs packages from the Python package index). It is meant to be a replacement for easy_install. So anything which can be installed with easy_install can also be installed with pip.

The pre-requisites for installing Pip are either setup_tools, or distribute. I read that setup_tools is not supported on Python 3.0. Even though this is not a problem, since I have Python 2.6, I figured it would be a good idea to go with 'distribute'

Distribute is also on the list of packages to install, so I install 'distribute' first.

curl http://python-distribute.org/distribute_setup.py | python

But I am also a bit curious about what distribute does. The documentation says it is a tool to 'Easily download, build, install, upgrade, and uninstall Python packages'. So basically this looks like a lower level tool which is needed by pip.

Now that I have installed 'distribute' I also installed Pip.

Installing VirtualEnv

Next I installed 'virtualenv' using pip:

'sudo pip install virtualenv'

I think there is a lot to virtualenv. I will write more about it in my next blog post. For now, I will link to the official virtualenv documentation page.

After installing virtualenv, I created an environment called ENV1.

I will now try to install nose in ENV1.

I activated the newly created environment running source bin/activate and then ran pip install nose.

Extra Credit:

So what do all of these software's do ?

1. pip - Is used for installing packages which are present in the PyPy package index
2. Distribute - Is a lower level tool for building, installing, and managing Python packages
3. VirtualEnv - Is a tool to create isolated environments for Python
4. Nose - Extends unittest to make testing easier

Understand setup.py

setup.py is used by the Python distutils as a standard way for installing third party Python modules. Before distutils, module creators would have to create install files for all the different platforms. Off course this is not easy, and I am sure there would be times when some module creators would not create such files. I believe since distutils uses Python, it is a way to use Python to install modules, rather than relying on the platforms way of installing Python packages. I also think this is good encapsulation :-)

If you are using distutils then you will know immediately if the package maintainer has packages their application in standard way or not. A Python package which is meant to be installed using distutils will always have a setup.py file bundled with it. This file will contain the calls it makes to distutils to install the package.

When we run

python setup.py install

There are really two things that happen. First is the build step, which puts all the files which need to be copied into the distribution root's build directory. I think by distribution root they mean the directory which is created for the distributed package. We can however change the directory in which the build process happens, if we so desire.

Once the build phase has completed, the install phase will copy the files to the chosen install directory. If no directory is specified then the default directory, which on *nix systems is usually /something/pythonX.x/site-packages. On my Ubuntu system it is /usr/local/lib/python2.6/site-packages However, it seems that modules are actually being installed in dist-packages.

It is possible to install the package using an alternate install scheme. There are various alternate install schemes. Even though I have not looked into them in any great detail, they seem to be able to not only customize the install location, but also determine who is able to access and use the package. Details for all of these install schemes is available here.

I will stop here as far as using disutils is concerned. This is a good introduction, and should be fine to start with.

Now I will understand the basics of using distutils for package creators.

Since distutils is also about packaging, let us try to understand some basic terminology. We will talk about general Python terminology followed by distutils terminology.

1. Module - This is the basic unit of code-reusability in Python. It is a block of code, which is usually imported by other code to be used. This is usually a single Python file.
2. Package - This is usually a module that contains other modules. It is usually contained in a directory. This directory must always contain a file called __init__.py
3. Root Package - This is the root of the heirarchies of packages. The root package usually contains the entire Python standard library, along with many stand alone modules which do not belong anywhere else. Now the documentation says something which totally freaked me out ... "Unlike regular packages, modules in the root package can be found in many directories: in fact, every directory listed in sys.pathcontributes modules to the root package"

distutils specific terminology:

1. Module Distribution is a collection of Python modules and packages, which are meant to be distributed en-masse.
2. Distribution Root is the directory which contains setup.py

Now let us understand the setup script from the perspective of a package creator. The main purpose of a setup script is to describe your module to distutils, so that the various commands that operate on your module do the right thing.

Without getting into too much detail, I will now create a simple module distribution and create a setup script for it. I will write about it in a separate blog post.

LPTHW Exercise 45

In this exercise we learn about IS-A and HAS-A relationships. We also learn how to invoke A superclass constructor from a subclass.