CE316 Laboratory: Assignment 1

Introduction

One of the many outstanding problems in computer vision involves classification, recognising the categories that features in images fall into. Images of galaxies are one type of feature that is difficult to classify on the basis of shape, as you can see from the images below: the left one is an elliptical galaxy while the one on the right is a spiral (actually an anti-clockwise spiral).

One thing that you should be able to see from these images is that there are more blue-coloured stars in the spiral galaxy than in the elliptical one — this characteristic can be used as the basis of an algorithm to distinguish the two types.

A galaxy recognition program

Your task is to write a program that reads in an image of a galaxy and identifies it as being elliptical or spiral. You are free to use whatever algorithm you like.  You may implement your program in any of C, C++ or Python, using the OpenCV library. You are free to use any OpenCV routines you like in your program.  If you are using C or C++, you should supply a Makefile that compiles your source code, generating an executable program called galaxy.

Your software must build and run under Linux on the machines in Computer Lab 1.  The executable version of your program must be called galaxy (with no extension) and it must be able to take the image to be classified on the command line; for example, to process the image galpic001.jpg it would be invoked as

./galaxy galpic01.jpg
The program must print on the standard output stream one of
ellipse
or
spiral
There must be no other output. These restrictions are so that it works with the test harness described below.

Comparing and assessing algorithms

The principles of evaluating a single algorithm and comparing the performances of several algorithms are described in the lecture notes. It is not especially difficult to perform this kind of thing by hand, though it is very tedious. For this reason, evaluations are usually performed using programs known as test harnesses. You will use a test harness and some image files to evaluate the performance of galaxy and compare its performance with that of agal, my own attempt at classifying galaxies.

The particular test harness you will use is FACT ("Framework for Algorithm Comparison and Testing"), demonstrated in lectures. You need to download FACT and the relevant data files. FACT separates the stages of executing a program on a series of tests and analyzing the results. This is because the execution stage is normally much slower than the analysis one and, as we shall see, there are several analyses that one might like to perform. The execution stage produces a transcript file, and all the analysis stages use transcript files as input.

The file galaxy-tests.fact contains the tests that are to be executed for this evaluation; it is human-readable and you are welcome to look at it. You should be able to run the test on your galaxy program using the command

./fact execute galaxy-tests

The execute tells FACT to use the test script galaxy-tests.fact and output a transcript; you can use run rather than execute if you prefer. FACT uses an "interface file" interface.py to run your program: this is a Python routine (FACT is written in Python) that runs your galaxy program and returns its output to FACT.

When you execute the above command, FACT will write output to your terminal window. The first line contains some identification information, used for checking in the analysis stages, followed by a single line per test. These lines are actually the transcript.

To create a transcript file, you simply use command-line redirection to make these lines go to a file

./fact execute galaxy-tests > galaxy.res
and twiddle your thumbs while it runs. Having generated the transcript, the next stage is to analyze it, either in isolation or by comparing it with other algorithms.

Analysing the transcript file is both quick and easy:

./fact analyse galaxy.res
Rather than analyse, you can write analyze or anal. If the name of the file that you wish to analyse ends in .res, you can omit the .res. The results of the analysis are written to your terminal window; you can use re-direction to save it in a file. You will see that the output contains two distinct tables, one summarising error rates etc. and the other a confusion matrix, which shows how false positives occur. You should interpret these in conjunction with the relevant part of the lecture notes.

Your next step is to compare the performance of your program with that of my own algorithm, agal. The transcript file from agal for the same tests is called agal.res and is part of the zip-file that you downloaded above. You perform the comparison using the command:

./fact compare galaxy.res agal.res

You can get more detail by appending --detail=2 to the command. Again, you should interpret the results in conjunction with the lecture notes.

Reporting your results

Having assessed the performance of galaxy in isolation and compared it with that of agal, you should summarize the results and interpret them in depth in the block of comments at the top of your program. I expect the results and interpretation to occupy a couple of pages of the print-out of your program, not just a few lines.

Note that your mark for the assignment does not depend on your program out-performing agal; your mark is determined by the quality of your coding and the correct interpretation of the performance evaluation results.

Summary

Submission deadline: Thu 10th November at 11:59:59 (noon)
What to submit: the source code of your program
any Makefile etc needed for compilation
galaxy.res
but not the images!
Please submit the files individually, not as a zip-file or similar.
Marks returned: three weeks
Assessment criteria: see the detailed description of the criteria

Remember to identify you work with your registration number only.  The coursework system allows you to upload your work as often as you like, so do keep uploading your files as you develop them.

Last updated on 2016-10-24 09:22:47 Web pages maintained by Adrian F. Clark [contact]