CE866 Laboratory: Assignment 1

Introduction

You will recall the discussion in the lecture notes of cbir, a program that performs content-based image retrieval. You will also remember that lectures discussed how vision algorithms are evaluated and compared. This assignment brings these two topics together: you are required to write a content-based image retrieval program of your own (not using exactly the same algorithm as the program in the notes) and compare its performance with that of cbir. These two stages in the assignment are described in more detail below.

Content-based image retrieval

The cbir program in the notes calculates the histogram of each image (i.e., there is one histogram, even if the image is a colour one). You are free to use any other viable algorithm that you can conceive — the purpose of the assignment is not to out-perform cbir, simply to have an alternative algorithm that you can compare with it. If you really cannot think of an algorithm to use, speak to a demonstrator who will suggest one.

Having chosen an algorithm, you are required to write a program that operates in the same way as cbir: your program must take a number of image filenames on its command line, the first of which is the "probe image" that you are trying to find in the other images supplied on the command line. Your program should generate precisely one line of output, the name of the image that best matches the probe image. Your program must explicitly ignore the case when the test image has the same name as the probe image — if you seem to be getting perfect results, you have probably forgotten to do this. For inspiration, the cbir program discussed in lectures is presented in the lecture notes.

You may implement your program in any of C, C++ or Python, and it must use the OpenCV library. You are free to use any OpenCV routines in your program.

Your software must build and run under Linux on the machines in Computer Lab 1.

Comparing and assessing algorithms

The principles of evaluating a single algorithm and comparing the performances of several algorithms is described in the lecture notes. It is not especially difficult to perform this kind of thing by hand — but it is very tedious. For this reason, evaluations are usually performed using programs known as test harnesses. For the second part of the assignment, you will use a test harness and some image files to evaluate mycbir and compare its performance with cbir.

The particular test harness you will use is FACT ("Framework for Algorithm Comparison and Testing"). You need to download FACT and the relevant data files. FACT separates the stages of executing a program on a series of tests and analyzing the results. This is because the execution stage is normally much slower than the analysis one and, as we shall see, there are several analyses that one might like to perform. The execution stage produces a transcript file, and all the analysis stages use transcript files as input.

To work with the test harness described below, your program must be called mycbir (no extension) and must be executable: if you're programming in Python, you do that by typing the command

  chmod +x mycbir
The demonstrators will help if you have trouble doing this. It is also essential that your program outputs nothing but the best-matching image filename to work with the test harness.

The file ass1.fact contains the tests that are to be executed; it is human-readable and you are welcome to look at it. You should be able to run the test on mycbir using the command

./fact execute ass1
The execute tells FACT to run the program on the test script ass1.fact and output a transcript; you can use run rather than execute if you prefer. (Note that FACT uses the file interface.py as an interface to the program being tested.) When you execute the above command, FACT will write output to your terminal window. The first line contains some identification information, used for checking in the analysis stages, followed by a single line per test. These lines are actually the transcript.

To create a transcript file, you simply use command-line redirection to make these lines go to a file

./fact execute ass1 > mycbir.res
and twiddle your thumbs while it runs. Having generated the transcript, the next stage is to analyze it, either in isolation or by comparing it with other algorithms.

Analysing the transcript file is both quick and easy:

./fact analyse mycbir.res
Rather than analyse, you can write analyze or anal. If the name of the file that you wish to analyse ends in .res, you can omit it. The results of the analysis are written to your terminal window; you can use re-direction to save it in a file. You will see that the output contains two distinct tables, one summarising error rates etc. and the other a confusion matrix, which shows how false positives occur.

FACT can generate HTML rather than plain text:

./fact --format=html --detail=2 -H -T analyse mycbir.res > mycbir.html
which you may find easier to read.

Your next step is to compare the performance of your program, mycbir, with the cbir program demonstrated in lectures. The transcript file from cbir, called cbir.res, is part of the assignment distribution that you downloaded above, and you should have generated mycbir.res above. You perform the comparison using the command

./fact compare mycbir.res cbir.res
Again, you can get more detail by appending --detail=2 to the command.

Having assessed the performance of mycbir in isolation and compared it with that of cbir, you should insert the FACT outputs into the comments at the top of your program and explain what they mean. This interpretation of the numerical results is an important part of the assignment.

When you upload your submission, remember to upload the transcript file from your algorithm too.

Summary

Submission deadline: Thursday 23th November at 12:59:59
What to submit: the source code of your program
any Makefile needed to compile your program
the transcript file from your program
but not the images!
Feedback returned: three weeks
Assessment criteria: see the detailed description of the criteria

Remember to identify your work with your registration number only.  The coursework system allows you to upload your work as often as you like, so do keep uploading your files as you develop them.

Last updated on 2017-10-06 10:58:46 Web pages maintained by Adrian F. Clark [contact]