The last time I raised the topic of image classification I discussed the basic attributes of images. This time I want to focus on the thornier issue of the content, or concepts, depicted in them.
There is a danger of treating an image like a piece of text and classifying its attributes: Who created it? When? What techniques were used? Then writing a title or caption and leaving it at that. Sometimes little more need be done to a document than record this kind of information, especially with free text searching, but lots more needs to be done to most images.
Image findability
Image findability is the process of using search and browse to access the images required. A major aspect of image findability relates to the things depicted in them. Image users often search for images based on the generic things in them and also the proper names of these things. Classifying images based on depicted content means considering anything and everything that is and can be depicted in an image. When considering this I like to focus my efforts on understanding the images I’m dealing with, the users who are trying to find and work with the images, and the ways in which these people need to search and browse for the images they need. After an assessment of these areas I then tailor my approach.
Broadly speaking people searching for depicted content are looking for a number of types:
- Places: cities, towns, villages, streets…
- Built works: parks, skyscrapers, cottages, walls, doors, windows…
- Topography: mountains, valleys…
- Groups and organisations: air forces, choirs, police departments…
- People: roles, occupations, ethnicity and nationality: mothers, doctors, Caucasians, French, Germans…
- Actions, activities and events: running, writing, laughing, smiling, birthdays, parties, book signings, meetings…
- Objects: a myriad of items…
- Animals and plants: common and scientific names…
- Anatomy and attributes of people, animals and plants: arms, legs, adults, leaves, trunks, paws, tails…
- Depicted text shown in images – often signs or writing shown in images…
Many of these generic types can also have proper named instances:
- Proper names of people, places, buildings, topography, organisations, animals etc
When dealing with depicted content I’ve found some of the biggest issues to be:
- Identification – knowing what is in an image
- Focus and specificity – knowing what to include and what to exclude
- Consistency – applying the same term in the same way for the same depicted content
Identification – knowing what is in an image
Depicted content is a relatively black and white area – a dog is depicted so a dog is tagged. However, it might sound a little weird, but working out what is actually in an image can be a lot harder than you think. Take a look at the image “Do You Know What This Is?” by Sister72
This depicted content is fairly simple to see, but understanding what you’re looking at is not that easy. Even if you know roughly what you’re looking at, do you know what it’s actually called?
One tip is to group similar images together when you’re classifying them. Also, always start by assembling as much information as possible before you begin to classify images. It is especially important to gather together the information you have from the creator or custodians of the images.
Also important, when you have the luxury, is to get the image creator to add key metadata about the image at the point of creation, or soon after.
Focus and specificity
Knowing what to include and what to exclude, what to mention and what to ignore, is also much harder than it sounds.
Firstly, some image users will want a piece of depicted content tagged whenever it appears in an image, others will only want it tagged when the image shows a very good representation of that content, and of course many people will want something in between the two extremes.
Different users have different requirements. You need to understand the domain in which you’re working and see the classification of depicted image content as supporting the needs of your users.
For example, Would you tag everything in this ‘Messy Room’ image?
What would you miss out and why?
Looking at the image of “Mountain Goats”, from Thorne Enterprises
Would you tag this with goats as well as mountains? Would this be helpful?
Let’s look at four images depicting windows:
What Light Through Yonder Window Breaks’?
and
Looking at these, it soon becomes clear that even deciding to apply a simple term like ‘Windows’ is not always easy.
Would you apply ‘Windows’ to the image of the cat looking out of the window? Is a window actually depicted in that image? If the image wasn’t tagged with ‘Windows’ how else would anyone find an image of a cat looking out of a window?
The other three images show windows as parts of buildings. but is a building always depicted? Deciding when to apply a building type or the name of a building can be hard. Should you do this every time a part of a building is shown? Only when the whole building is shown? When enough of the building is visible? Or when a section of the building that to most people would represent the build is visible? For example, what part of the Empire State Building would you consider to depict that building? Rarely does anyone see it all – how much is enough? Would you treat the images of windows in a similar way and classify them all with a building type of ‘Houses’, or would you ignore the structure and focus on the parts – the window, the roof?
Consistency
Achieving consistent application of terms to images revolves partly around clear term definitions, well defined application rules and guidelines, and a robust quality assurance process.
Term definitions are very important. Defining the meaning of a term, and ensuring the people choosing which term to assign understand that meaning, can be crucial to term application. For example, creating a term such as ‘Bow’ without defining its meaning is not going to make it easy to apply.
Application rules that are well considered, thorough and clear are also very useful. Even a simple concept often needs some form of guidance linked to it. I remember a while ago needing two terms, ‘Indoors’ and ‘Outdoors’ to allow users to find images of people who were outside and inside – a simple concept you might think, one that people often need, and one that’s easy to apply – who’d need guidelines for that? However, it soon became clear that guidelines were needed after I received a series of interesting questions: Is being on a train indoors? Should studio shots always be considered indoors? Does every shot of a person have to have indoors or outdoors assigned to it? If not, when should this term be used and when not? Is this a focus issue? If so, how much of a location needs to be seen before Indoors or Outdoors is used. A clear set of application guidelines followed an interesting meeting!
Strong quality assurance processes are very valuable. People make mistakes and images generate interesting issues. Appointing staff to review a percentage of classification work based on clear guidelines, and then sharing findings with the people who assigned the terms to the images, is an important way of assessing how well the image classification is progressing and keeping a classification team synchronised.
Today I’ve talked a lot about content depicted in images, next time I’ll focus on abstract concepts which are related to an images ‘aboutness’.
