New GUI for the review of annotated instances

It has been a long time since my last post and I have some interesting news about GAT. For the moment, in this post, I am going to write about its new graphical user interface (GUI). The new GUI of GAT has been designed to offer the user an intuitive environment not only for the semantic annotation and the ontology edition but also for the review of annotated instances.

The new GUI is divided in four tabs: Collection, Image, Annotation and Ontology.

The Collection tab and the Image tab are used for manual annotation of images at global and local level respectively. The ontology tab is used for the edition of ontologies encoded in MPEG-7/XML format. So, what's new? Now, the new Annotation tab allows the user to review all annotated instances of the current ontology and delete wrong instances.

As the following screenshot shows, the Annotation tab includes three components: on the top, a tool bar, below it, the instances panel on the left an the semantic tree on the right.

The "open" button from the tool bar allows the user read an MPEG7/XML file that contains the results of a previously annotation session. The "save" and "save as" button write the current annotation to the file system.

The tree offers a view of the semantic classes of the current ontology and its annotated instances. The root node represents the whole ontology and its children represents semantic classes. The third level of nodes represents instances and its children represents parts. For each node that represents an instance, its associated thumbnail is displayed in the panel.

The user reviews instances by selecting a node corresponding to an instance or a semantic class. If an instance node is selected, the area of support that represents the instance is highlighted in the associated thumbnail. As in the case of a global instance the area of support is the whole image, the thumbnail is recovered by a cyan mask. For a local instance the area of support is a part of the image. Although, at the moment, the thumbnail is also recovered by the mask. By selecting any thumbnail, it is recovered by the mask and the node corresponding to the instance is selected in the tree.

The screenshots below you shows an example. In the first one, the area of support of the instance "Instance 4 of the semantic class "Ramon Pellicer" is highlighted. In the second one, the area of support of all instances of the semantic class "Raquel Sans" are highlighted.


When the user left click on the tree, a popup menu is shown. The option "delete" allows the user to delete the instance represented by the current selected node in the tree.

In the next few days, a version with the new GUI will be available in the GAT web site.

Region selection using traces and rectangles

My name is Pia and this days I'm working on my Bachelor Thesis. It will take 4 months and it will be developed at th office of CCMA-ASI in Esplugues, Barcelona.

Now it's time to present it: The aim is to exploit some tools they already have for image search. Nowadays, typical query by text is the base of these tools but the idea is to help them grow and improve step by step their robustness and complexity by combining improvement of algorithms, similar images, region selection and greater simplicity for users. This means working hard on the interface design and continuous feedbacks with the end-user. My work will focus on region selection using traces and rectangles. It will be based on partition trees.

Partition trees

Partition trees firstly subject the image to a segmentation process depending on color, contour, texture, etc. that generates the initial partition and, secondly, single regions are combined into new regions. This interative process leads to a region-based hierarchical representation of the image. Specifically, we will use BPT (Binary Partition Trees) which are PTs that combine neighboring regions two by two. The figure at the left shows an example of BPT.

Implementation:

Starting on this concept, I met Xavi and Neus last week to discuss the region detection technique we wanted to implement for an easy and intuitive use. It will consist on traces and rectangles. Let me expose both:

• The region selection using traces consists on drawing lines with the mouse. The user will mark the parts of the image he is interested in, then the algorithm will take all the areas that the line is touching adding the ones who are children nodes of the same parent, i.e. brothers of the selected one.
• With rectangles, the notion will be the same but now the region of interest will be every region contained entirely within the rectangle. That is to say, partially contained region will be rejected. It will also cater the nodes with a common parent.
  

Example:

To understand this better, we will put an example: in this simplified drawing we see an anchorman. Behind him we find a white background with green designs. Supposing that we want to search the anchorman regardless of background, we may select him with our markers:

The black line at Figure 3 shows how to select the region using traces while the black rectangle at Figure 4 shows how to select the region using rectanles.

Negative traces and rectangles:

There is something missing that way: how to erase an object that seems to be on the region of interest. We can create a new concept: the negative traces or rectangles. Figure 3 and Figure 4 show this negative traces and rectangles in red.

In Figure 3, this green design at the left could be confused with the clothing of the anchorman. If we draw a negative trace on it, we are rejecting this region. Figure 4 shows the inclusion of two background figures within the rectangle of interest. If we draw new negative rectangles containing these figures we will avoid them.

Region selection using a set of points:

There is a third proposal: it could be interesting to select the areas we want to include in our search just with a set of points and also the areas we don't want to appear. This one introduces a new problem: what if the same region has positive and negative points? In this case we may dismiss the region or take a look at its position inside the root and decide.

IP streaming video on touch-screen mobile devices

Today I’ll focus on video players that allows IP streaming video on mobile devices. That’s one of the goals of my thesis, which consists of designing a touch interface for iPhone to player streaming videos from a server (that I also configure!).

Nowadays, there’s a growing mobile devices market, specially of devices with tactile interfaces. Most of them allows IP streaming video through wifi or 3G network such as smarthphones or tablets PC.

There’s a lot of smartphones that can play video: iPhone, Anrdoid Phones, Nokia touch screen series (with Symbian OS) or other touch screen mobile phones with Windows Mobile or other OS; although Apple’s iPhone and Google’s Androind are the main smartphones. These have lot of apps to download videos, player live streaming or video on demand, upload video from the camcorder... All of these are available to download from Android Market (Android Phones), App Store (Apple devices) or Cydia (only for a jailbroken iPhones).

Android Phones supports a lot of video formats such as MPEG4, H.264 or OGG. There’s a lot different devices that works with Android OS, such as devices from the companies like HTC, LG, Motorola, Samsung... So, all of these have different hardware and characteristics. Useful video apps are Yuliot, Qik, Ustream, Bambuser... all of these to stream live video to multiple Android Phones, iPhone and the Web. These are also available from iPhone, Symbian and Windows Mobile.

iPhone or iPod touch has a 3.5’’ screen at 320x480p resolution and allows H.264 Baseline Profile and MPEG-4 Simple Profile with m4v, mp4 and mov video formats. It has a lot of apps to player video on streaming, but the most used are Joost TV, Vuze (that allow video on demand and live streaming), Air Video, Qik and Ustream (that also allows stream live video to multiple platforms). All of these apps play video on the native formats, but the newest app Oplayer also plays avi, xvid and mpg files, so it’s called “the universal player”.

Moreover, for jailbroken iPhones, exists and alternative market called Cydia where users can download apps such as PwnPlayer or VLC4iphone, the VLC media player from videoland project.

Tablets PC refers to a slate-shaped mobile computer device, equipped with a touchscreen or stylus. All of them allows streaming video services as a computer. So, the main method is through internet browsers such as Youtube, Megavideo, Stage6, Joox, Yendit, Fase6... It also allows live stream from the webcam.

The newest apple device called iPad has revolutionized the Tablet PC market. Nowadays it doesn’t allow 3G connectivity and Wifi, but these functionalities will be available since next April. iPad supports all of iPhone formats and it also allows video H.264 Main Profile 720p and HD resolution. iPad work with almost all of the apps designed for the iPhone, but it’s expected that new applications or video servers exploits the large high-resolution screen.

Well, next week I’ll talk about video servers.

Twitter on TV? First steps.

Hello everybody! I am Cristina Ruiz, student of Telecommunications Engineering specialized in Sound and Image from EUETIT. This week I have started to develop my final bachelor thesis, that’s why I have decided to begin writing the blog today. I will try to post every Saturday in order to report the progress of my project and to explain new skills and knowledge I will get during the process.

Activa Multimedia Digital is the name of the company where I am doing my thesis, it is part of the CCMA (public group that leads the radio and television broadcasting services from Catalonia). Specifically, I am working with the R+D department in a project called CREA-IPTV which studies the possibilities of interactive and advanced services using IPTV. They have designed an application for the Netgem Netbox 8160 that provides television services over IP, such as extra content related with the delivered program, videos from 3alacarta or Facebook Connect, in any of the TVC (Television of Catalonia) multiplex channels.

My task will consist in supporting the process of the pilot testing that is going to be conducted in twenty different home users, and in expanding the application by adding a Twitter option.

Probably, most of you already know what Twitter is, however, I am going to make a brief introduction to those who have never heard about it or those who know that it exists but they have never used it.

Twitter is a microblogging tool that allows each registered user to write a text of 140 characters maximum called “tweet”, answering the question “What’s happening?”. People often publish their thoughts, interesting pieces of news, opinions and curious links.
Your followers can read your tweets and you can also choose people that you want to follow without the need of being accepted by them, this feature assigns Twitter a social network character.
There are two special characters used on Twitter:

• @ An at sing is used to reference a tweet to a specific user or to reply directly to it. For example: @user1 hello! The user who has posted the tweet is saying hello to user1.
• # A hash is written before a clue word when you want your tweet to take part in a global conversation channel; the clue word is like a label that defines the theme of the tweet. For instance, if you click on #hello or write it on the searching window, all tweets that include #hello appear on the screen.

In addition, there is a “retweet” option to post on your own user page a tweet of another user that you have found interesting and you want to share with your followers.

What is more, the widespread use of Twitter has led to the emergence of new services as the URL shortener provided by bit.ly, which lets you tweeting links without having to worry about its length.

After observing Twitter operation a little more, I concluded that my Twitter application will must be focused on TV3 programs that already have Twitter on the web and it should display on the TV all “tweets” posted by the specific program and also the ones that are targeted to it. Now it’s time to investigate Twitter API and to start to program the code.

I will see how it goes and I will tell you!

Playing with GAT

Yesterday, I learned how to use GAT’s application. It’s a really graphic and intuitive application but at the beginning it was a bit difficult for me although I’d got a good teacher.

If you usually read this blog, you should know for what is useful this application, if not, I’m gonna explain it to you. GAT has been created to annotate different parts of an image, but in my case, I just want to annotate if this image contains an anchor of it doesn’t.

Firstly, I went to Ontology’s and there I created a new ontology called Anchor. Then, in Glogal, I search in our database a folder that contained some TV3 anchor’s images. At that time, I selected all these images and thus, going to Annotation, you are able to see the selected images before even though they were in different folders!!

How you can see in the image, in the right side of it, you can see a list of instances. Each one of these instances corresponds an image. If you select an image, automatically the instance corresponding selected too and vice versa!!!

When we have to choose the different images of the anchors, we had had to create a criterion because there are some different images of anchors. The anchors had to stay in the foreground (how it’s required at the beginning) provided that are from tv news as TV3, 3/24, TN noon. That’s because the background scenery changes. An other decision has benn to discard the darks images. In that case, the seleccion of the image shown its incorrect because we should take out the image located in the second row third column that corresponds to instance 11.

Dynamic Search Spaces

The last few weeks of work GOS has experienced a significant change in reference to the selection of the search space.

As already known, GOS let run two different execution modes: local and remote. After setting the correct conection between the client and the Upseek server in remote mode, it is time for the user being able to define the search space manually and change it at any time. The two execution modes demand a different way to make this selection:

Local Mode
The search space panel have a one level system directory tree. By default, the contents are loaded by the directory specified in the configuration properties file.



To navigate the system and choose another directory for use as a root node of the search space, the user must click the right mouse button on the panel. This will open a secondary window that allows him to navigate the system selecting the new directory. The leaf selected when the user runs the query by example, along with all its children, will be the search space defined by the user and a ‘.txt’ file will be created containing all regions’ visual descriptors files path contained in these directories. This file will be sent to RANKER.



Remote Mode
We are currently experimenting with the creation of the search space file in the remote mode. To load the tree on the panel we will launch a query against the FedoraCommons upseek server database defining the ‘creator’ (Dublin Core metadata field). This will allow us to list all the ‘sources’ (Dublin Core metadata field) of the same creator. (Khristina wrote about Dublin Core metadata fields and how to use them).

When launching the query by example against the server a file is created containing all the pids with the ‘creator’ and, in case user has selected any of the tree nodes, the ‘source’ selected.

No doubt recent changes allow a greater freedom for the user and a very important restriction for the queries by examples.

Find, look, annotate

Today was a great day for team work. I met the developers of our CBIR system to re-invent the user interfaces once again. Our long-term goal: integrate UPLoad, GAT and GOS in a single and intuitive interface to the UPSeek server.

We started by reviewing the functionalities of the the three stand-alone graphical applications that connect with our UPSeek server. Firstly, UPLoad ingests images and Dublin Core metadata to the Fedora Commons Repository. Afterwards, GAT can annotate images and regions to generate more Dublin Core metadata as well as MPEG-7/XML (in the future, OWL). Finally, GOS is our search interace, that presently only supports query by example by in the future could implement query by text, sketch or any combination of them.

After two hours and a half of discussion, we came up with a design that would satisfy all current and future operations. It is inspired in the Eclipse interface and has three basic parts: a left panel to find images, a central panel to show their content and a right panel to display their metadata.

The search panel will consist on two tabs, one for navigation and another one to formulate queries on the database. Navigation will be supported both on the local filesystem (like UPLoad) and on the remote server, a new functionality that Manel is developing for GOS search spaces. The query tab will have a tree to define the search space and buttons to select the visual descriptors in case of query by example. Queries could be composite, combing texts or visual criteria.

The image panel will show the thumbnails of the content located thanks to the search panel. If a thumbnail is double-clicked, it will be zoomed to select a local region of it. This selection could be used for a local annotation or to formulate a region-based query by example.

Finally, the right metadata panel will include two tabs, one showing the Dublin Core fields and a second one with the MPEG-7/XML or OWL semantics. Users will be able to edit the images metadata according to the solutions implemented so far in UPLoad and GAT.

We are aware that these changes are massive and that it will take a long time before we do not have this all-in-one UPSeek client. Nevertheless, today's brainstorming was an excellent opportunity for everybody to clarify those aspects of the project that are not directly related to everyday's work. It is important that we all have a global view of the complete system before taking individual decisions that could help the rest of the team's work.