IdeaPanel

Xiao, Changcheng; Wang, Changhu; Zhang, Liqing; Zhang, Lei

IdeaPanel

Changcheng Xiao

2015, Proceedings of the 5th ACM on International Conference on Multimedia Retrieval - ICMR '15

visibility

…

description

2 pages

link

1 file

In this work, we introduce the IdeaPanel system, an interactive sketch-based image search engine with millions of images. IdeaPanel enables users to sketch the target image in their minds and also supports tagging to describe their intentions. After a search is triggered, similar images will be returned in real time, based on which users can interactively refine their query sketches until ideal images are returned. Different from existing work, most of which requires a huge amount of memory for indexing and matching, IdeaPanel can achieve very competitive performance but requires much less memory storage. IdeaPanel needs only about 240M B memory to index 1.3M images (less than 3% of previous MindFinder system). Due to its high accuracy and low memory cost, IdealPanel can scale up to much larger database and thus has larger potential to return the most desired images for users.

IdeaPanel: A Large Scale Interactive Sketch-based Image Search System Changcheng Xiao1∗, Changhu Wang2†, Liqing Zhang1 , Lei Zhang3 1 Shanghai Jiao Tong University, Shanghai, China 2 Microsoft Research, Beijing, China 3 Microsoft Corporation, Redmond, USA [email protected], [email protected], [email protected], [email protected] ABSTRACT In this work, we introduce the IdeaPanel system, an interactive sketch-based image search engine with millions of images. IdeaPanel enables users to sketch the target image in their minds and also supports tagging to describe their intentions. After a search is triggered, similar images will be returned in real time, based on which users can interactively refine their query sketches until ideal images are returned. Different from existing work, most of which requires a huge amount of memory for indexing and matching, IdeaPanel can achieve very competitive performance but requires much less memory storage. IdeaPanel needs only about 240M B memory to index 1.3M images (less than 3% of previous MindFinder system). Due to its high accuracy and low memory cost, IdealPanel can scale up to much larger database and thus has larger potential to return the most desired images for users. Categories and Subject Descriptors H.3.3 [Information Search and Retrieval]: Search Process, Query formulation; H.5.2 [User Interfaces]: Usercentered Design Keywords Sketch-based Image Retrieval, Shape Words, Interactive Search 1. INTRODUCTION Owing to the prevalence of Internet and mobile phones, billions of digital images are now freely accessible online, which bring a great opportunity for large scale image search. Usually, users search images with text queries. However, the ∗ Changcheng Xiao performed this work while being an intern at Microsoft Research Asia. † Corresponding author. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage, and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). Copyright is held by the author/owner(s). ICMR’15, June 23–26, 2015, Shanghai, China. ACM 978-1-4503-3274-3/15/06. http://dx.doi.org/10.1145/2671188.2749410 . Interac vely Sketching Tagging Target UI User Sketch Query Rank with Sketch Index Search Engine Figure 1: Illustration of the IdeaPanel system. Users can draw the major curves of the target image on the canvas panel. Then natural images with similar contours will be returned in real time. shape and location of the object cannot be easily formulated as a few keywords. Thus, query-by-example (QBE) was proposed. However, in QBE, a typical query is an example image, which is usually the reason of searching. To overcome existing problems of current image search engines. A natural solution is to enable users to flexibly express their intentions by providing them a pen and a drawing panel. With the popularity of touch-screen devices, all necessary prerequisites to sketch-based image retrieval (SBIR) are ready. As illustrated in Fig. 1 & 2, users can detail their ideas by interactively drawing sketches, which will be the queries to our system. Natural images with similar contours will be returned in real time, and users can interactively refine their sketches or add some tags until ideal images are returned. SBIR is complementary and thus can be combined with query-by-keyword and query-by-example modalities. Sketch based image retrieval has been extensively studied since 1990s [3], and stepped into large-scale scenarios in recent years [4, 2, 6, 5]. However, most of these methods focus on low level descriptors of sketches like local patches [4] or edge pixels [2], which requires huge amount of memory (e.g. MindFinder [2] requires 8GB memory for 2.1M images). Some hash-based methods [6, 5] have been proposed to save the memory cost of SBIR system while sacrificing some accuracy at the same time. In this work, we systematically solve the representation, matching and indexing problems of large scale sketch-based image retrieval. We try to step forward and see the sketches Canvas Panel Toolbox Sidebar (a) Input Page Image Panel (b) Result Page Figure 2: The Windows 8 Metro UI of IdeaPanel. Figure 3: Examples of hand-drawn sketches. in a higher view. Different from MindFinder [2] which indexes and matches with (sampled) edge pixels, we propose to represent both the query sketch and the natural image in the database as a bag of shape words. As shown in Fig. 2, when drawing a bike, most users will probably draw two circles as its wheels followed by a triangle as its frame. Ideally, it will be quite powerful if we can successfully extract the triangles and circles from database images, and match them with the shapes of the query sketch. However, it’s still a very challenging task to extract such shapes from natural images with high precision and recall. To balance the robustness and computational complexity, we leverage a middle level descriptor between edge pixels and shapes, i.e., line segments and circular arcs, as shape words. We propose an efficient algorithm for shape words extraction, followed by a generalization of the classical Chamfer Matching algorithm [1] for shape words’ matching. Finally, a novel inverted index structure is proposed to make shape words representation scalable to large scale image databases. Based on the shape words representation, we build the IdeaPanel system on two image databases, with 1.3M and 4.5M natural images respectively. Owing to the compact representation, novel matching and indexing strategies, IdeaPanel only requires 3% memory storage of the MindFinder system while holds very competitive accuracy at the same time. 2. SYSTEM OVERVIEW In this section, we introduce the major functions of the IdeaPanel system. As shown in Fig. 2, the UI of IdeaPanel contains two pages: the input page and the result page. The input page provides a canvas panel to help users flexibly express their search intentions. Below the canvas panel is the toolbox, which enables users to quickly edit their sketches or add some tags. When a search is finished, the returned images will be shown in the image panel of the result page. The major querying operation of IdeaPanel is sketching, which describes the main curves of the image in the user’s mind. As some complex objects are not easy to draw by normal users, our system also enables users to add some tags to constrain the semantic subject of the returned images. So our system support the following two query modes: Query by Sketch Draw the major curves of the target image within the canvas panel (Fig. 2 (a)). When finished, click the search button to start the search. Fig. 3 shows some example hand-drawn sketches. Query by Sketch + Tag Click the text box button in the toolbox, type in some Figure 4: Example queries and corresponding results. Images marked by red cross indicate that they are irrelevant with the query. keywords, and click the search button to start a new search. Users are also allowed to directly filter the returned images by choosing a proper tag in the slide bar of the result page. Fig. 4 shows several example sketch queries and corresponding top results, which are precise and structure sensitive. 3. ACKNOWLEDGMENTS The work of first and third authors was supported by the National Natural Science Foundation of China (Grant No. 61272251) and National Key Basic Research Program of China (Grant No. 2015CB856004). 4. REFERENCES [1] G. Borgefors. Hierarchical chamfer matching: A parametric edge matching algorithm. IEEE PAMI, 1988. [2] Y. Cao, H. Wang, C. Wang, Z. Li, L. Zhang, and L. Zhang. Mindfinder: Interactive sketch-based image search on millions of images. ACM MM, 2010. [3] R. Datta, D. Joshi, J. Li, and J. Z. Wang. Image retrieval: Ideas, influences, and trends of the new age. ACM Comput. Surv., 40:5:1–5:60, 2008. [4] M. Eitz, K. Hildebrand, T. Boubekeur, and M. Alexa. A descriptor for large scale image retrieval based on sketched feature lines. SBIM, 2009. [5] X. Sun, C. Wang, C. Xu, and L. Zhang. Indexing billions of images for sketch-based retrieval. ACM MM, 2013. [6] K.-Y. Tseng, Y.-L. Lin, Y.-H. Chen, and W. H. Hsu. Sketch-based image retrieval on mobile devices using compact hash bits. ACM MM, 2012.

Log In

IdeaPanel

Related papers

Related papers