External memory algorithms

Allen (2001) proposed the "Getting Things Done" (GTD) method for personal productivity enhancement, and reduction of the stress caused by information overload. This paper argues that recent insights in psychology and cognitive science support and extend GTD's recommendations. We first summarize GTD with the help of a flowchart. We then review the theories of situated, embodied and distributed cognition that purport to explain how the brain processes information and plans actions in the real world. The conclusion is that the brain heavily relies on the environment, to function as an external memory, a trigger for actions, and a source of affordances, disturbances and feedback. We then show how these principles are practically implemented in GTD, with its focus on organizing tasks into "actionable" external memories, and on opportunistic, situation-dependent execution. Finally, we propose an extension of GTD to support collaborative work, inspired by the concept of stigmergy.

Writing is analyzed as thinking that uses paper or other media to externalize and manipulate symbolic expressions. Mental operations of natural language can occur somewhat independently, and they communicate well with language that has been written, but for skilled writing these operations need elaborate installation in the mind. We explore four methods to see how expert writers externalize thoughts and interact with them: laboratory comparisons of novices and experts, interviews with accomplished writers (mostly of prose fiction), biographical analysis of Jane Austen's development as a writer, and consideration of Gustave Flaubert's notes and drafts. Writers can use paper to extend their thinking, and to create frameworks of cues that enable readers of a story to construct mental models that they may enter empathetically.

We present an approach for clique and quasi-clique computations in very large multi-digraphs. We discuss graph decomposition schemes used to break up the problem into several pieces of manageable dimensions. A semiexternal greedy randomized adaptive search procedure (GRASP) for finding approximate solutions to the maximum clique problem and maximum quasiclique problem in very large sparse graphs is presented. We experiment with this heuristic on real data sets collected in the telecommunications industry. These graphs contain on the order of millions of vertices and edges.

We show how to uniformly distribute data at random (not to be confounded with permutation routing) in two settings that are able to deal with massive data: coarse grained parallelism and external memory. In contrast to previously known work for parallel setups, our method is able to fulfill the three criteria of uniformity, work-optimality and balance among the processors simultaneously. To guarantee the uniformity we investigate the matrix of communication requests between the processors. We show that its distribution is a generalization of the multivariate hypergeometric distribution and we give algorithms to sample it efficiently in the two settings.

An effective way to process a graph that does not fit in RAM is to build a hierarchical partition of its set of vertices. This hierarchy induces a partition of the graph edge set. We use this partition to produce a macro view of the graph. A screen embedding of this macro view is a Graph Sketch. We describe the use of Rectangular FishEye Views to provide drill-down navigation of graph sketches at different levels of detail including the graph edges data. A higher level of detail of a sketch focus area is obtained by distorting the lower detail context. Alternative visual representations can be used at different sketch hierarchy levels. We provide two sketch screen embeddings. One is tree-map based and the other is obtained by a special sequence of graph edge contractions. We demonstrate the application of our current Unix/Windows prototype to telecommunication graphs with edge sets ranging from 100 million to 1 billion edges(Giga-Graphs). To our knowledge this is the first time that focus within context techniques have been used successfully for the navigation of external memory graphs.

The need and use of large scale distributed storage has rapidly increased in last few years. Organizations need Terabytes of storage for their operational data and backups. Large storage systems are the ultimate solution, but they are very expensive and require higher degree of skills for their operation and maintenance e.g. Storage Area Network (SAN). We propose "A Highly Scalable and Efficient Distributed File Storage System" that is reliable, inexpensive and easy to maintain. Our system is based on peer-to-peer network architecture. To ensure the reliability of the system we use a technique of erasure codes known as Luby Transform (LT). The System is designed for deployment in Local Area Networks (LAN) but with minimal changes it can be extended for Wide Area Networks (WAN) and Internet.

In web data, telecommunications traffic and in epidemiological studies, dense subgraphs correspond to subsets of subjects (ie users, patients) that share a collection of attributes values (ie ac-cessed web pages, email-calling patterns or disease diagnostic pro-files). Visual and ...

An implementation of an On Chip Memory (OCM) based Dual Data Rate external memory controller (OCM2DDR) for Virtex II Pro is described. The proposed OCM2DDR controller comprises Data Side OCM (DSOCM) bus interface module, read and write control logic, halt read module and Xilinx DDR controller IP core. The presented design supports 16MB of external DDR memory and 32 to 64 bits data conversion for single read and write operations. Our implementation uses 1063 slices of Virtex2Pro FPGA and runs at 100 MHz. The major bene ts of the proposed design are high bandwidth to external memory with reduced and more predictable access times compared to the Xilinx PLB DDR controller implementation. More specially, our read and write accesses are 2,44 and 4,25 times faster, than the PLB based solution respectively.

A suffix tree is a fundamental data structure for string searching algorithms. Unfortunately, when it comes to the use of suffix trees in real-life applications, the current methods for constructing suffix trees do not scale for large inputs. All the existing practical algorithms perform random access to the input string, thus requiring that the input be small enough to be kept in main memory.

We describe techniques that are useful for the detection of dense subgraphs (quasi-cliques) in massive sparse graphs whose vertex set, but not the edge set, fits in RAM. The algorithms rely on efficient semi-external memory algorithms used to preprocess the input and on greedy randomized adaptive search procedures (GRASP) to extract the dense subgraphs. A software platform was put together allowing graphs with hundreds of millions of nodes to be processed. Computational results illustrate the effectiveness of the proposed methods.

This paper reports the results of an extensive user study that examined information seekers' use of their memory and externally recorded search histories in searching for and using information in the legal domain. The ultimate goal of the research is to design search-historybased user interface tools to support information seeking. The results reported here focus on the management of complex information-seeking tasks. Computers can automatically record human-computer interaction events, allow the user to manipulate this information, and provide it back to the searcher through the user interface. In order to understand how this information can best support information seekers, the role of users' internal and external memory processes was examined using qualitative research methods (observations, interviews, and participatory user interface design sessions). The data collected were analyzed to identify potential task areas where search histories can support information seeking and use. The results show that many information-seeking tasks can take advantage of automatically and manually recorded history information. One of these areas is described in this paper: the management of complex tasks. Results of the study from the legal user group presented evidence of the utility of search histories and history-based interface tools.

This work presents an external memory approach to extract the maximal repeats from whole genome sequences with the statistics of these repeats across classes, where the definition of a class is determined from the statistics to be computed. A heuristic method consisting of a bucket-sort-like approach and the Chinese term extraction approach is adopted. The bucket-sorting method is used to

We present an external-memory algorithm to compute a well-separated pair decomposition (WSPD) of a given point set S in ℝd in O(sort(N)) I/Os, where N is the number of points in S and sort(N) denotes the I/O-complexity of sorting N items. (Throughout this paper we assume that the dimension d is fixed.) As applications of the WSPD, we show how to compute a linear-size t-spanner for S within the same I/O-bound and how to solve the K-nearest-neighbour and K-closest-pair problems in O(sort(KN)) and O(sort(N+K)) I/Os, respectively.

In this paper, we propose a novel external-memory algorithm to support view-dependent simplification for datasets much larger than main memory. In the preprocessing phase, we use a new spanned sub-meshes simplification technique to build view-dependence trees I/O-efficiently, which preserves the correct edge collapsing order and thus assures the run-time image quality. We further process the resulting view-dependence trees to build the meta-node trees, which can facilitate the run-time level-of-detail rendering and is kept in disk. During run-time navigation, we keep in main memory only the portions of the meta-node trees that are necessary to render the current level of details, plus some prefetched portions that are likely to be needed in the near future. The prefetching prediction takes advantage of the nature of the run-time traversal of the meta-node trees, and is both simple and accurate. We also employ the implicit dependencies for preventing incorrect foldovers, as well as main-memory buffer management and parallel processes scheme to separate the disk accesses from the navigation operations, all in an integrated manner. The experiments show that our approach scales well with respect to the main memory size available, with encouraging preprocessing and run-time rendering speeds and without sacrificing the image quality.

We consider the read/write streams model, an extension of the standard data stream model in which an algorithm can create and manipulate multiple read/write streams in addition to its input data stream. Like the data stream model, the most important parameter for this model is the amount of internal memory used by such an algorithm. The other key parameters are

Wireless Sensor Network (WSN) in combination with image sensors opens plethora of opportunities in the wildlife tracking. It provides a glimpse into previously unseen, remote and inaccessible world of some of the most endangered species on earth. tigerCENSE 1 is such an attempt to put sensor network technology in conserving one of the rarest and most elusive big cat species. The node, triggered by the Passive Infrared (PIR) sensor, captures the image of tiger using a CMOS image sensor and stores it in an external memory chip. To avoid any disturbance to animal, the node uses an Infrared (IR) flash, instead of white flash, to illuminate the target at night. The stored images get transferred to the base station via radio transceiver. This is transferred to the database server through Internet links for analysis by wildlife researchers. A solar energy harvesting system for recharging node's batteries is being added to avoid frequent human visit to change the batteries, making it highly non-intrusive system.

We develop I/O-efficient algorithms for diameter and all-pairs shortest-paths (APSP). For general undirected graphs G(V,E) with non-negative edge weights and E/V = o(B/ log V) our approaches are the first to achieve o(V 2) I/Os. We also show that for unweighted undirected graphs, APSP can be solved with just $O(V \cdot \textrm{sort}(E))$ I/Os. Both our weighted and unweighted approaches require O(V 2) space. For diameter computations we provide I/O-space tradeoffs. Finally, we provide improved results for both diameter and APSP computation on directed planar graphs.

Interfacing in Microprocessor-based Systems with an Advanced Physical Addressing. Mountassar Maamoun1, Boualem Laichi2, Abdelhalim Benbelkacem3, Daoud Berkani4 ... This solution consists in creating a new bus, made up of a data bus, an address bus and a control bus. ...

This paper reports the results of an extensive user study that examined information seekers' use of their memory and externally recorded search histories in searching for and using information in the legal domain. The ultimate goal of the research is to design search-history-based ...

We study the flow of water on fat terrains, that is, triangulated terrains where the minimum angle of any triangle is bounded from below by a positive constant. We give improved bounds for the worst-case complexity of river networks on fat terrains, and show how to compute the river network and other flow-related structures i/o-efficiently.

Home and office network gateways often employ a cost-effective embedded network processor to handle their network services. Such network gateways have received strong demand for applications dealing with intrusion detection, keyword blocking, antivirus and antispam. Accordingly, we were motivated to propose an appropriate fast scalable automaton-matching (FSAM) hardware to accelerate the embedded network processors. Although automaton matching algorithms are robust with deterministic matching time, there is still plenty of room for improving their average-case performance. FSAM employs novel prehash and root-index techniques to accelerate the matching for the nonroot states and the root state, respectively, in automation based hardware. The prehash approach uses some hashing functions to pretest the input substring for the nonroot states while the root-index approach handles multiple bytes in one single matching for the root state. Also, FSAM is applied in a prevalent automaton algo...

We present results related to satisfying shortest path queries on a planar graph stored in external memory. Let N denote the number of vertices in the graph and sort(N) denote the number of input/output (I/O) operations required to sort an array of length N: (1)We describe a blocking for rooted trees to support bottom-up traversals of these trees in O(K/B)

Skyline queries return the set of non-dominated tuples, where a tuple is dominated if there exists another with better values on all attributes. In the past few years the problem has been studied extensively, and a great number of external memory algorithms have been proposed. We thoroughly study the most important scan-based methods, which perform a number of passes over the database in order to extract the skyline. Although these algorithms are specifically designed to operate in external memory, there are many implementation details which are neglected, as well as several design choices resulting in different favors for these basic methods. We perform an extensive experimental evaluation using real and synthetic data. We conclude that specific design choices can have a significant impact on performance. We also demonstrate that, contrary to common belief, simpler skyline algorithm can be much faster than methods based on pre-processing

We have developed the University of Washington Image Computing Library (UWICL), the high-performance image processing library for a next-generation mediaprocessor currently under development, named the Media Accelerated Processor (MAP1000). The primary goal of this library is to provide the algorithm developers and application programmers with a flexible and efficient library of core image computing functions. The UWICL is organized as a set of three hierarchical layers. Each function in this multilayered framework consists of an application module, a function module, and a tight-loop module. The MAP has an intelligent DMA controller called the Data Streamer that allows efficient data flow management. In cache-based architectures, streaming image data from the external memory generates many costly data-cache misses, in many cases leading to a severe performance bottleneck. The MAP's Data Streamer is designed to address this problem. To reduce the number of the data-cache misses further, a ping-ponging data flow scheme is employed in UWICL functions, i.e., while execution units are processing a block of data currently in the data cache, the Data Streamer brings the next data block to the data cache before it is actually needed. We compare the performance of key imaging functions on the MAP and the Texas Instruments TMS320C80, one of the most powerful mediaprocessors currently available. Typically, a MAP function is 1.5 to 6.6 times faster than the corresponding TMS320C80 implementation. Also, we demonstrate the advantages of the UWICL's multilayered library organization over the single-layered approach with an example in implementing the Canny's edge detector. The multilayered implementation of this algorithm outperforms the single- layered version by 26%.

Writing is analyzed as thinking that uses paper or other media to externalize and manipulate symbolic expressions. Mental operations of natural language can occur somewhat independently, and they communicate well with language that has been written, but for skilled writing these operations need elaborate installation in the mind. We explore four methods to see how expert writers externalize thoughts and interact with them: laboratory comparisons of novices and experts, interviews with accomplished writers (mostly of prose fiction), biographical analysis of Jane Austen's development as a writer, and consideration of Gustave Flaubert's notes and drafts. Writers can use paper to extend their thinking, and to create frameworks of cues that enable readers of a story to construct mental models that they may enter empathetically.

We describe an efficient, high-level abstraction, multi-port memory-control unit (MCU) capable of providing data at maximum throughput. This MCU has been developed to take full advantage of FPGA parallelism. Multiple parallel processing entities are possible in modern FPGA devices, but this parallelism is lost when they try to access external memories. To address the problem of multiple entities accessing shared

Computer processing speeds are increasing rapidly due to the evolution of faster chips parallel processing of data and more e cient software Users today have access to an unprecedented amount of high quality high resolution data through various technologies This is resulting in a growing demand for higher performance input and output mechanisms in order to pass huge data sets from the external memory EM or disk system through the relatively small main memory of the computer and back again In recent years research into external memory algorithms has been growing to keep pace with the demand for innovation in this area EM algorithms for individual problems have been developed but few general purpose EM tools have been designed A fundamental tool is the bu er tree an external version of the a b tree It can be used to satisfy a number of EM requirements such as sorting priority queues range searching etc in a straightforward and I O optimal manner In this paper we describe an implementa...

In this paper we present an efficient FPGA implementation of the 'Set Partitioning in Hierarchical Trees' (SPIHT) algorithm of Said and Pearlman [1] in combination with an arithmetic coder. The FPGA implementation is applied within a partitioned approach for wavelet-based lossy image compression [2]. The basic SPIHT algorithm uses dynamic data structures that make a hardware realization difficult. We illustrate in detail how these dynamic data structures can be implemented in the FPGA without the use of external memory. We present a hardware realization which can be run with a frequency of 40 MHz in a Xilinx XC4000 device. The design requires 23% less internal memory as the recently published algorithm 'SPIHT Image Compression without Lists' of Wheeler and Pearlman [5].

We present the interpolation search B-tree (ISB-tree), a new cache-aware indexing scheme that supports update operations (insertions and deletions) in O(1) worst-case block transfers and search operations in O(logBlogn) expected block transfers, where B represents the disk block size and n denotes the number of stored elements. The expected search bound holds with high probability for a large class of

The performance and cost of an adjustable speed drive (ASD) depends largely on the PWM schemes and the implementation technique. In this paper, a new PWM implementation technique is proposed, where, an external ROM based circuit generates real time PWM patterns for the inverter on carrier cycle basis. Due to carrier cycle basis implementation, the scheme needs lesser external memory. The computational burden on the processor reduces much in the proposed scheme due to the use of Look-Up-Table (LUT), having little interaction with the processor and hence a low cost, low speed processor is capable of generating high-resolution PWM patterns using advanced PWM strategies. The proposed scheme offers wide range operation of an ASD with high resolution PWM implementable with a moderate speed processor, providing high performance at low cost.

String matching plays a central role in packet inspection applications such as intrusion detection, anti-virus, anti-spam and Web filtering. Since they are computation and memory intensive, software matching algorithms are insufficient to meet the high-speed performance. Thus, offloading packet inspection to a dedicated hardware seems inevitable. This paper presents a scalable automaton matching (SAM) coprocessor that uses Aho-Corasick (AC) algorithm with two parallel acceleration techniques, root-indexing and pre-hashing. The root-indexing can match multiple bytes in one single matching, and the pre-hashing can be used to avoid bitmap AC matching which is a cycle-consuming operation. In the platform-based SoC implementation of the Xilinx ML310 FPGA, the proposed hardware architecture can achieve almost 10.7 Gbps and support over 10,000 patterns for virus, which is the largest pattern set from among the existing works. On the average, the performance of SAM is 7.65 times faster than...

In this paper, we propose a new structure of FPGA based on MRAM technology; we name it MFPGA (Magnetic FPGA). FPGA i based on SRAM technology has been developed in the last years, because of its high speed and near limitless number of reprogramming, however SRAM is volatile thereby the configuration information and the intermediate data will be lost when power is turned off. By using MTJs (Magnetic Tunnel Junction) as the storage elements of FPGA, 1 the position of MTJs we can realize the non-volatility of FPGA, and then we will not need the external memory. In our simulation, the start-up time of circuit

The efficient simulation of spiking neural networks (SNN) remains as an open challenge. Current SNN computing engines are still far away of being able to efficiently simulate systems of millions of neurons. This contribution describes a computing scheme that takes full advantage of the massive parallel processing resources available at FPGA devices. The computing engine adopts an event-driven simulation scheme and an efficient next-event-to-go searching method to achieve high performance. We have designed a pipelined datapath in order to compute several events in parallel avoiding idle computing resources. The system is able to compute approximately 2.5 million spikes per second. The whole computing machine is composed only by an FPGA device and five external memory SRAM chips. Therefore the presented approach is of high interest for simulation experiments that require embedded simulation engines (for instance in robotic experiments with autonomous agents).

We present results related to satisfying shortest path queries on a planar graph stored in external memory. Let N denote the number of vertices in the graph and sort(N ) denote the number of input=output (I=O) operations required to sort an array of length N :

This paper describes a system able to acquire, process and eliminate noise in continuous streams of data in real-time. The signal processing algorithms were based on the discrete wavelet transform and employ a new approach to deal with border problems, allowing to process the data continuously. The system was implemented using a DSP coupled to a digitizer through its external memory bus to guarantee deterministic behavior while maintaining some degree of flexibility in its configuration. The achieved performance and potential applications are discussed at the end of the text.

We have developed the University of Washington Image Computing Library (UWICL), the high-performance image processing library for a next-generation mediaprocessor currently under development, named the Media Accelerated Processor (MAP1000). The primary goal of this library is to provide the algorithm developers and application programmers with a flexible and efficient library of core image computing functions. The UWICL is organized as a set of three hierarchical layers. Each function in this multilayered framework consists of an application module, a function module, and a tight-loop module. The MAP has an intelligent DMA controller called the Data Streamer that allows efficient data flow management. In cache-based architectures, streaming image data from the external memory generates many costly data-cache misses, in many cases leading to a severe performance bottleneck. The MAP's Data Streamer is designed to address this problem. To reduce the number of the data-cache misses further, a ping-ponging data flow scheme is employed in UWICL functions, i.e., while execution units are processing a block of data currently in the data cache, the Data Streamer brings the next data block to the data cache before it is actually needed. We compare the performance of key imaging functions on the MAP and the Texas Instruments TMS320C80, one of the most powerful mediaprocessors currently available. Typically, a MAP function is 1.5 to 6.6 times faster than the corresponding TMS320C80 implementation. Also, we demonstrate the advantages of the UWICL's multilayered library organization over the single-layered approach with an example in implementing the Canny's edge detector. The multilayered implementation of this algorithm outperforms the single- layered version by 26%.

We introduce a new variant of the popular Burrows-Wheeler transform (BWT) called Geometric Burrows-Wheeler Transform (GBWT). Unlike BWT, which merely permutes the text, GBWT converts the text into a set of points in 2-dimensional geometry. Using this transform, we can answer to many open questions in compressed text indexing: (1) Can compressed data structures be designed in external memory with similar performance as the uncompressed counterparts? (2) Can compressed data structures be designed for position restricted pattern matching [16]? We also introduce a reverse transform, called Points2Text, which converts a set of points into text. This transform allows us to derive the first known lower bounds in compressed text indexing. We show strong equivalence between data structural problems in geometric range searching and text pattern matching. This provides a way to derive new results in compressed text indexing by translating the results from range searching.

A suffix tree is a fundamental data structure for string searching algorithms. Unfortunately, when it comes to the use of suffix trees in real-life applications, the current methods for constructing suffix trees do not scale for large inputs. All the existing practical algorithms perform random access to the input string, thus requiring that the input be small enough to be kept in main memory. We are the first to present an algorithm which is able to construct suffix trees for input sequences significantly larger than the size of the available main memory. As a proof of concept, we show that our method allows to build the suffix tree for 12GB of real DNA sequences in 26 hours on a single machine with 2GB of RAM. This input is four times the size of the Human Genome, and the construction of suffix trees for inputs of such magnitude was never reported before.

Externalism holds that the individuation of mental content depends on factors external to the subject. This doctrine appears to undermine both the claim that there is a priori self-knowledge, and the view that individuals have privileged access to their thoughts. Tyler Burge’s influential inclusion theory of self-knowledge purports to reconcile externalism with authoritative self-knowledge. I first consider Paul Boghossian’s claim that the inclusion theory is internally inconsistent. I reject one line of response to this charge, but I endorse another. I next suggest, however, that the inclusion theory has little explanatory value.

Peer-to-peer networks have recently gained importance as environments for digital libraries. In this paper, we describe Pepper, a peer-to-peer network for supporting two important access modes, searching and browsing. Pepper seamlessly combines hierarchical networks with service-oriented architectures. Hierarchical networks improve efficiency by partitioning peers into high-end hubs which are responsible for routing the query, and leaf peers (typically running on desktop PCs or laptops) which host documents. Service-oriented architectures allow for loosely coupled systems, improve extensibility and reusability, and enable integration of the two access paradigms by combination of multiple services. For example, distributed searching and browsing both use the same statistical metadata, and thus the same storage service, for solving their tasks.

An implicit data structure for the dictionary problem maintains n data values in the first n locations of an array in such a way that it efficiently supports the operations insert, delete and search. No information other than that in Oð1Þ memory cells and in the input data is to be retained; and the only operations performed on the data values (other than reads and writes) are comparisons. This paper describes the implicit B-tree, a new data structure supporting these operations in Oðlog B nÞ block transfers like in regular B-trees, under the realistic assumption that a block stores B ¼ Oðlog nÞ keys, so that reporting r consecutive keys in sorted order has a cost of Oðlog B n þ r=BÞ block transfers. En route a number of space efficient techniques for handling segments of a large array in a memory hierarchy are developed. Being implicit, the proposed data structure occupies exactly Jn=Bn blocks of memory after each update, where n is the number of keys after each update and B is the number of keys contained in a memory block. In main memory, the time complexity of the operations is Oðlog 2 n=log log nÞ; disproving a conjecture of the mid 1980s.

This paper deals with the topic of learning through neuroevolutionary algorithms in non-stationary settings. This kind of algorithms that evolve the parameters and/or the topology of a population of Artificial Neural Networks have provided successful results in optimization problems in stationary settings. Their application to non-stationary problems, that is, problems that involve changes in the objective function, still requires more research. In this paper we address the problem through the integration of implicit, internal or genotypic, memory structures and external explicit memories in an algorithm called Promoter Based Genetic Algorithm with External Memory (PBGA-EM). The capabilities introduced in a simple genetic algorithm by these two elements are shown on different tests where the objective function of a problem is changed in an unpredictable manner.