Search

We argue that most of data will be stored and processed at the edge of the network in the next "big data" era. We call this phenomenon "Edge-Heavy Data", and we propose an architecture named "Krill" for it. One important characteristic of big data is its low value-density. If the data are never used, it is wasteful of sending it to the data centers (with a network cost) and storing them into expensive enterprise-grade data servers. Instead, the low value-density data will be stored near where they are generated, that is, the edge of the network. This paper discusses requirements and a possible architecture for efficiently dealing with the phenomenon of Edge-Heavy Data. We first consider a few scenarios where Edge-Heavy Data is desirable, or even inevitable, and identify its requirements. Then we propose an architecture based on the concept of Data Value Field (DVF), followed by an introduction of Jubatus, an open-source framework of online machine learning, as a first step towards the proposed architecture. [ABSTRACT FROM AUTHOR]

To develop an efficient way of generating innovation driven by big data is a central issue in academia as well as industry. Although such a demand grows rapidly, a human resource development that is a key factor for realizing it remains the rarely-discussed problem. A data scientist with sufficient knowledge in statistics and machine learning is exactly a right person who plays a crucial role in making effective use of big data. In fact, the business community, in particular, social network game company has made significant efforts to attract them. In this paper, we address a problem in developing data scientists in Japan, and give a brief introduction of interesting internship program for PhD candidate or post-doctoral researcher which the Silicon Valley technology companies initiated in summer, 2012. Its program is designed to bridge a gap between academia and a career data science. Finally, we consider how to increase the data scientist in Japan effectively, and propose to realize a Japanese version of that program as soon as possible with a financial support of Japanese government. [ABSTRACT FROM AUTHOR]