Search

Distributed machine learning (DML) is an increasingly important workload. In a DML job, each communication stage can comprise a coflow, and there are dependencies among its coflows. Thus, efficient coflow scheduling becomes critical for DML jobs. However, the majority of existing solutions focus on scheduling single-stage coflows with no dependencies. While there are a few studies schedule dependent coflows of multi-stage jobs, they suffer from either practical or theoretical issues. In this paper, we study how to schedule dependent coflows of multiple DML jobs to minimize the total job completion time (JCT) in a shared cluster. To solve this problem without any prior knowledge of job information, we present an online coflow-aware optimization framework called Parrot. The core idea in Parrot is to infer the job with the shortest remaining processing time (SRPT) each time and dynamically control the inferred job's bandwidth based on how confident it is an SRPT job while being mindful of not starving any other job. We have proved that Parrot algorithm has an approximation ratio of $O(M)$ , where M is the number of jobs. The results from large-scale trace-driven simulations further demonstrate that our Parrot can reduce the total JCT by up to 58.4%, compared to the state-of-the-art solution Aalo.

Modern resource management frameworks guarantee low tail latency for long-running services using the resource over-provisioning method, resulting in serious waste of resources and increasing the service costs greatly. To reduce the over-provisioning cost, we present HRSE, a hybrid resource scaling engine that enables much more efficient resource provisioning for both periodic and non-periodic workloads of long-running services while guaranteeing the tail latency Service Level Objective (SLO). HRSE employs a convolution-based time series analysis to identify periodic patterns in workloads. If periodic patterns are discovered, HRSE estimates the just-right amount of resources based on the periodic features through a top-K based collaborative filtering approach. Otherwise, it leverages wavelet-clustering to capture the short-term patterns in non-periodic workloads and predict the resource demands for the near future. To further enforce the tail latency SLO, HRSE uses an online reprovisioning mechanism that dynamically adjusts the resources to mitigate the performance uncertainty due to workload burstinesses. We fully implement HRSE on top of Docker and conduct extensive experiments using traces from production systems. Testbed experiments show that HRSE is able to increase the average resource utilization to 43% and 45% for periodic and non-periodic workloads respectively while guaranteeing the same tail latency objective.

The cloud resource provision policy of a content provider in the presence of competitors on globally distributed cloud platforms plays a significant role in maximizing its profit. However, developing an optimal resource provision policy is quite challenging, due to the difficult to capture the competition relationship between two competitive CPs and to obtain the budget of the competitors which is usually kept private. To solve this problem, in this paper, we propose a learning-driven cloud resource provision policy for a CP with competitors. We formulate the competition between the CPs as a lottery Colonel Blotto game in which the payoff of each region is positively related to the resource advantage achieved by the CP, formulate the budget allocation problem as a Markov decision process, and obtain the sub-optimal resource provision policy by reinforcement learning and deep reinforcement learning-based algorithms. We also prove the convergence of the sub-optimal solution. Finally, we validate our proposed method using real-world CPs statistics. The results show that the budget information is critical for a CP to make policy decisions, and it is better for CPs with smaller budget to focus their budget resources in regions with higher values.

This letter studies the virtualized network function (VNF) placement problem for edge systems, where the edge nodes can provide services that are originally deployed in cloud datacenters. As related to the end-users, edge supports low latency when the energy consumption of edge nodes and end users are the main concerns. Therefore, we formulate this problem with two performance metrics: latency and energy consumption. We present a joint VNF Placement optimization problem at the Edge, and propose the method of VPE. The numerical results show that our proposed VPE guarantees the energy consumption cost without damaging the performance on latency.

Latency to end-users and regulatory requirements push cloud providers to operate many datacenters all around the globe to host their cloud services. An emerging problem under such geo-distributed architecture is to assign each user request to an appropriate datacenter to benefit both cloud providers (e.g., low bandwidth cost) and end-users (e.g., low latency)—known as request allocation. However, prior request allocation solutions have significant limitations: they either focus only on optimizing the benefits for one entity (e.g., providers or users), or ignore some practical yet indispensable factors (e.g., heterogeneous latency requirements of different users and diverse per unit bandwidth cost among different datacenters) when optimizing benefits for both entities. In this paper, we study the problem of minimizing the total bandwidth cost for cloud service providers while guaranteeing the latency requirement for end-users. Specifically, we formulate an integer programming with consideration of the diversities in both the delay of requests and per unit bandwidth cost of datacenters. To efficiently and practically solve this problem, we first relax the integer programming into a continuous convex optimization and then take the advantages of random sampling to enforce the solution to be a feasible one for the original integer programming. We have conducted rigorous theoretical analysis to prove that our algorithm can provide a considerable good competitive ratio. Extensive simulations demonstrate that our proposed algorithm can reduce the total bandwidth cost by 30% while guaranteeing the latency requirements of all requests, as compared to conventional methods.

Interference among distributed cloud applications can be classified into three types: full, partial and zero. While prior research merely focused on full interference, the partial interference that occurs at parts of applications is far more common yet still lacks in-depth study. Serverless computing that structures applications into small-sized, short-lived functions further exacerbate partial interference. We characterize the features of partial interference in serverless as exhibiting high volatility, spatial-temporal variation, and propagation. Given these observations, we propose an incremental learning predictor, named Gsight, which can achieve high precision by harnessing the spatial-temporal overlap codes and profiles of functions via an end-to-end call path. Experimental results show that Gsight can achieve an average error of 1.71%. Its convergence speed is at least 3X faster than that in a serverful system. A scheduling case study shows that the proposed method can improve function density by ≥ 18.79% while guaranteeing the quality of service (QoS).

With the multi-tier pricing scheme provided by most of the cloud service providers (CSPs), the cloud users typically select a high enough transmission service level to ensure the quality of service (QoS), due to the severe penalty of missing the transmission deadline. This leads to the so-called over-provisioning problem, which increases the transmission cost of the cloud user. Given the fact that cloud users may not be aware of their traffic demand before accessing the network, the over-provisioning problem becomes more serious. In this paper, we investigate how to reduce the transmission cost from the perspective of cloud users, especially when they are not aware of their traffic demand before the transmission deadline. The key idea is to split a long-term transmission request into several short ones. By selecting the most suitable transmission service level for each short-term request, a cost-efficient inter-datacenter transmission service level selection framework is obtained. We further formulate the transmission service level selection problem as a linear programming problem and resolve it in an on-line style with Lyapunov optimization. We evaluate the proposed approach with real traffic data. The experimental results show that our method can reduce the transmission cost by up to 65.04%.

In mobile edge-cloud networks, multiple edge nodes form a mesh network to cooperate with each other. To maximize the benefit of resource-limited edge nodes, the content providers jointly optimize the content caching and recommendation decisions. However, the cooperation between edge nodes complicates both the content caching and recommendation decisions. To solve this problem, in this paper, we propose an efficient joint cooperative content caching and recommendation scheme in edge-cloud networks. Specifically, we formulate the joint cooperative content caching and recommendation problem as an integer-linear programming problem to minimize the average download delay, with controllable user preference distortion tolerance. We propose an efficient heuristic algorithm to solve the formulated problem due to its NP-hardness. We evaluate the performance of the proposed scheme with the MovieLens dataset. The simulation results demonstrate that the proposed scheme can decrease the average download latency by up to 37% and improve average cache hit rate by up to 24%, as compared with state-of-the-art solutions.

In the era of cloud computing, Content Providers (CPs) rent various resources, e.g., computing, storage, and network bandwidth, from Cloud Service Provider (CSP) to deploy their globally distributed services. Since multiple CPs may provide the same services and compete with each other to maximize their revenue, the business policies they adopt become very important, especially when the budget of each CP is limited and different. However, it is difficult to capture the competition characteristics and to find the optimal business policies of the CPs. In this paper, we investigate the business policies of competitive CPs, where each CP aims to maximize its revenue by optimally allocating its limited budget to resources including storage and bandwidth. First, we formulate the competition between the CPs as a Colonel Blotto game with elastic payoffs, in which the payoff of each region varies with the budget allocated to it. Then, we prove that its Nash Equilibriums are the optimal budget allocation policies for the CPs. We validate our method with real-world CPs and CSP's statistics. The results show that storage plays a key role in maximizing the revenue of the CP. It is also suggested that the CPs with less budget should concentrate their budget resource on a few specific regions.

Nowadays, service function chaining (SFC) becomes more and more widespread and profound to implement flexible and economical virtual network infrastructures for the Internet of Things (IoT). With the benefits of SFC, the IoT service providers can steer massive traffic through a sequence of heterogeneous virtual network function instances based on their business logic. SFC is viewed as an attractive solution for building virtualized IoT-dedicated network. However, the SFC orchestration in IoT is still a challenge problem. Existing work usually focuses on the performance guarantee while ignoring the issue of resource idleness. To meet the sharp increase in IoT traffic amounts and the diversification of IoT traffic requirements, it is necessary to implement the performance and resource aware SFC orchestration system. Motivated by this, we propose a novel linear programming model and an effective approximation optimization algorithm for SFC orchestration, in order to achieve performance guarantee while avoiding resource idleness. Based on the proposed model and algorithm, a new prototype system named performance and resource aware orchestration system of SFC for IoT (PRSFC-IoT) is built upon OpenStack for online SFC orchestration. A large number of simulation experiments show that the PRSFC-IoT outperforms existing solutions for SFC orchestration in IoT.

Heterogeneous Internet of Things (HetIoT) is an emerging research field that has strong potential to transform both our understanding of fundamental computer science principles and our future living. HetIoT is being employed in increasing number of areas, such as smart home, smart city, intelligent transportation, environmental monitoring, security systems, and advanced manufacturing. Therefore, relaying on strong application fields, HetIoT will be filled in our life and provide a variety of convenient services for our future. The network architectures of IoT are intrinsically heterogeneous, including wireless sensor network, wireless fidelity network, wireless mesh network, mobile communication network, and vehicular network. In each network unit, smart devices utilize appropriate communication methods to integrate digital information and physical objects, which provide users with new exciting applications and services. However, the complexity of application requirements, the heterogeneity of network architectures and communication technologies impose many challenges in developing robust HetIoT applications. This paper proposes a four-layer HetIoT architecture consisting of sensing, networking, cloud computing, and applications. Then, the state of the art in HetIoT research and applications have been discussed. This paper also suggests several potential solutions to address the challenges facing future HetIoT, including self-organizing, big data transmission, privacy protection, data integration and processing in large-scale HetIoT.

In this letter, we investigate an energy efficiency with performance guaranteed problem in mobile-edge computing. The mobile users desire lower energy consumption with better performance of tasks, for that we propose an energy minimizing optimization problem for mobile-edge cloud computing. We apply KKT conditions to solve it, and also present a request offloading scheme for this issue. In particular, the offloading scheme is determined by energy consumption and bandwidth capacity at each time slot. Numerical results demonstrate that our proposed offloading scheme outperforms local computing and entirely offloading method on energy consumption and performance on delay.

In large-scale Internet applications running on geographically distributed datacenters, such as video streaming, it is important to efficiently allocate requests among datacenters. To the best of our knowledge, existing approaches, however, either solely focus on minimizing total cost for provider, or guaranteeing QoS for end-users. In this paper, we apply the software defined network (SDN) controller to enable the central control of the entire network, and propose a joint optimization model to consider high bandwidth utilization for provider and low delay for users. We present the Nash bargaining solution (NBS) based method to model both requirements of provider’s high bandwidth utilization and end-users’ low delay. Specifically, we formulate the design of request allocation under those requirements as an optimization problem, which is NP-hard. To solve such hard optimization problem, we develop an efficient algorithm blending the advantages of Logarithmic Smoothing technique and the auxiliary variable method. According to the theoretical analysis, we verify the existence and uniqueness of our solution and the convergence of our algorithm. We conduct a large amount of experiments based on real-world workload traces and demonstrate the efficiency of our algorithm compared to both greedy and locality algorithms.

Resource sharing, multi-tenant interference and bursty workloads in cloud computing lead to high tail-latency that severely affects user quality of experience (QoE), where response latency is a critical factor. A lot of research efforts are dedicated to reducing high tail-latency and improving user QoE, such as software-defined cloud computing (SDC). However, the traditional availability analysis of cloud computing captures the pure failure-repair behavior with user QoE ignored. In this paper, we propose a conceptual framework, experience availability, to properly assess the effectiveness of SDC while taking into account both availability and response latency simultaneously. We review the related work on availability models and methods of cloud systems, and discuss open problems for evaluating experience availability in SDC. We also show some of our preliminary results to demonstrate the feasibility of our ideas.

Cloud storage systems frequently have a large user base that requires huge cloud resources. Sometimes, cloud devices become overloaded because of an imbalance in input/output (I/O) or space demand. How can data with different popularity be distributed over heterogeneous devices? The key to resolving this problem is to balance the workload of multi-dimension resources. A consistent hash-aware cloud storage system constitutes a good solution for data placement. It can achieve only 1-D balance, usually the balance of the space resource. However, it is not straightforward to obtain a balance of space, I/O, and other resources simultaneously. Many users have experienced the overloading of devices in these systems. We focus mainly on this problem in this paper. In this paper, we discuss the factors that cause the overload of devices that occurs in the hash-aware cloud. Furthermore, we design some schemes with three algorithms to facilitate the assignment of hybrid data of different size and popularity to the heterogeneous cloud. The system can reduce the probability of an overload occurring. Most systems do not easily accommodate the movement of data. However, we argue that relocating part of the necessary data is helpful. This relocation can achieve a balance of resource usage and use fewer resources, without the need for replicas. Our system can provide a better quality of service, because the imbalance in the usage of resources is reduced. We performed an evaluation using extensive simulations driven by real-world traces. We demonstrate that our system can effectively reduce the overload probability of devices in cloud storage systems.

In recent years, demands for rich multimedia services over mobile networks have been soaring at a tremendous pace. Traditional dedicated networking equipment may not be able to efficiently support the phenomenal growth of the traffic load and user demand dynamics while consuming an unnecessarily large amount of energy resources. Recently, mobile content caching, whereby popular contents are cached inside the mobile front-haul and back-haul networks so that demands for these contents from users in proximity can be easily accommodated without redundant transmissions from the remote sources, has emerged as an efficient technique for multimedia content delivery. Mobile content caching is particularly suitable for fifth generation (5G) mobile systems that are being designed to incorporate advanced cloud computing technologies and network function virtualization techniques. Therefore, in this paper, we first propose the concept of “Caching-as-a-Service” (CaaS) based on cloud-based radio access networks, and virtualized evolved packet core, which provides the capability to cache anything at anytime, anywhere in the cloud-based 5G mobile systems to satisfy user demands from any service location with high elasticity and adaptivity, and to empower third-party service providers with flexible controllability and programmability. Then, we study the potential techniques related to the virtualization of caching, and discuss the technical details of virtualization and optimization of CaaS in 5G mobile networks. Some novel schemes for CaaS are proposed to target different mobile applications and services. We also explore new opportunities and challenges for further research.

To reduce the cost of online cloud services, service providers often employ the elastic approach that allows tenants to "scale out" or "scale up" their applications at runtime. However, the traditional virtual machine-based approach cannot meet the fine-grained fluctuating demand due to its slow startup time and high reconfiguration cost. To address this challenge, we present ElaX, an online service manager that minimizes the resource provisioning cost for containerized online services while guaranteeing their tail latency requirement. ElaX designs a workload-aware resource allocation mechanism for containerized online services through the collaboration of three key components: First, Workload predictor is able to precisely predict the workload in periodic scenario, through a LSTM (Long Short-Term Memory) network; Second, Resource reservation allocates the just-right amount of resource supporting the predicted workload, using the combination of both scale-up and scale-out operations; Third, Online controller guarantees the tail latency requirement during prediction failures through a feedback-based control method, and further reduces the provision cost through resource reclamation. Our experiments on the two production workloads demonstrate that ElaX can reduce the average resource over-provisioning cost by more than 32.6% while guaranteeing the tail latency requirement.

Underlying network provides infrastructures for cloud computing in data centers. The server-centric architectures integrate network and compute, which place routing intelligence on servers. However, the existing multi-port server based architectures suffer from determined scale and large path length. In this paper, we propose FleCube, a flexibly-connected architecture on multi-port servers without using any switches. FleCube is recursively constructed on division of multiple ports in a server by means of complete graph. FleCube benefits data center networks by flexible scale and low diameter, as well as large bisection width and small bottleneck degree. Furthermore, we develop multi-path routing (MPR) to take advantage of parallel paths between any two servers. MPR adopts random forwarding to distribute traffic load and relieve network congestion. Analysis and comparisons with existing architectures show the advantages of FleCube. Evaluations under different degrees of network traffic demonstrate the merits of FleCube and the proposed routings.

In the health IoT (Internet of Things), the specialized sensor devices can be used to monitor remote health and notify the emergency information, e.g., blood pressure, heart rate, etc. These data can help the doctors to rescue the patients. In cloud-based health IoT, patients' medical/health data is managed by the cloud service providers. Secure storage and privacy preservation are indispensable for the outsourced medical/health data in cloud computing. In this paper, we study the integrity checking and sharing of outsourced private medical/health records for critical patients in public clouds (ICS). The patient can check his own medical/health data integrity and retrieve them. When a patient is in coma, some authorized entities and hospital can cooperate to share the patient's necessary medical/health data in order to rescue the patient. The paper studies the system model, security model and concrete scheme for ICS in public clouds. Based on the bilinear pairing technique, we design an efficient ICS protocol. Through security analysis and performance analysis, the proposed protocol is provably secure and efficient.

Recently a new performance metric called experience availability (EA) has been proposed to evaluate online cloud service in terms of both availability and response time. EA originates from the fact that from the prospective of quality of experience (QoE), an online cloud service is regarded as unavailable not only when it is inaccessible, but also when the tail latency is high. However, there still lacks analytic models for evaluating the EA of online services. In this paper, we propose an efficient EA-analytic model using stochastic reward net (SRN) to study the tail latency performance of online cloud services in the presence of failure-repair of the resources. Our EA-analytic model can predict the online service performance on EA, as well as support analysis on traditional availability and mean response time. We apply this model to an Apache Solr search service, and evaluate the prediction accuracy by comparing the results derived from the model to actual experimental results. It is shown that the proposed model overestimates the response time at lower percentiles and underestimates the response time at higher percentiles. Through attribution analysis, we further identify the list of factors that may affect the accuracy, and show that the 95th percentile latency prediction error can be reduced to as low as 2.45% by tuning the configurations suggested by the attribution.

There is enormous visitation pressure in most cloud systems, which is why they require so many devices and have such enormous structures. This pressure and size is accompanied by a huge cost problem. Enterprise IT departments must decide whether to build a private cloud themselves or lease space in a public cloud instead. To add to the complexity, most enterprise cloud systems are hybrids simultaneously consisting partly of a private cloud and partly of a public cloud. An enterprise needs to know the optimal mixing ratio to achieve the lowest cost. However, the optimal ratio is a dynamic value because the popularity of data changes constantly and the amount of data in both cloud types increases steadily. In this paper, we formulate maximizing the net present value (NPV) of the investment revenue as a dynamic decision-making problem and propose a model to simplify the decision-making problem. Furthermore, we provide a solution for locating data among the clouds and it can be employed again to determine how to best extend the cloud system at the lowest cost.

To deal with the severe challenge from explosively increasing mobile users' multimedia service requests in mobile networks, introducing cloud-based computing platform and content caching into radio access networks (RANs) has become an effective emerging technique. In this paper, we propose a collaborative hierarchical caching framework in cloud RANs (C-RANs), which minimizes the access delay of content delivery under certain specific network constraints to improve users' Quality of Service (QoS) as well as offload network traffic. Specifically, based on the formed hierarchical caching topology consisting of two tiers of content caching at the base stations and the cloud central unit (CCU), we decompose the formulated large-scale optimization problem into a series of simpler subproblems, and then propose the corresponding low-complexity distributed heuristic solutions as well as a content request routing scheme. Trace-based evaluation results demonstrate the effectiveness of the proposed framework.

Optimizations have been applied in many different engineering fields. Most of these applications may have similar characteristics: intensive on computing resources, time-consuming on calculation iterations, similar on computing environments and so on. This paper describes a generic cloud platform for engineering optimization by leveraging the compute resources hosted in cloud datacenters. The methodology developed was to decompose the engineering optimization processes into several interconnected sub-tasks, which were further converted and implemented as virtual applications for dynamic cloud deployment using OpenStack. The system can dynamically allocate and recycle the compute resources according to the specific engineering optimization applications. The research presented in the paper contributes on the generic engineering optimization process virtualization and cloud computing based implementation with innovative algorithms development. The system test results showed a way that potentially engineering optimization problems could be embedded into the put forward platform due to the developed large scale and intelligent cloud based optimization services. Further applications for building energy simulation and optimization, stents optimization, water distribution optimization are currently under development.

The total completion time of a task is also the major bottleneck in the big data processing applications based on parallel computation, since the computation and data are distributed on more and more nodes. Therefore, the total completion time of task is an important index to evaluate the cloud performance. The access latency between the nodes is one of the key factors affecting task completion time for cloud applications. Additionally, minimising total access time can reduce the overall bandwidth cost of running the job. This paper proposes an optimisation model focused on optimising VMs placement so as to minimise the total data access latency where the datasets have been located. According to the proposed model, our optimising VMs problem is linear programming. Therefore, we obtain the optimum solution of our model by the branch-and-bound algorithm with time complexity O(2NM). Simultaneously, we also present a greedy algorithm, which has O(NM) of time complexity, to solve our model. Finally, the simulation results show that all of the solutions of our model are superior to existing models and close to the optimal value.

This paper presents a scalable model-driven approach to quantify the availability of resources and optimal distribution of tasks over these resources, such that the average response time of tasks is minimized. To reduce the complexity of analysis and solution time, we use an integrated stochastic based approach. To achieve this, first we use clustering algorithm to group the tasks into distinct classes with similar characteristics in terms of resource and performance requirements. Second, we quantify the resource availability of cloud center among three states: active (running), idle (turned on, but not ready), and off (turned off). Third, we develop a queuing model for multiple heterogeneous multicore servers, and formulate and solve the optimal load distribution of tasks for multiple heterogeneous multicore servers in a cloud computing data centers. We derive equations that permit us to find optimal load distribution of tasks that their average response time is minimized. We obtain not only detailed assessment of cloud center performance, but also insights into equilibrium arrangement, capacity planning and power consumption to be kept under control.

Cloud computing is a new computing model, which provides various services through networks. Cloud customers can obtain resource and service on demand. There are a large number of storage services that can be used to manage the needs of customers in the cloud computing environment. The cloud storage services can let users in a convenient and reliable way store and share data from anywhere, on any device, and at any time. However, a lot of resources and energies are wasted in cloud computing environment. Especially, there is huge waste when it is a low-IO demand cloud system. We seek to save the resource and save the hardware cost at the same time. Nowadays, the important problem is how to save the resources and energies. It is hard to handle it in a high-IO demand system. But we find a way to handle it in a low-IO system. Maximising resource utilisation is the most popular approach which saves the resources and energies in cloud computing. In this paper, we propose to maximise resource utilisation of storage in the cloud environment. We assign the data which is stored in chunk devices of cloud by our data placement algorithms. Our algorithms are running on a central 'decision centre' which holds all devices' status of cloud. As the experiment has shown, minimising the disk usage can reduce the waste of the resources and energies. Finally, our experiment shows that our approach is efficient.

With the rapid growth of emerging applications like social network analysis, semantic Web analysis and bioinformatics network analysis, a variety of data to be processed continues to witness a quick increase. Effective management and analysis of large-scale data poses an interesting but critical challenge. Recently, big data has attracted a lot of attention from academia, industry as well as government. This paper introduces several big data processing technics from system and application aspects. First, from the view of cloud data management and big data processing mechanisms, we present the key issues of big data processing, including cloud computing platform, cloud architecture, cloud database and data storage scheme. Following the Map Reduce parallel processing framework, we then introduce Map Reduce optimization strategies and applications reported in the literature. Finally, we discuss the open issues and challenges, and deeply explore the research directions in the future on big data processing in cloud computing environments.

The implementation of cloud computing has attracted computing as a utility and enables penetrative applications from scientific, consumer and business domains. However, this implementation faces energy, power and associated cost concerns. In this paper, we present formulations and solutions for Green Cloud Environments (GCE) to minimize its energy consumption under new models by considering static and dynamic portions of cloud components, to reduce severe environmental impacts. To implement this objective, an in-depth knowledge of energy consumption patterns in cloud environment is necessary. We investigate energy consumption patterns. Specifically, we make use of measurable metrics based on runtime tasks to compare rationally the relation existing between energy consumption and cloud workload and computational tasks, as well as system performance. Our research results can be integrated into cloud computing systems to monitor energy consumption and support static and dynamic system level-optimization.

With the rapid growth of emerging applications like social network, semantic web, sensor networks and LBS (Location Based Service) applications, a variety of data to be processed continues to witness a quick increase. Effective management and processing of large-scale data poses an interesting but critical challenge. Recently, big data has attracted a lot of attention from academia, industry as well as government. This paper introduces several big data processing techniques from system and application aspects. First, from the view of cloud data management and big data processing mechanisms, we present the key issues of big data processing, including definition of big data, big data management platform, big data service models, distributed file system, data storage, data virtualization platform and distributed applications. Following the MapReduce parallel processing framework, we introduce some MapReduce optimization strategies reported in the literature. Finally, we discuss the open issues and challenges, and deeply explore the research directions in the future on big data processing in cloud computing environments.