Recent developments indicate that malware programs present a significant risk in the security and privacy of cloud systems. Existing research in malware detection encounters numerous significant challenges due to the constantly changing and advanced characteristics of malware. Malware detection systems frequently experience high rates of false positives and false negatives, where legitimate applications are incorrectly identified as malware or actual malware remains undetected, which results in operational inefficiencies. Traditional signature-based approaches struggle in recognizing new or modified malware. Additionally, sophisticated malware types, such as file less malware, ransom ware, and rootkits pose detection challenges as they integrate deeply into systems or alter their behaviour to evade detection. These challenges highlight the urgent need for ongoing advancements in this field. Existing methods of malware detection that rely on signatures have been found to be both inefficient and slow in the context of cloud environments. Also, Existing studies have focused on detecting malware by analysing input API calls. But, these models have encountered challenges such as limited accuracy and difficulties in effectively classifying malware types. In contrast, Deep Learning (DL) have shown success by analysing malware behaviour through API calls, which yields encouraging results. Additionally, the data produced by API calls necessitates more computational resources for training. To address these challenges, a new deep learning-based malware detection approach utilizes 2D grayscale images derived from API calls, along with an effective tuning strategy has been proposed. Initially, data are collected from cloud malware dataset. Then, API calls are converted into 2D gray scale images in order to construct gray scale image dataset. After getting the gray scale image, pre-processing is performed to reduce high level noise and to enhance the quality of image by weighted mean filter and anisotropic filter, which helps to improve the performance in classification. Next, these images are then passed into the feature extraction stage to extract sufficient features with an effective integrated densely connected squeeze MobileNet v2 (Ef-DeSMob2), which reduces the dimensionality issue and increase the computational complexity. Then, the collected features are passed into the classification phase to detect normal and malware classes from the samples using sparse attention with residual pyramidal depth wise separable convolutional neural networks (SA:ResPyDSC), which focus to enhance the security and reliability of the model. Finally, the hyper parameters in the classifier model like weights, bias are properly fine-tuned by utilizing hybrid white shark beluga optimization algorithm (Hy-WBeOp). The experimental findings illustrate that the proposed method attains accuracy of 98.06%, precision of 97.99%, recall of 97.05%, f1-score of 96.08% and error metrics like MSE AT 0.08, RMSE of 0.27 and MAE of 0.21, which shows that the proposed method helps to classify the malware classes accurately with less error rates. The proposed approach outperforms with the existing techniques because of its great efficiency. Overall, this approach establish a strong malware detection system classification and enhance the reliability and effectiveness of protection against malicious attacks. [ABSTRACT FROM AUTHOR]