Signal detection, parameter estimation and modulation classification are widely applied to many areas and plays a very important role in civilian and military, such as bio-science, criminal psychology, communication engineering, radar system, electronic warfare and so on. In the civilian field, with the increasing number of wireless electronic devices and higher transmission data rate demand, the problem of spectrum congestion becomes more and more highlighted and urgent. In recent years, wireless industry has shown great interest in Cognitive Radio (CR) and Dynamic Spectrum Access (DSA) network, whose primary function is to use limited frequency bands to transmit own signals without any interference with other primary users. Hence, the accuracy of signal detection and parameters estimation are particularly important and can provide reliable communication performance for cognitive radio users. In the military field, electronic warfare is crucial important part in modern war, such as own signal needs to be hidden, securely transmitted and received, enemy’s signals need to be identified, located and jammed. Thus, in such a non-cooperative environment, signal detection, parameter estimation and modulation classification technologies become more and more important and challenging. In the past few decades, several signal detection methods have been proposed, such as energy-based detection, matched filter-based detection and cyclostationary feature based detection. Energy based detection is simple to implement, but poorly performing at low SNR. Although the matched filter-based detection is the optimal detector, it needs to accurately know the prior information of the detected signal. Hence, matched filter-based detection is impractical to implement in real environment, such as non-cooperative environment. Cyclostationary feature based signal detection has high computational complexity, but it can be used for high-precision signal detection in low SNR environments. In recent years, there are many researchers show their interest and effort in signal detection, parameter estimation and modulation classification technologies. Most of them are working with single signal detection, parameter estimation and modulation classification. A few people consider time and frequency mixed signals as their target signals. In particular, some people assume that there is no any overlap between co-exited signals in time domain and frequency domain. In such case, we can easily separate those co-existed signals with a band-pass filter in frequency domain. Meanwhile, we can easily know the number of co-existed signals, estimate each signal’s parameters and classify their modulation types. However, in a spectrum congested environment, such as cognitive radio and electronic warfare, several signals are often mixed together with plenty of overlap in both time domain and frequency domain. In some special case, several signals are entirely overlapped in time domain and frequency domain, such as in-band full duplex communication signals. It is more challenge to enumerate and classify those kinds of mixed signals. Hence, studying mixed signal detection, parameter estimation and modulation classification is more practical significance. In this dissertation, we employ signal energy-based, mainly employ signal cyclostationary feature and machine learning technology-based methods to detect, estimate and classify mixed signal, which have significant overlap in both time domain and frequency domain. In particular, we employ energy-based detection to preliminary detect the signal is existed or not existed in the channel and use spectrum analysis roughly locate the interesting frequency band. Meanwhile, we employ different order signal cyclostationary features to detect, estimate and classify four popular digital communication signals, which includes low-order modulation type BPSK signal, high-order modulation type QPSK signal, 8-PSK signal and 16-QAM signals. According to our previous work, we can use second-order cyclostationary feature to detect and classify mixed signals, such as mixed BPSK signals, mixed QPSK signals. However, since some signals have no second-order cyclostationary feature, we unable to precisely estimate and classify them by using low order cyclostationary feature, such as we cannot use Spectral Correlation Function (SCF) to classify mixed QPSK signal and 16-QAM signal. So, in this dissertation, we consider some more challenged cases, include detecting, estimating and classifying mixed higher-order modulation signals, such as 16-QAM and 8-PSK signals, classifying mixed signals, which have similar cyclostationary features, such as QPSK and 16-QAM mixed signals, and analyze heavily overlapped mixed signals, such as two signals have same carrier frequency. Moreover, we employ low-order and high order cyclostationary features, i.e., cyclic moment and cyclic cumulants, to detect, estimate and classify more different combinations of mixed signals, such as BPSK and QPSK mixed signal, two QPSK mixed signal, BPSK and 16-QAM mixed signal, etc. In this dissertation, we also provide detailed performance analysis to demonstrate our proposed method can effectively detect mixed signals, estimate mixed signals’ parameters, such as carrier frequency, symbol rate and power, and classify mixed signals’ modulation types. In addition, our performance analysis is based on AWGN channel, flat fading channel and multi-path fading channels.