There has been an exponential growth in the number of malware in the cyber world in the last few years. Modern malware use sophisticated techniques such as polymorphism and metamorphism to thwart the malware detection and analysis.
Detecting malware on the basis of their features and behavior is critical for the computer security community. Most anti-virus depends on the signature based detection which is relatively easy to evade and is ineffective for zero-day exploit based malwares. Static analysis analyzes the executables without executing them whereas dynamic analysis actually executes the malware in a sand-boxed environment and the system changes are logged for further investigation. In this thesis, we are adopting a hybrid approach in which we integrate the feature vectors extracted from both static and dynamic analysis to detect unknown malware. Our experiments obtained an accuracy of 98.62% in detecting malware. Our detection system is robust and scalable as we have increased the amount of samples used for analysis and reduced the feature space compared to the existing approaches in the literature.