Competition Information This is the second Microsoft Malware Hosted competition on Kaggle. For more detials, please see here. The malware industry continues to be a well-organized, well-funded market dedicated to evading traditional security measures. Once a computer is infected by malware, criminals can hurt consumers and enterprises in many ways.With more than one billion enterprise and consumer customers, Microsoft takes this problem very seriously and is deeply invested in improving security. As one part of their overall strategy for doing so, Microsoft is challenging the data science community to develop techniques to predict if a machine will soon be hit with malware. As with their previous, Malware Challenge (2015), Microsoft is providing Kagglers with an unprecedented malware dataset to encourage open-source progress on effective techniques for predicting malware occurrences.

Goals The goal of this competition is to predict a Windows machine’s probability of getting infected by various families of malware, based on different properties of that machine. The telemetry data containing these properties and the machine infections was generated by combining heartbeat and threat reports collected by Microsoft’s endpoint protection solution, Windows Defender. Each row in this dataset corresponds to a machine, uniquely identified by a MachineIdentifier. HasDetections is the ground truth and indicates that Malware was detected on the machine. Using the information and labels in train.csv, you must predict the value for HasDetections for each machine in test.csv.

About This Kernel In this kernal, I would like to recored my personal EDA for Microsoft Malware competition.

Imported Packages

import numpy as np 
import pandas as pd 
import matplotlib.pyplot as plt
import warnings
import time
import sys
import datetime

plt.style.use('seaborn')
pd.set_option('display.max_columns', 500)

Data Types

This part was taken from the helpful kernel.

dtypes = {
        'MachineIdentifier':                                    'category',
        'ProductName':                                          'category',
        'EngineVersion':                                        'category',
        'AppVersion':                                           'category',
        'AvSigVersion':                                         'category',
        'IsBeta':                                               'int8',
        'RtpStateBitfield':                                     'float16',
        'IsSxsPassiveMode':                                     'int8',
        'DefaultBrowsersIdentifier':                            'float16',
        'AVProductStatesIdentifier':                            'float32',
        'AVProductsInstalled':                                  'float16',
        'AVProductsEnabled':                                    'float16',
        'HasTpm':                                               'int8',
        'CountryIdentifier':                                    'int16',
        'CityIdentifier':                                       'float32',
        'OrganizationIdentifier':                               'float16',
        'GeoNameIdentifier':                                    'float16',
        'LocaleEnglishNameIdentifier':                          'int8',
        'Platform':                                             'category',
        'Processor':                                            'category',
        'OsVer':                                                'category',
        'OsBuild':                                              'int16',
        'OsSuite':                                              'int16',
        'OsPlatformSubRelease':                                 'category',
        'OsBuildLab':                                           'category',
        'SkuEdition':                                           'category',
        'IsProtected':                                          'float16',
        'AutoSampleOptIn':                                      'int8',
        'PuaMode':                                              'category',
        'SMode':                                                'float16',
        'IeVerIdentifier':                                      'float16',
        'SmartScreen':                                          'category',
        'Firewall':                                             'float16',
        'UacLuaenable':                                         'float32',
        'Census_MDC2FormFactor':                                'category',
        'Census_DeviceFamily':                                  'category',
        'Census_OEMNameIdentifier':                             'float16',
        'Census_OEMModelIdentifier':                            'float32',
        'Census_ProcessorCoreCount':                            'float16',
        'Census_ProcessorManufacturerIdentifier':               'float16',
        'Census_ProcessorModelIdentifier':                      'float16',
        'Census_ProcessorClass':                                'category',
        'Census_PrimaryDiskTotalCapacity':                      'float32',
        'Census_PrimaryDiskTypeName':                           'category',
        'Census_SystemVolumeTotalCapacity':                     'float32',
        'Census_HasOpticalDiskDrive':                           'int8',
        'Census_TotalPhysicalRAM':                              'float32',
        'Census_ChassisTypeName':                               'category',
        'Census_InternalPrimaryDiagonalDisplaySizeInInches':    'float16',
        'Census_InternalPrimaryDisplayResolutionHorizontal':    'float16',
        'Census_InternalPrimaryDisplayResolutionVertical':      'float16',
        'Census_PowerPlatformRoleName':                         'category',
        'Census_InternalBatteryType':                           'category',
        'Census_InternalBatteryNumberOfCharges':                'float32',
        'Census_OSVersion':                                     'category',
        'Census_OSArchitecture':                                'category',
        'Census_OSBranch':                                      'category',
        'Census_OSBuildNumber':                                 'int16',
        'Census_OSBuildRevision':                               'int32',
        'Census_OSEdition':                                     'category',
        'Census_OSSkuName':                                     'category',
        'Census_OSInstallTypeName':                             'category',
        'Census_OSInstallLanguageIdentifier':                   'float16',
        'Census_OSUILocaleIdentifier':                          'int16',
        'Census_OSWUAutoUpdateOptionsName':                     'category',
        'Census_IsPortableOperatingSystem':                     'int8',
        'Census_GenuineStateName':                              'category',
        'Census_ActivationChannel':                             'category',
        'Census_IsFlightingInternal':                           'float16',
        'Census_IsFlightsDisabled':                             'float16',
        'Census_FlightRing':                                    'category',
        'Census_ThresholdOptIn':                                'float16',
        'Census_FirmwareManufacturerIdentifier':                'float16',
        'Census_FirmwareVersionIdentifier':                     'float32',
        'Census_IsSecureBootEnabled':                           'int8',
        'Census_IsWIMBootEnabled':                              'float16',
        'Census_IsVirtualDevice':                               'float16',
        'Census_IsTouchEnabled':                                'int8',
        'Census_IsPenCapable':                                  'int8',
        'Census_IsAlwaysOnAlwaysConnectedCapable':              'float16',
        'Wdft_IsGamer':                                         'float16',
        'Wdft_RegionIdentifier':                                'float16',
        'HasDetections':                                        'int8'
        }

%time train_data = pd.read_csv("data/train.csv", dtype=dtypes)
%time test_data = pd.read_csv("data/test.csv", dtype=dtypes)

Wall time: 2min 7s
Wall time: 1min 51s

Exploratory Data Analysis

train_data.head(20)

’HasDetections’ Distribustion

train_data['HasDetections'].value_counts()

0    4462591
1    4458892
Name: HasDetections, dtype: int64

Missing Vaules

Missing Values in Training Data

total = train_data.isnull().sum().sort_values(ascending = False)
percent = (train_data.isnull().sum()/train_data.isnull().count()*100).sort_values(ascending = False)
missing_train_data  = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])
missing_train_data.head(50)

{% raw %}

{% endraw %}

Missing Values in Testing Data

total = test_data.isnull().sum().sort_values(ascending = False)
percent = (test_data.isnull().sum()/test_data.isnull().count()*100).sort_values(ascending = False)
missing_test_data  = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])
missing_test_data.head(50)

{% raw %}

{% endraw %}

Unique Column Values

In this section, I would like to know the values of each column. Specifically, we would like to know the unique values and if there is any value that only exists on the training set.

training_data_columns = list(train_data.columns)
testing_data_columns = list(test_data.columns)

unique_val = pd.DataFrame(index = [], columns = ['TotalUniqueCount','SharedUniqueCount','TrainUniqueCount','OnlyInTrain','TestUniqueCount', 'OnlyInTest'])
for column in testing_data_columns[1:]:
    unique_val_train = train_data[column].unique()
    unique_val_test = test_data[column].unique()
    
    only_in_training = list(set(unique_val_train) - set(unique_val_test))
    only_in_testing = list(set(unique_val_test) - set(unique_val_train))

    total_unique_val_count = len(set(unique_val_train) | set(unique_val_test))
    shared_unique_val_count = len(set(unique_val_train) & set(unique_val_test))
    
    train_unique_val_count = len(unique_val_train)
    train_only_unique_val_count = len(only_in_training)
    
    test_unique_val_count = len(unique_val_test)
    test_only_unique_val_count = len(only_in_testing)
    
    unique_val.loc[column] = [total_unique_val_count, shared_unique_val_count,
                              train_unique_val_count, train_only_unique_val_count,
                              test_unique_val_count, test_only_unique_val_count]
    
#     print(f'{column}:')
#     print(f'----------------')
#     print(f'Total unique count: {total_unique_count}')
#     print(f'Train unique count: {train_unique_val_count}')
#     print(f'Test unique count: {test_unique_val_count}')
#     print(f'Only in train: {only_in_training[0:5]}{"..." if len(only_in_training) > 5 else ""}')
#     print(f'Only in test: {only_in_testing[0:5]}{"..." if len(only_in_testing) > 5 else ""}')
#     print('=================================================================')
#     print()

unique_val

{% raw %}

{% endraw %}