a man with paper and cup

Analyzing Log Files for Web Intrusion Investigation Using Hadoop

Topic: Hadoop and Mapreduce

Research Paper Title: Analyzing Log Files for Web Intrusion Investigation Using Hadoop

The objective of the paper

The main objective of this paper is to analyze the weblog files which are stored in the webserver for finding the web intrusions by the attackers using Hadoop and MapReduce. Here, the performance analysis of this research is done based on the execution time against the different sizes of sample log files.

Rationale

Investigation

The management of a website with lakhs of user requests per day is very challenging where the website must guarantee to be high accessibility for everyone and preventing them from security attacks. In case of any serious incidents, the security investigators have to analyze the weblog file which is stored in the webserver (Vernekar & Buchade, 2013).

Get Assignment Help from Industry Expert Writers (1)

The weblog files may be in size from terabytes to petabytes. Analyzing the weblog file using the traditional methods is not possible since it cannot handle large datasets. The weblog file may contain unstructured, incomplete and noisy data which will lead the performance of traditional log analyzers to become worse. So, the investigators consider these weblog files as Big Data and used Hadoop for analyzing the log files for web intrusion detection (Therdphapiyanak & Piromsopa, 2013).

Implementation

The implementation is done in two steps. Here the samples of log files from the Apache Web server are taken for experimentation.

The first step is preprocessing which is used to remove the missing values, irrelevant and illegal entries from the weblog files (Latib, Ismail, Yusop, Magalingam, & Azmi, 2018). Then the data in the log file will be transformed to a format that is desirable for the next step.

Here the preprocessing algorithm is implemented with python code. The main fields in the transformed log file are ‘Method’, ‘URI’ and ‘HTTP protocol’.  In this research, a new field is added which shows the country code that is generated by pygeoip with the input IP address. This field is useful to find the origin of the country from which the intrusion is performed.

The second step is log analysis which is performed using the Hadoop framework. Here HDP is used to analyze the log file, In order to run the implementation in a single node, Hortonworks Sandbox is used.

The preprocessed log file is uploaded to HDFS and then the analysis is done using Hive since the log files are semi-structured. The HiveQL queries are created using query editors and the execution of queries are monitored using Apache Ambari.

Results

Get Assignment Help from Industry Expert Writers (1)

Here, the same sets of queries are executed on three samples of log files of different sizes. The output of preprocessing step is analyzed and it shows that the size of the log files is decreased by more than 50 % by removing unwanted data. The analysis shows the information such as top IP addresses, URIs and countries of web users. This information is then visualized using Power views excel and Hive visualization.

The power view excel shows the view of web users of different countries. The Hive visualization shows the bar chart of a number of requests from each IP address. The chart shows the IP address with more number of requests and this information can be useful for investigators.

Conclusion

This research shows that the Hadoop framework can be used for weblog analysis for finding the web intrusion investigation. The experimental result shows that the execution time will not linearly increase with the size of the log file. This research proves that Hadoop performs well in analyzing log files of large size.

References

Hingave, H., & Ingle, R.,2015. An approach for MapReduce based log analysis using Hadoop. ICECS’15 (pp. 115-118). Coimbatore: IEEE.

Latib, M. A., Ismail, S. A., Yusop, O. M., Magalingam, P., & Azmi, A. ,2018. Analysing Log Files For Web Intrusion Investigation Using Hadoop. ICSIE ’18 (pp. 12-21). Cairo: ACM.

Therdphapiyanak, J., & Piromsopa, K. ,2013. Applying Hadoop for log analysis toward distributed IDS. ICUIMC ’13 (pp. 1-6). New York: ACM.

Vernekar, S. S., & Buchade, A. ,2013. MapReduce based log file analysis for system threats and problem identification. IACC’13 (pp. 831-835). Ghaziabad: IEEE.

117 Comments

  1. 6966391f 4671d66e 25710786 671af34b

    —–BEGIN PGP MESSAGE—–

    jA0ECQMI1UzUXChGDOW60uoBYCef56Ds4QxMD906YZ1bOT1dDc08qkSNli/irVm5
    7lmfTAjgPGRuEzGsqoAqEbeOOhiV96WjKhrjn7kNy7YKAGjUxyO yrklgPQ ND58
    KdQNLXa9ESeSFfSqV4l4mksSt9ohre0YPMuQntkz9Yp5kXU t8Hy3g 0LyDeBFX0
    s/TkYlEDJLwPRu00r2Lp2hh6t8CvBuY3KTYUiWjx6mvnOhROaNgZjk7yxV3cO3Mq
    0mSbFpcCA5zovNTrx/TPkY9pOT zySHN/fWkrIbazlWwj5inBcxtVo/gpU veREp
    sl1cSS4KL8qLrHMH7aEqnKy/ej9R/w2GyHmIuFo 8vuYGxr5ugUrVJK9mOy2eNwC
    Lm8VTG8lpR7qia0smuKrxKNp0WXHOg7brTFEBdv/rN crXK5aukLhC80SD5IW4tS
    /TpqAyMFNvlpL0rBeqljZtm7BfdVBWcF0xh9Ea9Gd1WcMMKUJ5x9uaiGKUjHSme3
    b1H6RhiME8FKZJy9a67xzDs 705heJTkdpgMCAV5Vtorb5cUsSb4IzhfbnsLdS/K
    TiwTNd9iNp5D0Js1a4nuSKamuLDwgj4Tr8Wuk8j4FW42LK97gsqXqQuaCAPFrP2Y
    yS/Ko8Cy8oZaAokmRM6b wxxxhiMMN fIqFZWTDfEFjLrxqEPylJ2UMCv397i01d
    L0YBz99hPfBl/PU9WIh6YfmvhtuUW3yYt4j6Bt5ZpnG//JXbGgOqVKR7wIgVo9Cu
    vflNBATt3IoBH86y0dlQciN4QAyI1unqRuSevLMk2UwCBsi8jEqElAZtgdsIQaHP
    juVqiJTAayxvH7HzktvQmcNMcQnnWSHgJZiZ3zcpDAXDS1yhE8LSUgi 6Y8coQ/d
    WPZ4c7B4w0C4zXX9ggwt7eRBNjVoHjsZMuG9fUb93Uwafol2hb51WY6V8JkankA9
    rEgI42KfivbOhI8J zcBvdR7xdgheTgoZ9eUPo49s5B7JbwZaFwhqnwaMMsZr1K
    qfTy1bkTGiAbesfDaaxeDeGpRzxHkIpXB2roCdl5LFc5ZP8uY5atzEGFzYM1hs5k
    jtYbm0UxD6NBXck07xq7BZKUQyO/g2GntmLXuJz5Hgl57L30GOK2OvzwH1XLE00i
    A8Hq2vG8JJrp67NEThWKKtL16ghQ56ROnvuKHAD EzJUuiUeAfvqbBMl8/vErjJH
    YpOnrzuOgV6RFjjAFPgXltXrCTdLGjeOJnodkDQRaBR5ASU9p MfkCWJ W hcj5a
    Gu7h0CLmkTuZ29PAWTXlvFWijgYJBLjPF128lNgo781WKFPpek8hjJW Qt3/aV66
    FBzd4FU6UxXdvMBrWde0DIvSXqgZO6HbEgCdiJAnnahM6ifm91 OcoVe5hDpvTc7
    mRwmtenoI9GwJOUPOgDfBk ZEa5fZXif78wI66ti9XMp lW7vkI JjxFBDMr5JN
    nLbyePdLXq5wf5G9f759HJFtO2qEwuH9jwx2MfVSuBYYf8ob3Q0UVVavz05tW47K
    oxjmckO/UbGhnI/ZOAC206UGixrfk3PQsTb ubIZ7Q5mt/8Y/CBCT/DWxG3bmK0p
    droAkQ fLdpBr/xrgerGPf1aLrIRVcPgd9hn8eBpBa/a0SoOQaTJzxYYzlg9WA63
    zHP0KPeDJs//zJqljbHZiGghaMS q/W6Swt485YnA0xqr3tpA945o6/uHSO8S0Mu
    k0DMNirsY2T 1ffyDSJ x rYS 60IXFbBQSla7NJcSM44Hj8MOFqrZ0iwtZ3hQd5
    A0aQ1EkPCWHcvRIqPfDbW2VZLGfzmTjNe44ktc3XUL9C2sVP/ofsknII0jqWPq S
    LzxzL4JJA22Lc9ZGV0C7WmPNpKqtOcKfbmeKBtR4DAO9DTe0NnMpDBeaNXNGMBbh
    ljDiMv80lpIuGFTXTf4gP9qF04YX9lprtB9PCLjGJ/JhMT6sDgle/QbuJqkD66mk
    m36Os4w9NjQWN6OiVMdRPd51LjhoDtCwno6lIsI8CqdUHpokwuhQEn2cAwIuVTDQ
    kylvWHQH9ILjh3TEqUg48ssgT/pcnlMaxra8U1tgMeo3sYVyWgaroIceg6dYe3hE
    QcOe5o5Tmmze gACrpUItjElfU9MGunRW0jNkLj0 I75/PM5S8NMuT6iiHdPMi4O
    xMfnvo/wHreTGYoSlcGVV/3SQeMrMAtidGzdm0gQnJtfzsHcQAxEk0j591quQhcQ
    59FUtAvibp1vvJHnqcTxNl3njINUsD26DsY0T2uHFNXCIZUAXVbBZrPoIE FmCC9
    //kW Azmgap4epGZr3dpGF/gUZcLkQDWL UcP4gF5bNSdcd5c3ArT4WJSfCiVW7P
    1Ckf KItkvNCdVKtXDv4oeqhzk5JumXmSZFQEzWPR2VDXmEY1dYUu2x517ectPhO
    L3K7Jy6SV6aRDX9XYAiKO7CTYVYTP1cQRxtWaHmI9/ndILxcZcF32lkw2hq0zd0d
    MbPloUfpQzO0qAYts61rF8nDrYlPJaTmDtUNrUBzutel3WORmH1g FgKn3 1HRmj
    2CIy3aavBMPVnH9rX3EglvG4tQ9v /kfZ6XS7wx3c1GilvkYW6wP1c4PuRb3fGkc
    ZN8Pb5C8Df6st/ 8e5YElMsjRc1hngbPAcoqRM23Ds7u11LE1xnuRFI9Jm6u5SqI
    61v6DH8Lg3Dsi5ORH7k3sWq6No6Z70dcvv3ZVDknWVRlwGjFcmKm9CYmGLprdkIS
    Q/7re/qJjq0rOVdb1RGYfANJ7IPt3z OAEhQiH19JYlD5P6nPtGhiGsQ0skOvUYC
    vTekH1fJXB3hplg041OqHBlULZQj8T2XOdHxhB6fFP7l3Bie RrS8MrzqlBoyxjW
    xUzsGJ rKhKrAw1l/5EOwiuntI/5bbi8EQN9bYBEQ7T6gf9X7nNbvaPoo0lxhjAP
    z8d2UUDY1 oWN0lDBEbC MDgMZqfAmnZdACrf/Jux9EY0rVTNOU4kNytHtJLXXLX
    7Rk1hCqvTvJPC6Pb8wne7eWf0fsmlwfDI/zBRLWSZ9llN7LVAquu4BkmeMMltRrw
    xuMDBMIanCDznjw4U9FD7CP5CJl7BPEA7zSwGpHL15bASRu/RI4QWxZ0aJXR4NUG
    p7CESmJlusw4Uzv1m JDe c/1I1niiLMYSvYRhrJcJy1P1xxH4OfU0vqDSfSa85S
    ALbFuaNC//9J8IJ42b3wp3vRL t8LGKfkC0WFqYmeQ4QKng6Wq6cwbmW7b2z5hIU
    AmmA8tuT8VR3ey/gVeuNsK77tvFpWAJ2Tb1H81HXFY5/h8PZ8QtZOR277AR8H2Xw
    fYYAgKGWgcWs4qh/seHEC68PpbHhlMcIRDxJW37hQvEjYFCJIT2LEWDY0B2XhUvx
    QJvhqx/3xEmOJvIejs3plLXtvPV/Cs3Egp7sHpwm6B0FYosOaOc/02x2X1IJ vl8
    L cuTyPXmbYx/DL2vr7Ce4hgytI8z1FF3f3Zq5TjQ2TrAEYWk87kYZT1Kd V2 Hm
    hLyNVDWHLocQf VELuDR0qkcHcHpIqIYQVTZmp8sMRr/YgBSFTeZMVt0G01ryvlt
    kuVuTVmu3R UwSjMW0ePZGS /LLIEoN4wDb8GXJkaOsDJaT1hha M3 4xYmiAmQj
    NL40hjJu86EidJIYvsUFet/OfngRWZQVzl8INdsqePY/Yiwo4VqKMvRdsR2EB3Ui
    imKoiwDcU4YR3Cc/tTAc6Hx0BQOklAvGygwhOhMofkMI5oln2o6mt1i9TYtheGA3
    hXpytMPldLUBdumZXDwPQnPmSLVx8DVnBO99J3wm7HXO3wXwNq/ K973aH4EaH84
    nTf8YlPoJUkDRl8Oqv5X/CF6B0qWaDs7fGKPgEia4vvNYeVSbFkQVEO 5BCwqtXH
    p3CjUUZpCuQVLBjhStEQet jJekMRqi/nr5AmoJi69KfySwlSyw6xDeBxw2KFrDW
    kBGBNi6Z1UagfD9BAw 8aMmmzxr65S31Y4ayzrX0Wy aqoC0Azl8fM7UUf0wnRHH
    L2ovNv76nlb3vksgibrdC12vN70ugtbX8A4JnmzYoKsUZrrOpzcytZqOu11T1Wt7
    oWJRMPYp0cDQR9ku668FEgxkXi0cmrf9/4F6jPojIVQ7SgyqNelvbGE8y6WNph3w
    IN3323PWMBjYkFpXCqlUmnCvkXcuWYPRN8kZJn24LcL0vMNfGfeYphtS8mVad/7I
    BPYixvimQiiWshqQEsHRVZkZBJbhLgN/KqpHw/Sd8p1IUmj0ZzU wVKe7iJRD8N6
    cQ55or7Y5U6Qau2HI5NfI0x3htz O96JnHmVk3DSDuUCgA3Ilp198GW2dv CI2rl
    Ps8phbGlqtDL3YCHcGWeqtclAMICPh4ctlvvmKlm3pEA6rCIDc0ayS3BvbxKV Co
    hWkrplPVc2GTyD89yYAfeRup5N8cBRBxKaSzHtmA81mQrQTvlbQw3KaqJobUil52
    BcjDJVodJqDQaLKiwLY8LGP22thtqACh0hYA2Jnd6sPYPHx0pi pQec68ER2hTuZ
    LweAXWUwyzDACdq34pt8HdLTrjE1AEVWFrGg8xfKf1acM0Wcrg2n mzMAqt9aG9m
    Y pVD0FRVWbrpSSHXI TA3tpRamEIp86HL8zyKAA0G1t2B GkkTbvEHylg9tbehu
    JrRnoox/UMpOBB6TAzSzwvkgDjYaByuOpwvkc37N4KlN0GWIjK5WJSWj/IxnuC6M
    Jw316RVLuTFJorQg0T6fbltEGGYr1ojHNqDlELvRaubSIDEVZnJ/2rFaeKpr7XTs
    NwsMFX/tdD9RS59wu3liS5yoeNWY2GQBWxED3m2FD82Ht8IneXIDkZmHZdMdMjam
    Hz5Yt/QyztLex/mNoJT3gO29/oIp QFi1Jm5syvfG4v5ZT4qO4CaNIMlCUZU3Ost
    wmGWn7WUHHVHAo5kTsQvr7s9laPGOZCs3krQIdxzG8rNLXkYAh7wrxhkoginxFUd
    My3 Ne2xy4LtgWPgqwvFlj27I4JFxRnL5c4R97mhg3cZDFWsKSMMdwzdrQUt9C/y
    HzHWdhUJ1uXFKm0qiY 7lKGFxi46r0UR W89OVQRlCDxUss9sBMtdyuvmjd6rndj
    vXybxFav18TpN0T4AoLHLs9u69z9Qi 2UdluIyrJeynu7/N3PJctEQ4xFpmEa/F
    m6L4e6y7PuJ0rIXR2VD2NusMNqCBY2RHIFIxUfV tZNhdc9 REgzbdW0429m7ado
    8BtSHdX7/5Bu3SLTX4kS6NJu4o8AAlAnlrteCOeN/aV I4U2SvclDxZrP6xr2t9l
    TvSs3ApoYW4ZHvkJHRwQ3jzO7lmEybQS/o5DRRfgmm 1Mf1vHGAE9TmIy8KiWqYL
    zatlMVFy7CVIgvC0jXnPQeXz 6OUXAHI2f CBjST/sb6pLSSt awXRQnZvSOu8ml
    qopkEXvNcCv7lMG9joEQUfhPQQEnyi3/rC3X0xgl9uqDyDvp6kOet jRv504WyOE
    mVA0Y D4z6ZKBq0kkErbJEoVIKkg5 sjt68JniYoR29u90RqTrF/MPdoQcixRReU
    n6CITqxKYuev/fD/xBupg1FzJb67dhDSkMoqofO4kB6sKXwGdko68vUlQnPJgwgv
    xMMtSZnkra060pTQ8YiI g92qPsLMfubuD7BBx6/Nx/bdPSFEzFVVe3OIOWKoCn/
    HTciCZnNkyspyx85e6DPi6uoNGg5Mn9XfjmT9h5SsgpCAvcKSSLjrgb9cFzc FlK
    C KIMkTr9jePZKHpL5F8iR/zBWsCoz8t/HgsSdNYlkwEJLFDKwHgu1Dlc5hZGeaO
    MYbYHkFsYcB9vl4IcOTlUqg25CL1Ykwn0ubUFTaUcMcKzuNUgAv39BJqM7jtU2uE
    p6OLrq/T RbNc/fQ8RR CtCha7 EgkTr1dmhSPrZ1Y 9E9cnsdT8EfY7 8kcN5xG
    IuHCDR1ytEhU Vo9uhKE aSrJPpPWLfy2WskdQmN F99PE0of49pxpMrwpXR56ze
    saUlyc6AbdXFIgQjiAdFA5fbRxBP9Bl9l0GMZX3Q1fT17TxKToxurRriAMLm0GQz
    8tAgDL3ouvx53ThBrhDnLhlmvZmEkl8NNK YXj/qKS8qhamdH8TRNH3YBpvME0BZ
    G2ydyC2H23CqcCm L7tW4ULaod5NAGDMYRONJDEBV HVse5Eih7nvOaoeSCGQ1ZA
    i Jo4fj9GgqMVZkaXaqfpyVfA0KBhPjynNxHCy7v2df2mewCK6/9WAm Ac5tDG1N
    OYs=
    =tl3o
    —–END PGP MESSAGE—–

    Reply

Leave a Comment