Using Python to Monitor a .Onion Dark Web Site

I have a few servers running on the dark web for my SANSSEC497 Practical OSINT course . The dark web is known for many things, but reliability isn’t necessarily one of them, which is why I have multiple. As the class becomes available in March, students will take it all over the world, at different times. Because of this, I needed a small program to monitor my dark web sites and let me know if they were offline. My first attempt was using a popular open-source website monitor. I made several attempts to route its traffic through Tor to monitor my .onion sites, but they weren’t successful. I finally decided to write a simple Python script to fit my needs. I then thought, why do that, when I can have ChatGPT do it for me? I went to the ChatGPT website and asked it to write me some python code to check if my .onion site was online and to alert me if it wasn’t. I had to switch the port it wanted to use for the SOCKS proxy (more on that later), but the code worked. I then asked it to ch

Themes for 2023

In a recent Discord chat, someone told me that they hated new year’s resolutions, and greatly preferred new year’s themes. I thought about it for a minute, and agreed that was a much better approach. Instead of a goal like “lose thirty pounds”, an overall theme of becoming a healthier person. That is one of my biggest themes for 2023 but another of the things I want to focus on this year is producing content.  I wrote a blog post last which normally isn’t a big deal, but that was my first blog post since... (checks notes) … May of 2020. I could blame it on Covid or some other excuses but the fact is I was just burnt out. I was working a fifty hour a week job with the federal government and spending my vacation time teaching SANS classes. These are all good things and I’m not complaining, I just didn’t really have much energy left for producing much content outside of the occasional conference talk.  Early in 2022 I received the opportunity to write an OSINT class for SANS which I knew

A Quick Look at What's inside the 1/4/2023 Twiter Leaked Data

You may have read that Twitter was hacked and hundreds of millions of user's data was stolen. In this post we'll talk about what happened, and what's in the data. This wasn't a breach in the way that most people think breach. Twitter's API had a flaw where if you provided an email address, it would reveal if that email address belonged to an account, and which account it belonged to. Someone used that to compile over 220,000,000 email addresses, and what user accounts those email addresses were tied to. Here is a (censored) look at what the data looks like: One of the biggest questions I had was if the data contained phone numbers for user's who used that method to authenticate instead of emails, but it doesn't look like that was the case, at least in this dataset. Everything that matches the pattern of a phone number looks to be part of the user's screen name. Hudson Rock co-founder Alon Gal pointed out on the @RockHudsonRock Twitter account tha

Nation State Quality OSINT on a Taco Bell Budget – Part 2

Welcome to the second post in our series on getting started using AWS services for OSINT. Last post we covered setting up an AWS account, getting the command line interface installed and configured with your credentials and using the Rekognition service to help solve some very common OSINT problems. In this post we’re going to build on that and: Register as a Twitter developer to get API credentials Setup up a Lightsail instance where we can run our code 24/7 for an extremely low cost (free at first) Setup an email with Amazon’s Simple Email Service (SES) which we’ll use to send email alerts Make a very simple database using Amazon’s DynamoDB Tie all of these together to make a persistent twitter monitor in Python To start off, let’s head over to Twitter and register as a developer so we can get an API key. Twitter does a great job supporting their community and you can sign up for a developer account here: The free “sandbox” account is

Nation State Quality OSINT on a Taco Bell Budget – Part 1

I remember taking digital forensics classes years ago and at the very end of the class feeling like I had learned a ton, had a great time, but was also wondering “Ok, what’s next?” When I’m teaching OSINT or other technical topics I often hear “This has been great! What’s next?” For some people, the next step is learning a programing language like Python to write your own custom tools such as web scrapers. Over the past year, I’ve found myself using Amazon Web Services (AWS) services more and more in OSINT tools that I write. They are very well documented but recently when I’ve been talking with other OSINT practitioners, I’ve realized that the part that often leaves people feeling overwhelmed is knowing what is available and how it can be used. In this multi-part blog series, I’ll provide a brief introduction to several of the AWS services as well as some code samples showing how we can use them on OSINT projects. Some of the services that we’re going to use in this series are: Awscli

Filelocator Pro Tips and Tricks for Indexing Large Breach Data Sets

Tomorrow I’ll be giving a talk on breach data including: Places where it’s located How to make large data sets searchable in a reasonable amount of time How some organizations are using breach data to improve their security posture Whenever I give conference talks I try to remove or reduce any barriers to entry. When I have given talks on memory forensics, I have always used the Windows standalone version of Volatility instead of Linux for my demos so attendees who were not really comfortable with Linux wouldn’t feel like they couldn’t try the techniques. With that idea in mind, I wanted to find a way to make large breach datasets searchable without the need to maintain huge databases, normalize hundreds (or more) of disparate datasets etc. Similar to a recent blog post I wrote where I used a forensics tool called bulk extractor to help quickly acquire selectors (emails, phone numbers etc) from a large dataset, I decided to use a common forensics technique of indexing for this problem.

Using Bulk Extractor for Quick OSINT Wins

Early this week, hosted a dump of a SQL databasehacked from a neo nazi forum online known as Iron March at .While there were some .CSV files, there was also an 750MB SQL database file. Withsome massaging, SQL databases can be queried for the data they contain.Sometimes all you’re looking for is a quick and dirty list of selectors andthis data dump seemed like the perfect opportunity to do a quick write-up onusing Bulk Extractor for OSINT. Bulk extractor is an open source tool that can be downloadedfrom .I first learned about it in a digital forensics class years ago and I’ve been afan ever since. It’s designed to quickly chew threw a pile of data and extractthe selectors (IP addresses, email addresses, phone numbers etc.) containedwithin that data. I’ve run it on hard drives, forensics image files, databasefiles, folders full of different file types, memory dumps from mobile phonesetc. I