Partial rant, partial useful blog post — I’m noticing that a lot of the “new” APIs for sites are starting to restrict access to content either by putting limits on content either by controlling how much of the data you’re allowed to access, or by not including the ability to access a certain amount of data over the API at all. This is different from a few years ago where sites like Twitter, would let you collect all the tweets from a user without issue. Maybe they’re being more privacy conscious (lulz) or maybe they want to charge a premium for this type of access, I don’t know.

One for example is Google Latitude. When I’m friends with someone, I can access their location. They have shared their location with me and have me as a trusted person. But going through the official Latitude API, you’re specifically blocked from collecting any kind of user’s private information. This sucks. I’ll be honest, I don’t have any good uses for stalking someone on Latitude but I think it’s funny to be able to track someone and run a report on where they’ve been over the past couple of days.

The newest “privacy aware” API is twitter. Their 1.1 API has been out for a while, but they recently blocked access to 1.0 (breaking all my scripts) which means that you no longer are able to easily collect tweets and other user information. That also means that those RSS/ATOM feeds you used to keep track of a person are gone. You now must have a Twitter client to access Twitter in some usable fashion.

OAuth Hates Scripters

To add to this situation, OAuth2 (compared to it’s predecessor)requires users to make a website interaction in order to collect the token and secret values meaning that scripts are annoying. We can get around this in a couple of ways, my favorite of which is using FoAuth.org. All this service does is facilitate the web requests necessary to  collect the OAuth values, and store them. Then using your python script and the new popular Requests library we can call FoAuth to get the info that we need and proxy the requests to the API we’re using. As designed, this has an expiration but it’s much easier to go back to FoAuth and re-authenticate a token rather than doing it for all of your scripts. Here’s an example from their main page:

 

Twitter 1.1 API Example

Anyways, here’s something potentially useful amid my rant. The Twitter 1.1 API uses Oauth to make requests and then gets rid of the whole pagination idea that was in the previous version, and relies on this “max_id” value. Basically, max_id is where you want to start collecting tweets from. So if I want collect all of a user’s tweets, I can collect the first 200, find the last one that I pulled, and make a request starting at that last one, looping until there are no more (or no more are given out).

Here’s how that looks:

https://gist.github.com/antitree/5835529

The problem with the above is that the Twitter API doesn’t give you _all_ of the tweets. Just whatever they feel like. Usually that’s a large amount (over 1000) but for users with lots-o-tweets, you’ll just hit an (AFAIK) arbitrary brick wall.

In Conclusion, screw you apis

IANADev, so to me, APIs are a polite way of accessing data but if we keep getting blocked, we can go back in time and collect the data in other ways. From an OSINT perspective, we would like to gather this content for whatever legitimate and illegitimate purposes that we want. I’ll be spending my time before Defcon updating whatever tools I have to make sure they’re not going to suddenly stop working.

Also to note, Recon-ng, a great tool for recon/OSINT/whatever, has had support for the 1.1 API for a while now which leads me to continue to believe it’s worth porting my tools into rather than trying to roll my own.

 

This Friday, I’ll be presenting a weird presentation at BSides Detroit. It’s titled “Corporate Intelligence: Lisbeth Salandar vs James Bond” and it’s on a subject that has been stuck in my head for a while. It’s a talk about corporate spying, competitive intelligence, industrial espionage — the type of thing where people are stealing information from one group, and selling it to another. As is the case with most of my presentations, it has little to do with what I do for work, and more to do with play.

In short, it’s going to be an anti-anti-corporate espionage training course where instead of learning of ways to protect yourself from being spied upon, you’ll learn all the tactics that spies are using and how to become one yourself. I’ll cover something called the Intelligence Lifecycle (that oddly enough a few other people have picked up on) and the Principal Motivators of Betrayal that shows how you can often convince people to become a spy for you. I’ve also created a little Android application that I wrote when I started playing these spy games.

Usually, when someone says “intel” they mean OSINT because there’s no way someone is going to spend the time trying to establish a human source of intel and no one wants to actually do something illegal like the tactics employed in technical intel gathering. OSINT, although referenced in the talk, is becoming somewhat of a tired subject. Not because I am an expert but because there’s nothing really new or novel about the way we use open sources of information. There is a lot of research about collection and very little about analysis so usually the output of most of the tools out there now is just downloading information online which results in a gigantic, unusable, pile of bloody data. I submitted a talk to Defcon Skytalks that goes into this subject even more (fingers crossed), but this talk, is more me laying the baseline to expand upon the idea of what we think of as “intel” to other sources of intelligence and see what kind of discussion it generates. 

The presentation is something I started while in Mexico reading The Grey Line. I moved on to other good books like Ira Winkler’s “Corporate Espionage” and “The Quiet Threat” by Ronald L. Mendell.  The full reading list is below if you’re interested.

The Grey Line: Modern Corporate Espionage and Counterintelligence, Andrew Brown

Corporate Espionage: What It Is, Why It’s Happening in Your Company, What You Must Do About It, Ira Winkler

The Quiet Threat, Ronald L. Mendell

Countering Industrial Espionage, Peter Heiims

Competitive Technical Intelligence: A Guide To Design, Analysis, And Action, Mathias M Coburn

Industrial Espionage: Intelligence Techniques and Countermeasures, Norman R Bottom, Jr., Robert R. J. Gallati

UPDATE: 6/8/2013 a version of my slide deck