Re: New D2D Finds - 4/16/2013
I'm not using anything crazy or black hat ops. Just your everyday browsers' included developer tools. Right-click on a page and, if you have them enabled, you should see a "Inspect Element" option. Click that and it'll open the tools in a frame within the same window, which you can move to a new window if you want. Firebug was the original, but now all the browsers should ship with them. (I use Chrome for most development.)
Then go to the 'Network' or 'Net' tab and you can watch all the little bits of data that a page requests. (You may need to refresh the current page if you didn't have the Dev Tools open at the beginning of the request.) vudu uses a JSON API for most/all of the 'data' on their site, which means it passes JavaScript objects...which luckily are pretty readable except that they normally are displayed as one long line of text. If you visit the disc to digital page, wait for the entire page to load, then hit the 'clear' button to empty out the current items (in the dev tools network list.) Then try a search and you should see a couple of requests show up in the list. You can click on any of the items for more details, but the 'Response' section is normally what you want. Try out a few searches and copy the responses to a text file and you'll soon get an idea of the data structure.
That's all I did to figure out how to pull the info from vudu. Then I just built a small web app that scrubs the data and builds a list for me. My app's not that smart because I can't pull any real metrics from the feeds besides whether or not there are more than 10 results for the current term. So it makes a ton of small requests that would definitely be alleviated with a public/documented API...especially if they had results ordered by dates of availability...one can dream.
Have they mentioned the University tie-in on the forums? I'd be interested in who-did-what if it's out there somewhere.
I'm not using anything crazy or black hat ops. Just your everyday browsers' included developer tools. Right-click on a page and, if you have them enabled, you should see a "Inspect Element" option. Click that and it'll open the tools in a frame within the same window, which you can move to a new window if you want. Firebug was the original, but now all the browsers should ship with them. (I use Chrome for most development.)
Then go to the 'Network' or 'Net' tab and you can watch all the little bits of data that a page requests. (You may need to refresh the current page if you didn't have the Dev Tools open at the beginning of the request.) vudu uses a JSON API for most/all of the 'data' on their site, which means it passes JavaScript objects...which luckily are pretty readable except that they normally are displayed as one long line of text. If you visit the disc to digital page, wait for the entire page to load, then hit the 'clear' button to empty out the current items (in the dev tools network list.) Then try a search and you should see a couple of requests show up in the list. You can click on any of the items for more details, but the 'Response' section is normally what you want. Try out a few searches and copy the responses to a text file and you'll soon get an idea of the data structure.
That's all I did to figure out how to pull the info from vudu. Then I just built a small web app that scrubs the data and builds a list for me. My app's not that smart because I can't pull any real metrics from the feeds besides whether or not there are more than 10 results for the current term. So it makes a ton of small requests that would definitely be alleviated with a public/documented API...especially if they had results ordered by dates of availability...one can dream.
Originally posted by LuzRinggold
View Post
Comment