Fan Fiction Fetcher, now with added LWP

[kerravonsen]

For those of you who were interested in my fanfic-fetching perl script, I've just released version 0.16 of WWW-FetchStory (http://search.cpan.org/~rubykat/WWW-FetchStory-0.16/) (well, that will be the URL when CPAN finishes processing it).

The big news: it no longer depends on wget! It uses the LWP perl module instead. This means that MS-Windows users should be able to use the script (fingers crossed).
I have retained the option to use wget, because some sites work with wget that don't work with LWP.(*)

There are a bunch of other improvements, and another new fetcher (Project Gutenberg), but the LWP stuff is the important bit.

(*) I have spent HOURS trying to get LWP + Cookies to work with LiveJournal, but no joy, and I have given up. LWP and Cookies work with other sites (I tried it on Ashwinder) but not with LJ. (throws hands in air) Anyone who can figure out why the cookies sometimes work and sometimes don't, that would be great. I have pored over debugging output, I have made observations with wireshark... The only difference seems to be that wget sends the right cookies and LWP only sends some of the right cookies.

Comments

[foxfirefey]

Hmm. Does it work with DW but not LJ? I have a guess but I'm not sure, and if it's true it'd probably be the same for both platforms.

[afuna]

Hmm, could it have something to do with adult content?

OH! Also, have you viewed the LJ jounal with the browser you exported the cookies form? There's a per-journal cookie that needs to be set (unless that's what you meant by per-session cookie)

[kareila]

The one time I scripted something similar to this, I prompted interactively for the LJ password and had the script do its own login. The resulting cookie worked with LWP for my own purposes (exporting locked posts from my own journal). However, that was around 2004 or so and LJ has probably changed quite a bit since then.

Perhaps they have done something that specifically blocks the LWP user agent. If that's the case, you could see if changing the user agent string helps. Good luck!

[brownbetty]

I just spent 38 hours trying to figure out how to make perl give me a nice epub from lj, only to discover you have already done so.

Which is super, as my fluency with perl is about on the level "Oh shit, hashes contain *references* to arrays? You can do that?"