Kathryn A. (![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png)
kerravonsen) wrote in ![[community profile]](https://www.dreamwidth.org/img/silk/identity/community.png)
perl2011-09-10 11:07 am
kerravonsen) wrote in perl2011-09-10 11:07 am
Fan Fiction Fetcher, now with added LWP
For those of you who were interested in my fanfic-fetching perl script, I've just released version 0.16 of WWW-FetchStory (http://search.cpan.org/~rubykat/WWW-FetchStory-0.16/) (well, that will be the URL when CPAN finishes processing it).
The big news: it no longer depends on wget! It uses the LWP perl module instead. This means that MS-Windows users should be able to use the script (fingers crossed).
I have retained the option to use wget, because some sites work with wget that don't work with LWP.(*)
There are a bunch of other improvements, and another new fetcher (Project Gutenberg), but the LWP stuff is the important bit.
(*) I have spent HOURS trying to get LWP + Cookies to work with LiveJournal, but no joy, and I have given up. LWP and Cookies work with other sites (I tried it on Ashwinder) but not with LJ. (throws hands in air) Anyone who can figure out why the cookies sometimes work and sometimes don't, that would be great. I have pored over debugging output, I have made observations with wireshark... The only difference seems to be that wget sends the right cookies and LWP only sends some of the right cookies.
The big news: it no longer depends on wget! It uses the LWP perl module instead. This means that MS-Windows users should be able to use the script (fingers crossed).
I have retained the option to use wget, because some sites work with wget that don't work with LWP.(*)
There are a bunch of other improvements, and another new fetcher (Project Gutenberg), but the LWP stuff is the important bit.
(*) I have spent HOURS trying to get LWP + Cookies to work with LiveJournal, but no joy, and I have given up. LWP and Cookies work with other sites (I tried it on Ashwinder) but not with LJ. (throws hands in air) Anyone who can figure out why the cookies sometimes work and sometimes don't, that would be great. I have pored over debugging output, I have made observations with wireshark... The only difference seems to be that wget sends the right cookies and LWP only sends some of the right cookies.
no subject
no subject
I have just now tested the script to fetch a locked post from Dreamwidth, and the same locked post made by the same person (for whom I have the same level of access on their DW and their LJ journals), both using the same exported cookie file which contains the appropriate session cookies... and it succeeds for Dreamwidth, but fails for Livejournal.
Fails in the sense that Dreamwidth gives me the page I want, and Livejournal gives me a "please log in" page.
8-(