14 Oct
2011
14 Oct
'11
12:37 a.m.
Just for the record.
The correct way to mirror a site with links corrected is
wget -c -m -k -np -e robots=off URL
Seems most people have problem remembering this incantation.
--
EMBnet/CNB
Scientific Computing Service
Solving all your computer needs for Scientific
Research.
http://bioportal.cnb.csic.es
http://www.es.embnet.org
14 Oct
14 Oct
12:43 a.m.
On Thu, Oct 13, 2011 at 08:37:27PM +0200, Jose R. Valverde wrote:
Just for the record.
The correct way to mirror a site with links corrected is
wget -c -m -k -np -e robots=off URL
Seems most people have problem remembering this incantation.
Wouldn't it be nice if it were
wget --mirror URL
?
--
---
Larry McVoy lm at bitmover.com http://www.bitkeeper.com
12:51 a.m.
I like shell scripts for hard to remember things like that.
Call it wget_mirror.sh or something like that:
#!/bin/bash
if [ $# -ne 1 ] ; then echo ERRR; exit 1 ; else url=$1 ; fi
wget -c -m -k -np -e robots=off $url
( that was off the top of my head and has NOT been tested ... it might have a typo or two
:)
- Derrik
On Oct 13, 2011, at 02:43 PM, Larry McVoy <lm(a)bitmover.com> wrote:
On Thu, Oct 13, 2011 at 08:37:27PM +0200, Jose R.
Valverde wrote:
Just for the record.
The correct way to mirror a site with links corrected is
wget -c -m -k -np -e robots=off URL
Seems most people have problem remembering this incantation.
Wouldn't it be nice if it were
wget --mirror URL
?
--
---
Larry McVoy lm at bitmover.com http://www.bitkeeper.com
_______________________________________________
TUHS mailing list
TUHS(a)minnie.tuhs.org
https://minnie.tuhs.org/mailman/listinfo/tuhs 
12:53 a.m.
Computing woudln't be the same :-)
--
Saludos - Greetings - Freundliche Grüße - Salutations
Sergio Pedraja
<http://www.linkedin.com/in/sergiopedraja>
twitter: @sergio_pedraja
http://www.linkedin.com/in/sergiopedraja
http://www.quora.com/Sergio-Pedraja
-----
2011/10/13 Larry McVoy <lm(a)bitmover.com>
On Thu, Oct 13, 2011 at 08:37:27PM +0200, Jose R.
Valverde wrote:
Just for the record.
The correct way to mirror a site with links corrected is
wget -c -m -k -np -e robots=off URL
Seems most people have problem remembering this incantation.
Wouldn't it be nice if it were
wget --mirror URL
?
--
---
Larry McVoy lm at bitmover.com
http://www.bitkeeper.com
_______________________________________________
TUHS mailing list
TUHS(a)minnie.tuhs.org
https://minnie.tuhs.org/mailman/listinfo/tuhs
1:35 a.m.
On Thu, Oct 13, 2011 at 1:43 PM, Larry McVoy <lm(a)bitmover.com> wrote:
On Thu, Oct 13, 2011 at 08:37:27PM +0200, Jose R.
Valverde wrote:
for a complete mirror, yes. but if you want to make sure that you
don't download any malware:
wget --mirror --evil=false
Just for the record.
The correct way to mirror a site with links corrected is
wget -c -m -k -np -e robots=off URL
Seems most people have problem remembering this incantation.
Wouldn't it be nice if it were
wget --mirror URL
9:59 p.m.
That is already included: -m == --mirror
But --mirror (or -m) does not include -k (convert links to local after
the transfer) nor -np (do not follow links upwards the parent directory),
nor an instruction to ignore 'robots.txt'.
The magic incantation I submitted will only download down the hierarchy,
in spite of robots.txt and fixing links, all three problems reported in
the thread.
Of course it is not polite to ignore robots.txt, but sometimes it may be
justified.
j
On Thu, 13 Oct 2011 11:43:37 -0700
Larry McVoy <lm(a)bitmover.com> wrote:
On Thu, Oct 13, 2011 at 08:37:27PM +0200, Jose R.
Valverde wrote:
--
EMBnet/CNB
Scientific Computing Service
Solving all your computer needs for Scientific
Research.
http://bioportal.cnb.csic.es
http://www.es.embnet.org
Just for the record.
The correct way to mirror a site with links corrected is
wget -c -m -k -np -e robots=off URL
Seems most people have problem remembering this incantation.
Wouldn't it be nice if it were
wget --mirror URL
?
--
---
Larry McVoy lm at bitmover.com http://www.bitkeeper.com
4817
days inactive
4818
days old
5 comments
5 participants
participants (5)
-
A. P. Garcia -
Derrik Walker -
Jose R. Valverde -
Larry McVoy -
SPC