Skip to main content

Wget: Mirroring Website

WGET is Ultimate Downloader. While you can use it simply to retrieve a single file from a server, it is much more powerful than that and offers many more features.One of its features is mirroring website.

Mirroring simply means downloading the entire contents of a Web site and uploading it somewhere else. So, we can make local copy of a website using wget.

The basic command to mirror website is

$ wget -m http://www.example.com

This command will allow you to download website but it can't make a complete local copy of a website. This command only download the page but the links still point to the real URLs including the images and stylesheets.

We have to add another option to make links not point to real URLs. Add option -k after -m

$ wget -m -k http://www.example.com

Now, the links will point to next page in our local copy website.

To delay your download, you can add option -w and how much time that you want to delay it. Because if we delay the mirroring, it can reduce the load of bandwidth of the site.

$ wget -m -k -w 25 http://www.example.com

From command above, -w 25 means we delay it for 25 seconds between requests. We can replace the number depend on our need. We can add suffix for the number like m for minutes, h for ours and d for days.

Now if you want to download your favorite website you can use wget to do that. This wget can be use on Linux, *Nix and Ms Windows


ref : http://fosswire.com

Popular posts from this blog

Extend a LVM partition after increasing its virtual disk on Virtualbox

No Linux machine at work? the easy way could be to simply install Virtualbox in one of the PC, create a VDI and install Ubuntu 14.04 in it. But the day will come when you need more space! Here is how to resize it: Resize Virtualbox VDI Open Virtualbox, make sure to shutdown your virtual machine. Then open a terminal (here windows): 1 2 3 4 cd “ C : \ Program Files \ Oracle \ VirtualBox ” VBoxManage list vms VBoxManage clonehd uuid 0000000000000000 backup . vdi VBoxManage modifyhd uuid 0000000000000000 -- resize 204800 This is the Virtualbox official manual  of available commands. And  this is a website  to convert Gb into Mb (–resize takes Mb as input, 200Gb = 204800Mb). Boot Gparted to resize the partition After a default install of Ubuntu Server on one physical disk, you will have a SWAP partition, and then an extended partition in which you’ll have your LVM partition (So 3 partitions).  Download Gparted  .iso,...

Website Using HTTPS Protocol

Last week, I changed protocol of my website from http to https. There were steps that i had to do. First, I installed openssl package and mod_ssl # yum install mod_ssl openssl Then, I configured the openssl  and created certificate # cd /etc/pki/tls/certs # make server.key # openssl rsa -in server.key -out server.key # make server.csr # openssl x509 -in server.csr -out server.crt -req -signkey server.key -days 3650 # chmod 400 server.* After that, I changed configuration of httpd/apache #nano /etc/httpd/conf.d/ssl.conf DocumentRoot "/var/www/html" ServerName www.server.world:443 SSLCertificateFile /etc/pki/tls/certs/server.crt SSLCertificateKeyFile /etc/pki/tls/certs/server.key Last, I restarted the webserver #service httpd restart That's all what i did last week to change my website protocol from http to https.

Privacy Policy

Another Pratama Privacy Statement What follows is the Privacy Statement for all Another Pratama websites (a.k.a. blogs) including all the websites run under the computer.pratama.us domain. Please read this statement regarding our blogs. If you have questions please ask us via our contact form. Email Addresses You may choose to add your email address to our contact list via the forms on our websites. We agree that we will never share you email with any third party and that we will remove your email at your request. We don’t currently send advertising via email, but in the future our email may contain advertisements and we may send dedicated email messages from our advertisers without revealing your email addresses to them. If you have any problem removing your email address please contact us via our contact form. Ownership of Information Another Pratama is the sole owner of any information collected on our websites. Comments/Message Boards Most Another Pratama websites contain com...