Caching Youtube with storeurl.pl Method (100% Working in 2014)

Happy new year. Hopefully in 2014, we are all be healthy and success, always.

This is my first posting in 2014. Now I will discuss about caching google youtube video with storeurl.pl. Until now, caching youtube video with storeurl method, still 100% working, although not perfectly. Previously, I actually gave up, because 2 weeks before, caching youtube does not work with all methods (including methods storeurl). When the client runs youtube (through a server proxy), youtube error, as shown below. This possibility is from youtube site, there may be changes in the algorithm. Fortunately now no error anymore. Besides youtube, some other video sites can also be cached with this method. For example, dailymotion and so on.

How did I know, that was taken from a proxy server cache / squid?? not cached from the client browser. Yes, of course, from the experiment. I see the traffic from the proxy server.
And in this experiment, I use the old version of squid, 2.7 stable 9. I've been using some of the new squid version, but still no luck. There are still a lot of errors. And in the client using the Firefox browser 26.0. (chrome browser, it did not work.)

Ok. let's get started. This is only brief explanation. Network configuration as shown below.   

Network configuration.

Install Ubuntu 13.10, and configure the ethernet.

For Linux installation, Ethernet settings, connection to the internet, I think you already know everything about it. Remember, there are 2 Lan Card used, 1 to the internet, and another heading toward lan / users.

Install Squid

apt-get update

apt-get install vim mc make gcc g++

Download squid and the example configuration from http://www.4shared.com/zip/bVgqjNQY/squid_n_shorewall_conf.html. And I assume the file that has been downloaded, is located in /usr/local. Then continue to the next steps.

cd /usr/local

unzip squid_n_shorewall_conf.zip

tar zxvf squid-2.7.STABLE9PATCH.tar.gz

cd squid-2.7.STABLE9

./configure --prefix=/usr --exec-prefix=/usr --bindir=/usr/sbin --sbindir=/usr/sbin --sysconfdir=/etc/squid --datadir=/usr/share/squid --includedir=/usr/include --libdir=/usr/lib --libexecdir=/usr/lib/squid --localstatedir=/var --sharedstatedir=/usr/com --mandir=/usr/share/man --infodir=/usr/share/info --x-includes=/usr/include --x-libraries=/usr/lib --enable-storeio=aufs --enable-removal-policies=heap,lru --disable-icmp --disable-delay-pools --enable-useragent-log --enable-referer-log --disable-wccp --enable-wccpv2 --disable-kill-parent-hack --enable-snmp --enable-cachemgr-hostname=localhost --enable-arp-acl --disable-htcp --disable-forw-via-db --disable-follow-x-forwarded-for --enable-cache-digests --disable-poll --enable-epoll --enable-linux-netfilter --disable-ident-lookups --enable-default-hostsfile=/etc/hosts --with-large-files --enable-http-violations --enable-gnuregex --enable-async-io=64 --with-aufs-threads=64 --with-pthreads --with-aio --enable-err-languages=English

make; make install

Continue to next steps below.

cd /usr/local/

mv storeurl.pl /etc/squid/

mv squid.conf /etc/squid/

mv squid /etc/init.d/

chmod 755 /etc/init.d/squid

chmod 755  /etc/squid/storeurl.pl

mkdir /cache

chown proxy:proxy /cache

mkdir /var/log/squid

touch /var/log/squid/access.log

touch /var/log/squid/cache.log

chown -Rf proxy:proxy /var/log/squid

update-rc.d squid defaults

squid -z

service squid restart

Until this point, squid should be ok. Next, install shorewall.

Install Shorewall.

cd /usr/local/

wget http://www.shorewall.com.au/CURRENT_STABLE_VERSION_IS_4.5/shorewall-4.5.21/shorewall-4.5.21.5.tgz

wget http://www.shorewall.com.au/CURRENT_STABLE_VERSION_IS_4.5/shorewall-4.5.21/shorewall-core-4.5.21.5.tgz

tar zxvf shorewall-core-4.5.21.5.tgz

tar zxvf shorewall-4.5.21.5.tgz

cd shorewall-core-4.5.21.5

./configure

./install.sh

cd ..

cd shorewall-4.5.21.5

./configure

./install.sh

Copy the example configuration that you downloaded from http://www.4shared.com/zip/bVgqjNQY/squid_n_shorewall_conf.html

cp /usr/local/shorewallconf/* /etc/shorewall/

update-rc.d shorewall defaults

At last, edit file in /etc/default/shorewall. Find startup=0, change to startup=1. If you think need it, you can change the shorewall configuration according your network. Done. Reboot. Then, It’s time to try from computer client. Using firefox, access to youtube video. There are some ways to check the youtube video successfully cached or not. 1, when reload video, 2, reload the browser, and 3, press enter on the URL. See the picture below. Other way open another browser from the same computer or on another computer, and then access the same video.

Then, check in file access.log (tail –f /var/log/squid/access.log), if you find TCP_HIT when access google youtube video, like picture shown below, it means you successfully configure squid to cache youtube.

Note. Update 24-06-2014.
I use Operating system, ubuntu 14.04 server 32 bit, and it works. Just a slightly different configurations, because it uses a 32 bit operating system. Although, there is no option ZPH, but ZPH (default) is enable.

./configure --prefix=/usr   --exec-prefix=/usr   --bindir=/usr/sbin   --sbindir=/usr/sbin   --sysconfdir=/etc/squid   --datadir=/usr/share/squid   --includedir=/usr/include   --libdir=/usr/lib   --libexecdir=/usr/lib/squid   --localstatedir=/var   --sharedstatedir=/usr/com   --mandir=/usr/share/man   --infodir=/usr/share/info   --x-includes=/usr/include   --x-libraries=/usr/lib  --enable-carp    --enable-storeio=aufs,ufs   --enable-removal-policies=heap,lru   --disable-icmp   --disable-delay-pools    --enable-useragent-log   --enable-referer-log   --disable-wccp   --enable-wccpv2   --disable-kill-parent-hack   --enable-snmp   --enable-cachemgr-hostname=localhost   --enable-arp-acl   --disable-htcp  --disable-forw-via-db   --disable-follow-x-forwarded-for   --enable-cache-digests    --disable-poll   --enable-epoll   --enable-linux-netfilter   --disable-ident-lookups   --enable-default-hostsfile=/etc/hosts    --with-large-files   --enable-http-violations    --enable-gnuregex --enable-async-io=64 --with-aufs-threads=64  --with-pthreads --with-aio  --enable-err-languages=English

Reference :
http://ubuntu-indonesia.com/forums/ubbthreads.php/topics/131277/1
http://aacable.wordpress.com/2012/01/19/youtube-caching-with-squid-2-7-using-storeurl-pl/