Thursday, January 02, 2014

Caching Youtube with storeurl.pl Method (100% Working in 2014)

Happy new year. Hopefully in 2014, we are all be healthy and success, always.
This is my first posting in 2014. Now I will discuss about caching google youtube video with storeurl.pl. Until now, caching youtube video with storeurl method, still 100% working, although not perfectly. Previously, I actually gave up, because 2 weeks before, caching youtube does not work with all methods (including methods storeurl). When the client runs youtube (through a server proxy), youtube error, as shown below. This possibility is from youtube site, there may be changes in the algorithm. Fortunately now no error anymore. Besides youtube, some other video sites can also be cached with this method. For example, dailymotion and so on.

an error occured, please try again later

How did I know, that was taken from a proxy server cache / squid?? not cached from the client browser. Yes, of course, from the experiment. I see the traffic from the proxy server.
And in this experiment, I use the old version of squid, 2.7 stable 9. I've been using some of the new squid version, but still no luck. There are still a lot of errors. And in the client using the Firefox browser 26.0. (chrome browser, it did not work.)
Ok. let's get started. This is only brief explanation. Network configuration as shown below. 

network configuration router & squid

Network configuration.
Install Ubuntu 13.10, and configure the ethernet.
For Linux installation, Ethernet settings, connection to the internet, I think you already know everything about it. Remember, there are 2 Lan Card used, 1 to the internet, and another heading toward lan / users.

Install Squid
apt-get update
apt-get install vim mc make gcc g++
Download squid and the example configuration from http://www.4shared.com/zip/bVgqjNQY/squid_n_shorewall_conf.html. And I assume the file that has been downloaded, is located in /usr/local. Then continue to the next steps.

cd /usr/local
unzip squid_n_shorewall_conf.zip
tar zxvf squid-2.7.STABLE9PATCH.tar.gz
cd squid-2.7.STABLE9
./configure --prefix=/usr --exec-prefix=/usr --bindir=/usr/sbin --sbindir=/usr/sbin --sysconfdir=/etc/squid --datadir=/usr/share/squid --includedir=/usr/include --libdir=/usr/lib --libexecdir=/usr/lib/squid --localstatedir=/var --sharedstatedir=/usr/com --mandir=/usr/share/man --infodir=/usr/share/info --x-includes=/usr/include --x-libraries=/usr/lib --enable-storeio=aufs --enable-removal-policies=heap,lru --disable-icmp --disable-delay-pools --enable-useragent-log --enable-referer-log --disable-wccp --enable-wccpv2 --disable-kill-parent-hack --enable-snmp --enable-cachemgr-hostname=localhost --enable-arp-acl --disable-htcp --disable-forw-via-db --disable-follow-x-forwarded-for --enable-cache-digests --disable-poll --enable-epoll --enable-linux-netfilter --disable-ident-lookups --enable-default-hostsfile=/etc/hosts --with-large-files --enable-http-violations --enable-gnuregex --enable-async-io=64 --with-aufs-threads=64 --with-pthreads --with-aio --enable-err-languages=English
make; make install

Continue to next steps below.
cd /usr/local/
mv storeurl.pl /etc/squid/
mv squid.conf /etc/squid/
mv squid /etc/init.d/
chmod 755 /etc/init.d/squid
chmod 755  /etc/squid/storeurl.pl
mkdir /cache
chown proxy:proxy /cache
mkdir /var/log/squid
touch /var/log/squid/access.log
touch /var/log/squid/cache.log
chown -Rf proxy:proxy /var/log/squid
update-rc.d squid defaults
squid -z
service squid restart

Until this point, squid should be ok. Next, install shorewall.

Install Shorewall.
cd /usr/local/
wget http://www.shorewall.com.au/CURRENT_STABLE_VERSION_IS_4.5/shorewall-4.5.21/shorewall-4.5.21.5.tgz
wget http://www.shorewall.com.au/CURRENT_STABLE_VERSION_IS_4.5/shorewall-4.5.21/shorewall-core-4.5.21.5.tgz
tar zxvf shorewall-core-4.5.21.5.tgz
tar zxvf shorewall-4.5.21.5.tgz
cd shorewall-core-4.5.21.5
./configure
./install.sh
cd ..
cd shorewall-4.5.21.5
./configure
./install.sh
Copy the example configuration that you downloaded from http://www.4shared.com/zip/bVgqjNQY/squid_n_shorewall_conf.html
cp /usr/local/shorewallconf/* /etc/shorewall/
update-rc.d shorewall defaults

At last, edit file in /etc/default/shorewall. Find startup=0, change to startup=1. If you think need it, you can change the shorewall configuration according your network. Done. Reboot. Then, It’s time to try from computer client. Using firefox, access to youtube video. There are some ways to check the youtube video successfully cached or not. 1, when reload video, 2, reload the browser, and 3, press enter on the URL. See the picture below. Other way open another browser from the same computer or on another computer, and then access the same video.
testing on youtube
Then, check in file access.log (tail –f /var/log/squid/access.log), if you find TCP_HIT when access google youtube video, like picture shown below, it means you successfully configure squid to cache youtube.
hit youtube

Note. Update 24-06-2014.
I use Operating system, ubuntu 14.04 server 32 bit, and it works. Just a slightly different configurations, because it uses a 32 bit operating system. Although, there is no option ZPH, but ZPH (default) is enable.
./configure --prefix=/usr   --exec-prefix=/usr   --bindir=/usr/sbin   --sbindir=/usr/sbin   --sysconfdir=/etc/squid   --datadir=/usr/share/squid   --includedir=/usr/include   --libdir=/usr/lib   --libexecdir=/usr/lib/squid   --localstatedir=/var   --sharedstatedir=/usr/com   --mandir=/usr/share/man   --infodir=/usr/share/info   --x-includes=/usr/include   --x-libraries=/usr/lib  --enable-carp    --enable-storeio=aufs,ufs   --enable-removal-policies=heap,lru   --disable-icmp   --disable-delay-pools    --enable-useragent-log   --enable-referer-log   --disable-wccp   --enable-wccpv2   --disable-kill-parent-hack   --enable-snmp   --enable-cachemgr-hostname=localhost   --enable-arp-acl   --disable-htcp  --disable-forw-via-db   --disable-follow-x-forwarded-for   --enable-cache-digests    --disable-poll   --enable-epoll   --enable-linux-netfilter   --disable-ident-lookups   --enable-default-hostsfile=/etc/hosts    --with-large-files   --enable-http-violations    --enable-gnuregex --enable-async-io=64 --with-aufs-threads=64  --with-pthreads --with-aio  --enable-err-languages=English


Reference :
http://ubuntu-indonesia.com/forums/ubbthreads.php/topics/131277/1
http://aacable.wordpress.com/2012/01/19/youtube-caching-with-squid-2-7-using-storeurl-pl/

46 comments:

  1. perfect Caching Youtube

    ReplyDelete
  2. it's a great artical thanks very much :)
    .....Please need 1 thing only i want to let my users use my proxy without configuring there browsers please replay?

    ReplyDelete
    Replies
    1. i have 2 lan in squid server eth0 ==> INTERNET ( 18.16.20.4/24 ) && eth1 ==> Mikrotik [(users)--( squid ip 192.168.10.1)]
      and i have 3 lan in mikrotik( version=5.20) eth0 == [Name:Lan (to my local users)]=> ip: 50.50.50.1/16 ----------- eth1 == [Name:WAN (INTERNET DSL)]=> ip: 18.16.20.6/24 ----------- eth2 == [Name:CACHE (connected to squid proxy server)]=> ip: 192.168.10.6/24

      Please Dont forget to replay to me i need this solution i needed it so much.....!!!! ?

      Delete
    2. I have made combination mikrotik and squid... look at my other posts...
      http://myconfigure.blogspot.com/2013/12/transparent-proxy-with-squid-3311.html
      http://myconfigure.blogspot.com/2013/03/transparent-squid-332-on-ubuntu-1210.html
      please read carefully,.. because there are links... and you must read them all..

      Delete
  3. mas, ini kenapa ya koq buat youtube nya selalu MISS

    192.168.100.200 - - [11/Jan/2014:16:25:45 +0700] "GET http://r6---sn-2uuxa3vh-cuis.googlevideo.com/videoplayback?algorithm=throttle-factor&burst=40&clen=1683777&cpn=yYmJ2nVxZy2QizT4&dur=105.975&expire=1389457781&factor=1.25&fexp=922518%2C938628%2C929432%2C916612%2C938630%2C936910%2C936913%2C907231%2C907240%2C921090&gir=yes&id=o-AEipw10jWtNObCbIowGlCD9mhnw3cLXK_J8YYFJKjWd_&ip=36.68.131.202&ipbits=0&itag=140&keepalive=yes&key=yt5&lmt=1388580726053529&ms=au&mt=1389432242&mv=m&pcm2fr=yes&range=475136-712703&ratebypass=yes&signature=63BE880D59FA5D62C68D591F239EB9E4E898A900.2ABB8DD6BA80D597A93182A641ED680DF2584BD7&source=youtube&sparams=algorithm%2Cburst%2Cclen%2Cdur%2Cfactor%2Cgir%2Cid%2Cip%2Cipbits%2Citag%2Clmt%2Cpcm2fr%2Csource%2Cupn%2Cexpire&sver=3&upn=cEdZiLKvH2s HTTP/1.1" 200 238160 TCP_MISS:DIRECT

    udah gitu, klo shorewall service nya lgi running, semua koneksi down. klo di stop, baru jalan.

    mohon tanggapannya mas

    ReplyDelete
    Replies
    1. 1. kalau akses youtube selalu berbeda, tentu miss... jika masih miss, walau mengakses youtube yang sama... artinya salah di storeurl.pl atau di squid.conf - nya... coba gunakan konfigurasi yang saya punya...
      2. kemungkinan shorewallnya salah konfigurasi...

      Delete
    2. 1.nah, disitu masalahnya mas. saya sudah ngikutin semua tahapan yang mas kasih. file yg saya pake pun yg mas punya. untuk caching yg lain, cntoh dari dailymotion, selalu HIT koq mas. cuma yg dari youtube yg MISS. untuk squid conf nya saya hanya merubah lokasi cache sesuai dengan network saya, yang lain tetap seperti aslinya.

      2.gunanya shorewall disini buat apa y mas?

      sekalian nanya mas,

      Delete
    3. 1. baru saja, saya coba lagi test dengan youtube,... masih tetap bisa...
      mungkin komputer untuk squidnya jangan yang jadul...
      tail -f /var/log/squid/cache.log --- >> udah tidak ada error ??
      apakah semua file youtube tidak bisa ??... perlu dicek kembali...
      2. shorewall untuk membuat firewall / iptables saja...

      Delete
    4. masih gak bisa juga mas,,, udah coba instal ulang lagi.
      komputer nya gak jadul koq, dual core, ram 2 gb, hdd 500gb.
      squid conf yg saya rubah, cuma buat cache dan penggunaan memory

      cache_dir aufs /cache1 70000 16 256
      cache_dir aufs /cache2 70000 16 256
      cache_dir aufs /cache3 70000 16 256
      cache_dir aufs /cache4 70000 16 256

      cache_mem 512 MB
      minimum_object_size 0 bytes
      maximum_object_size 1 GB

      klo store url nya tidak saya sentuh sama sekali. ato ada settingan di shorewall atau squid conf yg perlu saya rubah mas? saya pake mikrotik.

      mohon responnya mas???

      Delete
    5. udah saya coba lagi dari awal mas, tetap gak bisa untuk semua video youtube.

      di squid.conf saya hanya merubah buat cache dir dan memory use.

      store_dir_select_algorithm round-robin
      cache_dir aufs /cache1 70000 16 256
      cache_dir aufs /cache2 70000 16 256
      cache_dir aufs /cache3 70000 16 256
      cache_dir aufs /cache4 70000 16 256

      cache_mem 512 MB
      minimum_object_size 0 bytes
      maximum_object_size 1 GB

      apakah ada pengaruhnya mas memakai OS ubuntu yg salah. saya pakai ubuntu server 12.10 amd64 di mesin prosesor intel 32???

      Delete
    6. apakah sudah benar compile squid nya... apakah sudah ada referer-log ??
      kalau saya menggunakan core i5, dengan menggunakan virtual box... saya tadi coba kembali menggunakan ubuntu server 12.10... dan masih ok...
      mikrotik sejajar squid... atau didepan squid... ??
      cache1, cache2...--->> emang hardisknya banyak ??... cukup 1 saja... kalo hardisk hanya 1...

      Delete
  4. do i need the Shorewall for youtube cache or does youtube cache works without Shorewall to ?
    Regards
    Adi

    ReplyDelete
    Replies
    1. shorewall is for making firewall / iptables, making squid server is also as router.
      and then making squid as transparent proxy. so if you know iptables, you dont need shorewall.

      Delete
    2. thanks for your replay... i can work a bit with iptables, so for me shorewall not needed,
      version 3.4 of squid supports store-id.pl . have you tested it//

      Delete
    3. squid v 3.4 still has errors... i need more time to test it...

      Delete
  5. Mas kalau mau cache update patch game online, apakah config yang dari mas itu udah mencakup semua? atau ada tambahan setting lagi.. thx

    ReplyDelete
    Replies
    1. config -nya seperti sudah bagus untuk saat ini... termasuk untuk patch game online... ke depannya pastinya akan ada update-update game baru dan setting2 baru... tentu saja config-nya juga hrus menyesuaikan agar tetap mantapz... tongkrongin saja forum-forum untuk jika ingin lebih terupdate...

      Delete
    2. Ok mas, tutorial udh sy ikuti.. dan trims berhasil.. tp sy lihat di access.log sepertinya tidak ada hit untuk https/ ssl ya? yang terlihat hanya di http ..
      But for over all, thank u v m for this great post!

      Delete
    3. untuk https / ssl, ada settingannya tersendiri... tidak saya masukan ke dalam posting ini... https/ssl juga belum begitu bagus/perfect...

      Delete
  6. Udah sy coba di ubuntu 11.10, gk da masalah. Tp msh ada beberapa alamat video youtube yg gk muncul... Trims share nya ...

    ReplyDelete
  7. Mas, tutorial di atas saya sudah coba dan berhasil (trims). Padahal saya cuma pakai 1 ethernet tidak memakai 2 ethernet seperti yang sudah anda urai di awal post. Yang mau saya tanyakan:
    1. Bukankah squid di atas sudah transparant (sesuai squid.conf dari paket)? akan tetapi saya harus stel manual untuk setiap browser agar bisa menggunakan squid.
    2. Apakah squid bisa berjalan transparant dengan satu ethernet?
    3. Klo memang bisa, bagaimana pengaturan shorewallnya?
    Trimakasih atas postnya, dan semoga cepat dibalas.

    ReplyDelete
    Replies
    1. jawaban 1,2 &3,
      transparent itu, bahasa awamnya, (browser) si client jika menuju internet otomatis lewat proxy. (jadi tanpa disadari si client).
      anda menggunakan 1 ethernet, artinya si proxy sejajar dengan client ??
      artinya di gateway/router anda harus dibuat rules/aturan agar paket dari client seblum menuju internet harus lari ke proxy dulu....
      lihat contoh posting ini...
      http://myconfigure.blogspot.com/2013/12/transparent-proxy-with-squid-3311.html

      Delete
  8. mas, saya sudah berhasil buat tch hit nya,, tapi kok,, meskipun hit, bandwith juga kepake ya?? ato ada yang salah??
    dan juga youtube kok jalan nya setengah2 ya?? biasa nya kalo di pause bisa smpe akhir bar nya tp skrng di batasi,, itu knp ya??

    ReplyDelete
    Replies
    1. lihat cara saya melakukan test apakah berhasil atau tidak, di bagian akhir...
      bila perlu pastikan... dimatikan/dicabut kabel ke arah internet... agar benar2 pasti... bahwa saat menjalankan youtube yang kedua kalinya... dapatnya dari proxy....
      memang youtube jalannya setengah2... dari sononya... ada yang bilang file2 youtube,sengaja dibagi bagi...
      sebenarnya ini menguntungkan sekali untuk bandwidth2 yang kecil2... ndak kemakan semua bandwidth karena si youtube...

      Delete
  9. Cache Facebook emng g bsa y gan??
    Klo emng bsa gmn caranya

    ReplyDelete
    Replies
    1. masih tidak perfect...
      lihat disini... http://myconfigure.blogspot.com/2013/05/how-to-make-transparent-proxy-https-ssl.html

      Delete
  10. lapor gan buka web kaya modifikasi.com jadi stuk coba aja buka isinya nanti bakalan halaman itu aja yang tampil

    ReplyDelete
    Replies
    1. stuck karena pake proxy... salah satu kekurangannya ya seperti itu..,
      di bypass saja untuk situs tersebut... tidak lewat proxy.. disetting di squid.conf-nya...

      Delete
    2. makasih gan atas sarannya akhirnya nemu cara paling mudah untuk di kemudian hari di http://aacable.wordpress.com/2012/01/23/squid-howto-exclude-some-sites-exntension-from-caching/

      Delete
  11. does this configuration caches game patches?

    ReplyDelete
  12. game patches like dota 2 steam?

    ReplyDelete
  13. I read this post.This post is very informative and knowledgeable.i am very happy read this blog...
    More info:- Youtube Support



    ReplyDelete
  14. waktu compile keluar ini gan, kira2 kenapa yak, jadi gak ke cache itu si videonya...

    configure: WARNING: Unrecognized options: --enable-shared, --enable-static, --disable-esi, --enable-icap-client, --with-default-user, --enable-mit, --with-logdir, --enable-zph-qos, --with-filedescriptors, --enable-default-err-languages, --disable-hostname-checks, --enable-underscores

    ReplyDelete
    Replies
    1. itu kan udah jelas warningnya... ndak bener options-nya.... kok pake tanda koma (,) ???

      Delete
    2. woh tapi saya copas kok mas dari blog ini, coba saya koreksi satu2 lagi ya... tq berat atas infonya... oh iyah apakah jadi masalah jika command error diatas dengan tidak jalannya cache youtube tersebut?


      Makasih

      Delete
    3. ini yg saya jalankan... dan ada warning seperti komentar pertama saya... yg jadi pertanyaan apakah warning tersebut yg bisa bikin tidak jalan cache youtubenya yah?

      ./configure --prefix=/usr --exec-prefix=/usr --bindir=/usr/sbin --sbindir=/usr/sbin --sysconfdir=/etc/squid --datadir=/usr/share/squid --includedir=/usr/include --libdir=/usr/lib --libexecdir=/usr/lib/squid --localstatedir=/var --sharedstatedir=/usr/com --mandir=/usr/share/man --infodir=/usr/share/info --x-includes=/usr/include --x-libraries=/usr/lib --enable-shared=yes --enable-static=no --enable-carp --enable-storeio=aufs,ufs --enable-removal-policies=heap,lru --disable-icmp --disable-delay-pools --disable-esi --enable-icap-client --enable-useragent-log --enable-referer-log --disable-wccp --enable-wccpv2 --disable-kill-parent-hack --enable-snmp --enable-cachemgr-hostname=localhost --enable-arp-acl --disable-htcp --disable-forw-via-db --disable-follow-x-forwarded-for --enable-cache-digests --disable-poll --enable-epoll --enable-linux-netfilter --disable-ident-lookups --enable-default-hostsfile=/etc/hosts --with-default-user=squid --with-large-files --enable-mit=/usr --with-logdir=/data/log --enable-http-violations --enable-zph-qos --with-filedescriptors=65536 --enable-gnuregex --enable-async-io=64 --with-aufs-threads=64 --with-pthreads --with-aio --enable-default-err-languages=English --enable-err-languages=English --disable-hostname-checks --enable-underscores

      Delete
    4. jangan asal copas... perhatikan squid versi berapa yang digunakan... opsi tersebut khusus squid versi 2.79... perhatikan errornya...
      jika masih tidak mau... hilangkan saja beberapa opsi yang menurut anda tempatnya error...
      kalo juga tidak mau... masih error... ndak usah pake opsi... cukup pake perintah ./configure

      Delete
    5. waduw... saya ikutin banget smua yg ada dimari soalnya mas, sampe OSnya ajah sama ubuntu 13.10... klo hardware pake pc biasa sih, dual core, hd500gb, memory 4gb, pake osnya yg 64bit, ya lagi cobain pelan2 dari awal lagi... tnks atas masukannya

      Delete
    6. ini versinya... udah coba install baru lagi kenapa msh "TCP_MISS:DIRECT" yah, squid.conf dan storeull.pl gak ada yg saya rubah sama sekali... oh iyah saya gak pake shorewel sih... mungkin ada masukan lagi mas ?


      /usr/local# squid -v
      Squid Cache: Version 2.7.STABLE9
      configure options: '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/sbin' '--sbindir=/usr/sbin' '--sysconfdir=/etc/squid' '--datadir=/usr/share/squid' '--includedir=/usr/include' '--libdir=/usr/lib' '--libexecdir=/usr/lib/squid' '--localstatedir=/var' '--sharedstatedir=/usr/com' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--x-includes=/usr/include' '--x-libraries=/usr/lib' '--enable-shared=yes' '--enable-static=no' '--enable-carp' '--enable-storeio=aufs,ufs' '--enable-removal-policies=heap,lru' '--disable-icmp' '--disable-delay-pools' '--disable-esi' '--enable-icap-client' '--enable-useragent-log' '--enable-referer-log' '--disable-wccp' '--enable-wccpv2' '--disable-kill-parent-hack' '--enable-snmp' '--enable-cachemgr-hostname=localhost' '--enable-arp-acl' '--disable-htcp' '--disable-forw-via-db' '--disable-follow-x-forwarded-for' '--enable-cache-digests' '--disable-poll' '--enable-epoll' '--enable-linux-netfilter' '--disable-ident-lookups' '--enable-default-hostsfile=/etc/hosts' '--with-default-user=squid' '--with-large-files' '--enable-mit=/usr' '--with-logdir=/var/log/squid' '--enable-http-violations' '--enable-zph-qos' '--with-filedescriptors=65536' '--enable-gnuregex' '--enable-async-io=64' '--with-aufs-threads=64' '--with-pthreads' '--with-aio' '--enable-default-err-languages=English' '--enable-err-languages=English' '--disable-hostname-checks' '--enable-underscores'

      Delete
  15. Mas, kalo ./configure untuk ubuntu 14.04 64 bit ada ga mas, atau ga apa2 cukup pake yang 32 bit aja, soalnya punya saya miss miss melulu mas

    terimakasih mas, mohon bantuannya

    ReplyDelete
  16. Dear sir,
    everything is fine but the problem is : i can not install squid -z.The error message is LSB information is missing.How to solve this?.regards

    ReplyDelete
    Replies
    1. Reconfigure, then
      re-make and then
      re-install,...
      make sure there is no error when you do that...

      Delete
  17. I have canfig all the setting but now day all website work with https protocal and I unabel to cache all https website plz guide for the same

    ReplyDelete
  18. Dear i cannot download the files from 4shared link, can you upload them to alternate mirror, Regards

    ReplyDelete
  19. The top things about Clixsense's Work From Home Program:
    1. SURVEYS: 50+ 5-40 minute surveys paying out $0.5-$2.5 per survey.
    2. 12 Offer Walls - Make $0.5-$20 per offer.
    3. TASKS - Complete 1,000's of micro tasks from lots companies.

    ReplyDelete