Thursday, March 07, 2013

Cache Youtube Video with Squid 3.3.2 and Nginx

Note :
source : http://aacable.wordpress.com/2012/08/13/youtube-caching-with-squid-nginx/
I change some command, according with my network environment, and my experience.

( Update January 2th, 2014). Look at another posting, cache youtube video with storeurl method.  http://myconfigure.blogspot.com/2014/01/caching-youtube-with-storeurlpl-method.html
I assume you've installed :
1. Ubuntu 12.10,
2. Squid 3.3.2
3. Mikrotik, options to make it better
All of the above is installed and running nomally.If you have not installed some of the things mentioned above. Stop reading this section, and look for the section on this blog, about how to install squid. For example, you can read the
"Transparent Proxy with Squid 3.3.2 on Ubuntu 12.10 + Shorewall + Mikrotik".
Ok. Let's begin.

1. Add some command line at squid.conf
Add

# Youtube Cache Section [zaib]
url_rewrite_program /etc/nginx/nginx.rb
url_rewrite_host_header off
acl youtube_videos url_regex -i ^http://[^/]+\.youtube\.com/videoplayback\?
acl range_request req_header Range .
acl begin_param url_regex -i [?&]begin=
acl id_param url_regex -i [?&]id=
acl itag_param url_regex -i [?&]itag=
acl sver3_param url_regex -i [?&]sver=3
cache_peer 127.0.0.1 parent 8081 0 proxy-only no-query connect-timeout=10
cache_peer_access 127.0.0.1 allow youtube_videos id_param itag_param sver3_param !begin_param !range_request
cache_peer_access 127.0.0.1 deny all

Save & Exit.

2.      Install nginx
apt-get install nginx
vim /etc/nginx/nginx.conf

Remove all lines and paste the following data
user www-data;
worker_processes 4;
pid /var/run/nginx.pid;
events {
worker_connections 768;
}
http {
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
gzip on;
gzip_static on;
gzip_comp_level 6;
gzip_disable .msie6.;
gzip_vary on;
gzip_types text/plain text/css text/xml text/javascript application/json application/x-javascript application/xml application/xml+rss;
gzip_proxied expired no-cache no-store private auth;
gzip_buffers 16 8k;
gzip_http_version 1.1;
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
# starting youtube section
server {
listen 127.0.0.1:8081;
location / {
root /usr/local/www/nginx_cache/files;
#try_files "/id=$arg_id.itag=$arg_itag" @proxy_youtube; # Old one
#try_files  "$uri" "/id=$arg_id.itag=$arg_itag.flv" "/id=$arg_id-range=$arg_range.itag=$arg_itag.flv" @proxy_youtube; #old2
try_files "/id=$arg_id.itag=$arg_itag.range=$arg_range.algo=$arg_algorithm" @proxy_youtube;
}
location @proxy_youtube {
resolver 8.8.8.8;
proxy_pass http://$host$request_uri;
proxy_temp_path "/usr/local/www/nginx_cache/tmp";
#proxy_store "/usr/local/www/nginx_cache/files/id=$arg_id.itag=$arg_itag"; # Old 1
proxy_store "/usr/local/www/nginx_cache/files/id=$arg_id.itag=$arg_itag.range=$arg_range.algo=$arg_algorithm";
proxy_ignore_client_abort off;
proxy_method GET;
proxy_set_header X-YouTube-Cache "aacable@hotmail.com";
proxy_set_header Accept "video/*";
proxy_set_header User-Agent "YouTube Cacher (nginx)";
proxy_set_header Accept-Encoding "";
proxy_set_header Accept-Language "";
proxy_set_header Accept-Charset "";
proxy_set_header Cache-Control "";}
}
}
Save & Exit.

3.      Create directories to hold cache files
mkdir /usr/local/www
mkdir /usr/local/www/nginx_cache
mkdir /usr/local/www/nginx_cache/tmp
mkdir /usr/local/www/nginx_cache/files
chown www-data /usr/local/www/nginx_cache/files/ -Rf

4.      Create nginx .rb file
touch /etc/nginx/nginx.rb
chmod 755 /etc/nginx/nginx.rb
nano /etc/nginx/nginx.rb

Paste the following data in this newly created file
#!/usr/bin/env ruby1.9.1
require "syslog"
require "base64"

class SquidRequest
attr_accessor :url, :user
attr_reader :client_ip, :method

def method=(s)
@method = s.downcase
end

def client_ip=(s)
@client_ip = s.split('/').first
end
end

def read_requests
# URL <SP> client_ip "/" fqdn <SP> user <SP> method [<SP> kvpairs]<NL>
STDIN.each_line do |ln|
r = SquidRequest.new
r.url, r.client_ip, r.user, r.method, *dummy = ln.rstrip.split(' ')
(STDOUT << "#{yield r}\n").flush
end
end

def log(msg)
Syslog.log(Syslog::LOG_ERR, "%s", msg)
end

def main
Syslog.open('nginx.rb', Syslog::LOG_PID)
log("Started")

read_requests do |r|
if r.method == 'get' && r.url !~ /[?&]begin=/ && r.url =~ %r{\Ahttp://[^/]+\.youtube\.com/(videoplayback\?.*)\z}
log("YouTube Video [#{r.url}].")
"http://127.0.0.1:8081/#{$1}"
else
r.url
end
end
end
main
Save & Exit.

5.      Install ruby
apt-get install ruby
service squid restart
service nginx restart

Done. It's time to try. Through the client computer to browse the site youtube. If everything goes well, then video cache files can be found at /usr/local/www/nginx_cache/files
 
cache youtube with squid and nginx

If you find the file like the image above, it means you've successfully to cache youtube video. Congratulations, :)

Note Update : Now not working... I'm still looking to fix it. Please command if you have any information.

39 comments:

  1. Replies
    1. ...as a side effect, now I can use the cached files and play them offline with VLC!

      Delete
    2. good, I'm happy to know it... :)

      Delete
  2. I'm going try and rebuild an old squid box just to test this (if it works its an upgrade :-) )...
    i guess any linux with a working squid3 will just do if i skip 3.mikrotik options...

    I'll update my work later...

    ReplyDelete
    Replies
    1. hmm... I did build a working squid proxy. did all the instructions above.

      System is an AMD athlon 64 with 2 Gigs of DDR2 RAM. .. not quite old cpu. Linux flavour use is Zorin-32 bit. There are 2 HDD. OS hdd is a 160GB HDD and Storage is a 1TB hdd. Files and folders are properly configured.

      It works youtube got cached but.... system seems to be unstable... one moment i can cache video. the next moment it'll tell me it cant load notice... (some time i still can hear sound of the same video)... i guess its something to do with video quality option in youtube.

      the performance of squid proxy is also affected. i get broken picture link in throughout youtube pages and even other dynamic web pages like forums.

      Last i commented out all the squid.conf above things server act as a normal proxy.

      I guess i need more RAM....

      Delete
    2. squid 3.3.3 instance and installation taken from somewhere in this blog. its working.

      hmm... guess before upgrading ram i my best test Xubuntu and work from there.

      Delete
    3. cache video youtube with squid and nginx, is not the perfect way. sometimes some video youtube can not be cached/saved. maybe 70-80% can be cache. You must know this is illegal to cache video youtube. sure. youtube will renew away to protect their video. and the future, this way, cache video with squid and nginx will not work.

      Delete
    4. Solve problem (i hope)... i took codes from http://fendyhussain.wordpress.com/2012/11/30/youtube-caching-with-squid-nginx/

      it seems that his using an older version of Ruby in the rb file... things works currently... with extra 2 Gigs of ram... i try if i can lower back the ram to 2gigs again and do more test.

      Delete
    5. :) Depending ruby version. In my case, I used ruby version 1.9.1, so I changed head code.
      #!/usr/bin/env ruby1.9.1
      so next, if you are using the latest version of ruby, please adjust well the head code.

      Delete
  3. hmm... running to a wall...
    tried a fresh install of zorin OS...
    installed squid3 (version 3.1.19) from apt-get
    tweak squid.conf to allow http access. tested proxy setting to localhost and another computer to this proxy works.
    done nginx and ruby
    stuck after inserting # Youtube Cache Section [zaib] and other stuff to squid.conf
    squid3 service wount start.... if start... nothing happens.. browser says proxy unreachable..
    will work again after i commented out all the youtube cache part in squid.conf...

    guess i have to go with a squid 3.3.2
    i'll purge the current installed squid3 and install squid 3.3.2

    ReplyDelete
    Replies
    1. :), absolutely right, it is not compatible with the squid version 3.1.xx

      Delete
  4. The Movies that are stored in / usr / local / www / nginx_cache / files
    Is there any way Squid go deleting this file?

    ReplyDelete
    Replies
    1. deleting file /movie is manually, it's difficult to delete which movie we want it.

      Delete
    2. My old ones possible?

      Delete
    3. It's difficult, bcause video in cache is no title or clear identity.

      Delete
  5. I Have Tproxy and no load videos:
    iptables -t mangle -N DIVERT
    iptables -t mangle -A DIVERT -j MARK --set-mark 1
    iptables -t mangle -A DIVERT -j ACCEPT
    iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
    iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY --tproxy-mark 0x1/0x1 --on-port 3129

    Possible error using port 80?

    ReplyDelete
    Replies
    1. First, make sure squid is running well, then continue to install nginx and other script, to cache youtube video.

      Delete
    2. Squid without url_rewrite_program opens youtube movies correctly. but if I enable these settings does not work. NOTE: If I set up on another machine to access this proxy using port 3128 that is not redirect port 80 on the firewall works caching videos.

      Delete
    3. do you install squid version 3.3 ?? or older version ??

      Delete
    4. Yes, using 3.3

      Delete
    5. maybe your firewall/iptables is missing... look at http://myconfigure.blogspot.com/2013/03/transparent-squid-332-on-ubuntu-1210.html. Transparent proxy with option intercept, not tproxy. I'm going to experiment with tproxy. If successfull, I will post here...

      Delete
    6. Utilizing Tproxy error on Log:
      TCP_MISS/000 0 GET http://r5---sn-jucj-0qpe.c.youtube.com/videoplayback? - DIRECT/127.0.0.1

      Delete
    7. Try using intercept / transparent, for making transparent proxy.
      I'm not yet successfully using tproxy in centos...

      Delete
  6. can u upload ur squidconf ?
    many thx before

    ReplyDelete
    Replies
    1. http://myconfigure.blogspot.com/2013/03/squid-332-328-example-squidconf.html

      Delete
  7. Thank you for the guide, it is very helpful! Unfortunately I've got a problem with it. When I direct the browser to go through my proxy YouTube videos don't play smoothly, I'll try to explain... When I play a video, it starts playing and it plays ok until it finishes playing the first "chunk" of the video it has loaded, then it pauses for a couple of seconds then it loads the next "chunk" after it stars playing and so on. That doesn't happen if I try playing the video without the proxy, so it's not slow connection or something like that. Is there some configuration that I'm missing and which would make the video load without the pauses? Your help would be much appreciated.

    ReplyDelete
    Replies
    1. Your proxy make it slow your connection internet. Try running another video which quality is medium. Whether all the youtube video running slow? What version squid do you use ? Is it your configuration same like mine ? Are you using the old computer ? memory ? hardisk ? Generally proxy slow because memory and harddisk.

      Delete
    2. Hello Roel, I haven't been using a proxy before and I just installed it for a test on some not-at-all-powerfull computer with criticaly low memory. I haven't thought that could affect the squid performance so much , but now when you mentioned it I guess that is the cause since I followd your guide closely and no one else mentioned such a problem. Thank you for your reply, I will upgrade the computer and then see how it works. Best regards!

      Delete
  8. who can configure squid with good video cache on ubuntu 12.4, they tell me I will pay you well,
    this is my email spidernet@hotmail.de

    ReplyDelete
    Replies
    1. hahaha... nice joke...
      if you really need help, you can email me, roelngerii@gmail.com...

      Delete
  9. I did all steps but no youtube cache I dont know why??
    how can I see my squid version? I think it is 3.1

    ReplyDelete
    Replies
    1. check your squid with command
      squid -v

      Delete
  10. First, make sure squid is running well, then continue to install nginx and other script, to cache youtube video.Youtube Proxy

    ReplyDelete
  11. is it still working???/ coz there's an update note below that said it's nor working anymore

    ReplyDelete
    Replies
    1. not working anymore... please look at here http://myconfigure.blogspot.com/2014/01/caching-youtube-with-storeurlpl-method.html... for another method....

      Delete
  12. Youtube is a video website in which we can watch, listen and share the videos to others. There will be Live Streaming an update from the YouTube Product....
    Youtube Help

    ReplyDelete
  13. ./nginx.rb:14:in `client_ip=': undefined method `split' for nil:NilClass (NoMethodError)

    Does anyone could give me hand??

    ruby 1.9.3p484 (2013-11-22 revision 43786) [amd64-freebsd9]

    ReplyDelete
  14. this script does not work with plz do not visit this website

    ReplyDelete