Cache Youtube Video with Squid 3.3.2 and Nginx

Note :

source : http://aacable.wordpress.com/2012/08/13/youtube-caching-with-squid-nginx/

I change some command, according with my network environment, and my experience.

( Update January 2th, 2014). Look at another posting, cache youtube video with storeurl method.  http://myconfigure.blogspot.com/2014/01/caching-youtube-with-storeurlpl-method.html

I assume you've installed :
1. Ubuntu 12.10,
2. Squid 3.3.2
3. Mikrotik, options to make it better
All of the above is installed and running nomally.If you have not installed some of the things mentioned above. Stop reading this section, and look for the section on this blog, about how to install squid. For example, you can read the
"Transparent Proxy with Squid 3.3.2 on Ubuntu 12.10 + Shorewall + Mikrotik".

Ok. Let's begin.

1. Add some command line at squid.conf

Add

Transparent Proxy with Squid 3.3.2 on Ubuntu 12.10 + Shorewall + Mikrotik

 

This post explains how to setup a transparent proxy (intercept proxy) by combining mikrotik and squid eksternal. Look at a network above. a mikrotik as router, coupled with a proxy server (squid as transparent proxy). In the previous article has been discussed about this network. You can read at "Ubuntu 12.10 + Squid (Transparent Proxy) + Shorewall + Mikrotik". But the squid that was used is version 3.1.23. If you replace it with squid version 3.2.8 or 3.3.2, would be an error. The reason is, the proxy misrepresented the "redirect".
So.  If you want to use squid 3.2.8 or 3.3.2 as a transparent proxy, in a network as above, then there are some things that should be changed, the configuration on the router mikrotik and the Squid.
A few steps below, would be similar to the steps at "Transparent Proxy Server, Squid 3.3.2 / 3.2.8 on Ubuntu 12.10 + Shorewall"
We begin by configuring the Ethernet.

vi /etc/network/interfaces

Transparent Proxy Server, Squid 3.3.2 / 3.2.8 on Ubuntu 12.10 + Shorewall

Squid 3.1.xx (latest version 3.1.23), is a very simple and is used to make transparent / intercept. While the squid 3.3.2 / 3.2.8, to make transparent / intercept configuration is a bit different from the version of squid 3.1.23.  In this blog, I will explain how to create a transparent / intercept in two ways, the first is squid as well as a router, and the second, Mikrotik as a router coupled with squid external. Let's get to start, a transparent proxy server with squid as well as router.

Network configuration.
After installing the new Ubuntu, configure the ethernet. In my case, Ethernet to the Internet is dhcp, and another ethernet to the LAN, using the IP address 10.10.10.1. Of course, customize your network environment.

vi /etc/network/interfaces

Squid 3.3.2 / 3.2.8 : Example Squid.conf

#WELCOME TO SQUID
#       ----------------------------

acl snmppublic snmp_community public

acl bamboe src 192.168.10.0/24

#uncomment noway url, if necessary.
#acl noway url_regex -i "/etc/squid/noway"

acl SSL_ports port 443

acl Safe_ports port 80          # http

acl Safe_ports port 1935          # http

acl Safe_ports port 21          # ftp

acl Safe_ports port 443         # https

acl Safe_ports port 70          # gopher

acl Safe_ports port 210         # wais

acl Safe_ports port 1025-65535  # unregistered ports

acl Safe_ports port 280         # http-mgmt

acl Safe_ports port 488         # gss-http

acl Safe_ports port 591         # filemaker

acl Safe_ports port 777         # multiling http

CactiEZ, Cacti Make Easy

CactiEZ is a self installing Linux Distribution based off CentOS that sets up and configures a customized Cacti install. Everything is designed to be completely automated and working directly out of the box. This compact distro is loaded with extra features such as Syslog and Netflow data collection, Weathermaps, Reports, Auto Discovery, and much more!

Features

• Built on Centos 6 x64

Ubuntu Server as Domain Member Server

I have created a network like the picture above, a simulation, using VMWare 9.0, because of the limitations of the equipment. It would be nice if you could use in the real network. I assume you are already familiar with using linux and windows, including windows server 2008. And the network you created, has met the following conditions:

•   a windows server 2008 R2,

·         LAN IP, 10.10.0.2, hostname (DC name) = SERVER-AD-01.NGERI.CO.ID

·         domain: NGERI.CO.ID,

·         DNS is running well

·         More options, DHCP is already running.

Clear Memory in Linux

This is a script to clear the memory. This script is typically useful for servers that use squid. Squid as a proxy so greedy with memory. Create a file to put this script, for example file "clearmemory". And place it in the folder /etc/cron.daily/, in order to run automatically every day.
vim /etc/cron.daily/clearmemory.

Review Hotspot YFI Beta-6-1.1

This is the appearance of yfi Beta-6, that Installed in my vmware.
1. Install vmware.
2. Find the file Hotspot YFI Beta-6-1.1  from here, http://sourceforge.net/projects/hotcakes/
and download.
3. You will get file Beta-6-1.1.tar. extract the file and you will find 2 files, Beta-6-1.1.ovf and Beta-6-1.1-disk1.vmdk. Then, just click beta-6-1.1.ovf, and waiting... and run from vmware.
4. Login with default username, and default password,

Monitoring Squid with Cacti (Template Cacti for Squid... Complete...!!!)

I assume that you have installed snmp, squid and cacti, before. I will not explain again about how to install squid, snmp, cacti, etc.. there's so many article to discuss about that’s.
Then, if all installation is complete, let's make graph template for squid.

1. cek your squid configuration,
# squid -v
Squid Cache: Version 3.1.22

Install Snmp & Cacti on Ubuntu 12.10

What is Cacti ??

Cacti is an open-source, web-based network monitoring and graphing tool designed as a front-end application for the open-source, industry-standard data logging tool RRDtool. Cacti allows a user to poll services at predetermined intervals and graph the resulting data. It is generally used to graph time-series data of metrics such as CPU load and network bandwidth utilization. A common usage is to monitor network traffic by polling a network switch or router interface via simple network management protocol (SNMP). (wikipedia)
Cacti is a complete frontend to RRDTool, it stores all of the necessary information to create graphs and populate them with data in a MySQL database.

Ubuntu 12.10 + Squid 3.1.23 (Transparent Proxy) + Shorewall + Mikrotik

Installation & Configuration

Step-by-step installation, almost the same as previous versions of ubuntu, can be checked from the video install ubuntu 12:04, below. Thank you for the uploader.

Squid 3.1.23 : Example Squid.conf & Squid start up

#Squid.conf

#WELCOME TO SQUID 3.1.23
#       ----------------------------
acl snmppublic snmp_community public
acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl to_localhost dst 127.0.0.0/8 0.0.0.0/32

acl bamboe src 192.168.3.0/24
#uncomment noway url, if necessary
#acl noway url_regex -i "/etc/squid/noway"
acl SSL_ports port 443
acl Safe_ports port 80          # http

Membuat Grafik Sensors di Cacti

Sudah beberapa kali install cacti, tapi belum terarsipkan. Akhirnya ketika saya dimintain tolong sama tetangga untuk dibuatkan tampilan grafik, saya oke-oke saja, sekalian kesempatan mengingat-ingat dan mengarsipkan kembali... :)
Langsung saja. Kebetulan disini OS yang saya gunakan adalah OpenSuse 10.3.

1. Install sensors, beserta packet-packet pendukung lainnya. Lebih mudahnya gunakan yast.
Masuk ke yast --- software --- software management. Gunakan fasilitas search, untuk mencari packet sensors. Jangan lupa masukkan CD / DVD nya. Sangat mudah, karena itu, saya tidak akan menjelaskannya secara details.

Configure Shorewall (Not) Load Balance, Satu Speedy-nya Rusak

Kemarenan di warnet tetangga, internetnya lambat sekali... tidak seperti biasanya... setelah ditelusuri penyebabnya ternyata adalah modemnya rusak.
Di warnet tetangga ini, dia menggunakan load balance 2 buah speedy dengan menggunakan shorewall... sama persis dengan artikel configure shorewall... ya.. pastinya sama persis donk... lah saya buat sendiri... hehehehe...
Menurut teori load balance, dan berdasarkan pengalaman, jika salah satu speedy mati, maka speedy yang lainnya performancenya juga ikut drop.

BACKUP MY SHOREWALL CONFIGURASI (UPDATE)

Ini adalah configurasi shorewall yang saya gunakan di router, dimana di router, ada squid dan load balancing.
# shorewall version -a
shorewall-core: 4.5.5.3
shorewall: 4.5.5.3

shorewall.conf
STARTUP_ENABLED=Yes
VERBOSITY=1

Configure Shorewall ( Load Balance 2 WAN )

Berikut configure shorewall yang saya gunakan di warnet tetangga. Menggunakan 2 speedy load balance. Silahkan dimodif sendiri dengan mengganti IP nya dan tentunya sesuai dengan kebutuhan masing2. :)

Forwarding loop detected [Solve]

What is a forwarding loop?

A forwarding loop is when a request passes through one proxy more than once. You can get a forwarding loop if

a cache forwards requests to itself. This might happen with interception caching (or server acceleration) configurations.

a pair or group of caches forward requests to each other. This can happen when Squid uses ICP, Cache Digests, or the ICMP RTT database to select a next-hop cache.

Forwarding loops are detected by examining the Via request header. Each cache which "touches" a request must add its hostname to the Via header. If a cache notices its own hostname in this header for an incoming request, it knows there is a forwarding loop somewhere.

Optimizing and Security Hotspot YFI (part 2)

See previous post 

Squid

·         get the latest version squid, http://www.squid-cache.org and installing

cd /usr/local

wget http://www.squid-cache.org/Versions/v3/3.1/squid-3.1.19.tar.bz2

tar jxvf squid-3.1.19.tar.bz2

cd squid-3.1.19

./configure --prefix=/usr   --exec-prefix=/usr   --bindir=/usr/sbin  \

 --sbindir=/usr/sbin   --sysconfdir=/etc/squid   --datadir=/usr/share/squid  \

 --includedir=/usr/include   --libdir=/usr/lib   --libexecdir=/usr/lib/squid  \

 --localstatedir=/var   --sharedstatedir=/usr/com   --mandir=/usr/share/man  \

 --infodir=/usr/share/info   --x-includes=/usr/include   \ 

Optimizing and Security Hotspot YFI (part 1)

In this article, I try to optimize my server hotspot yfi, by adding squid, shorewall, and updating coova chilli. This post continued from previous post about Install Hotspot YFI. See the previous post in part 1 and part 2.

Upgrade Coova Chilli.

Important points to remember. The UAM Port is by default on 3990 when you compile the CoovaChilli package yourself, and on 3660 when using !CoovaChilliAP Firmware.

UAM Port	3990
COA Port	3799

 

Install CoovaChilli

• First, stop the service chilli, and remove the old coova chilli. Execute the following commands:

RRDtool 1.4.7 in ubuntu 10.04

Install RRDtool

Run the following commands as superuser:

sudo su

apt-get install libpango1.0-dev libxml2-dev

wget http://oss.oetiker.ch/rrdtool/pub/rrdtool-1.4.7.tar.gz
tar -zxvf rrdtool-1.4.7.tar.gz

cd rrdtool-1.4.7

mkdir /tmp/rrdbuild

export BUILD_DIR=/tmp/rrdbuild

mkdir /opt/rrdtool-1.4.7

export INSTALL_DIR=/opt/rrdtool-1.4.7