'Transparent Proxy'에 해당되는 글 3건

  1. 2010.10.27 Remove X-Forwarded-For header in Squid
  2. 2010.02.22 Varnish HTTP accelerator (state-of-the-art)
  3. 2009.08.24 mod_rpaf (proxy 및 Virtual IP 이용시) (1)
2010.10.27 15:30

Remove X-Forwarded-For header in Squid

Changes in 3.1 forwarded_for
New setting options. transparent, truncate, delete.

        If set to "transparent", Squid will not alter the
        X-Forwarded-For header in any way.

        If set to "delete", Squid will delete the entire
        X-Forwarded-For header.

        If set to "truncate", Squid will remove all existing
        X-Forwarded-For entries, and place itself as the sole entry.
        
Changes in 2.7 forwarded_for
Has several new modes, allowing one to finetune how/if the requesting client IP should be forwarded in X-Forwarded-For

Configuration Details:
Option Name: forwarded_for
Replaces:
Requires:
Default Value: forwarded_for on
Suggested Config:

If set to "on", Squid will append your client's IP address
in the HTTP requests it forwards. By default it looks like:

X-Forwarded-For: 192.1.2.3

If set to "off", it will appear as

X-Forwarded-For: unknown

If set to "transparent", Squid will not alter the
X-Forwarded-For header in any way.

If set to "delete", Squid will delete the entire
X-Forwarded-For header.

If set to "truncate", Squid will remove all existing
X-Forwarded-For entries, and place itself as the sole entry.



Some months ago I was working with Squid 3.0 and managed to get rid of the X-Forwarded-For header completely. Now I am trying to do the same thing but I do not seem to hit the nail.

I have tried forwarded_for off but that does not remove the header, it just inserts unknown in place of the client IP.

I tried header_access X-Forwarded-For deny all but I receive an error about invalid directive. Looking in the function reference it seems that this directive is deprecated in 3.0.

I tried forwarded_for delete but this is only available from 3.1.

Trackback 0 Comment 0
2010.02.22 20:40

Varnish HTTP accelerator (state-of-the-art)

VCL - Varnish configuation Language

The VCL language is a small domain-specific language designed to be used to define request handling and document caching policies for the Varnish HTTP accelerator. When a new configuration is loaded, the varnishd management process translates the VCL code to C and compiles it to a shared object which is then dynamically linked into the server process.

Examples

Other users have contributed examples from their VCLs to solve common problems.

Syntax overview

The VCL syntax is very simple, and deliberately similar to C and Perl. Blocks are delimited by curly braces,statements end with semicolons, and comments may be written as in C, C++ or Perl according to your own preferences.

  • In addition to the C-like assignment (=), comparison (==) and boolean (!, && and ||) operators, VCL supports regular expression and ACL matching using the ~ operator.
  • Unlike C and Perl, the backslash (\) character has no special meaning in strings in VCL, which use the (%xx) escape mechanism just like URLs, so it can be freely used in regular expressions without doubling.
  • Assignments are introduced with the set keyword. There are no user-defined variables; values can only be assigned to variables attached to backend, request or document objects. Most of these are typed, and the values assigned to them must have a compatible unit suffix.
  • VCL has if tests, but no loops.
  • The contents of another VCL file may be inserted at any point in the code by using the include keyword followed by the name of the other file as a quoted string.

Backend declarations

A backend declaration creates and initializes a named backend object:

backend www {
  .host = "www.example.com";
  .port = "http";
}

The backend object can later be used to select a backend at request time:

if (req.http.host ~ "^(www.)?example.com$") {
   set req.backend = www;
}

The timeout parameters can be overridden in the backend declaration. The timeout parameters are .connect_timeout for the time to wait for a backend connection, .first_byte_timeout for the time to wait for the first byte from the backend and .between_bytes_timeout for time to wait between each received byte.

These can be set in the declaration like this:

backend www {
   .host = "www.example.com";
   .port = "http";
   .connect_timeout = 1s;
   .first_byte_timeout = 5s;
   .between_bytes_timeout = 2s;
}

You can limit the amount of connections varnish will send to a backend like this:

backend www {
   .host = "www.example.com";
   .port = "http";
   .max_connections = 200; 
}

Directors

Directors choose from different backends based on health status and a per-director algorithm. There currently exists a round-robin and a random director.

Directors are defined using:

director b2 random {
    .retries = 5;
    {
     /* We can refer to named backends */
     .backend        = b1;
     .weight         = 7;
    }
    {
     /* Or define them inline */
     .backend        = {
         .host = "fs2";
     }
     .weight         = 3;
    }
}

The random director

The random director takes one per-director option .retries. This specifies how many tries it will use to find a working backend. The default is the same as the number of backends defined for the director.

There is also a per-backend option: weight which defines the portion of traffic to send to the particular backend.

The round-robin director

The round-robin does not take any options.

Backend probes

Backends can be probed to see whether they should be considered healthy or not. The return status can also be checked by using req.backend.healthy .window is how many of the latest polls we examine, while .threshold is how many of those must have succeeded for us to consider the backend healthy.

backend www {
   .host = "www.example.com";
   .port = "http";
   .probe = {
     .url = "/test.jpg";
     .timeout = 0.3 s;
     .window = 8;
     .threshold = 3;
   }
}

It is also possible to specify the raw HTTP request.

backend www {
   .host = "www.example.com";
   .port = "http";
   .probe = {
     # NB: \r\n automatically inserted after each string!
     .request ==
       "GET / HTTP/1.1"
       "Host: www.foo.bar"
       "Connection: close";
   }
}

ACLs

An ACL declaration creates and initializes a named access control list which can later be used to match client addresses:

acl local {
    "localhost"; /* myself */
    "192.0.2.0"/24; /* and everyone on the local network */
    ! "192.0.2.23"; /* except for the dialin router */
}

If an ACL entry specifies a host name which Varnish is unable to resolve, it will match any address it is compared to. Consequently, if it is preceded by a negation mark, it will reject any address it is compared to, which may not be what you intended. If the entry is enclosed in parentheses, however, it will simply be ignored.

To match an IP address against an ACL, simply use the match operator:

if (client.ip ~ local) {
   pipe;
}

Grace

If the backend takes a long time to generate an object there is a risk of a thread pile up. In order to prevent this you can enable grace. This allows varnish to serve an expired version of the object while a fresh object is being generated by the backend.

The following vcl code will make Varnish serve expired objects. All object will be kept up to two minutes past their expiration time or a fresh object is generated.

sub vcl_recv {
    set req.grace = 2m;
}

sub vcl_fetch {
    set obj.grace = 2m;
}

Functions

The following built-in functions are available:

regsub(str, regex, sub)

Returns a copy of str with the first occurrence of the regular expression regex replaced with sub. Within sub, \0 (which can also be spelled &) is replaced with the entire matched string, and \n is replaced with the contents of subgroup n in the matched string. VCL uses your libc's extended expression support, which in most cases is POSIX Extended Regular Expressions.

regsuball(str, regex, sub)

As regsub() but this replaces all occurrences.

purge( str )

Purge objects based on a list of conditions. For the purge to happen, every condition must be met. For example:

  purge( "req.url == / && obj.http.foo ~ bar" );

See VCLExamplePurging for more code examples

purge_url(regex)

Purge all objects in cache whose URLs match regex. For example:

  purge_url( "^/foo$" );

See VCLExamplePurging for more code examples

purge_hash(regex)

Purge all objects in cache whose hash strings match regex.

See VCLExamplePurging for more code examples

Subroutines

A subroutine is used to group code for legibility or reusability:

sub pipe_if_local {
    if (client.ip ~ local) {
       pipe;
    }
}

Subroutines in VCL do not take arguments, nor do they return values.

If multiple subroutines with the same name are defined, they are concatenated in the order in which the appear in the source.

To call a subroutine, use the call keyword followed by the subroutine’s name:

call pipe_if_local;

Special subroutines

There are a number of special subroutines which hook into the Varnish workflow. These subroutines may inspect and manipulate HTTP headers and various other aspects of each request, and to a certain extent decide how the request should be handled. Each subroutine terminates by calling one of a small number of keywords which indicates the desired outcome.

Graphical Overview

To see a description of the default VCL and to see a flow chart of how HTTP requests go through VCL please see the default VCL page.

vcl_recv

Called at the beginning of a request, after the complete request has been received and parsed. Its purpose is to decide whether or not to serve the request, how to do it, and, if applicable, which backend to use.

The vcl_recv subroutine may terminate with one of the following keywords:

error code [reason]

Return the specified error code to the client and abandon the request.

pass

Switch to pass mode. Control will eventually pass to vcl_pass.

pipe

Switch to pipe mode. Control will eventually pass to vcl_pipe.

lookup

Look up the requested object in the cache. Control will eventually pass to vcl_hit or vcl_miss, depending on whether the object is in the cache.

vcl_pipe

Called upon entering pipe mode. In this mode, the request is passed on to the backend, and any further data from either client or backend is passed on unaltered until either end closes the connection.

The vcl_pipe subroutine may terminate with one of the following keywords:

error code [reason]

Return the specified error code to the client and abandon the request.

pipe

Proceed with pipe mode.

vcl_pass

Called upon entering pass mode. In this mode, the request is passed on to the backend, and the backend’s response is passed on to the client, but is not entered into the cache. Subsequent requests submitted over the same client connection are handled normally.

The vcl_pass subroutine may terminate with one of the following keywords:

error code [reason]

Return the specified error code to the client and abandon the request.

pass

Proceed with pass mode.

Note: You must have a backend selected by the end of the vcl_pass function, otherwise you'll get a 503 response, as Varnish won't know where to direct the request.

vcl_hash

Use req.hash += req.http.Set-Cookie or similar to include the Set-Cookie HTTP header in the hash string. The vcl_hash subroutine may terminate with one of the following keywords:

hash

Proceed.

vcl_hit

Called after a cache lookup if the requested document was found in the cache.

The vcl_hit subroutine may terminate with one of the following keywords:

error code [reason]

Return the specified error code to the client and abandon the request.

pass

Switch to pass mode. Control will eventually pass to vcl_pass.

deliver

Deliver the cached object to the client. Control will eventually pass to vcl_deliver.

vcl_miss

Called after a cache lookup if the requested document was not found in the cache. Its purpose is to decide whether or not to attempt to retrieve the document from the backend, and which backend to use.

The vcl_miss subroutine may terminate with one of the following keywords:

error code [reason]

Return the specified error code to the client and abandon the request.

pass

Switch to pass mode. Control will eventually pass to vcl_pass.

fetch

Retrieve the requested object from the backend. Control will eventually pass to vcl_fetch.

vcl_fetch

Called after a document has been successfully retrieved from the backend.

The vcl_fetch subroutine may terminate with one of the following keywords:

error code [reason]

Return the specified error code to the client and abandon the request.

pass

Switch to pass mode. Control will eventually pass to vcl_pass.

deliver

Possibly insert the object into the cache, then deliver it to the client. Control will eventually pass to vcl_deliver.

esi

ESI-process the document which has just been fetched.

vcl_deliver

Called before a cached object is delivered to the client.

The vcl_deliver subroutine may terminate with one of the following keywords:

error code [reason]

Return the specified error code to the client and abandon the request.

deliver

Deliver the object to the client.

vcl_timeout

Called by the reaper thread shortly before a cached document reaches its expiry time.

The vcl_timeout subroutine may terminate with one of the following keywords:

fetch

Request a fresh copy of the object from the backend.

discard

Discard the object.

vcl_discard

Called by the reaper thread when a cached document is about to be discarded, either because it has expired or because space is running low.

The vcl_discard subroutine may terminate with one of the following keywords:

discard

Discard the object.

keep

Keep the object in cache.

If one of these subroutines is left undefined or terminates without reaching a handling decision, control will be handed over to the builtin default. See the EXAMPLES section for a listing of the default code.

vcl_error

XXX - needs documentation.

Variables

Although subroutines take no arguments, the necessary information is made available to the handler subroutines through global variables.

The following variables are always available:

Global variables

now

The current time, in seconds since the epoch.

Backend declarations

The following variables are available in backend declarations:

.host

Host name or IP address of a backend.

.port

Service name or port number of a backend.

Hashing a request

The following variables are available while determining the hash key of an object:

req.

req.hash

The hash key used to refer to an object in the cache. Used when both reading from and writing to the cache.

Processing a request

The following variables are available while processing a request:

client.

client.ip

The client’s IP address.

server.

server.ip

The IP address of the socket on which the client connection was received.

server.port

The port number of the socket on which the client connection was received.

req.

req.request

The request type (e.g. "GET", "HEAD").

req.url

The requested URL.

req.proto

The HTTP protocol version used by the client.

req.backend

The backend to use to service the request.

req.backend.healthy

Whether the backend is healthy or not.

req.http.header

The corresponding HTTP header.

Preparing a backend request

The following variables are available while preparing a backend request (either for a cache miss or for pass or pipe mode):

bereq.

bereq.request

The request type (e.g. "GET", "HEAD").

bereq.url

The requested URL.

bereq.proto

The HTTP protocol version used to talk to the server.

bereq.http.header

The corresponding HTTP header.

bereq.connect_timeout

The time in seconds to wait for a backend connection.

bereq.first_byte_timeout

The time in seconds to wait for the first byte from the backend. Not available in pipe mode.

bereq.between_bytes_timeout

The time in seconds to wait between each received byte from the backend. Not available in pipe mode.

Cache hit or backend retrieval

obj.

The following variables are available after the requested object has been retrieved from cache or from the backend:

obj.hits

The number of hits an object has received.

obj.proto

The HTTP protocol version used when the object was retrieved.

obj.status

The HTTP status code returned by the server.

obj.response

The HTTP status message returned by the server.

obj.cacheable

True if the request resulted in a cacheable response. A response is considered cacheable if it is valid (see above), the HTTP status code is 200, 203, 300, 301, 302, 404 or 410 and it has a non-zero time-to-live when Expires and Cache-Control headers are taken into account.

obj.ttl

The object’s remaining time to live, in seconds.

obj.lastuse

The approximate time elapsed since the object was last requests, in seconds.

Preparing response

resp.

The following variables are available while preparing a response to the client:

resp.proto

The HTTP protocol version to use for the response.

resp.status

The HTTP status code that will be returned.

resp.response

The HTTP status message that will be returned.

resp.http.header

Values may be assigned to variables using the set keyword:

sub vcl_recv {
  # Normalize the Host: header
  if (req.http.host ~ "^(www.)?example.com$") {
     set req.http.host = "www.example.com";
  }
}

HTTP headers can be removed entirely using the remove keyword:

sub vcl_fetch {
  # Don’t cache cookies
  remove obj.http.Set-Cookie;
}

 

원문 : http://varnish-cache.org


Trackback 0 Comment 0
2009.08.24 10:54

mod_rpaf (proxy 및 Virtual IP 이용시)

php 함수에서 x-forwarded-for 구문을 수정하는 대신 mod_rpaf 를 이용하면
간단하게 모듈만 올림으로써 IP 추적이 가능하게 된다.

가상서버내에서 클라이언트 IP 를 정상적으로 변환시켜 주는 모듈입니다.
이 모듈을 쓰지 않으면 메인서버의 게이트웨이 IP 가 찍히게 됩니다.
이 모듈을 사용하시게 되면 아파치 로그뿐만 아니라 프로그램상에서도
정상적으로 클라이언트 IP 추적이 가능합니다.

URL: http://stderr.net/apache/rpaf/

설치방법:
# wget http://stderr.net/apache/rpaf/download/mod_rpaf-0.6.tar.gz
# apt-get install apache2-threaded-dev (apxs2 설치, prefork 일 경우는 apache2-prefork-dev)
# apt-get install make (필요한 경우)
# apt-get install gcc (필요한 경우)
# make rpaf-2.0 (아파치 1.3 은 make rpaf)
# make install-2.0 (아파치 1.3 은 make install)

아차피 설정파일(apache2.conf)에

LoadModule rpaf_module /usr/lib/apache2/modules/mod_rpaf-2.0.so

RPAFenable On
RPAFsethostname On
RPAFproxy_ips 10.0.0.128
RPAFheader X-Forwarded-For

추가해 줍니다. 10.0.0.128 은 가상서버 게이트웨이 IP 입니다.
아파치 로그에 찍히는 아이피로 해 주시면 됩니다.
요렇게 하시고 아파치 로그등에서 확인해 보시면 정상적으로 IP 가 찍히는 걸 확인하실 수 있습니다.

페도라에서는

# yum install httpd-devel
# apxs -i -c -n mod_rpaf-2.0.so mod_rpaf-2.0.c

물론 gcc, make 등은 설치되어 있어야 합니다. 이렇게 컴파일하고 설치까지 하시면

/etc/httpd/modules/mod_rpaf-2.0.so

파일이 생성이 됩니다.

아파치 설정은 /etc/httpd/conf/httpd.conf 파일에 아래의 설정내용대로 설정하시면 됩니다.

LoadModule rpaf_module /etc/httpd/modules/mod_rpaf-2.0.so

RPAFenable On
RPAFsethostname On
RPAFproxy_ips 10.0.0.128
RPAFheader X-Forwarded-For


출처 : http://blog.naver.com/saint7710


Trackback 0 Comment 1
  1. monovision 2010.02.22 20:44 address edit & del reply

    varnish 에서는 헤더를 추가 및 삭제 하는 기능이 있으며 nginx 역시 해당 기능이 있습니다.

    X-Forwarded-For 헤더에 대한 로그는 거의 대부분의 웹서버(apache, nginx, lighttpd, iis 등) 에서 모두 로깅을 지원을 합니다.
    만약 진짜 안되는 웹서버가 있다면 웹서버 앞단에 nginx 등과 같은 성능 좋은 웹서버를 하나 더 두는 방법도 있습니다.