'proxy'에 해당되는 글 22건

  1. 2013.10.29 Firefox Proxy 통한 SSH 터널링 (2)
  2. 2013.08.27 Using cURL to automate HTTP jobs
  3. 2012.11.19 Burp Suite Free Edition v1.5 released (1)
2013.10.29 05:38

Firefox Proxy 통한 SSH 터널링

a fast, privately secured tunnel to transfer web pages and dns queries


Have you ever wanted to visit sites during the day from a location that denied access to those sites? Perhaps the company has denied access due to bandwidth considerations or you might have decided that the site you want to go to might not always be work safe depending on the story or pictures? What you need is the ability to create a secure and encrypted ssh connection to tunnel your browser traffic through.

Using a ssh tunnel to retrieve the data from websites is significantly faster than trying to use X forwarding to open a remote copy of Firefox on the remote machine. If a remote browser is used the connection will be saturated by the graphical front end of the remote browser window. Use the tunnel for the web site's data and leave the rendering of the browser to the local machine. This is the most efficient solution.

If you have access to a remote machine by way of ssh you can set up Firefox, or any other SOCKS v5 enabled application, to tunnel its connection through ssh. This way, if you were at work and wanted to browse your favorite sites like MySpace, Facebook or Maxim that are blocked at the company firewall you could.


Getting Started

First you must have ssh access to the remote machine you want to proxy to. Let it be a home machine or a free shell you signed up for on-line. You must also make sure you can ssh from where your browser is to where you want to tunnel to. No need to set this up if port 22 is not open to you from your location to your destination.

ATTENTION: We are proud to announce our Firefox add-on called, "Calomel SSL Validation". It will grade the security of your SSL connection. The link has screen shots too!

IMPORTANT NOTE: The Firefox tunnel using SOCKS5 (option 1) is the easiest and quickest proxy to setup. If you just want to get the proxy working then follow the SOCKS5 options.


Configure Firefox for the proxy

You need to configure Firefox to use the proxy. Find the section to add a proxy to the browser. On *nix systems of Firefox you will find the settings in File, Preferences, Advanced, Network, Settings. The setting by default is "Direct Connection to the Internet". We need to setup the "Manual proxy configuration".

You have two(2) options to pick from. You can proxy directly to the remote machine and then connect directly to web sites. This is the SOCKS5 method and is the easiest to setup. Or, you could use a Squid web proxy (if available) on the remote machine to accept the traffic from the ssh tunnel. Squid would then request the traffic from web sites. Pick one of the options below.

NOTE: For our example, ssh is going to listen on localhost (127.0.0.1) and port 8080 of the local machine.

Option 1: ssh and direct connect (SOCKS5) : If you are going to use the ssh tunnel with the option "-D 8080" then you need to setup the browser to use a SOCKS5 proxy. Setup the proxy config page with the following entries and leave the rest of the entries blank.

Manual proxy configuration:
  SOCKS Proxy  127.0.0.1  Port 8080
  check the box for "SOCKS v5"

Option 2: ssh tunnel to squid proxy (HTTP/SSL Proxy) : If you are going to use the ssh tunnel with the option "-L 8080:localhost:2020" to connect to the remote machine's Squid proxy then configure the browser to use a HTTP/SSL proxy. Setup the proxy config page with the following entries and leave the rest of the entries blank.

Manual proxy configuration:
  HTTP Proxy:  127.0.0.1  Port 8080
  SSL Proxy :  127.0.0.1  Port 8080 


Optional Step: DNS proxying through SOCKS5 is highly recommended

This step is optional, but since we are going to be proxying the data over the ssh tunnel then we should also proxy the DNS requests as well. The purpose of this exercise is to get to a site we might not otherwise be able to retrieve or just to anonymize our browsing from your location. If we tunneled our data through ssh and then asked the local DNS server for the ips it would defeat the purpose. So, add a boolean option into the URL "about:config" page in Firefox. Name the entry "network.proxy.socks_remote_dns" and set it to true.

This method will only take affect if you use the SOCKS5 proxy method. If you are proxying using the squid method (HTTP/SSL Proxy) you could always check if you can query another, independent DNS server like OpenDNS.

##Preference Name                 Status     Type      Value
  network.proxy.socks_remote_dns  user set   boolean   true


Making the ssh tunnel

Lastly, we need to start the ssh tunnel. You have two choices depending if you want the packets to be forwarded to squid on the remote machine or not.

Option 1: ssh and direct connect (SOCKS5) : The following line will start the ssh client and connect to username@remote_machine.com. Port 8080 on localhost (127.0.0.1) will listen for requests and send them to the remote machine. The remote machine will then send the packets out as if they originated from itself. The ssh options are in the man page of ssh, but to summarize them in order: Compression, SSH2 only, Quite, Force pseudo-tty allocation, Redirect stdin from /dev/null, and Place the ssh client into "master" mode for connection sharing.

ssh -C2qTnN -D 8080 username@remote_machine.com

Option 2: ssh to squid proxy (HTTP/SSL Proxy) : The following line will also start the ssh client and connect to username@remote_machine.com. Port 8080 on localhost (127.0.0.1) on the current machine will listen for requests and ssh tunnel them to the remote machine. On the remote machine ssh will forward the packets to localhost port 2020. If squid is listening on localhost port 2020 on the remote machine then all requests sent though the ssh tunnel will then be forwarded to squid. You can use squid to block ads and speed up web access. If you need assistance with squid, check out the Calomel.org Squid "how to" page.

ssh -C2qTnN -L 8080:localhost:2020 username@remote_machine.com


Testing the ssh tunnel

Once you execute the ssh line the encrypted and compressed ssh tunnel will be active in the xterm. We used the "quiet" options in ssh so there will not be any logging or output to the terminal.

Make sure Firefox is working by checking the proxy is active and then try to go to a web page. You can also try a site like WhatIsMyIp.com to verify the ip you have with the proxy is different than without.

If everything is working then you can be assured that all of your browsing traffic is being encrypted through the tunnel and no one at your current location will be able to see your traffic over the network.

Once you are done with the proxy just exit the ssh xterm or kill this instance of ssh with Ctrl-c. Remember to set Firefox back to "Direct Connection" if you want to directly browse from your location otherwise you will not be going anywhere.

Interested in setting up Squid or Samba? Check out our guides covering the Squid Proxy and Samba file share servers. We offer clear explanations and fully working example configurations.


Questions?

How can I setup two or more ssh tunnels through two or more machines ?

At some point you may need to tunnel through multiple ssh tunnels through multiple machines. This is quite easy to do as long as you have ssh access to every machine you want to tunnel through. In this example we will be tunneling from a desktop machine through a machine called host1 and then to a machine called host2 which will then access the internet. Something like so:

Firefox desktop -> host1 -> host2 -> internet

First, make sure you went through the beginning on this page and know how to get firefox to proxy through a SOCKS5 proxy on localhost port 8080. Then run the following ssh command on the desktop running Firefox. This will setup an encrypted ssh tunnel to host1 from the "Firefox desktop".

desktop$ ssh -C2qTnN username@host1 -L 8080:localhost:8080

Now, you need to ssh to host1 directly. Once you are on host1 run the following. This will collect any data from the first tunnel originating from the "Firefox desktop" to host1 and tunnel that data to host2.

host1$ ssh -C2qTnN -D 8080 username@host2

So, how does this setup work? Firefox on the desktop will initiate a SOCKS5 connection to localhost port 8080 on the desktop machine. Since a ssh tunnel is listening on localhost:8080 it will ssh tunnel the traffic to host1 which will forward this traffic to host1's localhost:8080. On host1 the second ssh command will tunnel all traffic it receives on localhost:8080 from the desktop machine to host2. On host2 the traffic will then be able to go out to the internet at large. If you have DNS SOCKS5 resolution on as well then all web traffic _and_ dns queries will goto host2 through both tunnels. From the view of the internet all queries originating from the "Firefox desktop" will look like they come from host2. Nice and anonymous.

What if I need to tunnel through more then two machines? Then just keep repeating "ssh -C2qTnN username@host1 -L 8080:localhost:8080" command for each incremental host. Once you decide you very last host you want the data to access the internet with then use the "ssh -C2qTnN -D 8080 username@host2" command.

To make sure you tunnel is working correctly using a site like ipchicken.com to see what ip address you are coming from. In the case of our example above ipchicken should report the ip address of host2.

Do you have any recommended modifications for Firefox in "about:config" ?

First, make sure to check out our Firefox Add-on "Calomel SSL Validation".

More open proxy connections: When you use a proxy, Firefox limits the amount of concurrent open connections to 8. This is too small for most users as many people open multiple tabs to many sites. When more then 8 connections are made the browser seems to be "stuck" because Firefox will wait till an open connection is closed before making a new one. To avoid this problem it is highly suggested to increase the persistent connections value from 8 to 25.

network.http.max-persistent-connections-per-proxy 25

Turn off pop-up tips: If you are annoyed by pop up text when your mouse hovers over a web element you can turn that function off.

browser.chrome.toolbar_tips  false

No animations: Stop all animated gifs and pictures like ads and annoying dancing cartoons characters.

image.animation_mode  none

No blinking text: Blinking text is annoying. Webmasters should not use it. In case they do, we will disallow the function in the browser.

browser.blink_allowed  false

Parallel connections: An easy way to speed up Firefox is to increase the amount of parallel connections the browser makes to the server. Open up Firefox and type in "about:config" in the URL. Then search for the string "conn" You should see the following entries listed. Modify them as follows:

network.http.max-connections                        25
network.http.max-connections-per-server             25
network.http.max-persistent-connections-per-proxy   25
network.http.max-persistent-connections-per-server  25

It is _not_ recommended to use more then 25 parallel connections due to abuse of the remote server and concurrency bottlenecks on the local system. Understand that if you have a slow system then more parallel connections can actually slow the browser down considerably. Also, if you try to open too many connections to a server then that server many consider you hostile and block or blacklist you.

Pipelining Enabled: The fastest and most efficient way to implement a browser is to use pipelining. This is where a single persistent connection is used, but instead of waiting for each response before sending the next request, several requests are sent out at a time. This reduces the amount of time the client and server are waiting for requests or responses to cross the network. Pipelined requests with a single connection are faster than multiple HTTP/1.0 requests in parallel, and considerably reduce the number of packets transmitted across the network. Apache supports both HTTP/1.0 keep-alive and HTTP/1.1 persistent connections. Pipelining is implement entirely at the browser end if supported by the remote web server, using persistent connections.

To enable pipelining in Firefox browser goto the url about:config . Then search for "pipe" and set the following:

network.http.pipelining              true
network.http.pipelining.maxrequests  8
network.http.pipelining.ssl          true
network.http.proxy.pipelining        true

TLSv1 with AES256, AES128 and 3DES 168 Only: When connecting to SSL based servers (https) you only want to use the strongest ciphers available. Most web server admins can setup their servers to prefer weak ciphers over strong ciphers for any reason; sometimes they want a less CPU intensive encryption or perhaps they just configured the server wrong. Even Google's encrypted pages prefer RC4 instead of AES and this is not our idea of good security. We want to make sure that our version of Firefox only uses AES 256 bit, AES 128 bit or 3DES 168 bit ciphers.

Open up a window and type "about:config". Then in the "Filter" bar at the top search for the following. Double clicking on each line will change the value.

  • tls and set the lines to true.
  • ssl2 and set every line entry to false.
  • ssl3 and set every line to false _except_ lines containing the strings "aes_256" and "aes_128".
  • security.ssl3.rsa_des_ede3_sha and set it to true. This is the weakest cipher and may be needed for some older SSL sites.

Now your browser will _only_ accept the TLSv1 protocol in AES256 bit cipher encryption no matter what previous weaker ciphers a web server prefers. This configuration also makes your browser FIPS 120-2 compliant (year 2030 specs).

Is there any way I can switch proxies faster?

There are add-ons, also called extensions, for Firefox called FoxyProxy or SwitchProxyTool you can use. They offer the ability to setup multiple proxy settings and choose the one you want, or turn them off, using a drop down menu.

I noticed you use compression in the ssh tunnel proxy. Why?

The majority of the data you are retrieving using the browser is text or HTML data. This type of data compresses very well at up to 80%. Using compression in the tunnel will speed up the delivery of the data considerably.



출처 : calomel.org



Trackback 5 Comment 2
  1. Favicon of https://blog.pages.kr 날으는물고기 2013.10.29 05:47 신고 address edit & del reply

    You do not need to use SOCKS proxy to connect to a HTTP proxy through SSH. You can use the following command:

    ssh -f -N -L $portlocal:$machineproxy:$portproxy $machinegateway

    $portlocal - the ssh client on your machine will listen on this port, you can chose for example 55555. In the web browser you will then set the HTTP proxy as localhost on port 55555.
    $portproxy - the port the Squid proxy is listening on.
    $machineproxy - address of the Squid proxy machine (as seen from the gateway).
    $machinegateway - address of the gateway machine as seen from your PC

    The SSH server on the gateway must allow port forwarding. If it is not allowed you will have to ask administrator of the server to do so.

  2. 2013.10.29 05:50 address edit & del reply

    비밀댓글입니다

2013.08.27 18:46

Using cURL to automate HTTP jobs

Date:    Jan 19, 2011
 
                The Art Of Scripting HTTP Requests Using Curl
                =============================================
 
 This document will assume that you're familiar with HTML and general
 networking.
 
 The possibility to write scripts is essential to make a good computer
 system. Unix' capability to be extended by shell scripts and various tools to
 run various automated commands and scripts is one reason why it has succeeded
 so well.
 
 The increasing amount of applications moving to the web has made "HTTP
 Scripting" more frequently requested and wanted. To be able to automatically
 extract information from the web, to fake users, to post or upload data to
 web servers are all important tasks today.
 
 Curl is a command line tool for doing all sorts of URL manipulations and
 transfers, but this particular document will focus on how to use it when
 doing HTTP requests for fun and profit. I'll assume that you know how to
 invoke 'curl --help' or 'curl --manual' to get basic information about it.
 
 Curl is not written to do everything for you. It makes the requests, it gets
 the data, it sends data and it retrieves the information. You probably need
 to glue everything together using some kind of script language or repeated
 manual invokes.
 
1. The HTTP Protocol
 
 HTTP is the protocol used to fetch data from web servers. It is a very simple
 protocol that is built upon TCP/IP. The protocol also allows information to
 get sent to the server from the client using a few different methods, as will
 be shown here.
 
 HTTP is plain ASCII text lines being sent by the client to a server to
 request a particular action, and then the server replies a few text lines
 before the actual requested content is sent to the client.
 
 The client, curl, sends a HTTP request. The request contains a method (like
 GET, POST, HEAD etc), a number of request headers and sometimes a request
 body. The HTTP server responds with a status line (indicating if things went
 well), response headers and most often also a response body. The "body" part
 is the plain data you requested, like the actual HTML or the image etc.
 
 1.1 See the Protocol
 
  Using curl's option --verbose (-v as a short option) will display what kind
  of commands curl sends to the server, as well as a few other informational
  texts.
 
  --verbose is the single most useful option when it comes to debug or even
  understand the curl<->server interaction.
 
  Sometimes even --verbose is not enough. Then --trace and --trace-ascii offer
  even more details as they show EVERYTHING curl sends and receives. Use it
  like this:
 
      curl --trace-ascii debugdump.txt http://www.example.com/
 
2. URL
 
 The Uniform Resource Locator format is how you specify the address of a
 particular resource on the Internet. You know these, you've seen URLs like
 http://curl.haxx.se or https://yourbank.com a million times.
 
3. GET a page
 
 The simplest and most common request/operation made using HTTP is to get a
 URL. The URL could itself refer to a web page, an image or a file. The client
 issues a GET request to the server and receives the document it asked for.
 If you issue the command line
 
        curl http://curl.haxx.se
 
 you get a web page returned in your terminal window. The entire HTML document
 that that URL holds.
 
 All HTTP replies contain a set of response headers that are normally hidden,
 use curl's --include (-i) option to display them as well as the rest of the
 document. You can also ask the remote server for ONLY the headers by using
 the --head (-I) option (which will make curl issue a HEAD request).
 
4. Forms
 
 Forms are the general way a web site can present a HTML page with fields for
 the user to enter data in, and then press some kind of 'OK' or 'submit'
 button to get that data sent to the server. The server then typically uses
 the posted data to decide how to act. Like using the entered words to search
 in a database, or to add the info in a bug track system, display the entered
 address on a map or using the info as a login-prompt verifying that the user
 is allowed to see what it is about to see.
 
 Of course there has to be some kind of program in the server end to receive
 the data you send. You cannot just invent something out of the air.
 
 4.1 GET
 
  A GET-form uses the method GET, as specified in HTML like:
 
        <form method="GET" action="junk.cgi">
          <input type=text name="birthyear">
          <input type=submit name=press value="OK">
        </form>
 
  In your favorite browser, this form will appear with a text box to fill in
  and a press-button labeled "OK". If you fill in '1905' and press the OK
  button, your browser will then create a new URL to get for you. The URL will
  get "junk.cgi?birthyear=1905&press=OK" appended to the path part of the
  previous URL.
 
  If the original form was seen on the page "www.hotmail.com/when/birth.html",
  the second page you'll get will become
  "www.hotmail.com/when/junk.cgi?birthyear=1905&press=OK".
 
  Most search engines work this way.
 
  To make curl do the GET form post for you, just enter the expected created
  URL:
 
        curl "http://www.hotmail.com/when/junk.cgi?birthyear=1905&press=OK"
 
 4.2 POST
 
  The GET method makes all input field names get displayed in the URL field of
  your browser. That's generally a good thing when you want to be able to
  bookmark that page with your given data, but it is an obvious disadvantage
  if you entered secret information in one of the fields or if there are a
  large amount of fields creating a very long and unreadable URL.
 
  The HTTP protocol then offers the POST method. This way the client sends the
  data separated from the URL and thus you won't see any of it in the URL
  address field.
 
  The form would look very similar to the previous one:
 
        <form method="POST" action="junk.cgi">
          <input type=text name="birthyear">
          <input type=submit name=press value=" OK ">
        </form>
 
  And to use curl to post this form with the same data filled in as before, we
  could do it like:
 
        curl --data "birthyear=1905&press=%20OK%20"         http://www.example.com/when.cgi
 
  This kind of POST will use the Content-Type
  application/x-www-form-urlencoded and is the most widely used POST kind.
 
  The data you send to the server MUST already be properly encoded, curl will
  not do that for you. For example, if you want the data to contain a space,
  you need to replace that space with %20 etc. Failing to comply with this
  will most likely cause your data to be received wrongly and messed up.
 
  Recent curl versions can in fact url-encode POST data for you, like this:
 
        curl --data-urlencode "name=I am Daniel" http://www.example.com
 
 4.3 File Upload POST
 
  Back in late 1995 they defined an additional way to post data over HTTP. It
  is documented in the RFC 1867, why this method sometimes is referred to as
  RFC1867-posting.
 
  This method is mainly designed to better support file uploads. A form that
  allows a user to upload a file could be written like this in HTML:
 
    <form method="POST" enctype='multipart/form-data' action="upload.cgi">
      <input type=file name=upload>
      <input type=submit name=press value="OK">
    </form>
 
  This clearly shows that the Content-Type about to be sent is
  multipart/form-data.
 
  To post to a form like this with curl, you enter a command line like:
 
        curl --form upload=@localfilename --form press=OK [URL]
 
 4.4 Hidden Fields
 
  A very common way for HTML based application to pass state information
  between pages is to add hidden fields to the forms. Hidden fields are
  already filled in, they aren't displayed to the user and they get passed
  along just as all the other fields.
 
  A similar example form with one visible field, one hidden field and one
  submit button could look like:
 
    <form method="POST" action="foobar.cgi">
      <input type=text name="birthyear">
      <input type=hidden name="person" value="daniel">
      <input type=submit name="press" value="OK">
    </form>
 
  To post this with curl, you won't have to think about if the fields are
  hidden or not. To curl they're all the same:
 
        curl --data "birthyear=1905&press=OK&person=daniel" [URL]
 
 4.5 Figure Out What A POST Looks Like
 
  When you're about fill in a form and send to a server by using curl instead
  of a browser, you're of course very interested in sending a POST exactly the
  way your browser does.
 
  An easy way to get to see this, is to save the HTML page with the form on
  your local disk, modify the 'method' to a GET, and press the submit button
  (you could also change the action URL if you want to).
 
  You will then clearly see the data get appended to the URL, separated with a
  '?'-letter as GET forms are supposed to.
 
5. PUT
 
 The perhaps best way to upload data to a HTTP server is to use PUT. Then
 again, this of course requires that someone put a program or script on the
 server end that knows how to receive a HTTP PUT stream.
 
 Put a file to a HTTP server with curl:
 
        curl --upload-file uploadfile http://www.example.com/receive.cgi
 
6. HTTP Authentication
 
 HTTP Authentication is the ability to tell the server your username and
 password so that it can verify that you're allowed to do the request you're
 doing. The Basic authentication used in HTTP (which is the type curl uses by
 default) is *plain* *text* based, which means it sends username and password
 only slightly obfuscated, but still fully readable by anyone that sniffs on
 the network between you and the remote server.
 
 To tell curl to use a user and password for authentication:
 
        curl --user name:password http://www.example.com
 
 The site might require a different authentication method (check the headers
 returned by the server), and then --ntlm, --digest, --negotiate or even
 --anyauth might be options that suit you.
 
 Sometimes your HTTP access is only available through the use of a HTTP
 proxy. This seems to be especially common at various companies. A HTTP proxy
 may require its own user and password to allow the client to get through to
 the Internet. To specify those with curl, run something like:
 
        curl --proxy-user proxyuser:proxypassword curl.haxx.se
 
 If your proxy requires the authentication to be done using the NTLM method,
 use --proxy-ntlm, if it requires Digest use --proxy-digest.
 
 If you use any one these user+password options but leave out the password
 part, curl will prompt for the password interactively.
 
 Do note that when a program is run, its parameters might be possible to see
 when listing the running processes of the system. Thus, other users may be
 able to watch your passwords if you pass them as plain command line
 options. There are ways to circumvent this.
 
 It is worth noting that while this is how HTTP Authentication works, very
 many web sites will not use this concept when they provide logins etc. See
 the Web Login chapter further below for more details on that.
 
7. Referer
 
 A HTTP request may include a 'referer' field (yes it is misspelled), which
 can be used to tell from which URL the client got to this particular
 resource. Some programs/scripts check the referer field of requests to verify
 that this wasn't arriving from an external site or an unknown page. While
 this is a stupid way to check something so easily forged, many scripts still
 do it. Using curl, you can put anything you want in the referer-field and
 thus more easily be able to fool the server into serving your request.
 
 Use curl to set the referer field with:
 
        curl --referer http://www.example.come http://www.example.com
 
8. User Agent
 
 Very similar to the referer field, all HTTP requests may set the User-Agent
 field. It names what user agent (client) that is being used. Many
 applications use this information to decide how to display pages. Silly web
 programmers try to make different pages for users of different browsers to
 make them look the best possible for their particular browsers. They usually
 also do different kinds of javascript, vbscript etc.
 
 At times, you will see that getting a page with curl will not return the same
 page that you see when getting the page with your browser. Then you know it
 is time to set the User Agent field to fool the server into thinking you're
 one of those browsers.
 
 To make curl look like Internet Explorer 5 on a Windows 2000 box:
 
  curl --user-agent "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" [URL]
 
 Or why not look like you're using Netscape 4.73 on an old Linux box:
 
  curl --user-agent "Mozilla/4.73 [en] (X11; U; Linux 2.2.15 i686)" [URL]
 
9. Redirects
 
 When a resource is requested from a server, the reply from the server may
 include a hint about where the browser should go next to find this page, or a
 new page keeping newly generated output. The header that tells the browser
 to redirect is Location:.
 
 Curl does not follow Location: headers by default, but will simply display
 such pages in the same manner it display all HTTP replies. It does however
 feature an option that will make it attempt to follow the Location: pointers.
 
 To tell curl to follow a Location:
 
        curl --location http://www.example.com
 
 If you use curl to POST to a site that immediately redirects you to another
 page, you can safely use --location (-L) and --data/--form together. Curl will
 only use POST in the first request, and then revert to GET in the following
 operations.
 
10. Cookies
 
 The way the web browsers do "client side state control" is by using
 cookies. Cookies are just names with associated contents. The cookies are
 sent to the client by the server. The server tells the client for what path
 and host name it wants the cookie sent back, and it also sends an expiration
 date and a few more properties.
 
 When a client communicates with a server with a name and path as previously
 specified in a received cookie, the client sends back the cookies and their
 contents to the server, unless of course they are expired.
 
 Many applications and servers use this method to connect a series of requests
 into a single logical session. To be able to use curl in such occasions, we
 must be able to record and send back cookies the way the web application
 expects them. The same way browsers deal with them.
 
 The simplest way to send a few cookies to the server when getting a page with
 curl is to add them on the command line like:
 
        curl --cookie "name=Daniel" http://www.example.com
 
 Cookies are sent as common HTTP headers. This is practical as it allows curl
 to record cookies simply by recording headers. Record cookies with curl by
 using the --dump-header (-D) option like:
 
        curl --dump-header headers_and_cookies http://www.example.com
 
 (Take note that the --cookie-jar option described below is a better way to
 store cookies.)
 
 Curl has a full blown cookie parsing engine built-in that comes to use if you
 want to reconnect to a server and use cookies that were stored from a
 previous connection (or handicrafted manually to fool the server into
 believing you had a previous connection). To use previously stored cookies,
 you run curl like:
 
        curl --cookie stored_cookies_in_file http://www.example.com
 
 Curl's "cookie engine" gets enabled when you use the --cookie option. If you
 only want curl to understand received cookies, use --cookie with a file that
 doesn't exist. Example, if you want to let curl understand cookies from a
 page and follow a location (and thus possibly send back cookies it received),
 you can invoke it like:
 
        curl --cookie nada --location http://www.example.com
 
 Curl has the ability to read and write cookie files that use the same file
 format that Netscape and Mozilla do. It is a convenient way to share cookies
 between browsers and automatic scripts. The --cookie (-b) switch
 automatically detects if a given file is such a cookie file and parses it,
 and by using the --cookie-jar (-c) option you'll make curl write a new cookie
 file at the end of an operation:
 
        curl --cookie cookies.txt --cookie-jar newcookies.txt         http://www.example.com
 
11. HTTPS
 
 There are a few ways to do secure HTTP transfers. The by far most common
 protocol for doing this is what is generally known as HTTPS, HTTP over
 SSL. SSL encrypts all the data that is sent and received over the network and
 thus makes it harder for attackers to spy on sensitive information.
 
 SSL (or TLS as the latest version of the standard is called) offers a
 truckload of advanced features to allow all those encryptions and key
 infrastructure mechanisms encrypted HTTP requires.
 
 Curl supports encrypted fetches thanks to the freely available OpenSSL
 libraries. To get a page from a HTTPS server, simply run curl like:
 
        curl https://secure.example.com
 
 11.1 Certificates
 
  In the HTTPS world, you use certificates to validate that you are the one
  you claim to be, as an addition to normal passwords. Curl supports client-
  side certificates. All certificates are locked with a pass phrase, which you
  need to enter before the certificate can be used by curl. The pass phrase
  can be specified on the command line or if not, entered interactively when
  curl queries for it. Use a certificate with curl on a HTTPS server like:
 
        curl --cert mycert.pem https://secure.example.com
 
  curl also tries to verify that the server is who it claims to be, by
  verifying the server's certificate against a locally stored CA cert
  bundle. Failing the verification will cause curl to deny the connection. You
  must then use --insecure (-k) in case you want to tell curl to ignore that
  the server can't be verified.
 
  More about server certificate verification and ca cert bundles can be read
  in the SSLCERTS document, available online here:
 
        http://curl.haxx.se/docs/sslcerts.html
 
12. Custom Request Elements
 
 Doing fancy stuff, you may need to add or change elements of a single curl
 request.
 
 For example, you can change the POST request to a PROPFIND and send the data
 as "Content-Type: text/xml" (instead of the default Content-Type) like this:
 
         curl --data "<xml>" --header "Content-Type: text/xml"               --request PROPFIND url.com
 
 You can delete a default header by providing one without content. Like you
 can ruin the request by chopping off the Host: header:
 
        curl --header "Host:" http://www.example.com
 
 You can add headers the same way. Your server may want a "Destination:"
 header, and you can add it:
 
        curl --header "Destination: http://nowhere" http://example.com
 
13. Web Login
 
 While not strictly just HTTP related, it still cause a lot of people problems
 so here's the executive run-down of how the vast majority of all login forms
 work and how to login to them using curl.
 
 It can also be noted that to do this properly in an automated fashion, you
 will most certainly need to script things and do multiple curl invokes etc.
 
 First, servers mostly use cookies to track the logged-in status of the
 client, so you will need to capture the cookies you receive in the
 responses. Then, many sites also set a special cookie on the login page (to
 make sure you got there through their login page) so you should make a habit
 of first getting the login-form page to capture the cookies set there.
 
 Some web-based login systems features various amounts of javascript, and
 sometimes they use such code to set or modify cookie contents. Possibly they
 do that to prevent programmed logins, like this manual describes how to...
 Anyway, if reading the code isn't enough to let you repeat the behavior
 manually, capturing the HTTP requests done by your browers and analyzing the
 sent cookies is usually a working method to work out how to shortcut the
 javascript need.
 
 In the actual <form> tag for the login, lots of sites fill-in random/session
 or otherwise secretly generated hidden tags and you may need to first capture
 the HTML code for the login form and extract all the hidden fields to be able
 to do a proper login POST. Remember that the contents need to be URL encoded
 when sent in a normal POST.
 
14. Debug
 
 Many times when you run curl on a site, you'll notice that the site doesn't
 seem to respond the same way to your curl requests as it does to your
 browser's.
 
 Then you need to start making your curl requests more similar to your
 browser's requests:
 
 * Use the --trace-ascii option to store fully detailed logs of the requests
   for easier analyzing and better understanding
 
 * Make sure you check for and use cookies when needed (both reading with
   --cookie and writing with --cookie-jar)
 
 * Set user-agent to one like a recent popular browser does
 
 * Set referer like it is set by the browser
 
 * If you use POST, make sure you send all the fields and in the same order as
   the browser does it. (See chapter 4.5 above)
 
 A very good helper to make sure you do this right, is the LiveHTTPHeader tool
 that lets you view all headers you send and receive with Mozilla/Firefox
 (even when using HTTPS).
 
 A more raw approach is to capture the HTTP traffic on the network with tools
 such as ethereal or tcpdump and check what headers that were sent and
 received by the browser. (HTTPS makes this technique inefficient.)
 
15. References
 
 RFC 2616 is a must to read if you want in-depth understanding of the HTTP
 protocol.
 
 RFC 3986 explains the URL syntax.
 
 RFC 2109 defines how cookies are supposed to work.
 
 RFC 1867 defines the HTTP post upload format.
 
 http://curl.haxx.se is the home of the cURL project


출처 : curl.haxx.se



Trackback 1 Comment 0
2012.11.19 18:49

Burp Suite Free Edition v1.5 released

Burp Suite Free Edition v1.5 is now available to download. 

This is a significant upgrade with a wealth of new features added since v1.4. The most notable of these are described below.

User Interface

Burp's UI has been completely overhauled, to improve looks and usability:
  • Fonts are now scalable throughout the UI, with corresponding resizing of all UI elements (tables, dialogs, buttons, etc.).
  • There are configurable hotkeys for all common functions.
  • Intruder and Repeater now have smart tabs, which you can drag to reorder, and click to create, close or rename.
  • Tables are natively sortable everywhere, except where the row ordering is part of the options you are configuring.
  • Text fields now have context-aware auto-complete memory.

Burp Proxy

You can now add comments and highlighting to items as they appear in the Proxy intercept window. This is useful when manually stepping through an application, allowing you to annotate interesting requests as they are made, and then return to these in the Proxy history for further investigation.

You can now bind Proxy listeners to specific IP addresses, in addition to the loopback interface and all interfaces.

Burp now implements sslstrip-style functionality, allowing you to use non-SSL-capable tools against HTTPS applications, or to perform active MITM attacks against users who begin browsing using HTTP:
  • You can force the Proxy to use HTTPS in outgoing requests, even if the incoming request did not use HTTPS. This is configured per-listener, in the request redirection settings.
  • You can configure the Proxy to convert all HTTPS links in responses and redirects to use HTTP.
  • You can configure the Proxy to remove the secure flag on set cookies, so that browsers will still submit them over HTTP.
  • You can disable the Proxy web interface (at http://burp) and suppress Burp error messages in responses. These options can be useful to mask the presence of Burp from users who connect via it.

Burp Intruder

There is a new ECB Block Shuffler payload type. This is designed for testing ECB-encrypted tokens and other data, to check their vulnerability to block shuffling attacks.

Burp Intruder now has improved extract grep functionality, which lets you define each extract grep location simply by selecting into the base response, or, during a live attack, by selecting into any result response that contains interesting content (such as an error message).

JSON is now fully supported, with automatic placement of payload positions and syntax colorizing in the message viewer.

Burp Repeater

The context menu now has a Paste URL as request item. This configures Repeater to make a GET request using the URL on the clipboard. The headers included within this request are taken from the request headers defined in the Spider options.

The context menu now has an Add to site map item, to facilitate manual content mapping.

Networking / HTTP

Burp now supports streaming HTTP responses, and handles these in a way that lets you and the application continue working. Streaming responses are often used for functions like continuously updating price data in trading applications, where the server keeps the response stream open, pushing further data in real time as this becomes available. Because intercepting proxies use a store-and-forward model, they can break these applications - the proxy waits indefinitely for the streaming response to finish, and none of it is ever forwarded to the client. Burp now lets you specify which URLs return streaming responses. The Proxy tool will pass these responses straight through to the client as data is received. The Repeater tool will update the response panel in real time as data is received. Other Burp tools will ignore streaming responses and will close the connections.

There is a new option to drop all out-of-scope requests. Using this option prevents Burp from issuing any requests to out-of-scope URLs, even if they are requested via the Proxy, Repeater etc. You can use this option based on the defined suite-wide scope or on a custom scope. 

Burp now handles Android SSL connections, implementing a workaround to accommodate the non-standard CONNECT requests issued by Android devices and the Android emulator.

Session handling

Various features have been added to the session handling support:
  • You can now define custom parameter locations within macro responses. In today's Ajax-heavy applications, items such as anti-CSRF tokens are often transmitted to the client within JavaScript or JSON structures; client-side code then processes the item and sends it within the relevant parameter in subsequent requests. You can now configure arbitrary named parameter locations within a macro response. For example, this allows you to specify that a particular JavaScript string contains a parameter with a specific name. When updating parameters in subsequent requests, Burp will update any parameter with that name using the value extracted from the configured location in the prior response
  • In the "Run macro" action, there is a new, default-off option to tolerate a mismatched URL when attempting to match parameters from the final macro response. This is useful for URL-agnostic anti-CSRF tokens, and enables you to configure a single macro to retrieve a valid token, which you can use in requests to multiple URLs, considerably simplifying the necessary Burp configuration in some applications.
  • In the "Run macro" action, there is a new, default-on option to URL-encode parameter values in the current request that have been derived from the final macro response.
  • The session handling cookie jar now tracks cookie expiration times.

Documentation

Burp now includes full help documentation within the software itself:
  • New help documentation is completely rewritten and up to date.
  • Comprehensive - 65,000 words.
  • Logically organized into 300 individual sections.
  • Includes every Burp function and configuration option.
  • Step-by-step "getting started" help for newbies.
  • Detailed help on using Burp in your testing methodology.
  • Advanced topics for Burp power users.
You can open the main help window via the Help menu. Contextual help is also provided throughout Burp. Next to any function or option, you can click the "?" button to view relevant help in a pop-up. And if necessary, you can drill down from there into the main help itself.


출처 : http://blog.portswigger.net/



Trackback 0 Comment 1
  1. Favicon of http://www.nascarlivetv.com/ Wtach Nascar Online 2012.11.30 15:12 address edit & del reply

    非常好的东西,好的想法和概念,很多伟大的信息和灵感,这两者我们都需要的,有用的信息。我想感谢你的努力,你在写这篇文章的。