ArcaneTek from LowTek

Setting charsets via IIS MIME types is harmful

noreply@blogger.com (LowTek) — Tue, 02 Dec 2008 23:40:00 +0000

Previously, in this post, I recommended that webmasters tweak their IIS MIME type settings so that static HTML files would specify a charset of UTF-8. It turns out that this causes some problems.

The issue is that some web crawlers / search engines send an HTTP request header like the following:

Accept: text/html

If you’ve configured IIS MIME types to send “text/html; charset=utf-8”, this does not match “text/html”, so IIS will actually return an error (i.e. a 40x or 50x HTTP status code), and the page contents will not be returned to the crawler.

Thus, one should not do as I’ve described in this post. Oh well, looks like setting the charset via a Meta Tag is the way to go for static HTML pages.

I happened to discover this issue by running LogParser on my IIS log files, looking for any unexpected HTTP status codes.

MySpace Account Hacking

noreply@blogger.com (LowTek) — Mon, 01 Dec 2008 23:25:00 +0000

After this post, I get a lot of questions like:

I think my spouse/SO/etc. is cheating on me. How do I hack their MySpace so I can find out what’s going on?
Help, someone hacked my MySpace and took over my account. How do I get it back? (Similarly: I forgot my password or somehow it got changed, what do I do now?)

Here’s some info from the MySpace site that may help you recover your account:

For more help, please see MySpace Top FAQs.

Configure IIS to redirect to your preferred URL

noreply@blogger.com (LowTek) — Wed, 27 Sep 2006 07:35:00 +0000

The preferred URL for most web sites includes "www." in the beginning of the URL (i.e. http://www.w3.org/). But sites should still work if you go to the URL without the "www." (i.e. http://w3.org/). But in that case, you don't want the user to bookmark the URL without the "www.".

The solution is to make the site without the "www." be a purely redirection-only site that just redirects to the site with "www.". You can do this in IIS by doing the following:

Create a new Web Site, right-click it and choose Properties, then navigate to the following dialog boxes and configure the site to use a Host Header that doesn't contain "www.".

Switch to the Home Directory tab and make it look like the following:

Explanation:

The special $S and $Q syntax is explained here. Basically, it makes it so that whatever the user specifies after the hostname (including file paths and query strings) is included in the redirection to the destination site. (BTW, that page also explained way more advanced and powerful syntax.)
When using this special syntax, it makes sense to use "The exact URL entered above" so that IIS doesn't append anything else to the destination URL.
"A permanent redirection for this resource" just makes IIS use a 301 Permanent Redirect instead of a 302 Temporary Redirect.

Save web server bandwidth by fixing IIS ETags

noreply@blogger.com (LowTek) — Tue, 26 Sep 2006 07:14:00 +0000

When you request a static file from a web site powered by IIS, IIS will return HTTP headers like the following:

HTTP/1.1 200 OK
Content-Length: 4835
Content-Type: text/html; charset=utf-8
Content-Location: http://servername/index.html
Last-Modified: Tue, 26 Sep 2006 03:35:29 GMT
Accept-Ranges: bytes
ETag: "704349d01ce1c61:d05"
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Date: Tue, 26 Sep 2006 06:29:11 GMT

The ETag header is meant to be a unique value that describes the file being retrieved. Note the "d05" in the ETag value. If you restart IIS and request the same file, the ETag won't be "d05", but will be another value. (It appears to be the IIS metabase "change number" value).

This change in value causes problems if you have a cluster of web servers serving up the same file content, but with different ETags. Because the same file essentially has different ETag values at different points in time, the file content may be retransmitted unnecessarily.

To fix this problem, you can explicitly set the ETag sub-value by setting the MD_ETAG_CHANGENUMBER property in the IIS metabase. It doesn't seem possible to do this by using the adsutil.vbs script, but you can do it by manually editing the metabase.xml file, or by following these instructions:

Download and install the IIS Resource Kit Tools to get the Metabase Explorer.
Run the Metabase Explorer, navigate to COMPUTERNAME -> LM -> W3SVC, right-click it, and choose New -> DWORD Record. Enter the following and hit OK:

Double-click the new "2039" property that was added to the list on the right half of the window. Enter the numeric value that you'd like to use in the ETag instead of a constantly-changing value.

Switch to the General tab and make it look like the following and hit OK:

Now IIS should use the ETag sub-value that you specified instead of of a constantly-changing value. Note that if you somehow make a server configuration change where you want the ETag value to change, you can go back into Metabase Explorer and change the value. Or, delete the "2039" property to go back to the old behavior.

Setting UTF-8 character encoding for all static HTML with IIS

noreply@blogger.com (LowTek) — Tue, 26 Sep 2006 05:34:00 +0000

NOTE: It is not advisable to follow these instructions for the reasons described in this newer post.

If you've done any programming with ASP.NET, you'll notice that by default it outputs this HTTP header:

HTTP/1.1 200 OK
Date: Tue, 26 Sep 2006 05:12:53 GMT
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
X-AspNet-Version: 2.0.50727
Cache-Control: private
Content-Type: text/html; charset=utf-8
Content-Length: 21

Note that UTF-8 is the default character encoding for ASP.NET. That's no surprise considering that W3C recommends it.

If you request a static HTML file from an IIS server, the default Content-type header does not specify any charset:

Content-Type: text/html

To fix this, run the IIS Manager, right-click your machine name, choose Properties, and navigate your way to this dialog box:

By default, there are 3 extensions with the Content-type of text/html: .html, .htm, .hxt. Tweak all of these to have a Content-type of "text/html; charset=utf-8".

Notes:

Contrary to what this page says, IIS does seem to allow spaces in the MIME type value.
Obviously only do this if your static HTML files are really UTF-8.
The cool thing about this technique is it requires no programming, no editing of HTML files, and it even works on static IIS error pages (i.e. 404, etc.).

Prevent IIS from revealing your web server IP address

noreply@blogger.com (LowTek) — Tue, 26 Sep 2006 04:54:00 +0000

If a HTTP client doesn't give a HTTP Host header when communicating with your web server, IIS 6 will output the IP address of your web server in the returned HTTP headers:

HTTP/1.1 200 OK
Content-Length: 4835
Content-Type: text/html
Content-Location: http://<your server IP address here>/index.html
Last-Modified: Tue, 26 Sep 2006 03:35:29 GMT
Accept-Ranges: bytes
ETag: "704349d01ce1c61:1"
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Date: Tue, 26 Sep 2006 04:32:57 GMT
Connection: close

In addition to revealing the IP in the HTTP Content-Location header, it may also display it in redirection responses in the HTTP Location header like the following:

HTTP/1.1 301 Moved Permanently
Content-Length: 153
Content-Type: text/html
Location: http://<your server IP address here>/subdir/
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Date: Tue, 26 Sep 2006 07:21:44 GMT
Connection: close

If you're behind a firewall or NAT, that IP address may be an internal address that you may not want to reveal to potential attackers (if they don't need it for anything legitimate, why reveal it?). To prevent IIS from revealing this info, there is a knowledgebase article.

Tips:

The KB article says to get a hotfix, but I was able to get it to work without installing any special hotfix.
To do the steps in the article, you need the IIS "site identifier" number. To get that, run:
cscript %windir%\system32\iisweb.vbs /query
Ultimately, to configure IIS to stop revealing this info, you just need to run:
cscript %SYSTEMDRIVE%\Inetpub\AdminScripts\adsutil.vbs SET W3SVC/<site identifier>/SetHostName <what you want shown instead of the IP>

How to block an IP address on Windows Server 2003

noreply@blogger.com (LowTek) — Sun, 24 Sep 2006 22:37:00 +0000

Sometimes you may want to prevent a specific IP address from accessing your machine or server. There's a variety of methods with pros and cons:

Application Level

If you really just want to prevent a specific IP from accessing a specific application on your machine, you may be able to reconfigure that app to block specific IPs. For example, if you're using IIS, you can configure it to block IPs in this window:

Note that if you use this IIS feature, you should be aware that it prevents HTTP.SYS kernel mode caching from being used (see the last bullet on this page), though most sites probably won't notice a difference.

Windows Firewall

The Windows Firewall control panel can be used to only allow specific IPs and IP-ranges:

The main problem here is that this is an Allow List, as opposed to a Deny List. So you'll have to "invert" your IP address to accomplish a "block".

Tip: To prevent from locking yourself out of your machine, it may be useful to start a batch file like the following before you change your firewall rules:

sleep 30
netsh firewall set opmode DISABLE

Provided that you have sleep.exe in your %PATH%, this will wait 30 seconds, then disable the Windows Firewall completely (just like clicking 'Off' in the Windows Firewall control panel). The idea is that if you tweak the UI and you accidentally lock yourself out, just wait 30 seconds and the firewall will be disabled. Or, if you tweaked it properly, just Ctrl-C the batch file to prevent the batch file from disabling the firewall.

IPsec

This tutorial explains how to use IPsec on Windows Server 2003 to block specific IPs.

Additional Tips:

If you don't want to use the MMC IP Security Policies snap-in GUI to "assign the policy", you can use this netsh command:
netsh ipsec static set policy name="Packet Filters - Your Web Host" assign=yes
Similarly, to unassign the policy (i.e. stop using these IPsec rules), run:
netsh ipsec static set policy name="Packet Filters - Your Web Host" assign=no
To prevent from locking yourself out of your machine, it may be useful to start a batch file like the following before you tweak the IPsec settings:

sleep 30
netsh ipsec static set policy name="Packet Filters - Your Web Host" assign=no

Provided that you have sleep.exe in your %PATH%, this will wait 30 seconds, then disable the IPsec policy (just like choosing "Un-assign" in the IP Security Policies snap-in GUI). The idea is that if you tweak the IPsec settings and you accidentally lock yourself out, just wait 30 seconds and the IPsec settings will be deactivated. Or, if you tweaked it properly, just Ctrl-C the batch file to prevent the batch file from deactivating the settings.
If you make your IPsec block list too big, you may encounter a perf slowdown. For example, as a test, I put 7,000 IP addresses in an IPsec block list and CPU usage went through the roof when the machine was under network stress. I think in practice if you keep the number low, you won't notice any perf change so this is still a very useful technique.

Ask your ISP

The main problems with all the approaches described above are the following:

You're still paying for the bandwidth for all the traffic reaching your box, even if you discard it using the methods above.
It still consumes resources of your machine to discard traffic.

If you ask your ISP/webhost/colo/upstream to block IPs, they may be able to solve these issues for you, though obviously then you may lose some agility in terms of how quickly you can add/remove IPs from the blocklist, etc.

Network Analyzers and Top Talkers for Windows

noreply@blogger.com (LowTek) — Sun, 24 Sep 2006 01:54:00 +0000

When diagnosing networking problems, it can be useful to have two tools on hand:

A "Top Talker" tool that will show what connections/hosts are using the most bandwidth.
A "Network Analyzer" that will capture and decode raw packets that are sent/received.

I recently tried out a few and here's my quick take. Really I was looking for a "network debugger" to help figure things out when things go bad.

Note: Many of these tools use the WinPcap kernel mode device driver. I'm typically pretty paranoid of kernel mode device drivers, but the WinPcap guys seem like they know what they're doing. If you ever want to turn off their driver, just run "net stop npf".

OmniPeek Personal

Powerful free version of WildPackets' commercial network analyzer. Has a lot of different, useful analysis views and "experts" to dig into packets, plus all analysis can be done while a capture is in progress. If you're going to only install one tool, this is probably it. OmniPeek installs its own kernel mode device driver, but at least it doesn't run unless you're actively using OmniPeek.

Wireshark

Formerly known as Ethereal, this is a popular Open Source network analyzer. Very feature filled, but I found the UI to be somewhat rougher than OmniPeek. Lots of nice features though, especially reconstruction of TCP conversations and a few Top Talker views. It did seem slower to analyze than OmniPeek, even for a surprisingly small capture. Uses WinPcap.

SmartSniff

Very clean UI with only one view of complete conversations (as opposed to individual packets). No real protocol decoding. Small and light, but not really meant to be used to dig into problems. Still, the conversation view is very easy to use. Uses WinPcap.

PRTG Traffic Grapher

PRTG's main focus is bandwidth consumption, so it handily solves the Top Talker scenario and has the most bandwidth reporting of all the tools I tried. Unfortunately, it has some quirks: it runs two separate processes on your machine (plus causes some Service Control Manager Event Log warnings due to this odd behavior); and, it doesn't use the regular WinPcap driver, but another instance of the driver under another name, WOEM_3_2. Ultimately, I think the OS integration could use a little more polish.

Show Traffic

Simple, straightforward UI to show Top Talkers. Small, lightweight tool, but the UI appears to update too often, slowing things down. Uses WinPcap.

ntop

Lots of powerful web-based bandwidth reports. The Windows version is only a limited-demo, unless you recompile it from source code or if you register the software (I couldn't exactly tell whether a donation is required or merely suggested). Or maybe this version doesn't have these requirements (but do you really want to run a non-official version?). Uses WinPcap.

Conclusion

For now I'm going to go with OmniPeek Personal as a real swiss-army knife to investigate any problem. As a backup, I'll install WinPcap ahead of time in case I want to run one of the other tools. :-)

Viewing your kernel mode device drivers

noreply@blogger.com (LowTek) — Sat, 23 Sep 2006 21:26:00 +0000

Poor quality kernel mode device drivers can cause system instability or crashes. You can view the kernel mode device drivers used on your system by doing the following:

Start Menu -> Run -> msinfo32
Go to System Summary -> Software Environment -> System Drivers

Speed up your Internet by limiting your upload speed with WinTC

noreply@blogger.com (LowTek) — Sat, 23 Sep 2006 20:55:00 +0000

Are you sick of reading bogus "speed up your Internet" tweaking instructions? Instead of bogus instructions, here's some actual free techniques that can be experimentally tested to see that they really work.

First, here's the benchmark that shows bad performance:

On two of your machines hooked up to the Internet, upload a file on both machines. i.e. Send an email with a big file attached, upload a file to some web site, etc.
While that's running on both machines, try to ping some close server on the Internet. For example, I ping a nearby university. You'll notice that the ping-times are 10x worse when both machines are uploading than when they're not uploading.

The instructions below explain how to use WinTC to limit the upload of the two machines so that your ping times will improve (i.e. when I do this, my ping times return back to their normal range even while the uploading is going on). Follow the instructions on both machines:

Follow these instructions to install the QoS Packet Scheduler on your system. Most likely you already have this (you'll probably already see it listed when you follow the instructions).
Download WinTC from this site. Expand the files to somewhere convenient like %ProgramFiles%\wintc
Run the following to create a NT Service on your machine that will apply the bandwidth rules we're about to configure:
wintc -kc
Edit the wintc.conf file to have contents like the following:

# Uncomment this line if you only want this to apply
# to your first network adapter. You may need to
# use something like this if you have a network
# adapter that is disconnected. Run wintc -i
# to see the number-to-NIC mapping that
# WinTC uses.
#
#default_netif = 0

# This limits all upload on the machine to 20K/sec.
# You may have to tweak this for your particular
# connection.
define_flow = myflow 20K prio=3
define_filter = myfilter srcport=0/0
assign_filter = myfilter myflow
Run services.msc and configure the WinTC service to Startup type: Manual if you don't want WinTC's bandwidth rules to automatically apply at startup.
Run the following to start the WinTC service to enable its bandwidth control rules:
net start wintc
Note that any error output will be saved to the event.log file in the directory where you put the WinTC files.
When you want to disable WinTC's bandwidth control rules, just run:
net stop wintc

Once you've got that setup on both machines, try the benchmark scenario described at the beginning of the document. If you don't see a major improvement, try reducing the upload until you see an improvement. Once you see an improvement, it's trivial to run "net stop wintc" and then see how the performance decreases in a few seconds. Then run "net start wintc" to re-enable the rules and watch performance increase, etc. etc.

Further Tweaking Tips

To make changes to the config, edit the wintc.conf file, then run net stop wintc, then net start wintc.
To prevent from hosing yourself, you may want to write a small batch file that does the following:

net start wintc
sleep 30
net stop wintc

This will only use the WinTC rules for 30 seconds (presuming you have sleep.exe in your %PATH% :-)). Thus, if you write a bad rule that kills your network access, it'll only kill it for 30 seconds instead of permanently locking you out of your system. (Obviously, this is most useful when you don't have physical access to your machine where you're using the rules)
Don't forget to check event.log for any error messages.
When the WinTC NT Service is running, run wintc -i -v for verbose information on what rules are in use.
The sample wintc.conf file above limits all traffic, including LAN traffic. The wintc_en.txt file that comes with WinTC explains more advanced syntax to allow full-speed for LAN traffic, but to limit Internet traffic.
WinTC can also be used on web servers, dedicated servers, colocation servers, etc. to limit bandwidth from excessive users, etc.

Making Secure Network Connections with stunnel

noreply@blogger.com (LowTek) — Thu, 21 Sep 2006 04:23:00 +0000

Stunnel is a tool that allows one to encapsulate existing protocols in a secure tunnel. This is similar to how one can do port forwarding with ssh. In addition to preventing the tunnel from being eavesdropped upon, it can also require that both ends be authenticated (i.e. each side can only connect if it proves who it is). The following is a tutorial to secure an example service on port 123 on Windows using stunnel.

On both machines:

Download the stunnel installer from this site and install it.
Download OpenSSL from this site and copy it somewhere on your machine (i.e. put it in %ProgramFiles%\OpenSSL or somewhere convenient). Get the latest version you can find. You need openssl.exe, libeay32.dll, libssl32.dll, zlib1.dll, etc.
Get a reasonable OpenSSL.cnf from somewhere, for example this site. Save it to where you put OpenSSL.

On the server machine that has an unsecured service that you want to secure with stunnel:

Make a directory to contain stunnel configuration files.
cd to the directory that contains OpenSSL.
Run:
openssl req -config openssl.cnf -new -newkey rsa:1024 -days 3650 -nodes -x509 -keyout "<dir you made>\stunnel.server.pem" -out "<dir you made>\stunnel.server.pem"
Answer the questions that it prompts you with. The data you enter will be stored in the certificate files you're making (and it'll be displayed in stunnel when you make connections with stunnel), so it is useful to make this text descriptive.
In the directory that you created, create a stunnel.server.conf file with these contents:

cert = <full path to stunnel.server.pem file in the dir you made>

; Some performance tunings
socket = l:TCP_NODELAY=1
socket = r:TCP_NODELAY=1

verify = 3
CAfile = <full path to stunnel.client.pem in the dir you made>

[myservice]
accept=124
connect=123

The full paths must not be quoted and must not contain spaces. You may have to specify short-filenames like C:\docume~1\SomeUs~1\MyDocu~1\stunnel.client.pem, etc.
Make sure to block port 123 using your firewall so that no one can directly talk to the unencrypted service directly. Also, make sure to open up port 124 which is the port that stunnel will listen on.

On the client machine that will connect to the server over the tunnel:

Make a directory to contain stunnel configuration files.
cd to the directory that contains OpenSSL.
Run:
openssl req -config openssl.cnf -new -newkey rsa:1024 -days 3650 -nodes -x509 -keyout "<dir you made>\stunnel.client.pem" -out "<dir you made>\stunnel.client.pem"
Answer the questions that it prompts you with.
In the directory that you created, create a stunnel.client.conf file with these contents:

cert = <full path to stunnel.client.pem file in the dir you made>

; Some performance tunings
socket = l:TCP_NODELAY=1
socket = r:TCP_NODELAY=1

verify = 3
CAfile = <full path to stunnel.server.pem in the dir you made>

; Use it for client mode
client = yes

[myservice]
; this accept should be local only
accept = 127.0.0.1:123
connect = <hostname or IP addr of server machine>:124

Again, the full paths must not be quoted or have spaces.
Securely copy the stunnel.server.pem file you created on the other machine to this machine, and put it in the directory you created on this machine.
Start stunnel.exe on the client with:
"%ProgramFiles%\stunnel\stunnel.exe" <full path to stunnel.client.conf>
Again, the full path must not be quoted or have spaces.

On the server machine:

Securely copy the stunnel.client.pem file you created to the server machine and put it in the directory you created on that server machine.
Start stunnel.exe on the server with:
"%ProgramFiles%\stunnel\stunnel.exe" <full path to stunnel.server.conf>
Again, the full path must not be quoted or have spaces.

Now everything is setup, so on the client machine you can just connect to localhost:123 and you'll really be connected to port 123 on the server machine, all via stunnel connections.

Explanations

By using verify=3 and specifying the CAfile options, it causes the server to validate the client and the client to validate the server. Only if they know each other (have each other's .pem files) is a connection made.
Once you block port 123 on the server with your firewall, the only exposed port is port 124 on the server and it only allows encrypted communication with known clients.
The way connecting works is your real client software connects to localhost:123. That is the 'client' instance of stunnel which connects to server:124 over an encrypted channel. Then the 'server' instance of stunnel connects unencrypted to server:123 (which is really on the same machine as the server).

Security Best Practices

When running stunnel on the server, run it as a low-privileged user (i.e. non-Administrator).
Configure the Windows Firewall to only allow connections to the stunnel server from specific IP addresses, if possible.
When you don't need the tunnel up, don't run the stunnel server process.

CPU Stress Tool Roundup

noreply@blogger.com (LowTek) — Sat, 16 Sep 2006 06:02:00 +0000

When putting together a new machine, it's a good idea to stress test the CPU (including all its cores) and get it to generate as much heat as possible. Here's some tools I've used

CPU Burn-in

This tool is specially designed to make the CPU emit as much heat as possible. Run one instance for each logical processor on your machine.
Stress Prime 2004 Orthos Edition

Prime95 is well-known to be a CPU stability stress tester. This is based on Prime95, but conveniently stresses all your logical processors in one package.
Super PI Mod

Yet another stress tool. Here's some same instructions on using it to stress multiple cores.

Test your Web Server Performance with WCAT

noreply@blogger.com (LowTek) — Sat, 16 Sep 2006 05:35:00 +0000

This is a quick tutorial on how to use WCAT to benchmark a web server. First, get the IIS 6.0 Resource Kit which includes the WCAT Client and Controller. Then, on your client machines that will help simulate load, do the following:

Install the Resource Kit you just downloaded (you don't need to install it on your web server).
Run "%ProgramFiles%\IIS Resources\WCAT Client\client.reg" to tweak some TCP/IP registry settings that will enable the client machines to make a lot of connections per second, etc. Again, you don't need to do this on the web server itself.
Pick one of the client machines to be the "controller" and run the following on each machine (including the controller itself):

cd /d "%ProgramFiles%\IIS Resources\WCAT Client"
clientloop.cmd <hostname or IP of controller machine>

On the controller machine, create a script.ubr file with the following contents:

[Configuration]
WarmupTime             5s
# Set this to the total number of client machines
NumClientMachines   2
MaxRecvBuffer          64K
CooldownTime           5s
ThinkTime                  0
# Set this to how many clients each actual machine should simulate
NumClientThreads     10
Duration                    30s
Comment                   script.ubr script
CloseMethod              RESET
ConnectEx                 TRUE
AsynchronousWait     TRUE

[Script]
# Try benchmarks with this TRUE or FALSE
SET KeepAlive = FALSE
SET Port = 80

NEW TRANSACTION
    classId = 1
    Weight = 100
    NEW REQUEST HTTP
# Set this to the file to request from the server
    URL = "/"

To run the benchmark, run the following on the controller machine:

"%ProgramFiles%\IIS Resources\WCAT Controller\wcctl.exe" -a <web server hostname or IP address> -z <path to script.ubr>

During the test, information will be output from wcctl.exe and also on the client machines. When the test is complete, a summary will be written to "%ProgramFiles%\IIS Resources\WCAT Controller\wcctl.exe.log.log" on the controller machine.

Tips:

To benchmark static file requests, but not using kernel caching, try accessing a directory URL like "/subdir/" instead of "/subdir/default.htm". Accessing "/subdir/" does not use kernel caching, but accessing "/subdir/default.htm" does use kernel caching. More info.
When benchmarking ASP.NET, try with and without Page Output Caching. When benchmarking Page Output Caching, try changing the VaryByParam value.

Test your Network Performance with ntttcp

noreply@blogger.com (LowTek) — Fri, 15 Sep 2006 07:56:00 +0000

UPDATE: ntttcp is available for download here.

This is a short tutorial on how to use the industry-standard ntttcp tool to bandwidth test a machine's network adapter (aka NIC), drivers, etc. I've used the process below to successfully test 100Mbps Fast Ethernet setups. The test simply simultaneously sends and receives data as fast as possible between two machines. With solid hardware and software, you should get pretty close to the theoretical max of 100Mbps both directions, simultaneously.

First, obtain ntttcp from this site.

Setup both of your machines, by doing the following on both:

Copy ntttcpr.exe and ntttcps.exe to the machine(ntttcpr.exe and ntttcps.exe are just renamed versions of the ntttcp_<arch>.exe file from the download above).
Configure the Windows Firewall to allow incoming connections to ntttcpr.exe at least on your local subnet.
Configure Task Manager's Networking tab to include the following columns:
- Bytes Sent Throughput
- Bytes Received Throughput
- Bytes Sent/Interval
- Bytes Received/Interval

Run the following commands to get ntttcpr listening on each machine:

On machine A, run:
- ntttcpr -p 5001 -w -m 1,0,<ip of machine A> -v -a 4
On machine B, run:
- ntttcpr -p 5101 -w -m 1,0,<ip of machine B> -v -a 4

Start the following commands approximately simultaneously on both machines to start the test:

On machine A, run:
- ntttcps -p 5101 -w -m 1,0,<ip of machine B> -v -a 4
On machine B, run:
- ntttcps -p 5001 -w -m 1,0,<ip of machine A> -v -a 4

TIP: When running ntttcps.exe, make sure you don't run it by giving a complex, full-path.

In other words, when running it, don't run it with a command line like the following:

"C:\dir with spaces\ntttcps.exe" -p 5001 ... <other args>

Instead, try to run it with something simpler like:

ntttcps -p 5001 ... <other args>

If you run it with a complex, full-path, you may encounter a bug in ntttcps where it thinks it is ntttcpr and it won't run correctly.

Anyway, while the commands are running, view Task Manager's Networking tab. Both "Bytes Sent Throughput" and "Bytes Received Throughput" should be at least 80-95% for each. Similarly, the "Bytes Sent/Interval" and "Bytes Received/Interval" should be at least 10,000,000 bytes each (for 100Mbps Fast Ethernet).

I was able to get such performance with cheap commodity machines, network cards, and network switches. On one machine, I could get about 85% throughput receiving, but only 25% throughput sending, when sending and receiving simultaneously. Needless to say, I switched to a better network adapter. :-)

Newegg's reasonable customer service refunds me $5

noreply@blogger.com (LowTek) — Mon, 11 Sep 2006 05:44:00 +0000

The other day I ordered two Seagate 250GB SATA drives from Newegg.com for $79.99 each, but I forgot to use the BUYBARRACUDA discount code which takes $5 off the order. I contacted Newegg Customer Support via their web site and told them that I've been a customer since 2001, plus I've spent over $1,000 at Newegg, including the $1,000 order where I bought the drives.

A few hours later, they emailed me back telling me that they'd refund $5 to my credit card! Thanks Newegg!

Diagnosing a hung system with Windows Server 2003's Emergency Management Services

noreply@blogger.com (LowTek) — Sun, 10 Sep 2006 23:54:00 +0000

The other day I setup Emergency Management Services on one of my machines. The idea is that you connect your machine to another machine via their COM ports, using a null modem cable. Then, from the other machine you can type in some limited commands when the system becomes unresponsive via normal routes (i.e. Remote Desktop/Terminal Services).

The OS setup involves using bootcfg.exe to edit the boot.ini file to add a few switches. Once that is setup, when you boot the OS, the following will come over the serial port:

Computer is booting, SAC started and initialized.

Use the "ch -?" command for information about using channels.
Use the "?" command for general help.

SAC>
EVENT: The CMD command is now available.
SAC>

Enter ? and press return for help:

SAC>?
ch Channel management commands. Use ch -? for more help.
cmd Create a Command Prompt channel.
d Dump the current kernel log.
f Toggle detailed or abbreviated tlist info.
? or help Display this list.
i List all IP network numbers and their IP addresses.
i <#> <ip> <subnet> <gateway> Set IP addr., subnet and gateway.
id Display the computer identification information.
k <pid> Kill the given process.
l <pid> Lower the priority of a process to the lowest possible.
lock Lock access to Command Prompt channels.
m <pid> <MB-allow> Limit the memory usage of a process to <MB-allow>.
p Toggle paging the display.
r <pid> Raise the priority of a process by one.
s Display the current time and date (24 hour clock used).
s mm/dd/yyyy hh:mm Set the current time and date (24 hour clock used).
t Tlist.
restart Restart the system immediately.
shutdown Shutdown the system immediately.
crashdump Crash the system. You must have crash dump enabled.

SAC>

At this point during my investigation my machine was inaccessible from the network, so I entered 'i' for IP address info:


SAC>i
Could not retrieve IP Address(es).

Hmm, that was really suspicious because a few minutes ago I was successfully using Remote Desktop with the machine. So then I figured I'd try to get a Command Prompt on the machine to do further diagnosis:

SAC>cmd
The Command Prompt session was successfully launched.
SAC>
EVENT: A new channel has been created. Use "ch -?" for channel help.
Channel: Cmd0001
SAC>

So far so good. To access the "new channel", I pressed <esc><tab>, which showed the following:

Name:                  Cmd0001
Description:           Command Prompt
Type:                  VT-UTF8
Channel GUID:          ead5a758-408c-11db-998c-0030485adfcb
Application Type GUID: 63d02271-8aa4-11d5-bccf-00b0d014a2d0 

Press <esc><tab> for next channel.
Press <esc><tab>0 to return to the SAC channel.
Use any other key to view this channel.

Then I pressed some key on my keyboard so that I could "view this channel". It allowed me to enter credentials:

Please enter login credentials.
Username: Administrator
Domain :
Password: ********

Attempting to authenticate...

But then it displayed:

The Command Console session is exiting.

??? Very strange. At this point, I thought I was stuck, so I entered "restart" to reboot the system:

SAC>restart
SAC>SAC failed to restart the system.
Failed with status 0xC000009A.

Yikes, what is going on? It sounds like 0xC000009A means out of paged pool. So I ran the 't' command to get more info:

SAC>t
memory: 2095456 kb uptime: 0 0:20:20.609

PageFile: \??\C:\pagefile.sys
Current Size: 2095104 kb Total Used: 3528 kb Peak Used 36040 kb

Memory:2095456K Avail:1527800K TotalWs: 323948K InRam Kernel: 1720K P: 9372K
Commit: 417392K/ 253420K Limit:4039728K Peak: 486732K Pool N:259904K P: 9792K

User Time Kernel Time Ws Faults Commit Pri Hnd Thd Pid Name
32864 44048 File Cache
0:00:00.000 0:13:18.484 28 0 0 0 0 2 0 Idle Process
0:00:00.000 0:00:24.968 236 4626 28 8 372 68 4 System
0:00:00.000 0:00:00.062 452 181 124 11 20 2 536 smss.exe
0:00:00.078 0:00:00.484 3200 1438 1600 13 361 12 624 csrss.exe
0:00:00.109 0:00:00.390 8580 2587 6032 13 476 20 664 winlogon.exe
0:00:00.093 0:00:00.484 3328 966 1496 9 285 16 708 services.exe
0:00:00.093 0:00:00.203 6764 1904 6900 9 393 25 720 lsass.exe
0:00:00.015 0:00:00.000 2524 696 840 8 79 5 900 svchost.exe
0:00:00.359 0:00:00.453 3172 859 1224 8 207 10 984 svchost.exe
0:00:00.000 0:00:00.031 4328 1354 3732 8 133 7 1084 svchost.exe
0:00:00.031 0:00:00.015 4792 1222 2912 8 156 13 1168 svchost.exe
0:00:07.609 0:01:33.750205392 58331 200384 8 909 69 1184 svchost.exe
0:00:00.000 0:00:00.015 4384 1250 3428 8 127 14 1712 spoolsv.exe
0:00:00.015 0:00:00.031 3892 1123 1604 8 149 13 1736 msdtc.exe
0:00:00.000 0:00:00.000 2036 505 524 8 56 2 1876 svchost.exe
0:00:00.000 0:00:00.000 1616 397 368 8 96 3 1912 IAANTmon.exe
0:00:00.000 0:00:00.000 1284 358 300 8 39 2 1956 svchost.exe
0:00:00.000 0:00:02.546 3972 1044 1348 8 64 3 2040 NTService.exe
0:00:00.015 0:00:00.140 4520 1470 2556 8 165 24 488 svchost.exe
0:00:00.000 0:00:00.000 2696 706 740 8 75 6 212 alg.exe
0:00:00.015 0:00:00.046 4384 1572 2464 8 119 4 432 wmiprvse.exe
0:00:00.109 0:00:00.031 4948 1510 1724 8 162 4 1232 wmiprvse.exe
0:00:00.453 0:00:00.359 12980 7658 12724 8 275 13 3716 HelpSvc.exe
0:00:00.000 0:00:00.015 1576 388 368 4 16 1 3380 logon.scr

Note the huge non-paged pool size of 259904K. It's no wonder networking doesn't work and I can't even create a new cmd.exe process on the machine.

At this point, I should have typed "crashdump" to take a full memory dump of the system to analyze later, but I forgot about that and instead I powercycled the system. :-)

But at least now I know what to look for and what to do next time.

Logon to the Console to use Intel PROSet software in Device Manager

noreply@blogger.com (LowTek) — Sun, 10 Sep 2006 22:27:00 +0000

TIP: To use the Intel PROSet software for an Intel Networking Adapter (NIC), logon to the console of the machine. If you're logging in via Remote Desktop (aka Terminal Services), pass the /console option to mstsc.exe to logon to the console session.

If you logon to a regular (non-console) Remote Desktop session, and go to Control Panel -> System -> Device Manager -> Network Adapters -> Intel(R) PRO/xxxx Network connection, it will display the following:

If instead you pass /console to mstsc.exe when logging in via Remote Desktop, you'll connect to the special console session and the Intel PROSet software will be available:

This seems a lot easier and safer than this other solution.

Slipstreaming Windows Server 2003 SP1

noreply@blogger.com (LowTek) — Wed, 06 Sep 2006 08:08:00 +0000

Here's a good guide on slipstreaming SP1 into Windows Server 2003, making a new CD that will directly install Windows Server 2003 SP1 (which is somewhat different than just installing Windows Server 2003 and applying SP1 afterwards).

A few tips I would add:

Update the Support Tools on the CD by getting the Windows Server 2003 SP1 Support Tools.
Update the Deployment Tools on the CD by getting the Windows Server 2003 SP1 Deployment Tools.
To make a bootable CD, use IsoBuster in conjunction with Nero (which probably already came with your burner) by following these instructions. The instructions are for Windows XP SP2, but the bootable CD instructions work for Windows Server 2003 as well.

Secret Dell hard drive diagnostics in the BIOS

noreply@blogger.com (LowTek) — Mon, 03 Jul 2006 23:20:00 +0000

My drive started to fail in my Dell PowerEdge 400SC, so Dell shipped me a new drive. I installed it and Dell Support told me to boot-up and repeatedly press Ctrl-Alt-D. Apparently this runs some sort of built-in BIOS hard drive diagnostics.

In my case, it happened to print out "Fail. Return Code: 4". The tech explained that this means "electrical failure".

I sent back the two bad drives, Dell shipped me a new one and it seems to be running fine. If this happens again on another Dell, I'll be sure to run Ctrl-Alt-D to start...

UPDATE: Looks like this is briefly documented here. I didn't find a list of return codes, but Return Code 7 seems to mean "bad tracks or sectors".

Memory tester software roundup

noreply@blogger.com (LowTek) — Mon, 03 Jul 2006 22:13:00 +0000

I recently got 2 sticks of 1GB memory for my system so the first thing I did was test it using some free memory testing software. I'm too paranoid, so I didn't want to rely on a single app, so I tried the following:

Windows Memory Diagnostic

Direct from Microsoft's Online Crash Analysis team. My guess is that these guys spend all day long looking at uploaded Watson blue-screen dumps and they've figured out "patterns" in crashes due to flaky memory.

Memtest86

Popular GPL memory testing software. One cool feature is that it'll tell you whether your memory is running in Dual Channel, whether PAT is enabled, etc. Though the screen-shot above doesn't show that probably because the screen-shot is from a virtual machine.

Memtest86+

An updated version of memtest86 by more contributors.

DocMemory

Years ago I used this software to detect some bad memory. I tried it again and it didn't seem to work right anymore (i.e. bizarre errors that didn't seem to be related to my memory). In fact, it even gave errors in a virtual machine.

CTU Newhire Guide to Opening a Socket

noreply@blogger.com (LowTek) — Sat, 01 Jul 2006 08:25:00 +0000

NOTE: See this post for information on MySpace Account Hacking.

In FOX's 24, whenever Jack Bauer is in a pinch, resident techgeek Chloe O'Brien has to "open a socket" or "open up a socket to the server". What does it all mean?

Well, I've been doing "network programming" (i.e. writing software that uses the Internet) since the 90's (I even have a web site about it) and I'm here to set everyone straight.

Sockets, schmockets, what's the deal?

Just like a house has multiple phone numbers for multiple land-lines (in a chatty house, one for uncle Bob, one for little Jimmy, one for grandma, etc.), a computer has multiple numbered "ports". For example, port 80 is often used for web sites.

When you make a phone call, you just pick up the phone and call someone's number. With computer "ports", you can't just contact a port on someone's computer -- they also have to be "listening" on that port. The way I think about it, the moment you plugin your phone handset into the wall, your phone is "listening". This web site you're reading right now has a web server program that is "listening" on port 80.

So how did your computer "call" this web site to download this page that you're reading right now?

Believe it or not, but the first thing it did was "open a socket". I kid you not.

Opening a socket is basically like picking up the phone handset to hear the dialtone. It's all just software bookkeeping on your local computer, until the next step:

The next thing your computer (your browser, really) did, was to "connect" that open socket to the listening port on my web site. This is basically the equivalent of connecting the phone call.

Once the socket connection is made, data can be transferred both ways. i.e. your computer requests a specific page, then this web site sends the page.

Socket Cheatsheet

To summarize:

Phone number = computer port number
Plugging in phone handset into wall = program listening on a port
Picking up the phone handset = opening a socket
Dialing the phone = connecting the socket

So what does the show really mean when they open a socket?

My guess is that the writers wanted something cryptic sounding, yet still based in reality. The good thing about using the phrase "socket" is that because sockets are fairly "low level", they don't really imply anything about what you do with them, in the same sense that who knows what people do with phones (i.e. ask for Mom's apple pie recipe, or find the hours of a store, etc.).

Of course, now that I've explained what "opening a socket" is, it just isn't as cryptic anymore. I'd suggest to the writers that they try something even more cryptic, yet still based in reality like this:

CTU newbie: I can't get it to work.
Chloe: You probably forgot to bind your socket.
(Chloe shoots the newbie that "lamer" look.)

Who wants to open a socket?

Everyone who uses the Internet opens up plenty of sockets, but it's all sort of hidden with complex software like browsers, etc. Anyone like Chloe would do something way more hardcore like... before I ramble on, does anyone want to see how to do these things hardcore? Drop me a comment and I'll write-up another part to this guide.

A Note to Hardcore Socket Programmers

Before someone comes down from Division or the Department of Homeland Security to escort me away, yes, I've omitted tons of details above like: IP addresses, binding, the socket used for listening, the socket returned from accept(), TCP vs. UDP, HTTP, protocols, how TCP works, etc., etc. Though I did try to use terminology like "listen" and "connect" based on the BSD socket APIs of the same names.

Use Remote Desktop to trick Virtual PC and Virtual Server into using USB smartcards

noreply@blogger.com (LowTek) — Fri, 30 Jun 2006 06:08:00 +0000

Summary: Microsoft Virtual PC and Virtual Server say that they don't work with USB smartcards, but you can trick it to work by using Remote Desktop.

The Microsoft site says the following about Virtual PC 2004:

Standard USB input devices such as keyboards and mice are supported through PS/2 emulation, but Virtual PC does not support USB devices that require their own drivers.

Similarly, about Virtual Server 2005, it says:

Virtual Server currently does not support USB hardware such as smart card readers and scanners. However, standard USB input hardware, such as keyboard and pointing devices, are supported.

Although it may be true that Virtual PC and Virtual Server don't directly support USB smartcards, you can work-around this issue by doing the following:

In the Guest OS virtual machine (i.e. inside The Matrix if you know what I mean), go into the System control panel, Remote tab and enable Remote Desktop. You'll also need to make sure any firewall in the Guest OS allows port 3389 (Remote Desktop). The System control panel automatically configures the Windows Firewall appropriately.

In your real Host OS, run Start Menu -> All Programs -> Accessories -> Communications -> Remote Desktop Connection, or run mstsc.exe from the command line.

Enter in the hostname or IP address of the Guest OS virtual machine.
Click Options, then click the Local Resources tab and make sure there's a checkbox next to "Smart cards".

Click Connect, login to the Guest OS and run apps, etc. that use your smartcard via the Remote Desktop connection.

Debugging the Internet Explorer View Source problem with Fiddler and fixing ASP.NET

noreply@blogger.com (LowTek) — Wed, 28 Jun 2006 22:44:00 +0000

Summary: This article shows how to figure out why Internet Explorer's View Source command didn't work on an XML document, how it was debugged using Fiddler, and the ASP.NET fix.

You ever try "View Source" on an XML document and get this dialog box that says "The XML source file is unavailable for viewing."?

I got this while creating an ASP.NET page that was outputting RSS. I ran Fiddler and spied on what the web server was returning:

I didn't really see anything wrong with the output, so the next step was Process of Elimination to try removing HTTP headers until the problem didn't occur anymore. So I chose Rules -> Automatic Breakpoints -> After Responses from Fiddler's menus. This causes Fiddler to break-in and effectively pause the HTTP session so that you can change the response before the browser gets it:

Eventually, I tried deleting the Vary: * HTTP header by selecting it, right-clicking, and choosing "Remove Header". This was the header causing the problem! I did a quick check around various RSS feeds on the Internet and I didn't see any that used this header. For the detailed info on what this header is for, see the HTTP spec.

It seems that ASP.NET was outputting this Vary: * HTTP header because I used the following in my page:

<%@ Page Language="C#" CodeFile="t.aspx.cs" Inherits="t_aspx" %>
<%@ OutputCache Duration="300" VaryByParam="*" %>

The reason I used VaryByParam="*" was because my page is dependent on all of the QueryString parameters.

Eventually, after digging around, I found these two pieces of documentation:

So if you put the following in your ASP.NET web.config file, it will no longer output the Vary: * header and then your cached XML output will be View Source-able in Internet Explorer.

<?xml version="1.0"?>
<configuration>
  <system.web>
    <caching>
      
      <outputCache
        omitVaryStar="true">
      </outputCache>
    </caching>
  </system.web>
</configuration>

Proof that Digg has at least 6 web servers

noreply@blogger.com (LowTek) — Tue, 27 Jun 2006 05:22:00 +0000

The Straight p00p: Anyone on the Internet can experimentally determine how many web servers Digg.com has, due to their server configuration. If Digg tweaks their web servers to fix this, they can speed up the site and save everyone bandwidth. Anyone can try these investigative techniques on other sites, too.

The Details: A quick check with Fiddler shows that Digg.com seems to serve all its content from www.digg.com. That hostname resolves to a single IP address (they probably use a load balancer like BigIP), but we can still figure out what's going on by requesting a static file from their site and looking at the HTTP headers. I used the HEAD perl script that comes with perl and the LWP library:

>head http://www.digg.com/img/comment-1.png
200 OK
Cache-Control: max-age=3600
Connection: Keep-Alive
Date: Tue, 27 Jun 2006 05:02:41 GMT
Accept-Ranges: bytes
ETag: "870384-a72-41414d62cbc00" <-----------------
Server: Apache
Content-Length: 2674
Content-Type: image/png
Last-Modified: Thu, 18 May 2006 19:13:52 GMT

The entity tag value by default seems to be composed of the file inode value, the file size, and the modification time.

File inodes are values that make sense on a particular file-system (i.e. they're unique to the volume/partition/file-system on the box). So if the server cluster has multiple filesystems (i.e. content copied around and on each box in the cluster), it's highly likely that the inode values will be different. (of course, this assumption could be wrong if network filesystems or SANs are in use, etc.).

To test this theory, I wrote a short quick-hack perl script that makes 200 HEAD requests and shows how many unique inode values are returned. Here's the output:

>perl -w p.pl
"5c01db-a72-41414d62cbc00"
... // lot more output
"4f01c8-a72-41414d62cbc00"
INODES:

870384 = 24
430251 = 30
4f01c8 = 42
6481c8 = 31
981c0 = 34
5c01db = 39

SIZES:

a72 = 200

MTIMES:

41414d62cbc00 = 200

As you can see, there's 6 unique inode values, so that's how I deduced that Digg probably has at least 6 web servers. Back in April, it was reported that Digg had 3 web servers and just today Digg was stocking up on BestBuy boxes.

What's wrong with differing ETag values? The problem is that if you happen to access server #1 in the cluster and it returns one ETag for that file, then later you go back to the site, your browser will ask the server "give me that file, but don't give it to me if your file matches the ETag I got earlier". Well, if you're talking to server #2 this time around, it has a different ETag (even though the file content is the same), so then it has to retransmit the content to you again, for basically no reason.

Digg could reconfigure their web servers to use the same ETags across their cluster, then this wouldn't be an issue, then guys like me couldn't write these kinds of investigative reports. :-)

The technique I explained in this write-up could be used on other sites as well. Found anything interesting?