Assigning Basic Authorization HTTP Header to HttpWebRequest

If you're making a call to a HTTP resource that requires a Basic Authorization HTTP header, you can use the following code:

HttpWebRequest request = (HttpWebRequest)WebRequest.Create("url-here");
request.Credentials = new NetworkCredential("username", "password");

Then post the request in the usual way.

The confusion comes because on the first call the HTTP header will not be present on the request. The framework is relying on the first call receiving a 401 response, with a WWW-Authenticate header present, giving a Basic realm=<realm-name> value. A second call will then be made with the correct headers in place.

If you are just consuming the resource, chances are the server will respond in a way that allows this behaviour. However, if it doesn't respond with a 401, or that 401 response doesn't contain the WWW-Authenticate header (or you just don't want to make 2 calls), then you have to manually add the Authorization header to the request:

byte[] authBytes = Encoding.UTF8.GetBytes("user:password".ToCharArray());
request.Headers["Authorization"] = "Basic " + Convert.ToBase64String(authBytes);

Writing XML with UTF-8 Encoding using XmlTextWriter and StringWriter

If you want to use XmlTextWriter to write XML into a StringBuilder you can create the XmlTextWriter like this:

StringBuilder builder = new StringBuilder();
XmlWriter writer = new XmlTextWriter(new StringWriter(builder));

But this generates a declaration on the resulting XML with the encoding of UTF-16 (the encoding of a .Net String). There doesn't seem to be a straightforward way of making this declaration UTF-8 in this set up.

You can, of course, use a MemoryStream instead of a StringWriter, and then use Encoding.UTF8.GetString(...) to convert the bytes to a string, but doing this made the resulting string have non-printable characters in it, which we don't want.

The solution is to subclass StringWriter and override the Encoding property. Sounds a bit overkill for a solution, but it works very well. Just create the following class (based on Jon Skeet's class):

public class StringWriterWithEncoding : StringWriter
{
Encoding encoding;

public StringWriterWithEncoding (StringBuilder builder, Encoding encoding)
:base(builder)
{
this.encoding = encoding;
}

public override Encoding Encoding
{
get { return encoding; }
}
}

Then use StringWriterWithEncoding instead of StringWriter in your XmlTextWriter.

This operation requires IIS integrated pipeline mode

If you're getting an error message

This operation requires IIS integrated pipeline mode

when trying to add headers to a HttpResponse object in an ASP.Net web application, chances are you're trying something like this:

response.Headers["myHeader"] = "someValue";

or

response.Headers.Add("myHeader", "someValue");

A quick Google reveals mentions of bugs in the 3.5 framework, but I'm using .Net 2. Nevertheless you can work around this problem by adding your header this way instead:

response.AddHeader("myHeader", "someValue");

And your code should now add the header successfully.

Custom ThreadPool

Last year in a post from the last day at Tech Ed I mentioned a talk about building Scalable ASP.NET Web Applications, in which a warning was given about using the normal .Net ThreadPool in web apps. A suggestion was made to instead use a custom thread pool so that ASP.NET's worker threads weren't all used up.

In that talk it was said there was code for such a class on the Wintellect website, but I remember looking at the time and couldn't find anything.

I've since found Mike Woodring's .NET Sample Page, on which there is a custom thread pool inplementation available for download.

I've not needed to use it yet, so can't comment on its usage, but I'll give it a go when I need control over the ThreadPool.

Force a C# Web Service Proxy to use HTTP 1.0

(Code at the bottom)

Esendex customers can submit SMS using .Net Web Services. While the simple code samples we provide will get most people up and running, the samples aren't always enough when volumes start to increase.

The most common error report we receive from these high usage customers usually contains a complaint that our APIs are down, and aren't responding, but this is rarely the case. Our server array offers all our customers the responsiveness and the reliability that they will need, but the error report always seems to look like this is the case.

Normally the problem is at the customers side, and more often than not it's due to them consuming our Web Services using HTTP 1.1.

A limitation of HTTP 1.1 is that it only support 2 simultaneous connections to another server. This is usually OK, but if a customer has a multi-threaded application then this can quickly become a problem. The problem manifests itself with an error that looks like the server is down.

In reality what you'll have is a bunch of threads all waiting for a chance to connect, and when they don't you tend to get timeout errors that make it look like the server didn't respond.

To get around this you need to connect using HTTP 1.0, but by default Visual Studio creates a web reference using 1.1. So you need to do some tweaking.

Basically we need to override the GetWebRequest method on our generated web reference and alter the properties on the HttpWebRequest so that we can assign the right version.

When you add a Web Reference in Visual Studio you effectively add an auto generated class to your project, which (if you look down the directory tree in Windows Explorer) is saved in a Reference.cs file. This file changes every time you tell Visual Studio to update the web reference.

Because of this it isn't always possible to override that particular class. If you working with an API that can change and you want to be able to update it easily, then you'll have to use partial classes in order to override the method. Or you could sub class the proxy and override it in there.

Personally I like to move the Reference.cs file into the project as a normal file (renamed of course), and then edit the generated code. I can do this as I know that the API isn't going to change (if we do need to offer new functionality we release new Web Services rather than change existing ones).

This allows you to override the method in the normal way, and also allows you to change how the proxy gets the URL to connect to.

Whichever way you do it, here's the code you need to add in:

protected override WebRequest GetWebRequest(Uri uri)
{
HttpWebRequest request =
  (HttpWebRequest)base.GetWebRequest(uri);
request.ProtocolVersion = System.Net.HttpVersion.Version10;

return request;
}

And, as you've now got access to the HttpWebRequest you can also disable Keep Alives if they're causing you problems as well.

What skills should a graduate have?

There's a question What Skills Should Undergrads Have running on slashdot from the aftermath of the whole "Java is damaging students" article. There's a lot of answers already but I think one of the important ones is being able to visualise how software connects to each other.

As a graduate programmer it's highly unlikely that you will be given a green field development project. It's more likely that you'll be maintaining or enhancing an existing system, so it's important to be able to see in your mind how and where the different components connect.

I've tried to answer questions like this before, but it's so difficult as a professional developer needs to have so many skills it's impossible to narrow down even a few as being the most important. But I guess when you're starting out in the business you need to:

• Realise that you don't know anything
• Be willing to learn
  
• Not be afraid to ask questions
• Be able to analyse problems and visualise solutions

Technical ability is of course important, but you need to interview well and show some potential to get a decent job out of it.

A friend of a friend I met in sixth form college once claimed that he didn't need any computing qualifications to get a job, as his proven ability at writing his own programs would set him apart from everyone else. And admittedly he had written a lot of programs.

Yet from an employer's point of view all that work he's done hasn't been in a commercial environment where change requests are common, priorities change, your projects are moved to other developers, and you're asked to work on projects of which you have no knowledge.

The experience he had of writing programs didn't cover any of that, and being flexible enough to handle those situations is a very desirable attribute to employers.

I don't know what he's doing now. Maybe he's carrying on doing what he wants and making a living out of it. But even if that worked for him it's certainly not a path I would recommend for people wishing to get a foothold in the industry.

Needle in a haystack

Been a bit quiet on the blogging front recently. That's mainly because I'm currently running through some tests for the first iteration of an update that will drastically change the architecture of the Esendex Messaging System. I can't go into detail about the change, but my current tests run into a very intermittent failure.

At irregular intervals one of the services hangs: just totally stops responding, and because it's multithreaded the log files aren't much help. Actually, its worse than looking for a needle in a haystack, as at least in that case you know what you're looking for.

It first presented itself after sending only 100 messages, then after 2,000. An initial investigation didn't shine any light on anything so I sent some more messages. It hasn't failed again yet, and I've currently sent 15,000 through. No code change either, so the error is still in there waiting to be found.

Frustrating is not the word.

Neat Pattern for Cloning with Inheritance

For once I've come across an MSDN forum post which exactly solves a problem I had. I'm not having the same problem as the person who asked the question, but the solution given was spot on for me.

Suppose you have two classes, SuperClass and SubClass, where SubClass inherits from SuperClass. If I want to create a deep copy of SubClass I should really implement a Copy method in SuperClass, and override that in SubClass.

But if I do that I can't really use the Copy method in SuperClass, because I can't cast a SuperClass to a SubClass.

The solution (as described in the above post: I can claim zero credit for this neat pattern), is to instead create a CopyFrom method, in which a SuperClass can be populated from another SuperClass. Then in the SuperClass's Copy method I create a new SuperClass, and call the CopyFrom method on it, passing in "this" (or "Me", if in VB.Net).

Then SubClass can override CopyFrom and call into the base method, adding any further properties that SubClass has. SubClass then overrides the Copy method, creates a new SubClass, and calls CopyFrom before returning it.

This is the code from the post above (credit to "nobugz"):

Class A
  Private DataA As String = "Data-A"
  Public Sub New()
  End Sub
  Public Overridable Sub CopyFrom(ByVal obj As A)
    Me.DataA = obj.DataA
  End Sub
  Public Overridable Function Clone() As A
    Dim NewA As New A
    NewA.CopyFrom(Me)
    Return NewA
  End Function
End Class

Class B
  Inherits A
  Private DataB As String = "Data-B"
  Public Sub New()
  End Sub
  Public Overrides Sub CopyFrom(ByVal obj As A)
    obj.CopyFrom(Me)
    Me.DataB = CType(obj, B).DataB
  End Sub
  Public Overrides Function Clone() As A
    Dim obj As B = New B
    obj.CopyFrom(Me)
    Return obj
  End Function
End Class

Log Visualiser using a particle system

I carried on my attempts to make logging output look a bit interesting, and what I think I wanted was a stream of pixels flowing down from a server name to give a waterfall effect. Each pixel streaming down would represent one log event, and the colour of the pixel would be determined by the log level of that event.

I set out to implement this, but then thought that someone else must have come up with a particle system for XNA already. And they have, check out this particle sample on the XNA Creators Club.

This sample shows how you can make explosions and smoke effects in XNA, and the end result is surprisingly realistic. I took this example and derived my own class from the abstract ParticleSystem class. It took some experimentation with the parameters I used, but now I've got a stream of pixels flowing in the way that I planned.

I've also got a mock application that just generates log entries for the XNA application, and I've generated 10,000 logging events in a very quick sequence. As the XNA application doesn't store the events for very long there doesn't appear to be much slow down, but it would be interesting to load test it against a number of servers generating a high volume of logs. At the moment I think there would be a slow down but I'm pretty sure I can refactor the event handling so that it will be more scalable.

There's some cleaning up to do, mainly with the positioning of new server entries and externalising a number of parameters to a config file, but I think I'm mostly there.

Visualising Logs - Progress Report

Just a short one as it's getting late now. I've continued a little with my XNA log viewer mentioned previously.

I've set up a mock application that generates logs on command, and I've set up log4net in the same way you would if you wanted to look at the logs in Chainsaw. So this fires off UDP packets containing the entry in XML. I've then got a class that listens asynchronously to a socket. When it receives data it deserialises it to an object, and then raises an event.

My XNA "game" has one of these socket listening classes, and provides an event handler to receive the logs.

On screen right now I'm just scrolling the messages. To do this I have a GameComponent that contains a rough implementation of a generic circular queue. The contents of this queue are drawn to the screen, and as it's always wrapping around, it gives the effect of scrolling without having to move the position of the string.

Next I need to figure out how I want the logs to be visualised.

Do you use goto in C#?

Looking through the code for the SmtpClient.Send method in Reflector and found this snippet:

switch (this.DeliveryMethod)
{

case SmtpDeliveryMethod.SpecifiedPickupDirectory:

if (this.EnableSsl)
{

throw new SmtpException(SR.GetString(
"SmtpPickupDirectoryDoesnotSupportSsl"));

}
break;

case SmtpDeliveryMethod.PickupDirectoryFromIis:

if (this.EnableSsl)
{

throw new SmtpException(SR.GetString(
"SmtpPickupDirectoryDoesnotSupportSsl"));

}
goto Label_021C;

default:

goto Label_022B;

}
MailWriter fileMailWriter = this.GetFileMailWriter(
this.PickupDirectoryLocation);
goto Label_025D;

Label_021C:

fileMailWriter = this.GetFileMailWriter(
IisPickupDirectory.GetPickupDirectory());
goto Label_025D;

Label_022B:

this.GetConnection();
fileMailWriter = this.transport.SendMail((message.Sender != null) ?
message.Sender : message.From, recipients,
message.BuildDeliveryStatusNotificationString(),
out exception);

Label_025D:

this.message = message;
message.Send(fileMailWriter, this.DeliveryMethod != SmtpDeliveryMethod.Network);
fileMailWriter.Close();
this.transport.ReleaseConnection();
if ((this.DeliveryMethod == SmtpDeliveryMethod.Network) && (exception != null))
{

throw exception;

}

I've always been taught to avoid goto like the plague, and it's something I've followed for years. I've never even considered it to be an option when writing code.

Thinking about it though, I did read an article a few years ago (which I can't find now) which said you could credibly use gotos only if you used it to jump down code, and not up: a rule that is stuck to here.

The thing is, I know this code can be written without using goto. But also, it might not be as readable as it is now.

I'm conflicted. But I'm sticking to the "no goto" rule until someone can come up with an example that really can't be coded without using a goto.

Documenting assumptions with Debug.Assert

Another one from Greg Beech today: Document your assumptions with Debug.Assert instead of comments.

I've never used Debug.Assert, but after reading this post I'm trying it out today. The post makes some excellent points:

When writing methods people frequently make assumptions, such as that the internal state of the class is correct, or that the arguments passed into a private method are valid (because they should have been checked by the non-private method calling it)

[...]

[If these assumptions are wrong] the program will carry on regardless until it fails at some point in the future. And because you didn't trap the problem at the point where it became clear that something was already wrong (i.e. at the comment documenting the assumption) then it's much harder to work out where the problem really occurred because the failure itself could occur further down the line in a largely unrelated method.

[...]

So every time you find yourself making an assumption "I know this will be there", "this should be in that state", "the argument should not be null as the public method will have validated it", document it by using Debug.Assert rather than a comment. Then, if your assumption is incorrect, your application will come to a grinding halt at the point where the problem became apparent.

So I'm giving it a go. It can only help during development, and won't affect anything in production code as the Debug class will only compile in debug builds.

There are a few warnings in Greg's post about when not to use it, so make sure you read those before giving it a go.

Visual Studio 2005 Automation Samples

Guess I should have looked at creating a Visual Studio plugin first rather than figuring out how to convert RTF to HTML. I might not need to do that judging by the description of one of the Visual Studio 2005 Automation Samples:

Generate HTML
This sample demonstrates how to take text from the editor window and build a HTML file using this text, including font and color syntax highlighting information.

Haven't downloaded this yet, but I will do. Sounds like the Blogger code upload plugin could be quite easy to implement.

I found this page linked from the Customize and Extend Visual Studio page on MSDN.

UPDATE: Ahh, the Generate HTML sample is actually in C++, not C#. Should still be able to use it as a guide though, just need to figure out what objects to use.

RTF to HTML and the GData API

Finally got around to properly extracting the RTF conversion code from AutoFormatter last night. It turned out to be as easy as I thought, after last time's wasted efforts. I've had a look on Robert Verpalen's blog for some contact details, but I can't find any and the AutoFormatter post doesn't seem to allow comments.

I might have to leave a comment on another one of his other posts, but the date on the AutoFormatter post is December 2005, so I don't know if he's maintaining it now. Doesn't seem to be any posts since then about the application anyway.

I think I'll wait until I have something usable first though.

Anyway, with that extracted I downloaded the GData API SDK and tried to connect to my blog. The sample code included is very helpful, but it still took a little time to get my head around. I eventually knocked up some code that would add a new draft to my blog, so I think I have everything I need to make the plugin now.

Now I just need to learn about how to make a decent plugin.

Trying to Extract Features from AutoFormatter

I took a look at the AutoFormatter code last night, and tried to make a start on extracting the code that does the conversion into a class library.

I can't say it was an easy task though, and at the end of it all I found I'd done it wrong anyway. What I really need is a class that just converts RTF to HTML, but I had extracted the method that creates the div blocks around the code as well. Surrounding the code in this way does make the end result nice and pretty, but for the purposes of a generic conversion library it's not needed.

It was about 11 o'clock last night when I figured this out. It's always a little disheartening when you find you've just spent the last 3 hours doing something that you now need to discard.

Guess that's what happens when you try to modify code without truely understanding it. I can't see any unit tests in the code to give an indication of the intention of some of the code, or to provide a safety net to the quite severe refactoring that I am attempting.

But I think extracting the RTF to HTML code should be easier than what I did last night. There were a number of options I needed to extract into a class and pass into the method last night, but I'm sure I don't need all of this for the straight conversion.

I'll see how it goes.

Formatting Currency to Pence and Cents

Almost anyone will know how to format a decimal to a currency string. It's nice and easy to make a number like 1.25 format like £1.25, (or $1.25 if you're using a CultureInfo that has a dollar CurrencySymbol in its NumberFormatInfo.

But, say I have the decimal 0.06, and I want this to format to 6p. Or I might have 0.065 which I want to show as 6.5p.

How do I do that?

I understand I may need to multiply this decimal by 100 to get the whole number of pence, but how do I know which symbol to append? Plus, how do I know which end of the string the symbol needs to go?

There is nothing in a NumberFormatInfo that describes these things. The CurrencySymbol gives us our $, £, €, etc, but where's the pence and cents symbols?

Any ideas?

Right now we've got a workaround in place that works, but shouldn't this be a part of the NumberFormatInfo?

I've posted a question on the MSDN Forums about this.

UPDATE: Kevin posted a question on the MSDN Forums for this as well, and got a more satisfactory answer than I did.

Read that post here.

RTF to HTML: AutoFormatter 2.0

Visual Studio copies code from the editor in RTF format, so technically all you need to do is convert RTF to HTML in order to upload it to a blog post, which will form part of the plugin, if I ever get around to making it.

Looking around last night I found AutoFormatter 2.0. This is a .Net 2.0 application written in C# that does a neat job of doing exactly what I need. The HTML it produces is in a scrolling DIV, formatted in Visual Studio colours, and looks quite nice.

In fact, as a standalone application this is just what I need to post code samples, but I started down this path to try the GData API with a Visual Studio plugin. I'm hoping to use part of AutoFormatter to do it though.

If I can extract the conversion code from the AutoFormatter app into a standalone assembly, then theoretically I should be able to use it from the plugin. The code is available for download, and I made a start at extracting the required elements last night. If I can get this working then that's a big chunk of the work done I think.

I'll get in touch with AutoFormatter's author if I manage to get the core functionality extracted to a separate assembly. Hope he doesn't mind.

Creating Visual Studio Plugins

This is more of a reminder to myself than anything else: Exploring Extensibility: Your First Visual Studio Add-in.

I found it at about the same time I was looking into the GData API, and I had an idea about creating a plugin to Visual Studio which would allow you to upload a code snippet to a blog post, and have it fully formatted so it was readable.

Still might do it, but I'm not sure how indepth the wizard approach will be. I'll probably end up with a standalone application that you paste code into and then upload it.

How C# 3.0 and LINQ Evolved

Starting Visual Studio today the Start Page's MSDN feed pointed to this article: C# 3.0: The Evolution Of LINQ And Its Impact On The Design Of C#.

It describes the separate language additions that were required to make LINQ work, and is well worth a read in my opinion.

Having been introduced to C# and LINQ at Tech Ed earlier this year I'm itching to try some of it out when I get the chance.