Essential HTTP Client Checklist and CFHTTP (Pt. 2)

In a previous post I discussed a list of criteria for a modern HTTP client and compared them to the out of the box features of the Adobe ColdFusion tag CFHTTP.  CFHTTP supports all of the 'must-have' features but provides little in the way of 'optional extras'. Critically common authentication options (such as digest and NTLM) are not supported.

So what alternatives are there to CFHTTP?

Before I look at Java libraries I thought I would do a quick google for a familiar face: The custom tag CFX_HTTP5, available from the CFTagStore. It looks like this ancient custom tag has recently been retested for ColdFusion 8 and shows less signs of being the abandonware it threated to become a few years ago. I can not speak to the tags merits directly since I have never used it, but it claims to support NTLM authentication among other things (like multi-threading, which is pretty redundant now ColdFusion has the CFTHREAD tag).

Here is a list of features from the TagStore. It looks like CFX_HTTP5 is (or was) the top selling ColdFusion tag.

• Take complete control over HTTP request and response (like Referer, Content-type, cookies, redirects, and so on).
  
  
• Control and report authentication scheme to be used in HTTP request: Basic, NTLM, Digest, and others.
  
  
• Fixes most of known CFHTTP-related problems in ColdFusion 4.5, 5.0, and MX.
  
  
• Use tag's asynchronous mode to execute many HTTP requests SIMULTANEOUSLY within the SAME ColdFusion page.
  
  
• Execute HTTP requests on background, while reading a database, for example.
  
  
• Progress monitoring of long-running downloads.
  
  
• Selective downloads based on specified content types.



• Tag maintains its own independent of ColdFusion threading and requests queuing.



• Increase the performance of your Web-site by deploying true multithreaded applications.



• Sessions - chained HTTP requests that share state and authetication credentials.



• Trouble-free HTTPS communications.



• Advanced timeout settings - both TCP/IP and HTTP-based.



• Very small executable. No external dependencies, except WinHttp API.



• Built-in GZIP decompression of the content.



• Built-in base64 encoding of binary content.



• Client-side Digital Certificates support.



• Unicode and Code Pages support (CF MX and later).



• Able to support access to any Web-service.

As you can see the tag supports both NTLM and GZIP compression, but adherence to some of the 'Must-have' features is not readily apparent.

Stay tuned; The next post I will look at closer at the available Java libraries.

About this Blog

This is the personal blog of Ben Davies. Ben is passionate about using ColdFusion to integrate, automate and report the enterprise. Ben lives in Brisbane, Australia, and can offer his services via his company, Helium 3 IT Solutions.

Essential HTTP Client Checklist and CFHTTP (Pt. 1)

I was reading "Restful Web Services" (Leonard Richardson and Sam Ruby, May 2007) and I came across a useful checklist of the 'must-have' and 'optional-extra' features of HTTP client software libraries, along with a comparison of HTTP client libraries for different platforms.

Out of curiosity I am going to measure ColdFusion 8's CFHTTP functionality against the check list to see how it fares, and then compare it to the Java libraries presented (because if CFHTTP fails to deliver your next step will no doubt be Java).

The following is their list of 'must-have' features, along with notes about the support within CFHTTP in ColdFusion 8.

1. Support for HTTPS and SSL (Supported)


2. Support for Methods GET HEAD PUT POST DELETE OPTIONS TRACE and optionally WebDAV extensions (like MOVE) ( Support exists for all HTTP spec methods but no extensions)


3. Customisation of Entity Body of PUT and POST (Supported)


4. Customisation of Request Headers (Supported)


5. Access to the headers and the response code of the provided response(Supported)


6. Support for HTTP Proxies (Supported - So long as you use basic proxy authentication- more later)

Here are the 'optional-extras':

1. Automatic, Transparent HTTP Compression(Not Supported)


2. Automatic, Transparent Response Caching (Using response headers) (Not Supported)


3. Support for Basic, Digest and WSSE Authentication Basic is supported, but not Digest. WSSE is not supported, and NTLM authentication definitely is not either. This one bites!


4. Support for Redirects (Supported - So long as you do not redirect more than 4 times in a valid use case.)


5. Support for Client-side cookie management/acceptance (Despite superb Server-side Cookie capabilities, there is no client-side capabilities here)

In my experience, support for NTLM (or any) authentication should be in the must-have list (especially for corporate environments) but from the context of the author's pespective - RESTful Web Services - I thought the list was very good.

Note that CFHTTP has all the 'must have' features from this list, which is a surprising and pleasing result.

What about the other Java libraries? You'll need to wait for the next blog entry!

Book Review: Managing Software Development with Trac and Subversion

Trac is a Project Management tool written in Python with an integrated issue tracker, wiki and strong integration into the Subversion version control system. By any yardstick the open source project does pretty well when it comes to maintaining documentation.

Like all reasonably complex tools, however, there is an advantage in getting a different or fuller perspective on how to use the tool. And for the time poor (me!), a single, authoritative book describing everything an average user would need to know about a product is well worth the cost.

When I first saw Managing Software Development with Trac and Subversion, by David J Murphy on Amazon I was excited. I had recently set up Trac at work, and although I had stumbled through the installation and configuration, I am left with lingering doubts about how I configured it and how I try to use it day to day.The book arrived last week and I had very quickly read it back to front.

Unfortunately, this is very easy to do since the book is only 105 pages long. That is perhaps my biggest disappointment in the book, since it only provided a light coverage of easiest material to find on the internet.

The book covers:

• Basic software project management advice


• Installing and setting up Trac


• Integrating Subverion


• Basic usage of the tool


• How to 'customise' Trac by modifying the drop-down lists used for forms

For someone already running Trac at a very basic level of expertise, I found little information in the book that I wasn't familiar with. The author made hints throughout the book of some of the more interesting aspects of Trac (email integration, for example, or running multiple projects in side-by-side) but never expanded on them. I found this frustrating.

The version of Trac covered was a .10 release. For such a recent release (2007) I am not sure why the newer .11 release was not covered instead, especially since large strides have been made in removing the need to use the command line to administer the site.

Coverage of Subversion in the book was very brief and not very satisfactory. Effective use of source control is a key aspect of software development - and integration with subversion is a key aspect of Trac - so the lack of detail was puzzling. On the other hand, there are many online excellent quality subversion reference sites and a couple of excellent books, so a reader could always turn to these in a pinch. More worrying was the carte blanch recommendation to always branch working copies and copies and a miniscule reference to the update command.

Many interesting topics were not addressed in any useful depth. I would have like to have seen some coverage of:

• Trac Administration, especially security


• Some of the more popular macros and plugins (there are many), and how to install them


• Alternative configurations (Trac supports a few)


• Using Trac with multiple projects


• Using Trac with Eclipse via the Mylyn Eclipse plugin and Trac connector


• Customising the Trac page templates


• Syndicating the content using RSS, and integrating email

Despite a certain amount of disappointment with this book, if a second edition were released that included items from the list above I would definitely consider buying it.

The book is very easy to read, and includes enough screenshots and command lines examples such that the reader is never lost while reading the text. Having said that, the print quality is on the poor side. Despite a nice looking cover the pages appear more photocopied than printed.

If you have never installed or used Trac before this book would be a useful starting point, but be advised most of this information is available on the Trac website, and you will soon be asking questions.

Gettings Things Done

Recently I spotted a couple of blog entries in ColdFusion blogosphere relating to personal time management and I thought I would chip in a process that has worked for me.

I value time management, because I never seem to have enough. Some people make it look easy (Ray Camden, I'm looking at you). Currently I'm working long hours at my current workplace, establishing a business (red tape, anyone?), trying to keep up to date with what is happening in the ColdFusion community, reading books on the latest technologies, enhancing my basic business site, fixing the glaring problems with this blog, actually adding content to this blog, working 110% on my first company project, doing my tax, doing the housework and somehow keeping my relationships functional. I am pretty busy, and if I can find a way to better keep on top of all my pressing issues I will.

Recently I was definitely not feeling like I was on top of anything. I found myself distracted at work because I kept remembering other things I had to do, many of which were also at work, but also in my other arenas as well. Despite the fact I was managing my work task list, and separate lists for my other things, I found I was not managing my time well.

I took the opportunity to think about why I was struggling so much more now. The easy answer was that I was more busy, but in the my current job that wasn't new. Then I stumbled upon a particular piece of software, which made me think back a few months and realise my problem was not something I was doing now that was the problem. It was something I was not doing.

Cue drumroll...What had happened was that I had stopped using a technique that I had unknowingly become reliant on. Two or more years ago I bought a book from the bargain bin of the local bookstore called Getting Things Done: The Art of Stress Free Productivity by David Allen. I had bastardised the techniques he provided somewhat to the extent I was only using core principles, but had successfully held on to it for a long time. But bad habits had crept into my daily routine, and my email inbox had gone from being always empty to being in dire need of an auto-archive.

So reflecting back, I determined that the methodology had really worked wonders for my productivity, stress levels, and therefore my time management. So I've launched back into using it everyday, and not just for work, but for all areas of my life.

I wouldn't want to try to summarise the way the "Get Things Done" approach works (try here instead), but I will highlight the key points that I believe help me the most:

Immediate thought collection

As soon as your remember something, receive an email you need to think about, or get asked to do something, you record it as a thought in your Trusted System. Being able to immediately know a thought or item for consideration is safe for later contemplation lets you move on and not be bothered about it at random times throughout your working week.

Processing your thoughts in one hit.

You need to come back to those thoughts you've captured, and ideally you set aside a specific time once or twice a day to review them. When you do so, you file them as next actions, reference items or someday/maybe items you can not action now but might want to in the future.

The great thing about this approach is it is incredibly easy once you put the decision making hat on to decide what needs to be done next (if anything) for every single thought. It is very time efficient too - Instead of wrenching your mind away from your current task to assess the implications of a particular off-topic thought, you have a set period where you set aside this planning and decision making process and you save a lot of time in doing it in one spell.

Working Happily

With the system in place, you find you immerse yourself more in your current tasks, because you trust you have not forgotten anything else. You are not trying to remember anything and you do not have any extra 'stuff' in your head beyond the current task. I think the biggest truth exposed by the system is that all of the other baggage in our minds - the internal post it notes to think about this or that- Have a large impact on our productivity. So does the 'context-shifting' we need to do even to think about another topic briefly. If the operation of a person's brain could be compared with the a computer (and it can, because I'm about to :)) you might say that you really notice the difference in performance when the background services are kept to a minimum and you do not need to swap the bulk of your virtual memory to disk every few minutes.

Not Just Me

You might wonder what the website was that clued me into the source of my un-orderedness and reminded me of my previous successful. I stumbled on to the Thinking Rock site and found free software someone had cared to write in support for what I thought was a shonky little bargain bin time management book.

A quick google corrected my misapprehensions though. Getting Things Done - also now abbreviated to GTD - is actually a very well supported movement, and hundreds of web sites have been devoted to it.

I am always very sceptical about these techniques and strategies (and Nigerian princes needing a local 'banking agent' too), because I am very aware that they sell not necessarily based on how effective they are, but how effective they feel, often based on little more than the blurb on the book. But this is one instance where I found my experience bore the promise out - I am managing my work and personal time a lot better now that I have re-implemented some Get Things Done principles, just like I was managing my work time better when I original tried it out.

So for me the methodology is successful. Your mileage might vary (indeed, I still haven't fixed the numerous problems with this blog) but I know I am getting a lot more done and hell, even managing to post a blog entry :)

SVNAnt Undocumented Attribute

Just a quick note to mention that while incorporating SvnAnt into my Ant build scripts I needed to find a way to pass the '-force' switch as I would to the command line client for an export operation.

Although the SvnAnt documentation doesn't mention the ability to pass this switch to the <export> nested tag, I noticed the <delete> tag supported a separate force attribute. I tried it out with the <export> tag and it worked a treat.

SVNAnt on Vista Gotcha

I thought I would quickly blog this issue for anyone else struggling to get SvnAnt working on Windows Vista. I have been trying to streamline my deployments using Ant running within Eclipse and as an important part of that task I wanted to integrate my source control. I attempted to use the SvnAnt task to perform operations (such as export, commit, etc) on my Subversion repository.

However I've just spent over an hour trouble-shooting a problem with this task. The cliff notes version of the resolution is this simple realisation:

You can not use the 'command line' (as opposed to JNI) interface on Vista. Read on for the background.

SvnAnt allows you to specify two ways of connecting to SVN. Using the command line interface (effectively executing the command line binary) or using a native library (javahl) through JNI.

My initial approach was to try to use the command line. the subversion binary was already on my path and I knew it worked OK. Although the JNI implementation is reportedly more performant I tried to attack the simplest approach first.

However while experimenting I kept getting the simple error:

Cannot use javahl nor command line svn client


Documentation on the web (provided by my friend and yours, Google) provided me many references to this problem. It seems that with respect to the command line client interface, the PATH environmental variable must include the subversion binary directory. This had me scratching my head, since the subversion binaries were on my path. I began investigating whether the path variable was not available or different, either because I was running Ant via Eclipse run as administrator, or because Eclipse was spawning Ant in its own JRE.

After a lot of investigation, I discovered in my case this was a complete red-herring. On Vista, the subversion executable svn.exe requires an environmental variable SystemDrive that the SvnAnt task purposefully strips as described in this post by John E. Leon Guerrero. So the bottom line is that it is not possible to use the command line interface into Subversion from SvnAnt on Microsoft Vista.

John goes on to describe how to use the DLL distributed with Subclipse for the JNI interface by adding it to the path. This worked immediately for me.

To restate: You must use the JNI/Javahl interface into Subversion if you intend to use SvnAnt on Microsoft Vista.

As a side note, when debugging a possible Ant environment issue, either within Eclipse or from the command line, consider passing the ‘-diagnostics’ argument, which produces a nice output of the environment Ant is running in.

Hope this helps someone!

Using NTLM Authentication with CFHTTP

Anyone who has done a significant amount of work within company intranets in a microsoft network environment will at some point have lamented the lack of the NTLM support in CFHTTP and the administrator scheduled task and system probe functions.

Intranets within microsoft environments have always had a key advantage - that when webservers are set to use integrated security, no user logins are required. Clients using Internet Explorer are automatically authenticated, which is a usability and security benefit of immense value. This is due to NTLM authentication, which automatically secures HTTP requests when webservers or web hosted files are set to use integrated security. This is easily done on IIS and achievable on Apache as well. However the ColdFusion tag CFHTTP - the out of the box HTTP client - does not support NTLM authentication, which means that typically special measures have been required to allow HTTP to be viable communication and information gathering tool for ColdFusion servers in MS environments. This problem affects the HTTP requests sent by native ColdFusion Administrator functions such as Scheduled Tasks and System probes as well.

There are two common approaches to allowing HTTP in NTLM and integrated security environments:

• Use an alternative CFHTTP implementation. For example the CFX_HTTP5 custom tag supports NTLM and provides a couple of extra features as well. It is also possible to roll your own implementation using COM objects or .NET assemblies. Of course solving the CFHTTP issue does not help with CF Administrator Scheduled Tasks or System probes.
• Change the security around specific resources in an adhoc fashion to Basic or another less security setting. With many applications using a 'front-end controller' pattern this can mean creating a proxy page. The disadvantage of this approach the complication to your code base, the overall poorer security outcome and the fact that at times you do not have this level administration access to other internal resources you wish to connect to.
  

While installing Trac the other day (a long an drawn out process I should blog about) I was reminded of another, more elegant solution - Using a dedicated NTLM-aware proxy server. A dedicated proxy server can convert your CFHTTP basic authentication to NTLM and since it is dedicated it can be configured to only accept requests from your ColdFusion server machines. The proxy server required can be lightweight and some circumstances can easily share your ColdFusion server machine.

All ColdFusion HTTP services (CFHTTP, Scheduled Tasks and System Probes) support the definition of proxy server settings, so your end point URLs can be maintained and on the receiving end your can use the automatic authentication to determine with confidence the end user (the CF server) that is accessing the resource.

The specific NTLM-aware proxy server that I experimented with was a Python based server called NTLM Authorization Proxy Server. Other implementation surely exist however, which I'll endeavor to do some more research on and blog about here.

Overall I think the usage of an NTLM proxy service as I describe is a more elegant and consistent approach which minimises the impact of this unfortunate issue on your code base and your approaches to security. Let me know what you think.

Printable Web Pages with CSS

Many blogs and websites I view do not print well directly from the browser. Having printable pages is a good thing© for many reasons that all revolve around making your pages accessible to a wider audience in more situations. Besides, we like the radical notion that a browser's print button might actually be used now and then.

Designing your pages so that they print acceptably direct from the browser is not impossible, but you will find the ease will vary from site to site and the types of data that need to be printed. Flowing, textual data is easily printed; wide tables, inline scroll bars and pre-formatted text are not.

If you have developed your site with semantic mark up there may be an easy solution you have not considered. When importing stylesheets into your pages you have the option of specifying the media (output contexts) the stylesheet should be applied for. If you don't specify this, the browser assumes your stylesheet is valid in all situations, which can lead to undesirable outcomes (that is, completely unreadable printed web pages).

By specifying the media attributes of the stylesheet imports, you can control how the browser attempts to style printed output. Vast improvements in the output quality can often be achieved simply by not providing a stylesheet for print media and letting the browser defaults style your page.

Choose Where your Stylesheets Apply

How does this work? Lets assume you currently import a single stylesheet in your HTML. You might have done so using the following markup:

<link href="screen-style.css" rel="stylesheet" type="text/css">

When a user attempts to print your page directly from the browser, the browser attempts to use your styles in the rendered output. If this stylesheet was designed for online use, this can include CSS rules that are difficult to render in the fixed-width, fixed background colour environment of the printed page. The result can be terrible. A Possible Quick Fix - Disable Stylesheets for Printing

A possible quick and easy improvement is to turn off defined styling - relying on the browser's defaults - for non-screen media. We do this by being more specific in our link tag:

<link href="style.css" rel="stylesheet" type="text/css" media="screen">

For a list of recognised media types, refer to the media types W3C web page.

The media attribute specifies that this stylesheet is only suitable for use on a computer screen, so the stylesheet is ignored when the document is to be printed. Often this is provides a dramatic improvement.

How Does it Look?

For an example of page that uses this technique, open your print preview on this blog page.

UPDATE 30-May-2009: Since this blog is now on Blogger this is no longer true. Blogger needs smarter printing!

TODO: Ask Blogger to improve printing

Your mileage may vary, depending on the quality of the mark up you have used in your page. The order of your HTML markup, its semantic value and your use of CSS for screen layouts will all have a large effect on the readability of your page sans-stylesheet. (This is where you find out why best-practice rocks).

These factors and why impact stylesheet-less printing are described below:

Considered Source Order

Source order - the order in which items appear in your markup - becomes important, because the browser lays elements out in the order they appear in your HTML. Ideally, your markup is already ordered to optimise viewing in mobile devices or text browsers anyway. If it is not, you should consider changing it. CSS layout techniques can achieve a range of effects with any source order, so ordering your markup effectively should not be restrictive. You will want to consider placing elements like sidebars and vertical advertising at the bottom of your document, after the content.

CSS Layouts

If you are still using table based layouts you will find this tip is not very helpful. Without overriding the default styles with a stylesheet, tables used for layout appear as they would had they semantic meaning. CSS Layouts simply disappear on the other hand, generally serialising the page output. In the fixed width setting of the paper page, this is often the best way to prevent content being cut off at the edge of the page.

Semantic Markup

Semantic use of HTML has its advantages, and this is one of them. The default styles applied by browsers are generally highly readable, so if you have used lists, headers, blockquotes and other tags semantically in your document, it will still be readable without the stylesheet rules.

Stylesheets Specifically for Printing

So if option 1 is simply to suppress use of the a stylesheet for printing, the next option is to provide a stylesheet dedicated to printed output only alongside the screen-only sheet.

<link href="screen-style.css" rel="stylesheet" type="text/css" media="screen">
<link href="print-style.css" rel="stylesheet" type="text/css" media="print">

A more practical approach might be to selectively override rules from a master spreadsheet for print situations only. This drastically reduces the amount of duplicate code you need to maintain.

<link href="common.css" rel="stylesheet" type="text/css">
<link href="print.css" rel="stylesheet" type="text/css" media="print">

Many sites already separate layout stylesheets from typography. This can be especially useful to further minimise repetition:

<link href="typography.css" rel="stylesheet" type="text/css" media="screen,print">
<link href="layout.css" rel="stylesheet" type="text/css" media="screen">
<link href="print.css" rel="stylesheet" type="text/css" media="print">

Regardless of the approach, the differences you might want to implement in printed outputs might include:

• Hiding unwanted elements, like sidebars, forms and advertising
• Removing background and foreground images that would not print well
• Ensuring readable fonts with high-contrast font colours and point (pt) specified font sizes
• Removing or resetting fixed widths on elements
• Using advanced (and increasingly supported) CSS rules to insert URLs inline after hyperlinks, or special print-only messages about the context of the printout.

Having a print-only stylesheet is an effective way to retain some custom styles for printing, but this method does increase the amount CSS code that needs to be maintained, so it may not be appropriate for all situations.

Not the Answer for Everything

Some printing requirements will not be helped by these stylesheet methods. When printed outputs must have explicit pagination or wide areas of tabulated data, other approaches (such as providing a PDF) may be more appropriate.

Having said that, this approach to managing your print outputs is relatively easy to understand and can be provided very quickly to every page on a website. Printing web pages directly from the web browser is an easy operation for users of your sites and applications to understand, and generally does not break the flow of their browsing experience. For textual pages such as articles and blog entries, this approach is ideal.

Let the Users Know

If you do go to the effort of ensuring your page can be printed, make sure you let the users know. Years of unsuccessfully printing web documents has permanently dissuaded many users from ever even attempting to print a web page directly.

Letting the users know that the document can be printed directly can be as simple as providing a link which opens the print dialog, such as:

Print this document

I would suggest placing this link at the top and bottom of the page, especially if the page content stretches "below the fold".

In Summary

We can gain a lot of control over how our pages appear when printed directly from the browser, as we have seen. Using semantic, intelligently ordered markup, simply excluding our stylesheet from print operations can produce an effective result in many cases. For more control, a print-only stylesheet can be used. There may still be some pages that are difficult to master the printing of from CSS alone, but most textual content will print with excellent results. Remember to let the users know that they can print your page.

My Article on SitePoint

For anyone interested SitePoint have published an article I wrote titled Creating Scalable Applications with ColdFusion Components.

If you have any feedback or comments, let me know for next time.

Many thanks to Matt and Amy (from Sitepoint and Adobe respectively) who edited my work.

Brush Up on your Web Standards

ColdFusion developers need to know and keep up with Web Standards

For some time I have been thinking about blogging something to this effect.. but I have not had a blog (which is an impediment to this sort of thing).

Recently Richard Davies (which is a brilliant last name) blogged here about his own puzzlement at the lack of web standards talk within the CF community. I agree. The discussions simply do not seem to come up. After looking at some open source CF projects recently I am beginning to believe we should talk more about Web standards and how to used them in building web applications.

Note 30-May-2009: There was originally another page of commentary on this topic but it did not come across in the import process.

Welcome and an Introduction

The below welcome message seems dated today and not completely accurate. I leave it here for posterity!

Hello and welcome to my new blog: Fuel For Fusion. In this blog I intend to talk about (and hopefully contribute meaningfully to) development in the enterprise, with a technical emphasis on ColdFusion and web applications in general.

Let me introduce myself. My name is Ben Davies, and I have been working with ColdFusion in an enterprise environment for just over 6 years. I'm the Lead Systems Developer on the Vale Inco Goro Nickel Project, where I have had a fantastic opportunity to test myself and try new and exciting things.

I have ideas I would like to make a reality, so I have set up a consultancy to help fund some time for private projects. Consultancy based work is challenging but rewarding. I enjoy having control of my own professional development and getting more opportunities to see different businesses, workplaces, problems and solutions.

I'm very interested in software development. Or did you already get that impression?

I hope you enjoy my blog. I have learnt a lot from the blogs of others - especially in the past few months as I have made strong effort to stay up to date - and I hope I can contribute something back.

If you have any comments about feel free to leave them against the entry or contact me directly using the contact details on this page. Ciao!

Opening New Windows with the window.open() Method

I was reviewing the do's and don'ts of opening new browser windows using javascript when I came across this useful (and possibly ancient page about the the window.open() method in javascript and how it should be applied.

I am recording the page address here since it brought up many useful considerations that otherwise do not seem to get much attention in code examples.

http://jennifermadden.com/javascript/window1.html