Threaded Indy News Reader - Felix John COLIBRI.

• abstract : a NewsReader which presents the articles sorted by thread and in a logical hierarchical way: who answers to whom. It uses the basic Indy NewsReader demo as a starting point
  
• key words : NNTP reader - news reader - threaded newsreader
  
• software used : Windows XP, Delphi 6, Indy 8.025
  
• hardware used : Pentium 2.800Mhz, 512 M memory, 140 G hard disc
  
• scope : Delphi 1 to 2006, Turbo Delphi for Windows, Kylix
  
• level : Delphi developer
  
• plan :
  
  • Reading News
    
  • The Threaded Indy NewsReader
    
  • Download the Sources
    

1 - Reading News

You will find plenty of newsreaders around, even some free, and with sources. Having very specific needs, we built our own, which originated in a Compuserve BBS (Bulletin Board: the ancestor of the newsgroups) project. Recently, we had to read SSL (Secure Socket Layer) protected newsgroups, but we had not implemented the SSL layer. Since Indy includes the SSL part, we decided to trade our old Windows Sockets engine for and Indy based one.

We first looked at the Delphi demo project:

    C:\Program Files\Borland\Delphi6\Demos\Indy\NewsReader

which is a very fine demo. We used it as the starting point of this project.

Why didn't we use the Indy demo "as is". There are a couple of reasons

• first articles are presented in a sequential manner. For daily overview of messages, that's fine. But for in-depth analysis, a presentation by thread (by message topic, or subject), the hierarchical presentation is obviously better suited
  
• we have added additional functionalities on top of the basic newsreader
  

2 - The Threaded Indy NewsReader

2.1 - The Basic Indy NewsReader

	drop a tIdNNtp Client on your tForm Set the Host (forums.borland.com) and Port (119) properties
	to fetch all the newsgroup from an Host, use: IdNNTP1.Connect; IdNNTP1.GetNewsgroupList(my_tStrings); where my_tStrings is any tStrings descendent (tListBox.Items, tMemo1.Lines, my_tstringlist etc)
	you will get an entry for each news group, like   borland.public.delphi.jobs 22692 22481 y borland.public.delphi.multimedia 7236 7046 y borland.public.delphi.non-technical 607311 600469 y borland.public.delphi.oleautomation 63539 62670 y borland.public.delphi.oodesign 26836 26153 y borland.public.delphi.opentoolsapi 12827 12641 y borland.public.delphi.reporting-charting 40787 40504 y
	once you have selected one of the newsgroups (borland_public_delphi_oodesign for instance), you can request all the messages within some id range: IdNNTP1.SelectGroup('borland.public.delphi.oodesign'); IdNNTP1.SendXOVER('26153-26836', my_message_tStrings); and ItNNTP1 will send the XOVER command to the news server, with the message id range. The headers will be placed in the my_header_tStrings tStrings descendent
	here is a part of the answer (truncated in string length and message count):   26809 | Why has OODB failed? | "William Egge" <begge@eggcentric.com> | Mon, 4 Dec 2006 26810 | Re: Why has OODB failed? | Michael Baytalsky <mike@contextsoft.com> | Mon, 04 Dec 200 26811 | Re: Why has OODB failed? | Franz-Leo Chomse <franz-leo.chomse@samac.de> | Mon, 04 Dec 200 26812 | Re: Why has OODB failed? | "Wayne Niddery [TeamB]" <wniddery@chaffaci.on.ca> | Mon, 4 Dec 26813 | Re: SmallTalk | "Peter Morris [Droopy eyes software]" <pete@NO_droopyeyes_SPAM. 26814 | Re: Why has OODB failed? | Andreas Dorn <adornno1@web.de> | Mon, 04 Dec 200 26815 | Re: SmallTalk | "Jarle Stabell" <jarle@remove_stuff_dlogikk_spam_kills_email.co 26816 | Re: Why has OODB failed? | "Wayne Niddery [TeamB]" <wniddery@chaffaci.on.ca> Tue, 5 Dec 26817 | Re: Why has OODB failed? | "Bob Dawson" <RBDawson@prodigy.net> | Tue, 5 Dec 2006 26818 | Onion | "Peter Morris [Droopy eyes software]" <pete@NO_droopyeyes_SPAM. 26819 | TTreeView.ValidateSelection not virtual nor dynamic?! | Ricardo Villela Coppola <coppola@c 26820 | Re: SmallTalk | "Peter Morris [Droopy eyes software]" <pete@NO_droopyeyes_SPAM. 26821 | Re: SmallTalk | "Bart" <@> | Fri, 8 Dec 2006 26822 | Re: SmallTalk | "Peter Morris [Droopy eyes software]" <pete@NO_droopyeyes_SPAM. 26823 | Re: SmallTalk | "Bart" <@> | Fri, 8 Dec 2006 26824 | Re: Why has OODB failed? | "John Herbster" <herb-sci1_AT_sbcglobal.net> | Fri, 8 Dec 2006 26825 | Re: Why has OODB failed? | "Harley Pebley" <harley_pebley@idahotech.com> | 8 Dec 2006 16:4 26826 | Re: Why has OODB failed? | Michael Baytalsky <mike@contextsoft.com> | Sat, 09 Dec 200 26827 | Re: TTreeView.ValidateSelection not virtual nor dynamic?! | "Bob Dawson" <RBDawson@prodigy We have truncated the length of each line, and replaced the Tab (Chr(9)) with a "|" A full header line will look like   26809   Why has OODB failed?   "William Egge"   <begge@eggcentric.com>   Mon, 4 Dec 2006 08:47:08 -0500   <457426d2$1@newsgroups.borland.com>   2156   39   Xref: newsgroups.borland.com borland.public.delphi.oodesign:26809
	to donwload a single message (article), we drop a tIdMessage on your tForm: request the message (Wayne Niddery's answer to William Egge in our case) using IdNNTP1.GetArticle(26812, '', IdMessage1); my_sting:=  TIdText(IdMessage1.MessageParts.Items[0]).Body.Text;
	if we display the body of the message in a tMemo, we would see (truncated):   William Egge wrote: > I am curious why OODB has failed to take hold.  It can"t be because of > needing compaitibility with other systems because an OODB should be > able to provide a relational interface for legacy applications. The model simply doesn"t work at the persistence level - one does not need  to store *objects* (which include behaviour) one just needs to store the  data of those objects. Storing entire objects as some kind of blob is counter-productive because it  cannot be effectively and efficiently queried, and it also inefficient since  not all of attributes of an object *should* be stored - one of the nice  things about objects is they can have derived, non-persistent attributes and  relations.

would have brought back the following text (truncated):

Therefore, using the Indy components has the following benefits:

• we do not have to wait for and assemble the different TCP/IP packets
  
• the status code (220) is analyzed for us
  
• the intersting parts of the header (FROM, SUBJECT etc) are placed in different properties of the component
  
• if the message is a multipart message, they are also separated in the tIdMessage.MessageParts.Items array. As I understand, Borland newsgroup Netiquette discourage using attachments (they are stored in a separate newsgroup), but they could be handy for some other Server
  

2.2 - Threaded NewsReader

• several topic messages are intermingled
  
• there is no indication to "who answers to whom"
  

You will notice that the "OODb" messages are mixed with the other messages ("Smalltalk" in our case)

To somehow represent the thread, we have to

• sort the messages by thread
  
• in each thread, display the hierarchical nature of the discussion
  

This is the purpose of the present project.

In fact, we presented already how to build the message tree structure in a detailed paper, with the algorithm we used, UML Class diagram and the UNIT which builds the tree. We are simply going to incorporate the CLASSes in the current project

2.3 - The Delphi Threaded NewsReader

• we removed the SSL parts, not required in our case
  
• we simplified the newsgroup tListView, and added a simple optional "Delphi" filter
  
• we removed the list of messages from a group
  

• we replaced the list of message by a thread list
  
• clicking on an Item of the thread list displays
  
  • either a tTreeView of the messages, and clicking on on of those fetches the content of the message
    
  • or we fetch all the messages of the thread and present them indented
    

2.4 - Mini User Guide

	compile and run the p_threaded_news_reader project
	the project is displayed
	connect the reader to your server by clicking "connect_"
	all available newsgroups are displayed, along with the currently available message count: Note that since we checked the delphi_ tCheckbox, our project only presents the Delphi newsgroups. We have no need for any C++ or other Java stuff. we also removed the "borland.public.delphi." prefix which does not add anything to the display and we removed the empty groups
	to display the messages from a given newsgroup, click on the target newsgroup
	the project sends the "XOVER" NNTP command to fetch the message headers:
	after 5 seconds (ADSL, 512K), they are here, and our U_C_NEWS_THREAD_LIST UNIT orders them by thread, and presents them, along with their starting date (month/year in mm-yy format):
	to the message from a thread on by one, select the "article_" tab from the right tNoteBook, and click on one of the thread in the middle tTreeview. In our case, we chose "UML Class Diagram From Source"
	the tTreeView displays all message headers in an indented fashion:
	clicking on any message in the tTreeview requests the message content and displays it in a tMemo:
	to view all the message from a thread, select the "articleS_" tab from the right tNoteBook, and click on one of the thread in the middle tTreeview. In our case, we chose "UML Class Diagram From Source"
	the tMemo displays all the messages in an indented fashion. Here are the first message, and the start of Gerrit Beuze's answer:

• Mr Remo Candeli is somehow hard with Model Maker. I still use it for our UML Trainings, when customers do not wish to use Together
  
• Gerrit Beuze from ModelMaker Tools answers very nicely and even provides links to Model Maker UML analysis tools
  
• to the list provided by Gerrit Beuze, I would also add the excellent, free, open source .PAS to UML Class diagram Ess-Model tool
  

2.5 - What's Next ?

Reading messages every morning is fine, but you cannot build a knowledge base unless you somehow save the text, and create either an indexing scheme, or a search engine.

Whenever I look for some information, I simply go to Tamarack an try to find some keyword that their Rubicon engine will use to locate relevant messages. The main reason I am using Tamarack, like so many of you, is that Tamarack has saved all the Delphi messages, and offers an Internet querying system to get the information out of this database.

What I would prefer is to download the archive, of, say, the last 5 years, and use our own Full Text Search engine which has a better requesting system (in my opinion). You have the full Pascal logic at your fingertips. To get tools other than ModelMaker which extract the UML Class diagram from source code we could ask:

I am not sure that Rubicon can handle such requests.

So why havn't we already implemented this machinery ? Well, our Delphi newsgroup archives are incomplete. When some customer projects dragged us into heavy programming, donwloading the news was often neglected. Maybe we will build an automatic tool which will do the downloading each morning, to avoid those gaps in our archives.

But this brings me to another pet idea of mine: why did Borland never offered the FULL archives as .ZIP files ? Are there some legal reasons (copyrights, maybe ?). Or disk space ? Or lack of interest from the developers ?

I mentioned it when I met some CodeGear executives, early October 2006, but have not received any answer so far. Maybe if many of you push for this .ZIP archive, and if the previous stuff has not yet been scrapped, we could get this fabulous knowledge base ? And if I can put my hands on the .ZIPs, I will publish the adapted search engine.

3 - Download the Sources

• threaded_news_reader.zip: the newsreader with all necessary units, bundled in a single .ZIP (32 K)
  

The .ZIP file(s) contain:

• the main program (.DPR, .DOF, .RES), the main form (.PAS, .DFM), and any other auxiliary form
  
• any .TXT for parameters, samples, test data
  
• all units (.PAS) for units
  

• are self-contained: you will not need any other product (unless expressly mentioned).
  
• for Delphi 6 projects, can be used from any folder (the pathes are RELATIVE)
  
• will not modify your PC in any way beyond the path where you placed the .ZIP (no registry changes, no path creation etc).
  

• create or select any folder of your choice
  
• unzip the downloaded file
  
• using Delphi, compile and execute
  

The Pascal code uses the Alsacian notation, which prefixes identifier by program area: K_onstant, T_ype, G_lobal, L_ocal, P_arametre, F_unction, C_lasse etc. This notation is presented in the Alsacian Notation paper.

As usual:

• please tell us at fcolibri@felix-colibri.com if you found some errors, mistakes, bugs, broken links or had some problem downloading the file. Resulting corrections will be helpful for other readers
  
• we welcome any comment, criticism, enhancement, other sources or reference suggestion. Just send an e-mail to fcolibri@felix-colibri.com.
  
• or more simply, enter your (anonymous or with your e-mail if you want an answer) comments below and clic the "send" button

  Name :
  E-mail :
  Comments * :

• and if you liked this article, talk about this site to your fellow developpers, add a link to your links page ou mention our articles in your blog or newsgroup posts when relevant. That's the way we operate: the more traffic and Google references we get, the more articles we will write.
  

4 - References

• the basic Indy NewsReader demo is in the Delphi Demo folder
  
• the construction of a thread tree is presented our the message tree structure paper, with the algorithm we used, UML Class diagram and the UNIT which builds the tree.
  
• Indy components are on Delphi's Palette, but you can find additional information, a knowledgebase and additional demos at the Indy Project web site
  

5 - The author