Parsing RSS Feeds

Out of the box, the parser only understands XML tags that have the same name as HTML tags. So this example:

    import org.htmlparser.Parser;
    import org.htmlparser.filters.NodeClassFilter;
    import org.htmlparser.tags.TitleTag;
    import org.htmlparser.util.NodeIterator;
    import org.htmlparser.util.NodeList;
    import org.htmlparser.util.ParserException;

    /*
     *  RSS (RDF Site Summary - formerly called Rich Site Summary) is a method of
     * describing news or other Web content that is available for "feeding"
     * (distribution or syndication) from an online publisher to Web users.
     * RSS is an application of the Extensible Markup Language (XML) that adheres
     * to the World Wide Web Consortium's Resource Description Framework (RDF).
     * Originally developed by Netscape for its browser's Netcenter channels,
     * the RSS specification is now available for anyone to use.
     */
    public class ResourceDescriptionFrameworkSiteSummary
    {
        public static void main (String[] args) throws ParserException
        {
            Parser parser;
            NodeList list;

            parser = new Parser ("http://sourceforge.net/export/rss2_sftopstats.php?feed=mostactive_weekly");
            list = parser.extractAllNodesThatMatch (new NodeClassFilter (TitleTag.class));
            for (NodeIterator iterator = list.elements (); iterator.hasMoreNodes (); )
                System.out.println (iterator.nextNode ().toPlainTextString ());

        }
    }

    Will only find the TITLE tags, which may be what we want:

    Rank 1: Gaim (100% activity)
    Rank 2: Azureus - BitTorrent Client (99.9934% activity)
    Rank 3: eGroupWare: Enterprise Collaboration (99.9867% activity)
    Rank 4: WinMerge (99.9801% activity)
    Rank 5: phpMyAdmin (99.9735% activity)
    Rank 6: guliverkli (99.9668% activity)
    Rank 7: phpGedView (99.9602% activity)
    Rank 8: AMSN (99.9536% activity)
    Rank 9: dotproject (99.9469% activity)
    Rank 10: ScummVM (99.9403% activity)

    However, with some custom tags defined, it can handle the heirarchy of the XML:

    import org.htmlparser.Node;
    import org.htmlparser.Parser;
    import org.htmlparser.PrototypicalNodeFactory;
    import org.htmlparser.filters.NodeClassFilter;
    import org.htmlparser.tags.CompositeTag;
    import org.htmlparser.tags.Tag;
    import org.htmlparser.tags.TitleTag;
    import org.htmlparser.util.NodeIterator;
    import org.htmlparser.util.NodeList;
    import org.htmlparser.util.ParserException;

    /*
     *  RSS (RDF Site Summary - formerly called Rich Site Summary) is a method of
     * describing news or other Web content that is available for "feeding"
     * (distribution or syndication) from an online publisher to Web users.
     * RSS is an application of the Extensible Markup Language (XML) that adheres
     * to the World Wide Web Consortium's Resource Description Framework (RDF).
     * Originally developed by Netscape for its browser's Netcenter channels,
     * the RSS specification is now available for anyone to use.
     */
    class Item        extends CompositeTag { public String[] getIds () { return (new String[] { "ITEM" }); } }
    class Title       extends CompositeTag { public String[] getIds () { return (new String[] { "TITLE" }); } }
    class Description extends CompositeTag { public String[] getIds () { return (new String[] { "DESCRIPTION" }); } }
    class Link        extends CompositeTag { public String[] getIds () { return (new String[] { "LINK" }); } }
    class Guid        extends CompositeTag { public String[] getIds () { return (new String[] { "GUID" }); } }
    class PubDate     extends CompositeTag { public String[] getIds () { return (new String[] { "PUBDATE" }); } }

    public class ResourceDescriptionFrameworkSiteSummary
    {
        public static void main (String[] args) throws ParserException
        {
            Parser parser;
            PrototypicalNodeFactory factory;
            NodeList list;
            Item item;
            NodeList kids;
            Node node;
            Tag tag;
            String name;

            parser = new Parser ("http://sourceforge.net/export/rss2_projsummary.php?group_id=24399");
            factory = new PrototypicalNodeFactory (true); // empty
            factory.registerTag (new Item ());
            factory.registerTag (new Title ());
            factory.registerTag (new Description ());
            factory.registerTag (new Link ());
            factory.registerTag (new Guid ());
            factory.registerTag (new PubDate ());
            parser.setNodeFactory (factory);
            list = parser.extractAllNodesThatMatch (new NodeClassFilter (Item.class));
            for (NodeIterator iterator = list.elements (); iterator.hasMoreNodes (); )
            {
                item = (Item)iterator.nextNode ();
                kids = item.getChildren ();
                if (null != kids)
                    for (int i = 0; i < kids.size (); i++)
                    {
                        node = kids.elementAt (i);
                        if (node instanceof Tag)
                        {
                            tag = (Tag)node;
                            name = tag.getTagName ();
                            if (name.equals ("TITLE") || name.equals ("DESCRIPTION"))
                                System.out.println (tag.toPlainTextString ());
                        }
                    }
            }
        }
    }

    This isn't as pretty as it could be, but you get the idea:

    Project name: HTML Parser
    Project description: HTML Parser is a library, written in Java, which allows you to parse HTML (HTML 4.0 supported).
    It has been used by people on live projects. Developers appreciate how easy it is to use. The architecture is flexible, allowing you to extend it easily.
    Developers on project: 16
    Project administrators: &#60;a href=&#34;http://sourceforge.net/users/derrickoswald/&#34;&#62;derrickoswald&#60;/a&#62;, &#60;a href=&#34;http://sourceforge.net/users/somik/&#34;&#62;somik&#60;/a&#62;
    Activity percentile (last week): 98.3413%
    Most recent daily statistics (24 Jan 2004): Ranking: 251, Activity percentile: 98.34%,
    Downloadable files: 25615 total downloads to date
    Most recent daily statistics (24 Jan 2004): Download count: 19
    Mailing lists (public): 4
    Public mailing lists: htmlparser-developer, htmlparser-announce, htmlparser-user, htmlparser-cvs
    Discussion forums (public): 2, containing 110 messages
    Public discussion forums: Open Discussion, Help, htmlparser-user, htmlparser-developer
    Tracker: Bugs (1 open/158 total)
    Tracker description: Bug Tracking System
    Tracker: Support Requests (1 open/20 total)
    Tracker description: Tech Support Tracking System
    Tracker: Patches (0 open/0 total)
    Tracker description: Patch Tracking System
    Tracker: Feature Requests (2 open/10 total)
    Tracker description: Feature Request Tracking System
    CVS (8169 commits/809 adds)
    Most recent daily statistics (24 Jan 2004): Commit count: 0; Add count: 0 &#60;br&#62;&#60;a href=&#34;http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/htmlparser/&#34;&#62;[Web-based access to repository]&#60;/a&#62;

    Last edited on Tuesday, January 27, 2004  6:04:21 pm.