So, this really is challenging in XSLT 1.0. You basically have to break it down into phases of sorting, grouping, then processing. In 2.0, you can do all of this in basically one pass.
The solution posted below, is broken into three steps. The first step is to take the input and sort them into the correct order. These are all the templates that run in no mode at all. We store that result tree in a variable, and use it as the input to the next part. Next we take the sorted result and divide them into "Groups" and wrap each group with a "Group" element. This was the difficult part that can be easily done in XSLT 2.0. These are the templates in the "Group" mode. We store that result in a variable tree, and use it as the input to the next part. This part take the sorted groups and generates an html output grabbing the first 3 agents of each sorted group. I'm sure there's a better way to do this, even in 1.0, but this illustrates the idea and steps you'll have to take to get there.
Input Document.
<agencies>
<agency count="133">GOV.NIH.XYZ.465.56</agency>
<agency count="42">GOV.USDA.1411.16.10</agency>
<agency count="55">GOV.FDA.1411.56.213</agency>
<agency count="60">GOV.FDA.1412.86.45</agency>
<agency count="129">GOV.NIH.ABC.245.21</agency>
<agency count="48">GOV.USDA.4412.345.08</agency>
<agency count="46">GOV.USDA.1411.345.00</agency>
<agency count="39">GOV.USDA.5512.16.12</agency>
<agency count="216">GOV.NIH.NCI.456.23</agency>
<agency count="32">GOV.USDA.1411.17.03</agency>
<agency count="60">GOV.FDA.4412.33.56</agency>
<agency count="59">GOV.FDA.4410.45.87</agency>
<agency count="51">GOV.USDA.5512.231.05</agency>
<agency count="55">GOV.FDA.1431.32.45</agency>
</agencies>
Transformation
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output indent="yes" method="html"/>
<xsl:template match="/">
<xsl:variable name="sortedList">
<xsl:apply-templates select="agencies"/>
</xsl:variable>
<xsl:variable name="groupedList">
<xsl:apply-templates select="$sortedList" mode="Groups"/>
</xsl:variable>
<xsl:apply-templates select="$groupedList/agencies" mode="getTop3"/>
</xsl:template>
<xsl:template match="agencies">
<xsl:copy>
<xsl:apply-templates select="agency">
<xsl:sort select="."/>
</xsl:apply-templates>
</xsl:copy>
</xsl:template>
<xsl:template match="agency">
<xsl:copy>
<xsl:copy-of select="@*"/>
<xsl:value-of select="."/>
</xsl:copy>
</xsl:template>
<xsl:template match="/" mode="Groups">
<agencies>
<xsl:apply-templates select="agencies" mode="Groups">
<xsl:with-param name="pos" select="'1'"/>
</xsl:apply-templates>
</agencies>
</xsl:template>
<xsl:template match="agencies" mode="Groups">
<xsl:param name="pos"/>
<xsl:variable name="groupKey" select="substring-before(agency[position() = $pos],concat('.',substring-after(substring-after(agency[position() = $pos],'.'),'.')))"/>
<xsl:variable name="matchedSetNo" select="count(agency[starts-with(.,$groupKey)])"/>
<Group>
<xsl:for-each select="agency[starts-with(.,$groupKey)]">
<xsl:copy-of select="."/>
</xsl:for-each>
</Group>
<xsl:if test="($pos + $matchedSetNo) <= (count(agency))">
<xsl:apply-templates select="../agencies" mode="Groups">
<xsl:with-param name="pos" select="$pos + $matchedSetNo"/>
</xsl:apply-templates>
</xsl:if>
</xsl:template>
<xsl:template match="agencies" mode="getTop3">
<html>
<body>
<h1>Top 3 Agencies</h1>
<br/>
<xsl:apply-templates select="Group" mode="getTop3"/>
</body>
</html>
</xsl:template>
<xsl:template match="Group" mode="getTop3">
<h2>
<xsl:value-of select="concat(substring-before(agency[1],'.'),'.',substring-before(substring-after(agency[1],'.'),'.'))"/>
</h2>
<br/>
<xsl:apply-templates select="agency[1]|agency[2]|agency[3]" mode="getTop3"/>
</xsl:template>
<xsl:template match="agency" mode="getTop3">
<xsl:value-of select="concat(.,'(',@count,')')"/>
<br/>
</xsl:template>
</xsl:stylesheet>
Output HTML Code
<html>
<body>
<h1>Top 3 Agencies</h1>
<br/>
<h2>GOV.FDA</h2>
<br/>GOV.FDA.1411.56.213(55)
<br/>GOV.FDA.1412.86.45(60)
<br/>GOV.FDA.1431.32.45(55)
<br/>
<h2>GOV.NIH</h2>
<br/>GOV.NIH.ABC.245.21(129)
<br/>GOV.NIH.NCI.456.23(216)
<br/>GOV.NIH.XYZ.465.56(133)
<br/>
<h2>GOV.USDA</h2>
<br/>GOV.USDA.1411.16.10(42)
<br/>GOV.USDA.1411.17.03(32)
<br/>GOV.USDA.1411.345.00(46)
<br/>
</body>
</html>
Good luck!