The custom categories feature is half-baked but I don't really intend to develop it further, so if it doesn't do what you need I'm sorry in advance; I hope you can find another solution. The only real 'actions' you can add to the HUD yourself are macros, or you can link to certain compendiums (Macro, RollTable, JournalEntry, or Playlist compendiums). Having said that, here is how you can do that.
(1) You can add new subcategories to the existing core categories (the HUD buttons that are always visible: inventory, spells, etc) by clicking on the category button. or
(2)If you want to add them in a new category, first click on the 'book' icon on the right side of HUD and enter the name of any new categories you'd like to add. There's not a lot of control as to where they appear but you can choose to place them on the right or the left of the core categories.
When you click on the category, you'll be presented with a dialogue that will suggest subcategories. These suggestions are just compendiums you have permission to view and are supported (see above).
If ignore the suggestions and just enter your own text, it will add an empty subcategory to the HUD category. This is probably what you want.
Click on the subcategory title and you can then add Macros to the new subcategory. These macros are drawn from the Macros Directory and the user should have permission to see them/run them. So any actions you want to add, first make a macro for it.
That's about it. If you add a compendium, clicking on the subcategory title will also allow you to filter the compendium entries.