combine that with a companion OR You, having a high trainer skill (or multiple companions with high trainer skills) and you train Serjeants instantly. then, retake it with the 78 man army, it should be easy (now, at this point, you get over 200 troop limit right after claiming the fief) and go recruiting like crazy (Having big relations to two to three villages helps a lot, i get second tier troops instantly with +50 relations with villages) and you only need two to four of the villages to have high relations with them, in order to recruit 50 first tier troops on lucky go per one village. first, wait till the faction that you want a castle or a town from, goes to siege a town or a castle. Thats how i got enough troops to take my first real estate. One way, is to go and capture a castle that has just been captured (your army size will be tripled when you become a monarch) and it stays like that even though you lose the one castle or town you just sieged. In clash of kings (ive played it for ages) it takes a lot of time and effort, three to four times more time to have enough big army to support you.

Clash of kings castle upgrade