Panthers pressed strongly in the first-half but their only points were a penalty goal from the boot of Jamie Benn.
They were foiled by a resolute Oldham defence, with Matty Embleton and Luke Varley being held up over the line.
Panthers were also their own worse enemies because several times the ball went to ground with the line wide open and it became apparent it was going to be one of those days.
St Annes led 6-2 at the break after Chris Carter's 30th minute try was converted by Paul Ashton.
Panthers made positional changes for the second-half but they came unstuck almost immediately as St Anne's latched onto a stray pass and Lee Forsyth scored a converted try to put them ten points ahead.
With less than ten minutes left, Panthers suddenly sprung to life.
A cross field kick from Dave Probert saw Embleton rise above everyone else to collect the ball and touch down.
Shortly afterwards, Matthew Bacon (pictured) darted over at the corner and Panthers were only two points behind.
But it turned out to be too little too late as Oldham managed to hold out in the dying minutes.
Chris Hall, Varley and man of the match Embleton put in big efforts for Panthers but it wasn't enough to rescue the Castleford side who will have to regroup as they face a spell of four successive away matches.