Jak znaleźć top 3 topper każdego przedmiotu w danej tabeli?

Możesz to zrobić za pomocą zmiennych.

select t.*
from (select t.*,
             (@rn := if(@s = subject, @rn + 1,
                        if(@s := subject, 1, 1)
                       )
             ) as rn
      from t cross join
           (select @rn := 0, @s := '') params
      order by subject, marks desc
     ) t 
where rn <= 3
order by t.subject, t.rn;