You'd have to use the EXISTS operator instead of the IN operator for that, IN will only work with single column results:
SELECT
courseID
, studentID
FROM
whatever W
WHERE
EXISTS (
SELECT
1 -- doesn't matter what you return here, even NULL is OK, the EXISTS operator is satisfied as soon as anything is returned
FROM
somewhereelse S
WHERE
S.courseID = W.courseID
AND S.studentID = W.studentID
)
Another option would be to do an inner join on the two columns, but that's not a subquery so beyond the scope of this discussion :)