Drilling on an angle without a mill can be pretty tricky. As mentioned, cutting a key way, especially in brass, is pretty easy. Brass is a wonderful material to machine. The only problem, really, is it tends to grab unless the tool has a proper rake. This includes drills. The best option is to grind a relief on any drills used for machining brass. Alternately, one can simply drill in small stages, starting with a fairly small drill and moving up in small increments. This is especially true on a drill press, which probably does not have a semi-fixed spindle. On a lathe with a solidly set tail stock it is not quite as much of an issue.
A set screw hole does not have to be drilled with extreme precision . If you take care in finding the Top Dead Center of the cylinder along a radial line in the vice, a drill press should be easily accurate enough. Be sure to start the hole with a center drill, going very slowly. Check your work several times, and don't just pull down on the quill handle. Rather, poke gently up and down to get the hole started. Once started, all subsequent drilling operations will follow the original path faithfully, as long as you do not use too small a drill bit at the outset. If the depth of he hole is too much more than ten times the diameter of the drill bit, the drill can begin to wander as it moves down in the material.
I would not attempt CA adhesive unless the torque on this cylinder is going to be quite low.