I remember having the same problem back when I was goofing off with OpenGL. I'm not sure how much it will help you, but I'll contribute this anyway. I wrote a brute-force method that allows me to rotate along any object's local axis, and then I can use that as the object's orientation dimensions.
Disclaimer: This is C code.
Further Disclaimer: This is before I learned how to write code properly.
/* 'orientation' is a 3x3 float matrix that contains a unit vector for each of the local z, x, y (in that order) axes relative to the origin. In my example, it represents the camera, and the camera looks out the z axis with the y axis representing its up vector). */
/* Rotate a number of degrees around axis 0 (z), 1 (x) or 2 (y) */
void cam_rotate(float degrees, int axis)
{
float matrixM[3][3];
float Xo, Yo, Zo, cosd, sind;
int nxt = (axis+1)%3;
float new_x,new_y,new_z, mag;
/* The vector we rotate around does not change. We store its values in smaller variables to make the code more legible. */
Xo = orientation[axis][0];
Yo = orientation[axis][1];
Zo = orientation[axis][2];
cosd = cos(degrees*3.1415/180);
sind = sin(degrees*3.1415/180);
/* Multiply one of the vectors by the transformation matrix representing the degree of transformation. */
matrixM[0][0] = Xo*Xo + cosd*(1-Xo*Xo);
matrixM[0][1] = Xo*Yo + cosd*(0-Xo*Yo) + sind*-Zo;
matrixM[0][2] = Xo*Zo + cosd*(0-Xo*Zo) + sind*Yo;
matrixM[1][0] = Yo*Xo + cosd*(0-Yo*Xo) + sind*Zo;
matrixM[1][1] = Yo*Yo + cosd*(1-Yo*Yo);
matrixM[1][2] = Yo*Zo + cosd*(0-Yo*Zo) + sind*-Xo;
matrixM[2][0] = Zo*Xo + cosd*(0-Zo*Xo) + sind*-Yo;
matrixM[2][1] = Zo*Yo + cosd*(0-Zo*Yo) + sind*Xo;
matrixM[2][2] = Zo*Zo + cosd*(1-Zo*Zo);
new_x = matrixM[0][0]*orientation[nxt][0]+
matrixM[0][1]*orientation[nxt][1]+
matrixM[0][2]*orientation[nxt][2];
new_y = matrixM[1][0]*orientation[nxt][0]+
matrixM[1][1]*orientation[nxt][1]+
matrixM[1][2]*orientation[nxt][2];
new_z = matrixM[2][0]*orientation[nxt][0]+
matrixM[2][1]*orientation[nxt][1]+
matrixM[2][2]*orientation[nxt][2];
mag = sqrt(new_x*new_x+new_y*new_y+new_z*new_z);
if (new_x == 1 || new_x == -1) { new_y = 0; new_z = 0; }
if (new_y == 1 || new_y == -1) { new_x = 0; new_z = 0; }
if (new_z == 1 || new_z == -1) { new_x = 0; new_y = 0; }
orientation[nxt][0] = new_x/mag;
orientation[nxt][1] = new_y/mag;
orientation[nxt][2] = new_z/mag;
/* Take the cross product of the first two matrices to reorient the third axis. */
nxt = (nxt+1)%3;
new_x = matrixM[0][0]*orientation[nxt][0]+
matrixM[0][1]*orientation[nxt][1]+
matrixM[0][2]*orientation[nxt][2];
new_y = matrixM[1][0]*orientation[nxt][0]+
matrixM[1][1]*orientation[nxt][1]+
matrixM[1][2]*orientation[nxt][2];
new_z = matrixM[2][0]*orientation[nxt][0]+
matrixM[2][1]*orientation[nxt][1]+
matrixM[2][2]*orientation[nxt][2];
mag = sqrt(new_x*new_x+new_y*new_y+new_z*new_z);
if (new_x == 1 || new_x == -1) { new_y = 0; new_z = 0; }
if (new_y == 1 || new_y == -1) { new_x = 0; new_z = 0; }
if (new_z == 1 || new_z == -1) { new_x = 0; new_y = 0; }
orientation[nxt][0] = new_x/mag;
orientation[nxt][1] = new_y/mag;
orientation[nxt][2] = new_z/mag;
} So your start direction is orientation[0] and up is orientation[2].
This code is unnecessarily verbose. If you replace my hard-coded math with matrix multiplication, then you get the same effect (I did this in Python using Numpy one time). The idea is to have 3 vectors all at right angles with one another to permit arbitrary rotation along the object's local axes at any given moment.
I certainly hope this helps you - I spent days trying to figure this out, and the effect was rather nice. If you can't understand my code, the entire thing's logic was derived from the OpenGL Red Book. Here's a link to the relevant page: http://fly.cc.fer.hr/~unreal/theredbook/chapter03.html
The sines and cosines came from a Wikipedia page on the subject http://en.wikipedia.org/wiki/Transformation_matrix