I've realized that it is an advantage if the polygon is convex. You don't need to determine if a point is inside the polygon, but clipping a concave polygon can result in more than one polygon, and if you can assume that the convex polygon has clockwise vertices then you can connect intersection points in the clockwise direction. Otherwise you would probably need to use a ray from one of the intersection points parallel to the side of the rectangle to decide which direction was the inside direction.
Instead of using two float[] arrays to represent the polygon, I would use a list of java.awt.geom.Point2D or some equivalent point class. I would also use a java.awt.geom.Rectangle2D to represent the rectangle instead of 4 float parameters. Rectangle2D already has a method for determining if a point is inside, but if I didn't have that because I was using a different rectangle class, then my first step would be to make that. Step two would be to make a method that takes two points and a rectangle, and assuming one point is inside and one point is outside, it returns the point at which the edge between the two points intersects the outline of the rectangle.
Then I would probably make a method that takes two points and a rectangle and returns a list of all the places where the edge between the points intersects the outline of the rectangle. That could be zero, one, or two places, but I would only use that method when both points are outside the rectangle, so the result would always be zero or two places.
I would create a private class for organizing my results, something like:
private final class Intersect {
public final int edgeIndex;
public final Point2D intersection;
public final boolean isInward;
public Intersect(int edgeIndex, Point2D intersection, boolean isInward) {
this.edgeIndex = edgeIndex;
this.intersection = intersection;
this.isInward = isInward;
}
}
Then I would write a method that takes a list of those, the polygon, and the rectangle, and returns a collection of lists of points. I would make use of a method that takes an Intersect, the polygon, and the rectangle and returns an instance of a private enum:
private enum Direction { CLOCKWISE, COUNTERCLOCKWISE }
It would do this by taking a line through the intersection point that is parallel to the outline of the rectangle and then iterating through the polygon to see how many times the other edges of the polygon intersect the line on either side of the intersection point. It should be even on one side and odd on the other side, unless we've done something wrong. Return the Direction of the side that is odd.
I would also want a method that takes a list of Intersects and sorts it into clockwise order around the outline of the rectangle. I expect the list of Intersects would start in either clockwise or counterclockwise order, but it is important to know which one, so a sorting is called for. (You might alternatively just compare the first two Intersects and then reverse the order of the list if necessary.) I would use java.util.Collections.sort to do the sorting, and I would define a java.util.Comparator something like this:
private final class OutlineOrder implements Comparator<Intersect> {
private final Rectangle2D rectangle;
public OutlineOrder(Rectangle2D rectangle) {
this.rectangle = rectangle;
}
@Override
public int compare(Intersect a, Intersect b) {
...
}
}
I'll leave the details of determining which Intersect is further clockwise to you.
Then all we have to do is go through the list of Intersects in clockwise order until we find the first inward Intersect, and we start building up a list of vertices from the polygon. Each time we get to an outward Intersect we add its intersection point to the list and determine the next inward Intersect by using that method that returns CLOCKWISE or COUNTERCLOCKWISE above. If the next one is clockwise then we just continue, but if the next one is counterclockwise then we end this list of vertices, add it to the collection, and start a new list of vertices at the next clockwise inward Intersect.
I hope that helps.